Normal people dont even use the client, they use an online wallet service. The client is for geeks.  I think the future of bitcoin is dependent on making lightweight clients. I am glad to see developement moving that direction. Lightweight clients will stuff suffer from having to get their initial block scan from somewhere. Either through a trusted service online, or your computer or device still downloading the entire chain, and throw away the 99.9% that isn't yours. Although, a lot of this can be avoided by making sure addresses are tagged with a first date and/or block number, so that your client doesn't have to download the entire chain to find its txs. Btw, my Linux/Ubuntu client was unusable in 0.3.24 because it would take days to update, even if it was just a couple hundred blocks behind. I just updated to 0.4.0 and now it catches up almost immediately. I downloaded most of the chain through 0.4.0 and it took less than a day. Obviously not ideal for the casual user, but it's better than it was before... |
|
|
|
NOTE: Please see: Armory Bitcoin Client which was forked from this project to be the most advanced Bitcoin client to date! This project remains as a useful set of pure-python tools. But Armory is a full Bitcoin client! I am releasing my Python/C++/SWIG code into the wild under the GNU Affero General Public License: PyBtcEngine on Github by etotheipiif you are in Linux/Ubuntu, you can easily compile and execute (Windows is available too, but it's a lot of work to get it compiling): The PBE block-explorer demo.In a nutshell: PyBtcEngine is a computational backend that could be used as a starting point for Python-based BTC tools & software. It does not include any networking code at all, but has a fairly complete set of everything else unrelated to networking. Most library components are heavily unit-tested, so there should be a fair degree of robustness built-in. This library enables you to read, scan and organize the entire blockchain, perform all ECDSA operations, evaluate most scripts, detect non-std scripts, collect balances and Tx lists for wallets/addresses, detect/handle blockchain reorganizations gracefully, and can even be used to test your blk0001.dat for bit-errors. And it can do all this ridiculously fast! (see timings below). In its current state it is perfect for an offline BlockExplorer (in progress), but could easily be expanded into other tools, or used for the backend of an alternative client. See below for a more-complete list of implemented features. Below, I have copied the "STATUS" section of the README which shows the current capabilities in each language (which are combined in SWIG): ********************************************************************************
* STATUS: Last Updated - 28 Oct, 2011
* Legend:
* _ not implemented
* . implemented but not tested
* + implemented and partially tested
* X implemented and tested
*
* C++ Python SWIG
* ---------------------------------------------------------------
* (01) Ser/Unser Block Objects X X X
* (02) Hash160/Hash256 X X X
* (03) Difficulty calcs X X X
* (04) Address Generation X X
* (05) Address Verify/Manip X X
* ---------------------------------------------------------------
* (06) BlkHeaders read/scan/org X X X
* (07) BlkHeaders reorgs X X
* (08) Blockchain read/scan/org X X
* (09) Blockchain reorgs X X
* (10) Blockchain verify integrity X X
* ---------------------------------------------------------------
* (11) NonStd Tx Detection + + +
* (12) Script pprint X X X
* (13) Script OP_CHECKSIG X X
* (14*) Arbitrary script eval X X
* (15) ECDSA Sign/Verify X X X
* ---------------------------------------------------------------
* (16) Address/Wallet tracking X X X
* (17) Scan blkchain for Tx X X
* (18) Scan blkchain for NonStd X X
* (19) Reorg w/ double-spend X X
* (20) Add new blockdata real-time X X
* ---------------------------------------------------------------
* (20) SelectCoins for tx X X
* (21) Tx construct given inputs X X
* (22+) Distr Proposals for multi-sig + +
* (23) Tx broadcast
* (24) Tx fee detect/calc/handle X X
* (25) Blockchain download
+ please see https://gist.github.com/1321518
* all scripts have been implemented, most of them are tested.
OP_IF/NOTIF/ELSE/ENDIF are the only codes not implemented yet.
******************************************************************************** The current implementation holds everything in memory, and so it takes up about 1.2 GB of RAM right now. I plan to improve this in the future, but my computer has 8GB so I'm not in any hurry to make it more lightweight. On the other hand, because of this, and my painstakingly-careful memory management, the library is extremely fast. Here's the timings, measured on a single thread of an AMD Phenom X4 840 CPU with 8GB of 1333 MHz DDR3. - Read entire blockchain into RAM: 5s
- Scan entire blockchain, collect headers/txs: 10s
- Organize and find longest chain: 0.5s
- Verify blkfile integrity: 2.5s
- Get balances/ledger for a set of addresses/wallets, from scratch: ~0.75s/wallet
Yes, you can load, organize and scan all 600 MB of blockchain, and find transactions for a given wallet in less than 20s. My careful memory management guarantees that there are virtually no extraneous copy operations at any step. As such, some of the code is a bit complicated, but no one can say it isn't fast! Most of the C++ code is documented in the base directory, in the file, Using_PyBtcEngine.README. There is also a ton of example/unit-testing code that will be critical for anyone wanting to use it. In particular, three files contains examples of nearly every available method: - (C++) BlockUtilsTest.cpp
- (Python) unittest.py
- (Together) testswig.py
Recent Updates (08 Dec, 2011): Development has been forked to Armory which will be used for a client will all sorts of new, innovative features. If you are looking for pure-python tools/code for Bitcoin, keep following PyBtcEngine, but otherwise switch to Armory. I should have finished GUI development and have an alpha client released by the end of the year! (multiple encrypted wallets, address import, watching-only wallets, multi-sig tx, and even more!) Recent Updates (16 Nov, 2011): - Added lots of new C++ features: secure binary data handling, AES encryption, ECDSA signing, and time-and-memory-bound key-derivation function! A wallet using this library can now set the target compute time and memory for the key-derivation function to decrypt the private keys (the Satoshi client only has time-bound KDF)
- Added elaborate SelectCoins algorithm, which actually works extremely well! Created a SelectCoins-solution evaluator, and then threw in a few simple, a few elaborate coin selection algorithms. And some random ones. The SelectCoins evaluator gives each of them a sequence of scores, weights the scores by user preferences (with a default), and then chooses the best solution. The idea is to throw in a ton of solutions, each one of which may be better for an initial starting condition. Whichever one is best for the moment will be used
- Started secure wallet format and serialization. Will be using a simple binary format for the wallets, with all data encrypted via AES-256, and using a key-derivation function that nominally takes 0.5s of computation and 8 MB of RAM on the users' computer. The timing calibration makes it difficult to brute-force a solution, and the memory requirement completely disarms GPUs from being able to help with such a search.
- Created BIP 0010 to proactively figure out how clients can deal with multi-signature transactions. The core concept is "Tx Distribution Proposals." As such, I have implemented TxDPs and actually made them the basis for all transaction operations, even for single-signer transactions. If the private key is on your computer, it will be signed and broadcast immediately. If not, it will give you the TxDP which can be signed by the offline computer without needing access to the blockchain. If multiple signatures are required, the TxDPs are easily copied ASCII blocks that to be include inline in emails or as attachments, and easily combined when multiple signatures are received.
I have done my best to make this code "usable," meaning well-formatted code and lots of comments. Unfortunately, Bitcoin is complicated, and so there's only so much one can do to make the code easier to comprehend. Feel free to offer recommendations for improving it -- but it is a lot of code, so any major refactorings will probably not happen unless you're volunteering. License:GNU Affero General Public License v3 (AGPL) for this project. The license was picked to allow users to use it for free if they plan to create more OSS, but require a dual-licensing negotiation if someone wants to use it in closed-source software. Please contact me if you're interested.
|
|
|
|
FYI: I have officially released my code under the AGPL v3. My thread about it is here: https://bitcointalk.org/index.php?topic=46485The example code shows you how to do everything you wanted (@BkkCoins), except for updating the blockchain file in real time. However, it only takes about 20s to reload and rescan the blockchain, so you could set your code to call BlockDataManager.Reset() then BDM.readBlkFile_FromScratch() again, periodically. |
|
|
|
I have spent much of the past two months developing all the utilities I needed to write BTC software in Python. As I quickly found out, python is dirt slow, and so in order to process the blockchain I made a super-optimized C++ layer which is pulled into Python through SWIG. The end result has been fantastic! And here I release it into the wild under the GNU Affero General Public License: PyBtcEngine on Github by etotheipiTo be clear: This is a computational back-end that could be used as a base for Python-based BTC software. It does not include any networking code at all. I have done my best to make this code "usable," meaning well-formatted code and lots of comments. Unfortunately, Bitcoin is f***ing complicated, and so there's only so much one can do to make the code easier to comprehend in this situation. Also, I'm not a software engineer: I'm a mathematician with an engineering background and generally write algorithms to go into software, not the software itself. I'm sure many people will disagree with some of my software design (such as some obvious places I should've used inheritance but chose not to for the sake of simplicity). Oh well. It's OSS now, so clone it and refactor it all you want. Library Details:Be aware that the current implementation holds everything in memory, and so it takes up about 1.2 GB of RAM right now. I plan to improve this in the future, but my computer has 8GB so I'm not in any hurry to make it more lightweight. On the other hand, because of this (and my painstakingly careful memory management), it is ridiculously fast. Here's the timings, measured on a single thread of an AMD Phenom X4 840 CPU with 8GB of 1333 MHz DDR3. - Read blockchain into RAM: 5s
- Scan blockchain, collect headers/txs: 10s
- Organize and find longest chain: 0.5s
- Verify blkfile integrity (merkle roots and leading zeros on header hashes): 2.5s
- Get balances/ledger for a set of addresses/wallets, from scratch: 5s
Yes, you can use this tool to scan your entire blockchain file, verify there's no errors in it, find the longest chain and find all transactions for a given address(s) in under 30s! Yes, all 600 MB of blockchain. One of the reasons it's so fast is because I copy all data once into a bulk, 600MB chunk of RAM, and then everything else is pointers to locations in this chunk, and frequently pointers to these pointers. It may be complicated, but no one can say it isn't fast! There is some complexity in the fact that the C++ code does not have any bignum or ECDSA capabilities, but the python library doesn't have any of it's utilities for scanning/maintaining the blockchain. So, the libraries must be combined to get a full backend, but I have a ton example/unit-testing code which should highlight how it is used. In particular, three files contains examples of nearly every available method: - (C++) BlockUtilsTest.cpp
- (Python) unittest.py
- (Together) testswig.py
Below, I have copied the "STATUS" section of the README which shows the current capabilities * STATUS: Last Updated - 01 Oct, 2011
* Legend:
* _ not implemented
* . implemented but not tested
* + implemented and partially tested
* X implemented and tested
*
* C++ Python
* -----------------------------------------------------------------
* (01) Ser/Unser Block Objects X X
* (02) Hash160/Hash256 X X
* (03) Difficulty calcs X X
* (04) Address Generation X
* (05) Address Verify/Manip X
* -----------------------------------------------------------------
* (06) BlkHeaders read/scan/org X X
* (07) BlkHeaders reorgs X
* (08) Blockchain read/scan/org X
* (09) Blockchain reorgs +
* (10) Blockchain verify integrity X
* -----------------------------------------------------------------
* (11) NonStd Tx Detection + +
* (12) Script pprint X X
* (13) Script evaluation +
* (14) Script OP_CHECKSIG X
* (15) ECDSA Sign/Verify X
* -----------------------------------------------------------------
* (16) Address/Wallet tracking X X
* (17) Scan blkchain for Tx X
* (18) Scan blkchain for NonStd X
* (19) Reorg w/ double-spend .
* (20) Add new blockdata real-time .
* -----------------------------------------------------------------
* (21) Networking/Sockets
* (22) Blockchain download
* (23) Tx construction
* (24) Tx broadcast
* (25) Tx fee detect/calc/handle
As a whole, PyBtcEngine can perform all the ECDSA operations, evaluate most scripts, detect non-std scripts, and should handle blockchain reorganizations gracefully upon adding blockdata (though, the reorg code hasn't been tested much). The last important thing for me to add before I start using instead of just developing it, is transaction construction and signing. All the pieces are there, I just need to write the code to collect available TxOuts from the C++ wallet, construct the tx packet, and apply the python ECDSA code to sign it. License:I naively picked the GNU Affero General Public License (AGPL) for this project. I have never developed OSS before, and thus I'm a newbie to the world of OSS licenses. I picked the AGPL because it seems to be more appropriate to networking-related code, and contains copyleft -- all derivatives works of this library must also be OSS with copyleft. If you believe I don't fully understand the licensing I have selected, please let me know. |
|
|
|
Testnet-in-a-box looks like a great tool. I'm going to have to play with that. You proposed an interesting method, but I'm not sure if it will actually work, unless I misunderstood how testnet-in-a-box works. Won't all of the wallets be using the same blk0001.dat file? If so, then the moment a wallet is "connected," it will read the blkfile and see the original transaction and reject the attempt to make a new, conflicting transaction. I probably don't totally understand and/or there's probably a way to hack it... but as you said, it's a lot of work. I might just try to hand-construct headers and transactions and inject them into a special test blk0001.dat file, since I don't actually need the header hashes to have leading zeros for the test. Maybe I should just pay someone else to do this... I'll pay a 5 BTC bounty for a small blk0001.dat file containing this unit test. It should resemble the regular blk0001.dat file: contain a sequence of block data with the same format: MagicBytes(4B)
NumBlkBytes(4B)
Header(80B)
numTx(VAR_INT)
Tx1
Tx2
...
TxN
I don't care how the blocks are constructed, and the headers don't have to have leading zeros. Preferably this blkfile would have 3 regular blocks to start, with a few different addresses and a few regular transactions. Then the next block will be 4a which will contain two transactions, one to be invalidated. Next will be block 4b which will have one tx using the same outputs as one of the tx in 4a, but to a different address. Then block 5 which will extend 4b and force a reorg. I'm sure I'll break down and do something like this myself, eventually, but I'd rather pay someone to do it for me |
|
|
|
EDIT: I added a bounty to this thread, but then did it myself. See 4 posts down.I'm working on making sure that my code will properly handle a blockchain reorganization, especially with a tx that is valid on a chain but then invalidated by a reorg that doesn't involve that transaction. What I really need is some header&block data that represents a blockchain reorg, and create a unit-test out of it. I found some invalid blocks in my blk0001.dat file, and started trying to construct a unit test based on that. The problem is, all the transactions in the invalid block were included in the replacement block, with the exact same Tx hash. Thus, I have no examples of a real double-spend to know if my code will properly find and handle invalidated transactions. I am left with either leaving that code untested, or having to construct a fake chain by hand -- I'm hoping someone else constructed something like this, since it seems like a bit of work.
|
|
|
|
At the moment transactions can be created just given the intended recipient's address and knowledge of the details of unspent TxOuts for which one is able to satisfy the scriptPubKey. Your client can create the transaction and you could potentially print it out and send it in a letter.
Could be useful in cases where only the recipient has full access to the blockchain and network, like POS situations. The buyer needs access to neither the Internet nor the blockchain, which could make a very light handheld wallet possible. The wallet device needs only create the transaction, keep its own accounts and be able to undo a transaction when the user knows it's not actualized. This is a great idea in concept, until you actually try using it. Your phone will have to occasionally synchronize with the network, because it needs to know if it received money/refill, and to know what transactions actually made it into the blockchain. Requiring the user to do the record keeping is a recipe for unusable software... the user always fails at this : forgets to click a button, clicks the wrong button, etc. If the phone has the wrong information, it will start trying to sign transactions with TxOuts that have already spent and you spend 10 minutes trying to figure out why 7-11 won't accept your transaction for a candy bar -- not to mention that you just created 10 new transactions while trying to figure it out, and now the your phone wallet is hosed. Or you phone tells you have no money because the you forgot to cancel a transaction that didn't actually go through, and now you can't remember which one it was. If you get it wrong, you end up with the other problem. Users are busy/clumsy/lazy and always breaks a system that requires any kind of regular maintenance/record-keeping. On the other hand, most phones now (smartphones, at least) have internet access, so periodic synchronization isn't too bad. But I do believe that the ability to use BTC from your handheld device is critical to the future of BTC. Therefore, it's critical that BTC is usable without the entire blockchain. But I think periodic synchronization to find out the "truth" state of your wallet will be required. (EDIT: synchronization doesn't mean storing the whole blockchain... just occasionally connecting to some peers and downloading new blockdata and only keeping what's relevant) |
|
|
|
I'm not really sure what context you're talking about. The "standard" use-case for Bitcoin is address-to-address transactions requiring the 20-byte hash of the public key of the recipient. This is what Satoshi referenced in his paper when talking about the future and lightweight clients. If you don't have at least this much information, you're not going to be using standard scripts to send money without the whole blockchain. If this isn't acceptable, you shouldn't be using something like FirstBits in the first place (which, btw, I don't know for sure, but I'm assuming it shortens a BTC address by only keeping the first X bits of the recip address) If you want to go outside the box and use something else, you could make this work with non-standard transactions: I believe you could use OP_LSHIFT/OP_RSHIFT in the TxOut script if all you have is the first few bits (I'm pretty sure this works) -- i.e. create the script to check only the first X bits of the publickey hash match the data in the TxOut script . Something like OP_DUP
OP_HASH160
100
OP_LSHIFT
<Right60BitsOfRecipAddressWith100ZeroBitsAtEnd>
OP_EQUALVERIFY
OP_CHECKSIG If you're asking about the necessity of future BTC protocols to support not holding the whole blockchain, I would argue that it is critical. Any degree of success in BTC will lead to blockchain files way too big for users' computers, and it's even currently too big now for smartphones--which are also critical to the BTC success. I think we'd be burning a huge bridge if we allowed "standard" to include any procedure requiring the entire blockchain to operate. EDIT: I just notices OP_LSHIFT and OP_RSHIFT are disabled, so the above script wouldn't work |
|
|
|
|
I won't be able to help you, then. When I started working on the tool, I never had any intention of merging it with the main client. It's eventually going to be a completely separate client, but at the moment it's just a utility library that can scan the blockchain very fast and pick out relevant information.
I never imagined a system in call in the main client would ever be acceptable in the master branch. But it sounded like you were willing to accept a special build for your own purposes.
|
|
|
|
Is it in something other than C++?
The core components that do all the blockchain scanning is in C++. For everything you have mentioned in bounty #2, it would all be C++ code. Anything requiring a bignum library or ECDSA would have to be done with python code which imports said C++ code via SWIG. But the C++ code works as a standalone "BlockUtils" modules, and compiles both in Windows/MSVS2010 and Linux (the swig stuff only works in Linux, at the moment). I just have to create an example main() that does what you're looking for. |
|
|
|
|
Re: Bounty 2
I recognize you're looking for these features to be implemented in the client itself, but I have an alternative tool ready to go, if you are interested. It reads the entire blkchain file in 10s, organizes it and finds the longest chain in about 1s. It would be trivial to delete and rewrite the blockchain without the orphan blocks (haven't done it yet, but I'm sure it's in under a minute). Also, it can check the integrity of the blk0001.dat file in about 4s: verifies that all headers have hashes with leading zeros and that all merkle trees match the headers. If there are any inconsistencies, it will tell you exactly which block it is in.
Secondly, the tool will also, after the blockchain is loaded, scan for all instances of a 20-byte address and report a list of blocks in which it appears, with the capability to query any other related information. By default, it converts the public keys in coinbase-related transactions to the 20-byte address. The only thing required for me to do would be to convert that list of Txs/TxOuts/TxIns into whatever format you're looking for. When you populate a BtcWallet object with 20-byte addresses, it can find all relevant transactions in about 5s.
I could put together a special main()/binary for you that would do exactly what you're looking for. Perhaps you could then even recompile the main client with a button that makes a system call to execute this binary for scanning, cleaning, searching and indexing the blockchain.
Unfortunately, this is completely disjoint from the main client codebase, so there's really no way to mix them otherwise. If this isn't acceptable, I understand.
|
|
|
|
BkkCoins, I was thinking about what you said you wanted the library for. I realized that I didn't have a good way to find the sender of a given input: the sender is not always identified in a TxIn, so you have to use the BlockDataManager to go find the TxOut. Anyways, I squashed a bug and added some methods to make this a lot easier, definitely worth updating if you already checked it out. (and if you prefer this over genjix's tool... I'm sure his sql database would be easy to access in python, too). Here's some sample python code to get you started with my code if you plan to use it (this is scattered throughout testswig.py, but I extracted the important parts). from sys import path
path.append('..') # I'm running this from the cppForSwig directory
from pybtcengine import *
from datetime import datetime
from BlockUtils import *
# Create the BlockDataManager, scan the chain, organize the block headers
bdm = BlockDataManager_FullRAM.GetInstance()
bdm.readBlkFile_FromScratch('../blk0001.dat') # point this to your blkfile!!!
bdm.organizeChain();
# At this point all blockheaders and txs are loaded into memory.
# Can access instantaneously by hash or by height
# Get block 100,014 because it's got 6 diverse Tx
someBlk = bdm.getHeaderByHeight(100014)
# A helper method to convert header times to a pretty format
def unixTimeToFormatStr(unixTime, formatStr='%Y-%b-%d %I:%M%p'):
dtobj = datetime.fromtimestamp(unixTime)
dtstr = dtobj.strftime(formatStr)
return dtstr[:-2] + dtstr[-2:].lower()
# Another helper to convert raw, 20-byte address values to Base58
def hash160ToAddr(hash160):
b = PyBtcAddress().createFromPublicKeyHash160(hash160)
return b.getAddrStr()
# Going to print senders, receivers
print 'TxList for block #', someBlk.getBlockHeight()
topTxPtrList = someBlk.getTxRefPtrList()
print 'NumTx:', len(topTxPtrList)
for txptr in topTxPtrList:
# We print big-endian because we like to to be able to put it in BlockExplorer to verify
print '\nTx:', binary_to_hex(txptr.getThisHash().toString(), BIGENDIAN)[:16],
# Each Tx has a pointer to the header of the block it's included in
blkHead = txptr.getHeaderPtr()
print 'Blk:', blkHead.getBlockHeight(),
print 'Timestamp:', unixTimeToFormatStr(blkHead.getTimestamp())
# Print the TxIns
for i in range(nIn):
txin = txptr.getTxInRef(i)
if txin.isCoinbase():
print '\tSender:', '<COINBASE/GENERATION>'.center(34),
print 'Value: 50 [probably]';
else:
print '\tSender:', hash160ToAddr(bdm.getSenderAddr20(txin).toString()),
print 'Value:', coin2str(bdm.getSentValue(txin))
# Print the TxOuts
for i in range(nOut):
txout = txptr.getTxOutRef(i)
print '\tRecip: ', hash160ToAddr(txout.getRecipientAddr().toString()),
print 'Value:', coin2str(txout.getValue())
You can probably construct your BTC webs pretty easily by combining calls from above. If you have problems, make sure you're passing binary forms, not hex, and whether you converted to or from a BinaryData object. All python methods require binary python strings. All C++/SWIG methods require BinaryData objects (create with BinaryData(pyBinaryStr) and back via bindata.toString()). -Eto P.S. -- I also cleaned up some code and put most methods in the .cpp files so you can scan the .h files easier to find the methods you're looking for. |
|
|
|
Gavin, that's a very good idea. I have never even heard of mmap(), but it sounds like a very useful tool. In fact, I had some projects at work where this could've been very useful. Why didn't you tell me about this sooner?! (and apparently it works for interprocess communication? also useful!) At the moment, I think I have plenty of time before this becomes a necessity, but I'll definitely keep it in mind as work to expand it. Though, in the long run I plan to convert/fork it into a headers-only-but-save-my-own-tx-data implementation. |
|
|
|
|
Yes, I am very pleased with its performance. It uses about 1.2 GB in RAM to hold everything, and it's speed comes from the idea of copying the entire blockchain into RAM in a single copy operation (120MB/s for my HDD), then using only references and pointers to locations in that chunk of data for everything else--removing the need for extraneous copy operations.
At the moment, this FullRAM implementation was the easiest way for me to do everything (and also the fastest), and perfectly okay for at least a year (at the moment, it's for my own use and I have 8GB of RAM). So I implemented it with BlockObjRef objects to reference RAM, but can be later updated to reference file locations instead of RAM. Then when the blockchain is too big, I can do my initial scan in pieces, and leave the bulk of the data on the disk, retrieving it on demand. The maps of headers and tx hashes/refs will fit into RAM no problem for a very long time.
Bear in mind there is some learning curve to the way I organized things. But I don't know any full BTC implementation that would avoid this (BTC is complicated...). Everything is held in maps of pointers/references -- you can get from any one piece of information to another, it just may take a couple hops through memory. Luckily, SWIG handles the pointers/references very well. I haven't played with it too much (just got it working yesterday), but so far I haven't found any issues with it.
I was hoping to open-source this project eventually, but hadn't decided on it yet until I read your post. So it's probably short on documentation. But I'll be working on that in the near future. Don't be afraid to ask questions, and if there's some functionality it's missing, please let me know so that I can add it--whatever it is should probably be there anyway...
|
|
|
|
Funny you bring this up. I just got my Python/C++/SWIG implementation of blockchain scanning in place, yesterday. You can access it here. It successfully reads the entire blockchain into RAM (while the blockchain is still small enough to fit there), populates maps of blockheaders/hashes and txs/hashes, and will even do a fresh scan of the entire wallet finding all the txouts/txins and calculating balances. The python library has the ECDSA signing, verification and address calculations, but is not very good at scanning the blockchain which is why I did it in C++ and pulled it into python with SWIG. I just finished some testing last night: I can do all the scanning, blockchain organization, indexing, and wallet balances/unspenttxouts from scratch in about 30s (which is pretty good considering the blockchain is currently 600MB+). EDIT: I misspoke when I said "wallet": I'm not talking about the official BTC wallet files, I'm talking about creating a C++ BtcWallet object filling it with 20-byte hashed-publickey addresses. All the tools are there and mostly tested. For now, there is a "unittest.py" file which should demonstrate most of the python code that is there, BlockUtilsTest.cpp to see how to use the C++ code, and testswig.py for examples on mixing the two. The makefile has a "make" for making the C++ code, and "make swig" to run swig and compile the shared object. This is all in Linux, but I have run and tested the C++ code (alone) in Windows XP with MSVS 2010. This is what I was looking for 2 months ago and decided to do it myself. Holy hell has it been educational! -Eto P.S. - It's funny you bring this up, because my next goal was to write a PyQt application for exploring the blockchain. |
|
|
|
|
This specific system doesn't have to be the "one." I just like the idea of having the ideas structured in this way and allowing people to comment and vote. This is, assuming it's whitelisted access in some way to bring up the signal-to-noise and avoid it turning into another incarnation of the forums (but yes, you already said that).
It looks like the folks using this have already been using "gist", but just making it a bit more formal with BEP/BIP numbers to reference them.
|
|
|
|
|
I don't agree that less formalism is the answer. It's the answer to getting less-serious people involved...
But let's stop here and get back on topic: I like the idea informer you set up. It's a slick, professional-looking interface. I also like the topics you seeded it with. I will start adding comments.
Genjix, I apologize for letting this thread go a bit off-topic. But I do think what Alex set up is similar to what you were originally suggesting in this thread... is it not? Or were you thinking more about a wiki-style system?
|
|
|
|
|
Bitcoin is not just the original software created by Satoshi. It's all the future design of the Bitcoin software, as well as all the variants that will be built among all participating parties. Satoshi is a beast to be able to have created and implemented the original idea by himself... but it's going to take the collaboration of a lot of people to expand it and develop it for the real world.
Your argument is like saying Relativity isn't all that complicated, it was created by one guy! Sure, the original concept was proven by Einstein, but it took thousands of other smart people to find all its applications and make it useful beyond its original incarnation.
We don't need to harp on this any more. I'm just pointing out that it's a huge mistake to somehow think Bitcoin is "complete." We need other people and businesses to get involved if it's really going to succeed, and they can't unless there's a more-formal environment for learning about Bitcoin and suggesting improvements. Any system that requires going to a forum to get the information you need is not mature enough for "the real world."
|
|
|
|
My advanced apologies to Theymos... (I don't mean to DDoS blockexplorer, but I'm posting this anyway...)I was playing with my blockchain scanning tools. I decided to see what was the most "active" address in the blockchain, by most number of TxOuts. What I found was that there's a pretty big jackass out there executing thousands of pointless transactions. Worst of all, he's doing it without any transaction fees.I am going to post a few addresses here but I ask for Theymos' sake you don't try searching blockexplorer for them. Click on the links below to see a sample transaction, but don't try to follow them: they have so many tx that I got the following error after waiting a few minutes for the page to load: ERROR: Address ledger is extremely large. Contact me if you really need the data. This guy perpetually is submitting transactions with 10-50 inputs of 0.01 or 0.1 each between a few addresses. The address with the most activity is 159FTr7Gjs2Qbj4Q5q29cvmchhqymQA7of with 16,850 TxOuts spread out among 8500 transactions. Almost all of them go to 1E29AKE7Lh1xW4ujHotoT4JVDaDdRPJnWu. Here's an example block with this address--about Tx #20.These coins get juggled between lots of different addresses constantly. For instance, they are linked to another pair of addresses with a ton of activiity: 1AKW1jFcv49EN4g87BN7BdcyztQydihWUG constantly sending dozens of inputs of 0.01 to 1BSTJU32hFj7Uda3qbYCJ9pXXFG6xn9bBX. This block has 8 tx with 10 inputs each of this type - about Tx#20-30. Clearly, whoever this is, is trying hard to bloat the chain and test the limits of the network. For the first time, I feel better that the flood-defense protocols are strict, because I bet he could be doing a lot more damage without it. But I wonder if there was more the client could be doing to prevent this...? This guy is probably responsible for many tens of MB of space used on my HDD, and it'd be nice if it was costing him something to do this. Until it does, I imagine he's going to continue eating my harddrive space...
|
|
|
|
|
I'm talking about my personal client development. I can set NODE_NETWORK to whatever I want... though it sounds like 0 or 1 are the only options that are meaningful.
|
|
|
|
|
Show Posts
