100 (25+25+40+10) BTC Bounty: Blkindex.dat Rebuild Function & others

[casascius]

STATUS: AVAILABLE

I am offering a 25 BTC bounty, payable by me, to the first person who does the following:

Create a patch to the most recent version of Bitcoin, to offer the ability to reindex the block chain, as follows:

• A reindex should occur upon bitcoin startup if --reindex is supplied as a parameter.
• The patch should read blk0001.dat, and copy only the longest chain to a new file, discarding any orphan blocks below the last hardcoded checkpoint, as well as anything that is obviously corrupted.  (then delete blk0001.dat and rename the new file into its place)
• Rebuild blkindex.dat from scratch based entirely on the contents of the new blk0001.dat file.
• Re-creation of blkindex.dat should occur automatically, any time bitcoin is started and blk0001.dat is found but blkindex.dat is missing.  (In this case, if --reindex is not supplied, blk0001.dat need not be rewritten).

To claim the bounty, the patch needs to be submitted to github and test successfully.  Not necessarily to the main Bitcoin client source, but just where everyone can get at it.  (either as a pull request or as a fork)

- - - - - - - - - - -
SECOND BOUNTY (ANOTHER 25 BTC):

if the reindex function does the following:

In addition to adding all the usual data to blkindex.dat, also add an optional index that, given the binary form of a bitcoin address, returns a list of all the blocks where a reference to that bitcoin address can be found.

• This index shall not exist by default, unless the user asks for it to be built (e.g. --reindex full).
• There needs to be a function in the code that allows the index to be queried.  (if the index doesn't exist, then it's OK for this function to simply report the index doesn't exist.)
• Pubkeys appearing in transactions (e.g. coinbase) need to be converted to a binary bitcoin address before being added as keys to this index.
• The index does not need to be concerned with whether the reference is to a standard transaction, nonstandard, txin, txout, whatever.  Just, if there's a reference of any kind within a transaction, make an entry.
• The index needs to be updated each time a new block is written to disk.

Same rule: it must be on github and testable for the bounty to be payable.

- - - - - - - - - - -
THIRD BOUNTY (ANOTHER 40 BTC... TOTAL SO FAR 90 BTC)

Implement the sweepprivkey function as I have proposed in the Wiki.  https://en.bitcoin.it/wiki/Sweepprivkey

The sweepprivkey needs to be able to take advantage of the index that is the subject of the second bounty, so that it completes quickly (typically under 2 seconds).  If the index does not exist, it may (at your option) either report failure, or conduct a scan of the block chain if needed.


- - - - - - - - - - -
FOURTH BOUNTY (ANOTHER 10 BTC... TOTAL 100 BTC)

Add the sweep function to the user interface.  The sweep function should only be visible if the extra index is built.

The sweep should have its own dialog.  Upon filling in a bitcoin address and pressing Enter, if the bitcoin address has funds, a "YES/NO" dialog box should pop up, tell the user the amount of available funds, and then ask a YES/NO, "do you want to sweep".  If the bitcoin address is invalid or has no funds, the only choice should be "OK" to dismiss the dialog box saying so, not yes/no.

If the user clicks yes, a transaction should be immediately broadcast that sweeps the funds into a new wallet address.

[1714065594]

1714065594

[1714065594]

1714065594

[etotheipi]

Re:  Bounty 2

I recognize you're looking for these features to be implemented in the client itself, but I have an alternative tool ready to go, if you are interested.  It reads the entire blkchain file in 10s, organizes it and finds the longest chain in about 1s.  It would be trivial to delete and rewrite the blockchain without the orphan blocks (haven't done it yet, but I'm sure it's in under a minute).   Also, it can check the integrity of the blk0001.dat file in about 4s: verifies that all headers have hashes with leading zeros and that all merkle trees match the headers.  If there are any inconsistencies, it will tell you exactly which block it is in.  

Secondly, the tool will also, after the blockchain is loaded, scan for all instances of a 20-byte address and report a list of blocks in which it appears, with the capability to query any other related information.  By default, it converts the public keys in coinbase-related transactions to the 20-byte address.  The only thing required for me to do would be to convert that list of Txs/TxOuts/TxIns into whatever format you're looking for.  When you populate a BtcWallet object with 20-byte addresses, it can find all relevant transactions in about 5s.

I could put together a special main()/binary for you that would do exactly what you're looking for.  Perhaps you could then even recompile the main client with a button that makes a system call to execute this binary for scanning, cleaning, searching and indexing the blockchain.  

Unfortunately, this is completely disjoint from the main client codebase, so there's really no way to mix them otherwise.  If this isn't acceptable, I understand.

[casascius]

Is it in something other than C++?

[etotheipi]

Is it in something other than C++?

The core components that do all the blockchain scanning is in C++.  For everything you have mentioned in bounty #2, it would all be C++ code.  Anything requiring a bignum library or ECDSA would have to be done with python code which imports said C++ code via SWIG.

But the C++ code works as a standalone "BlockUtils" modules, and compiles both in Windows/MSVS2010 and Linux (the swig stuff only works in Linux, at the moment).  I just have to create an example main() that does what you're looking for.

[casascius]

The purpose for this index would be to implement things in the client that depended on it, that otherwise would require a minutes-long (and growing) scan of the blockchain.  Unfortunately, this would have to be code that would be acceptable for inclusion into the main client.  The developers are unlikely to accept the requirement for a system call into another binary.

However, not only would bignums and ECDSA already be available within the codebase of the client, they would probably be unnecessary for the sake of building this index.  All the index needs to be is something that says address X appears in blocks A,B,C, and would need to be kept up to date each time a new block was saved.

[etotheipi]

I won't be able to help you, then.  When I started working on the tool, I never had any intention of merging it with the main client.  It's eventually going to be a completely separate client, but at the moment it's just a utility library that can scan the blockchain very fast and pick out relevant information.

I never imagined a system in call in the main client would ever be acceptable in the master branch.  But it sounded like you were willing to accept a special build for your own purposes.

[casascius]

This bounty will probably get awarded sooner rather than later: I am seeing significant progress toward earning it at github: jarpiain/bitcoin... way to go!