Faster initial block download (5x faster)

[satoshi]

By making some adjustments to the database settings, I was able to make the initial block download about 5 times faster.  It downloads in about 30 minutes.

The database default had it writing each block to disk synchronously, which is not necessary.  I changed the settings to let it cache the changes in memory and write them out in a batch.  Blocks are still written transactionally, so either the complete change occurs or none of it does, in either case the data is left in a valid state.

I only enabled this change during the initial block download.  When you come within 2000 blocks of the latest block, these changes turn off and it slows down to the old way.

I built a test build if you'd like to start using it:

http://www.bitcoin.org/download/bitcoin-0.3.2.5-win32.zip
http://www.bitcoin.org/download/bitcoin-0.3.2.5-linux.tar.gz

These binaries also include Gavin Andresen's JSON-RPC HTTP authentication feature and the other important security improvements from 0.3.2.

I've been running a test over the last 24 hours that kills and restarts it randomly every 2-60 seconds (poor thing) while it's trying to do an initial block download and it's been fine.

There are no changes to the way it handles wallet.dat.  This change is only for blk*.dat and the non-critical addr.dat.  You can always delete blk*.dat if it gets screwed up and let it re-download.

[1714187226]

1714187226

[1714187226]

1714187226

[1714187226]

1714187226

[1714187226]

1714187226

[knightmb]

Are these for use with the test network you have setup or the current public network?

[NewLibertyStandard]

Are these for use with the test network you have setup or the current public network?

This is for the regular network. The test network doesn't have enough blocks to test it effectively. From what I can tell, satoshi's builds are always for the live network unless he specifies otherwise.

[knightmb]

Awesome, got some extra slow machines to wipe/test with then. Will be curious to see what it does on faster machines as well. 

[knightmb]

So far, awesome increase for the Windows client. Doing about 500 blocks / second using a private node. After the timer is done, I'll try it on a wild node out on the Internet to see if the speed is just as good.

[knightmb]

By making some adjustments to the database settings, I was able to make the initial block download about 5 times faster.  It downloads in about 30 minutes.

The database default had it writing each block to disk synchronously, which is not necessary.  I changed the settings to let it cache the changes in memory and write them out in a batch.  Blocks are still written transactionally, so either the complete change occurs or none of it does, in either case the data is left in a valid state.

I only enabled this change during the initial block download.  When you come within 2000 blocks of the latest block, these changes turn off and it slows down to the old way.

The first part of the 70k downloaded in about 20 minutes, the rest of the remaining 2000 blocks in about 7 minutes, so only 27 minutes from 0 to 100%, very nice! Is there a safety reason to stop within the last 2000 blocks or can it be tweaked to stop at remaining 500 blocks for example?

In case anyone was curious, average download speed was around 3 Mbps for each burst of download.

[knightmb]

Blocks download just as fast in the wild (about 3 Mbps), so a big thumbs up, this will certainly get new users up and running much quicker.

[satoshi]

Is there a safety reason to stop within the last 2000 blocks or can it be tweaked to stop at remaining 500 blocks for example?

Not really.  I'll change it to 1000 next time.

[knightmb]

Is there a safety reason to stop within the last 2000 blocks or can it be tweaked to stop at remaining 500 blocks for example?

Not really.  I'll change it to 1000 next time.

Either way, it's an amazing improvement over the older version! Great work!

I ran two side by side (using the current release, and your new faster one) and in the time it took the newer one to download (twice) after a wipe, the original was still only about a few thousand blocks into it's download. 

[lachesis]

Congrats! I ran with libeatmydata for the initial download I had to do after I accidentally wiped my blk*.dat and it helped a bit, but this sounds even better!

[Bitcoiner]

Nice; perhaps it's only necessary to immediately flush transactions that are "self-originated"; i.e. if you spent or received some money?

[jgarzik]

IMO it should be time-based ("1 day"), not an arbitrary number.

[knightmb]

No one mentioned anything about the Linux client, but I've tested it on mine, runs 500 blocks / second during download, everything else works great, no issues to report so far.