Blockchain synchronization speed improvements incoming

[gmaxwell]

I thought people would be excited to hear that I finally got around to really investigating why the initial chain sync-up was so slow (at least for some users) and that I found an easily fixed problem.  

It's well known that the reference software does a lot of unneeded synchronous disk IO during syncup (in addition to the large amount of needed async IO), so syncup is expected to be slow right now on systems without fast disks... but even on systems with super fast disks and CPUs, the syncup is still quite slow. People frequently make all kinds of wild network related suggestions that they think will help syncup which I've been discounting: https://bitcointalk.org/index.php?topic=53389.msg640824#msg640824, but I hadn't had a chance to sit down and work on any of the actual causes of slowness.

Well, it turns out that when the wallet encryption was introduced some usage of mlock() was added to keep private key data out of swap. This is a good thing, but the mlock is used naively which is bad because it's slow (results in a TLB flush). That wouldn't matter too much, except a side effect of the change was to mlock all the memory used by a very common data type used all through bitcoin. This is pointless because most of the usage doesn't contain private data. (and may have even adversely impacted security of the encrypted wallets on systems with limited mlockable memory)

This was somewhat hard to track down— it doesn't show up in the oprofile cycle counter (it would have shown up in TLB flushes, but thats only obvious in retrospect) and didn't show up in some other performance tools like valgrind/callgrind (which is too simplistic an emulation to know mlock is slow). I eventually caught it in ltrace output.

There is still some ongoing discussion about the best path to fix this, but the performance improvement at least on some nodes is quite significant. YMMV, if you're not on a super fast SSD this improvement will be much smaller (at least until the excessive fsync usage is resolved), and my AMD AMD Phenom(tm) II X6 1055T  with kernel 2.6.35.14-106.fc14.x86_64 might mlock slower than some others.

This should also speed up the client generally, though its especially obvious for the chain syncup.

Enough talk, How about some pictures:

The blue and red lines are variants of this fix, green is the stock client.

(note the log scale)

There is more discussion on the pull request (https://github.com/bitcoin/bitcoin/pull/740)

[1715244780]

1715244780

[1715244780]

1715244780

[Mushoz]

This is amazing, great work!  Do you happen to have any idea when this is going to get merged into a release of the mainclient? Thanks!

[Transisto]

Awesome ! I couldn't understand why syncing the blockchain was taking longer than downloading a 30gb torrent.

[gmaxwell]

This is amazing, great work!  Do you happen to have any idea when this is going to get merged into a release of the mainclient? Thank!

It was merged about an hour ago, and will ship in the next release of the client.

[finway]

Thank you for your contribution, gmaxwell.

BTW: Is it safe? I mean, is zero-after-free still there ?

[gmaxwell]

Thank you for your contribution, gmaxwell.

BTW: Is it safe? I mean, is zero-after-free still there ?

Yes, the zero after free was kept.  And it still mlocks the private key data (which is slow because its still doing it for every allocation but it only adversely impacts key pool filling, this will be fixed later as it provides less benefit and needs a more complicated fix to maintain a pool of mlocked memory for that purpose).

[pusle]

Thank you, this is really nice!  I'm sending you some coins for this

[prof7bit]

Why does it do synchronized disk IO anywhere anyways? Its not used with removable media (its even almost impossible using it on external removable media when it is implemented this way anyways.

[gmaxwell]

Why does it do synchronized disk IO anywhere anyways? Its not used with removable media (its even almost impossible using it on external removable media when it is implemented this way anyways.

Because the sync disk IO is important to avoid losing database updates or corrupting the database when the power goes out during a write—  which matters a lot for the wallet but not everything else.  However, it currently uses the same database setup (though different files) for all things.

So an update on this— After having some people test the RC builds on lower performing windows boxes (e.g. random spinning disk systems with only 1GB ram) they're still seeing 18 hour syncs, so at least on those systems this particular doesn't make as big of a difference as it does on fast/linux/ssd system.   I'm still working on finding additional improvements for all systems.

If you'd like to benchmark it on your own system(s), you should add logtimestamps=1  to your bitcoin.conf so your debug.log file will get a timestamp for every entry. Then you can start it up and check later to see how long it took (this is how I generated the graphs used on this page).

There are RC builds with this fix (as well as some of the recent deadlock fixes), made by Matt Corallo <matt@bluematt.me> using the reproducible build process:
http://dl.dropbox.com/u/29653426/bitcoin-0.5.2rc1.tar.bz2
http://dl.dropbox.com/u/29653426/bitcoin-0.5.2rc1-win32.tar.bz2

(though, er, personally I wouldn't run binaries from dropbox urls, except for the fact that I knew Matt posted them)

If you do try out the RC builds please let me know (either here, in #bitcoin-dev on irc, or in PM) as getting a few more people trying them out is the only thing delaying their official release right now. Always make sure to backup your wallet before trying new software (and periodically even if you're on old software!).