Experimental pre-0.8 builds for testing

[grue]

using 2012-12-13-turbo-ldb17
bitcoin-qt.exe -par=4 -connect=178.18.90.41 -dbcche=1536 -logtimestamps -benchmark
AMD Phenom II X4 @ 3.9 GHZ
4 GB RAM

Cpu usage starts off at 2%-25%, then goes down to 1%-5% about half way through block download.
Bitcoin-qt was downloading blocks at 540 kB/s.

debug.log:
http://www.4shared.com/archive/O3Jh456U/debug.html

[1715000661]

1715000661

[finway]

bitcoin-qt.exe -reindex -par=4 -dbcche=500 -logtimestamps -benchmark

it took 37 minutes to reach 193000
and it took 2 hours 27 minutes to reach 211865
before 193000, CPU usage:  almost 50%
after 193000 , CPU usage:   85% ~ 95%

Laptop, intel core i3, 4GB ram

[Rogue Star]

Here's my results from the 15th. I was syncing solely from a local peer and did not use block loading

Configuration
-------
Version: bitcoin-0.7.1-283-gf6aadb1-win32 (aka "turbo" build)
Flags: -connect=x.x.x.x -keypool=0
Hardware: VM w/full hw acceleration
vOS: Windows 7 64-bit latest
vCPU: AMD A8-3850 2 cores @ 2.9 GHz
vRAM: 4GB @ 1333 MHz
vDisk: 7200 rpm
vNetwork: 1 gigabit link

performance syncing up to block 193K
-----------
Duration: 56m23s
CPU Utilization: ~50%
CPU threads: ~32
Network Utilization: ~1% over gigabit link

RAM usage:
100MB @ ~130K blocks
110MB @ ~148K blocks
120MB @ ~157K blocks
130MB @ ~176K blocks
140MB @ ~190K blocks

performance for full sync up to network (212433)
-----------
Duration: 5h 15m total
CPU Utilization: up to 100%
Network Utilization: ~0.25% over gigabit link

[finway]

Error report:
After a crush, i can't start bitcoin-qt, so i installed the newest build, still can't.

Windows7 32bit   AMD CPU  4GB ram.

debug.txt

Code:

Bitcoin version v0.7.1-283-gf6aadb1-beta (2012-12-13 11:11:02 +0100)
Using OpenSSL version OpenSSL 1.0.1c 10 May 2012
Startup time: 12/25/12 10:52:08
Default data directory C:\Users\Administrator\AppData\Roaming\Bitcoin
Used data directory C:\Users\Administrator\AppData\Roaming\Bitcoin
Using 2 threads for script verification
dbenv.open LogDir=C:\Users\Administrator\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\Administrator\AppData\Roaming\Bitcoin\db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
Loading block index...
Opening LevelDB in C:\Users\Administrator\AppData\Roaming\Bitcoin\blktree
Opened LevelDB successfully
Opening LevelDB in C:\Users\Administrator\AppData\Roaming\Bitcoin\coins
Opened LevelDB successfully
LoadBlockIndex(): last block file = 33
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=246, size=35583097, heights=212710..212955, time=2012-12-18..2012-12-20)
 block index            3982ms
Loading wallet...
nFileVersion = 79900
 wallet                  329ms
InvalidChainFound: invalid block=000000000000030ced7d02b925d6a8709a4d1559afe535d863cbc56d0e569894  height=212955  work=672233111659878960356  date=12/20/12 23:05:13

[smtp]

Hi

If you think this message is the wrong place, place shift it to the correct topic/thread:

    Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360  regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client.

And if you think it is appropriate, you are free to move it to this (or a better) board/thread.

BTW: Thank you for taking me out off the newbie-posting-restriction jail.

Regards
smtp

[Pieter Wuille]

    Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360  regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client.

And if you think it is appropriate, you are free to move it to this (or a better) board/thread.

BTW: Thank you for taking me out off the newbie-posting-restriction jail.

Hello there,

I can't take you out of the jail, but I can explain what we're doing in 0.8.

The old blkindex.dat file is a (very non-efficient, 12 bytes per txout) index that stores the disk position of all transactions in the blk00*.dat files, and for each transaction output whether and where it got spent. During block validation, this means that for every transaction that needs to be validated, we need to look up all input's previous transactions in the index, verify all inputs aren't spent yet, then read those transactions from disk, and then evaluate scripts, and finally update the index with spends and new transaction outputs created. This means the working set (the amount of data that needs to be readily available) is most of the block index, and part of the block data itself too (especially the recent/last part needs frequent access). This means fast access to gigabytes, and you're right to point out that this is slow. Too slow, because it is wasteful in two ways: 1) we need access to full transactions (even the parts that are already spent, or aren't necessary for validation of inputs) and 2) the index to search through contains all transactionsm even those that are already completely spent.

For 0.8, this is changed completely. There is no transaction index at all anymore. Instead, we keep - in addition to the block files - a database (LevelDB, not BDB) that contains just all unspent outputs (specifically, for every not-fully-spent transaction: its version, its height in the chain, and for all its unspent outputs: the value and the script) using an efficient encoding (so not pointers into the block files, the actual data itself). This database is right now (including all overhead and indexing structures) 146 MiB (126 MiB of which is actual data). This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory.

There is some overhead in maintaining a database, and encoding and decoding the txout data, but on my laptop, if I disable signature verification and increase the db cache size to 200 MB, reindexing to block 213769 (including all block and transaction validation, just not ECDSA verification) takes around 10 minutes (mostly at 100% CPU usage on one core). Using hash tables instead of trees for the cache, and increasing its size would further reduce that number. That's not the next limiting factor though. The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined).

I do plan to make some improvements to the in-memory unspent-output-database-cache still, so that if made large enough, one shouldn't ever wait on disk, but I don't expect huge improvements there anymore. The most important changes for speeding up are reworking the block download mechanism, and parallellizing signature verification (which is already implemented, but not merged in mainline).

[notme]

Trying to compile the turbo build on linux I get these errors:
/usr/bin/ld: cannot find -lshlwapi
/usr/bin/ld: cannot find -ldbghelp

It seems to be complaining about windows only libraries being missing.

[Pieter Wuille]

Trying to compile the turbo build on linux I get these errors:
/usr/bin/ld: cannot find -lshlwapi
/usr/bin/ld: cannot find -ldbghelp

It seems to be complaining about windows only libraries being missing.

Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows).

[smtp]

I can't take you out of the jail, but I can explain what we're doing in 0.8.

The first was no longer necessary (else I could not habe posted the previous notice in THIS topic/board) and
thx for the later.

So, reindexing and the "-loadblock" option in 0.7.1 not only made a new blkindex.dat file but furthermore do a fully validation (including the expensive signature check). I think there is no secret option in 0.7.1 or 0.7.2 to disable the signature verification when using "-loadblock", isn't it?

Else a new bitcoin-client user could simple load a valid blockchain from a URL he trusted and simply load it to get this "blkindex.dat" to save MUCH real time -- this was my aim. For history: I deleted the only 6 orphan blocks form my personal blk00*.dat and thus wanted/needed  a new blkindex.dat.

For 0.8, this is changed completely. There is no transaction index ..... This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory.

This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. To check/validate a NEW block/transaction, the bitcoin-client gets only this data in RAM is sufficient, but sadly not for my blockparser.
Moreover the idea to keep the whole out-scripts in memory is very good because there is only a very small number of effectively different response-scripts - they mainly differ only in different RMD-160-bit hashes for their public key. So, I guess in > 99% a table-no and this 20 byte hash is sufficient data to store the script. I also looked at all the input- and output-scripts and there are only very few effectively different ones -- so your "encoding and decoding the txout data" should be done quick and you can keep the coding-table small -- as long as the number of different out-scripts stay small (as in the past).

The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined).

The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on https://en.bitcoin.it/wiki/OP_CHECKSIG. Thus I can't judge how cpu-intensive this OP_CHECKSIG evaluation really is.

To be frank - I should add that I had hoped 0.8 would be released as a Christmas present and after this, once more as a New Years present, but my hope faded when I read more on the bitcoin forum. Thus I posted my proposal in the hope it could help to improve the 0.8 version.

Thank you for your explanation. Now we can only continue to hope that 0.8 with the new data-structure will be released as reasonable quick as possible.

smtp

[notme]

Trying to compile the turbo build on linux I get these errors:
/usr/bin/ld: cannot find -lshlwapi
/usr/bin/ld: cannot find -ldbghelp

It seems to be complaining about windows only libraries being missing.

Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows).

I'm using the makefile generated by the qmake command.

The branch is the most up to date one in your github repository (sipa/bitcoin) head is:
fd95a8ac216b447f429409fde9f8eaef2ae4c339

[niko]

I decided to try out the low extreme, so I dug out a fossil and fired it up:

Windows XP SP3
Pentium M, 1.6 GHz
760 MB RAM
Disk space available initially: ~35 GB

Installed bitcoin-0.7.1-315-gfd95a8a-win32-setup.exe

bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark

Started up fine, connected through LAN to the specified peer, started downloading. Typical CPU load (essentially no competing processes) was 20%-65%, rarely over 70%. Peak memory usage ~180 MB.

It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB.  What is this?

I cleaned up some old files to make additional 7GB of space available, and started again as above. It continued downloading, then progress bar disappeared. The animated "sync" icon in the corner was still active, and downloading and verification was still going on fine. The CPU usage appeared somewhat lower then earlier, 20%-40% typically. Three hours into it, ran out of disk space again! The "coins" folder now grew to >37GB.

Cleaned up more GB, restarted the client, and two hours later and few hundreds of blocks before the end it ran out of disk space again! "Coins" folder now 40.1 GB, 21,734 files.

Cleaned up more space, but bitcoin-qt now crashes immediately after start. Rebooted, still the same. Windows crash report:

Code:

<?xml version="1.0" encoding="UTF-16"?>
<DATABASE>
<EXE NAME="bitcoin-qt.exe" FILTER="GRABMI_FILTER_PRIVACY">
    <MATCHING_FILE NAME="bitcoin-qt.exe" SIZE="20755456" CHECKSUM="0x604C3D23" BIN_FILE_VERSION="0.7.99.0" BIN_PRODUCT_VERSION="0.7.99.0" PRODUCT_VERSION="0.7.99.0" FILE_DESCRIPTION="Bitcoin-Qt (OSS GUI client for Bitcoin)" COMPANY_NAME="Bitcoin" PRODUCT_NAME="Bitcoin-Qt" FILE_VERSION="0.7.99.0" ORIGINAL_FILENAME="bitcoin-qt.exe" INTERNAL_NAME="bitcoin-qt" LEGAL_COPYRIGHT="2009-2012 The Bitcoin developers" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x1" MODULE_TYPE="WIN32" PE_CHECKSUM="0x13D233D" LINKER_VERSION="0x10000" UPTO_BIN_FILE_VERSION="0.7.99.0" UPTO_BIN_PRODUCT_VERSION="0.7.99.0" LINK_DATE="01/30/2011 00:00:00" UPTO_LINK_DATE="01/30/2011 00:00:00" VER_LANGUAGE="Language Neutral [0x0]" />
    <MATCHING_FILE NAME="uninstall.exe" SIZE="366985" CHECKSUM="0x7224CC4B" BIN_FILE_VERSION="0.7.99.0" BIN_PRODUCT_VERSION="0.7.99.0" PRODUCT_VERSION="0.7.99" FILE_DESCRIPTION="" COMPANY_NAME="Bitcoin project" PRODUCT_NAME="Bitcoin" FILE_VERSION="0.7.99" LEGAL_COPYRIGHT="" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x4" VERFILETYPE="0x1" MODULE_TYPE="WIN32" PE_CHECKSUM="0x1B641" LINKER_VERSION="0x60000" UPTO_BIN_FILE_VERSION="0.7.99.0" UPTO_BIN_PRODUCT_VERSION="0.7.99.0" LINK_DATE="02/19/2012 15:01:49" UPTO_LINK_DATE="02/19/2012 15:01:49" VER_LANGUAGE="Language Neutral [0x0]" />
    <MATCHING_FILE NAME="daemon\bitcoind.exe" SIZE="6174208" CHECKSUM="0x2B4948A" MODULE_TYPE="WIN32" PE_CHECKSUM="0x5ECEB6" LINKER_VERSION="0x10000" LINK_DATE="01/30/2011 00:00:00" UPTO_LINK_DATE="01/30/2011 00:00:00" />
</EXE>
<EXE NAME="kernel32.dll" FILTER="GRABMI_FILTER_THISFILEONLY">
    <MATCHING_FILE NAME="kernel32.dll" SIZE="990208" CHECKSUM="0xCC2C4544" BIN_FILE_VERSION="5.1.2600.6293" BIN_PRODUCT_VERSION="5.1.2600.6293" PRODUCT_VERSION="5.1.2600.6293" FILE_DESCRIPTION="Windows NT BASE API Client DLL" COMPANY_NAME="Microsoft Corporation" PRODUCT_NAME="Microsoft® Windows® Operating System" FILE_VERSION="5.1.2600.6293 (xpsp_sp3_gdr.121001-1622)" ORIGINAL_FILENAME="kernel32" INTERNAL_NAME="kernel32" LEGAL_COPYRIGHT="© Microsoft Corporation. All rights reserved." VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x2" MODULE_TYPE="WIN32" PE_CHECKSUM="0xFBCBC" LINKER_VERSION="0x50001" UPTO_BIN_FILE_VERSION="5.1.2600.6293" UPTO_BIN_PRODUCT_VERSION="5.1.2600.6293" LINK_DATE="10/03/2012 04:58:13" UPTO_LINK_DATE="10/03/2012 04:58:13" VER_LANGUAGE="English (United States) [0x409]" />
</EXE>
</DATABASE>

Debug.log:  http://www.filehosting.org/file/details/407213/debug.log

All in all, almost complete download via LAN and verification took ~10 hours.

[Pieter Wuille]

It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB.  What is this?

This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data?

[Pieter Wuille]

This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more.

So it would seem:

(a bit outdated, but the trends are clear)

[niko]

It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB.  What is this?

This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data?

[Diapolo]

Seems to be a Windows related issue with LevelDB...

Dia

[Pieter Wuille]

It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB.  What is this?

Thanks for reporting this. We (mostly Gavin) found a general problem with the LevelDB code on older Windowses, The last build (0.7.1-313-g4fb7372) should fix this.

Please try again if you can.

[niko]

It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB.  What is this?

Thanks for reporting this. We (mostly Gavin) found a general problem with the LevelDB code on older Windowses, The last build (0.7.1-313-g4fb7372) should fix this.

Please try again if you can.

Tried again with bitcoin-0.7.1-313-g4fb7372-win32.zip - successfully downloaded blocks and verified, from scratch. It took about 11 hours. Setup the same as in my earlier posts here:
Windows XP SP3
Pentium M, 1.6 GHz
760 MB RAM
bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark

Peak memory usage ~180MB.

There may have been a WiFi problem towards the end of download (or client got stuck doing nothing). Finally I restarted it with port forwarding and made ~20 connections within hours. Received and sent coins.

All in all, even this weakling can download (via LAN) and verify complete blockchain overnight.

Minor issues:

- download progress bar still disappearing towards the end of download
- it says "processed X of 210000 blocks" - even though we are well past 210000
- CPU usage for the first ~third of blocks is 40%-100%, then it slows down to 0%-40% for the most part - not sure what the bottleneck is
- verification slows down progressively; would it make sense to base progress bar on megabytes, not on the block count?

[Diapolo]

Minor issues:

- download progress bar still disappearing towards the end of download
- it says "processed X of 210000 blocks" - even though we are well past 210000

As the number of blocks on the network is only a rough estimate, we can't really rely on that number, I'm going to think about how this could be handled more user-friendly .

Dia

[notme]

Trying to compile the turbo build on linux I get these errors:
/usr/bin/ld: cannot find -lshlwapi
/usr/bin/ld: cannot find -ldbghelp

It seems to be complaining about windows only libraries being missing.

Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows).

I'm using the makefile generated by the qmake command.

The branch is the most up to date one in your github repository (sipa/bitcoin) head is:
fd95a8ac216b447f429409fde9f8eaef2ae4c339

I tried again tonight.  The dbghelp error is indeed gone now, but the shlwapi error persists.  Head is 4fb73728b2a924c8b940042b3b7fbe8ef7a4b274

Removing -lshlwapi from line 100 of bitcoin-qt.pro (LIBS line) allows the build to complete.  It is now running with a fresh (empty) datadir and seems to be syncing nicely.

[Pieter Wuille]

Removing -lshlwapi from line 100 of bitcoin-qt.pro (LIBS line) allows the build to complete.  It is now running with a fresh (empty) datadir and seems to be syncing nicely.

Thanks, fixed.

Seems I forgot testing building bitcoin-qt on Linux, after all tinkering to get the Windows build working again.