grue
Legendary
Offline
Activity: 2058
Merit: 1434
|
|
December 15, 2012, 06:51:34 PM |
|
using 2012-12-13-turbo-ldb17 bitcoin-qt.exe -par=4 -connect=178.18.90.41 -dbcche=1536 -logtimestamps -benchmark AMD Phenom II X4 @ 3.9 GHZ 4 GB RAM Cpu usage starts off at 2%-25%, then goes down to 1%-5% about half way through block download. Bitcoin-qt was downloading blocks at 540 k B/s. debug.log: http://www.4shared.com/archive/O3Jh456U/debug.html
|
|
|
|
finway
|
|
December 22, 2012, 05:41:58 AM |
|
bitcoin-qt.exe -reindex -par=4 -dbcche=500 -logtimestamps -benchmark
it took 37 minutes to reach 193000 and it took 2 hours 27 minutes to reach 211865 before 193000, CPU usage: almost 50% after 193000 , CPU usage: 85% ~ 95%
Laptop, intel core i3, 4GB ram
|
|
|
|
Rogue Star
Member
Offline
Activity: 89
Merit: 10
|
|
December 22, 2012, 08:02:39 AM |
|
Here's my results from the 15th. I was syncing solely from a local peer and did not use block loading
Configuration ------- Version: bitcoin-0.7.1-283-gf6aadb1-win32 (aka "turbo" build) Flags: -connect=x.x.x.x -keypool=0 Hardware: VM w/full hw acceleration vOS: Windows 7 64-bit latest vCPU: AMD A8-3850 2 cores @ 2.9 GHz vRAM: 4GB @ 1333 MHz vDisk: 7200 rpm vNetwork: 1 gigabit link
performance syncing up to block 193K ----------- Duration: 56m23s CPU Utilization: ~50% CPU threads: ~32 Network Utilization: ~1% over gigabit link
RAM usage: 100MB @ ~130K blocks 110MB @ ~148K blocks 120MB @ ~157K blocks 130MB @ ~176K blocks 140MB @ ~190K blocks
performance for full sync up to network (212433) ----------- Duration: 5h 15m total CPU Utilization: up to 100% Network Utilization: ~0.25% over gigabit link
|
you can donate to me for whatever reason at: 18xbnjDDXxgcvRzv5k2vmrKQHWDjYsBDCf
|
|
|
finway
|
|
December 25, 2012, 10:55:19 AM |
|
Error report: After a crush, i can't start bitcoin-qt, so i installed the newest build, still can't. Windows7 32bit AMD CPU 4GB ram. debug.txt Bitcoin version v0.7.1-283-gf6aadb1-beta (2012-12-13 11:11:02 +0100) Using OpenSSL version OpenSSL 1.0.1c 10 May 2012 Startup time: 12/25/12 10:52:08 Default data directory C:\Users\Administrator\AppData\Roaming\Bitcoin Used data directory C:\Users\Administrator\AppData\Roaming\Bitcoin Using 2 threads for script verification dbenv.open LogDir=C:\Users\Administrator\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\Administrator\AppData\Roaming\Bitcoin\db.log Bound to [::]:8333 Bound to 0.0.0.0:8333 Loading block index... Opening LevelDB in C:\Users\Administrator\AppData\Roaming\Bitcoin\blktree Opened LevelDB successfully Opening LevelDB in C:\Users\Administrator\AppData\Roaming\Bitcoin\coins Opened LevelDB successfully LoadBlockIndex(): last block file = 33 LoadBlockIndex(): last block file: CBlockFileInfo(blocks=246, size=35583097, heights=212710..212955, time=2012-12-18..2012-12-20) block index 3982ms Loading wallet... nFileVersion = 79900 wallet 329ms InvalidChainFound: invalid block=000000000000030ced7d02b925d6a8709a4d1559afe535d863cbc56d0e569894 height=212955 work=672233111659878960356 date=12/20/12 23:05:13
|
|
|
|
smtp
Member
Offline
Activity: 70
Merit: 10
|
|
December 27, 2012, 10:08:19 PM |
|
Hi If you think this message is the wrong place, place shift it to the correct topic/thread: Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360 regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client. And if you think it is appropriate, you are free to move it to this (or a better) board/thread. BTW: Thank you for taking me out off the newbie-posting-restriction jail. Regards smtp
|
|
|
|
Pieter Wuille (OP)
|
|
December 27, 2012, 11:29:38 PM |
|
Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360 regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client. And if you think it is appropriate, you are free to move it to this (or a better) board/thread. BTW: Thank you for taking me out off the newbie-posting-restriction jail. Hello there, I can't take you out of the jail, but I can explain what we're doing in 0.8. The old blkindex.dat file is a (very non-efficient, 12 bytes per txout) index that stores the disk position of all transactions in the blk00*.dat files, and for each transaction output whether and where it got spent. During block validation, this means that for every transaction that needs to be validated, we need to look up all input's previous transactions in the index, verify all inputs aren't spent yet, then read those transactions from disk, and then evaluate scripts, and finally update the index with spends and new transaction outputs created. This means the working set (the amount of data that needs to be readily available) is most of the block index, and part of the block data itself too (especially the recent/last part needs frequent access). This means fast access to gigabytes, and you're right to point out that this is slow. Too slow, because it is wasteful in two ways: 1) we need access to full transactions (even the parts that are already spent, or aren't necessary for validation of inputs) and 2) the index to search through contains all transactionsm even those that are already completely spent. For 0.8, this is changed completely. There is no transaction index at all anymore. Instead, we keep - in addition to the block files - a database (LevelDB, not BDB) that contains just all unspent outputs (specifically, for every not-fully-spent transaction: its version, its height in the chain, and for all its unspent outputs: the value and the script) using an efficient encoding (so not pointers into the block files, the actual data itself). This database is right now (including all overhead and indexing structures) 146 MiB (126 MiB of which is actual data). This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory. There is some overhead in maintaining a database, and encoding and decoding the txout data, but on my laptop, if I disable signature verification and increase the db cache size to 200 MB, reindexing to block 213769 (including all block and transaction validation, just not ECDSA verification) takes around 10 minutes (mostly at 100% CPU usage on one core). Using hash tables instead of trees for the cache, and increasing its size would further reduce that number. That's not the next limiting factor though. The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined). I do plan to make some improvements to the in-memory unspent-output-database-cache still, so that if made large enough, one shouldn't ever wait on disk, but I don't expect huge improvements there anymore. The most important changes for speeding up are reworking the block download mechanism, and parallellizing signature verification (which is already implemented, but not merged in mainline).
|
I do Bitcoin stuff.
|
|
|
notme
Legendary
Offline
Activity: 1904
Merit: 1002
|
|
December 28, 2012, 12:01:29 PM |
|
Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp
It seems to be complaining about windows only libraries being missing.
|
|
|
|
Pieter Wuille (OP)
|
|
December 28, 2012, 02:59:08 PM |
|
Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp
It seems to be complaining about windows only libraries being missing.
Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows).
|
I do Bitcoin stuff.
|
|
|
smtp
Member
Offline
Activity: 70
Merit: 10
|
|
December 28, 2012, 09:39:35 PM |
|
I can't take you out of the jail, but I can explain what we're doing in 0.8.
The first was no longer necessary (else I could not habe posted the previous notice in THIS topic/board) and thx for the later. So, reindexing and the "-loadblock" option in 0.7.1 not only made a new blkindex.dat file but furthermore do a fully validation (including the expensive signature check). I think there is no secret option in 0.7.1 or 0.7.2 to disable the signature verification when using "-loadblock", isn't it? Else a new bitcoin-client user could simple load a valid blockchain from a URL he trusted and simply load it to get this "blkindex.dat" to save MUCH real time -- this was my aim. For history: I deleted the only 6 orphan blocks form my personal blk00*.dat and thus wanted/needed a new blkindex.dat. For 0.8, this is changed completely. There is no transaction index ..... This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory.
This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. To check/validate a NEW block/transaction, the bitcoin-client gets only this data in RAM is sufficient, but sadly not for my blockparser. Moreover the idea to keep the whole out-scripts in memory is very good because there is only a very small number of effectively different response-scripts - they mainly differ only in different RMD-160-bit hashes for their public key. So, I guess in > 99% a table-no and this 20 byte hash is sufficient data to store the script. I also looked at all the input- and output-scripts and there are only very few effectively different ones -- so your "encoding and decoding the txout data" should be done quick and you can keep the coding-table small -- as long as the number of different out-scripts stay small (as in the past). The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined). The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on https://en.bitcoin.it/wiki/OP_CHECKSIG. Thus I can't judge how cpu-intensive this OP_CHECKSIG evaluation really is. To be frank - I should add that I had hoped 0.8 would be released as a Christmas present and after this, once more as a New Years present, but my hope faded when I read more on the bitcoin forum. Thus I posted my proposal in the hope it could help to improve the 0.8 version. Thank you for your explanation. Now we can only continue to hope that 0.8 with the new data-structure will be released as reasonable quick as possible. smtp
|
|
|
|
notme
Legendary
Offline
Activity: 1904
Merit: 1002
|
|
December 28, 2012, 10:41:27 PM |
|
Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp
It seems to be complaining about windows only libraries being missing.
Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows). I'm using the makefile generated by the qmake command. The branch is the most up to date one in your github repository (sipa/bitcoin) head is: fd95a8ac216b447f429409fde9f8eaef2ae4c339
|
|
|
|
niko
|
|
December 30, 2012, 05:13:52 PM Last edit: December 30, 2012, 05:39:38 PM by niko |
|
I decided to try out the low extreme, so I dug out a fossil and fired it up: Windows XP SP3 Pentium M, 1.6 GHz 760 MB RAM Disk space available initially: ~35 GB Installed bitcoin-0.7.1-315-gfd95a8a-win32-setup.exe bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark Started up fine, connected through LAN to the specified peer, started downloading. Typical CPU load (essentially no competing processes) was 20%-65%, rarely over 70%. Peak memory usage ~180 MB. It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this? I cleaned up some old files to make additional 7GB of space available, and started again as above. It continued downloading, then progress bar disappeared. The animated "sync" icon in the corner was still active, and downloading and verification was still going on fine. The CPU usage appeared somewhat lower then earlier, 20%-40% typically. Three hours into it, ran out of disk space again! The "coins" folder now grew to >37GB. Cleaned up more GB, restarted the client, and two hours later and few hundreds of blocks before the end it ran out of disk space again! "Coins" folder now 40.1 GB, 21,734 files. Cleaned up more space, but bitcoin-qt now crashes immediately after start. Rebooted, still the same. Windows crash report: <?xml version="1.0" encoding="UTF-16"?> <DATABASE> <EXE NAME="bitcoin-qt.exe" FILTER="GRABMI_FILTER_PRIVACY"> <MATCHING_FILE NAME="bitcoin-qt.exe" SIZE="20755456" CHECKSUM="0x604C3D23" BIN_FILE_VERSION="0.7.99.0" BIN_PRODUCT_VERSION="0.7.99.0" PRODUCT_VERSION="0.7.99.0" FILE_DESCRIPTION="Bitcoin-Qt (OSS GUI client for Bitcoin)" COMPANY_NAME="Bitcoin" PRODUCT_NAME="Bitcoin-Qt" FILE_VERSION="0.7.99.0" ORIGINAL_FILENAME="bitcoin-qt.exe" INTERNAL_NAME="bitcoin-qt" LEGAL_COPYRIGHT="2009-2012 The Bitcoin developers" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x1" MODULE_TYPE="WIN32" PE_CHECKSUM="0x13D233D" LINKER_VERSION="0x10000" UPTO_BIN_FILE_VERSION="0.7.99.0" UPTO_BIN_PRODUCT_VERSION="0.7.99.0" LINK_DATE="01/30/2011 00:00:00" UPTO_LINK_DATE="01/30/2011 00:00:00" VER_LANGUAGE="Language Neutral [0x0]" /> <MATCHING_FILE NAME="uninstall.exe" SIZE="366985" CHECKSUM="0x7224CC4B" BIN_FILE_VERSION="0.7.99.0" BIN_PRODUCT_VERSION="0.7.99.0" PRODUCT_VERSION="0.7.99" FILE_DESCRIPTION="" COMPANY_NAME="Bitcoin project" PRODUCT_NAME="Bitcoin" FILE_VERSION="0.7.99" LEGAL_COPYRIGHT="" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x4" VERFILETYPE="0x1" MODULE_TYPE="WIN32" PE_CHECKSUM="0x1B641" LINKER_VERSION="0x60000" UPTO_BIN_FILE_VERSION="0.7.99.0" UPTO_BIN_PRODUCT_VERSION="0.7.99.0" LINK_DATE="02/19/2012 15:01:49" UPTO_LINK_DATE="02/19/2012 15:01:49" VER_LANGUAGE="Language Neutral [0x0]" /> <MATCHING_FILE NAME="daemon\bitcoind.exe" SIZE="6174208" CHECKSUM="0x2B4948A" MODULE_TYPE="WIN32" PE_CHECKSUM="0x5ECEB6" LINKER_VERSION="0x10000" LINK_DATE="01/30/2011 00:00:00" UPTO_LINK_DATE="01/30/2011 00:00:00" /> </EXE> <EXE NAME="kernel32.dll" FILTER="GRABMI_FILTER_THISFILEONLY"> <MATCHING_FILE NAME="kernel32.dll" SIZE="990208" CHECKSUM="0xCC2C4544" BIN_FILE_VERSION="5.1.2600.6293" BIN_PRODUCT_VERSION="5.1.2600.6293" PRODUCT_VERSION="5.1.2600.6293" FILE_DESCRIPTION="Windows NT BASE API Client DLL" COMPANY_NAME="Microsoft Corporation" PRODUCT_NAME="Microsoft® Windows® Operating System" FILE_VERSION="5.1.2600.6293 (xpsp_sp3_gdr.121001-1622)" ORIGINAL_FILENAME="kernel32" INTERNAL_NAME="kernel32" LEGAL_COPYRIGHT="© Microsoft Corporation. All rights reserved." VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x2" MODULE_TYPE="WIN32" PE_CHECKSUM="0xFBCBC" LINKER_VERSION="0x50001" UPTO_BIN_FILE_VERSION="5.1.2600.6293" UPTO_BIN_PRODUCT_VERSION="5.1.2600.6293" LINK_DATE="10/03/2012 04:58:13" UPTO_LINK_DATE="10/03/2012 04:58:13" VER_LANGUAGE="English (United States) [0x409]" /> </EXE> </DATABASE>
Debug.log: http://www.filehosting.org/file/details/407213/debug.logAll in all, almost complete download via LAN and verification took ~10 hours.
|
They're there, in their room. Your mining rig is on fire, yet you're very calm.
|
|
|
Pieter Wuille (OP)
|
|
December 31, 2012, 12:53:44 AM |
|
It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?
This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data?
|
I do Bitcoin stuff.
|
|
|
Pieter Wuille (OP)
|
|
December 31, 2012, 01:01:39 AM |
|
This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. So it would seem: (a bit outdated, but the trends are clear)
|
I do Bitcoin stuff.
|
|
|
niko
|
|
December 31, 2012, 01:30:22 AM |
|
It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?
This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data?
|
They're there, in their room. Your mining rig is on fire, yet you're very calm.
|
|
|
Diapolo
|
|
January 01, 2013, 11:23:06 AM |
|
Seems to be a Windows related issue with LevelDB...
Dia
|
|
|
|
Pieter Wuille (OP)
|
|
January 02, 2013, 10:34:50 PM |
|
It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?
Thanks for reporting this. We (mostly Gavin) found a general problem with the LevelDB code on older Windowses, The last build (0.7.1-313-g4fb7372) should fix this. Please try again if you can.
|
I do Bitcoin stuff.
|
|
|
niko
|
|
January 03, 2013, 06:06:10 PM |
|
It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this?
Thanks for reporting this. We (mostly Gavin) found a general problem with the LevelDB code on older Windowses, The last build (0.7.1-313-g4fb7372) should fix this. Please try again if you can. Tried again with bitcoin-0.7.1-313-g4fb7372-win32.zip - successfully downloaded blocks and verified, from scratch. It took about 11 hours. Setup the same as in my earlier posts here: Windows XP SP3 Pentium M, 1.6 GHz 760 MB RAM bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark Peak memory usage ~180MB. There may have been a WiFi problem towards the end of download (or client got stuck doing nothing). Finally I restarted it with port forwarding and made ~20 connections within hours. Received and sent coins. All in all, even this weakling can download (via LAN) and verify complete blockchain overnight. Minor issues: - download progress bar still disappearing towards the end of download - it says "processed X of 210000 blocks" - even though we are well past 210000 - CPU usage for the first ~third of blocks is 40%-100%, then it slows down to 0%-40% for the most part - not sure what the bottleneck is - verification slows down progressively; would it make sense to base progress bar on megabytes, not on the block count?
|
They're there, in their room. Your mining rig is on fire, yet you're very calm.
|
|
|
Diapolo
|
|
January 03, 2013, 06:13:29 PM |
|
Minor issues:
- download progress bar still disappearing towards the end of download - it says "processed X of 210000 blocks" - even though we are well past 210000
As the number of blocks on the network is only a rough estimate, we can't really rely on that number, I'm going to think about how this could be handled more user-friendly . Dia
|
|
|
|
notme
Legendary
Offline
Activity: 1904
Merit: 1002
|
|
January 04, 2013, 12:52:31 AM |
|
Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp
It seems to be complaining about windows only libraries being missing.
Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows). I'm using the makefile generated by the qmake command. The branch is the most up to date one in your github repository (sipa/bitcoin) head is: fd95a8ac216b447f429409fde9f8eaef2ae4c339 I tried again tonight. The dbghelp error is indeed gone now, but the shlwapi error persists. Head is 4fb73728b2a924c8b940042b3b7fbe8ef7a4b274 Removing -lshlwapi from line 100 of bitcoin-qt.pro (LIBS line) allows the build to complete. It is now running with a fresh (empty) datadir and seems to be syncing nicely.
|
|
|
|
Pieter Wuille (OP)
|
|
January 04, 2013, 02:39:44 AM |
|
Removing -lshlwapi from line 100 of bitcoin-qt.pro (LIBS line) allows the build to complete. It is now running with a fresh (empty) datadir and seems to be syncing nicely.
Thanks, fixed. Seems I forgot testing building bitcoin-qt on Linux, after all tinkering to get the Windows build working again.
|
I do Bitcoin stuff.
|
|
|
|