|
Title: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 09, 2012, 12:18:32 AM Hello all,
I've uploaded a new set of builds of bitcoind/Bitcoin-Qt to my builds page (http://bitcoin.sipa.be/builds/pre-0.8). The new "turbo" build is based on current master on git, plus some more experimental patches (Hal Finney's optimized secp256k1 code (https://bitcointalk.org/index.php?topic=3238.msg45795#msg45795), and parallel script verification). The source code for these can be found here (https://github.com/sipa/bitcoin/tree/turbo) (.tgz archive here (https://github.com/sipa/bitcoin/archive/turbo.tar.gz)). Disclaimer: don't use these for any serious stuff - I am not responsible for destroyed wallets or exploded CPU's. Testing of these is very welcome. As the most invasive change is in the block/transaction validation system, this is the most interesting part to test. I am very interested in reports about speed, CPU usage, memory usage, ... during initial block synchronization, in particular on Windows systems. Obviously, crashes or other unexpected behaviour is even more important. As these builds use the new 0.8 database code ("ultraprune", as I called it before), they will not use the existing block files/database from 0.7.2 or earlier. There are several ways to import the data:
As these builds use parallel script verification (only enabled after block 193000, the last checkpoint), you may wish to control how many threads are used (don't use more than the number of CPU cores you have). By default, it will try to autodetect, but you can use the -par=N option to select the number of threads. One interesting thing you can test is comparing -par=1 with higher numbers, and check the speedup (in particular of the blocks after 193000). Other options which may be useful to play with:
Title: Re: Experimental pre-0.8 builds for testing Post by: justusranvier on December 09, 2012, 02:47:10 AM This is v0.7.1-244-g3ccb06f-beta starting from a completely bare directory, downloading the blockchain from a peer on the same LAN to remove internet connection speed as a factor.
Quote time bitcoin-qt -par=8 -connect=192.168.XXX.XXX real 48m58.429s user 114m35.786s sys 2m33.817s Again with a larger cache. Quote time bitcoin-qt -par=8 -connect=192.168.XXX.XXX -dbcache=4096 real 42m10.028s user 109m33.141s sys 1m9.843s Title: Re: Experimental pre-0.8 builds for testing Post by: Rudd-O on December 09, 2012, 04:43:17 AM This is v0.7.1-244-g3ccb06f-beta starting from a completely bare directory, downloading the blockchain from a peer on the same LAN to remove internet connection speed as a factor. Quote time bitcoin-qt -par=8 -connect=192.168.XXX.XXX real 48m58.429s user 114m35.786s sys 2m33.817s Again with a larger cache. Quote time bitcoin-qt -par=8 -connect=192.168.XXX.XXX -dbcache=4096 real 42m10.028s user 109m33.141s sys 1m9.843s Multithreading seems to be working well! Title: Re: Experimental pre-0.8 builds for testing Post by: finway on December 09, 2012, 06:02:31 AM "C:\Program Files\Bitcoin\bitcoin-qt.exe" -reindex -par=4 -dbcache=500 -logtimestamps -benchmark
3h22m @ Windows7 Title: Re: Experimental pre-0.8 builds for testing Post by: dserrano5 on December 09, 2012, 02:22:27 PM Starting from an empty ~/.bitcoin directory (having only bootstrap.dat and a bitcoin.conf with an rpc password) and running this:
Code: nice bitcoind -logtimestamps -loadblock=dot-bitcoin/blk0001.dat -loadblock=dot-bitcoin/blk0002.dat -loadblock=dot-bitcoin/blk0003.dat -dbcache=50 -benchmark & Took 1 hour 57 to reach height 211494 when at that moment the latest block was 211518. As soon as 211519 was found some minutes later the remaining blocks were processed/downloaded/whatever. 4 CPU cores. RAM usage reached 420 Mb RSS, after synchronization it ended up at 375 Mb. Restarting the daemon leaves it at 130 Mb. I've never used -dbcache until now, if that matters. Usual RAM usage until now was 80-90 Mb just after starting, then going up slowly as days passed (like in, 130 Mb after ~10 days). Title: Re: Experimental pre-0.8 builds for testing Post by: gmaxwell on December 09, 2012, 05:46:08 PM "C:\Program Files\Bitcoin\bitcoin-qt.exe" -reindex -par=4 -dbcache=500 -logtimestamps -benchmark 3h22m @ Windows7 Were you already synced with 0.8 before attempting that reindex? If so that was horribly slow. Title: Re: Experimental pre-0.8 builds for testing Post by: finway on December 10, 2012, 12:46:53 AM "C:\Program Files\Bitcoin\bitcoin-qt.exe" -reindex -par=4 -dbcache=500 -logtimestamps -benchmark 3h22m @ Windows7 Were you already synced with 0.8 before attempting that reindex? If so that was horribly slow. Yes, before reindex, i'm running pre0.8 1025build version, synced. Laptop: Intel Core i3, 4GB ram Title: Re: Experimental pre-0.8 builds for testing Post by: fornit on December 10, 2012, 01:12:11 AM turbo build just crashed on win 7 64. ran for quite a few hours and then came weirdo "this program requested visual studio runtime library to terminate it in an unusual way"-or something-message, which i successfully closed by reflex :-\
was doing normal blockchain download stuck around 93% with 30 connections. Title: Re: Experimental pre-0.8 builds for testing Post by: goodbc on December 10, 2012, 04:51:41 AM I downloaded the bitcoin-0.7.1-244-g3ccb06f-win32-setup.exe, installed on my test laptop (Dual core 2GHz Intel, 2GB DDR, with XP) and, without any other extra parameters, it took ~6:40 hours to process 193395 blocks of transaction history. The cpu use was so far ~ 25%.
After this I had to restart the client (the download apparently stopped) and the cpu usage increased to ~75% (sometimes 97-99%). Later it dropped to ~65% while download speed was ~ 250 KB/s. A new restart was needed after 9:43 hours. The download bar at the bottom of the window disappeared, hopefully this won't scare a regular user. Finally, the block chain fully synchronized after ~13:00 hours... I have to admit I expected a way shorter time than that. Disclaimer: your mileage may vary. Title: Re: Experimental pre-0.8 builds for testing Post by: gmaxwell on December 10, 2012, 07:44:44 AM I have to admit I expected a way shorter time than that. If you're syncing from the network even if your own connection is fast you're subject to luck of the draw on which peers you pull from. This is one reason Pieter was suggesting loadblock runs above.Title: Re: Experimental pre-0.8 builds for testing Post by: Mushoz on December 10, 2012, 09:25:35 PM I have to admit I expected a way shorter time than that. If you're syncing from the network even if your own connection is fast you're subject to luck of the draw on which peers you pull from. This is one reason Pieter was suggesting loadblock runs above.Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers? Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release :) Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 11, 2012, 12:01:48 AM Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers? When someone implements an improvement. I plan to have a look at this myself soon, but no promises. Quote Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release :) It still needs a lot of testing, and several things are missing before we can have a new stable release. Title: Re: Experimental pre-0.8 builds for testing Post by: Mushoz on December 11, 2012, 03:47:37 PM Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers? When someone implements an improvement. I plan to have a look at this myself soon, but no promises. Quote Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release :) It still needs a lot of testing, and several things are missing before we can have a new stable release. Sounds great, keep up the good work :) And no need to rush :) Title: Re: Experimental pre-0.8 builds for testing Post by: jgarzik on December 12, 2012, 03:43:49 AM Sounds great, keep up the good work :) +1, Pieter has been doing the (majority?) of the Satoshi client heavy lifting. He deserves lots of kudos and free beer. Quote And no need to rush :) Actually, we do! We need to stabilize 0.8 as quickly as is reasonably possible, because people are beginning to avoid 0.7.x due to slowness, choosing instead less secure but faster bitcoin clients. Any and all testing, on any and all platforms, is greatly appreciated. Even posting "it works, my platform is [Ubuntu version X | Fedora version Y | Windows version Z]..." is helpful. Title: Re: Experimental pre-0.8 builds for testing Post by: paybitcoin on December 12, 2012, 07:05:14 AM Hello,
Testing with bitcoin-0.7.1-244-g3ccb06f-win32-setup.exe and a bootstrap.dat from the torrent. All defaults, except using my server as the peer. CPU is Win7 x64 Athlon II X4 2.8GHz (4 cores), 4GB RAM, HDD. bootstrap.dat finished in 53 minutes at 25% CPU, ~120 MB RAM max. With no connection so its not downloading stuff in the background. The rest of the blocks took 79 minutes. Download was constant at 350-450KBps (about 30% of my total DSL bandwidth.) 70-80% CPU the whole time, ~163 MB RAM max. blocks dir is now 4.53 GB. Went pretty smoothly. Much, much better than the previous days-long sync. Title: Re: Experimental pre-0.8 builds for testing Post by: kwukduck on December 12, 2012, 07:16:52 AM Clean setup in win7_x64
Still syncing the blockchain (over 15 hrs now) Barely any CPU usage, but using 1.3GB memory. Also seems to have a lot of disk activity making the system a bit slow. Title: Re: Experimental pre-0.8 builds for testing Post by: m0mchil on December 12, 2012, 02:14:25 PM intel core i7 860 (4 physical, 8 logical cores)
bitcoin-qt built with gitian-builder from sipa/turbo rev 3ccb06f5ae538260817b6ef474f93029be41a64d Ubuntu 12.04 64 bit, SSD A-DATA SX910 256 GB 64 bit, no -par (default), default cache first 193000 blocks from bootstrap.dat - 11m:40s CPU time real 9m50.565s user 10m47.800s sys 0m56.548s 64 bit, no -par (default), default cache loading from blk000x.dat, up to arround block 211100 -loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat 1 GB peak memory, ~80% average CPU usage, total CPU time 3h:16m real 41m30.004s user 196m43.378s sys 2m47.834s debug.log.ubuntu.12.04.x64.7z (http://debian.fmi.uni-sofia.bg/~maurice/bitcoin/debug.log.ubuntu.12.04.x64.7z) Windows 7 Professional x64, two Seagate 512 GB HDD in RAID 0 -loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat default cache, with '-par 8', initially it seemed to run in single thread mode - first 193 000 blocks loaded significantly slower than on Ubuntu (same hardware), reason perhaps is slower storage. After block 193000 the average CPU utilization rose to ~75%. Looking at threads, there were 16 threads executing same code of which 8 were actually active. 92 minutes real time, 6h:05m CPU time, 72% average CPU utilization, 235 MB peak memory working set I then tried with '-par=4' and setting CPU affinity to physical cores only. Now there were initially 12 threads at transaction crunching, with again only one actively processing pre-SatoshiDice blocks. debug.log.win32.7z (http://debian.fmi.uni-sofia.bg/~maurice/bitcoin/debug.log.win32.7z) Title: Re: Experimental pre-0.8 builds for testing Post by: slothbag on December 13, 2012, 09:55:52 AM I upgraded to the latest Jenkins build of bitcoin-qt ( pre-0.8 ) on windows. I tried for a week and a half running -loadblock and -reindex, stopping restarting etc.. it was horribly slow, and going no where.
So I eventually downloaded the SIPA turbo build linked in the OP, deleted my existing blockchain, downloaded the bootstrap and let it run from scratch with an empty wallet.. Took about 6 hours to load in the bootstrap file while CPU sat at 30% and disk I/O not that high, so not sure what the bottleneck was there. then after block 190000 CPU jumped to 100% and it took about 10 hours to get back in sync to current block. Then I added my old wallet and rescanned that which went for maybe 10-20mins.. So significantly faster than the first time I sync'd 18 months ago.. but still quite frustrating. I'm running a Atom CPU based nettop with windows 7, hence the slower than average speeds. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 13, 2012, 12:05:28 PM I added yet another - and even more experimental - build (in the "-turbo-ldb17" directory here (http://bitcoin.sipa.be/builds/pre-0.8)) with levelDB 1.7, and several Windows-only changes (new Windows LevelDB backend, Boost 1.52, newer compiler suite). Tests/benchmarks/comparison with the other turbo build are very welcome, particularly on Windows (there are very few changes for other systems).
M0mchil already reported some GUI problems, but I'm confident those can be fixed if necessary. The question is really whether it's stable and performant. Also things like testing whether it recovers from sudden power loss or forceful quits of the application are nice. Again a disclaimer: Use at your own risk, I'm not responsible for overheating systems or stolen wallets. PS: If you report synchronization benchmarks, please mention on what kind of system (CPU/RAM) and CPU usage (after block 193000). Also, don't just synchronize from arbitrary peers over the internet - the variation between hitting a good or a bad peer makes the timing completely unreliable. Title: Re: Experimental pre-0.8 builds for testing Post by: justusranvier on December 13, 2012, 12:56:11 PM Can you add a command line option for benchmarking that will cause the program to exit immediately after a successful synchronization?
Title: Re: Experimental pre-0.8 builds for testing Post by: grue on December 15, 2012, 06:51:34 PM using 2012-12-13-turbo-ldb17
bitcoin-qt.exe -par=4 -connect=178.18.90.41 -dbcche=1536 -logtimestamps -benchmark AMD Phenom II X4 @ 3.9 GHZ 4 GB RAM Cpu usage starts off at 2%-25%, then goes down to 1%-5% about half way through block download. Bitcoin-qt was downloading blocks at 540 kB/s. debug.log: http://www.4shared.com/archive/O3Jh456U/debug.html Title: Re: Experimental pre-0.8 builds for testing Post by: finway on December 22, 2012, 05:41:58 AM bitcoin-qt.exe -reindex -par=4 -dbcche=500 -logtimestamps -benchmark
it took 37 minutes to reach 193000 and it took 2 hours 27 minutes to reach 211865 before 193000, CPU usage: almost 50% after 193000 , CPU usage: 85% ~ 95% Laptop, intel core i3, 4GB ram Title: Re: Experimental pre-0.8 builds for testing Post by: Rogue Star on December 22, 2012, 08:02:39 AM Here's my results from the 15th. I was syncing solely from a local peer and did not use block loading
Configuration ------- Version: bitcoin-0.7.1-283-gf6aadb1-win32 (aka "turbo" build) Flags: -connect=x.x.x.x -keypool=0 Hardware: VM w/full hw acceleration vOS: Windows 7 64-bit latest vCPU: AMD A8-3850 2 cores @ 2.9 GHz vRAM: 4GB @ 1333 MHz vDisk: 7200 rpm vNetwork: 1 gigabit link performance syncing up to block 193K ----------- Duration: 56m23s CPU Utilization: ~50% CPU threads: ~32 Network Utilization: ~1% over gigabit link RAM usage: 100MB @ ~130K blocks 110MB @ ~148K blocks 120MB @ ~157K blocks 130MB @ ~176K blocks 140MB @ ~190K blocks performance for full sync up to network (212433) ----------- Duration: 5h 15m total CPU Utilization: up to 100% Network Utilization: ~0.25% over gigabit link Title: Re: Experimental pre-0.8 builds for testing Post by: finway on December 25, 2012, 10:55:19 AM Error report:
After a crush, i can't start bitcoin-qt, so i installed the newest build, still can't. Windows7 32bit AMD CPU 4GB ram. debug.txt Code: Bitcoin version v0.7.1-283-gf6aadb1-beta (2012-12-13 11:11:02 +0100) Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on December 27, 2012, 10:08:19 PM Hi
If you think this message is the wrong place, place shift it to the correct topic/thread: Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360 regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client. And if you think it is appropriate, you are free to move it to this (or a better) board/thread. BTW: Thank you for taking me out off the newbie-posting-restriction jail. Regards smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 27, 2012, 11:29:38 PM Please have a look at my posting at https://bitcointalk.org/index.php?topic=15911.msg1422360#msg1422360 regarding "blkindex.dat" and "reindexing" real time speed of the bitcoin client. And if you think it is appropriate, you are free to move it to this (or a better) board/thread. BTW: Thank you for taking me out off the newbie-posting-restriction jail. Hello there, I can't take you out of the jail, but I can explain what we're doing in 0.8. The old blkindex.dat file is a (very non-efficient, 12 bytes per txout) index that stores the disk position of all transactions in the blk00*.dat files, and for each transaction output whether and where it got spent. During block validation, this means that for every transaction that needs to be validated, we need to look up all input's previous transactions in the index, verify all inputs aren't spent yet, then read those transactions from disk, and then evaluate scripts, and finally update the index with spends and new transaction outputs created. This means the working set (the amount of data that needs to be readily available) is most of the block index, and part of the block data itself too (especially the recent/last part needs frequent access). This means fast access to gigabytes, and you're right to point out that this is slow. Too slow, because it is wasteful in two ways: 1) we need access to full transactions (even the parts that are already spent, or aren't necessary for validation of inputs) and 2) the index to search through contains all transactionsm even those that are already completely spent. For 0.8, this is changed completely. There is no transaction index at all anymore. Instead, we keep - in addition to the block files - a database (LevelDB, not BDB) that contains just all unspent outputs (specifically, for every not-fully-spent transaction: its version, its height in the chain, and for all its unspent outputs: the value and the script) using an efficient encoding (so not pointers into the block files, the actual data itself). This database is right now (including all overhead and indexing structures) 146 MiB (126 MiB of which is actual data). This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory. There is some overhead in maintaining a database, and encoding and decoding the txout data, but on my laptop, if I disable signature verification and increase the db cache size to 200 MB, reindexing to block 213769 (including all block and transaction validation, just not ECDSA verification) takes around 10 minutes (mostly at 100% CPU usage on one core). Using hash tables instead of trees for the cache, and increasing its size would further reduce that number. That's not the next limiting factor though. The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined). I do plan to make some improvements to the in-memory unspent-output-database-cache still, so that if made large enough, one shouldn't ever wait on disk, but I don't expect huge improvements there anymore. The most important changes for speeding up are reworking the block download mechanism, and parallellizing signature verification (which is already implemented, but not merged in mainline). Title: Re: Experimental pre-0.8 builds for testing Post by: notme on December 28, 2012, 12:01:29 PM Trying to compile the turbo build on linux I get these errors:
/usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp It seems to be complaining about windows only libraries being missing. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 28, 2012, 02:59:08 PM Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp It seems to be complaining about windows only libraries being missing. Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows). Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on December 28, 2012, 09:39:35 PM I can't take you out of the jail, but I can explain what we're doing in 0.8. The first was no longer necessary (else I could not habe posted the previous notice in THIS topic/board) andthx for the later. :) So, reindexing and the "-loadblock" option in 0.7.1 not only made a new blkindex.dat file but furthermore do a fully validation (including the expensive signature check). I think there is no secret option in 0.7.1 or 0.7.2 to disable the signature verification when using "-loadblock", isn't it? Else a new bitcoin-client user could simple load a valid blockchain from a URL he trusted and simply load it to get this "blkindex.dat" to save MUCH real time -- this was my aim. For history: I deleted the only 6 orphan blocks form my personal blk00*.dat and thus wanted/needed a new blkindex.dat. Quote from: Pieter Wuille For 0.8, this is changed completely. There is no transaction index ..... This means this 146 MiB is the only thing that needs to be readily available during block validation, and it easily fits in caches in memory. This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. To check/validate a NEW block/transaction, the bitcoin-client gets only this data in RAM is sufficient, but sadly not for my blockparser. :)Moreover the idea to keep the whole out-scripts in memory is very good because there is only a very small number of effectively different response-scripts - they mainly differ only in different RMD-160-bit hashes for their public key. So, I guess in > 99% a table-no and this 20 byte hash is sufficient data to store the script. I also looked at all the input- and output-scripts and there are only very few effectively different ones -- so your "encoding and decoding the txout data" should be done quick and you can keep the coding-table small -- as long as the number of different out-scripts stay small (as in the past). Quote from: Pieter Wuille The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined). The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on https://en.bitcoin.it/wiki/OP_CHECKSIG. Thus I can't judge how cpu-intensive this OP_CHECKSIG evaluation really is.To be frank - I should add that I had hoped 0.8 would be released as a Christmas present and after this, once more as a New Years present, but my hope faded when I read more on the bitcoin forum. Thus I posted my proposal in the hope it could help to improve the 0.8 version. ;) Thank you for your explanation. Now we can only continue to hope that 0.8 with the new data-structure will be released as reasonable quick as possible. ;) smtp Title: Re: Experimental pre-0.8 builds for testing Post by: notme on December 28, 2012, 10:41:27 PM Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp It seems to be complaining about windows only libraries being missing. Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows). I'm using the makefile generated by the qmake command. The branch is the most up to date one in your github repository (sipa/bitcoin) head is: fd95a8ac216b447f429409fde9f8eaef2ae4c339 Title: Re: Experimental pre-0.8 builds for testing Post by: niko on December 30, 2012, 05:13:52 PM I decided to try out the low extreme, so I dug out a fossil and fired it up:
Windows XP SP3 Pentium M, 1.6 GHz 760 MB RAM Disk space available initially: ~35 GB Installed bitcoin-0.7.1-315-gfd95a8a-win32-setup.exe bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark Started up fine, connected through LAN to the specified peer, started downloading. Typical CPU load (essentially no competing processes) was 20%-65%, rarely over 70%. Peak memory usage ~180 MB. It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this? I cleaned up some old files to make additional 7GB of space available, and started again as above. It continued downloading, then progress bar disappeared. The animated "sync" icon in the corner was still active, and downloading and verification was still going on fine. The CPU usage appeared somewhat lower then earlier, 20%-40% typically. Three hours into it, ran out of disk space again! The "coins" folder now grew to >37GB. Cleaned up more GB, restarted the client, and two hours later and few hundreds of blocks before the end it ran out of disk space again! "Coins" folder now 40.1 GB, 21,734 files. Cleaned up more space, but bitcoin-qt now crashes immediately after start. Rebooted, still the same. Windows crash report: Code: <?xml version="1.0" encoding="UTF-16"?> Debug.log: http://www.filehosting.org/file/details/407213/debug.log All in all, almost complete download via LAN and verification took ~10 hours. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 31, 2012, 12:53:44 AM It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this? This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data? Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on December 31, 2012, 01:01:39 AM This sounds well - I like this idea. I just checked: nearly exactly a 7.5-th of all deposits/txout-channels is not redeemed currently. And I believe this fraction will increase in the future further more. So it would seem: http://bitcoin.sipa.be/pruning-txout.png (a bit outdated, but the trends are clear) Title: Re: Experimental pre-0.8 builds for testing Post by: niko on December 31, 2012, 01:30:22 AM It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this? This is very unexpected and wrong. The size of the coins directory shouldn't exceed 150 MB or so (unless you increase -dbcache, in which case it may grow to something like 150 + dbcache/2 MB). Can you tell me which files (extensions) are the bulk of the data? http://imageshack.us/a/img17/5889/19491900.png http://imageshack.us/a/img42/6451/12301958.png http://imageshack.us/a/img708/1621/77206552.png Title: Re: Experimental pre-0.8 builds for testing Post by: Diapolo on January 01, 2013, 11:23:06 AM Seems to be a Windows related issue with LevelDB...
Dia Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 02, 2013, 10:34:50 PM It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this? Thanks for reporting this. We (mostly Gavin) found a general problem with the LevelDB code on older Windowses, The last build (0.7.1-313-g4fb7372) should fix this. Please try again if you can. Title: Re: Experimental pre-0.8 builds for testing Post by: niko on January 03, 2013, 06:06:10 PM It took 4.5 hours to arrive to 5800 blocks left, when it ran out of disk space. It popped a small error window, with "ok" as the only option, when clicked the client closed. It turned out the "coins" folder grew to over 30 GB. What is this? Thanks for reporting this. We (mostly Gavin) found a general problem with the LevelDB code on older Windowses, The last build (0.7.1-313-g4fb7372) should fix this. Please try again if you can. Tried again with bitcoin-0.7.1-313-g4fb7372-win32.zip - successfully downloaded blocks and verified, from scratch. It took about 11 hours. Setup the same as in my earlier posts here: Windows XP SP3 Pentium M, 1.6 GHz 760 MB RAM bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark Peak memory usage ~180MB. There may have been a WiFi problem towards the end of download (or client got stuck doing nothing). Finally I restarted it with port forwarding and made ~20 connections within hours. Received and sent coins. All in all, even this weakling can download (via LAN) and verify complete blockchain overnight. Minor issues: - download progress bar still disappearing towards the end of download - it says "processed X of 210000 blocks" - even though we are well past 210000 - CPU usage for the first ~third of blocks is 40%-100%, then it slows down to 0%-40% for the most part - not sure what the bottleneck is - verification slows down progressively; would it make sense to base progress bar on megabytes, not on the block count? Title: Re: Experimental pre-0.8 builds for testing Post by: Diapolo on January 03, 2013, 06:13:29 PM Minor issues: - download progress bar still disappearing towards the end of download - it says "processed X of 210000 blocks" - even though we are well past 210000 As the number of blocks on the network is only a rough estimate, we can't really rely on that number, I'm going to think about how this could be handled more user-friendly :). Dia Title: Re: Experimental pre-0.8 builds for testing Post by: notme on January 04, 2013, 12:52:31 AM Trying to compile the turbo build on linux I get these errors: /usr/bin/ld: cannot find -lshlwapi /usr/bin/ld: cannot find -ldbghelp It seems to be complaining about windows only libraries being missing. Which makefile are you using? And which version of the branch (A recent change removed dbghelp altogether, also for windows). I'm using the makefile generated by the qmake command. The branch is the most up to date one in your github repository (sipa/bitcoin) head is: fd95a8ac216b447f429409fde9f8eaef2ae4c339 I tried again tonight. The dbghelp error is indeed gone now, but the shlwapi error persists. Head is 4fb73728b2a924c8b940042b3b7fbe8ef7a4b274 Removing -lshlwapi from line 100 of bitcoin-qt.pro (LIBS line) allows the build to complete. It is now running with a fresh (empty) datadir and seems to be syncing nicely. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 04, 2013, 02:39:44 AM Removing -lshlwapi from line 100 of bitcoin-qt.pro (LIBS line) allows the build to complete. It is now running with a fresh (empty) datadir and seems to be syncing nicely. Thanks, fixed. Seems I forgot testing building bitcoin-qt on Linux, after all tinkering to get the Windows build working again. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 12, 2013, 09:14:11 PM Quote from: Pieter Wuille The bottlenecks are now the crappy block download mechanism (which sometimes stops for several minutes) and signature verification (which is needed in practice, and is several orders of magnitude slower than all the rest combined). The last point can only be critical, if CPU-usage tends to 100% not the waiting time of the bitcoin client. The only checking my blockparser doesn't do is the merkeltree-hash of each block (I was still too lazy to do the coding) and this OP_CHECKSIG (resp. OP_CHECKMULTISIG), but I had read and understood its detailed description on https://en.bitcoin.it/wiki/OP_CHECKSIG. Thus I can't judge how cpu-intensive this OP_CHECKSIG evaluation really is.Indeed, roughly 99% of the CPU time is used for evaluation of all these OP_CHECKSIG (about 8 hours real time on my 2 GHz Athlon64-PC) when checking the total block chain. My (first) guess: Either we need a much faster implementation of the openssl function ECDSA_verify or abandon this OPCHECK_SIG. E.g. a hash computation (and its verification) only should be of equal security, but it is much faster than verifying a signature. Already some bitcoin-transactions in the block chain have used a "OP_HASH160 <14> OP_EQUAL"-script as txout-script. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 12, 2013, 10:40:46 PM Using hashes for signatures, where you reveal a secret to spend an output that contained it hash, you give the world the ability to steal your coins as soon as you spend them. For example, any miner could change the transaction to send it them self by using your revealed secret. This is an inherent problem with symmetric cryptography. You need some secret that only you know to prove your identity.
I have seen systems validate over 5000 tx/s (on a 6-core, hyperthreading machine with hal's optimized ECDSA code), so I think in the near future, we're good (at least until the block size limit would be lifted). Not every machine will be a full node, but if hardware to keep up and sync at least at over 1 block/s, is available to consumers, we probably don't have that much to worry about. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 12, 2013, 11:56:34 PM Using hashes for signatures, where you reveal a secret to spend an output that contained it hash, you give the world the ability to steal your coins as soon as you spend them. For example, any miner could change the transaction to send it them self by using your revealed secret. This is an inherent problem with symmetric cryptography. You need some secret that only you know to prove your identity. I c, Thx. But to spend something I need NOT to prove my identity principially. The problem in the bitcoin-model are the miners, because only this special subcommunity can put my transactions (or what ever they want) into blocks. Thus I must trust them not to change my transaction or need another algorithm (signature) to check/prove that theses did not change my transaction. This I had forgoten. :-/ I always feel uncomfortable by relaying on these miners in this bitcoin-world - I mean, to be unable to write my wanted transactions by myself into the block chain. Quote I have seen systems validate over 5000 tx/s (on a 6-core, hyperthreading machine with hal's optimized ECDSA code), so I think in the near future, we're good (at least until the block size limit would be lifted). Not every machine will be a full node, but if hardware to keep up and sync at least at over 1 block/s, is available to consumers, we probably don't have that much to worry about. Fine, and what about the guys still have half the CPU-power (or less) than me? :) I did a bit preciser estimation: I needed 160 sec to read the block chain up to height 21500 (and get all data in place), but then needed 1031 sec (the wrongly 1191 sec was including the 160 secs) to read and verify the next 1283 blocks (which contain a total of 838243 OP_CHECKSIG). Which shows a speed of 813 tx/s (not 704 tx/s) (script-verifications) -- far more realistic/typical than your 6-core hyperthreading parallel machine, IMHO.BTW: I believe the most cpu-time consuming part of the signature-verification is the modulo-n arithmetic (with n a 256-bit number). smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 12:02:17 AM Thus I must trust them not to change my transaction or need another algorithm (signature) to check/prove that theses did not change my transaction. Not just miners. Anyone relaying your transaction could modify it without invalidating it, with hash-locked transactions. Quote Fine, and what about the guys still have half the CPU-power (or less) than me? :) I did a bit preciser estimation: I needed 160 sec to read the block chain up to height 21500 (and get all data in place), but then needed 1191 sec to read and verify the next 1283 blocks (which contain a total of 838243 OP_CHECKSIG). Which shows a speed of 704 tx/s (script-verifications) -- far more realistic/typical than your 6-core hyperthreading parallel machine, IMHO. I was talking about the near future - not about this very moment. The fact that the hardware exists today means it is accessible for consumers, which is what matters (i.e., one doesn't need a big investment to run/maintain a validating node). Also, not everyone needs to run a validating node (but the more accessible we can make it to do so, the better). Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 01:35:50 PM I've uploaded a new set of builds of bitcoind/Bitcoin-Qt to my builds page (http://bitcoin.sipa.be/builds/pre-0.8). The new "turbo" build is based on current master on git, plus some more experimental patches (Hal Finney's optimized secp256k1 code (https://bitcointalk.org/index.php?topic=3238.msg45795#msg45795), and parallel script verification). The source code for these can be found here (https://github.com/sipa/bitcoin/tree/turbo) (.tgz archive here (https://github.com/sipa/bitcoin/archive/turbo.tar.gz)). I just have downloaded the current source version (1869678 bytes), "tar -xf bitcoin-turbo.tar.gz" it and call in the new created directory bitcoin-turbo the "qmake" . But gets lots of error messages about being unable to find all these local language files: Project MESSAGE: Building with UPNP support Project MESSAGE: Building with UPNP support lrelease warning: Met no 'TRANSLATIONS' entry in project file '/home/achim/Downloads/bitcoin-turbo/bitcoin-qt.pro' RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_bg.qm' RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_ca_ES.qm' RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_cs.qm' .... RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_zh_CN.qm' RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_zh_TW.qm' So, what are the exact preconditions/dependencies for compiling this turbo-bitcoin source successfully? (Are there linux-binaries availible?). I want to make a speed measurement of blockchain verification with this bitcoin-turbo release compared to my own blockparser. Currently I'm able to make 939.88 script-verification/sec with my 2 GHz AMD Athlon(tm) 64 Processor 3200+ stepping 00 processor (a few years old) using gcc 4.4.5 with (my default) options -march=athlon64 -O2. Thus I wonder, a new, say 3 GHz cpu with 6 cores should even be able to acomplish 10000 script-verification / sec or more today ... not to mention optimized secp256k1 code. Appendum: "qmake -v" results in QMake version 2.01a Using Qt version 4.7.2 in /usr/lib64/qt4 smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 03:02:29 PM I just have downloaded the current source version (1869678 bytes), "tar -xf bitcoin-turbo.tar.gz" it and call in the new created directory bitcoin-turbo the "qmake" . But gets lots of error messages about being unable to find all these local language files: Project MESSAGE: Building with UPNP support Project MESSAGE: Building with UPNP support lrelease warning: Met no 'TRANSLATIONS' entry in project file '/home/achim/Downloads/bitcoin-turbo/bitcoin-qt.pro' RCC: Error in 'src/qt/bitcoin.qrc': Cannot find file 'locale/bitcoin_bg.qm' .... So, what are the exact preconditions/dependencies for compiling this turbo-bitcoin source successfully? (Are there linux-binaries availible?). Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'. Quote Thus I wonder, a new, say 3 GHz cpu with 6 cores should even be able to acomplish 10000 script-verification / sec or more today ... not to mention optimized secp256k1 code. I've certainly seen more than that - 6000tx/s is over 12000txins/s. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 03:21:42 PM Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'. Seriously, when calling "qmake" in the directory $HOME/tmp/bitcoins/bitcoin-0.7.1-linux/src I get no errors but the message Project MESSAGE: Building with UPNP support Project MESSAGE: Building with UPNP support Removed plural forms as the target language has less forms. If this sounds wrong, possibly the target language is not set or recognized. I don't know whether this should sound wrong or not, but obviously something has changed regarding "bitcoin-qt.pro" compared to the old release 0.7.1. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 03:36:32 PM Building works fine on my laptop, but I believe I hit the same problem when trying to build release binaries for Linux in gitian. It may require the package 'qt4-linguist-tools'. That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 04:27:30 PM That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why. There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 05:02:30 PM There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now. A typo -- a single lost ". Unbelievable that this did not trigger a syntax error message in qmake. Thanks Pieter, and sorry that I have pressed you effectively to debug it for me. :-/ smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 05:58:31 PM Thanks Pieter, and sorry that I have pressed you effectively to debug it for me. :-/ I wonder whether I should point you to my next problem? "qmake" seems to finish fine and then I type "make" resulting in: .... make[1]: Leaving directory `/home/achim/Downloads/bitcoin-turbo/src/leveldb' cd /home/achim/Downloads/bitcoin-turbo; /bin/sh share/genbuild.sh /home/achim/Downloads/bitcoin-turbo/build/build.h fatal: Not a git repository (or any parent up to mount parent ) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). .... and a bit later it aborts in src/db.h:14:20: Error: db_cxx.h was not found. So I presume this bundled bitcoin-turbo.tar.gz assumes still some Git-structure/environment/dependency to work correctly. /usr/include/db4.8/db_cxx.h exists - but this is not on the /home filesystem and I dislike to test it on the (readonly) /usr filesystem. BTW: If you dislike that I'm able to compile your pre-release without Git-knowledge, no problem for me if you thus refuse to support me. :) Appendum: Meantime I have set/create an environment variable GIT_DISCOVERY_ACROSS_FILESYSTEM=1 , but this has no effect. Same failure when calling "make". smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 08:26:41 PM The error is harmless. It uses git to determine the current version, but it has a fallback in case that fails.
The db_cxx error means you don't have libdb4.8++-dev or libdb5.1++-dev installed. See doc/build-unit.txt for instructions. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 09:25:48 PM The error is harmless. It uses git to determine the current version, but it has a fallback in case that fails. Sorry, that I like to disagree. The error message is from the compiler or preprocessor (not linker) that it don't find the (source) file db_cxx.h (which is in /usr/include/db4.8 as I wrote).The db_cxx error means you don't have libdb4.8++-dev or libdb5.1++-dev installed. See doc/build-unit.txt for instructions. Meanwhile I discovered in bitcoin-qt.pro these comments: # Dependency library locations can be customized with: # BOOST_INCLUDE_PATH, BOOST_LIB_PATH, BDB_INCLUDE_PATH, # BDB_LIB_PATH, OPENSSL_INCLUDE_PATH and OPENSSL_LIB_PATH respectively and after inserting BDB_INCLUDE_PATH=/usr/include/db4.8 directly after these comments lines. It worked. Now it stops later complaining about src/net.cpp:18:32: Error: miniupnpc/miniwget.h: So it looks like I must check where all the needed PATHS really are and write it into bitcoin-qt.pro .... wondering that the default include PATH is not searched by qmake. Well, thanks to only 3 different used libraries I need hopefully at most to customize 6 paths. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 10:24:20 PM Dear Pieter
I'm a very little bit upset due to your quick and dirty patching of your current bitcoin-turbo/bitcoin-qt.pro file. ;) Why? After disabling UPNP-support via the line UPNP=- in the bitcoin-qt.pro file everything works fine until the final linkage. There I got the error message: g++ -static -fstack-protector-all -Wl,-O1 -Wl,-rpath,/usr/lib64/qt4 -o bitcoin-qt build/bitcoin.o build/bitcoingui.o ...... ...... -L/usr/X11R6/lib -lQtCore -lgthread-2.0 -lrt -lglib-2.0 -lpthread /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.5/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lboost_system But these library seems to be in the standard /usr/lib directory. Thus I thought either the line was too long (2486 characters) or some kind of order problem with the -L$LIBPATH -llibrary was the reason. After about a quarter hour -- I couldn't believe why g++ suddenly doesn't see these libraries -- my eyes catched at the beginning of the line: "-static " -- and remembered the diff of the new made bitcoin-qt.pro file from today and its predecessor in my downloaded *.tar.gz file. You have inserted # use static linking QMAKE_LFLAGS *= -static in the bitcoin-qt.pro newly! I removed the "-static" from the above g++ line and no problem to do the default dynamic library linkage, getting a 13 MB huge dynamical linked 64bit-executable bitcoin-qt! You should tell everybody, that you now expect all libraries to be statically linked! A new feature of the upcoming 0.8.0-release? ;) BTW: Of course I like to encourage (dispite of all my critic) to continue your valuable work. Best wishes, achim Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 11:08:05 PM Oh, that's not a change I made, though I should have caught it.
The -static was intended to only be added for Windows. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 11:09:52 PM Running full node means major consumption of upload bandwidth. With my 32 kB total upload, after 30+ nodes connect to me I can hardly surf the Internet, even very simple websites. Few people I know have even bigger problem because they actualy pay per GB transfered in any direction, so they stopped running full node. In such a setup I'd advise you to disable listening, as that will reduce the amount of nodes trying to fetch block data from you significantly. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 13, 2013, 11:45:00 PM Oh, that's not a change I made, though I should have caught it. The -static was intended to only be added for Windows. Because the bitcoin-turbo.tar.gz file was not changed which I downloaded after you just had announced the remove of the syntax error in bitcoin-qt.pro today in this thread, I looked into your git-repository and copied the new changed bitcoin-qt.pro and made a diff to my downloaded in the tar-ball. A last question for today (it got very late and I must get up tomorrow early): What is the x11-resource name of the color which backgrounds the "This is a pre-release ...." warning in the wallet-window and which controls the red of "(not synchron)" , such that I can change these specific colors? smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 13, 2013, 11:51:33 PM A last question for today (it got very late and I must get up tomorrow early): What is the x11-resource name of the color which backgrounds the "This is a pre-release ...." warning in the wallet-window and which controls the red of "(not synchron)" , such that I can change these specific colors? I have no clue at all about GUI stuff. Wladimir wrote that. Title: Re: Experimental pre-0.8 builds for testing Post by: Diapolo on January 14, 2013, 07:32:38 AM That's why I add that builds in gitian show the same problem, so I confirm there is a problem here. I just haven't spent the time investigating why. There was a syntactic error in the bitcoin-qt.pro file (which was fixed before, but it seems the fix got lost). Newer versions of lrelease didn't break on it, but older ones did. It should be fixed now. So it was related to that single " I mentioned a few days ago on Github, HA ;). Dia Title: Re: Experimental pre-0.8 builds for testing Post by: Diapolo on January 14, 2013, 07:56:51 AM Oh, that's not a change I made, though I should have caught it. The -static was intended to only be added for Windows. Sorry, that is fixed in the last revision of the stack-protector pull now :). Dia Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 09:22:25 AM Oh, that's not a change I made, though I should have caught it. Sorry, that is fixed in the last revision of the stack-protector pull now :).The -static was intended to only be added for Windows. Sorry, I deduced (wrongly) from the fact that if it are your files (URL https://github.com/sipa/bitcoin/tree/turbo) then at least you were responseable for this "-static" patch, sipa. Please remove this total misleading message: fatal: Not a git repository (or any parent up to mount parent ) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set) from the build-process, especially tragic if a different error occurs later which could have been founded (for an outsider) on this reason. And one more proposal for the next official bitcoin-release: Mention in the doc/build-unix.txt the possibility to customize theses 6 variables # Dependency library locations can be customized with: # BOOST_INCLUDE_PATH, BOOST_LIB_PATH, BDB_INCLUDE_PATH, # BDB_LIB_PATH, OPENSSL_INCLUDE_PATH and OPENSSL_LIB_PATH respectively in the bitcoin-qt.pro file. BTW: If noone made errors, nobody could learn. :) smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 11:04:32 AM Hi
Last night I only started this bitcoin-qt with an empty $HOME/.bitcoin to see whether it principle worked. :) Now the bitcoin-qt 0.7.99 is running first time seriously on my hardware (2 GHz Athlon-64 3200+ stepping 00 with a total of 4 GB ECC-Ram). Firstly (an experiment) I had set two softlinks to the existing blk0001.dat and blk0002.dat and started with "./bitcoin-qt -reindex -detachdb", but it immediately crashed without error-message (no bootstrap.dat exists). Next I built the new bootstrap.dat of size 4883237032 (block height 216283), deleting theses links to blk00*.dat and started "/usr/bin/time ./bitcoin-qt -detachdb -benchmark" Now it runs and seems to process all these blocks. After 15 min CPU-time consumed, it still had ~ 59000 left . I wonder why it is so slow, for signature-checkng too fast but still uses mostly 98% CPU-time, some drops from time to time, but I estimate still > 90% CPU-time on average. Periodcially disk-writes occurs, but because it has taken 3.3 GB system-cache this doesn't really matter to real-time behaviour. Currently, I just look, at remaining ~ 31600 blocks, it still processes as about 10 - 20 blocks /sec (difficult to estimate). I wonder what it is doing internally with such a high cpu-usage. Disk-reads are fairly sparse in real-time. ... to be continued (now 39 min CPU-time used and 23400 blocks left to be processed). BTW: System is a flavor of Linux 2.6.32-gentoo-r7 x86_64. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 12:44:13 PM block height 21000 was reached after 82 min cpu-time, currently it is at block height 214045 needed 128 min Cpu-time and got 93 % CPU-time in the real-time. And I discovered that the new blocks are stored in the sub-directory blocks together with rev*.dat files. I still have the feeling no signature-verification is done . It procesess currently with 439 tx/sec (no txin-count availible in debug.log). I think it will finish (reaching block height 216283) in the next 20 minutes when I'm at lunch. :)
smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 02:59:56 PM Finished:
/usr/bin/time ./bitcoin-qt -detachdb -benchmark Application asked to unregister timer 0x3a000004 which is not registered in this thread. Fix application. 8680.81user 338.18system 4:18:23elapsed 58%CPU Elapsed time is much too high, because it waits 1.5 hours until I finished it manually - 2 h 45 min real-time for loading the bootstrap.dat, 93.8 % CPU time looks right and about 12 minutes I/O-time until finish loading. It only needs 1-2 seconds to finish bitcoin-qt after I have clicked the Quit-Button (-detachdb no longer needed?). Total size of directory "blocks" is 5417516 kB containing 2*37 files (about 11% overhead, old blkindex.dat had 32% overhead compared to the blk00*.dat). Directory "coins" has size 164324 kB. I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat. Moreoever I see no effect of this "-benchmark" option (maybe this ".....Fix application." message?) BTW: I run it with "maxconnections=0" set in the bitcoin.conf file such that at the end of the load no further action was done. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 14, 2013, 08:31:53 PM I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat. I have never claimed otherwise. That is not the bottleneck now, however (crappy block download and signature validation are). Quote Moreoever I see no effect of this "-benchmark" option (maybe this ".....Fix application." message?) It should report block connect/validation speeds in debug.log. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 08:50:02 PM Hi
There is highly probable a bug in the "-reindex" option called code. After updating with the bitcoin-qt-0.7.99 to the current main blockchain , this works fine, I restarted it again: ./bitcoin-qt -maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark and it crashed after a few seconds without any error message. Here is the tail of the debug.log 2013-01-14 20:37:15 Opened LevelDB successfully 2013-01-14 20:37:15 block index 51ms 2013-01-14 20:37:15 Loading wallet... 2013-01-14 20:37:15 nFileVersion = 79900 2013-01-14 20:37:16 wallet 976ms 2013-01-14 20:37:16 Loading addresses... 2013-01-14 20:37:16 Reindexing block file blk00000.dat... 2013-01-14 20:37:16 Loaded 10094 addresses from peers.dat 40ms 2013-01-14 20:37:16 mapBlockIndex.size() = 0 2013-01-14 20:37:16 nBestHeight = -1 2013-01-14 20:37:16 setKeyPool.size() = 100 2013-01-14 20:37:16 mapWallet.size() = 0 2013-01-14 20:37:16 mapAddressBook.size() = 1 2013-01-14 20:37:16 Done loading 2013-01-14 20:37:16 refreshWallet Maybe you get an idea. BTW: Could you explain the new format of these blk*.dat files (and if easy, the rev*.dat) files or is this anywhere already documented? Appendum: I just want to give you more debugging info and therefore started "gdb ./bitcoin-qt" and then inside run "-maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark" and it runs ... :-/ smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Diapolo on January 14, 2013, 10:00:37 PM You mean -dbcache, right? I also observed a crash while reindexing with current master, but I didn't yet take a closer look at it.
Dia Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 10:37:27 PM You mean -dbcache, right? I also observed a crash while reindexing with current master, but I didn't yet take a closer look at it. I don't know, but I believe :)I used the same options as bitcoin-qt.exe -reindex -par=4 -dbcche=500 -logtimestamps -benchmark except this -par=4smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 10:54:09 PM and it runs ... 2013-01-14 22:38:44 Reindexing finished:-/ 98:20 min CPU-time until block-height 216823 and 99:16 min until 21650 height (the maxium no of blocks in the blk*dat files). I lookup up in the debug.log : Start at 2013-01-14 20:42:55 which means 116:11 min total real-time or 85.4 % CPU-time. Compare this with the bootstrap.dat of 216823 blocks using 150 min CPU-time. It should be about equal - but I did a simple trick. *gg* I recompiled the turbo-bitcoin/sources with the g++ additional options -march=athlon64 -O2 :-) Nearly a boost of 50% in speed up! I wonder why there is also in the official sources not at least a -O2 option for the g++-compiler. Everything will profit from -O2. BTW: This "./bitcoin-qt -maxconnections=0 -reindex -dbcche=500 -logtimestamps -benchmark" (also with -dbcache inplace of -dbcche) crashes the client reproducable after start. Sadly not, if invoked with gdb. :-( smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 14, 2013, 11:19:46 PM I just notice this reindex-bug is a really nasty bug. If I restart bitcoin-qt next (only with -maxconnection=0) it starts immediately with reindexing. :-(
I quite it and start it again, it starts reindexing ... Lots of lines "ProcessBlock: ORPHAN BLOCK, prev=0000000000000640dd32c57bbda456631968a2d57ccc3d40190c767ab2ce7e99" now writen to debug.log. Ups! debug.log was truncated by bitcoin-qt -- only 77030 lines of these "ProcessBlock: ORPHAN BLOCK, prev=...." inside now. Well, I better should throw all away now .... and wait for Pieter to debug it. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: 2112 on January 14, 2013, 11:55:35 PM Sadly not, if invoked with gdb. :-( I just wanted to remind you about "ulimit -c unlimited" to be able to meaningfully use the debugger after the crash. You're probably running with the default of "ulimit -c 0".Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 15, 2013, 10:52:07 AM Sadly not, if invoked with gdb. :-( I just wanted to remind you about "ulimit -c unlimited" to be able to meaningfully use the debugger after the crash. You're probably running with the default of "ulimit -c 0".1) my ulimit has no -c option (only a -f option) on Linux (not only on my gentoo, but also on Ubuntu 10.2 I just checked) 2) called "ulimit" gives the default of unlimited but most important 3) There occurs no crash in gdb, thus it is not meaningfull (in my context) to talk about "use the debugger after the crash". ;) BTW: I just discovered the ulimit with many options on SunOS. "ulimit -c" sets the "core" limit of the process there. I wonder what core precisely is meant. Anyway, thanks for your thoughts smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 15, 2013, 11:15:53 AM I think there is still a lot of room for speed-optimization of the load of the initial bootstrap.dat. I have never claimed otherwise. That is not the bottleneck now, however (crappy block download and signature validation are).Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt. You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format! I also dislike to see it to be replaced (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and blktree-diretory?), if these stay undocumented or to be over-complex. Again making it impossible to use directly this raw block chain data. I think I have shown with my block-parser, that conversion of this data into a bitcoin-qt usable structure can be done only in a few minutes on a typical hardware and not in more than 20 times real time for bitcoin-turbo and checking everything inside the main block chain except the script-verification! Yes, before, 0.7.?, it were more than 200-times of real time, so a great achievement of your new experimental 0.7.99 version. Please remember: I only started activity in this Bitcoin Forum, because the VERY SLOW initialization with the main block chain by the (most spread) bitcoin-qt client might put off increasingly many want-to-be starters into the bitcoin world. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 15, 2013, 11:45:46 AM 3) There occurs no crash in gdb, thus it is not meaningfull (in my context) to talk about "use the debugger after the crash". ;) I got another idea. You could compile the source with the "-g" flag and then start your client as normal with the "-reindex"-option hoping still(!) the crash occurs and force the OS to produce a core-file (image of all his allocated memory into a file). Then you could investigate this core-file with gdb afterwards.But I think, before realizing these thoughts, at first sipa must tell us his oppinion in this "-reindex" case. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: 2112 on January 15, 2013, 12:28:28 PM I wonder what core precisely is meant. I apologize I wrote like everyone knows and understands the postmortem debugging.$ cat bitcoind.c #include <stdio.h> int main() { return printf("Hello %s!\n",2112); } $ make bitcoind $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited $ ulimit -c unlimited $ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited $ ./bitcoind Segmentation fault (core dumped) $ ls core* core $ gdb ./bitcoind core GNU gdb (GDB) 7.1-ubuntu Reading symbols from /home/2112/bitcoind...(no debugging symbols found)...done. Reading symbols from /lib/libc.so.6...done. Core was generated by `./bitcoind'. Program terminated with signal 11, Segmentation fault. #0 0x00007f4f24cd7e32 in _IO_vfprintf_internal (s=0x7f4f2500c780, format=<value optimized out>, ap=0x7fffcba92680) at vfprintf.c:1617 1617 vfprintf.c: No such file or directory. in vfprintf.c (gdb) where #0 0x00007f4f24cd7e32 in _IO_vfprintf_internal (s=0x7f4f2500c780, format=<value optimized out>, ap=0x7fffcba92680) at vfprintf.c:1617 #1 0x00007f4f24cdf65a in __printf ( format=0xffffe906 <Address 0xffffe906 out of bounds>) at printf.c:35 #2 0x000000000040053f in main () (gdb) So on a reasonable OS program doesn't have to be compiled with debugging and/or run under the debugger to approximately locate the place where it crashed. I think that it is a terrible legacy of Microsoft (and/or Apple) that various Linux distributions switched to "disable core dumps by default". The old "remove core dumps older than a week by default" was much better, because nobody would use the "program crashes, but not under the debugger" excuse. Again, I want to apologize for writing like everyone already knows and just needed a reminder. Hopefully there will be at least one reader who wasn't familiar with the postmortem concept who now will learn how to override the default in their Linux installation to re-enable to perform good old core dumps. Nowadays core memory can probably be found only in museums, but the core dumps are still useful. Lets debug like it is 1970 again! Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 15, 2013, 02:59:05 PM I wonder what core precisely is meant. I apologize I wrote like everyone knows and understands the postmortem debugging.1) I did not grasp your idea, that I must/should force the OS to produce this core-file. 2) You seem to have a different environment (I even don't know your OS), where ulimit has this -c option. 3) We have the rare case, that a program which crashes when invoked by the shell doesn't crash when invoked by gdb. BTW: I still remember well, when , about 20 years ago, nearly every crash of a program left a core file. Thanks, smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 15, 2013, 03:39:22 PM I think that it is a terrible legacy of Microsoft (and/or Apple) that various Linux distributions switched to "disable core dumps by default". The old "remove core dumps older than a week by default" was much better, because nobody would use the "program crashes, but not under the debugger" excuse. The main problem is people don't express precisely what they are talking of or assume environments/dependencies others don't have/use. :-/When checking for "ulimit" on my sytem/environment , I did "man ulimit" and later "which ulimit". So I got my informations (also for ubuntu & SunOS) from the man-pages and from the which/PATH-search. So, should I apologize for not doing "whence ulimit", but "which ulimit", don't I? ulimit (as typed in the command-line) is a shell-built-in (if you use an appropriate shell!) with all these many options, including option -c for size of core-limit, even in (my) Linux (if you use sh or ksh or bash, but not e.g. in csh or tcsh). Appendum: There exists also the (obsolete) system/library-call ulimit, see "man ulimit 3". smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 15, 2013, 09:28:37 PM Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt. You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format! I'm not sure what you mean. The current downloading mechanism often takes hours just downloading (not including any verification or storage), a lot of which is waiting and not doing anything. Compare that to (order of magnitude) an hour of verification to reach block 210000 (without signature checks). After block 210000, signature checking becomes the bottleneck. So if we want to improve the experience for new users, the first step should be speeding up the download itself, and the signature verification. After that, we can have a look at further improvements for the verification itself. Quote I also dislike to see it to be replaced (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and blktree-diretory?), if these stay undocumented or to be over-complex. Again making it impossible to use directly this raw block chain data. It's being replaced to improve performance. The file format of the block files (blk000?.dat before, blocks/blk000??.dat now) hasn't changed: it's a binary concatenation of blocks in network format. The coins directory is a LevelDB database in a cusitom compact format that represents the set of unspent transaction output. See the source code (specifically: main.h above, class CCoins) for the exact specification. The rev* files contain data to be able to disconnect blocks afterwards in case of reorganisation. Bitcoin needs exclusive access to the database anyway, so everything except the block files isn't intended to be used by third party code. If you need access to the raw blockchain/utxo data, use the getblock, getrawtransaction and gettxout RPC commands. Quote I think I have shown with my block-parser, that conversion of this data into a bitcoin-qt usable structure can be done only in a few minutes on a typical hardware and not in more than 20 times real time for bitcoin-turbo and checking everything inside the main block chain except the script-verification! Yes, before, 0.7.?, it were more than 200-times of real time, so a great achievement of your new experimental 0.7.99 version. My priority is correctness/completeness of the verification, consistency of the database on disk, and minimizing the working set size. That on itself leads to improved performance, but there are certainly a lot of things possible to improve it. As said, not a priority now. Yes if you do everything in RAM using a hash map, I'm sure it will be faster (it should!). Quote Please remember: I only started activity in this Bitcoin Forum, because the VERY SLOW initialization with the main block chain by the (most spread) bitcoin-qt client might put off increasingly many want-to-be starters into the bitcoin world. I'm very aware of the slowness, but changing in the reference code things takes time (to implement, to review and to test) - this slows development down, but I'm sure the community wouldn't want even a blazingly fast release with subtle bugs in it that lead to a split of the network after a large part of the community adopts it. We do what we can, and most of us (including me) just do this in our free time. The real bottleneck is testing, so thanks for helping out here (though I really can't do much about a segfault without more details...). Oh and by the way, use `qmake RELEASE=1` to build optimized binaries. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 15, 2013, 11:44:49 PM Hi
I should lead-in by saying, "sorry that I pressed you to spend so much real time in this reply - only for me". :-/ Sorry, but I think I should disagree strongly! This might to be an (or better the!) important bottleneck for each newbie who want to start to use bitcoin-qt. You are true, it is no bottleneck for you and probably for > 95% of all forums posters here who has done already the initial download of the blockchain data. "crappy block download and signature validation" becomes the next bottle neck AFTER you have your personal block chain in a bitcoin-client-conform format! I'm not sure what you mean. The current downloading mechanism often takes hours just downloading (not including any verification or storage), a lot of which is waiting and not doing anything. Compare that to (order of magnitude) an hour of verification to reach block 210000 (without signature checks). After block 210000, signature checking becomes the bottleneck. So if we want to improve the experience for new users, the first step should be speeding up the download itself, and the signature verification. After that, we can have a look at further improvements for the verification itself. You (the newbie) can get the download with maximal speed (his band-width limitation) by simply download a trusted bootstrap.dat! It can't be faster. It is our responsibility to provide in regular (short) time distances these block chains. E.g. each week. So simply! BTW: I did load bootstrap.dat of the main blockchain until height 216283 once again with your 0.7.99 release (and compiled with -march=athlon64 -O2) and got 36.5 minutes cpu-time until height 210000 and 96.5 minutes cpu-time until height 216283 which uses a total of 108 min real-time. (My testing of this -reindex-bug had finally corrupted my blockchain data). The last 6283 blocks needed exactly 1 h cpu-time or 0.57 sec/block. But better comparing transaction counts not block counts. The first 9344206 tx until block height 210000, could be loaded from bootstrap.dat into the bitcoin-qt data structures with 4266 tx/sec the remaining 1731667 transactions with 481 tx/sec (due to verification checks). Thus, if there would have been a new bootstrap.dat until block height 216283, total real-time would have been only 43 min. Well, my blockparser needs less than 2 min (without sig-checks yields > 92000 tx/sec) , but maybe your deletion of redeemed deposits is expensive (but I doubt). So I guess there is surely a factor 20 (or more) which could be improved (I did no aggressive coding in my parser to reduce cpu or real time). And I already praised you -- you are faster already by a factor 10 (or even a bit more) compared to version 0.7.1. :) BTW: Writing the data new to disk, needs neglectable cpu-time and no further real-time (I see it at the running 0.7.99 because it is done in parallel) , but this I can't do as long as I neither know the structure of blkindex.dat nor of your new structured files in 0.7.99 (I asked for). (A following item). Sorry, it becomes too late in the night. I will continue tomorrow, discussing the/your following items/responses. Regards, smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 16, 2013, 04:02:18 PM 2nd part:
Quote I also dislike to see it to be replaced (the currently undocumented "blkindex.dat"-format) by new formats in rev*dat, blk*dat, coins-directory (and blktree-diretory?), if these stay undocumented or to be over-complex. Again making it impossible to use directly this raw block chain data. It's being replaced to improve performance. The file format of the block files (blk000?.dat before, blocks/blk000??.dat now) hasn't changed: it's a binary concatenation of blocks in network format. The coins directory is a LevelDB database in a cusitom compact format that represents the set of unspent transaction output. See the source code (specifically: main.h above, class CCoins) for the exact specification. The rev* files contain data to be able to disconnect blocks afterwards in case of reorganisation. Bitcoin needs exclusive access to the database anyway, so everything except the block files isn't intended to be used by third party code. If you need access to the raw blockchain/utxo data, use the getblock, getrawtransaction and gettxout RPC commands. I did (already yesterday) a "cmp $HOME/.bitcoin.0.7.99/blocks/blk00000.dat $HOME/.bitcoin/blk0001.dat" which results in the (first) difference at byte 134214364. So they can not be simply concatenated to get the block-chain back, right? But maybe I have discovered a bug in the data structure/files. Well, "everything except the block files isn't intended to be used by third party code", together with "It's being replaced to improve performance" this both sounds like old Mircosoft-windows argumentation. :-/ "Bitcoin needs exclusive access to the database" Huch? only if it is run - but this line of argumentation matches the previous. BTW: Why do I sound so negatively? Because I must think of my horrible long initialiation of the block chain which lasts far more than 23 h real-time last december and you tell me that it is no bottleneck. Sorry, to be frank. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 16, 2013, 04:17:34 PM I did (already yesterday) a "cmp $HOME/.bitcoin.0.7.99/blocks/blk00000.dat $HOME/.bitcoin/blk0001.dat" which results in the (first) difference at byte 134214364. So they can not be simply concatenated to get the block-chain back, right? But maybe I have discovered a bug in the data structure/files. The old files are limited to (almost) 2GB. The new files are limited to 128MiB. Apart from that, they are exactly the same format, they just store fewer blocks per file. If you concatenate either (in the right order), you get a usable bootstrap.dat Quote Well, "everything except the block files isn't intended to be used by third party code", together with "It's being replaced to improve performance" this both sounds like old Mircosoft-windows argumentation. :-/ "Bitcoin needs exclusive access to the database" Huch? only if it is run - but this line of argumentation matches the previous. Obviously you can use it for whatever you like while Bitcoin isn't running. I just mean that compatibility with external applications was not a design priority (except for the block files - they could have been made smaller as well, but we chose not to, in order not to break external applications). Quote BTW: Why do I sound so negatively? Because I must think of my horrible long initialiation of the block chain which lasts far more than 23 h real-time last december and you tell me that it is no bottleneck. Sorry, to be frank. You're sort of hijacking this thread, ranting about things that are not priorities now. In 0.7.x, block verification WAS the bottleneck (except when you were on SSD or a tmpfs), and that is the reason for all changes that will be in 0.8, and what this thread exists for. I'm just saying that after those, it isn't. Block downloading and signature verification are the bottleneck. And yes, using bootstrap.dat for initialization avoids the problem with slow block downloading, but that's not what you want to be telling new users. The priority should be making automatic syncup usable again. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 16, 2013, 04:42:07 PM 3rd part:
I think I have shown with my block-parser, that conversion of this data into a bitcoin-qt usable structure can be done only in a few minutes on a typical hardware and not in more than 20 times real time for bitcoin-turbo and checking everything inside the main block chain except the script-verification! Yes, before, 0.7.?, it were more than 200-times of real time, so a great achievement of your new experimental 0.7.99 version. My priority is correctness/completeness of the verification, consistency of the database on disk, and minimizing the working set size. That on itself leads to improved performance, but there are certainly a lot of things possible to improve it. As said, not a priority now. Yes if you do everything in RAM using a hash map, I'm sure it will be faster (it should!). "completeness of the verification" let the user decide what degree he wants. There is an option -checklevel (introduced by yourself, add e.g. a 7 for signature-check if you like, or whatever you like personally to fine-tune control the verification process when block loading) -- I tried do do all these with my blockparser I know bitcoin-0.7.99 does when bootstraping data (to get a fair real-time comparison). But initialization with the bootstrap.dat which you got from a trusted source, should need no (deep?) verification to judify long real-times! "As said, not a priority now" -- well I present you quantitative real-time measurement and you judge this is of no importance. "Yes if you do everything in RAM using a hash map, I'm sure it will be faster (it should!)." Sorry, your 0.7.99 with bootstrap.dat also (could/should?) do everything in RAM, and I wrote already: the write back of the new data-structure in 0.7.99 happens in parallel to the CPU on my system, so no (a very, very negligible) real-time win possible, if no writing back to disk occurs. Your qualitative comment "it will be faster (it should!)", I already estimated quantitatively to be > 10 to give the true magnitudes. :-( Yes, you can argue, that minimizing your working set has highest priority, and then I should shut up (I use about 2.3 GB totally, because I store all txout-scripts in memory, you need only to keep the unspent in memory, 1/7.5 or less). BTW: I discovered today I can simply improve my performance by about 15 % if I only suppress all the statistics output for each block. But then my blockparser is limited by the disk-access speed when reading the bootstrap.dat, checking and building up the data structures, not the CPU-speed anymore. smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 16, 2013, 04:52:08 PM I did (already yesterday) a "cmp $HOME/.bitcoin.0.7.99/blocks/blk00000.dat $HOME/.bitcoin/blk0001.dat" which results in the (first) difference at byte 134214364. So they can not be simply concatenated to get the block-chain back, right? But maybe I have discovered a bug in the data structure/files. The old files are limited to (almost) 2GB. The new files are limited to 128MiB. Apart from that, they are exactly the same format, they just store fewer blocks per file. If you concatenate either (in the right order), you get a usable bootstrap.dat Sorry, I get tired, I just wanted to reply with the 4th, last part, but it looks superflously. Therefore only one obviously point: even if you don't know what "cmp" do, and what you can conclude from my cited, given output, the sum of bytes in all these blocks/blk*.dat files is larger than the original bootstrap.dat. So you tell us false. (and because all except the last are 2^27 bytes in size) you even need not to add up, only consider size modulo 2^27 of bootstarp.dat for a check). smtp Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 16, 2013, 04:56:17 PM "completeness of the verification" let the user decide what degree he wants. There is an option -checklevel (introduced by yourself, add e.g. a 7 for signature-check if you like, or whatever you like personally to fine-tune control the verification process when block loading) -- I tried do do all these with my blockparser I know bitcoin-0.7.99 does when bootstraping data (to get a fair real-time comparison). But initialization with the bootstrap.dat which you got from a trusted source, should need no (deep?) verification to judify long real-times! -checkblocks is about how much verification you want at startup about the data you already have. I find disabling checks when importing a bootstrap.dat very risky - someone distributing "trusted" files which have an accidental (or deliberate!) error in them is enough to cause a block chain split, which in case of widespread deployment could be devastating for the bitcoin economy. If there was something significant to be gained, it could be made optional with a flag with a big warning on that, but I doubt it will help much (I don't have numbers, and you're welcome to do benchmarks yourself if you want to help), as most time is maintaining the database and not doing checks. Quote "As said, not a priority now" -- well I present you quantitative real-time measurement and you judge this is of no importance. "Yes if you do everything in RAM using a hash map, I'm sure it will be faster (it should!)." Sorry, your 0.7.99 with bootstrap.dat also (could/should?) do everything in RAM, and I wrote already: the write back of the new data-structure in 0.7.99 happens in parallel to the CPU on my system, so no (a very, very negligible) real-time win possible, if no writing back to disk occurs. Your qualitative comment "it will be faster (it should!)", I already estimated quantitatively to be > 10 to give the true magnitudes. :-( Yes, thank you very much for those numbers. I know there are performance improvements possible, I've told you a zillion times now. This thread is about doing tests, so we can get 0.8 out as soon as possible, because 0.7.x's performance is terrible, and it won't be released before it is stable. Can we please stop this discussion, and get this thread back to test results? Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 16, 2013, 04:58:59 PM Sorry, I get tired, I just wanted to reply with the 4th, last part, but it looks superflously. Therefore only one obviously point: even if you don't know what "cmp" do, and what you can conclude from my cited, given output, the sum of bytes in all these blocks/blk*.dat files is larger than the original bootstrap.dat. So you tell us false. (and because all except the last are 2^27 bytes in size) you even need not to add up, only consider size modulo 2^27 of bootstarp.dat for a check). You make the wrong assumption that because the file format is the same, the concatenation will be binary identical. In the 0.8 code, the files are pre-allocated in blocks of 16 MiB, to avoid high fragmentation. That means that at the end of those files, you'll have some zero bytes. So the concatenated files will have these zeroes too, and you'll get a slightly larger bootstrap.dat. Both the loading code in 0.7 and 0.8 should be able to deal with holes in the files, though. See, I didn't lie. Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 16, 2013, 05:01:46 PM And yes, using bootstrap.dat for initialization avoids the problem with slow block downloading, but that's not what you want to be telling new users. Are you joking? Your whole attitude and the reality (0.7.99 is experimental!) press new people to use bootstrap.dat and this will increase as longer you wait.And if a newbie complains to me about long first initialization, of course I will point them to bootstrap.dat method, who won´t? smtp Title: Re: Experimental pre-0.8 builds for testing Post by: smtp on January 16, 2013, 05:06:01 PM See, I didn't lie. Then you have your own definition of "concatenation". But the reader of these postings can judge by themselves.smtp Title: Re: Experimental pre-0.8 builds for testing Post by: notme on January 16, 2013, 05:14:02 PM And yes, using bootstrap.dat for initialization avoids the problem with slow block downloading, but that's not what you want to be telling new users. Are you joking? Your whole attitude and the reality (0.7.99 is experimental!) press new people to use bootstrap.dat and this will increase as longer you wait.And if a newbie complains to me about long first initialization, of course I will point them to bootstrap.dat method, who won´t? smtp It took my machine less than 4 hours to download from scratch, and most of that was waiting on peers to send the data. What is the problem? Really, if you don't have any bug reports or testing results, can you please start a new thread to discuss potential future improvements. This thread is for testing 0.8 so that it can be released sometime this decade. Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on January 16, 2013, 06:00:57 PM Are you joking? Your whole attitude and the reality (0.7.99 is experimental!) press new people to use bootstrap.dat and this will increase as longer you wait. And if a newbie complains to me about long first initialization, of course I will point them to bootstrap.dat method, who won´t? Of course. I'll point them to bootstrap.dat now too. But that doesn't mean we don't have to fix the real problem (the fact that the built-in block download code is very crappy). If things were working as they should, we wouldn't need to tell new users about bootstrap.dat (although it may still be useful). My attitude? I don't want to be responsible for a network fork. The purpose of the reference client is implementing a full node, and that means zero trust: it will verify all incoming data. If you can't live with the resource requirements that are needed for that, and consider trusting some instance to do checking for you, you're better off using a lightweight client (which only checks block headers - see SPV mode in Satoshi's paper), as those don't need trust in the data source and don't risk creating a network fork if the trusted data is wrong. I think everything that had to be said, is said now. You may disagree with me, but if so, please take it elsewhere. I won't respond anymore to messages here which are not about getting 0.8 tested. Title: Re: Experimental pre-0.8 builds for testing Post by: niko on January 16, 2013, 07:14:22 PM Is there anything in particular we should try with the latest build (Jan 15)?
Title: Re: Experimental pre-0.8 builds for testing Post by: notme on January 16, 2013, 07:15:44 PM Is there anything in particular we should try with the latest build (Jan 15)? Try loading from scratch with an empty data directory. It's fast :). Title: Re: Experimental pre-0.8 builds for testing Post by: niko on January 18, 2013, 06:36:25 AM Tried again the low-end PC with the latest build:
Windows XP SP3 Pentium M, 1.6 GHz 760 MB RAM wired LAN connection, bitcoin-qt.exe -connect=192.168.0.xxx -logtimestamps -benchmark Took about nine hours. Here are the ~30-minute excerpts from debug.log: Code: 2013-01-17 15:06:47 Bitcoin version v0.7.1-353-g3afefd8-beta (2013-01-14 23:38:10 +0100) Title: Re: Experimental pre-0.8 builds for testing Post by: Gabi on January 20, 2013, 05:48:23 PM I'm trying this client, downloaded it yesterday. I don't have precise data since i didn't run it continuosly but i can say that it was much much faster and much less heavy on hard disk. I started it without a blockchain and let it download from peers, yes i know, not optimal but it worked well. Memory usage is a bit high, it was at 600MB during the chain download, now it's at like 400 but it's fine
Title: Re: Experimental pre-0.8 builds for testing Post by: mb300sd on January 22, 2013, 01:12:24 AM Running 2 nodes, everything seems to work and downloading is definitely faster
Title: Re: Experimental pre-0.8 builds for testing Post by: bracek on January 22, 2013, 03:57:17 PM downloaded bitcoin-0.7.1-353-g3afefd8-win32-setup
win 7, 64 bit i7 8 gb ram laptop uninstalled version 0.7.2. the last one from bitcoin.org I did not delete folder or any files from where older version was installed this bitcoin-0.7.1-353-g3afefd8-win32-setup it sees my wallet as encrypted and locked (like it should be) started at 15:43 ventilator is not silent, but moderate, with ocasional overdrives that last 5 to 10 seconds the message says "reindexing blocks on disk" it never went over 8 active connections, I connect to internet through nearby hotspot, map port using upnp is checked by default, connect through socks proxy was unchecked by default i did not touch that at all I don't know how to portforward bitcoin and never did that, but I noticed the green bar on the last released version 0.7.2, and it oscilated between 19 and 36 connections the other day when I looked 15:53 --> 168000 blocks 16:03 --> 193500 blocks 16:13 --> 209300 blocks ventilator started spinning loudly non-stop after 200k blocks 16:23 --> 213000 blocks blockchain was last updated on my disk with version 0.7.2 , a few days ago, so last 300 will be updated with this new version, 16:33 --> 216200 blocks ventilator did not slow down from block 200k 16:37 --> 500 blocks remaining 16:38 --> 239 blocks remaining, message changed to "synchronizing with network" ventilator slowed down significantly, almost silent, ocasional overdrives of few seconds 16:41 --> 95 blocks remaining 16:43 --> 10 blocks remaining looks like it is ok I shut it down with end process tree after 15 sec i started it again it took about 30-35 seconds to open gui connected to someone within 5 seconds. I dont like the idea of sending coins into it. armory 0.87 16:47 started scanning blockchain 16:50 armory shows the same number of blocks as bitcoin gui... ------------------------------------------------------------------------------------------- bitcoin was turned off for at least half an hour before restart, after computer restart, I manually started bitcoin (option to start with windows is turned off) it got stuck at "verifying block database integrity" and after a few minutes error message prompted on screen "corupted block database detected please restart the client with -reindex" how to even do that ? :) I guess I will delete all from bitcoin folder and start actual full blockchain download ------------------------------------------------------------------------------------------- laptop was turned off after start, bitcoin opened with no problems ------------------------------------------------------------------------------------------ today I deleted all files from bitcoin folder, started download after 15 minutes it processed 102k blocks after 30 min 120k blocks after 1 hour 133k 90 min 139k 2 and half hours 155k 4 hours 185k 6 hours 206k 7 hours 214k finished exactly after 8 hours I started armory 0.87 offline icon from start menue while bitcoin still had 350 blocks to verify I pressed a button in armory to go online, it started scanning the blockchain, finished showed current block count, and it said it is connected, but watching wallets had balances -- btc and I was not able to create unsigned transactions I don't know if this is armory or bitcoin problem after I turned armory off, and started the regular icon of armory (not the one with word offline in name) armory scanned and connected and balances were in place, and I was able to create unsigned transactions --------------------------------------------------------------------------------------------------------------- today again bitcoin "stopped responding" on startup, nothing unusual was done yesterday, or forced shutdown, or settings changed but, after a minute it did actually start, connected, loadaded blocks ----------------------------------------------------------------------------------------------- Title: Re: Experimental pre-0.8 builds for testing Post by: inglian on January 24, 2013, 12:49:21 AM Great results with bitcoin-0.7.1-353-g3afefd8-win32
bootstrap.dat was in place. I didn't see much multithreading, but the blockchain bootstrap and update ran fast. debug.log excerpts: Code: 2013-01-23 03:11:54 SetBestChain: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 work=8590065666 tx=2 date=2009-01-09 02:54:252013-01-23 03:44:50 Worked fine with a small transaction in and back out. Title: Re: Experimental pre-0.8 builds for testing Post by: niko on January 24, 2013, 05:24:02 AM Obviously, the new code offers a great improvement in the speed of initial verification. New users can now set up a full node and start using it much faster then opening and verifying a PayPal account. Great.
I have to ask now - excuse my ignorance - what is the worst thing that could happen if there is a mistake or an oversight in the new algorithm? Obviously, block hashes are identical to those from the current reference client. Does this mean all unspent coins are safe and secure? Is there any serious risk in moving forward with 0.8? Title: Re: Experimental pre-0.8 builds for testing Post by: dansmith on January 31, 2013, 12:36:44 PM Tested bitcoin-0.7.1-353-g3afefd8-linux from scratch on a low-spec Ubuntu 12.04 w/ 0,9GHz Celeron M CPU & 1GB ram on ext4 on external USB2.0 HDD.
It took me 2 computer resets, because bitcoin-qt would gobble up all RAM bringing the machine to a non-responsive state. Approximately 48 hours is what I needed to finish syncing from scratch. CPU was loaded @ 60% throughout. I always had 8-9 connections on a fast internet link. Datadir weighed in at 6,2 GB. The bright side is that bitcoin-qt now shuts down in 3 secs. (the GUI still takes 2+ mins to show up when launching bitcoin-qt) Title: Re: Experimental pre-0.8 builds for testing Post by: notme on February 01, 2013, 08:00:15 AM Tested bitcoin-0.7.1-353-g3afefd8-linux from scratch on a low-spec Ubuntu 12.04 w/ 0,9GHz Celeron M CPU & 1GB ram on ext4 on external USB2.0 HDD. It took me 2 computer resets, because bitcoin-qt would gobble up all RAM bringing the machine to a non-responsive state. Approximately 48 hours is what I needed to finish syncing from scratch. CPU was loaded @ 60% throughout. I always had 8-9 connections on a fast internet link. Datadir weighed in at 6,2 GB. The bright side is that bitcoin-qt now shuts down in 3 secs. (the GUI still takes 2+ mins to show up when launching bitcoin-qt) Dude, memory is way too cheap to put yourself through the torture of running a machine with 1 GB of ram. Title: Re: Experimental pre-0.8 builds for testing Post by: BkkCoins on February 03, 2013, 03:24:03 AM Dude, memory is way too cheap to put yourself through the torture of running a machine with 1 GB of ram. Unless you're paying for a server such as a VPS. In this case every MB of memory pushes the monthly cost up. I keep getting crashes on my bitcoind install running an Electrum server. From the debug log: Code: 02/03/13 02:35:30 CTxMemPool::accept() : accepted a3fbde782d (poolsz 713) This seems to happen every few days for me right now. Is there a memory leak that is causing eventual depletion? Title: Re: Experimental pre-0.8 builds for testing Post by: niko on February 03, 2013, 03:35:52 AM ^^^I've installed and verified pre-0.8 from scratch several times on a 768 MB fossil running WinXP (see earlier posts). Never got close to running out of RAM. IIRC peak usage was 190 MB. Might be a problem specific to Linux?
Title: Re: Experimental pre-0.8 builds for testing Post by: dserrano5 on February 03, 2013, 01:29:20 PM Dude, memory is way too cheap to put yourself through the torture of running a machine with 1 GB of ram. Unless you're paying for a server such as a VPS. In this case every MB of memory pushes the monthly cost up. I keep getting crashes on my bitcoind install running an Electrum server. From the debug log: Code: 02/03/13 02:35:30 CTxMemPool::accept() : accepted a3fbde782d (poolsz 713) This seems to happen every few days for me right now. Is there a memory leak that is causing eventual depletion? My "memory" problems in my VPS turned out to be not related to the available RAM, but to the number of running processes/threads (I have a limit of 128 threads). You may be finding the same issue. Title: Re: Experimental pre-0.8 builds for testing Post by: notme on February 04, 2013, 06:34:31 AM Dude, memory is way too cheap to put yourself through the torture of running a machine with 1 GB of ram. Unless you're paying for a server such as a VPS. In this case every MB of memory pushes the monthly cost up. I keep getting crashes on my bitcoind install running an Electrum server. From the debug log: Code: 02/03/13 02:35:30 CTxMemPool::accept() : accepted a3fbde782d (poolsz 713) This seems to happen every few days for me right now. Is there a memory leak that is causing eventual depletion? bad_alloc.... do you have no swap? Title: Re: Experimental pre-0.8 builds for testing Post by: rasputin on February 05, 2013, 12:19:44 AM I keep getting crashes on my bitcoind install running an Electrum server. Absolutely, at least for me (64-bit Linux). The memory leak predates the 0.8 build, just for the record.*snip* This seems to happen every few days for me right now. Is there a memory leak that is causing eventual depletion? I've worked around it by installing monit and creating this config file: Code: $ cat /etc/monit/conf.d/bitcoind Monit starts bitcoind whenever it isn't running, and forces a restart once the process is using more than 420MB of physical memory. Title: Re: Experimental pre-0.8 builds for testing Post by: BkkCoins on February 05, 2013, 05:56:20 AM Thank you for the multiple suggestions. I haven't had time to explore it more fully yet. It is running on a 1GB VPS under OpenVZ. I believe this has no swap or at least is handled differently. At this moment it's using 686 MB / 1024MB available, and this is typical. This includes electrum and bitcoind. About 468 MB of that is bitcoind. I think it does increase over time but haven't kept records.
There probably is a threads/process limit set by the virtual host. I don't know how to check that off hand but I recall there being some vz cmds and beancounter type stuff I could find out and check. I could probably fix it by just restarting bitcoind and electrum every 24 hours. But for now I'll let it go and check now and then to see if the memory use climbs up. Title: Re: Experimental pre-0.8 builds for testing Post by: dserrano5 on February 05, 2013, 11:57:22 AM There probably is a threads/process limit set by the virtual host. I don't know how to check that off hand but I recall there being some vz cmds and beancounter type stuff I could find out and check. Code: # grep numproc /proc/user_beancounters Title: Re: Experimental pre-0.8 builds for testing Post by: BkkCoins on February 05, 2013, 02:48:01 PM There probably is a threads/process limit set by the virtual host. I don't know how to check that off hand but I recall there being some vz cmds and beancounter type stuff I could find out and check. Code: # grep numproc /proc/user_beancounters But it's running fine atm, and steady at 660 MB total. Title: Re: Experimental pre-0.8 builds for testing Post by: nexusakachus on February 18, 2013, 12:58:01 AM runs smooooooth
:) macbook pro 6.2 osx 10.8.3 build 12D58 Title: Re: Experimental pre-0.8 builds for testing Post by: notme on February 18, 2013, 10:38:04 PM Any word on when this will be released?
Title: Re: Experimental pre-0.8 builds for testing Post by: Diapolo on February 19, 2013, 06:10:49 AM Any word on when this will be released? I'm sure we will see an RC2 and perhaps an RC3 before this version gets released. Dia Title: Re: Experimental pre-0.8 builds for testing Post by: notme on February 19, 2013, 06:47:45 AM Any word on when this will be released? I'm sure we will see an RC2 and perhaps an RC3 before this version gets released. Dia Actually, this wasn't even RC1. RC1 is out now though: https://bitcointalk.org/index.php?topic=142151.0 Not many complaints, solid for me too. Hopefully we can get enough testing and get this pushed out before too long. It really is a huge improvement. Title: Re: Experimental pre-0.8 builds for testing Post by: notme on February 19, 2013, 10:12:09 PM Any word on when this will be released? I'm sure we will see an RC2 and perhaps an RC3 before this version gets released. Dia Actually, this wasn't even RC1. RC1 is out now though: https://bitcointalk.org/index.php?topic=142151.0 Not many complaints, solid for me too. Hopefully we can get enough testing and get this pushed out before too long. It really is a huge improvement. 0.8.0 is out: https://bitcointalk.org/index.php?topic=145184.msg1540252#msg1540252 Title: Re: Experimental pre-0.8 builds for testing Post by: Pieter Wuille on February 25, 2013, 09:06:02 PM Since 0.8.0 final is released, I'd like to thank everyone who participated in testing the pre-release versions. Several problems were found after user reports, and others at least showed that many people noticed the improvements. There is certainly still room for improvement, but we're making progress in small (and sometimes bigger) steps.
Thanks! |
