Bitcoin Forum
August 20, 2019, 09:34:42 PM *
News: Latest Bitcoin Core release: 0.18.0 [Torrent] (New!)
 
   Home   Help Search Login Register More  
Pages: [1] 2 3 4 5 6 »  All
  Print  
Author Topic: Experimental pre-0.8 builds for testing  (Read 19620 times)
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1064
Merit: 1036


View Profile WWW
December 09, 2012, 12:18:32 AM
Last edit: December 09, 2012, 01:39:54 AM by Pieter Wuille
 #1

Hello all,

I've uploaded a new set of builds of bitcoind/Bitcoin-Qt to my builds page. The new "turbo" build is based on current master on git, plus some more experimental patches (Hal Finney's optimized secp256k1 code, and parallel script verification). The source code for these can be found here (.tgz archive here).

Disclaimer: don't use these for any serious stuff - I am not responsible for destroyed wallets or exploded CPU's.

Testing of these is very welcome.

As the most invasive change is in the block/transaction validation system, this is the most interesting part to test. I am very interested in reports about speed, CPU usage, memory usage, ... during initial block synchronization, in particular on Windows systems. Obviously, crashes or other unexpected behaviour is even more important.

As these builds use the new 0.8 database code ("ultraprune", as I called it before), they will not use the existing block files/database from 0.7.2 or earlier. There are several ways to import the data:
  • Put bootstrap.dat in your data directory.
  • Start the program with options -loadblock=path/to/old/blk0001.dat -loadblock=path/to/old/blk0002.dat.
  • If you already have a full 0.8 database (perhaps from a next-test build), you can use the new -reindex option to rebuild your index from scratch, redoing the block validation process, reusing the block data you already have on disk.
  • Obviously, you can also just let it synchronize from network, but as the current block-fetch logic is somewhat of a hack, you may get unlucky, and hit a slow peer, or have downloads that stop for long periods of time. It is advised to use -connect=IP to connect to a single fast peer (you can use my server, 178.18.90.41, for this purpose if needed) for that.

As these builds use parallel script verification (only enabled after block 193000, the last checkpoint), you may wish to control how many threads are used (don't use more than the number of CPU cores you have). By default, it will try to autodetect, but you can use the -par=N option to select the number of threads. One interesting thing you can test is comparing -par=1 with higher numbers, and check the speedup (in particular of the blocks after 193000).

Other options which may be useful to play with:
  • -dbcache=N tweaks the amount of cache used. N is a number in megabytes (default is 25), but higher numbers may result in significant speedups.
  • -logtimestamps adds timestamps to the output in the debug.log file, which may be very useful to compare speed.
  • -benchmark produces detailed timing information during block validation (not very accurate on Windows, unfortunately).

I do Bitcoin stuff.
1566336882
Hero Member
*
Offline Offline

Posts: 1566336882

View Profile Personal Message (Offline)

Ignore
1566336882
Reply with quote  #2

1566336882
Report to moderator
1566336882
Hero Member
*
Offline Offline

Posts: 1566336882

View Profile Personal Message (Offline)

Ignore
1566336882
Reply with quote  #2

1566336882
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1566336882
Hero Member
*
Offline Offline

Posts: 1566336882

View Profile Personal Message (Offline)

Ignore
1566336882
Reply with quote  #2

1566336882
Report to moderator
1566336882
Hero Member
*
Offline Offline

Posts: 1566336882

View Profile Personal Message (Offline)

Ignore
1566336882
Reply with quote  #2

1566336882
Report to moderator
1566336882
Hero Member
*
Offline Offline

Posts: 1566336882

View Profile Personal Message (Offline)

Ignore
1566336882
Reply with quote  #2

1566336882
Report to moderator
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile
December 09, 2012, 02:47:10 AM
 #2

This is v0.7.1-244-g3ccb06f-beta starting from a completely bare directory, downloading the blockchain from a peer on the same LAN to remove internet connection speed as a factor.

Quote
time bitcoin-qt -par=8 -connect=192.168.XXX.XXX

real    48m58.429s
user    114m35.786s
sys     2m33.817s

Again with a larger cache.

Quote
time bitcoin-qt -par=8 -connect=192.168.XXX.XXX -dbcache=4096

real    42m10.028s
user    109m33.141s
sys     1m9.843s
Rudd-O
Newbie
*
Offline Offline

Activity: 56
Merit: 0



View Profile WWW
December 09, 2012, 04:43:17 AM
 #3

This is v0.7.1-244-g3ccb06f-beta starting from a completely bare directory, downloading the blockchain from a peer on the same LAN to remove internet connection speed as a factor.

Quote
time bitcoin-qt -par=8 -connect=192.168.XXX.XXX

real    48m58.429s
user    114m35.786s
sys     2m33.817s

Again with a larger cache.

Quote
time bitcoin-qt -par=8 -connect=192.168.XXX.XXX -dbcache=4096

real    42m10.028s
user    109m33.141s
sys     1m9.843s

Multithreading seems to be working well!
finway
Hero Member
*****
Offline Offline

Activity: 714
Merit: 500


View Profile
December 09, 2012, 06:02:31 AM
Last edit: December 09, 2012, 04:55:06 PM by finway
 #4

"C:\Program Files\Bitcoin\bitcoin-qt.exe" -reindex -par=4 -dbcache=500 -logtimestamps -benchmark

3h22m
@ Windows7

dserrano5
Legendary
*
Offline Offline

Activity: 1904
Merit: 1010



View Profile
December 09, 2012, 02:22:27 PM
 #5

Starting from an empty ~/.bitcoin directory (having only bootstrap.dat and a bitcoin.conf with an rpc password) and running this:

Code:
nice bitcoind -logtimestamps -loadblock=dot-bitcoin/blk0001.dat -loadblock=dot-bitcoin/blk0002.dat -loadblock=dot-bitcoin/blk0003.dat -dbcache=50 -benchmark &

Took 1 hour 57 to reach height 211494 when at that moment the latest block was 211518. As soon as 211519 was found some minutes later the remaining blocks were processed/downloaded/whatever.

4 CPU cores. RAM usage reached 420 Mb RSS, after synchronization it ended up at 375 Mb. Restarting the daemon leaves it at 130 Mb. I've never used -dbcache until now, if that matters. Usual RAM usage until now was 80-90 Mb just after starting, then going up slowly as days passed (like in, 130 Mb after ~10 days).
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2772
Merit: 2374



View Profile
December 09, 2012, 05:46:08 PM
 #6

"C:\Program Files\Bitcoin\bitcoin-qt.exe" -reindex -par=4 -dbcache=500 -logtimestamps -benchmark

3h22m
@ Windows7

Were you already synced with 0.8 before attempting that reindex?  If so that was horribly slow.
finway
Hero Member
*****
Offline Offline

Activity: 714
Merit: 500


View Profile
December 10, 2012, 12:46:53 AM
 #7

"C:\Program Files\Bitcoin\bitcoin-qt.exe" -reindex -par=4 -dbcache=500 -logtimestamps -benchmark

3h22m
@ Windows7

Were you already synced with 0.8 before attempting that reindex?  If so that was horribly slow.


Yes, before reindex, i'm running pre0.8 1025build version, synced.

Laptop:  Intel Core i3, 4GB ram

fornit
Hero Member
*****
Offline Offline

Activity: 988
Merit: 1001


View Profile
December 10, 2012, 01:12:11 AM
 #8

turbo build just crashed on win 7 64. ran for quite a few hours and then came weirdo "this program requested visual studio runtime library to terminate it in an unusual way"-or something-message, which i successfully closed by reflex  Undecided

was doing normal blockchain download stuck around 93% with 30 connections.

goodbc
Member
**
Offline Offline

Activity: 100
Merit: 10



View Profile
December 10, 2012, 04:51:41 AM
 #9

I downloaded the bitcoin-0.7.1-244-g3ccb06f-win32-setup.exe, installed on my test laptop (Dual core 2GHz Intel, 2GB DDR, with XP) and, without any other extra parameters, it took ~6:40 hours to process 193395 blocks of transaction history. The cpu use was so far ~ 25%.

After this I had to restart the client (the download apparently stopped) and the cpu usage increased to ~75% (sometimes 97-99%). Later it dropped to ~65% while download speed was ~ 250 KB/s.

A new restart was needed after 9:43 hours. The download bar at the bottom of the window disappeared, hopefully this won't scare a regular user.

Finally, the block chain fully synchronized after ~13:00 hours...


I have to admit I expected a way shorter time than that.

Disclaimer: your mileage may vary.

Check your IQ! Send any amount to this address:
1GoodBTCiGyd1J1LkDhCThfTHG8n9WJnNn
gmaxwell
Moderator
Legendary
*
qt
Offline Offline

Activity: 2772
Merit: 2374



View Profile
December 10, 2012, 07:44:44 AM
 #10

I have to admit I expected a way shorter time than that.
If you're syncing from the network even if your own connection is fast you're subject to luck of the draw on which peers you pull from. This is one reason Pieter was suggesting loadblock runs above.
Mushoz
Hero Member
*****
Offline Offline

Activity: 686
Merit: 500


Bitbuy


View Profile WWW
December 10, 2012, 09:25:35 PM
Last edit: December 10, 2012, 09:46:20 PM by Mushoz
 #11

I have to admit I expected a way shorter time than that.
If you're syncing from the network even if your own connection is fast you're subject to luck of the draw on which peers you pull from. This is one reason Pieter was suggesting loadblock runs above.


Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers?

Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release Smiley

www.bitbuy.nl - Koop eenvoudig, snel en goedkoop bitcoins bij Bitbuy!
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1064
Merit: 1036


View Profile WWW
December 11, 2012, 12:01:48 AM
 #12

Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers?

When someone implements an improvement. I plan to have a look at this myself soon, but no promises.

Quote
Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release Smiley

It still needs a lot of testing, and several things are missing before we can have a new stable release.

I do Bitcoin stuff.
Mushoz
Hero Member
*****
Offline Offline

Activity: 686
Merit: 500


Bitbuy


View Profile WWW
December 11, 2012, 03:47:37 PM
 #13

Do you happen to know when the way the blockchain gets downloaded gets changed? Perhaps even downloading in parallel from multiple peers?

When someone implements an improvement. I plan to have a look at this myself soon, but no promises.

Quote
Edit: And is there an ETA for the 0.8 release? I'm so looking forward to all these great changes to hit to main release Smiley

It still needs a lot of testing, and several things are missing before we can have a new stable release.

Sounds great, keep up the good work Smiley And no need to rush Smiley

www.bitbuy.nl - Koop eenvoudig, snel en goedkoop bitcoins bij Bitbuy!
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1568
Merit: 1005


View Profile
December 12, 2012, 03:43:49 AM
 #14

Sounds great, keep up the good work Smiley

+1, Pieter has been doing the (majority?) of the Satoshi client heavy lifting.  He deserves lots of kudos and free beer.

Quote
And no need to rush Smiley

Actually, we do!  We need to stabilize 0.8 as quickly as is reasonably possible, because people are beginning to avoid 0.7.x due to slowness, choosing instead less secure but faster bitcoin clients.

Any and all testing, on any and all platforms, is greatly appreciated.

Even posting "it works, my platform is [Ubuntu version X | Fedora version Y | Windows version Z]..." is helpful.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
paybitcoin
Member
**
Offline Offline

Activity: 85
Merit: 10


1h79nc


View Profile WWW
December 12, 2012, 07:05:14 AM
 #15

Hello,

Testing with bitcoin-0.7.1-244-g3ccb06f-win32-setup.exe and a bootstrap.dat from the torrent. All defaults, except using my server as the peer.
CPU is Win7 x64 Athlon II X4 2.8GHz (4 cores), 4GB RAM, HDD.

bootstrap.dat finished in 53 minutes at 25% CPU, ~120 MB RAM max. With no connection so its not downloading stuff in the background.
The rest of the blocks took 79 minutes. Download was constant at 350-450KBps (about 30% of my total DSL bandwidth.) 70-80% CPU the whole time, ~163 MB RAM max.

blocks dir is now 4.53 GB.

Went pretty smoothly. Much, much better than the previous days-long sync.
kwukduck
Legendary
*
Offline Offline

Activity: 1941
Merit: 1000


View Profile
December 12, 2012, 07:16:52 AM
 #16

Clean setup in win7_x64
Still syncing the blockchain (over 15 hrs now)
Barely any CPU usage, but using 1.3GB memory. Also seems to have a lot of disk activity making the system a bit slow.

14b8PdeWLqK3yi3PrNHMmCvSmvDEKEBh3E
m0mchil
Full Member
***
Offline Offline

Activity: 171
Merit: 101


View Profile
December 12, 2012, 02:14:25 PM
 #17

intel core i7 860 (4 physical, 8 logical cores)

bitcoin-qt built with gitian-builder from sipa/turbo rev 3ccb06f5ae538260817b6ef474f93029be41a64d


Ubuntu 12.04 64 bit, SSD A-DATA SX910 256 GB

64 bit, no -par (default), default cache

first 193000 blocks from bootstrap.dat - 11m:40s CPU time

real    9m50.565s
user    10m47.800s
sys     0m56.548s


64 bit, no -par (default), default cache

loading from blk000x.dat, up to arround block 211100

-loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat

1 GB peak memory, ~80% average CPU usage, total CPU time 3h:16m

real    41m30.004s
user    196m43.378s
sys     2m47.834s

debug.log.ubuntu.12.04.x64.7z




Windows 7 Professional x64, two Seagate 512 GB HDD in RAID 0

-loadblock=blk0001.dat -loadblock=blk0002.dat -loadblock=blk0003.dat

default cache, with '-par 8', initially it seemed to run in single thread mode - first 193 000 blocks loaded significantly slower than on Ubuntu (same hardware), reason perhaps is slower storage.

After block 193000 the average CPU utilization rose to ~75%. Looking at threads, there were 16 threads executing same code of which 8 were actually active.

92 minutes real time, 6h:05m CPU time, 72% average CPU utilization, 235 MB peak memory working set

I then tried with '-par=4' and setting CPU affinity to physical cores only. Now there were initially 12 threads at transaction crunching, with again only one actively processing pre-SatoshiDice blocks.

debug.log.win32.7z

slothbag
Sr. Member
****
Offline Offline

Activity: 369
Merit: 250



View Profile
December 13, 2012, 09:55:52 AM
 #18

I upgraded to the latest Jenkins build of bitcoin-qt ( pre-0.8 ) on windows.  I tried for a week and a half running -loadblock and -reindex, stopping restarting etc.. it was horribly slow, and going no where.

So I eventually downloaded the SIPA turbo build linked in the OP, deleted my existing blockchain, downloaded the bootstrap and let it run from scratch with an empty wallet..

Took about 6 hours to load in the bootstrap file while CPU sat at 30% and disk I/O not that high, so not sure what the bottleneck was there. then after block 190000 CPU jumped to 100% and it took about 10 hours to get back in sync to current block. Then I added my old wallet and rescanned that which went for maybe 10-20mins..

So significantly faster than the first time I sync'd 18 months ago.. but still quite frustrating.

I'm running a Atom CPU based nettop with windows 7, hence the slower than average speeds.
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1064
Merit: 1036


View Profile WWW
December 13, 2012, 12:05:28 PM
 #19

I added yet another - and even more experimental - build (in the "-turbo-ldb17" directory here) with levelDB 1.7, and several Windows-only changes (new Windows LevelDB backend, Boost 1.52, newer compiler suite). Tests/benchmarks/comparison with the other turbo build are very welcome, particularly on Windows (there are very few changes for other systems).

M0mchil already reported some GUI problems, but I'm confident those can be fixed if necessary. The question is really whether it's stable and performant. Also things like testing whether it recovers from sudden power loss or forceful quits of the application are nice.

Again a disclaimer: Use at your own risk, I'm not responsible for overheating systems or stolen wallets.

PS: If you report synchronization benchmarks, please mention on what kind of system (CPU/RAM) and CPU usage (after block 193000). Also, don't just synchronize from arbitrary peers over the internet - the variation between hitting a good or a bad peer makes the timing completely unreliable.

I do Bitcoin stuff.
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1006



View Profile
December 13, 2012, 12:56:11 PM
 #20

Can you add a command line option for benchmarking that will cause the program to exit immediately after a successful synchronization?
Pages: [1] 2 3 4 5 6 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!