[ANN] Bitcoin blockchain data torrent

[PatrickMacH]

The torrent is still a faster way of sync for the newbies. I have recommended the seed to the Chinese:
[...]

With new versions, the torrent is ABSOLUTELY NOT FASTER. It is faster and easier to allow Bitcoin Core to sync normally, no need to download a massive file, and the sync is faster due to headers-first synchronization.

Sorry, but I can not confirm this; at least not if you have access to a very fast internet connection.

  I've just downloaded the latest block chain (bootstrap.dat: 2015-12-10) from Flo's Press 4u: https://flo.sh/bitcoin-qt-bootstrap-dat/
uncompressed the zipfile to my ~/.bitcoin application directory and restartet a brand new Bitcoin-Qt v0.11.2.

Bitcoin-Qt imported the bootstrap file within minutes. So I'm ready to go (download and unpacking of the bootstrap zip file included) within less than two hours using recent equipment! 

Just starting the plain Bitcoin-Qt client would have taken many hours if not days to complete the same task

Nevertheless downloading a bootstrap file may be counterproductive using a slow internet access technology.

[achow101]

Sorry, but I can not confirm this; at least not if you have access to a very fast internet connection.

The difference is probably negligible if you have a fast connection.

I've just downloaded the latest block chain (bootstrap.dat: 2015-12-10) from Flo's Press 4u: https://flo.sh/bitcoin-qt-bootstrap-dat/
uncompressed the zipfile to my ~/.bitcoin application directory and restartet a brand new Bitcoin-Qt v0.11.2.

Bitcoin-Qt imported the bootstrap file within minutes. So I'm ready to go (download and unpacking of the bootstrap zip file included) within less than two hours using recent equipment! 

Just starting the plain Bitcoin-Qt client would have taken many hours if not days to complete the same task

Nevertheless downloading a bootstrap file may be counterproductive using a slow internet access technology.

That is not the actual official bootstrap.dat. That is the file produced by a third party and has also been compressed from 50+ Gb to around 35+. The smaller size makes it faster to download (duh). I was referring to the official bootstrap.dat which is now obsolete and is still the full size of the blockchain, not compressed.

[PatrickMacH]

That is not the actual official bootstrap.dat. That is the file produced by a third party and has also been compressed from 50+ Gb to around 35+. The smaller size makes it faster to download (duh). I was referring to the official bootstrap.dat which is now obsolete and is still the full size of the blockchain, not compressed.

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.

[achow101]

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.

The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.

There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

[hamiltino]

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.

The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.

There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.

[achow101]

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.

The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.

There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.

I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.

[rienelber]

quote author=knightdk link=topic=145386.msg13357972#msg13357972 date=1451088033]
quote author=hamiltino link=topic=145386.msg13357882#msg13357882 date=1451086860]
quote author=knightdk link=topic=145386.msg13272697#msg13272697 date=1450322869]
quote author=PatrickMacH link=topic=145386.msg13272577#msg13272577 date=1450321761]
It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
/quote]
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

quote author=PatrickMacH link=topic=145386.msg13272577#msg13272577 date=1450321761]
Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
/quote]
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.
/quote]

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.
/quote]
I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.
/quote]

This just makes no sense, since torrent clients download at full ISP speed, while the btc server is subject to several bottlenecks.

[achow101]

This just makes no sense, since torrent clients download at full ISP speed, while the btc server is subject to several bottlenecks.

Bottlenecks where? Bitcoin nodes download at full ISP speed as well. It doesn't have the additional bottleneck of having to verify all of the blocks after the download. The node will verify the blocks as they are downloaded, not after they have all been downloaded.

[hamiltino]

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.

The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.

There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.

I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.

I guess it depends on your location and connection speeds. I still think the torrent is useful for some, not all. It's also nice to have a single file that i can use whenever i need to quickly bootstrap my client. Also when downloading the blockchain with the client i get corrupt database error's all the time, forcing me to constantly reindex the blockchain... what gives?

[Chancellor]

I've just set up a fresh full node using 0.11.2 Core client. Syncing the whole blockchain took 4 days. What is interesting, downloading blocks was not a bottleneck. My 20 Mbit/s connection was never fully utilized. What is slow, is population of the database and probably building base of UTXO. I admit that my setup is rather slow: Core2Duo @ 2.4 GHz and external 1 TB HDD (5400 rpm) on USB 2.0. With this setup HDD seems to be the major bottleneck.

[jackg]

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

[BillyBobZorton]

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?

[jackg]

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?

Once the old software is installed, and the blockchain downloaded. I can then update it to the new wallet!
It will take much less time and there is no need to re syncronise the blockchain after this is done with the newer version. It will just runn the veryify blocks process.

[rav3n_pl]

NEW version is syncing FASTER than ANY older one!

[achow101]

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?

Once the old software is installed, and the blockchain downloaded. I can then update it to the new wallet!
It will take much less time and there is no need to re syncronise the blockchain after this is done with the newer version. It will just runn the veryify blocks process.

Newer software should be able to sync faster than downloading and verifying the torrent will on an older client.

However, if you insist on installing an old client and upgrading, here is a copy of 0.9.4 that I built and uploaded: https://github.com/achow101/bitcoin/releases/tag/v0.9.4

[Amph]

NEW version is syncing FASTER than ANY older one!

the new version 0.12 will synch even faster, thanks to the capacity increased, which will reduce  orphan, and reduce bandwidth

[rav3n_pl]

HDD is still bottleneck, my system shows 10-20MB of data written every block synced...

[hamiltino]

I'm currently maintaining an up to date bitcoin blockchain data torrent here: https://bitcointalk.org/index.php?topic=1310261.0

[washuu]

With new versions, the torrent is ABSOLUTELY NOT FASTER. It is faster and easier to allow Bitcoin Core to sync normally, no need to download a massive file, and the sync is faster due to headers-first synchronization.

Let me share my story.

I have a small NAS, used also as set-top box to my TV. It runs debian. I decided to give the Bitcoin a try, and set up a node - I have plenty of space there, and this box is used sporadically.

I downloaded fresh version of Bitcoin (at that moment, 0.11.02), and started downloading the blocks normally. I have a decent speed to the internet.

I took over a month to download the whole blockchain, and I have logs to prove it. I have monitored its progress - bitcoind used just a small bandwith, but always 100% of my CPU, on the NAS. It os not very powerful CPU, I admit, so I wasn't surprised. It was the computation that caused the blockchain to load slowly.

When it finished, and started running "normally", the CPU consumption was about 20-40%. It was quite expected. But I decided to give the torrent a try, just for sake of comparison.

The torrent I took from the above linked thread. The torrent downloaded on the same box, with the same CPU and network,  in about 7 hours. Then, I needed to download missing blocks - it took about 36 hours. In total - 42 hours.

In both cases the blockchain is about 62 GB big. The MD5 of the files are different, though.

Now my points:

 - there are computers with slower CPU. Keep this in mind. My case was extreme, but it's just proof, that CPU is also very important during downloading the blockchain, and people having problems with slow downloads perhaps have probelem with CPU, not network.
 - precomputation is just reasonable. Why everyone should compute the same blocks every time they start a new node?
 - if you prepare "certified" blockchain (can be GPG signed, etc.), you will show the people that you CARE about them, and about their time, and freedom of choice of downloading method. This is important for people that are hesitating about running a bitcoin daemon.

Thanks for reading.

[whizz94]

January 11th 2016 blockchain full download
~~~~~~~~~~~~~~~~~~~~~~~~~
Here I report my most recent performance as it will help others to decide what hardware is sufficient.  
It was to a VBOX virtual machine in linux on a quad-core i5 throttled to <75% x 4cpu cores.
Download of the whole blockchain from block 1 to present took about 2 days.  /.bitcoin/ is now about 65GB
2009 to 2013 parts of the blockchain maxed out the 4x cpu usage.
After that it slowed down, with 4x cpu usage decreased from about 95% x 4 x 2.3 GHz to (4 x 2.3 GHz about 40% of the time).
I suspect that the most recent 40GB of the blockchain were synching at a rate limited by my HDD rw speed and seek time, so a faster HDD such as a SSD would be the best available speedup, and might get a full blockchain download down to less than 40 hours.  What have others been seeing?

I don't think that the original bootstrap.dat is much help now as it only does the "easy" third at the start of the blockchain.