Bitcoin Forum
November 22, 2026, 09:43:20 AM *
News: Latest Bitcoin Core release: 32.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 [21] 22 »  All
  Print  
Author Topic: [ANN] Bitcoin blockchain data torrent  (Read 211339 times)
achow101
Staff
Legendary
*
Offline

Activity: 4032
Merit: 7898


Just writing some code


View Profile WWW
December 17, 2015, 03:27:49 AM
 #401

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

hamiltino
Hero Member
*****
Offline

Activity: 644
Merit: 500


P2P The Planet!


View Profile
December 25, 2015, 11:41:00 PM
 #402

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.

stacking coin
achow101
Staff
Legendary
*
Offline

Activity: 4032
Merit: 7898


Just writing some code


View Profile WWW
December 26, 2015, 12:00:33 AM
 #403

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.
I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.

rienelber
Full Member
***
Offline

Activity: 154
Merit: 100


View Profile
December 26, 2015, 12:06:21 AM
Last edit: February 22, 2016, 08:14:21 PM by rienelber
 #404

quote author=knightdk link=topic=145386.msg13357972#msg13357972 date=1451088033]
quote author=hamiltino link=topic=145386.msg13357882#msg13357882 date=1451086860]
quote author=knightdk link=topic=145386.msg13272697#msg13272697 date=1450322869]
quote author=PatrickMacH link=topic=145386.msg13272577#msg13272577 date=1450321761]
It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
/quote]
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

quote author=PatrickMacH link=topic=145386.msg13272577#msg13272577 date=1450321761]
Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
/quote]
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.
/quote]

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.
/quote]
I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.
/quote]

This just makes no sense, since torrent clients download at full ISP speed, while the btc server is subject to several bottlenecks.
achow101
Staff
Legendary
*
Offline

Activity: 4032
Merit: 7898


Just writing some code


View Profile WWW
December 26, 2015, 12:14:11 AM
 #405

This just makes no sense, since torrent clients download at full ISP speed, while the btc server is subject to several bottlenecks.
Bottlenecks where? Bitcoin nodes download at full ISP speed as well. It doesn't have the additional bottleneck of having to verify all of the blocks after the download. The node will verify the blocks as they are downloaded, not after they have all been downloaded.

hamiltino
Hero Member
*****
Offline

Activity: 644
Merit: 500


P2P The Planet!


View Profile
December 26, 2015, 01:03:35 AM
Last edit: December 31, 2015, 08:00:30 AM by hamiltino
 #406

It's a standard zip compressed bootstrap.dat file, combining all blk*.dat files out of the blocks directory.
The actual bootstrap.dat is not compressed. The one you are downloading is, but it isn't the official one that was maintained by jgarzik.

Maybe we should discuss adding a zip level compression (or better suitable) algorithm to the Bitcoin-Qt client to reduce the amout of data needed to be transferred to update the block chain.
There is talk on the mailing list about block compression and ways to speed up syncing, but I don't think that any of it will be coming in the next release. It either isn't ready or there isn't any particular need for it. Using the actual bootstrap.dat (not the one that you used) is not actually faster than simply letting it sync.

stop talking out of ur ass, im getting speeds up to 5mb/s with the torrent, i can't get anything near this with the client. This has been a huge time saver for me.
I'm not talking out of my ass. From my experience it has been faster to let the client (version 0.10+) sync by itself than it is to download the bootstrap.dat and to import that.

I guess it depends on your location and connection speeds. I still think the torrent is useful for some, not all. It's also nice to have a single file that i can use whenever i need to quickly bootstrap my client. Also when downloading the blockchain with the client i get corrupt database error's all the time, forcing me to constantly reindex the blockchain... what gives?

stacking coin
Chancellor
Full Member
***
Offline

Activity: 154
Merit: 100


View Profile
December 27, 2015, 05:36:06 PM
 #407

I've just set up a fresh full node using 0.11.2 Core client. Syncing the whole blockchain took 4 days. What is interesting, downloading blocks was not a bottleneck. My 20 Mbit/s connection was never fully utilized. What is slow, is population of the database and probably building base of UTXO. I admit that my setup is rather slow: Core2Duo @ 2.4 GHz and external 1 TB HDD (5400 rpm) on USB 2.0. With this setup HDD seems to be the major bottleneck.

BTC: 1GRx2H5esyFTucZCt1eX9tjpoqfMrbVHMT
jackg
Copper Member
Legendary
*
Offline

Activity: 2856
Merit: 3076


https://bit.ly/387FXHi lightning theory


View Profile
December 28, 2015, 10:24:32 PM
 #408

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?
BillyBobZorton
Legendary
*
Offline

Activity: 1204
Merit: 1028


View Profile
December 28, 2015, 10:26:26 PM
 #409

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?
jackg
Copper Member
Legendary
*
Offline

Activity: 2856
Merit: 3076


https://bit.ly/387FXHi lightning theory


View Profile
December 28, 2015, 10:39:55 PM
 #410

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?

Once the old software is installed, and the blockchain downloaded. I can then update it to the new wallet!
It will take much less time and there is no need to re syncronise the blockchain after this is done with the newer version. It will just runn the veryify blocks process.
rav3n_pl
Legendary
*
Offline

Activity: 1361
Merit: 1003


Don`t panic! Organize!


View Profile WWW
December 28, 2015, 10:53:58 PM
 #411

NEW version is syncing FASTER than ANY older one!

1Rav3nkMayCijuhzcYemMiPYsvcaiwHni  Bitcoin stuff on my OneDrive
My RPC CoinControl for any coin https://bitcointalk.org/index.php?topic=929954
Some stuff on https://github.com/Rav3nPL/
achow101
Staff
Legendary
*
Offline

Activity: 4032
Merit: 7898


Just writing some code


View Profile WWW
December 29, 2015, 03:21:22 AM
 #412

Is there a way that I can install an old verstion of bitcoin core.
Import the torrent file into that and then update the software to the latest version.

Would this work better to syncronise to the network, as I have been told that, when switching between versions, the initial blockchain import is not required? Have I got this correct?

I think t's best that you download the entire blockchain from scratch no matter what, it's worth investing into the extra time for the extra security. Why do you want to install an old version anyway tho?

Once the old software is installed, and the blockchain downloaded. I can then update it to the new wallet!
It will take much less time and there is no need to re syncronise the blockchain after this is done with the newer version. It will just runn the veryify blocks process.
Newer software should be able to sync faster than downloading and verifying the torrent will on an older client.

However, if you insist on installing an old client and upgrading, here is a copy of 0.9.4 that I built and uploaded: https://github.com/achow101/bitcoin/releases/tag/v0.9.4

Amph
Legendary
*
Offline

Activity: 3290
Merit: 1072



View Profile
December 29, 2015, 07:45:17 AM
 #413

NEW version is syncing FASTER than ANY older one!

the new version 0.12 will synch even faster, thanks to the capacity increased, which will reduce  orphan, and reduce bandwidth
rav3n_pl
Legendary
*
Offline

Activity: 1361
Merit: 1003


Don`t panic! Organize!


View Profile WWW
December 29, 2015, 11:29:12 AM
 #414

HDD is still bottleneck, my system shows 10-20MB of data written every block synced...

1Rav3nkMayCijuhzcYemMiPYsvcaiwHni  Bitcoin stuff on my OneDrive
My RPC CoinControl for any coin https://bitcointalk.org/index.php?topic=929954
Some stuff on https://github.com/Rav3nPL/
hamiltino
Hero Member
*****
Offline

Activity: 644
Merit: 500


P2P The Planet!


View Profile
January 04, 2016, 06:13:04 PM
 #415

I'm currently maintaining an up to date bitcoin blockchain data torrent here: https://bitcointalk.org/index.php?topic=1310261.0

stacking coin
washuu
Newbie
*
Offline

Activity: 3
Merit: 0


View Profile
January 14, 2016, 11:00:52 AM
 #416

With new versions, the torrent is ABSOLUTELY NOT FASTER. It is faster and easier to allow Bitcoin Core to sync normally, no need to download a massive file, and the sync is faster due to headers-first synchronization.


Let me share my story.

I have a small NAS, used also as set-top box to my TV. It runs debian. I decided to give the Bitcoin a try, and set up a node - I have plenty of space there, and this box is used sporadically.

I downloaded fresh version of Bitcoin (at that moment, 0.11.02), and started downloading the blocks normally. I have a decent speed to the internet.

I took over a month to download the whole blockchain, and I have logs to prove it. I have monitored its progress - bitcoind used just a small bandwith, but always 100% of my CPU, on the NAS. It os not very powerful CPU, I admit, so I wasn't surprised. It was the computation that caused the blockchain to load slowly.

When it finished, and started running "normally", the CPU consumption was about 20-40%. It was quite expected. But I decided to give the torrent a try, just for sake of comparison.

The torrent I took from the above linked thread. The torrent downloaded on the same box, with the same CPU and network,  in about 7 hours. Then, I needed to download missing blocks - it took about 36 hours. In total - 42 hours.

In both cases the blockchain is about 62 GB big. The MD5 of the files are different, though.

Now my points:

 - there are computers with slower CPU. Keep this in mind. My case was extreme, but it's just proof, that CPU is also very important during downloading the blockchain, and people having problems with slow downloads perhaps have probelem with CPU, not network.
 - precomputation is just reasonable. Why everyone should compute the same blocks every time they start a new node?
 - if you prepare "certified" blockchain (can be GPG signed, etc.), you will show the people that you CARE about them, and about their time, and freedom of choice of downloading method. This is important for people that are hesitating about running a bitcoin daemon.

Thanks for reading.
whizz94
Full Member
***
Offline

Activity: 149
Merit: 100


Solar Bitcoin Specialist


View Profile WWW
January 14, 2016, 11:18:45 AM
Last edit: February 14, 2016, 12:32:44 PM by whizz94
 #417

January 11th 2016 blockchain full download
~~~~~~~~~~~~~~~~~~~~~~~~~
Here I report my most recent performance as it will help others to decide what hardware is sufficient.  
It was to a VBOX virtual machine in linux on a quad-core i5 throttled to <75% x 4cpu cores.
Download of the whole blockchain from block 1 to present took about 2 days.  /.bitcoin/ is now about 65GB
2009 to 2013 parts of the blockchain maxed out the 4x cpu usage.
After that it slowed down, with 4x cpu usage decreased from about 95% x 4 x 2.3 GHz to (4 x 2.3 GHz about 40% of the time).
I suspect that the most recent 40GB of the blockchain were synching at a rate limited by my HDD rw speed and seek time, so a faster HDD such as a SSD would be the best available speedup, and might get a full blockchain download down to less than 40 hours.  What have others been seeing?

I don't think that the original bootstrap.dat is much help now as it only does the "easy" third at the start of the blockchain.
shorena
Copper Member
Legendary
*
Offline

Activity: 1498
Merit: 1644


No I dont escrow anymore.


View Profile
January 14, 2016, 02:29:13 PM
 #418

-story-
Now my points:

 - there are computers with slower CPU. Keep this in mind. My case was extreme, but it's just proof, that CPU is also very important during downloading the blockchain, and people having problems with slow downloads perhaps have probelem with CPU, not network.

Yes, the overal speed depends on more than just bandwith.

- precomputation is just reasonable. Why everyone should compute the same blocks every time they start a new node?

You either used the "bootstrap.dat" in which case you did not safe any computation time, but rather wasted it by seperating download and verification. From the times you states and "precomputation" however I assume you downloaded a preverified blockchain. This is naturally faster, because you dont do all the work. Thus the comparisson is invalid and you need to trust the other party to not have messed with the blockchain.

You should verifiy the blocks to avoid this exact problem, trusting someone else. The same way you verify the binaries you download or the signed message you receive.

- if you prepare "certified" blockchain (can be GPG signed, etc.), you will show the people that you CARE about them, and about their time, and freedom of choice of downloading method. This is important for people that are hesitating about running a bitcoin daemon.

Thanks for reading.

Which will require you to trust that person and their ability to secure their PGP key. People are not hesitating to run a full node because of the initial time it takes, but because of the constant time it needs to download and verify data. If it would just be 1-3 days (relativly modern desktop/laptop computer) it would not be an issue.

Im not really here, its just your imagination.
washuu
Newbie
*
Offline

Activity: 3
Merit: 0


View Profile
January 16, 2016, 09:37:59 PM
 #419


You either used the "bootstrap.dat" in which case you did not safe any computation time, but rather wasted it by seperating download and verification. From the times you states and "precomputation" however I assume you downloaded a preverified blockchain. This is naturally faster, because you dont do all the work. Thus the comparisson is invalid and you need to trust the other party to not have messed with the blockchain.

You should verifiy the blocks to avoid this exact problem, trusting someone else. The same way you verify the binaries you download or the signed message you receive.

That's exactly my point - why would I have needed trust someone else, if I had had the opportunity to download pre-verified blockchain as a package, certified by bitcoin.org ? That's just giving the people another way of getting bitcoin node up and running very fast. Precomputing means not only wasted CPU cycles, but also HDD operations, as there was reported that every block requires much more HDD transfer, than just size of the blocks.

There still will be option to run bitcoind from scratch, for any people that just want to "fire and forget", and that is good.
jackg
Copper Member
Legendary
*
Offline

Activity: 2856
Merit: 3076


https://bit.ly/387FXHi lightning theory


View Profile
January 16, 2016, 11:33:02 PM
 #420

January 11th 2016 blockchain full download
~~~~~~~~~~~~~~~~~~~~~~~~~
Here I report my most recent performance as it will help others to decide what hardware is sufficient. 
It was to a VBOX virtual machine in linux on a quad-core i5 throttled to <75% x 4cpu cores.
Download of the whole blockchain from block 1 to present took about 2 days.  /.bitcoin/ is now about 65GB
2009 to 2013 parts of the blockchain maxed out the 4x cpu usage.
After that it slowed down, with 4x cpu usage decreased from about 95% x 4 x 2.3 GHz to (4 x 2.3 GHz about 40% of the time).
I suspect that the most recent 40GB of the blockchain were synching at a rate limited by my HDD rw speed and seek time, so a faster HDD such as a SSD would be the best available speedup, and might get a full blockchain download down to less that 40 hours.  What have others been seeing?

I don't think that the original bootstrap.dat is much help now as it only does the "easy" third at the start of the blockchain.

I was considering running the bitcoin core of a hard drive on a raspbery pi and transferrring the data over afterward to my laptop.

Would this save power? It only has to power a hard drive, raspberry pi and WiFi adapter.
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 [21] 22 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!