Downloading the blockchain

[Mushoz]

Hey guys,

I just installed Bitcoin on a new computer and moved my wallet.dat and started the client so it could begin downloading the chain. At the beginning it was downloading blocks really fast, but now that it is at 80% it's slowing down to a crawl with loads of HDD activity. Why is that? Why is it slowing down so much at the end? Are the blocks that much bigger? And why is it causing so much HDD activity? AFAIK the blockchain was only about 700ish MB last time I checked, so I'm not really sure why the HDD is a bottleneck when downloading the chain over a 25mb/s internet connection. Thank you very much in advance for explaining my questions

[1714008003]

1714008003

[Gabi]

Are the blocks that much bigger?

Yup

And why is it causing so much HDD activity?

Dunno but it's normal, same happens here

when downloading the chain over a 25mb/s internet connection.

Yup but you don't download blocks at 25mb/s, but only at the speed as the peers you are connected to send them to you, so much slower.

[SgtSpike]

Hey guys,

I just installed Bitcoin on a new computer and moved my wallet.dat and started the client so it could begin downloading the chain. At the beginning it was downloading blocks really fast, but now that it is at 80% it's slowing down to a crawl with loads of HDD activity. Why is that? Why is it slowing down so much at the end? Are the blocks that much bigger? And why is it causing so much HDD activity? AFAIK the blockchain was only about 700ish MB last time I checked, so I'm not really sure why the HDD is a bottleneck when downloading the chain over a 25mb/s internet connection. Thank you very much in advance for explaining my questions

The earlier blocks only have a single transaction - the transaction to generate a new bitcoin address.  Laters blocks have hundreds, sometimes even thousands of transactions.  So yes, they're going to take a lot longer to download.

[SaintFlow]

Sorry to threatjack,

Can someone tell me how long it now takes to download the entire blockchain if you are new to this
on average.

Will this become something of a scalability issue to bitcoin in the future?

[SgtSpike]

My estimate is 4-24 hours, depending on your connection speed and the connection speed of the nodes you are connected to.

Yes, it is a scalability issue, but it will be solved with lite clients that don't store the full blockchain.

[ovidiusoft]

My estimate is 4-24 hours, depending on your connection speed and the connection speed of the nodes you are connected to.
Yes, it is a scalability issue, but it will be solved with lite clients that don't store the full blockchain.

My way of doing it is to connect to a few fallback nodes ( https://en.bitcoin.it/wiki/Fallback_Nodes#IPv4_Nodes ). For example, btcnode.novit.ro would be a good choice for people in Romania or a few hops away (Europe, Russia), it is located in Bucharest and has 100 Mbit bandwidth, soon (read: next year) to be upgraded to 1 Gbit.

[DeathAndTaxes]

Also be aware that even w/ unlimited nodes and unlimited bandwidth it takes some time to validate each block (and every transaction in the block) and write it to the database.  The client (properly) assumes anything the network tells it can't be trusted until the data is validated against the portion of block chain already validated. In essence is is rebuilding the entire transaction history of Bitcoin from the genesis block forward.

[deepceleron]

You don't need "fallback nodes", as every connection is a node. The best thing to speed downloading is to have lots of connections, such as properly setting up port forwarding. 30+ connections instead of eight is normal.

You can also look at the CPU usage while downloading; unless a CPU core is maxed to 100%, CPU usage likely isn't the limiting factor. Disk access is also an order of magnitude faster than the average high-speed connection.

My direct-download blockchain thread was un-stickied, it seems, but you can find it in the newbie forum.

[ovidiusoft]

You don't need "fallback nodes", as every connection is a node. The best thing to speed downloading is to have lots of connections, such as properly setting up port forwarding. 30+ connections instead of eight is normal.

Not necessarily. Due to the random way you get connections, it's possible that a significant portion of these are on slow connections or unstable and the client will work a lot on finding another node to send you those blocks, and so on.

Plus, it always pissed me off that on every Bitcoin client startup I had to wait tens of seconds (sometimes minutes) until it would start downloading. This is worse if a lot of time passed since you shutdown (most of the IPs in your local cache are no longer available).

[racerguy]

http://eu1.bitcoincharts.com/blockchain/
download that it's much faster, and current to within 24hrs.

[bb113]

http://eu1.bitcoincharts.com/blockchain/
download that it's much faster, and current to within 24hrs.

Yes, this is good advice. In fact, I couldn't even get the client to download the blockchain. Once I downloaded the nightly from bitcoincharts it updated fine.

[gmaxwell]

AFAIK the blockchain was only about 700ish MB last time I checked, so I'm not really sure why the HDD is a bottleneck when downloading the chain over a 25mb/s internet connection. Thank you very much in advance for explaining my questions

Bitcoin is a system designed so that you don't have to trust anyone— so your client doesn't trust your peers to give you valid blocks, it doesn't even trust that the miners followed the rules, it processes each one of them performing all the transaction processing the bitcoin system does on its own to validate that the rules of the bitcoin system have been respected and that a couple of big miners didn't just quietly agree to give themselves a million coin out of thin air.

Of course, it's possible to build bitcoin end user software that doesn't do all this checking— bitcoinj for example, doesn't even need to keep a copy of the historical transactions— but then it can't also participate as a full peer in the network. It has to have trust— that there aren't sizable miners out there willing to insert complete rubbish into the blockchain (or a non-trivial fork of it).

The hard disk activity is because the later blocks have many more transactions and for each transaction your system must find the transactions that paid into it to make sure the figures add up... the software then does synchronous database writes to make sure that the data isn't lost— a good thing during the normal processing, but kinda silly during the initial synchup.  I expect future versions will speed this up quite a bit, though initial sync on a full node will always be somewhat slow.

The actual downloading time is almost never an issue for users— its the validation thats slow. Though there still are a lot of pre-0.3.24 nodes out there which slow the syncup because they randomly hang up on peers asking for a couple of the more recent blocks at once.

Contrary to the claims in posts upthread, for almost everyone the disk IO is the bottleneck. If you run your node on a ramdisk or a very fast SSD you can get CPU to be the bottleneck.  You'd probably have to be on dialup before the network would be the bottleneck.

[fivemileshigh]

How does one connect to a particular node from the list at

https://en.bitcoin.it/wiki/Fallback_Nodes ?

 I'm assuming the " -addnode=ip " switch is for windows users?

I'm on OSX 10.6.8

thanks!

[racerguy]

How does one connect to a particular node from the list at

https://en.bitcoin.it/wiki/Fallback_Nodes ?

 I'm assuming the " -addnode=ip " switch is for windows users?

I'm on OSX 10.6.8

thanks!

add that line into your bitcoin.conf file, wherever that is on a mac.

[Red Emerald]

How does one connect to a particular node from the list at

https://en.bitcoin.it/wiki/Fallback_Nodes ?

 I'm assuming the " -addnode=ip " switch is for windows users?

I'm on OSX 10.6.8

thanks!

Code:

$ /Applications/Bitcoin.app/Contents/MacOS/bitcoin --help