Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: JimmyGre on May 16, 2014, 02:38:48 AM



Title: downloading blocks is too slow
Post by: JimmyGre on May 16, 2014, 02:38:48 AM
I followed this -- http://linux.ringingliberty.com/bitcoin/ tutorial to install my daemon. But I can not get the bitcoin balance.
I checked here -- https://bitcointalk.org/index.php?topic=595916.0 . It says there are 299134 blocks and would take a day to download. It takes me 2 weeks to download but still less than 290000.
The number of blocks is increasing but really really slow.

Any reasons?

Thank you,
Jim


Title: Re: downloading blocks is too slow
Post by: Shogen on May 16, 2014, 03:46:41 AM
It says there are 299134 blocks and would take a day to download. It takes me 2 weeks to download but still less than 290000.
The number of blocks is increasing but really really slow.

The initial sync shouldn't take such a long period of time. Do you have a very limited internet bandwidth?


Title: Re: downloading blocks is too slow
Post by: JimmyGre on May 16, 2014, 03:55:53 AM
It says there are 299134 blocks and would take a day to download. It takes me 2 weeks to download but still less than 290000.
The number of blocks is increasing but really really slow.

The initial sync shouldn't take such a long period of time. Do you have a very limited internet bandwidth?

I don't think I have very limited...
it's like adding one block every min....


Title: Re: downloading blocks is too slow
Post by: cp1 on May 16, 2014, 03:57:49 AM
Is your cpu maxed? It has to process the blocks too.  Low disk space?  Some kind of firewall problem? How many connections do you have?


Title: Re: downloading blocks is too slow
Post by: 21st-century on May 16, 2014, 03:58:03 AM
You might have poor node connectivity. How many connections do you have? Check that you've setup port forwarding correctly in your router so other nodes can "find" yours easier. Failing that, you can change the source code's outgoing connection # and incoming connection # to be 100 instead of ~8. Improves connectivity and block downloads if you're running a server. Just be careful it doesn't rape your bandwidth.

In future you might want to use a bootstrap.dat. It's a pre-downloaded list of blocks that can vastly speed up how fast you can get a client "up to date." There are some good torrents for this.


Title: Re: downloading blocks is too slow
Post by: Shogen on May 16, 2014, 04:19:27 AM
In future you might want to use a bootstrap.dat. It's a pre-downloaded list of blocks that can vastly speed up how fast you can get a client "up to date." There are some good torrents for this.

The link to the bootstrap torrent can be found directly on bitcoin.org download page. https://bitcoin.org/en/download

Quote
If you know how to download a torrent file, you can speed up this process by putting bootstrap.dat (https://bitcoin.org/bin/blockchain/bootstrap.dat.torrent) (a previous copy of the block chain) in the Bitcoin Core data directory before starting the software.


Title: Re: downloading blocks is too slow
Post by: JimmyGre on May 16, 2014, 05:59:55 AM
In future you might want to use a bootstrap.dat. It's a pre-downloaded list of blocks that can vastly speed up how fast you can get a client "up to date." There are some good torrents for this.

The link to the bootstrap torrent can be found directly on bitcoin.org download page. https://bitcoin.org/en/download

Quote
If you know how to download a torrent file, you can speed up this process by putting bootstrap.dat (https://bitcoin.org/bin/blockchain/bootstrap.dat.torrent) (a previous copy of the block chain) in the Bitcoin Core data directory before starting the software.


Sorry, really new for bitcoin. How can I do that on CentOS server?
Many many thanks~
J


Title: Re: downloading blocks is too slow
Post by: grue on May 17, 2014, 01:22:14 AM
You might have poor node connectivity. How many connections do you have? Check that you've setup port forwarding correctly in your router so other nodes can "find" yours easier. Failing that, you can change the source code's outgoing connection # and incoming connection # to be 100 instead of ~8. Improves connectivity and block downloads if you're running a server. Just be careful it doesn't rape your bandwidth.
this is bullshit. bitcoin core only downloads from one peer at a time. having more peers does not improve download speed.


Title: Re: downloading blocks is too slow
Post by: cp1 on May 18, 2014, 02:41:18 AM
You'll have to install a torrent client, like transmission.


Title: Re: downloading blocks is too slow
Post by: cahirlet on May 18, 2014, 05:36:20 AM
Mine took a few days. If you want it to be faster, download the bootstrap from the official site.
Go to https://bitcoin.org/en/download (https://bitcoin.org/en/download) this and scroll to the bottom.
The QT will verify the blocks when starting up. If you do not want to download the blocks, try a light weight client like multibit  :)


Title: Re: downloading blocks is too slow
Post by: JimmyGre on May 18, 2014, 10:19:25 PM
Mine took a few days. If you want it to be faster, download the bootstrap from the official site.
Go to https://bitcoin.org/en/download (https://bitcoin.org/en/download) this and scroll to the bottom.
The QT will verify the blocks when starting up. If you do not want to download the blocks, try a light weight client like multibit  :)

Thank you Cahirlet. I'm not sure where I should put the bootstrap.dat.torrent file. under /root/bitcoin I have .git; contrib; doc; qa; share; src folders. Which one should contain bootstrap? And to start this bootstrap, can i just run '$bitcoin = new jsonRPCClient('http://admin:password@127.0.0.1:8332/');' ?

Many many thanks,
J


Title: Re: downloading blocks is too slow
Post by: cahirlet on May 19, 2014, 09:17:49 AM
Mine took a few days. If you want it to be faster, download the bootstrap from the official site.
Go to https://bitcoin.org/en/download (https://bitcoin.org/en/download) this and scroll to the bottom.
The QT will verify the blocks when starting up. If you do not want to download the blocks, try a light weight client like multibit  :)

Thank you Cahirlet. I'm not sure where I should put the bootstrap.dat.torrent file. under /root/bitcoin I have .git; contrib; doc; qa; share; src folders. Which one should contain bootstrap? And to start this bootstrap, can i just run '$bitcoin = new jsonRPCClient('http://admin:password@127.0.0.1:8332/');' ?

Many many thanks,
J
Note: You need to use torrent client in order to get the blockchain file. Try using a torrent client. As to how to import it, navigate to ~/.bitcoin/ and paste your bootstrap.dat or any other name that u have finished torrenting. Personally I like this: http://www.utorrent.com/downloads/linux
Wish you good luck!


Title: Re: downloading blocks is too slow
Post by: JimmyGre on May 19, 2014, 10:41:58 PM
Mine took a few days. If you want it to be faster, download the bootstrap from the official site.
Go to https://bitcoin.org/en/download (https://bitcoin.org/en/download) this and scroll to the bottom.
The QT will verify the blocks when starting up. If you do not want to download the blocks, try a light weight client like multibit  :)

Thank you Cahirlet. I'm not sure where I should put the bootstrap.dat.torrent file. under /root/bitcoin I have .git; contrib; doc; qa; share; src folders. Which one should contain bootstrap? And to start this bootstrap, can i just run '$bitcoin = new jsonRPCClient('http://admin:password@127.0.0.1:8332/');' ?

Many many thanks,
J
Note: You need to use torrent client in order to get the blockchain file. Try using a torrent client. As to how to import it, navigate to ~/.bitcoin/ and paste your bootstrap.dat or any other name that u have finished torrenting. Personally I like this: http://www.utorrent.com/downloads/linux
Wish you good luck!
Cool~ Thank you!!!


Title: Re: downloading blocks is too slow
Post by: Cryddit on May 20, 2014, 09:53:54 PM
Actually a high latency, or a slow computer, can cause very slow downloads in the bitcoin blockchain, even if you have good bandwidth.

The way the initial block download works, your client goes and asks somebody, "what's the current block", and they say, BLOCK ID XXXXX.  And then since you don't have XXXXX, you ask them for that and they send it to you, but after you download it, then you look at its block header and discover that the previous block was WWWWW - and you don' t have that one either!  So you ask, "Hey could you give me  a copy of Block WWWWW?"  And they send that, and you look at it, and its parent was VVVVVV, and you don' t have that either, so you ask.....  and so on, back to the Genesis block.

The deal here is that you don't know which additional block you need next (and therefore can't ask for it) until you've gotten the current block, so you can only have one request for a block out there at a time.  If you have a high latency, or the peer you asked the block for has a high latency, it can take a few seconds to get each and every block, even if you both support very high transfer rates.

Then there's a CPU bottleneck to worry about.  Your peer does all kinds of cryptographic checks on each downloaded block, so if your CPU is slow, that may even dominate the latency time. 

It's worthwhile to get the bitcoin blockchain using bittorrent; with the bittorrent protocol at least you can have requests out there and active for more than one block at a time. 






Title: Re: downloading blocks is too slow
Post by: softtissue on May 21, 2014, 01:48:05 AM
Trying to use SSD, it will improve the download speed obviously. :)


Title: Re: downloading blocks is too slow
Post by: cahirlet on May 21, 2014, 09:04:14 AM
Trying to use SSD, it will improve the download speed obviously. :)
Verification and write speed, after downloading, your computer need to verify the blocks individually. SSD greatly improves the speed. Download speed is dependent on your internet speed, not on hardware.  :)


Title: Re: downloading blocks is too slow
Post by: swapcoiner on May 21, 2014, 03:17:45 PM
Check your network bandwidth. Block downloads depends on network speed.


Title: Re: downloading blocks is too slow
Post by: zvs on May 21, 2014, 11:31:39 PM
Use

bitcoin-qt (or bitcoind) --maxconnections=1 --connect=5.9.24.81

until you finish syncing the entire blockchain.

This will help w/ two things, 1) you won't use any upstream (& potentially have downstream limited by saturated upstream) by having or connecting to a client that's also downloading the blockchain, 2) you'll receive blocks as fast as your CPU can process them (I have gigabit connection on 5.9.24.81).


Title: Re: downloading blocks is too slow
Post by: DeathAndTaxes on May 22, 2014, 06:40:10 AM
Trying to use SSD, it will improve the download speed obviously. :)
Verification and write speed, after downloading, your computer need to verify the blocks individually. SSD greatly improves the speed. Download speed is dependent on your internet speed, not on hardware.  :)

Network speed is rarely the bottleneck unless you just have the bad luck of continually connecting to a really slow peer.   Downloading 20 GB @ 5Mbps is only ~8 hours.

Downloading and verification aren't independent steps.  To avoid being spoofed by other nodes your client will download a block, verify it, and then request the next block.  The blockchain is downloaded, verified, and indexed one block at a time.


Title: Re: downloading blocks is too slow
Post by: Frz on May 22, 2014, 09:41:08 AM
Are there plans to do block verification (of multiple blocks) in parralel?
Especially during bootstrap.dat loading this would be very useful (because we all have multi core machines) - possibly even utilising the gpu to speed up the process some more.


Title: Re: downloading blocks is too slow
Post by: TierNolan on May 22, 2014, 11:36:09 AM
The way the initial block download works, your client goes and asks somebody, "what's the current block", and they say, BLOCK ID XXXXX.  And then since you don't have XXXXX, you ask them for that and they send it to you, but after you download it, then you look at its block header and discover that the previous block was WWWWW - and you don' t have that one either!  So you ask, "Hey could you give me  a copy of Block WWWWW?"  And they send that, and you look at it, and its parent was VVVVVV, and you don' t have that either, so you ask.....  and so on, back to the Genesis block.

No it doesn't.

If you had blocks from 0 to 10000, you would send something like.

I have (hashes of)

block 10000
block 9999
block 9998
....
block 9990
block 9900
block 9000
block 8000
block 5000
block 2000
block 500
block 0

The peer then scans the list and starts with the one first on the main chain.  If your block 10000 wasn't on the main chain, but block 9990 was, then it would start at block 9990.

Normally, it starts from the most recent block you have, unless you are somehow on a fork of the main chain.

Quote
Then there's a CPU bottleneck to worry about.  Your peer does all kinds of cryptographic checks on each downloaded block, so if your CPU is slow, that may even dominate the latency time.  

This is generally the big one.


Title: Re: downloading blocks is too slow
Post by: zvs on May 22, 2014, 06:19:53 PM
Quote
Then there's a CPU bottleneck to worry about.  Your peer does all kinds of cryptographic checks on each downloaded block, so if your CPU is slow, that may even dominate the latency time.  

Maybe it's 'cause I have a fast CPU (i7-4700), but my problem was getting peers that uploaded too slow.  Each time it requests blocks from some peer w/ little or no upstream, you waste several minutes..

That's why it's better to just connect to one fast peer.


Title: Re: downloading blocks is too slow
Post by: DeathAndTaxes on May 22, 2014, 06:38:54 PM
You would need a really slow CPU for it to be the bottleneck.  zvs is right, your bootstrapping speed will depend heavily based on the peer you connect to.  If that client has sub dialup speed uploads bandwidth (maybe because they are "leet" and have 860 inbound connections on their 1Mbps DSL line).  

It hasn't gotten much focus because it works and it isn't a critical upgrade but as the network becomes larger and the bootstrap times longer the clients will need to become better multitaskers.  Queueing up blocks, analyzing peer connection speeds, downloading in parallel, etc.   Optimally there should be a solid queue of blocks building up because other bottlenecks (probably disk unless you have a fast SSD) can't keep up.  Other ideas to explore would be to download headers only first, find, and verify the best chain.  The node now has a complete set of validated headers from the genesis block to current and can request block bodies from multiple peers and fill in the framework.  It also would make sense to throttle the number of connections a peer has based on its connectivity and maybe have peers distinguish between up to date nodes and bootstrapping nodes (like seeder and leacher terms used in bittorrent).  For a given amount of upstream bandwidth a node can support more "synced" peers than it can "bootstrapping" peers.   One could define the difference as a node who is more than two hours behind the best chain as bootstrapping and nodes would share that with other nodes when the status changes.  


Title: Re: downloading blocks is too slow
Post by: Sukrim on May 23, 2014, 01:46:54 AM
Headers first sync is now talked about for a long time, apparently sipa is working on it at the moment (or not, he didn't push commits regarding this to github in the last 2 weeks). Hopefully it'll be ready for 0.10.0, it was already rumored to be in 0.9 but apparently didn't make the cut.