Stupidly slow bitcoin core syncing

[Prelude]

The issue is not with my computer. I started noticing this with the release of 0.10 or so. 0.11.2 still has the same issue for me. Being 24 hours behind can take half an hour to sync! Completely unacceptable. The wallet is installed on a Samsung SSD in a 5960X rig with a gigabit fiber connection, so don't tell me my computer is at fault. Seems I'm not the only one with this issue, but haven't been able to find a solution. Will I have to stop using the core wallet in favor of something else?

[1715001835]

1715001835

[unamis76]

Bitcoin Core being "slow" isn't a defect, it's just how it is. Maybe the nodes you're connected to aren't really that fast? I don't know. Either way that's something that one cannot really improve...

[Parazyd]

Well, there are really a few factors that affect this...
Your CPU is one of them, notice how usage goes up when you are syncing. It's also a huge amount of data.
Other thing, the nodes you're connected to probably don't have a Gbit fiber connection, so that might be a reason for slowness too.

[CIYAM]

I have been unable to sync for the last six months at least (if I run Bitcoin it will never catch up).

At first I thought it was just because I live in China but when I visited Australia for a month it still never caught up (despite having a reasonable home bandwidth connection there).

It might simply be that the number of txs and the fact that I am using an average laptop (that is now at least 5 years old) which is preventing it from catching up. I understand in the next release sigs will be processed much faster so hopefully I can still run a full node after that (otherwise I am going to have to use a SPV wallet).

[shorena]

What does the core build in bandwith monitor show? Does it show a large spike of downloading data at the beginning? This is usually the case for me, once the data is downloaded it takes a bit (not 30 minutes though) until its shown as fully synced. I have at max 32MBit/s downstream, but I never saturate that with core anyway.

If that is indeed the case, the reason would be disk or CPU.

If the data trickles in instead, its probably your connection to other nodes.

[CIYAM]

What does the core build in bandwith monitor show?

Unfortunately when I was in Australia I didn't bother to look at that - from here I typically see problems that are due to the GCF (very slow ping times, etc.).

I wish I had paid more attention to it in Australia (I was busy with some "real life" stuff at that time) but I would be very surprised if connectivity and bandwidth where the issue when I was there (am pretty sure I ran Bitcoin with and without a SOCKS proxy whilst I was there and my proxy is not in China).

[shorena]

What does the core build in bandwith monitor show?

Unfortunately when I was in Australia I didn't bother to look at that - from here I typically see problems that are due to the GCF (very slow ping times, etc.).

I wish I had paid more attention to it in Australia (I was busy with some "real life" stuff at that time) but I would be very surprised if connectivity and bandwidth where the issue when I was there (am pretty sure I ran Bitcoin with and without a SOCKS proxy whilst I was there and my proxy is not in China).

For your case I wonder if you cant connect to one of the miner nodes. They probably dont have their IPs public, but maybe you can get one if you ask. Connecting directly to them with -connect=IP could help you catch up.

[CIYAM]

For your case I wonder if you cant connect to one of the miner nodes. They probably dont have their IPs public, but maybe you can get one if you ask. Connecting directly to them with -connect=IP could help you catch up.

I did actually try adding nodes manually (via the console window) to help but unfortunately that didn't work either.

In general when it comes to bandwidth issues I tend to assume that because I live in China I will have troubles but because I could not sync the Bitcoin blockchain in Australia (when I had a month to do so) I do think that bandwidth can't be the only reason.

[Parazyd]

You can find extra-fast nodes to connect to here: https://blockchain.info/hub-nodes
Might help.

[CIYAM]

You can find extra-fast nodes to connect to here: https://blockchain.info/hub-nodes
Might help.

Yup - used that when I was in Aus - didn't help.

[unamis76]

What does the core build in bandwith monitor show? Does it show a large spike of downloading data at the beginning? This is usually the case for me, once the data is downloaded it takes a bit (not 30 minutes though) until its shown as fully synced. I have at max 32MBit/s downstream, but I never saturate that with core anyway.

If that is indeed the case, the reason would be disk or CPU.

If the data trickles in instead, its probably your connection to other nodes.

This is also the case for me, bandwidth monitor looks the same. ~ 24 hours worth of blockchain data usually takes from 30 to 45 minutes (old Core 2 Duo, 2GB Ram, 1TB WD Caviar Blue), but I've had it take an hour.

Core just takes time, it's just how it is

Not syncing at all is indeed strange, though...

[Jet Cash]

We need more full nodes, and you won't get them unless they get some reward for helping other nodes to sync.

[shorena]

I was away for a few days so perfect time to clock sync.

Was behind ~40 hours when I started. It was clearly working out of order, clearing several hours at once after some time. CPU is i5 4. Gen and stayed at 80% on all 4 cores. Disk is a cheap Crucial SSD.

Code:

2 min. - 105 MB -  3 nodes - 26h
3 min. - 150 MB -  4 nodes - 19h
4 min. - 153 MB -  4 nodes - done
now*   - 206 MB - 23 nodes
* few hours later

Net monitor:

[LFC_Bitcoin]

The issue is not with my computer. I started noticing this with the release of 0.10 or so. 0.11.2 still has the same issue for me. Being 24 hours behind can take half an hour to sync! Completely unacceptable. The wallet is installed on a Samsung SSD in a 5960X rig with a gigabit fiber connection, so don't tell me my computer is at fault. Seems I'm not the only one with this issue, but haven't been able to find a solution. Will I have to stop using the core wallet in favor of something else?

I agree that the current speed to sync is unacceptable but what can we do other than change to a crappy web wallet or lightweight client?

Not much.

It's just the way it is, as others have said we should see several improvements in the next release.

One of the older releases used to take me about 5 minutes to even open the wallet, I much prefer being able to see what % I'm on before the wallet & addresses load.

You still have the ridiculous sync wait then though

[zvs]

Mine always crashes around the same block.

Then starts over from block 1, 'reading the blockchain', with 1 core.  Which takes forever, and then gets an error and aborts again after 10 hours or so.  

like this

Code:

2016-01-05 11:53:27.625338 LoadBlockIndexDB: last block file = 410
2016-01-05 11:53:27.629531 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=94, size=60445069, heights=391569...391856, time=2016-01-03...2016-01-05)
2016-01-05 11:53:27.629589 Checking all blk files are present...
2016-01-05 11:53:27.700450 LoadBlockIndexDB: transaction index enabled
2016-01-05 11:53:27.700552 LoadBlockIndexDB: hashBestChain=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f height=0 date=2009-01-03 18:15:05 progress=0.000000
2016-01-05 11:53:27.701030 init message: Verifying blocks...
2016-01-05 11:53:27.701201  block index           16563ms
2016-01-05 11:53:27.701415 No wallet support compiled in!
2016-01-05 11:53:27.701433 init message: Activating best chain...
2016-01-05 11:53:27.764422   - Load block from disk: 0.36ms [0.00s]
2016-01-05 11:53:27.764506     - Sanity checks: 0.03ms [0.00s]
2016-01-05 11:53:27.764534     - Fork checks: 0.03ms [0.00s]
2016-01-05 11:53:27.764586       - Connect 1 transactions: 0.05ms (0.050ms/tx, 0.000ms/txin) [0.00s]
2016-01-05 11:53:27.764602     - Verify 0 txins: 0.07ms (0.000ms/txin) [0.00s]
2016-01-05 11:53:27.764653     - Index writing: 0.05ms [0.00s]
2016-01-05 11:53:27.764678     - Callbacks: 0.03ms [0.00s]
2016-01-05 11:53:27.764695   - Connect total: 0.30ms [0.00s]
2016-01-05 11:53:27.764710   - Flush: 0.01ms [0.00s]
2016-01-05 11:53:27.764725   - Writing chainstate: 0.01ms [0.00s]
2016-01-05 11:53:27.764782 UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048  height=1  log2_work=33.000022  tx=2  date=2009-01-09 02:54:25 progress=0.000000  cache=
0.0MiB(1tx)

It verifies the blocks starting from 1, with a single core.  Attempting a 'bitcoin-cli stop' says that it's currently reading block index, lol.

I had more success w/ the XT client, but I don't think they're really much different at all?

[cr1776]

You could have a block that is corrupted on disk which might mean re-downloading the block chain.

Mine always crashes around the same block.

Then starts over from block 1, 'reading the blockchain', with 1 core.  Which takes forever, and then gets an error and aborts again after 10 hours or so.  

like this

Code:

2016-01-05 11:53:27.625338 LoadBlockIndexDB: last block file = 410
2016-01-05 11:53:27.629531 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=94, size=60445069, heights=391569...391856, time=2016-01-03...2016-01-05)
2016-01-05 11:53:27.629589 Checking all blk files are present...
2016-01-05 11:53:27.700450 LoadBlockIndexDB: transaction index enabled
2016-01-05 11:53:27.700552 LoadBlockIndexDB: hashBestChain=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f height=0 date=2009-01-03 18:15:05 progress=0.000000
2016-01-05 11:53:27.701030 init message: Verifying blocks...
2016-01-05 11:53:27.701201  block index           16563ms
2016-01-05 11:53:27.701415 No wallet support compiled in!
2016-01-05 11:53:27.701433 init message: Activating best chain...
2016-01-05 11:53:27.764422   - Load block from disk: 0.36ms [0.00s]
2016-01-05 11:53:27.764506     - Sanity checks: 0.03ms [0.00s]
2016-01-05 11:53:27.764534     - Fork checks: 0.03ms [0.00s]
2016-01-05 11:53:27.764586       - Connect 1 transactions: 0.05ms (0.050ms/tx, 0.000ms/txin) [0.00s]
2016-01-05 11:53:27.764602     - Verify 0 txins: 0.07ms (0.000ms/txin) [0.00s]
2016-01-05 11:53:27.764653     - Index writing: 0.05ms [0.00s]
2016-01-05 11:53:27.764678     - Callbacks: 0.03ms [0.00s]
2016-01-05 11:53:27.764695   - Connect total: 0.30ms [0.00s]
2016-01-05 11:53:27.764710   - Flush: 0.01ms [0.00s]
2016-01-05 11:53:27.764725   - Writing chainstate: 0.01ms [0.00s]
2016-01-05 11:53:27.764782 UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048  height=1  log2_work=33.000022  tx=2  date=2009-01-09 02:54:25 progress=0.000000  cache=
0.0MiB(1tx)

It verifies the blocks starting from 1, with a single core.  Attempting a 'bitcoin-cli stop' says that it's currently reading block index, lol.

I had more success w/ the XT client, but I don't think they're really much different at all?

[ethought]

Running 0.12 on an Amazon instance and it is taking days to sync.

I noticed my daemon is only showing 8 connections.

The last few thousand blocks are going at a snails pace.

[shorena]

Running 0.12 on an Amazon instance and it is taking days to sync.

I noticed my daemon is only showing 8 connections.

The last few thousand blocks are going at a snails pace.

8 connections is pretty normal during sync. If 0.12 is slow its very likely the CPU. I managed to fully sync 0.12 in ~8.5 hours on 2 virtual i5 cores.

[grue]

Are you using windows? If so, while you're syncing, can you open resource monitor and post a screenshot of the overview tab?

[shorena]

Are you using windows? If so, while you're syncing, can you open resource monitor and post a screenshot of the overview tab?

I just remembered that it might also be interesting to see whether you are still downloading (can be seen via Help -> Debug Window -> Network Traffic).