Ridiculously slow synchronization

[stdset]

I am trying to setup a full node. I use a VPS with Ubuntu 16.04, 8 GB RAM. It's already more than 2 months since I installed bitcoind from ppa:bitcoin/bitcoin. Since then I upgraded it 2 times to 0.13.0 and to 0.13.1. The problem is: blocks downloading stalls from time to time. In this condition (without downloading a single block) bitcoind stays for days. When I first noticed a stall somewhere around block 300000, I thought that maybe some downloaded data was damaged, removed downloaded blockchain completely and started synchronization again, but the problem occured again. Then I tryed restarting bitcoind with -reindex option. Reindexing takes significant time, but when it finishes downloading resumes at normal speed (tens, even hundreds of blocks per hour), but soon stalls again. Stalls don't last forever though, from time to time downloading resumes, but not for long. I configured the node to accept incoming connections and it does accepts them. I increased max number of open connections to 32, usually it shows 31 active connections. But it doesn't help.
Currently my node is in stalled mode, for more than a day, at block 422530, hash of the block:

Code:

000000000000000001fb4b21fce4d41ea07adff678ac1080fd2d5d3d5f9b08de

matches what block explorers show for that block.
Here are some 100 lines of debug.log:

Code:

2016-10-30 20:31:26 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19348
2016-10-30 20:31:27 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19349
2016-10-30 20:31:27 receive version message: /Satoshi:0.13.0/: version 70014, blocks=436655, us=ip_of_the_vps:34062, peer=19350
2016-10-30 20:31:30 Peer=19285 is stalling block download, disconnecting
2016-10-30 20:31:30 receive version message: /Satoshi:0.8.5/: version 70001, blocks=436655, us=ip_of_the_vps:35916, peer=19351
2016-10-30 20:31:31 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19353
2016-10-30 20:31:31 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19354
2016-10-30 20:31:32 Peer=19292 is stalling block download, disconnecting
2016-10-30 20:31:37 receive version message: /Satoshi:0.11.2/: version 70002, blocks=436655, us=ip_of_the_vps:42666, peer=19355
2016-10-30 20:31:40 Peer=19350 is stalling block download, disconnecting
2016-10-30 20:31:40 receive version message: /Satoshi:0.13.1/: version 70014, blocks=436655, us=ip_of_the_vps:45882, peer=19356
2016-10-30 20:31:42 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19357
2016-10-30 20:31:45 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19359
2016-10-30 20:31:48 receive version message: /bitcoin-seeder:0.01/: version 60000, blocks=350000, us=ip_of_the_vps:8333, peer=19360
2016-10-30 20:31:49 receive version message: /bitcoin-seeder:0.01/: version 60000, blocks=350000, us=ip_of_the_vps:8333, peer=19362
2016-10-30 20:31:53 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19363
2016-10-30 20:31:53 receive version message: /bitcoin-seeder:0.01/: version 60000, blocks=350000, us=ip_of_the_vps:8333, peer=19364
2016-10-30 20:31:57 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19365
2016-10-30 20:31:58 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19366
2016-10-30 20:32:02 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=402957, us=127.0.0.1:8333, peer=19367
2016-10-30 20:32:05 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=424596, us=127.0.0.1:8333, peer=19368
2016-10-30 20:32:08 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=402958, us=127.0.0.1:8333, peer=19369
2016-10-30 20:32:11 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19370
2016-10-30 20:32:11 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19371
2016-10-30 20:32:15 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19372
2016-10-30 20:32:15 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19373
2016-10-30 20:32:17 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19374
2016-10-30 20:32:17 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19375
2016-10-30 20:32:19 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19376
2016-10-30 20:32:19 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=402957, us=127.0.0.1:8333, peer=19377
2016-10-30 20:32:19 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19378
2016-10-30 20:32:20 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19379
2016-10-30 20:32:20 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19380
2016-10-30 20:32:21 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19381
2016-10-30 20:32:22 receive version message: /Satoshi:0.13.1/: version 70014, blocks=436655, us=ip_of_the_vps:8333, peer=19382
2016-10-30 20:32:22 socket recv error Connection reset by peer (104)
2016-10-30 20:32:25 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19383
2016-10-30 20:32:26 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19384
2016-10-30 20:32:27 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19385
2016-10-30 20:32:32 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19386
2016-10-30 20:32:32 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19387
2016-10-30 20:32:35 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19388
2016-10-30 20:32:42 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19390
2016-10-30 20:32:43 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19391
2016-10-30 20:32:47 receive version message: /Satoshi:0.13.1/: version 70014, blocks=436656, us=ip_of_the_vps:48816, peer=19392
2016-10-30 20:32:49 receive version message: /Satoshi:0.13.1/: version 70014, blocks=335678, us=ip_of_the_vps:8333, peer=19393
2016-10-30 20:32:51 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19394
2016-10-30 20:32:55 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19395
2016-10-30 20:32:56 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19396
2016-10-30 20:32:57 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19397
2016-10-30 20:32:57 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19398
2016-10-30 20:33:00 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19399
2016-10-30 20:33:03 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19400
2016-10-30 20:33:03 receive version message: /bitcoin-seeder:0.01/: version 60000, blocks=230000, us=ip_of_the_vps:8333, peer=19401
2016-10-30 20:33:05 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19403
2016-10-30 20:33:11 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19404
2016-10-30 20:33:13 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19405
2016-10-30 20:33:15 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19406
2016-10-30 20:33:18 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19407
2016-10-30 20:33:21 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19408
2016-10-30 20:33:21 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19409
2016-10-30 20:33:21 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19410
2016-10-30 20:33:23 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19411
2016-10-30 20:33:23 socket recv error Connection timed out (110)
2016-10-30 20:33:24 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19412
2016-10-30 20:33:29 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19413
2016-10-30 20:33:31 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=424601, us=127.0.0.1:8333, peer=19414
2016-10-30 20:33:37 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19416
2016-10-30 20:33:38 receive version message: /Satoshi:0.13.0/: version 70014, blocks=436656, us=ip_of_the_vps:34076, peer=19417
2016-10-30 20:33:40 Peer=19351 is stalling block download, disconnecting
2016-10-30 20:33:40 receive version message: /BitcoinUnlimited:0.12.1(EB16; AD4)/: version 80002, blocks=436656, us=ip_of_the_vps:39256, peer=19418
2016-10-30 20:33:42 Peer=19355 is stalling block download, disconnecting
2016-10-30 20:33:42 receive version message: /Satoshi:0.13.0/: version 70014, blocks=436656, us=ip_of_the_vps:53284, peer=19419
2016-10-30 20:33:44 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19420
2016-10-30 20:33:45 Peer=19417 is stalling block download, disconnecting
2016-10-30 20:33:46 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436655, us=127.0.0.1:8333, peer=19421
2016-10-30 20:33:49 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=424493, us=127.0.0.1:8333, peer=19422
2016-10-30 20:33:49 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19423
2016-10-30 20:33:50 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19424
2016-10-30 20:33:51 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19425
2016-10-30 20:33:58 receive version message: /bitcoin-seeder:0.01/: version 60000, blocks=350000, us=ip_of_the_vps:8333, peer=19426
2016-10-30 20:33:59 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19427
2016-10-30 20:33:59 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19428
2016-10-30 20:34:01 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19429
2016-10-30 20:34:02 receive version message: /bitcoin-seeder:0.01/: version 60000, blocks=350000, us=ip_of_the_vps:8333, peer=19430
2016-10-30 20:34:03 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19431
2016-10-30 20:34:04 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19432
2016-10-30 20:34:05 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19433
2016-10-30 20:34:07 receive version message: /bitcoinj:0.12.2/Bitcoin Wallet:2.9.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19434
2016-10-30 20:34:07 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=425211, us=127.0.0.1:8333, peer=19435
2016-10-30 20:34:08 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19436
2016-10-30 20:34:09 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19437
2016-10-30 20:34:10 receive version message: /bitcoinj:0.13.1/Bitsquare:0.3.3/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19438
2016-10-30 20:34:11 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19439
2016-10-30 20:34:17 receive version message: /bitcoinj:0.13.2/MultiBitHD:0.1.4/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19440
2016-10-30 20:34:17 receive version message: /bitcoinj:0.13.4/Bitcoin Wallet:4.46/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19441
2016-10-30 20:34:26 receive version message: /bitnodes.21.co:0.1/: version 70002, blocks=436655, us=ip_of_the_vps:8333, peer=19442
2016-10-30 20:34:31 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=436656, us=127.0.0.1:8333, peer=19443
2016-10-30 20:34:32 connect() to 193.15.16.4:8333 failed after select(): Connection refused (111)
2016-10-30 20:34:37 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=426679, us=127.0.0.1:8333, peer=19444

Again and again it explicitly states stalled block downloading. I guess in 2-4 weeks it will finally synchronize, but I'd prefer to sort out this problem anyway if possible.

[1715610843]

1715610843

[1715610843]

1715610843

[1715610843]

1715610843

[1715610843]

1715610843

[1715610843]

1715610843

[cr1776]

... I use a VPS with Ubuntu 16.04, 8 GB RAM ....

What are the other stats on the VPS?  Number of cores? Load average?  Is the host throttling it?  (Perhaps it is none of those things, but best to be thorough).

[achow101]

What are the specs of your vps? Do you happen to be using Amazon Web Services for your VPS?

Part of the issue is probably that you have a ton of connections from spam nodes. You may be under a DoS attack.

Can you please provide ALL of your debug.log? What you have given provides almost 0 useful information.

[stdset]

... I use a VPS with Ubuntu 16.04, 8 GB RAM ....

What are the other stats on the VPS?  Number of cores? Load average?  Is the host throttling it?  (Perhaps it is none of those things, but best to be thorough).

There's just one core, but current load average is: 0,07, 0,02, 0,00. I don't think it's throttling because I install and update packages without any problems.
About memory usage, that's what top says:

Code:

KiB Mem :  8175544 total,   131828 free,  1280856 used,  6762860 buff/cache
KiB Swap:  8385532 total,  8361176 free,    24356 used.  6558060 avail Mem 

As far as I understand this, almost all memory is in use. That's because I also use this VPS for NXT full node, it's written in Java, therefore high memory usage. But swap is barely used, so I guess it shouldn't be a problem. NXT node works without apparent issues btw, but their blockchain size and tps aren't comparable to Bitcoin of course.
To be more sure that the problem isn't traffic throttling, I just did this test (downloaded 1000Mb):

Code:

$ time wget -O /dev/null http://mirror.nl.leaseweb.net/speedtest/1000mb.bin

real	8m17.184s
user	0m2.172s
sys	0m6.616s

Meanwhile, current blocks count is still 422530.

[stdset]

Do you happen to be using Amazon Web Services for your VPS?

No, my VPS isn't from Amazon, it's from guys whom I know in person.

Can you please provide ALL of your debug.log? What you have given provides almost 0 useful information.

I hope first and last 1000 lines of debug.log are enough? You can see them here.
Looking at this I see that something changed when I manually stopped bitcoind (using 'bitcoin-cli stop' of course) probably to change some settings, maybe to manually add some nodes. Before the stop it was updating the tip and preallocating something, after the stop the log consists mostly of 'receive version' and 'stalling block download' messages.

[achow101]

Do you happen to be using Amazon Web Services for your VPS?

No, my VPS isn't from Amazon, it's from guys whom I know in person.

Ok. The concern there was that if you used AWS you could have been blocked by a ton of nodes due to node spammers using AWS. But that is not a concern.

Can you please provide ALL of your debug.log? What you have given provides almost 0 useful information.

I hope first and last 1000 lines of debug.log are enough? You can see them here.
Looking at this I see that something changed when I manually stopped bitcoind (using 'bitcoin-cli stop' of course) probably to change some settings, maybe to manually add some nodes. Before the stop it was updating the tip and preallocating something, after the stop the log consists mostly of 'receive version' and 'stalling block download' messages.

From what I can tell, your node just seems to be having a really hard time connecting to a proper node to give it blocks. It looks like your node is being spammed connections, probably the same ones that are using AWS.

Try going into the Bitcoin Core Data Directory and deleting the peers.dat file. Then start Bitcoin Core with the -listen=0 option set so that you don't receive any incoming connections. See if you sync then. Once you are fully synced, try again without that option. You should probably also en masse ban connections from AWS nodes.

[stdset]

From what I can tell, your node just seems to be having a really hard time connecting to a proper node to give it blocks. It looks like your node is being spammed connections, probably the same ones that are using AWS.

Try going into the Bitcoin Core Data Directory and deleting the peers.dat file. Then start Bitcoin Core with the -listen=0 option set so that you don't receive any incoming connections. See if you sync then. Once you are fully synced, try again without that option. You should probably also en masse ban connections from AWS nodes.

I did what you suggested. Now synchronization is slowly advancing. Here you can see new debug.log. It seems to me though, that blocks downloading resumed not due to measures you suggested, but thanks to some other reason. Because in the beginning of debug.log you can see familiar 'stalling block download' messages. That corresponds to my observation: downloading resumed not right after changing settings and restarting bitcoind.
Is it OK that my node states stalled block downloading just several seconds after connecting to a peer:

Code:

2016-10-31 21:46:02 receive version message: /Satoshi:0.13.1/: version 70014, blocks=436814, us=ip_of_the_vps:52858, peer=4
2016-10-31 21:46:11 Peer=4 is stalling block download, disconnecting

?

[achow101]

I did what you suggested. Now synchronization is slowly advancing. Here you can see new debug.log. It seems to me though, that blocks downloading resumed not due to measures you suggested, but thanks to some other reason. Because in the beginning of debug.log you can see familiar 'stalling block download' messages. That corresponds to my observation: downloading resumed not right after changing settings and restarting bitcoind.
Is it OK that my node states stalled block downloading just several seconds after connecting to a peer:

Code:

2016-10-31 21:46:02 receive version message: /Satoshi:0.13.1/: version 70014, blocks=436814, us=ip_of_the_vps:52858, peer=4
2016-10-31 21:46:11 Peer=4 is stalling block download, disconnecting

?

It looks like something is preventing you from receiving blocks. I don't think this is related to the software but rather something on your network. I am not quite sure. Perhaps your ISP is rate limiting you? What I am seeing is that the connections between you and the peers are being broken causing the problems here.

[stdset]

It looks like something is preventing you from receiving blocks. I don't think this is related to the software but rather something on your network. I am not quite sure. Perhaps your ISP is rate limiting you? What I am seeing is that the connections between you and the peers are being broken causing the problems here.

As far as I understand, with rate limiting, downloading 1 GB of data:

Code:

$ time wget -O /dev/null http://mirror.nl.leaseweb.net/speedtest/1000mb.bin

real	8m17.184s
user	0m2.172s
sys	0m6.616s

which I tested yesterday, when sync was stalled completely, should have taken much more time.