bitcoin core full node download blockchain slow

[22bits]

I have a fairly robust dedicated pc running ubuntu:
Processor Intel Core i5
Memory 8gb
Disk 500 gb ssd

The progress started quickly with about 70% in a couple of days.  Then it slowed down dramatically to where now  like today for example it has downloaded not even 1% in the last 12 hours, maybe getting like 2% per day at this point.

Been checking to debug.log no errors, running continuously with percentage complete increasing as expected (just slow). 
Config:
prune=3000
dbcache=16384

Looking for any suggestions to try out.  Seems like at this rate it would take a few weeks to load, that seems too slow, or it this how long it seems to take nowadays?

[DaveF]

1) I5 is a bit vague. The original Intel Core i5 was released almost 15 years ago. What gen is yours?

2) The 500 GB drive is not large enough you need at least 1 TB now. https://www.blockchain.com/explorer/charts/blocks-size

3) How fast is your connection? Downloading 540GB on a 1M connection is still going to take a while no matter what else you have.

-Dave

[BitMaxz]

Memory 8gb

Config:
dbcache=16384

It seems that you set the dbcache more than your current RAM memory size since you have 8GB of memory you shouldn't set it to more than 4096MB.

I suggest try to change it to 4096 but before you do that make sure to stop the syncing process first if you are using a GUI just exit the GUI, and it will shut down automatically if not you can use "bitcoin-cli stop" to avoid corruption reboot your PC and rerun everything then monitor it again if the syncing process still slow or not.

[Cricktor]

2) The 500 GB drive is not large enough you need at least 1 TB now. https://www.blockchain.com/explorer/charts/blocks-size

OP has prune=3000 in his bitcoin.conf, limiting his blockchain data to approx. 3GB. The directory of chainstate will still occupy ~12GB. Depending on how full the 500GB drive is with other stuff, it should still suffice.

You're correct when you speak of an unpruned, full archival, node which needs 611GB for the blockchain data alone. Here's my current storage usage:

Code:

$ du -hd1 .bitcoin/
611G	.bitcoin/blocks
12G	.bitcoin/chainstate
59G	.bitcoin/indexes
216M	.bitcoin/wallets
682G	.bitcoin/

As BitMaxz already pointed out, dbcache is way too large for only 8GB RAM. Use approx. half the RAM for dbcache. For an Initial Blockchain Download (IBD) you can also set the option blocksonly=1, so that your node is not bothered with transaction verification and relay and mempool stuff.

[22bits]

1) I5 is a bit vague. The original Intel Core i5 was released almost 15 years ago. What gen is yours?

Intel® Core™ i5-3470T CPU @ 2.90GHz × 4

2) The 500 GB drive is not large enough you need at least 1 TB now. https://www.blockchain.com/explorer/charts/blocks-size

OK, good info, thanks

3) How fast is your connection? Downloading 540GB on a 1M connection is still going to take a while no matter what else you have.

Just did a test, down = 197.94 Mbps, wired fiber

[22bits]

It seems that you set the dbcache more than your current RAM memory size since you have 8GB of memory you shouldn't set it to more than 4096MB.

I suggest try to change it to 4096 but before you do that make sure to stop the syncing process first if you are using a GUI just exit the GUI, and it will shut down automatically if not you can use "bitcoin-cli stop" to avoid corruption reboot your PC and rerun everything then monitor it again if the syncing process still slow or not.

Thanks, great suggestion.  I will do this.  I am running the bitcoind as a systemd service and am able to gracefully shut it down and restart it.

[22bits]

OP has prune=3000 in his bitcoin.conf, limiting his blockchain data to approx. 3GB. The directory of chainstate will still occupy ~12GB. Depending on how full the 500GB drive is with other stuff, it should still suffice.

OK, thanks, that tracks what I thought.

[22bits]

So just as an update, I am running the bitcoind as a systemd job.  I also have a crontab job to track hourly the progress.  The internet speed is at some times better than others.  I get between .03% and .15% per hour of progress.

I will update the cache parameter setting and retry.  

[I updated dbcache=4096 and will continue to monitor progress hourly, and update here after a while, thanks all, number one priority on my list to get this working:)]

[LoyceV]

I updated dbcache=4096 and will continue to monitor progress hourly

How is your memory load? My guess is you've reached the part where your SSD-speed (both reads and writes) is the limiting factor due to the low RAM. There's a huge difference between SSDs: some have very bad performance when it comes to combined reads and writes. If it's a budget SSD, that may be the cause.

On a similar setup, it took me less than 2 days to complete the IBD. See this post on my progress.

Considering the low price of RAM, if at all possible, add some That should make things run much smoother.

[22bits]

How is your memory load? My guess is you've reached the part where your SSD-speed (both reads and writes) is the limiting factor due to the low RAM. There's a huge difference between SSDs: some have very bad performance when it comes to combined reads and writes. If it's a budget SSD, that may be the cause.

Code:

               total        used        free      shared  buff/cache   available
Mem:         7909288     1471828      127476      235508     6309984     5935240
Swap:        2097148     1010240     1086908

I have not used #free before but the numbers don't seem to look to bad and are similar to your outputs.  Still not explaining my slow times as far as I can tell. 


Also after updating the dbcache setting still ridiculously slow like .09% per hour.  Like 10 times too slow. 

[LoyceV]

Also after updating the dbcache setting still ridiculously slow like .09% per hour.  Like 10 times too slow.

Have a look at hard disk io/load. I forgot what I used exactly, but Google gives many options.

Quick (but inconclusive) test: how long does this take?

Code:

time sync

[nc50lc]

Also after updating the dbcache setting still ridiculously slow like .09% per hour.  Like 10 times too slow.  

That new setting is fine.
Setting the database cache to 16GB while having 8GB of RAM is bad for your SSD since your system will be relying too much on page file.
Bitcoin Core use almost all of the allotted dbcache (about 15GB with your previous setting) to save the UTXO set to RAM before writing it on disk.
Since more than half is being moved from/to disk as page file, plus the data being dumped to the data directory,
your SSD must had been on a lot of strain before that could make IBD slower.
(apparent on HDD, not much on SSD, not on separate physical drives for OS and bitcoin datadir)

How about your peers? try: bitcoin-cli getpeerinfo
Is it populated with nodes with bad connections? Peers keep disconnecting? ('conntime' just recent)

[22bits]

How about your peers? try: bitcoin-cli getpeerinfo

shows a blank list.  I see continual but slow progress, does this indicate a problem that the list is blank?

Code:

core@core:~/.bitcoin$ bitcoin-cli getpeerinfo
[
]

time sync

Code:

core@core:~/.bitcoin$ time sync

real    0m0.416s
user    0m0.002s
sys     0m0.000s

not sure what this is showing me.

After thinking about it for a while my thinking at this point is I am getting somehow ISP throttled which fits the facts that initially I was seeing fairly normal progress.  It is being done in such a way as to not affect other activities I am doing on a separate computer.  

[nc50lc]

How about your peers? try: bitcoin-cli getpeerinfo

shows a blank list.  I see continual but slow progress, does this indicate a problem that the list is blank.

That outputs your list of peers, zero means that you're currently not connected to any Bitcoin node.
That must be the issue, low to zero peer means slow IBD since there's few/no node to download the blockchain from.

You must check your system for anything that blocks any connection to Bitcoin Core.
Start with your firewall.

[LoyceV]

time sync

Code:

core@core:~/.bitcoin$ time sync

real    0m0.416s
user    0m0.002s
sys     0m0.000s

not sure what this is showing me.

I have a very slow HDD in my system. When doing the above, it can take minutes to sync. It it would be many seconds for your SSD, I would have called it bad. At 0.4s, I can't really tell. For an idle system it's a few ms.

After thinking about it for a while my thinking at this point is I am getting somehow ISP throttled which fits the facts that initially I was seeing fairly normal progress.

That's easy to test: rename your Bitcoin Core directory and restart from scratch. If it's fast again, it's not your ISP.

[22bits]

That's easy to test: rename your Bitcoin Core directory and restart from scratch. If it's fast again, it's not your ISP.

I saved off the blocks, chainstate and debug.log and restarted.  Will know soon, good idea

To start with I see a bunch of peers in the getpeerinfo now....

[22bits]

I am confused.  So now at the start of running fresh it is quick again.

First hour = 4.18% which is good.

Not sure what is happening.

getpeerinfo still showing several peers.

Any clues or hints on why it would run fast at the beginning and then slow down?  I am not using a firewall.

[LoyceV]

Any clues or hints on why it would run fast at the beginning and then slow down?

Your chainstate directory gets bigger, and eventually doesn't fit your RAM anymore. That large increases the disk reads and writes. But still, about 10 times slower than mine shouldn't happen. What SSD do you have?

I started at the same rate:

Converted to progress per hour:

[22bits]

What SSD do you have?

500GB TOSHIBA MQ01ABF0

Interesting about the slow down under normal circumstances, makes sense.  I will see on this go round how progress is in a while.  I might get a 1000 mbps fiber dedicated separate line to troubleshoot further.

[LoyceV]

500GB TOSHIBA MQ01ABF0

That's not an SSD, it's a HDD. That explains everything! It needs to write several TB of data, read a lot of data, and have random access all the time. My SSD was already the limiting factor, it makes total sense your HDD is a factor 10 slower.

I might get a 1000 mbps fiber dedicated separate line to troubleshoot further.

Your internet connection isn't the bottleneck. There are only 2 things that can improve performance:
1. Get a SSD, preferrably a good brand (Samsung Evo for instance), not a budget disk. Check some reviews on Tom's Hardware to make sure you get a good one. But even a budget SSD will be much faster than your current HDD.
2. Add more RAM.

If you do both, you're entire computing experience will get much smoother. Without those upgrades, you should probably just use Electrum instead of Bitcoin Core.