Bitcoin blockchain sync very slow on HDD

[LoyceV]

It must be said that this computer is not new, and my cryptocurrency stuff are running in a virtual machine.

May I ask why? I'd use a VM if I don't trust the software (such as Fork wallets). For daily wallets it's much more convenient to run them on your main system (or as a different user to separate file access).

[NotATether]

I will soon get 1gb internet and try to see how fast I can do this.

A 1Gbps line that consistently hits that speed can finish the initial block download in a couple hours as long as you also have an enterprise HDD (or SSD).

[philipma1957]

I will soon get 1gb internet and try to see how fast I can do this.

A 1Gbps line that consistently hits that speed can finish the initial block download in a couple hours as long as you also have an enterprise HDD (or SSD).

Yeah I am thinking of doing it using this ssd

https://www.samsung.com/us/computing/memory-storage/solid-state-drives/980-pro-w-heatsink-pcie-4-0-nvme-ssd-2tb-mz-v8p2t0cw/?

Oh BTW I found a pretty good deal on an elite mini 800 g9 with an i7 14700T cpu 64gb ram and a 4tb nvme.2 ssd the eth port has a 1Gbps line

https://www.ebay.com/itm/126560449790?

this pc would pretty much be a perfect node for the next 5 years

[DireWolfM14]

Oh BTW I found a pretty good deal on an elite mini 800 g9 with an i7 14700T cpu 64gb ram and a 4tb nvme.2 ssd the eth port has a 1Gbps line

https://www.ebay.com/itm/126560449790?

this pc would pretty much be a perfect node for the next 5 years

That's a pretty good system.  I like the size of it, too.  

My latest build was a on 1u server, which can be cumbersome if you're not set up to house servers.  Enterprise servers that have met their End of Life cycle get replaced regularly, and can be had for pretty darn cheap on Ebay.  With a 256 megabaud internet connection I was able to sync the blockchain in under 6 hours.  Syncing Monero still took a few days for some reason.

System: HP ProLiant DL360 G7 Server
Processors: 2x Intel Xeon X5650 @ 2.67GHz
Ram: 18x 4GB DDR3 (74GB total)
Storage: 4x 1TB SSD Raid 5 (2.8TB usable storage)
OS: Debian 12

The total build cost me about $330 a little more than a year ago.

[ranochigo]

A 1Gbps line that consistently hits that speed can finish the initial block download in a couple hours as long as you also have an enterprise HDD (or SSD).

My experience is that bandwidth isn't the key factor with your synchronization, because your peers are probably not going to be able to saturate your bandwidth. I've had similar performance with 2GBPS, 1GBPS and 200mbps. Most significant improvement probably came around when I got an NVMe.

[LoyceV]

A 1Gbps line that consistently hits that speed can finish the initial block download in a couple hours as long as you also have an enterprise HDD (or SSD).

With sufficient RAM, the type of disk barely matters. All it does is save the blocks, and save chainstate when you shut it down.

[ranochigo]

With sufficient RAM, the type of disk barely matters. All it does is save the blocks, and save chainstate when you shut it down.

While that is mostly true, I still find I/O to be the bottleneck at times after the default assumevalid block. With the default dbcache, my synchronization time is around ~9 hours but 20GB of dbcache brings it down to 7.5hours. That is with NVMe, when comparing SSD and HDD, the difference is far bigger.

I've also discovered one of the cons of maintaining the entire chainstate (maxing out the dbcache) recently. Having all of the chainstate in the dbcache will result in the chainstate having to be rebuilt upon ungraceful shutdown due to it not being flushed to disk. It also depends if it really is feasible to add more ram for IBD, if they have no use for all that extra ram.

[NotATether]

With sufficient RAM, the type of disk barely matters. All it does is save the blocks, and save chainstate when you shut it down.

While that is mostly true, I still find I/O to be the bottleneck at times after the default assumevalid block. With the default dbcache, my synchronization time is around ~9 hours but 20GB of dbcache brings it down to 7.5hours. That is with NVMe, when comparing SSD and HDD, the difference is far bigger.

I've also discovered one of the cons of maintaining the entire chainstate (maxing out the dbcache) recently. Having all of the chainstate in the dbcache will result in the chainstate having to be rebuilt upon ungraceful shutdown due to it not being flushed to disk. It also depends if it really is feasible to add more ram for IBD, if they have no use for all that extra ram.

I think there is a command in Bitcoin Core's RPC that will allow you to manually flush things like the chainstate to the disk though? savemempool or something like that.  It obviously saves the mempool data but I don't know if it also saves the chainstate as well.

You could do something like gracefully shutting down the node after the IBD and then restarting it though.

[ranochigo]

I think there is a command in Bitcoin Core's RPC that will allow you to manually flush things like the chainstate to the disk though? savemempool or something like that.  It obviously saves the mempool data but I don't know if it also saves the chainstate as well.

It doesn't. Chainstate and mempool are different. I'm not aware of any commands that flushes the chainstate automatically because the frequency should presumably should be governed by the dbcache capacity.

You could do something like gracefully shutting down the node after the IBD and then restarting it though.

So long as you don't suffer any hiccups during the IBD. It would be quite a pain to rebuild the chainstate again.

[LoyceV]

You could do something like gracefully shutting down the node after the IBD and then restarting it though.

Let's say you do this, and restart Bitcoin Core. Then, your system crashes and chainstate is messed up. Don't you still need to rebuild it from scratch? Or can it continue from the block where it was last saved to disk?

[satscraper]

I still find I/O to be the bottleneck at times after the default assumevalid block. With the default dbcache, my synchronization time is around ~9 hours but 20GB of dbcache brings it down to 7.5hours. That is with NVMe, when comparing SSD and HDD, the difference is far bigger.

True. I myself have experienced   the significant  reduction of synch time after  replacing my HDD (7200 rpm) by NVMe SSD.

 I/O value does matter when the case in hand is blockchain synchronization. Those who deny this fact were never fiddling with SSD.

BTW, the optimal value of dbcache is 1/4 RAM.

[nc50lc]

I think there is a command in Bitcoin Core's RPC that will allow you to manually flush things like the chainstate to the disk though? savemempool or something like that.  It obviously saves the mempool data but I don't know if it also saves the chainstate as well.

There are: dumptxoutset and loadtxoutset RPC commands, maybe these are the commands that you saw?
But dumptxoutset doesn't actually dump to the original chainstate, it just creates a file that can be loaded with loadtxoutset.

Although, it's useful for anyone who want to use the wallet without waiting for IBD since the loaded UTXO set (from someone/backup) will be used temporarily instead.
(currently only in RegTest, TestNet or Signet; ref: https://github.com/bitcoin/bitcoin/blob/master/src/kernel/chainparams.cpp#L176-L178)
Ref: https://github.com/bitcoin/bitcoin/blob/master/doc/design/assumeutxo.md

[ABCbits]

BTW, the optimal value of dbcache is 1/4 RAM.

I've seen some people make this suggestion, but what are the reason behind choosing 1/4 RAM? I would rather say majority available RAM capacity or current serialized UTXO size[1].

[1] https://statoshi.info/d/000000009/unspent-transaction-output-set

[satscraper]

BTW, the optimal value of dbcache is 1/4 RAM.

I've seen some people make this suggestion, but what are the reason behind choosing 1/4 RAM? I

The reason for such value is very simple. Bitcoin Core runs on machine with OS which needs RAM for its routine operations. Besides node itself perform plenty of tasks outside those ones related with its databases. Sure this value of 1/4 RAM is machines-average one, thus if one needs the best value for the given device/OS he must analyze the RAM usage before starting the synch procedure.  But if he prefer not to bother himself with such kind of analysis 1/4 RAM would be the best option for him.

[ABCbits]

BTW, the optimal value of dbcache is 1/4 RAM.

I've seen some people make this suggestion, but what are the reason behind choosing 1/4 RAM? I

The reason for such value is very simple. Bitcoin Core runs on machine with OS which needs RAM for its routine operations. Besides node itself perform plenty of tasks outside those ones related with its databases. Sure this value of 1/4 RAM is machines-average one, thus if one needs the best value for the given device/OS he must analyze the RAM usage before starting the synch procedure.  But if he prefer not to bother himself with such kind of analysis 1/4 RAM would be the best option for him.

I don't think Bitcoin Core do other intensive task (which use lots of RAM) during sync. But other than that, your suggestion makes sense, although it's on safer side.

[bizeodal]

With HDD and 16GB for dbcache, 5 weeks in 2 to 3 hours. GUI still stalled badly I think.

Some notes:

The lopp.net blog has some articles on sync speed

If you have 16GB RAM do not allocate all of it to dbcache. I allow 1.2GB for OS and other functions, and I don't use GUI. This means set dbcache to 14800

I've only ever used HDD
With 4GB RAM and an I3 CPU, I set dbcache to 2800 and the IBD took 12 days
With 8GB RAM, I set dbcache to 6800 and the IBD finished in 55 hours
With 32GB RAM, I set dbcache to 16000 and the IDB finished in 16 hours
I have 2 HDD. It is faster to have chainstate on one HDD and blocks on a different HDD
According to the documentation, dbcache can't be more than 16000
Speed is best monitored as transactions per minute/hour, not blocks
I never use the GUI
Blocks are only written sequentially, never updated after closing each BLK**.DAT
Chainstate is constantly inserted and deleted. Using bigger dbcache (up to 16GB) stores more of chainstate in RAM, reducing the time-consuming insert/delete writes to disk
IBD does not verify txinput signatures if "assumevalid" is enabled, until the most recent blocks. IBD slows down a lot when it gets to this block, but by then it's nearly finished

Things not yet tried ...
Chainstate is about 12GB, so it would be cheap to use a small SSD for chainstate and keep blocks on HDD

...

Actual numbers are useful, but few people bother to make measurements. This means the generic "get a SSD" advice is based in ignorance
The biggest determinant is dbcache. One of the recent lopp.net articles has some measurements
Most people asking about slow node initialization do not specify how long it's taking, or what they expected, do not mention how much RAM they have, and never mention dbcache
Most "how to" articles fail to mention the initialization time. This is because most of those bloggers never actually finish the IBD, and do not admit it

[bizeodal]

You could do something like gracefully shutting down the node after the IBD and then restarting it though.

Let's say you do this, and restart Bitcoin Core. Then, your system crashes and chainstate is messed up. Don't you still need to rebuild it from scratch? Or can it continue from the block where it was last saved to disk?

You need to reindex, which takes 3 to 5 hours and rebuilds chainstate

A clean shutdown in the middle of IBD will write chainstate to disk. Then if you backup chainstate plus the most recent BLKnnn.DAT, you have a checkpoint where chainstate is synced as at the most recent block in that BLK file. This assumes none of the older now-closed BLK files ever gets corrupted. Chainstate is about 12GB and BLK files are never bigger than 128MB, so it's a fairly light backup