Faster sync would be nice (i.e. soooo slow)

[galmok]

During windows update, my pc crashed for some unknown reason and that apparently affected my bitcoin data as it decided synchronize everything. During this it completely trashes my disk with seeks (rotating disk), and currently claims it will need 2 weeks to 5 years to finish synchronizing!

This simply wont do. It is a complete deterrent for anyone wanting to host full nodes.

Current progress increase per hour is at -0.00% (yes, it is negative). Just before it was at 0.05%.

There must be a quicker way to synchronize this. I mean, I could download the full archive in 2 hours, maxing my internet and disk speed. If only I could find someone offering a recent and reliable copy of the bitcoin data for me to download.

bitcoin.exe consumes approx 5% cpu, about 1 GB mem, and has IO approx 5-10MBps.

It feels like bitcoin is attempting to work on several blocks at the same time, which will absolutely kill a rotating disk. Is that something it would do? Can it be made to just take one block at a time?

I don't know how to speed this up, and at this point, I am considering just to stop hosting this node.

Bitcoin Client Software and Version Number: 19.0.1
Operating System: Windows 10 Pro
System Hardware Specs: Core i7 920 @ 3.6GHz, 500 Mbps internet, 12GB RAM, 7200 rpms 8TB disk for bitcoin data.
Description of Problem: So very slow at synchronizing.
Any Related Addresses: -
Any Related Transaction IDs: -
Screenshot of the problem: -
Log Files from the Bitcoin Client: -

[1714971815]

1714971815

[1714971815]

1714971815

[1714971815]

1714971815

[1714971815]

1714971815

[BitMaxz]

Can you edit the bitcoin.conf and add this value below?

Code:

dbcache=4000

Then test if there is an improvement in syncing performance.

Since your memory is 12gb you can increase the dbcache more to 10,000mb or less. Test them every time you change the dbcache just don't set it higher than 10,000mb it will use all of your ram memory and it will crash.

[galmok]

I managed to stop bitcoin (it was pretty much unresponsive while trashing the disk, allowing for a mouse click every minute or so) and had to find the configuration settings, which are not in bitcoin.conf. That file is empty.

I found the settings in the registry, and I modified this setting to read 4096:

nDatabaseCache

Then I started bitcoin again. After the initial loading of the index, it started synchronizing again, but much faster. Currently it is using 900MB memory (less than before) yet it performs much better.

Could there be a memory leak filling up the cache with needless information seeing as it is running "fine" (still slow) now and uses less memory than before where it ran horrible? Maybe a simple restart would have been enough?

Anyway, I hope you guys figure out a way to make synchronization be completed way faster. It should take hours, not days.

[galmok]

Well, "much faster" was an overstatement. Currently it claims it needs 3 days to complete the synchronization (with 0.54% increase per hour).

It still trashes the disk a lot, so I guess most of the time is spent waiting for IO (seeks).

It uses 1.5GB ram, no network IO, and disk averages about 50MBps (which is much more than before).

I guess this is as good as it gets.

I did find a webpage offering torrent downloads of the bitcoin database, but it was discontinued and only an old database was available. That could have solved the synchronization issue in a few hours...

[philipma1957]

 well a lot depends on why you run your core.  is it to be a node and support the network?  Is it a wallet?

Is it both?

Some advice is use a pc like this one

https://www.ebay.com/itm/Lenovo-ThinkCentre-M93p-Tiny-Desktop-PC-2-90GHz-Core-i5-4570T-16GB-RAM-No-HDD/383427102572?

stick in a 1tb ssd load the core.

then clone it on a spare hdd.

so if it crashes the spare will not be far behind and take forever to synch.

[nc50lc]

Try to add: assumevalid=0000000000000000000b804d0a7c68afaf198cbae1cc16ea3613435ce77a9521 to your bitcoin.conf file.
That's the block hash of block height 618,873; with that setting, core will not verify all the blocks below the specified height.
But with your CPU/RAM, I think this will not dramatically increase your syncing speed [for blocks higher than 453354 (commit 3c02b95)]

Also temporarily add, blocksonly=1 So your node will not relay transactions that could add a very little strain to your HDD.

But as always pointed out to the same issue, it all comes down to the limitations of an HDD;
Consider upgrading to SSD (SLC: single-level cell type).

[HCP]

I did find a webpage offering torrent downloads of the bitcoin database, but it was discontinued and only an old database was available. That could have solved the synchronization issue in a few hours...

Synchronization is NOT simply downloading the blocks... everything needs to be verified. Otherwise, the main tenet "Don't trust, verify" is broken.

The large disk thrashing is likely your machine reading/writing chainstate info. You can also achieve a relatively good speed increase by moving the "chainstate" onto a high speed storage device like an SSD.

Have a read of this: https://en.bitcoin.it/wiki/Splitting_the_data_directory

[ABCbits]

That's the limitation of the HDD, it won't get much faster without using SSD (either to store whole blockchain or only store chainstate folder).

Consider upgrading to SSD (SLC: single-level cell type).

SSD which uses SLC isn't easy to find and quite expensive, unless you're talking SSD which uses SLC only for cache/buffer.

[NeuroticFish]

stick in a 1tb ssd load the core.

Actually a small SSD would do.
It's enough to temporarily symlink the chainstate folder to the SSD until the first/big sync is done, then that data can be moved to the HDD too.
And that's only some 4GB now.

[LFC_Bitcoin]

If you have a problem with syncing then maybe running a full node & or using bitcoin core as a wallet isn’t for you. There are plenty of lightweight wallets that can do a good job. I recommend electrum but make sure you download it from electrum.org

Hardware devices like Trezor are great too.

[LoyceV]

For reference: I recently re-synced Bitcoin Core on my 5 year old i3 laptop, running Linux, with 16 GB RAM. I set dbcache to 4096, chainstate on SSD, blocks on HDD, and while also using my laptop for other tasks, it completed syncing in about a day. During the first hours, my internet speed was the limiting factor, later on I think my CPU became limiting.

I've never been able to test it, but from what I've seen, Linux beats Windows on syncing times.

[galmok]

Thanks for all the comments.

I can see I was right to assume the harddisk was the problem.

But I can't replace it with a SSD as I have no more harddisk slots available and I cannot spare the large space offered by the rotating disks.

Anyway, I could be looking into moving the chainstate to a SSD, but I'd like to know the amount of writes it receives before doing so. Is it going to wear down an SSD fast (seeing as SLC is suggested)?

As mentioned, I set the buffer to 4096, and bitcoin now uses 5.6GB of memory. I guess there must be some extra memory consumed that scales with the buffer setting (besides the buffer itself).

Progress is now at 80% (up from 58% yesterday) so I guess a week more and I am done (if I don't do the chainstate optimisation). And if the chainstate is only about 4GB, maybe offer a fast sync, having bitcoin temporarily cache the chainstate in memory and only do bulk writes occasionally? That way, rotating disks would be able to keep up during synchronization.

[nc50lc]

(seeing as SLC is suggested)?

If you can get your hands on an industrial grade SSD which has 'SLC' "flash type", that'll be a great replacement for your HDD.
Because it's expected to last for more than 20 decades even with extreme read/write processes.

If not, you can settle with consumer grade SSD with 'SLC cache' and 'MLC' flash, the life expectancy for that type is about a decade.
The key is not filling the drive with data over 80%, the lower the free space, the better.

Also, the old SSD issue "expensive price per GB" is almost solved by now,
For example: You can buy a 1TB SLC Cache/MLC flash SSD for $100 today (consumer grade).

[philipma1957]

yeah op’s sin is he want to use the pc for multiple tasks with a large hdd.

his first full sized synch is a bear.

I was trying to see just how fast I could do a new full wallet.

i used a 2tb gen 4 cosair ssd
16gb ram
a ryzen 9 3900x cpu.

200mb internet connection

took me around six hours.

this is a high end gaming machine.

i guess it would cost around 2k to build it.

i suspect the 200mb internet is why it took six hours.

i would need a new modem from cable and i could do up to 400 mb i am sure this pc would load the core quicker then six hours.


but people do not want to spend 2k for a full node.

[galmok]

yeah op’s sin is he want to use the pc for multiple tasks with a large hdd.

his first full sized synch is a bear.

I have been running bitcoin core for many years with default settings. As I mentioned, the database died due to a windows update causing a BSOD.

And everyone claiming fast speeds are using SSD's. I have a rotating disk (for bitcoin data). I have several other disks in the system, mostly rotating, but also 2 SSD's (that I will not overload with wear from bitcoin, as it seems to want to trash the disk quite a bit).

I merely feel there is basis for optimizations so bitcoin synchronization could run much better on rotating disks, and to use less memory than 5+GB while doing so.

Anyway, my DB is synchronized and I will now reduce DB setting to the default 300MB again.

[Philipma1957cellphone]

Out of curiosity I set up another pc with 2 sticks of 16gb ram. 32gb total but 2400 speed. I used a 1 tb nvme 2 western digital black a fatality x370 mobo and an amd 2400g same 200 mb internet this took 13 hours for full startup.  So even good dedicated pc takes at best 4 hours. More likely 6 to 12.  I would say clone the drive every three months make two copies of it.  You will never be more then three months behind.  It is cheaper then any other method.

[achow101]

I managed to stop bitcoin (it was pretty much unresponsive while trashing the disk, allowing for a mouse click every minute or so) and had to find the configuration settings, which are not in bitcoin.conf. That file is empty.

I found the settings in the registry, and I modified this setting to read 4096:

nDatabaseCache

You probably shouldn't be modifying the registry values. Those are Qt specific settings, if you ever run the bitcoind rather than bitcoin-qt, then those settings won't transfer. Only bitcoin.conf is universal to both.

The bitcoin.conf file is supposed to be empty, it has no settings to start with. You add things to it. So you would add the dbacache setting as it is not already there.

Anyway, I could be looking into moving the chainstate to a SSD, but I'd like to know the amount of writes it receives before doing so. Is it going to wear down an SSD fast (seeing as SLC is suggested)?

There are fewer writes with a higher dbcache. While I don't have exact numbers, with modern hardware you will probably move on to newer hardware before seeing any noticeable degraded performance. And SLC lasts longer too.

As mentioned, I set the buffer to 4096, and bitcoin now uses 5.6GB of memory. I guess there must be some extra memory consumed that scales with the buffer setting (besides the buffer itself).

Yes. dbcache is just the database cache. There is other memory being used that do not have user configurable parameters.

Progress is now at 80% (up from 58% yesterday) so I guess a week more and I am done (if I don't do the chainstate optimisation). And if the chainstate is only about 4GB, maybe offer a fast sync, having bitcoin temporarily cache the chainstate in memory and only do bulk writes occasionally? That way, rotating disks would be able to keep up during synchronization.

IIRC the database uncompressed in memory (leveldb does some compression on disk), is 6 or 7 GB. With dbcache=8000, the whole thing fits in memory. This significantly boosts performance because it basically never gets flushed to disk. Perhaps there could be a dynamic dbcache thing where Core tries to determine how large it should be, but no one has done that yet. For now, the default is low so that Core works on low end hardware too.

I merely feel there is basis for optimizations so bitcoin synchronization could run much better on rotating disks, and to use less memory than 5+GB while doing so.

Feel free to try to do this yourself. Every release contains several optimizations but at the same time, we are fighting against blockchain growth. If we improve sync time by 20%, but the blockchain grows 20% larger, then it's a wash and the sync time doesn't change. It's hard to make IBD go faster, but there are continual improvements to try to make it faster.