bitcoin core full node download blockchain slow

[philipma1957]

an I 5  cpu
32gb ram
a 2tb ssd

and 200 or more internet

[LoyceMobile]

an I 5  cpu
32gb ram
a 2tb ssd

and 200 or more internet

That's excessive for a pruned node.

[22bits]

That's not an SSD, it's a HDD.

Ohh... had to double check that.  It is in a ThinkCentre M92P.  Need an upgrade, thanks

just use Electrum

Most stuff yes.  For this I am going to use JoinMarket once I get this stable.  I have it installed but haven't started using it yet.

Progress running after 13 hours this time around is 53.13%.

[LoyceV]

It is in a ThinkCentre M92P.  Need an upgrade, thanks

Google says it can handle 32 GB RAM. If that's correct, Bitcoin Core will even work fine with an HDD (I did it on a server recently).

For this I am going to use JoinMarket once I get this stable.  I have it installed but haven't started using it yet.

I haver no experience with JoinMarket.

Progress running after 13 hours this time around is 53.13%.

The thing is.... The first part isn't that hard on your hardware: your chainstate directory is probably still smaller than your available RAM. At the end of the IBD, it'll be 12 GB.

[22bits]

So I am doing a pretty extensive test on existing system as follows:
1.)  Save off previous chainstate, blocks (bitcoin-core db) and restart from 0.
2.)  Now noted that initial progress to 50% or so is good, normal.
3.)  Once it starts slowing down I am stopping the bitcoind.service for 12 hours, giving it a rest.  This is just in case there is some kind of weird throttling going on.
4.)  After rest period restarting bitcoind.service.
5.)  If slow, then I am going to order this -> Beelink SER5 MAX RYZEN 7 5800H Mini pc office + Ram32GB + Storage 1TB

It is looking like about to order it, just waiting to confirm that progress is now slow .....

The surprising aspect for me is normally things either work or they don't.  This is one of those particular things that becomes more and more resource consuming as it goes along, still working but going slower and slower.

OK just now:  hmmm, inconclusive.  First full hour after restart = .98%.  Total now at 70.58%. 

[nc50lc]

OK just now:  hmmm, inconclusive.  First full hour after restart = .98%.  Total now at 70.58%.  

The first hour may had been spent discovering peers, syncing "block headers", then the first few blocks.
Normally it should be finished in a few minutes but if you keep on getting slow peers and limited to 11 (when no inbound peers), that could take a while.

That will even be inconsistent in the later blocks.
Take note that Bitcoin Core has a hard-coded "assumed valid" block which dictates to skip the script verifications from the blocks before it.
Means that it would take longer to verify blocks newer than that indicated blocks hash.
Each older versions of Bitcoin Core have lower block height than the newer versions but you can set it with assumevalid=<block_hash> setting or --assumevalid=<block_hash> arg.
Recommended to use the latest version's default (link) if you're using an older Bitcoin Core version.

[LoyceV]

3.)  Once it starts slowing down I am stopping the bitcoind.service for 12 hours, giving it a rest.  This is just in case there is some kind of weird throttling going on.

This step doesn't help.

5.)  If slow, then I am going to order this -> Beelink SER5 MAX RYZEN 7 5800H Mini pc office + Ram32GB + Storage 1TB

That's more than sufficient to do a quick download.

This is one of those particular things that becomes more and more resource consuming as it goes along, still working but going slower and slower.

Correct.

[22bits]

The first hour may had been spent discovering peers, syncing "block headers", then the first few blocks.
Normally it should be finished in a few minutes but if you keep on getting slow peers and limited to 11 (when no inbound peers), that could take a while.

Actually when I said the first hour it is because I track from a crontab job each hour and before the first 'full' hour there was actually about 1/2 hour.

Still continuing to plug along, now at 74.59 doing just over 1% per hour. 

However another symptom of the system hardware being maxed out is now to do anything on the system I need to temporarily shutdown the bitcoind process.

So will report back after new hardware arrives and is set up. 

That's more than sufficient to do a quick download.

This was at the upper end of my hardware search, I am a bit annoyed I can't really find anything without windows on it, but so be it.  I would build one but for itx mini pc that looks like it is not fun. 

Thanks for all help + suggestions. 

[LoyceV]

This was at the upper end of my hardware search

Why not just add some RAM?

I am a bit annoyed I can't really find anything without windows on it, but so be it.

The "Microsoft tax", I know your pain

[22bits]

Why not just add some RAM?

The particular reason I have these older lenovo systems is that I use them as off grid tails systems and they don't have any networking (no wifi, bluetooth) built in.  Not so good for adding RAM or making any modifications.  Pain is done now, money deducted from CC, waiting for the under 20 hour system to arrive  Will time the event once it gets here.....

The "Microsoft tax", I know your pain

lol, glad I am not the only one !

[DaveF]

...
5.)  If slow, then I am going to order this -> Beelink SER5 MAX RYZEN 7 5800H Mini pc office + Ram32GB + Storage 1TB
...

Depending on where you are in the world since prices vary a lot you can get the 16GB Intel N100 version with a 512GB drive and then buy a 1TB and a m.2 to USB adapter for the 512GB that comes with the unit for 1/2 the price of what you are looking at. Yes it's 16GB vs 32GB but you get to reuse the 512GB m.2 for something.

AND the Intel uses a fraction of the power of the AMD so it's cheaper to run and less noise.

YES it's slower, but once the initial sync is done and it's just running the node & JoinMarket and a few other things it's more then enough.

-Dave

[LoyceV]

I use them as off grid tails systems

I get the feeling there's an interesting story behind this. Starting with: why Tails, if it's offline?

[22bits]

I get the feeling there's an interesting story behind this. Starting with: why Tails, if it's offline?

So my setup with the no wifi linux box is that I do a electrum transaction on one computer, then save it off and sign it on the second computer.  The second computer is an air-gapped not on-line linux that boots tails via a usb.  I use electrum on tails to sign anything.  It is just a way to have air gapped box. 

[22bits]

Depending on where you are in the world since prices vary a lot you can get the 16GB Intel N100 version with a 512GB drive and then buy a 1TB and a m.2 to USB adapter for the 512GB that comes with the unit for 1/2 the price of what you are looking at.

Thanks for that, I had looked at less expensive options also.

Status update:  the new system rips speed wise.  It got to 100% overnight.  So definitely was hw problem.  I really like the beelink SER5 max, looks well made.  I do not have hour by hour, if I re-index for any reason will post one. 

[Cricktor]

Some stats are readily written in the debug.log file of Bitcoin Core (time stamps and percentage progress when blocks are digested). If you're lucky, your whole IBD might be still logged.

By default this file will be pruned from time to time (not sure which threshold triggers shrinkage of it), unless you put shrinkdebugfile=0 in your bitcoin.conf. This tells Core to only ever append log entries to the debug.log and never shrink that file.

Glad you're happy with your new hardware and that it works smoothly.

[22bits]

Some stats are readily written in the debug.log file of Bitcoin Core (time stamps and percentage progress when blocks are digested). If you're lucky, your whole IBD might be still logged.

Thanks, yes I see that, can get exact times with likes that show 'progress='...

Glad you're happy with your new hardware and that it works smoothly.

Yeah, has really been a learning curve.  I initially would not have expected hardware to have such a prominent effect.

Here are the latest times.  I was hoping to finish in under 48 hours, was pleasantly surprised.
Hour 1: 8%
Hour 2: 15%
Hour 3: 28%
Hour 4: 40%
Hour 5: 52%
Hour 6: 62%
Hour 7: 70%
Hour 8: 78%
Hour 9: 87%
Hour 10: 95%
Hour 11: 100%

[LoyceV]

Here are the latest times.  I was hoping to finish in under 48 hours, was pleasantly surprised.
Hour 1: 8%
Hour 2: 15%
Hour 3: 28%
Hour 4: 40%
Hour 5: 52%
Hour 6: 62%
Hour 7: 70%
Hour 8: 78%
Hour 9: 87%
Hour 10: 95%
Hour 11: 100%

So with enough RAM, the progress stays more or less linear. Did you increase dbcache settings? I'm not sure if it would still matter: Linux can handle file cache on it's own.

[22bits]

So with enough RAM, the progress stays more or less linear. Did you increase dbcache settings?

I was going to set it to 12000 but ended up setting it to 8000 for the above run.  Yes looks basically linear