Running a node too slowly

[deepskydiver]

Bitcoin Client Software and Version Number: 22.0.0
Operating System: Ubuntu 20.04.3 LTS
System Hardware Specs: Intel NUC w i3-4010u / 4 Gb RAM / 60Gb HDD / 1TB external Samsung SSD for blockchain
Description of Problem:

Rather than go over the many problems I've had let me focus on the problem now. I had to reindex because I had an error in one of the headers. This appears not to be finished despite running for 3 days now. It seems to be going progressively slower and getting nowhere. But I'd like to avoid a complete resync as I've already restarted once. I don't understand what the two %ages apply to and why it's still going so incredibly slowly when it's pulling between 200 and 300Mb a second off the HDD.
There are 10 peers connected (though little to no traffic to or from them) and no errors in the log.



.. help please.

[1714496094]

1714496094

[1714496094]

1714496094

[1714496094]

1714496094

[1714496094]

1714496094

[nc50lc]

There may be issues with the latest header(s) that you've received and for some reason, Bitcoin Core may have invalidated it.
Try to check your logs for error lines about "block headers".

Log file is in the 'data directory' named "debug.log".

[deepskydiver]

Yes - you're right, thank you. What are my options?
(I believe though that thus entry was from before my reindex)

[nc50lc]

Try to set the specific block hash as 'valid' again by using the command reconsiderblock followed by the marked-invalid block hash.
You can input the command in the console tab (Window->Console)

Example:

Code:

reconsiderblock 000000000000000000006d768186e0119892ce2d2ca74215637300dd1ecabebb

[deepskydiver]

Thanks, I'll try that. Though I don't have the hash value. The header error references a file position.
"Errors in block header at FlatFilePos(nFile=2248, nPos=75228559)"
Incidentally why does it have two % complete values?

[nc50lc]

Thanks, I'll try that. Though I don't have the hash value. The header error references a file position.
"Errors in block header at FlatFilePos(nFile=2248, nPos=75228559)"

Are there any other errors beside that line? It indicates a corrupted file.
Usually, you just have to -reindex to fix it but you already on the process so there must be an issue somewhere else.
Maybe it's still not done with the reindex because of the hardware since you're using a NUC device and an external SSD.

You can try to speed it up by using more of your RAM by increasing the "database cache" by a reasonable amount (Settings->Options...->Size of database cache).
Set a value depending on your usual RAM usage (e.g. Half), but do not set it too high.

While you're in the settings, see if you have "pruning" setting enabled since you might not be needing it because you have enough space in your external SSD.
Plus reindex will trigger a full resync if your blockchain is pruned.

Incidentally why does it have two % complete values?

The % above is just for the "block headers", the one below is for the entire blockchain (the actual sync progress).

[deepskydiver]

Thanks again.

There are no other errors in the log file, no. It was because of that error that I reindexed.
The database cache is set to half of my RAM too, thanks.
But I don't understand why it can be so slow when it's basically rebuilding the headers from the data on disk, yes?
I can see watching the process that it's reading at a rate between 3 and 400 Mb a second. So disk access is fine. But it's crawling through it. At this rate it's rebuilding the headers - from the data it already has - at 2% per day. I can't see how that adds up reading 300 Mb a second..?

And to be clear on the two percentages to check my understanding: The top one (the entire blockchain) is not moving. The lower one (rebuilding the headers) is going at a rate of a couple of percent per day.

[nc50lc]

There are no other errors in the log file, no. It was because of that error that I reindexed.
The database cache is set to half of my RAM too, thanks.
But I don't understand why it can be so slow when it's basically rebuilding the headers from the data on disk, yes?

I can only assume that there's a bottleneck in your hardware,
I haven't used an NUC device but it seems like comparable to same-generation RPi models when it comes with specs,
I've heard that it'll take weeks to fully sync Core with it, -reindex on the other hand is also resource intensive minus the internet bandwidth.

Sadly, I can't help much if it's the hardware since I don't use either devices.

[LoyceV]

60Gb HDD / 1TB external Samsung SSD for blockchain

In my experience, the chainstate directory is a bigger performance problem than the blocks directory. But even if it's on your SSD already, the fact that it's external isn't good for performance. Any chance you can exchange those disks? Take the SSD from it's enclosure and mount it at the place of the HDD. (Note that I've never used a NUC so I can't tell if this is possible).

it's pulling between 200 and 300Mb a second off the HDD

Any chance it's low on RAM and actively swapping? That would be terrible for performance.

[deepskydiver]

Any chance it's low on RAM and actively swapping? That's would be terrible for performance.

I think you've got it!

I reduced the database cache by half a Gb and it's transformed. System monitor shows it's reading at most a tenth the data it was, yet the bitcoin-qt process is at 70% CPU instead of between just 1 and 5. And it's working through over 1.3% per hour rather than 0.08%.
Thanks for that suggestion. You know when hard drives used to make noise swapping might have occurred to me, but it didn't here! Even though now I can see that the combination of memory used and swap was at maximum in system monitor.

Thank you so much.

[NotATether]

Is that 10 outgoing peers? Because that's the number that Bitcoin Core sets up for you when it first starts, using stuff like peer list files and DNS seeds.

You need to have additional, incoming peers connect with you in order to pull blocks. When you are verifying the blocks too slowly (whether by slow network speeds or slow CPU or read/write disk speeds), incoming peers might start dropping their connections with you thinking that your node is not replying to KeepAlive messages fast enough.