CPU overheating during node sync + checksum errors

[takuma sato]

Im syncing a Bitcoin Knots node from scratch on Debian and I have encountered 2 errors already that have to do with the

Code:

"Fatal LevelDB error: Corruption: block checksum mismatch"

error. As the blocks downloaded I came to see the progress and there was an error window. This happened again after some hours. Luckily im able to keep syncing after I did reset the OS and I did not need to reindex the entire thing or god forbid redownload the entire thing which would be hell. So now it's syncing normally. Im 53.45% in, and according to the GUI it says 18 hours to go but that probably fluctuates as more block activity is encountered.

Anyway, the thing is, im running on an i5 laptop and the CPU cores are at a constant wooping 95ºC/203ºF. sensors say that 100+ is critical.

I was wondering if this is what is causing the checksum errors. Is there a way to lower the amount of CPU power being used by Bitcoin so that I can run this at a lower temperature?

Im running with a dbcache of 2048 which I think is reasonable for 8GB RAM and I have already lowered par to =1.

Another thing to consider is that this is running from an USB SSD portable disk, so I wonder if USB I/O isn't that great for this. But I really wanted the portability of having a node on the go as you travel, and I thought it would be better than doing this in an USB pendrive. The disk also has FDE.

Another thing would be the RAM. I have not tested with memtest.

So something is causing these errors. Hopefully I can finishing syncing without further errors wtih the par change but who knows. I would like to lower the temperature at least to 75ºC/ºF176 because the CPU is taking a beating from 3 days already of maximun speed.

There's a command in linux called cpulimit that allows you to use a % of the cpu given a maximun value. Has anyone tried this with bitcoin-qt?

Edit: Perhaps move this to the Technical Support subforum.

[nc50lc]

Anyway, the thing is, im running on an i5 laptop and the CPU cores are at a constant wooping 95ºC/203ºF. sensors say that 100+ is critical.
-snip-
I would like to lower the temperature at least to 75ºC/ºF176 because the CPU is taking a beating from 3 days already of maximun speed.

How about the hardware? When did you last cleaned your laptop's internals?
That overheating is a striking hardware issue probably with your laptop's cooling since Bitcoin Core already set not to utilize it to its limit.
With throttling and proper cooling, it shouldn't reach critical levels so check your laptop's heatsink, fan and thermal paste for dust buildups.

If if's clean and the thermal paste isn't crumbly yet... that may be a symptom of failing CPU if it's not at 100% utilization.
Or course, check your CPU's spreadsheet for its normal/max temp since there are a lot of different generations of i5.

[ABCbits]

Im syncing a Bitcoin Knots node from scratch on Debian

Im running with a dbcache of 2048 which I think is reasonable for 8GB RAM and I have already lowered par to =1.

If your laptop only used to sync Bitcoin Core, i'm sure you can use higher dbcache value since Debian doesn't use that much RAM to improve sync speed.

Another thing would be the RAM. I have not tested with memtest.

I don't see how RAM can make CPU overheat. But you may want to memtest it since you mentioned Bitcoin Knots exited itself.

Anyway, the thing is, im running on an i5 laptop and the CPU cores are at a constant wooping 95ºC/203ºF. sensors say that 100+ is critical.
-snip-
I would like to lower the temperature at least to 75ºC/ºF176 because the CPU is taking a beating from 3 days already of maximun speed.

How about the hardware? When did you last cleaned your laptop's internals?
That overheating is a striking hardware issue probably with your laptop's cooling since Bitcoin Core already set not to utilize it to its limit.
With throttling and proper cooling, it shouldn't reach critical levels so check your laptop's heatsink, fan and thermal paste for dust buildups.

If if's clean and the thermal paste isn't crumbly yet... that may be a symptom of failing CPU if it's not at 100% utilization.
Or course, check your CPU's spreadsheet for its normal/max temp since there are a lot of different generations of i5.

In addition, i would recommend OP to research about how to perform CPU undervolt. It's another way to reduce CPU temperature.

[DaveF]

...Or course, check your CPU's spreadsheet for its normal/max temp since there are a lot of different generations of i5....

Like over a dozen of them.

@takuma sato  what are the full specs of the laptop.
An early 2010 i5 is a lot different then a 12th gen

Same with the drive is it a new ssd is going to run a lot cooler then a spinning drive or even an old SSD.

-Dave

[takuma sato]

I have run the node syncing all night successfuly by using -par=1. So it looks like limiting it to 1 core did the trick. The progress speed didn't seem to be affected that much. However when I hit around 70% it has become incredibly slow. It's pushing 0.12% an hour. At this rate it will take 8 days. This is like a year ago remaining of blocks. The net activity graph shows huge gaps between the green bars. Hopefully this is not the result of schnorr signatures and then weird usage of the blockchain with more complex signatures that take more time. Because that looks like that (gaps between activity). In any case, something started happening about a year of blocks remaining where it has become so damn slow. I've tried restarting a few times to swap peers and no difference. Any ways around this?

[philipma1957]

I have run the node syncing all night successfuly by using -par=1. So it looks like limiting it to 1 core did the trick. The progress speed didn't seem to be affected that much. However when I hit around 70% it has become incredibly slow. It's pushing 0.12% an hour. At this rate it will take 8 days. This is like a year ago remaining of blocks. The net activity graph shows huge gaps between the green bars. Hopefully this is not the result of schnorr signatures and then weird usage of the blockchain with more complex signatures that take more time. Because that looks like that (gaps between activity). In any case, something started happening about a year of blocks remaining where it has become so damn slow. I've tried restarting a few times to swap peers and no difference. Any ways around this?

yeah tell us how big is the external ssd.

ie 1tb is not the best choice

2tb would be better.

tells us what gen is the i5

if the cpu is 8th gen or better it should be good enough

are you using wifi to load or eth cable?

[nc50lc]

Hopefully this is not the result of schnorr signatures and then weird usage of the blockchain with more complex signatures that take more time. Because that looks like that (gaps between activity). In any case, something started happening about a year of blocks remaining where it has become so damn slow.

Maybe not, your node probably caught up to the block height where your node starts validating all of the scripts compared to the lighter verifications done in the older blocks.

I'm talking about the default "assumevalid" block value.
If you're using v29.0, it's block height 886157 onwards: github.com/bitcoin/bitcoin/blob/v29.0/src/kernel/chainparams.cpp#L121

Since it's almost done syncing, I suggest not to touch that default value.

[LoyceV]

There's a command in linux called cpulimit that allows you to use a % of the cpu given a maximun value. Has anyone tried this with bitcoin-qt?

Instead of cpulimit, I'd look at reducing the CPU frequency. Read about scaling_governor powersave and cpufrequtils (but I've never tested it for this purpose).

so check your laptop's heatsink, fan and thermal paste for dust buildups.

When I run a laptop long term, I clean the fan from dust a few times per year.

However when I hit around 70% it has become incredibly slow. It's pushing 0.12% an hour. At this rate it will take 8 days.

The only way to solve this: get more RAM. My chainstate is 11 GB, your RAM is 8. Even with 16 GB it heavily relies on it's SSD to sync.

Look up the TDP for your laptop CPU. I never buy them >15W, but many laptop CPUs burn 35 to 45W (or even more). All this power is turned into heat. Intel built CPUs that burn a maximum of 6W and don't even need a fan, but not many laptops use them.

[takuma sato]

There's a command in linux called cpulimit that allows you to use a % of the cpu given a maximun value. Has anyone tried this with bitcoin-qt?

Instead of cpulimit, I'd look at reducing the CPU frequency. Read about scaling_governor powersave and cpufrequtils (but I've never tested it for this purpose).

so check your laptop's heatsink, fan and thermal paste for dust buildups.

When I run a laptop long term, I clean the fan from dust a few times per year.

However when I hit around 70% it has become incredibly slow. It's pushing 0.12% an hour. At this rate it will take 8 days.

The only way to solve this: get more RAM. My chainstate is 11 GB, your RAM is 8. Even with 16 GB it heavily relies on it's SSD to sync.

Look up the TDP for your laptop CPU. I never buy them >15W, but many laptop CPUs burn 35 to 45W (or even more). All this power is turned into heat. Intel built CPUs that burn a maximum of 6W and don't even need a fan, but not many laptops use them.

This is not normal. At current speed it will take now like 7 weeks. The speed just gets worse. Something going on here. I may have to give up on the laptop and try on my desktop by plugging the USB and hopefully it works (since Debian was installed on the laptop and it would be a different hardware)

I have run the node syncing all night successfuly by using -par=1. So it looks like limiting it to 1 core did the trick. The progress speed didn't seem to be affected that much. However when I hit around 70% it has become incredibly slow. It's pushing 0.12% an hour. At this rate it will take 8 days. This is like a year ago remaining of blocks. The net activity graph shows huge gaps between the green bars. Hopefully this is not the result of schnorr signatures and then weird usage of the blockchain with more complex signatures that take more time. Because that looks like that (gaps between activity). In any case, something started happening about a year of blocks remaining where it has become so damn slow. I've tried restarting a few times to swap peers and no difference. Any ways around this?

yeah tell us how big is the external ssd.

ie 1tb is not the best choice

2tb would be better.

tells us what gen is the i5

if the cpu is 8th gen or better it should be good enough

are you using wifi to load or eth cable?

It's a Sandy Bridge, it's enough, and it's eth cable plugged to a fast connection and 2 TB SSD. There's something going on that is making it insanely slow. It should be faster even on an older computer.

[LoyceV]

This is not normal. At current speed it will take now like 7 weeks. The speed just gets worse. Something going on here. I may have to give up on the laptop and try on my desktop by plugging the USB and hopefully it works (since Debian was installed on the laptop and it would be a different hardware)

Actually, it is normal when there's a bottleneck. Find the bottleneck, and it should improve (until the next bottleneck pops up).
What disk is in the laptop? If it's SSD: symlink chainstate to your local disk.

[ABCbits]

--snip--

It's a Sandy Bridge, it's enough, and it's eth cable plugged to a fast connection and 2 TB SSD. There's something going on that is making it insanely slow. It should be faster even on an older computer.

Yeah, it's far too slow. I would guess it's caused either by
1. Very low dbcache value (only 2GB), while peak UTXO size was at 12.8GB on late 2024[1].
2. If you use your 2TB SSD to store other data, it's worth to know that some SSD become noticeably slower when it's getting full.
3. Too much data have written on your SSD and it switched from pSLC to either TLC/QLC mode which makes it slower. Below image example is for sequential write, but i expect random I/O activity would be severally impacted as well.

But if you want to find the exact bottleneck/issue, you probably should start by using tool such as htop or iotop.

[1] https://statoshi.info/d/000000009/unspent-transaction-output-set.

[takuma sato]

This is not normal. At current speed it will take now like 7 weeks. The speed just gets worse. Something going on here. I may have to give up on the laptop and try on my desktop by plugging the USB and hopefully it works (since Debian was installed on the laptop and it would be a different hardware)

Actually, it is normal when there's a bottleneck. Find the bottleneck, and it should improve (until the next bottleneck pops up).
What disk is in the laptop? If it's SSD: symlink chainstate to your local disk.

The entire OS + node is installed within the same external SSD. I don't want anything leaking outside of the encrypted drive. The idea was to be able to have a full node synced on the go and carry it with you if you travel. The SSD is dedicated only to running the node, so considering the blockchain is around 685MB,  and there is only the Debian OS installed, it should last until I need to upgrade.

Thankfully, it seems that after turning off the laptop for some hours and trying again, it's not at 2% per hour, which means at this rate it may finish within 10 hours, assuming it doesn't stop downloading again and huge gaps happen. So maybe the high temperature was making a problem or something. But it fluctuates, now it says 13 hours to go and 1.54%... but as long as it's not 7 weeks then I will consider this as expected performance. Edit: oh well now it's going slower again at 0.88%, 24 hours to go adnd it keeps getting slower.. hopefully it picks up speed again.

--snip--

It's a Sandy Bridge, it's enough, and it's eth cable plugged to a fast connection and 2 TB SSD. There's something going on that is making it insanely slow. It should be faster even on an older computer.

Yeah, it's far too slow. I would guess it's caused either by
1. Very low dbcache value (only 2GB), while peak UTXO size was at 12.8GB on late 2024[1].
2. If you use your 2TB SSD to store other data, it's worth to know that some SSD become noticeably slower when it's getting full.
3. Too much data have written on your SSD and it switched from pSLC to either TLC/QLC mode which makes it slower. Below image example is for sequential write, but i expect random I/O activity would be severally impacted as well.

But if you want to find the exact bottleneck/issue, you probably should start by using tool such as htop or iotop.

[1] https://statoshi.info/d/000000009/unspent-transaction-output-set.

Like I said above, the problem seems to have been fixed by turning off the laptop for some hours and trying again. So perhaps it was the high temps or who knows. Now temps are around 65 to below 75, which is manageable given the high summer temperature.

From what i've read, as far as dbcache goes you should use 1/3 of your ram.

In any case, this shows that we need Knots to become the #1 node and we need to stop the spam. We don't need all this non-monetary crap making the initial blockchain download an even bigger nightmare that scares away potential nodes that could be online from people that have laptops that they don't use lying around that could be used as nodes. It's clear now they are ruining Bitcoin with this nonsense being hosted on the blockchain.

[Satofan44]

Others have answered your thread pretty well, I would like to add something.

Another thing to consider is that this is running from an USB SSD portable disk, so I wonder if USB I/O isn't that great for this. But I really wanted the portability of having a node on the go as you travel, and I thought it would be better than doing this in an USB pendrive. The disk also has FDE.

While I do understand the use case, I would never recommend this. Putting the issue of speed aside, USB is not a reliable interface for something that is so intensive such as Bitcoin Core, neither the hardware nor software. All it takes is a small intermittent fault, a halt in the software or a twitch of the cable and you can have a big corruption.

Code:

"Fatal LevelDB error: Corruption: block checksum mismatch"

error.

I've done some experiments in the past and I could never have a reliable node on a USB drive. Neither in the earlier days on a HDD, nor these days on an SSD. The chance for something to go wrong is just way too high compared to internal storage. This doesn't mean that you can't still try, just beware that issues are expected.
Have you ever considered upgrading the laptop storage directly instead?

[LoyceV]

The entire OS + node is installed within the same external SSD. I don't want anything leaking outside of the encrypted drive. The idea was to be able to have a full node synced on the go and carry it with you if you travel.

Are you saying you'll run your own OS on external USB on someone else's computer? Why would you trust them, and why would they trust you? I'd never let anyone do that on my hardware! Why not also bring a laptop on your travels? I paid just over €100 for my disposable second hand travel laptop, including SSD upgrade. It's quiet, small, light, tough and ugly (that last one is a deliberate feature). The biggest drawback is the 8 GB RAM that can't be upgraded.

I still wouldn't recommend it for a full node though: if you use it once a week and it takes an hour to catch up syncing while slowing down your laptop to the point you can't do anything else, you can't really use it to make a quick payment.

[ABCbits]

From what i've read, as far as dbcache goes you should use 1/3 of your ram.

People who recommend that probably assume you also use your computer/laptop for other tasks. If you only sync Bitcoin Core and doesn't run other app that use lots of RAM, you can set it higher based on portion of free/available RAM.

I still wouldn't recommend it for a full node though: if you use it once a week and it takes an hour to catch up syncing while slowing down your laptop to the point you can't do anything else, you can't really use it to make a quick payment.

I agree. Some privacy can be preserved if you use SPV wallet that connect to your node or use wallet that use BIP 157/158 (such as Wasabi and Ginger wallet).

[Cricktor]

I've done some experiments in the past and I could never have a reliable node on a USB drive. Neither in the earlier days on a HDD, nor these days on an SSD. The chance for something to go wrong is just way too high compared to internal storage. This doesn't mean that you can't still try, just beware that issues are expected.

From first-hand experience I've been running a Raspiblitz and Umbrel node on two Raspi 4B with 8GiB RAM for years rock-solid reliably where the main blockchain data storage was a SATA SSD connected via an USB3-SATA-adapter to the Raspi. Never had any issues with the storage (of course the SSD (SanDisk SSD Plus) was a low-power model suitable for the limited total USB power budget of the Raspi 4B).

It doesn't contradict your statement. Anyway, I'm aware that a Raspi 4B isn't comparable to a laptop or desktop computer. If possible, I'd prefer to use something else than USB for reliable 24/7/365 storage attachment and use.

In June 2023 I did an experiment how fast Umbrel could sync the blockchain from scratch with a dbcache value near ~4800 for one of my above mentioned Raspis: it finished after 95h at blockheight 796033. And a Raspi 4B isn't known to have rocket fast access with such an USB3 adapter to a rather normal SATA 2.5in SSD.

[Vod]

I don't see how RAM can make CPU overheat.

If the RAM is faulty and the CPU needs to keep refreshing caches, it would affect it but in modern systems with ECC, the issue is usually caught before the CPU becomes involved.  Another thing for the OP to look at would be his BIOS - disabling certain features could cause his CPU extra work.

[takuma sato]

Others have answered your thread pretty well, I would like to add something.

Another thing to consider is that this is running from an USB SSD portable disk, so I wonder if USB I/O isn't that great for this. But I really wanted the portability of having a node on the go as you travel, and I thought it would be better than doing this in an USB pendrive. The disk also has FDE.

While I do understand the use case, I would never recommend this. Putting the issue of speed aside, USB is not a reliable interface for something that is so intensive such as Bitcoin Core, neither the hardware nor software. All it takes is a small intermittent fault, a halt in the software or a twitch of the cable and you can have a big corruption.

Code:

"Fatal LevelDB error: Corruption: block checksum mismatch"

error.

I've done some experiments in the past and I could never have a reliable node on a USB drive. Neither in the earlier days on a HDD, nor these days on an SSD. The chance for something to go wrong is just way too high compared to internal storage. This doesn't mean that you can't still try, just beware that issues are expected.
Have you ever considered upgrading the laptop storage directly instead?

I don't see this being an issue with 3.0+ interfaces. I managed to finish syncing the node with the desktop and once signed thankfully the laptop is able to keep up.
I like the portability of a small drive. Gives me more options.

The entire OS + node is installed within the same external SSD. I don't want anything leaking outside of the encrypted drive. The idea was to be able to have a full node synced on the go and carry it with you if you travel.

Are you saying you'll run your own OS on external USB on someone else's computer? Why would you trust them, and why would they trust you? I'd never let anyone do that on my hardware! Why not also bring a laptop on your travels? I paid just over €100 for my disposable second hand travel laptop, including SSD upgrade. It's quiet, small, light, tough and ugly (that last one is a deliberate feature). The biggest drawback is the 8 GB RAM that can't be upgraded.

I still wouldn't recommend it for a full node though: if you use it once a week and it takes an hour to catch up syncing while slowing down your laptop to the point you can't do anything else, you can't really use it to make a quick payment.

No, you can own diffferent laptops on different spots and avoid carrying a laptop crossing borders with important information which is not a great idea.