Bitcoin Knots Update -> SSD Daily TBW Skyrockets

[vxyz123456]

Below I have included an image of a chart which overlays what software my node was running on top of the system's cumulative gigabyte/terabyte written value on the SSD. The OS is Xubuntu on an 8th gen Intel Celeron with a SATA SSD.

https://i.imgur.com/KHnpk9X.png

Whenever I update to a version of Bitcoin Knots greater than 28.1, the disk writes skyrocket. Previously disk writes were at just under 20 GB / day, now they are at 230 GB / day, over a tenfold increase. I believe something about the new node version is causing the indexer (I'm using Fulcrum) to work extra hard or something.

I have no idea why. Can anybody assist? Why would using a newer version of Bitcoin knots cause so many more disk writes? I'm running on an SSD and I don't want to kill it prematurely.

Is there some setting I can tweak to prevent this? What changed between 28.1 and 29.3 that would cause the indexer to do so many extra disk writes?

I asked a similar question a while ago, but about the indexer electrumX (rather than the node version), as I tried it initially and found it was doing high disk writes. That was the reason I switched to Fulcrum:
https://bitcointalk.org/index.php?topic=5501257.0

Other relevant system info:
- Fulcrum 1.11.0.
- Xubuntu 24.04 LTS
- 8 GB of RAM
- Intel 8th gen Celeron CPU (2 cores, 2 threads).
- With Fulcrum running and Knots 29.3 20260508 going, the task manager only shows 2.0 / 8 GB of RAM in use, so I don't think this is a SWAP issue.

[LoyceV]

I asked a similar question a while ago ~
https://bitcointalk.org/index.php?topic=5501257.0

Your problem is the same as it was 2 years ago:

- 8 GB of RAM

- With Fulcrum running and Knots 29.3 20260508 going, the task manager only shows 2.0 / 8 GB of RAM in use, so I don't think this is a SWAP issue.

The problem isn't swap, it's chainstate. If it doesn't fit in RAM, it needs to be largely rewritten for every new block. Two years ago it was 12 GB, now it's 13 GB.

What changed between 28.1 and 29.3 that would cause the indexer to do so many extra disk writes?

I can't tell you this, I've never used Bitcoin Knots. But RAM would be the first thing to upgrade.

[ABCbits]

- With Fulcrum running and Knots 29.3 20260508 going, the task manager only shows 2.0 / 8 GB of RAM in use, so I don't think this is a SWAP issue.

The problem isn't swap, it's chainstate. If it doesn't fit in RAM, it needs to be largely rewritten for every new block. Two years ago it was 12 GB, now it's 13 GB.

For additional information, UTXO/chainstate size passed 8GB in middle of 2023[1]. That's why the total write suddenly skyrocket. It's when Ordinal gaining popularity, so Fulcrrum also works harder to index more address and UTXO.

[1] https://statoshi.info/d/000000009/unspent-transaction-output-set?orgId=1&from=now-5y&to=now&timezone=browser&refresh=10m

[nc50lc]

I asked a similar question a while ago ~

Your problem is the same as it was 2 years ago:

- 8 GB of RAM

I'm not sure if it's actually his ram.
Because in the graph that he provided, the average disk usage reduced again when he downgraded his client to the unaffected version and then spiked again after another upgrade.
If it's hardware, any of those versions should exhibit similar performance issues.

@OP how accurate are those client version labels in the graph?

Why would using a newer version of Bitcoin knots cause so many more disk writes? I'm running on an SSD and I don't want to kill it prematurely.

I'm not using Knots to help you accurately but those versions have BIP110 activated, perhaps it has something to do with its new consensus.

[LoyceV]

I'm not sure if it's actually his ram.
Because in the graph that he provided, the average disk usage reduced again when he downgraded his client to the unaffected version and then spiked again after another upgrade.

Even the lower value of 20 writes per day is high, with sufficient RAM I'd expect it to be a lot less.

I'm not using Knots to help you accurately but those versions have BIP110 activated, perhaps it has something to do with its new consensus.

Could it be they flush dbcache to disk a lot more frequently in the newer version?

[vxyz123456]

I asked a similar question a while ago ~

Your problem is the same as it was 2 years ago:

- 8 GB of RAM

I'm not sure if it's actually his ram.
Because in the graph that he provided, the average disk usage reduced again when he downgraded his client to the unaffected version and then spiked again after another upgrade.
If it's hardware, any of those versions should exhibit similar performance issues.

@OP how accurate are those client version labels in the graph?

Why would using a newer version of Bitcoin knots cause so many more disk writes? I'm running on an SSD and I don't want to kill it prematurely.

I'm not using Knots to help you accurately but those versions have BIP110 activated, perhaps it has something to do with its new consensus.

Those labels are perfectly accurate, down to the day. I know because I remember when I switched versions.

Your observation is correct, as soon as I downgrade back to 28.1, the writes are reduced significantly, back to their prior trend (20 GB/day).

Perhaps it is related to the new consensus rules, but I don't really understand why that would be related to this. I'd assume it would be some other change that is causing the indexer to work harder. But I don't know what.

Also, regarding this comment by LoyceV:

"Even the lower value of 20 writes per day is high, with sufficient RAM I'd expect it to be a lot less."

Keep in mind that this is total system disk writes, not only the node, but also the indexer (Fulcrum). The indexer is responsible for most of those. Would the indexer be writing less if the system had more RAM?

I can test this with 16 GB of RAM next weekend, but I doubt this is the problem, considering the task manager isn't showing high RAM utilization and I'm not aware of what the newer versions of Knots would be doing that would require so much more.

[nc50lc]

I can test this with 16 GB of RAM next weekend, but I doubt this is the problem, considering the task manager isn't showing high RAM utilization and I'm not aware of what the newer versions of Knots would be doing that would require so much more.

Even with the issue, this might help to reduce your disk usage since you can keep the whole or a large chunk of the UTXO-set in your memory.
Consider bumping your dbcache value after you upgrade your RAM.

IICRC, they also updated how their dbcache is set by default.
If it backported updates from Bitcoin Core, its default should have been increased to 1GB and I've also read that Knots auto-sets it depending on how much RAM the system has. (CMIIAM)
However, the first v29.x version that you've used which also exhibit the issue doesn't have those updates yet so it shouldn't be related to the new dbcache defaults.

[Cricktor]

@OP
Have you tested how current Bitcoin Core versions behave in your environment? Would be interesting if there's a similar change in daily disk writes.

I'm not a fan of Knots and the nonsense ^{in my opinion} it shills (YMMV!), so take my words as someone who wouldn't run Knots at all. If you have reasons to use Knots, not my business. Well, because I'm pretty much Knots agnostic, I've no idea what causes the change of disk write pressure you observe.

I don't want to endorse to change too many variables, but I have this question: why do you run outdated version of Fulcrum? In the v1.xx branch your version isn't the last, IIRC Fulcrum 1.12 was latest. Version 2.x of Fulcrum brought more database reliability where database corruption should be a thing of the past. Upgrading to the v2 branch requires either a database upgrade or starting from scratch by cleaning the datadir of Fulcrum (slow multiday rebuild from block 0).

At the moment I run Bitcoin Core v30.2 and Fulcrum v2.1.0 in a virtual machine with allocated 24GiB RAM (will reduce it to 20 or even 16GiB soon as the VM host would benefit from a bit more RAM back). I don't look at the daily disk write stats. Maybe I should? 

[vxyz123456]

@OP
Have you tested how current Bitcoin Core versions behave in your environment? Would be interesting if there's a similar change in daily disk writes.

Good idea to test this, I just switched to Core 31.0 tonight (still 8 GB RAM). I'll let it go for a few days and see if the writes are still high and report back here.

why do you run outdated version of Fulcrum

I don't have a good reason. I've just never bothered to update it. I'm assuming the database migration will take a while to complete, so I think it makes sense to wait to try this until I've explored other possibilities first.

At the moment I run Bitcoin Core v30.2 and Fulcrum v2.1.0 in a virtual machine with allocated 24GiB RAM (will reduce it to 20 or even 16GiB soon as the VM host would benefit from a bit more RAM back). I don't look at the daily disk write stats. Maybe I should?  

Yeah, I'd recommend monitoring this. I have a python script that runs nightly which uses smartmontools to pull the disk writes and append it to a csv file.

[LoyceV]

I don't look at the daily disk write stats. Maybe I should? 

Most users don't need to worry about this. Even at 230 GBW/day, that's still only 84 TBW/year, which means it takes much longer to wear out a modern SSD than it's economic lifespan. Simply put: you'd want a bigger one long before it reaches it's limit.
I just checked mine, and SMART shows this after writing 50 TB to the disk:

Code:

Percentage Used:                    15%

Wasn't the whole part of "SMART" that it gives you a warning when you need to start worrying?

I've had old disks with much lower TBW-numbers, I think I've seen 60 or 120 TBW limits, but those are far too small to run full nodes with Fulcrum anyway.

[Cricktor]

Yeah, I'm not worried at all, my VM host's NVMe SSD will likely last longer than this mad RAM and flash memory shortage crisis will last.

Here are my current S.M.A.R.T. stats of it, if anyone is interested.

Code:

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-111-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Lexar SSD NM790 4TB
Serial Number:                      <redacted>
Firmware Version:                   12237
PCI Vendor/Subsystem ID:            0x1d97
IEEE OUI Identifier:                0xcaf25b
Total NVM Capacity:                 4.096.805.658.624 [4,09 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       2.0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          4.096.805.658.624 [4,09 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            caf25b 03200016d9
Local Time is:                      Thu May 14 11:28:16 2026 CEST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     90 Celsius
Critical Comp. Temp. Threshold:     95 Celsius

...

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        39 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    2%
Data Units Read:                    239.437.560 [122 TB]
Data Units Written:                 134.516.667 [68,8 TB]
Host Read Commands:                 1.128.564.896
Host Write Commands:                1.264.606.967
Controller Busy Time:               3.801
Power Cycles:                       24
Power On Hours:                     14.086
Unsafe Shutdowns:                   7
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    16
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               39 Celsius
Temperature Sensor 2:               33 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

This SSD can very likely endure quite a bit more data written to it. I didn't see much of a need to monitor this before and am not convinced to need it in the future. I'm fine to look every few months or so at the data and be good with it.

The main action hogs on this machine are docker containers with Bitcoin Core and Fulcrum which run 24/7/365. There are a few other docker containers with local mempool.space and bitcoinexplorer.com instances that I occasionally use for privacy when I browse the blockchain.

[LoyceV]

Code:

Model Number:                       Lexar SSD NM790 4TB
Data Units Written:                 134.516.667 [68,8 TB]
Power On Hours:                     14.086

This SSD can very likely endure quite a bit more data written to it.

Endurance:    3000 TBW

68.8 TB in 14.086 hours is 0.00488 TB/hour. At this rate, you'll reach the limit in 68 years. Long before that, you'll need to upgrade because the blockchain doesn't fit anymore

[vxyz123456]

@OP
Have you tested how current Bitcoin Core versions behave in your environment? Would be interesting if there's a similar change in daily disk writes.

It looks like Core 31.0 has this problem too.

https://i.imgur.com/LMKeNcQ.png

So that eliminates it being Knots-specific or related to the new consensus rules.

Any ideas now? Has anything in Core changed recently that would cause this?

I will still try putting more RAM in it this weekend, just haven't had time to do that yet. I can also try updating Fulcrum, but I will try that last.

If there are any settings I could configure that might address this, I think that would also be worth trying first.

[LoyceV]

It looks like Core 31.0 has this problem too.

If I have to guess: they may have changed something to make it faster if you have enough RAM. Last time I did a full IBD with Bitcoin Core 26, it barely used any RAM despite setting a high dbcache value. I've seen performance improvements from upgrades before, and I assume this is the same. Unless you're low on RAM, then it's counter productive, apparently.

[Cricktor]

It looks like Core 31.0 has this problem too.

https://i.imgur.com/LMKeNcQ.png

So that eliminates it being Knots-specific or related to the new consensus rules.

Hm, interesting. I would then assume that a similar version of Core to a Knots version that has lower daily write pressure should perform likely the same.

I'm not sure if it makes sense to determine which exact version of Core has lower write pressure under otherwise constant conditions. I can't quite recall changes of the versions of Core that would point to significantly more write pressure (maybe limited to lower RAM environments as LoyceV hints at).

Don't change too many variables, so keeping your current Fulcrum version makes sense at first.

[vxyz123456]

OK, I just put in another 8GB DIMM so now the system has 16 GB total. Unfortunately the system only has two RAM slots and I don't have any 16 GB DIMMs, so if 16 isn't enough, I won't have a way to test more than this at the moment. Still running Core 31.0 for testing purposes.

From looking at task manager, I'm not seeing a meaningful difference in the amount of RAM used (still hovering around 2GB) but maybe I'll see a reduction in writes. I'll give it another day or two and we'll see.

[nc50lc]

From looking at task manager, I'm not seeing a meaningful difference in the amount of RAM used (still hovering around 2GB) but maybe I'll see a reduction in writes. I'll give it another day or two and we'll see.

Have you increased your node's database cache settings?
If not, it wont fully utilize those added RAM.

Try to set your dbcache to 8GiB or higher if your system doesn't have much background/other processes.
Then restart your node.

[ABCbits]

Your observation is correct, as soon as I downgrade back to 28.1, the writes are reduced significantly, back to their prior trend (20 GB/day).

Out of curiosity, is the sync speed noticeably slower (as in total downloaded and verified block) compared with Knots 29.3 or Core 31? You can use debug.log to know the overall sync speed.

I can also try updating Fulcrum, but I will try that last.

I skimmed release note from 1.12 to latest version, but it's mostly about performance improvement, preventing corruption and other general fix/improvement. There's no mention it would reduce disk write. But I still recommend you to update if you have spare time.

[vxyz123456]

From looking at task manager, I'm not seeing a meaningful difference in the amount of RAM used (still hovering around 2GB) but maybe I'll see a reduction in writes. I'll give it another day or two and we'll see.

Have you increased your node's database cache settings?
If not, it wont fully utilize those added RAM.

Try to set your dbcache to 8GiB or higher if your system doesn't have much background/other processes.
Then restart your node.

Thanks for the reminder. This morning I set

Code:

dbcache=12288

and restarted the machine. In all my prior tests so far with 8GB RAM (I was just switching node versions), this had been left as unconfigured. We'll see if that makes a difference.

I also started looking at the fulcrum logs to see if I could notice any difference between when I was running Knots 28.1 (low writes) and Knots 29.3 or Core 31.0, but unfortunately nothing is jumping out at me as meaningfully different.