Are High Disk Writes Expected?

[vxyz123456]

I have been running Bitcoin Core 27.0 with txindex=1 and ElectrumX 1.16.0 on a Samsung 860 Evo SSD on Xubuntu. This is a dedicated machine doing nothing but running those two programs all day. I've also been using smartmontools to monitor the total disk writes once a day over time, and I've noticed the numbers are shockingly high. Obviously this is a concern since SSDs can only perform so many writes before they die.

Unfortunately, I only have a few data points right now, but all of these were recorded AFTER both the initial block download and after the transaction index was finished being built:

30 total gigabytes written to the SSD from June 24th to June 25th
108 total gigabytes written to the SSD from June 25th to June 26th
158 total gigabytes written to the SSD from June 26th to June 27th

To be clear, this is the difference in total GiB (512 bytes * Total LBAs Written /1024**3) written to the disk from day to day, not actual capacity being used. The amount of disk space being used is about what I would expect (increasing a few hundred megabytes per day).

I'm getting the LBAs written from smartmontools using:
sudo smartctl -a /dev/sda
and pulling the Total_LBAs_Written field, in case anybody else wants to try to monitor this for themselves and compare.

Obviously there's a lot more testing I need to do before I can determine what's causing this for sure but of course my intuition is that it's either related to Bitcoin Core or ElectrumX since the machine is dedicated to running them.

It is also possible I have the LBA size wrong for this drive and I'm inflating the gigabytes written without realizing it, but 512 bytes is standard from what I can tell.

Do these disk write numbers seem reasonable to you, or are these way too high?

[LoyceV]

I have been running Bitcoin Core 27.0 with txindex=1 and ElectrumX 1.16.0 on a Samsung 860 Evo SSD on Xubuntu. This is a dedicated machine doing nothing but running those two programs all day.

Can you share how much RAM you have, how much swap is in use, and Bitcoin Core's dbcache setting?

Unfortunately, I only have a few data points right now, but all of these were recorded AFTER both the initial block download and after the transaction index was finished being built:

30 total gigabytes written to the SSD from June 24th to June 25th
108 total gigabytes written to the SSD from June 25th to June 26th
158 total gigabytes written to the SSD from June 26th to June 27th

It looks like something changed in between those days. I'd expect Bitcoin Core to write approximately the same amount each day (at least in days so close together).

It is also possible I have the LBA size wrong for this drive and I'm inflating the gigabytes written without realizing it, but 512 bytes is standard from what I can tell.

Correct. I've used this script (use at your own risk) to check it in the past.

Here's my findings: on a standard pruned Bitcoin Core (so no txindex I guess, and no ElectrumX) with only 8 GB of RAM, my SSD wrote just under 5 TB for the entire IBD.
It looks like you're writing a lot more than that.

a Samsung 860 Evo SSD

The 500 GB version can handle 300 TBW. Even at 150 GB per day, you're looking at 5 years.

[vxyz123456]

I have been running Bitcoin Core 27.0 with txindex=1 and ElectrumX 1.16.0 on a Samsung 860 Evo SSD on Xubuntu. This is a dedicated machine doing nothing but running those two programs all day.

Can you share how much RAM you have, how much swap is in use, and Bitcoin Core's dbcache setting?

Unfortunately, I only have a few data points right now, but all of these were recorded AFTER both the initial block download and after the transaction index was finished being built:

30 total gigabytes written to the SSD from June 24th to June 25th
108 total gigabytes written to the SSD from June 25th to June 26th
158 total gigabytes written to the SSD from June 26th to June 27th

It looks like something changed in between those days. I'd expect Bitcoin Core to write approximately the same amount each day (at least in days so close together).

It is also possible I have the LBA size wrong for this drive and I'm inflating the gigabytes written without realizing it, but 512 bytes is standard from what I can tell.

Correct. I've used this script (use at your own risk) to check it in the past.

Here's my findings: on a standard pruned Bitcoin Core (so no txindex I guess, and no ElectrumX) with only 8 GB of RAM, my SSD wrote just under 5 TB for the entire IBD.
It looks like you're writing a lot more than that.

a Samsung 860 Evo SSD

The 500 GB version can handle 300 TBW. Even at 150 GB per day, you're looking at 5 years.

The system has 8GB of RAM on an Intel 8th gen Celeron CPU (2 cores, 2 threads). The node isn't pruned. Xubuntu's built-in task manager is showing Memory at 25% (1.9 GiB / 7.6 GiB) and Swap at 34% (1.4 GiB / 4 GiB) at this exact moment, having had both Bitcoin Core and ElectrumX running 24/7 for about 3 or 4 days. I guess I don't really understand why the Swap is being used significantly given how much memory appears to be available, but maybe this is normal. If not, let me know.

I don't have anything in the Bitcoin.conf file specifically changing the database cache. However, from what I can see in the GUI using bitcoin-qt, under Settings -> Options -> Size of database cache, I see it is set to 450 MiB.

Also, I checked smartmontools again and confirmed the sector size is 512 bytes, so my math should be correct.

[ABCbits]

Do these disk write numbers seem reasonable to you, or are these way too high?

At a glance, i find it's reasonable since you only have 8GB of RAM when current UTXO size is about 11.6GB according to https://statoshi.info/d/000000009/unspent-transaction-output-set and you need to share with both OS and ElectrumX. But for starter, you probably want to run iotop --accumulated and wait for few days to know which process perform most write.

Here's my findings: on a standard pruned Bitcoin Core (so no txindex I guess, and no ElectrumX) with only 8 GB of RAM, my SSD wrote just under 5 TB for the entire IBD.
It looks like you're writing a lot more than that.

In addition, according to Jameson Lopp benchmark, the write is much lower if you have much bigger RAM and configure Bitcoin Core to use lots of RAM. Although his benchmark doesn't enable txindex either.

Bitcoin Core 26.0

The full sync used:

    16.7 GB RAM
    60 MB disk reads
    561 GB disk writes

[LoyceV]

The system has 8GB of RAM on an Intel 8th gen Celeron CPU (2 cores, 2 threads).

Adding more RAM should largely reduce your disk writes, but that may not be possible considering it's a Celeron.

The node isn't pruned. Xubuntu's built-in task manager is showing Memory at 25% (1.9 GiB / 7.6 GiB) and Swap at 34% (1.4 GiB / 4 GiB) at this exact moment, having had both Bitcoin Core and ElectrumX running 24/7 for about 3 or 4 days. I guess I don't really understand why the Swap is being used significantly given how much memory appears to be available, but maybe this is normal. If not, let me know.

That's normal. The other memory is not empty, it's probably in use as file cache. Writing to your swap file is meant to improve overall system performance. You can adjust this behaviour by changing "swappiness", but be careful not to make it worse.

Size of database cache, I see it is set to 450 MiB.

That's the default, and usually a larger value is recommended. Ideally, it should be enough to fit chainstate, but since that's 12 GB, most people don't have that luxory. I've never tested how increasing dbcache affects performance. I assume Linux file cache is quite good at it by default.

In addition, according to Jameson Lopp benchmark, the write is much lower if you have much bigger RAM and configure Bitcoin Core to use lots of RAM.

    561 GB disk writes

That makes sense: with enough RAM, chainstate can stay in memory until you shut down Bitcoin Core. So all that's written is the full blockchain, which must have been 561 GB back then.

[vxyz123456]

The system has 8GB of RAM on an Intel 8th gen Celeron CPU (2 cores, 2 threads).

Adding more RAM should largely reduce your disk writes, but that may not be possible considering it's a Celeron.

The node isn't pruned. Xubuntu's built-in task manager is showing Memory at 25% (1.9 GiB / 7.6 GiB) and Swap at 34% (1.4 GiB / 4 GiB) at this exact moment, having had both Bitcoin Core and ElectrumX running 24/7 for about 3 or 4 days. I guess I don't really understand why the Swap is being used significantly given how much memory appears to be available, but maybe this is normal. If not, let me know.

That's normal. The other memory is not empty, it's probably in use as file cache. Writing to your swap file is meant to improve overall system performance. You can adjust this behaviour by changing "swappiness", but be careful not to make it worse.

Size of database cache, I see it is set to 450 MiB.

That's the default, and usually a larger value is recommended. Ideally, it should be enough to fit chainstate, but since that's 12 GB, most people don't have that luxory. I've never tested how increasing dbcache affects performance. I assume Linux file cache is quite good at it by default.

In addition, according to Jameson Lopp benchmark, the write is much lower if you have much bigger RAM and configure Bitcoin Core to use lots of RAM.

    561 GB disk writes

That makes sense: with enough RAM, chainstate can stay in memory until you shut down Bitcoin Core. So all that's written is the full blockchain, which must have been 561 GB back then.

I'm getting the impression that adding more RAM mostly reduces disk writes before the node is fully synced, not after. But I have extra RAM and I can bump it up to 16GB so I can give it a shot. The Celeron supports up to 64 GB of RAM according to Intel's website. I could also put in 32 GB later (I can swap it out from another old computer), but I'll start with 16 for now.

From what I can tell, I should set this line in bitcoin.conf (12288 = 12 * 1024):

Code:

dbcache=12288

Is that the only thing I need to change? Or are there other settings I should try too?

[LoyceV]

I'm getting the impression that adding more RAM mostly reduces disk writes before the node is fully synced, not after.

Each new block requires verification, which includes reading and writing to chainstate.

But I have extra RAM and I can bump it up to 16GB so I can give it a shot. The Celeron supports up to 64 GB of RAM

Usually the mainboard is the limiting factor.

Code:

dbcache=12288

With 16 GB RAM, that doesn't leave much for the rest of your OS. I'd go for no more than half your RAM, so try 8 GB dbcache if you install 16 GB RAM.

Is that the only thing I need to change? Or are there other settings I should try too?

That should be enough. For what it's worth: my system has 16 GB RAM (and the mainboard can't handle more), and during IDB I increase dbcache, but when normally running I reduce it to 1024 again. I don't want Bitcoin Core to occupy the majority of my RAM, I need it for other things.
I just checked: I've written 7.5 TB in the past year (I didn't do a new IBD on it). That's 21 GB written per day, for Bitcoin Core, swap and everything else running on my system. I keep chainstate on this SSD, and blocks on my HDD. At this rate, I'll reach my SSD's rated writes after 80 years.

[vxyz123456]

Do these disk write numbers seem reasonable to you, or are these way too high?

At a glance, i find it's reasonable since you only have 8GB of RAM when current UTXO size is about 11.6GB according to https://statoshi.info/d/000000009/unspent-transaction-output-set and you need to share with both OS and ElectrumX. But for starter, you probably want to run iotop --accumulated and wait for few days to know which process perform most write.

Here's my findings: on a standard pruned Bitcoin Core (so no txindex I guess, and no ElectrumX) with only 8 GB of RAM, my SSD wrote just under 5 TB for the entire IBD.
It looks like you're writing a lot more than that.

In addition, according to Jameson Lopp benchmark, the write is much lower if you have much bigger RAM and configure Bitcoin Core to use lots of RAM. Although his benchmark doesn't enable txindex either.

Bitcoin Core 26.0

The full sync used:

    16.7 GB RAM
    60 MB disk reads
    561 GB disk writes

Thanks for suggesting iotop. I've been running it for about 10 minutes and I already get the impression electrumx is doing all the disk writes, not even Bitcoin core. After just 10 minutes I'm seeing:

Disk Writes          Command
3.36 G                 python3 /home/user/Desktop/electrumx/electrumx_server
17.97 M               [jdb2/sda2-8]
6.52 M                 systemd-journald
3.84 M                 bitcoin-qt -min -chain=main [b-scheduler]
3.75 M                 bitcoin-qt -min -chain=main [b-msghand]
3.36 M                 tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

So it's almost certainly my elecrumx configuration I'll need to scrutinize, not Bitcoin core.

Edit: Here's what I'm seeing after one hour:

Disk Writes          Command
8.49 G                 python3 /home/user/Desktop/electrumx/electrumx_server
56.66 M               [jdb2/sda2-8]
23.80 M               systemd-journald
20.05 M               bitcoin-qt -min -chain=main [b-msghand]
16.27 M               bitcoin-qt -min -chain=main [b-scheduler]
13.89 M               tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

[alexeyneu]

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 are cheap now as  are sas2 hdd's, it's enterprise-grade things.
by hba i mean raid card flashed to IT mode (it'll be LSI one most probably). Because if it's not flashed already you'll need to do this https://www.reddit.com/r/DataHoarder/comments/lck2ou/comment/gm93zpp/      on your own.

[vxyz123456]

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 is cheap now, it's enterprise-grade things.

Funny you say that because the electrumx devs specifically recommended using an SSD. (As I pointed out above, it's electrumx that is doing all the disk writes, not Bitcoin Core.)

[alexeyneu]

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 is cheap now, it's enterprise-grade things.

Funny you say that because the electrumx devs specifically recommended using an SSD. (As I pointed out above, it's electrumx that is doing all the disk writes, not Bitcoin Core.)

and that's after he'd checked out factory specs. hopeless case

[LoyceV]

Edit: Here's what I'm seeing after one hour:

Disk Writes          Command
8.49 G                 python3 /home/user/Desktop/electrumx/electrumx_server

I've never used Electrum server, but I'm curious: what could it possibly be doing that requires so much disk writes?

say monero clearly stated that their node will ruin ssd.

This has nothing to do with Monero.

[ABCbits]

Edit: Here's what I'm seeing after one hour:

Disk Writes          Command
8.49 G                 python3 /home/user/Desktop/electrumx/electrumx_server
56.66 M               [jdb2/sda2-8]
23.80 M               systemd-journald
20.05 M               bitcoin-qt -min -chain=main [b-msghand]
16.27 M               bitcoin-qt -min -chain=main [b-scheduler]
13.89 M               tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

Assuming the output comes after you use 16GB RAM and set higher dbcache value, it's not surprising since Bitcoin Core would use disk less frequently. I saw you asked on GitHub whether changing CACHE_MB value may solve the problem. But description under CACHE_MB also mention ElectrumX write something called "unflushed history" frequently which doesn't seem could be changed.

say monero clearly stated that their node will ruin ssd. i'll rather go with sas hba and sas hdd. used sas1 is cheap now, it's enterprise-grade things.

I recall their node software also recommends using SSD if you store the data on HDD. Although i don't know if they still show that recommendation today.

[vxyz123456]

At the moment I cannot find any settings to change for electrumx that would lower the disk writes. As a temporary solution for now I'm thinking I might use an extra 2.5 inch HDD from one of my old laptops and dedicate it to just the electrumx db folder (this is where all the writes are happening) so I can keep it separate from the SSD the OS and Bitcoin Core are installed on. Obviously performance will be worse but I'm not sure it will be bad enough to really matter for me. Either that or I will use a much cheaper extra 256 GB SSD instead.

[LoyceV]

As a temporary solution for now I'm thinking I might use an extra 2.5 inch HDD from one of my old laptops and dedicate it to just the electrumx db folder (this is where all the writes are happening) so I can keep it separate from the SSD the OS and Bitcoin Core are installed on. Obviously performance will be worse but I'm not sure it will be bad enough to really matter for me. Either that or I will use a much cheaper extra 256 GB SSD instead.

I wouldn't move the writes to a HDD. That's terrible for performance. Note that cheap SSDs are usually lower quality, both in endurance as well as performance. I'd just let it be, keep backups of the data, and by the time your SSD is "worn" in a few years, replace it with a bigger one.

[ABCbits]

At the moment I cannot find any settings to change for electrumx that would lower the disk writes. As a temporary solution for now I'm thinking I might use an extra 2.5 inch HDD from one of my old laptops and dedicate it to just the electrumx db folder (this is where all the writes are happening) so I can keep it separate from the SSD the OS and Bitcoin Core are installed on. Obviously performance will be worse but I'm not sure it will be bad enough to really matter for me. Either that or I will use a much cheaper extra 256 GB SSD instead.

Using HDD seems to be poor choice. If i were you, i would try different Electrum server implementation such as Fulcrum or Electrs. Sparrow Wallet developer even write a benchmark about it on https://www.sparrowwallet.com/docs/server-performance.html.

[vxyz123456]

I'm giving Fulcrum a try right now. The database is currently building. It uses rocksdb while electrumx (by default) used leveldb, so maybe the write activity will be different. I will report back once it's all finished.

[philipma1957]

1tb ssd
32gb ram
I5 8500  or newer

using the Celeron and 8gb ram is simply a shit show.


I have run a ton of different  wallets in fact I will be doing a new one as I now have 1 gb  vs my old internet

[vxyz123456]

1tb ssd
32gb ram
I5 8500  or newer

using the Celeron and 8gb ram is simply a shit show.


I have run a ton of different  wallets in fact I will be doing a new one as I now have 1 gb  vs my old internet

I'm not sure about this. As I mentioned, not very much RAM was being used according to Xubuntu's built-in task manager, and when I put in 16 GB instead of 8 GB of RAM, I noticed absolutely no difference. I highly doubt the processor would impact the amount of disk writes.

Once again, it's not like the performance is bad; it's actually been entirely sufficient for me. I just can't figure out why ElectrumX is writing so much to disk. This very well could simply be how it was designed, and everybody, regardless of hardware, would experience this.

[LoyceV]

not very much RAM was being used according to Xubuntu's built-in task manager

Did you miss this part?

The other memory is not empty, it's probably in use as file cache.

and when I put in 16 GB instead of 8 GB of RAM, I noticed absolutely no difference.

No difference in memory usage, or no difference in disk writes? I guess it depends on what your Electrum server is doing: Bitcoin Core would benefit from larger dbcache, I don't know anything about Electrum server.

I highly doubt the processor would impact the amount of disk writes.

Agreed.