Pruned mode syncing doesn't cache file writes?

[Lauda]

Stop using HDDs, problem solved and no dev-time gets wasted.

Yeah, people should adapt to software rather than the other way around. Great way to encourage full nodes, too.

Yes, that's exactly the best way. You think you are entitled to demand Core to implement something for you, well you are not. There is more important work that needs to be doing. What makes matters worse here, you complain about the suggestion yet you run in pruned mode. You need maybe $10-worth-of-SSD to run it in the pruned mode with your settings.

[1713874205]

1713874205

[1713874205]

1713874205

[1713874205]

1713874205

[1713874205]

1713874205

[1713874205]

1713874205

[jnano]

You think you are entitled to demand

I'm not demanding anything. I'm describing a problem that I (and others) encounter, the same as any other report from the userbase. What's done with that is up to whoever is capable and willing.

Even if your suggestion works for you, it doesn't work for everyone. Besides myself, I also find it an issue when I suggest to people, especially layman, how they should use Bitcoin. I do mention Core for some of its benefits, but on the other hand it's a burden to normal users. 200GB+ of storage, or pruned mode that brings your computer down if you're on an HDD.

Currently not, but in past i use pruned mode where i didn't experience such problem. Take note i use 3.5" 7200 RPM HDD on Linux OS. Meanwhile you're using 2.5" 5400 RPM which is slower mine

Perhaps something changed in how pruned mode or caching is handled since v0.15? Or maybe some difference in settings (I don't remember the details, but I think Sjors saw cases where a larger dbcache degrades performance). Anyway, the specific HDD might make a small difference but it doesn't change the big picture. The problem is documented well enough and was also encountered by core devs.

You need maybe $10-worth-of-SSD

Regardless of this issue, if you could point me in the right direction I'd love to get a reliable 1TB SSD for $10 to take the place of the 1TB 2.5" HDD.

[Lauda]

You think you are entitled to demand

I'm not demanding anything. I'm describing a problem that I (and others) encounter, the same as any other report from the userbase. What's done with that is up to whoever is capable and willing.

Even if your suggested solution works for you, it doesn't work for everyone. Besides myself, I also find it an issue when I suggest to people, especially layman, how they should use Bitcoin. I do mention Core for some of its benefits, but on the other hand it's a burden for normal users. 200GB+ of storage, or pruned mode that brings your computer down if you're on HDD.

Alright, no demanding. You need to keep in mind that pruning is not a priority like full nodes are.

You need maybe $10-worth-of-SSD

Regardless of this issue, if you could point me in the right direction I'd love to get a reliable 1TB SSD for $10 to take the place of the 1TB 2.5" HDD.

You are using pruned mode, you said that yourself. Therefore: You don't need 1 TB, you need <10-20 GB. FYI you can get a 1 TB SSD for ~$130, and I'm talking about the best-reliable SSDs (e.g. Samsung Evo). It's actually a great time to pick up one!

[jnano]

Reading some discussions on Github, it seems the importance of pruned mode (and other space or time optimization methods like assumeutxo) increases as the size of the blockchain increases. That, assuming there's the goal of not driving more users to SPV or centralized bitcoin web wallets. Bitcoin Core is really not user-friendly when you add it all up, which is a shame.

About that SSD, I'd need the capacity to at least match that of the HDD it would replace in the laptop.

[Lauda]

Reading some discussions on Github, it seems the importance of pruned mode (and other space or time optimization methods like assumeutxo) increases as the size of the blockchain increases. That, assuming there's the goal of not driving more users to SPV or centralized bitcoin web wallets. Bitcoin Core is really not user-friendly when you add it all up, which is a shame.

Nah, pruning is a done deal. The other things that are being worked on are not related to improving pruning (but they might, down the road).

About that SSD, I'd need the capacity to at least match that of the HDD it would replace in the laptop.

The problem is your laptop HDD, i.e. the 5400 RPM that ETFbitcoin mentioned.

[jnano]

It could be noticeable, but it would be far smaller than the 10 times difference you get with proper caching (simulated using a RAM drive).

I haven't done real reaserch, but my brief search shows that 2.5" 5400 RPM drives might actually be pretty similar to 3.5" 7200 RPM drives as far as random 4K I/O is concerned, depending on the specific models compared. See the following in StorageReview (a rather reputable storage-centric site): 2.5" HDD review and 3.5" HDD review.

Never mind the specific models, the reviews also include comparative data for other drives. The IOMeter 4K Random I/O tables show 2.5" drives, probably 5400 RPM, getting 50-160 IOPS. The 3.5" 7200 drives are getting 50-180 IOPS. The top 7200 drives (WD RE4 and Caviar Black) are 80-180, so faster, but only by about 15%. The only drives showing real difference are the 10K RPM Velociraptors, with 140-370 IOPS, but these drives are far from typical.

It could be that 2.5" drives gain some advantage over 3.5" drives due to the shorter distance the heads need to travel (equivalent to short-stroking), then lose some due to mobile-use optimizations. Also the RPM difference could contribute.

SSDs are of course much faster (though even there proper caching may have some benefits).

[jnano]

I bet some tweaks could improve HDD performance, but I doubt any would be enough to turn it into a non-issue.

Still hoping for a caching solution one day. In the meantime, I think the best solution is for me to script the RAM drive procedure.

[LoyceV]

~, but on the other hand it's a burden to normal users. 200GB+ of storage, or pruned mode that brings your computer down if you're on an HDD.

I can't believe this is still an issue after almost 2 years!

It could be noticeable, but it would be far smaller than the 10 times difference you get with proper caching (simulated using a RAM drive).

I still think this might be an Operating System problem. Are you still only using Windows for this?

Reading some discussions on Github, it seems the importance of pruned mode (and other space or time optimization methods like assumeutxo) increases as the size of the blockchain increases.

I can confirm this for my case: I pruned my blockchain to 99 GB (the maximum amount allowed) because I was running low on disk space. I don't want the hassle of getting a bigger disk until I replace my laptop entirely.

I'm going to run some tests myself It will be a very rough estimate, as my PC is currently processing Merit data and will be processing Trust data tomorrow.

This is what I'm going to do:
1. Test syncing a pruned blockchain on HDD (starting now).
2. Test on RAM drive (starting when the previous is done):

On Linux, this works for me:

Code:

mkdir /dev/shm/prunedBitcoin                            # create new directory in /dev/shm, which by default uses up to 50% of available RAM
chmod 700 /dev/shm/prunedBitcoin                        # basic security on a multi-user-system
bitcoin -datadir=/dev/shm/prunedBitcoin -prune=550      # now we wait
mv /dev/shm/prunedBitcoin ~                             # move to your home-directory after you close Bitcoin Core

I'm using database cache 300 MB (my usual).

Update: The pruned download to hdd is now 48% done. Most of the day it downloaded several MB per second, with around 6% progress increase per hour. Then, I started a torrent download (Knoppix DVD) on the same hdd, which completely took all my bandwidth, and dropped the progress increase to 0.06% per hour.
When the torrent was done, Bitcoin Core didn't speed up. I restarted it, and after a while was still barely using any CPU power, but eventually the download speed went up, later CPU usage went up, and it's now processing several blocks per second again (Last block time Feb 2017).
I'm not using the latest version of Bitcoin Core yet, but this seems to be faster than I remember from when I tested the same on a RAM drive. I think my Wifi got faster.

If it keeps going like this, the pruned download to hdd will be finished tomorrow morning. In that case I won't bother to compare doing the same to a RAM drive.
I can't reproduce OP's problem with very slow syncing of a pruned blockchain on hdd so far.

[LoyceV]

Bump with an update: I've turned off Bitcoin Core for many hours today (to speed up processing Trust data), then turned it on again. It has 7500 blocks and 2 hours to go, progressing just under 1% per hour.
Could it be SegWit blocks take longer to check?

If I wouldn't touch my PC, I think it'll complete a pruned download to hdd in approximately 24 hours. At well over 200 GB, I'm not disappointed!
I tested this via Wifi, on a system that's over 4 years old and was nowhere near state of the art when I bought it.

To conclude:

I've launched it after one offline day, and syncing really grinds the HDD and barely progresses. It's doing something like 1 block per minute.

The bottleneck is probably somewhere else in your system, and not with Bitcoin Core.

[Lauda]

Could it be SegWit blocks take longer to check?

Could it be that bigger blocks take longer to check?[1] I don't know. Use your brain? Luke would be very angry with you.
[1] FTFY.

[jnano]

Thanks for the testing LoyceV. When it's all done, it'd be interesting to know how long it all took.

This got me curious, so I'm going to compare a fresh IBD to RAM-drive versus HDD. It's going to a take a while, as I won't be able to dedicate a computer 24/7. Just for good measure, for the HDD tests I'll defrag before each separate launching of Core.

But again, regardless of how much you can improve it on HDD, there is an inherent inefficiency in how the caching works.
I'm not sure if anyone checked the GitHub issue I linked earlier , so I'll quote Sjors directly, who seems like a pretty good authority on Bitcoin Coin behavior.

It often takes more than half an hour to catch up on less than a day of blocks.

[LoyceV]

LoyceV, for reference, could you tell us about these things when you sync :
1. CPU usage? Was it near 100%?
2. Was Bitcoin Core use most of your internet bandwidth?

1. Most of the time it took all available CPU power.
2. On and off: it started at 3-5 MB/s continuously, but close to the end it took 8 MB/s for a couple of seconds, then stopped downloading to process the data.

[jnano]

I decided to benchmark it more carefully, on a 2.5" 5400 rpm laptop HDD, on Windows.

I did two full IBDs, one with the chainstate in a RAM drive, the other only on HDD. Block data was on the HDD in both cases. Rather than a straight run it was a few hours each time, then quitting Core. For the HDD run I defragged before every run. For the RAM run I may have as well, but not sure if every time. So as far as fragmentation goes I think it's better than typical real-life scenarios.

Occasionally I used the computer for other things at the same time, but light usage.

I partially watched CPU usage. It wasn't generally a bottleneck. For most part it looked like Core limits itself to about 50% CPU, and only in more recent blocks it reached 80%+ and then maybe hit some CPU limits. Network speed was not a bottleneck.

On pure-HDD, besides the slower IBD times, it also seriously harms the usability of the computer. In the RAM drive case it's rarely if ever a major issue.

The most interesting result is how the total RAM and HDD runtimes diverge:


It seems the HDD run is greatly affected by maximum block size, or perhaps SegWit I/O patterns.

Earlier blocks were mostly indifferent to HDD. It was even slightly faster part of the time. My guess is fragmentation differences (I may have been less careful in the RAM run). Either way, it's insignificant.


A graph of how much slower the HDD run was:


Average time/block:

[Lauda]

or perhaps SegWit I/O patterns.

This is not the case.

[jnano]

Might help eventually:
https://github.com/bitcoin/bitcoin/pull/17487

[jnano]

By the way, another problem with these uncached writes is that it wastes SSD write cycles.

I'm seeing about 2.4 GB chainstate data written per each 1 day of blockchain (blocks from recent months). That's chainstate data only, not blocks.

Can anyone confirm?

[NotATether]

By the way, another problem with these uncached writes is that it wastes SSD write cycles.

I'm seeing about 2.4 GB chainstate data written per each 1 day of blockchain (blocks from recent months). That's chainstate data only, not blocks.

Can anyone confirm?

Interesting. My chainstate folder is capped at 4 gigabytes, but my node has been running for a couple days now. I don't have a way to monitor write amounts from only the chainstate folder, but I made a script that monitors write amounts from the whole process if you're interested in it. How did you capture the amount of chainstate data written?

[jnano]

I symlinked only the chainstate directory to the SSD, then watched the drive's SMART stats.
(Maybe there's also a configuration switch to specify the chainstate directory separately from the whole data dir?)

That 2.4GB/day I'm seeing is "client writes" from the PC. I haven't checked yet SSD write amplification. And would be interesting to compare to a non-pruned node.

Assuming 2.4GB/day, that's approaching 1TB/year without block and index data, and ignoring WA. If you're not testing multiple full IBDs maybe it's acceptable, but it's not great, especially if you use the SSD for more than just Bitcoin Core. And it isn't inherently "needed", it's just a side effect of the write caching problem.

most SSD have hundred TB write endurance

Some 120GB SSDs, like the Crucial BX500 or Kingston A400, have only 40 TBW. Twice that much in the 240GB models. A nice mid-ranger like the MX500 500GB has 150 TBW.

My chainstate folder is capped at 4 gigabytes

4GB is more or less its current storage size, but Core does a whole lot of small fragmented writes there while running. At least in pruned mode.

[ABCbits]

Here's the result with command sudo iotop -a with txindex enabled for about 80 minutes.

Code:

    TID  PRIO  USER     DISK READ DISK WRITE   SWAPIN      IO    COMMAND                 
  14736 be/4 user      0.00 B     20.50 M  0.00 %  0.00 % bitcoin-qt [b-scheduler]
  15497 be/4 user   1904.77 M      8.77 M  0.00 %  4.67 % bitcoin-qt [b-msghand]                                       

Unfortunately iotop separate based PID and remove total read/write if the PID is closed, so i doubt my result is useful at all.

SSD have hundred TB write endurance

Some 120GB SSDs, like the Crucial BX500 or Kingston A400, have only 40 TBW. Twice that much in the 240GB models. A nice mid-ranger like the MX500 500GB has 150 TBW.

I forget entry-level and small capacity SSD have lower write endurance.

[jnano]

I forget entry-level and small capacity SSD have lower write endurance.

Some QLC drives are even worse. Corsair MP400 has 200 TB TBW for the 1TB model. I'm not aware of small QLC drives currently, but that might change.