Pruned mode syncing doesn't cache file writes?

[jnano]

I have Bitcoin Core in prune mode.
I've launched it after one offline day, and syncing really grinds the HDD and barely progresses. It's doing something like 1 block per minute.

In the chainstate directory only 250MB of files were touched, so writes are easily cacheable.
Is there a way to get write caching?

[achow101]

Increase the size of the dbcache by starting Bitcoin Core with the -dbcache=<n> option or adding dbcache=<n> to your bitcoin.conf file where <n> is the amount of RAM in MB that you want to dedicate to the database cache. Increasing this will reduce the amount of disk IO that is done as the database will be flushed less frequently. A dbcache of ~6000 should allow the entire database cache to be held in memory so only on flush is required.

[jnano]

dbcache was set to 3000. I don't think I saw the process use anywhere close to that much memory, but I'll look more closely next time.

Is dbcache only for the UTXO set? Is it for reads or writes?

[achow101]

dbcache was set to 3000. I don't think I saw the process use anywhere close to that much memory, but I'll look more closely next time.

Is dbcache only for the UTXO set? Is it for reads or writes?

The dbcache is used for holding the UTXO set in memory. Once the dbcache is full, the UTXO set will be flushed to the disk. Having a higher dbcache means that it is flushed less so it will reduce disk IO from the UTXO set side of things. However during the syncing process, blocks are being written and deleted from disk so that is a major source of disk IO and slowdown.

[LoyceV]

Is there a way to get write caching?

In general, your Operating System should take care of this on a disk level.
I have good experiences running a pruned Bitcoin Core from a RAM drive. If you have enough RAM, you can do this, and copy it to HDD once it's done syncing.

On Linux, this works for me:

Code:

mkdir /dev/shm/prunedBitcoin                            # create new directory in /dev/shm, which by default uses up to 50% of available RAM
chmod 700 /dev/shm/prunedBitcoin                        # basic security on a multi-user-system
bitcoin -datadir=/dev/shm/prunedBitcoin -prune=550      # now we wait
mv /dev/shm/prunedBitcoin ~                             # move to your home-directory after you close Bitcoin Core

It's now downloading at the maximum speed my Wifi allows, Estimated time left: 9 hours (although I know from experience it will take a bit longer, as it's limited by my CPU later on).

[jnano]

Another check, this time with dbcache=6000. The cache isn't working. I'm using 15.0.1 x64 on Windows 8.

After 15 initial minutes, sync rate is less than 1 block/minute. Memory usage peaked at about 300 MB.
I/O is more reads than writes, and there's a constant opening and closing of chainstate files.

Watching file accesses for about a minute, there were a few 100s of I/Os to block data (one file and its rev file),
and more than 100K I/Os to chainstate files.

Startup:


After a few more minutes:

[LoyceV]

Another check, this time with dbcache=6000. The cache isn't working. I'm using 15.0.1 x64 on Windows 8.

After 15 initial minutes, sync rate is less than 1 block/minute. Memory usage peaked at about 300 MB.

What hardware specs do you have? It seems the site was under DDOS earlier, by the time I could edit my post, Bitcoin Core had downloaded 15% already. And that's just an old i3, but with 12 GB ram.

How far behind are you? 1 block/minute is only 10 times real time, that means it could take months to catch up.

After a few more minutes:

I see 159,271 Page Faults in just 3 minutes. That's 843 per second. I'm no expert on Windows, and I'm not sure what this means exactly, but this seems high to me. From Wiki:

Major page faults on conventional computers (which use hard disk drives for storage) can have a significant impact on performance. An average hard disk drive has an average rotational latency of 3 ms, a seek time of 5 ms, and a transfer time of 0.05 ms/page. Therefore, the total time for paging is near 8 ms (= 8,000 μs). If the memory access time is 0.2 μs, then the page fault would make the operation about 40,000 times slower.

Your sync speed looks a lot like my old Atom netbook with 1 GB ram (and SSD). Are you limited on RAM?

[jnano]

That particular sync was to catch up after 12 hours offline.

The only bottleneck here is the HDD, combined with the lack of caching, which leads to a lot of random small reads and writes.
RAM is not a problem. Bitcoin Core anyway never used much more than the 300MB you see in the screenshot.

Page faults are normal. I guess it's the disk accesses due to what the software does. The 3 minutes are CPU time, not total runtime.
A random reference point: launching Notepad entails about 2000 page faults in 50ms.

[LoyceV]

The only bottleneck here is the HDD, combined with the lack of caching, which leads to a lot of random small reads and writes.

I have the same setup on HDD (although I didn't prune it), and barely turn it off. I've turned it off now, tomorrow morning I'll turn it on again and tell you how long it takes to sync.

On my RAM drive experiment, I'm still doing several blocks per second. It's now less than 2 years behind, when blocks were not yet full.

RAM is not a problem. Bitcoin Core anyway never used much more than the 300MB you see in the screenshot.

Your OS uses RAM too, and the more you have, the more file cache it can use. Hence my question: how much RAM do you have? I noticed a huge overall improvement when I went from 4 to 12 on Linux.

Page faults are normal. I guess it's the disk accesses due to what the software does. The 3 minutes are CPU time, not total runtime.
A random reference point: launching Notepad entails about 2000 page faults in 50ms.

I see. It was worth a try

[jnano]

RAM's 8GB. BitcoinCore isn't restricting itself to 300MB because it tries to be polite, most memory is free.
I don't think it limits the cache size dynamically, does it? The explicit dbcache setting suggests it just uses as much as it wants, up to that limit.

I think a RAM drive will indeed solve it, but shouldn't the software just know how to use RAM directly?

[LoyceV]

RAM's 8GB. BitcoinCore isn't restricting itself to 300MB because it tries to be polite, most memory is free.

The OS uses the free memory for file caching.

I don't think it limits the cache size dynamically, does it? The explicit dbcache setting suggests it just uses as much as it wants, up to that limit.

I'm not sure how it works internally, click "Help > Debug window" to see it's Memory usage for the Memory Pool, I always assumed that's the amount of cache it uses (but I'm not entirely sure), and memory pool doesn't have so many unconfirmed transactions.

I think a RAM drive will indeed solve it, but shouldn't the software just know how to use RAM directly?

One would say so! I don't know why it's so slow for you, I tested all fine:

I have the same setup on HDD (although I didn't prune it), and barely turn it off. I've turned it off now, tomorrow morning I'll turn it on again and tell you how long it takes to sync.

It took 70 seconds to sync 11 hours of blocks. That's about 1 block per second

[jnano]

Thanks for checking. 70 seconds is much more reasonable (though still could be quicker). Were similar syncs quicker on your SSD install?

The memory pool is unconfirmed transactions not yet in blocks, not the total memory usage of the client.

Windows might use free memory for its own caching, but it's relinquished if software needs it.

[achow101]

RAM is not a problem. Bitcoin Core anyway never used much more than the 300MB you see in the screenshot.

Then you have probably set the parameter incorrectly. How did you set dbcache? Can you post the contents of your bitcoin.conf file? Can you post the contents of your debug.log file?

I'm not sure how it works internally, click "Help > Debug window" to see it's Memory usage for the Memory Pool, I always assumed that's the amount of cache it uses (but I'm not entirely sure), and memory pool doesn't have so many unconfirmed transactions.

The mempool and the dbcache are unrelated to each other. They have their own memory allocations and are configured with different options.


OP, what kind of HDD do you have? What is your CPU? CPU can also be a major bottleneck given the amount of computation that needs to be done to validate blocks.

[LoyceV]

Thanks for checking. 70 seconds is much more reasonable (though still could be quicker). Were similar syncs quicker on your SSD install?

I've never installed a full Bitcoin Core on my SSD (it's not big enough). Syncing also uses a lot of CPU, so I'm very happy with 1 block (1 MByte) per second.

I had turned off my pruned test on the RAM drive, testing it now (blocks from April 10, 2017): My 200 block test took just over 5 minutes, that's even slower than my HDD-test this morning.

It's an interesting pattern though: sometimes it stops using much CPU, I think it then downloads many blocks ahead at 5 Mbyte/s, and after this continues processing them. CPU load goes from 20 to 350 % (of virtual 4 cores). Then, after a while, it stops downloading, while still processing blocks. It seems inefficient not downloading and syncing continuously at the same time. I'm still using Bitcoin Core version v0.14.2.0, so this might be faster in the latest version.

I can't tell you what's the cause of your slow sync, it's a huge speed difference with my test. If you ever figure it out, please post your results here.

[achow101]

I've never installed a full Bitcoin Core on my SSD (it's not big enough). Syncing also uses a lot of CPU, so I'm very happy with 1 block (1 MByte) per second.

I had turned off my pruned test on the RAM drive, testing it now (blocks from April 10, 2017): My 200 block test took just over 5 minutes, that's even slower than my HDD-test this morning.

It's an interesting pattern though: sometimes it stops using much CPU, I think it then downloads many blocks ahead at 5 Mbyte/s, and after this continues processing them. CPU load goes from 20 to 350 % (of virtual 4 cores). Then, after a while, it stops downloading, while still processing blocks. It seems inefficient not downloading and syncing continuously at the same time. I'm still using Bitcoin Core version v0.14.2.0, so this might be faster in the latest version.

I can't tell you what's the cause of your slow sync, it's a huge speed difference with my test. If you ever figure it out, please post your results here.

When you have pruning enabled with the default settings (prune=550), Bitcoin Core will try to keep at most 550 MB of block and undo data on disk. This means that ~225 MB of blocks are actually stored. After that, it begins deleting blocks starting with the oldest.

When you are syncing, Bitcoin Core is downloading blocks as fast as possible, doing a basic check on them, and then writing them to disk. But validating the blocks and connecting them takes longer than the download and quick check, so the 225 MB of blocks allocated fills up quickly, but blocks can't then be deleted because the validation has not yet completed on the older blocks, so the download pauses so that the validation can catch up. Then it resumes and repeats this process. Because of this, syncing a pruned node may take longer.

When you don't have pruning enabled, Bitcoin Core can just keep downloading and checking blocks as fast as possible without any consideration as to where the validation is at.

[fabioganga]

When you have pruning enabled with the default settings (prune=550), Bitcoin Core will try to keep at most 550 MB of block and undo data on disk. This means that ~225 MB of blocks are actually stored. After that, it begins deleting blocks starting with the oldest.

When you are syncing, Bitcoin Core is downloading blocks as fast as possible, doing a basic check on them, and then writing them to disk. But validating the blocks and connecting them takes longer than the download and quick check, so the 225 MB of blocks allocated fills up quickly, but blocks can't then be deleted because the validation has not yet completed on the older blocks, so the download pauses so that the validation can catch up. Then it resumes and repeats this process. Because of this, syncing a pruned node may take longer.

When you don't have pruning enabled, Bitcoin Core can just keep downloading and checking blocks as fast as possible without any consideration as to where the validation is at.

Thanks for the clear explanation. I have raised my prune value to prune=2048. Would this speed things up a bit or not?

[jnano]

I had turned off my pruned test on the RAM drive, testing it now (blocks from April 10, 2017): My 200 block test took just over 5 minutes, that's even slower than my HDD-test this morning.

So, pruned on RAM drive is slower than the HDD test? Is the HDD test pruned as well?

But two differences between us: you're on v14.2.0 and Linux, I'm on 15.0.1 (x64) and Windows.

Then you have probably set the parameter incorrectly. How did you set dbcache? Can you post the contents of your bitcoin.conf file?

The dbcache setting is acknowledged in the settings window:


This is bitcoin.conf with comment lines removed:

Code:

prune=2000
minimizetotray=1
dbcache=6000

Can you post the contents of your debug.log file?

Lots of lines. Anything specific? The cache parts read:

Code:

2017-10-20 14:21:35 * Using 2.0MiB for block index database
2017-10-20 14:21:35 * Using 8.0MiB for chain state database
2017-10-20 14:21:35 * Using 5990.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)

Do the small figures refer to something other than caching? I thought the UTXO set is the same as chainstate db.

What is your CPU? CPU can also be a major bottleneck given the amount of computation that needs to be done to validate blocks.

CPU usage is single-digit on average, with peaks crossing 10% (the screenshot in the earlier post shows a typical pattern. Here's another startup.)

what kind of HDD do you have?

Internal 2.5" 5400rpm. Not quick, but all things considered, definitely not much different than any other HDDs when it comes to seeking.

The I/O patterns and lack of caching are the root cause. It's constantly reading at a few MB/sec, mixed with writes at a few 100s KB/s, with an occasional >1MB write.

The following ticket mentions changes in caching in v15. Maybe the issue is related?
https://github.com/bitcoin/bitcoin/issues/10647

Another ticket says prune mode flushes UTXO cache constantly in a v15 beta:
https://github.com/bitcoin/bitcoin/issues/11315

[jnano]

Temporary solution:

I tried what LoyceV suggested.

I copied the whole "chainstate" directory to a RAM drive, which I symlinked to the "chainstate" directory under Core's data dir.
That way, sync speed turned an order of magnitude faster. When syncing was done I copied the RAM drive contents back into the physical "chainstate" directory, and restarted Core.

[jnano]

I had some hope that v0.17's PR #11658 would help, but there's no appreciable difference.

Without using a RAM drive, and unless a proper caching system is implemented for the chainstate, it seems that decent performance requires an SSD. Maybe HDD is sufficient when using non-pruned, but I haven't tried that.

[jnano]

In my experience, HDD alone is sufficient for IBD as long as the HDD is used exclusively for IBD.

Perhaps you're not running pruned?

In your case, IBD most likely is very slow because  it's also used for Windows OS

It's purely due to Core's caching behavior, there's no other appreciable disk activity.

Stop using HDDs, problem solved and no dev-time gets wasted.

Yeah, people should adapt to software rather than the other way around. Great way to encourage full nodes, too.


Anyway, it's mainly a question of whether there's developer interest, and luckily that seems to exist.

In particular, a while back Sjors has been spending time benchmarking and trying things out. There was also a PR from esotericnonsense. What got merged into 0.17 was from luke-jr. MarcoFalke or jamesob wanted to regularly benchmark pruned behavior, and seem to care about HDD performance as evident on BitcoinPerf, which graphs IBD time on HDD versus SSD, though currently just non-pruned.

There are two current open tickets:
#11315 Prune undermines the dbcache
#12058 Slow catchup for recent blocks on non-SSD drive