how-to clean up chainstate folder

[citb0in]

In short: a pruned node that uses the minimum possible value "prune=550" shows a folder structure of

./blocks = 704M
./chainstate = 11G

This of course goes beyond the actual purpose of a pruned node. Why does the folder content of chainstate rise to this immense height? And how can this immensely high disk usage of the chainstate folder be cleaned up? Simply deleting it is certainly not a good option. What is the best way to proceed in such a case?

[nc50lc]

And how can this immensely high disk usage of the chainstate folder be cleaned up? Simply deleting it is certainly not a good option. What is the best way to proceed in such a case?

You can't clean it, it contains the "UTXO set" based from the blocks that you've verified.

If your node is running for a few years now, re-syncing Core or --reindex-chainstate may produce a more "optimized" chainstate
but it wont be a significant change in size, maybe a reduction of 1GB at best.

This of course goes beyond the actual purpose of a pruned node. Why does the folder content of chainstate rise to this immense height?

Your pruned node is a full node too, it needs the chainstate to verify all the transactions that it continuously receiving from its peers.
It's increasing (at a very slow rate) because of UTXOs that are being spent and created.

[NotATether]

This of course goes beyond the actual purpose of a pruned node. Why does the folder content of chainstate rise to this immense height?

Your pruned node is a full node too, it needs the chainstate to verify all the transactions that it continuously receiving from its peers.
It's increasing (at a very slow rate) because of UTXOs that are being spent and created.

Especially Ordinals. I think all this minting is contributing to make the UTXO set bigger at an even faster rate since selling and buying an Ordinal makes the same outputs as it does inputs (IIRC).

Pretty soon, I think we will need drop provably-unspendable outputs from the chainstate.

[achow101]

Pretty soon, I think we will need drop provably-unspendable outputs from the chainstate.

We already do.

[pooya87]

Especially Ordinals. I think all this minting is contributing to make the UTXO set bigger at an even faster rate since selling and buying an Ordinal makes the same outputs as it does inputs (IIRC).

Pretty soon, I think we will need drop provably-unspendable outputs from the chainstate.

Unfortunately the junk outputs Ordinals attackers create are not provably unspendable. A lot of them are just dust outputs that the standard rules prevent from being spent. Other broken scripts* (including those used for Ordinals Attack) don't contribute enough to the bloat to justify the effort of writing the code to "purge" them from the database.

The OP_RETURN outputs (the only provably unspendable outputs) have always been dropped as a.chow said.

* Example of a provably unspendable broken script that I'm talking about:

Code:

OP_DUP OP_HASH160 <push 21 bytes> OP_EQUALVERIFY OP_CHECKSIG

[ABCbits]

If your node is running for a few years now, re-syncing Core or --reindex-chainstate may produce a more "optimized" chainstate
but it wont be a significant change in size, maybe a reduction of 1GB at best.

Do you have personal experience or know documentation that --reindex-chainstate actually reduce chainstate size? After all, chainstate basically just LevelDB (key/value) database.

This of course goes beyond the actual purpose of a pruned node.

But it's technical limitation of pruned node, since you need all "spendable" UTXO to verify all new TX/block.

Why does the folder content of chainstate rise to this immense height?

1. Ordinals (mainly BRC-20 token).
2. Rising Bitcoin popularity.

What is the best way to proceed in such a case?

Allocate more storage. I don't expect Bitcoin Core will implement compressed chainstate feature.

[nc50lc]

Do you have personal experience or know documentation that --reindex-chainstate actually reduce chainstate size? After all, chainstate basically just LevelDB (key/value) database.

His chainstate folder is 11GB in size and based from his post history, he's been using his node for a few years now.
On the other hand, I have a recently fully synced node with 9.8GB chainstate folder.

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Here's a post in bitcoin stackexchange that explains how it work: https://bitcoin.stackexchange.com/a/117435

[ABCbits]

Do you have personal experience or know documentation that --reindex-chainstate actually reduce chainstate size? After all, chainstate basically just LevelDB (key/value) database.

His chainstate folder is 11GB in size and based from his post history, he's been using his node for a few years now.
On the other hand, I have a recently fully synced node with 9.8GB chainstate folder.

Good point. I just found out size of my chainstate folder has smaller size.

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Here's a post in bitcoin stackexchange that explains how it work: https://bitcoin.stackexchange.com/a/117435

I already know LevelDB has some optimization, but didn't know it doesn't fully optimize/reduce size of it's DB files. Although as reminder, --reindex-chainstate will redownload whole blockchain.

[citb0in]

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Although as reminder, --reindex-chainstate will redownload whole blockchain.

So will that result in a smaller or significant difference folder size after such a reindex ? Please provide some folder sizes before/after if possible. Thanks a bunch in advance. What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

citb0in

[BlackHatCoiner]

So will that result in a smaller or significant difference folder size after such a reindex ?

It will result in a small difference. If you have a storage problem, reindexing for the sake of restructuring your chainstate is definitely not a recommended option.

What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

Due to heuristics, two LevelDB databases may have the same content, but it is not guaranteed that they are byte-to-byte identical, with the exact same size. Nodes running for different durations exhibit different write patterns. For instance, a node running for years may have more frequent but smaller writes, whereas a node completing the IBD may have fewer but larger writes.

Maybe a Bitcoin Core developer can provide us a better insight.

[ABCbits]

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Although as reminder, --reindex-chainstate will redownload whole blockchain.

So will that result in a smaller or significant difference folder size after such a reindex ? Please provide some folder sizes before/after if possible. Thanks a bunch in advance. What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

citb0in

Sorry, but i don't want to run reindex-chainstate since i only use HDD. But i don't expect much difference, where you only gain about 1GB free space. You better upgrade your storage, clear all cache on your OS and delete unused files.

[nc50lc]

The command: --reindex-chainstate deletes the chainstate and creates a new chainstate from the blockchain.

Thanks a bunch in advance. What I still don't understand how two fully-synchronized non-pruned nodes can have different sizes of the chainstate. That makes no sense to me.

Go to the link in my previous reply to see the comment of a Bitcoin Developer in bitcoin.stackexchange site regarding the chainstate database size difference.

So will that result in a smaller or significant difference folder size after such a reindex ? Please provide some folder sizes before/after if possible.

I've already mentioned in the first reply that it'll not be a significant difference.