Absolute minimal node disk use configuration

[user32485724]

Hi,

I am working on a web app for bitcoiners. The app needs to make rpc calls to a node to get information from the node. I am only interested in the mempool, noting else.

To that end, I would like to run a minimal node configureation on the webserver. I played around with configurations locally an found that pruning to the minimum still leaves a 12 gig heavy .bitcoin folder, because chainstate is 11 gigs.

Can that be shrunk further? I am looking for the minimum hard drive footprint because my cloud droplet only comes with 25 gigs and 12 gigs just aren't available. Is there a setting in bitcoin.conf that get's rid of chainstate?

This is the config that gets me 12gigs today:

server=1
daemon=1
prune=550
maxmempool=100
rpcuser=redacted
rpcpassword=alsoredacted
rpcallowip=127.0.0.1
rpcport=8332


Alternatively, is there a public rpc server that allows calls every 5 minutes or so?

[1714520844]

1714520844

[1714520844]

1714520844

[1714520844]

1714520844

[LoyceV]

Is there a setting in bitcoin.conf that get's rid of chainstate?

I don't think that's possible. Without chainstate, Bitcoin Core can't know which transactions in your mempool can't exist.

The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

[Amphenomenon]

Here's what I use on my server:

Code:

~/bitcoin-26.0/bin/bitcoind -maxuploadtarget=500000 -dbcache=4096

Note that 4 GB dbcache is low for the IBD, but I don't want to give it 16 GB continuously. Normal file cache can handle it anyway. I simply extracted the compressed file in a home directory, and run it with user permissions. There's no need to edit config files, all options can be added to the starting command.
The 500 GB upload target is a daily limit, only set to prevent crazy things, but never came close to that amount.

I believe the maxuploadtarget is what you need for this

[LoyceMobile]

I believe the maxuploadtarget is what you need for this

Wrong. Bandwidth has nothing to do with disk usage.

[user32485724]

Is there a setting in bitcoin.conf that get's rid of chainstate?

I don't think that's possible. Without chainstate, Bitcoin Core can't know which transactions in your mempool can't exist.

The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

thanks. If you're right, it boils down to mounting an extra drive or finding a public bitcoin node that allows external RPC calls. I need one call every 5 minutes, that's 288 per day

[ABCbits]

Can that be shrunk further? I am looking for the minimum hard drive footprint because my cloud droplet only comes with 25 gigs and 12 gigs just aren't available. Is there a setting in bitcoin.conf that get's rid of chainstate?

Impossible, chainstate (which contain UTXO) is necessary to verify whether transaction violate Bitcoin protocol or not.

Alternatively, is there a public rpc server that allows calls every 5 minutes or so?

There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

[ranochigo]

Most blockchain explorers offer a public API for users to query but that'll mean that you would have to trust their information to be accurate. If that fits your purpose, then you can try to use those: BlockCypher, SoChain for example. Caveat is that since you're not able to run your own node and verify the data, there is no way of knowing if they are accurate.

[NotATether]

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.)

Alternatively, is there a public rpc server that allows calls every 5 minutes or so?

There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

Getblock is not a fake service, I have used it before.

[ABCbits]

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.)

It'd require OP to rent VPS which have at least 12GB though. My Bitcoin Core's chainstate currently has size about 10.8GB, you also need 300MB of RAM for mempool and some more for OS.

There are few third party services which let you perform RPC operation which supported by full node software they run. I'm only aware of getblock.io which offer such thing, but take note i never use that service and i don't know whether it's legit or fake service. So i strongly recommend you to do your own research.

Getblock is not a fake service, I have used it before.

That's good to know, i said that since i recall they used to perform somewhat aggressive advertising here (where some of them deleted by moderator).

[LoyceV]

It'd require OP to rent VPS which have at least 12GB though. My Bitcoin Core's chainstate currently has size about 10.8GB, you also need 300MB of RAM for mempool and some more for OS.

And you can add 1 or 2 GB RAM per year to keep up with the growth

[ranochigo]

I *think* that if you set the dbcache to a high enough value then it's just going to keep the entire chainstate in RAM and then only flush it to disk on program exit (which I know is not very helpful but maybe you can delete the folder when the program is finished. Caveat: I'm don't know if this will make Bitcoin Core complain.)

Storage is far cheaper than RAM, and note that since your entire chainstate is in RAM, any corruption to the RAM would result in a catastrophic failure and thus another reindex. I doubt it wouldn't try to reserve the space for chainstate on the disk at all times. So it'll be far easier to just get more storage.

[LoyceV]

Storage is far cheaper than RAM

That's true, but if you're low on RAM, more storage isn't going to help you.

note that since your entire chainstate is in RAM, any corruption to the RAM would result in a catastrophic failure and thus another reindex.

Any failure in your RAM pretty much fucks up your computer. Luckily, it's rare for RAM to fail after it's properly installed.

I doubt it wouldn't try to reserve the space for chainstate on the disk at all times. So it'll be far easier to just get more storage.

Each time Bitcoin Core verifies a block, it reads/writes a lot to chainstate. If you're low on RAM, the disk will be the bottleneck.

[ranochigo]

Any failure in your RAM pretty much fucks up your computer. Luckily, it's rare for RAM to fail after it's properly installed.

Yeah, that's why mission critical configurations have ECC and not for normal consumers. Even a slight corruption in the chainstate would be bad.

Each time Bitcoin Core verifies a block, it reads/writes a lot to chainstate. If you're low on RAM, the disk will be the bottleneck.

Bitcoin Core only stores as much chainstate on the ram as you allow it. Keeping the entire chainstate on the RAM, permanently and not as a cache, would probably cause more problems than it solves.

Regardless, the behavior for the chainstate flushing during IBD of a pruned node should be that the flushing occurs everytime that the blocks are removed from disk. Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

[LoyceV]

Regardless, the behavior for the chainstate flushing during IBD of a pruned node should be that the flushing occurs everytime that the blocks are removed from disk.

I've never tested it. It would be interesting to try: compare disk writes during IBD on a system with little RAM, to a system with 32 GB dbcache. I don't have the RAM to test this though.

Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

I've synced a pruned node in /dev/shm in the past. Works fine

[ranochigo]

Even if RAM is extremely reliable and that the server is never shutdown, Core would still attempt to flush chainstate to disk as the IBD progresses. Hence, it is necessary for the disk to have sufficient space to accomodate the chainstate, no way around that.

I've synced a pruned node in /dev/shm in the past. Works fine

That's including swap I suppose. If you're trying to keep your chainstate in your RAM, then that's the expected behavior (flushing) which is quite different from using your ram as a tmpfs or ramdisk. If you're using /dev/shm as a ram disk, then you can only use half of the RAM, and you'll have to consider the different overheads as well. Likely would require over 32GB of ram, which would probably mean that it's going to be a couple of times more than just adding half a terabyte to your server.

[DaveF]

Did you TEST your app with testnet 1st? Why play with getting it working on the live chain and dealing with these issues before you know it is going to work.
If you have issues now you will not know if it's a programming / app problem or a configuration issues because you tried to squeeze it into something too small.

Also, you really should be using a hosted solution with a lot more storage & power. If it does work and you release it if you are that limited on storage and RAM it's going to get clobbered once it's in the real world.

-Dave

[BlackHatCoiner]

Is there a setting in bitcoin.conf that get's rid of chainstate?

There's a reason why you can't prune the chainstate. It's this one thing you absolutely need to verify that the transactions do not go against the protocol. The most you could do to save up space is enable prune=550 (doesn't go less than that).

It is theoretically possible to run a full node from home (where you probably have sufficient storage), and configure your private server to communicate with your home node. But, that's just too much effort. Just buy more storage on your private server.