Bitcoin Core - Full backup of everything

[apogio]

Hello,

I'm running a couple of full nodes, but I'll have to stick with one now.

I'm thinking about maintaining a full backup of the entire blockchain on an HDD, in case I ever need to set up a new node, so that I don't have to do the IBD again.

More precisely, I'm thinking about :
1. backing up the entire ~/.bitcoin/ directory, with everything that it includes (blocks, chainstate, wallets etc).
2. keeping the HDD fully encrypted.
3. make periodical update backups (I don't know how often).

Questions:
(a) Is it a good thought? If not, why isn't it a good thought.
(b) What's the ideal process?

Note: The wallets are also separately backed up on other fully encrypted drives. So this won't be my only backup.

[LoyceV]

I'm thinking about maintaining a full backup of the entire blockchain on an HDD, in case I ever need to set up a new node, so that I don't have to do the IBD again.

But why? Taken slightly out of context, it reminds me of this:

“Only wimps use tape backup. REAL men just upload their important stuff on ftp and let the rest of the world mirror it.”

(note: he wasn't talking about wallets)
Unless you're restricted on bandwidth or RAM, I wouldn't worry about backing up blockchain data as long as thousands of nodes have a backup online!

3. make periodical update backups (I don't know how often).
~
(b) What's the ideal process?

I would use this:

Code:

rsync - a fast, versatile, remote (and local) file-copying tool
       --link-dest=DIR          hardlink to files in DIR when unchanged

This way, you don't need to overwrite old backups, while keeping multiple "snapshots" of older backups barely takes more disk space than your latest backup.

[ABCbits]

(b) What's the ideal process?

If you want to be more ideal, i have these suggestion.
1. Write down details of the backup, such as latest version of Bitcoin Core used to create/load the ~/.bitcoin/ directory.
2. Compute and store checksum for each files. It should be much faster than asking Bitcoin Core to perform reindex if you also worry about file corruption/integrity.
3. If you wish to reduce storage usage on your HDD, compress entire ~/.bitcoin/ directory. I remember old thread that discuss it could reduce the size by about 20%, although i can't find the thread.

[LoyceV]

3. If you wish to reduce storage usage on your HDD, compress entire ~/.bitcoin/ directory. I remember old thread that discuss it could reduce the size by about 20%, although i can't find the thread.

I tested this with 7 blocks-files and pigz:
135 MB was reduced to 94 MB.

[apogio]

But why? Taken slightly out of context, it reminds me of this:

“Only wimps use tape backup. REAL men just upload their important stuff on ftp and let the rest of the world mirror it.”

(note: he wasn't talking about wallets)
Unless you're restricted on bandwidth or RAM, I wouldn't worry about backing up blockchain data as long as thousands of nodes have a backup online!

It's not only the bandwidth restrictions, but, isn't it also a pain when you need to wait 3 days to have a fully functional node up & running again?

2. Compute and store checksum for each files. It should be much faster than asking Bitcoin Core to perform reindex if you also worry about file corruption/integrity.

Can you elaborate on this please? Computing and storing checksums will certainly help with integrity, but how will they help in speed?

3. If you wish to reduce storage usage on your HDD, compress entire ~/.bitcoin/ directory. I remember old thread that discuss it could reduce the size by about 20%, although i can't find the thread.

Yeah, that's cool. I could also add an encryption password on the compressed file, instead of messing with encrypting the entire disk which takes time. So, one layer of encryption on the .tar file, another on the wallets themselves.

[LoyceV]

It's not only the bandwidth restrictions, but, isn't it also a pain when you need to wait 3 days to have a fully functional node up & running again?

I know I'm just 16 GB of RAM short from making this 2 days less Other than that, it's up to you to decide whether or not that wait is a pain. I'm not using Bitcoin Core for anything that couldn't wait for 3 days.

Yeah, that's cool. I could also add an encryption password on the compressed file, instead of messing with encrypting the entire disk which takes time. So, one layer of encryption on the .tar file, another on the wallets themselves.

Compressing files makes incremental backups a lot more difficult. I used pigz for my test, which compressed better than bzip, but there's probably compression software that offers better results at the expense of slower compression.

If you'd be running a pruned node, I'd say a backup (~15 GB) is worth saving a ~700 GB download.

All this is coming from someone who had to restart a new IBD a few times over the years after I did something that fucked up my data directory.... I still don't have a backup

[apogio]

I know I'm just 16 GB of RAM short from making this 2 days less Other than that, it's up to you to decide whether or not that wait is a pain. I'm not using Bitcoin Core for anything that couldn't wait for 3 days.

To be absolutely honest with you, I'm lucky enough not to have very urgent payments that I can't process otherwise, in a scenario where my Core nodes are totally destroyed, suddenly. I'm just lazy to wait for 3 days 

If you'd be running a pruned node, I'd say a backup (~15 GB) is worth saving a ~700 GB download.

I am running a pruned node, just for hot wallet purposes with small amounts.

All this is coming from someone who had to restart a new IBD a few times over the years after I did something that fucked up my data directory.... I still don't have a backup

Haha, yeah I feel you. That's what I'd like to avoid.

[takuma sato]

The main problem isn't even the big size of the blockchain, downloading or storing it (which is a problem), but the actual validation of increasingly complex blocks due spammers.
Check the charts:
https://statoshi.info/d/000000009/unspent-transaction-output-set?orgId=1&from=now-5y&to=now&timezone=browser&refresh=10m

If you go there you'll see that when the Ordinals protocol started and all this nonsense became popular (Ordinal Punks, Taproot Wizards, Bitcoin Rocks, etc.) followed throughout Jan–Feb 2023 and into 2023+ you'll notice it becomes increasingly difficult for the node to validate blocks.

"Size of Serialized UTXO Set" is the total disk size of the serialized Unspent Transaction Output set (UTXO set) and so when you are syncing the blockchain, specially if you are using a non-infected computer (a computer from 2008 or earlier, without an NSA embedded OS such as Intel ME) then it will become a nightmare to sync it as it will struggle to process those blocks.

Bitcoin Core v30 will only make this worse, which is why the logical conclusion is that Bitcoin Core v30 is a de-facto attack on the network and Knots 29 is the only thing we've got to corner spammers into irrelevance (and not to mention the whole CSAM-on-the-blockchain attack vector as well)

As far as the backup, I would backup everything, but for a sanity check, I like to generate fresh chainstate files when running a brand new backup. The problem once again is, it's becoming a nightmare to do so on any useful laptops you may have. You pretty much need a gaming computer these days to verify the full blockchain. This is a disaster when it comes to running Bitcoin with maximum security.

[LoyceV]

You pretty much need a gaming computer these days to verify the full blockchain.

That's not true. The thing that separates a "gaming computer" from a "normal" PC is a GPU, and Bitcoin Core doesn't use that.

If you're limiting yourself to 17 years old hardware, then indeed Bitcoin Core becomes more and more challenging to run.

[apogio]

You pretty much need a gaming computer these days to verify the full blockchain. This is a disaster when it comes to running Bitcoin with maximum security.

Although I get your point, I kinda disagree.

I think what makes a difference is:
1. the RAM. If you've got enough RAM you can fit in more data during initial blockchain download. This will ensure that you're going faster than someone else who is limited with half of your RAM.
2. the bandwidth. Obviously you need a good internet connection.
3. the type of storage medium. SSDs are faster than HDDs and NVMEs are faster than SSDs.

So, these 3 things exist on most modern (non-gaming) computers. At least I've got a decent laptop that fits most of these criteria. If I had an NVME, I believe I could decrease the IBD to 1/2 of the time.

[LoyceV]

If I had an NVME, I believe I could decrease the IBD to 1/2 of the time.

I think a system with 32 GB RAM and HDD could be faster than the same system with 8 GB RAM and NVMe

[philipma1957]

If I had an NVME, I believe I could decrease the IBD to 1/2 of the time.

I think a system with 32 GB RAM and HDD could be faster than the same system with 8 GB RAM and NVMe

yeah a wd gold 8tb hdd is fast enough with a good cpu and ram of

32gb or more.

https://www.amazon.com/gp/product/B0CVMZTBJ4/ref=ox_sc_act_title_1?smid=ATVPDKIKX0DER&th=1

I have done a lot of pcs running  nodes.

 Ram is a big deal

[apogio]

I think a system with 32 GB RAM and HDD could be faster than the same system with 8 GB RAM and NVMe

yeah a wd gold 8tb hdd is fast enough with a good cpu and ram of
...
 Ram is a big deal

I guess both of you assume that the dbcache will be adjusted accordingly, otherwise I suppose 8GB vs 32GB doesn't make a difference, if you only use 600 MB of it.

[LoyceV]

I guess both of you assume that the dbcache will be adjusted accordingly, otherwise I suppose 8GB vs 32GB doesn't make a difference, if you only use 600 MB of it.

I've never tested it, and I don't have spare systems with 32 GB, but I assume Linux file system cache can handle most of the work dbcache does. Other than that, I did indeed assume the correct settings to use the available RAM.

[ABCbits]

2. Compute and store checksum for each files. It should be much faster than asking Bitcoin Core to perform reindex if you also worry about file corruption/integrity.

Can you elaborate on this please? Computing and storing checksums will certainly help with integrity, but how will they help in speed?

I mean computing checksum/hash than using one of Bitcoin Core features (such as verifychain[1] or reindex).

[1] https://bitcoincore.org/en/doc/29.0.0/rpc/blockchain/verifychain/

Bitcoin Core v30 will only make this worse, which is why the logical conclusion is that Bitcoin Core v30 is a de-facto attack on the network and Knots 29 is the only thing we've got to corner spammers into irrelevance (and not to mention the whole CSAM-on-the-blockchain attack vector as well)

As reminder,
1. OP_RETURN doesn't enter UTXO set/chainstate.
2. It's possible to add malicious arbitrary data on Bitcoin blockchain with other approach and it already happened long time ago.

[Forsyth Jones]

I've always done this and it's a great idea, of course!

However, since I'm running out of storage space and don't have any backups of the Bitcoin folder/data, I need to resolve this soon.

Nowadays, if I need to redownload the database and blockchain again, I don't think I'll have many problems. However, with each passing year (every day, actually), the size increases, and it will take longer to synchronize a full node.

I just want a way to make these backups, and when restoring, for example, from one laptop to another, move the (updated) backup to the laptop with the outdated Bitcoin folder, without having to delete the blockchain/blocks/database, allowing the updated backup to complement the outdated laptop's database.

I don't know if that was clear, but that would be amazing.

[apogio]

I just want a way to make these backups, and when restoring, for example, from one laptop to another, move the (updated) backup to the laptop with the outdated Bitcoin folder, without having to delete the blockchain/blocks/database, allowing the updated backup to complement the outdated laptop's database.

If your node isn't running, can't you simply drag & drop the updated blockchain to the other node's database directory? Won't this simply replace the missing files?

[LoyceV]

However, with each passing year (every day, actually), the size increases, and it will take longer to synchronize a full node.

On the other hand: with every hardware upgrade, it's a lot faster again

I just want a way to make these backups, and when restoring, for example, from one laptop to another, move the (updated) backup to the laptop with the outdated Bitcoin folder, without having to delete the blockchain/blocks/database, allowing the updated backup to complement the outdated laptop's database.

For Linux, that's what rsync is for. But note that different Bitcoin Core downloads don't have the exact same blocks files, so if you're trying to "complement" one node with data from another note, it could still require copying almost everything.

[philipma1957]

Well lately I have been using a Mac Studio with a 2tb internal ssd. And 36gb ram with a 10000 speed eth cable. Although I use 1000 speed internet.

And I cloned that to an external thunderbolt 5 ssd 4tb nvme .

So I have a backup and the original.

But all of above = money.

No one needs this storage or speed for a full node.

[Forsyth Jones]

If your node isn't running, can't you simply drag & drop the updated blockchain to the other node's database directory? Won't this simply replace the missing files?

Honestly, I've never tried this for fear corrupting the data, so I copy the entire current blockchain to an external HD or SSD (whichever I have available), but first I delete the outdated backup because it's less than 2tb available.

For Linux, that's what rsync is for. But note that different Bitcoin Core downloads don't have the exact same blocks files, so if you're trying to "complement" one node with data from another note, it could still require copying almost everything.

So there's nothing I can do to optimize how backups are handled, right? I've already considered that possibility.