Allocating chainstate files into an SSD

[cellard]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

There's also the problem of space obviously, since SSD's are way more expensive.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I was wondering if anyone knows how to do this in Linux. (im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files, so could save a lot of money by not having to store block files inside SSD drives).

[jackg]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

There's also the problem of space obviously, since SSD's are way more expensive.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I was wondering if anyone knows how to do this in Linux. (im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files, so could save a lot of money by not having to store block files inside SSD drives).

You can't easily do this and anyway you did it could potentially cause lag and more database corruption.
It is worthwhile getting a good size ssd if you want to go that way to store everything on it (copying your wallet.dat to other places to try to save it should the ssd need formatting).

[cellard]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

There's also the problem of space obviously, since SSD's are way more expensive.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I was wondering if anyone knows how to do this in Linux. (im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files, so could save a lot of money by not having to store block files inside SSD drives).

You can't easily do this and anyway you did it could potentially cause lag and more database corruption.
It is worthwhile getting a good size ssd if you want to go that way to store everything on it (copying your wallet.dat to other places to try to save it should the ssd need formatting).

Why would it add lag?

As far as I know, chainstate files are the files that get accessed the most by Bitcoin Core. blocks folder files aren't accessed that often. So if this is true then you would be making it a lot faster if the files that get accessed the most are sitting on an SSD.

Im not sure if it's worth investing on a bigger SSD, when im just going to be using it so sync the blockchain. Sure I could re-use it a couple years from now to store data in the future but I dont want to invest unnecessary money.

[jackg]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

There's also the problem of space obviously, since SSD's are way more expensive.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I was wondering if anyone knows how to do this in Linux. (im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files, so could save a lot of money by not having to store block files inside SSD drives).

You can't easily do this and anyway you did it could potentially cause lag and more database corruption.
It is worthwhile getting a good size ssd if you want to go that way to store everything on it (copying your wallet.dat to other places to try to save it should the ssd need formatting).

Why would it add lag?

As far as I know, chainstate files are the files that get accessed the most by Bitcoin Core. blocks folder files aren't accessed that often. So if this is true then you would be making it a lot faster if the files that get accessed the most are sitting on an SSD.

Im not sure if it's worth investing on a bigger SSD, when im just going to be using it so sync the blockchain. Sure I could re-use it a couple years from now to store data in the future but I dont want to invest unnecessary money.

I mean the blocks and chainstate directory are designed to be in the same parent data directory and unless you can do something fancy with the code or the partition architecture of how everything runs, you could get it working normally. However if not, then it'd either require files to trick Bitcoin Core into putting the chainstate in a different place.

Thus, you'll have to stick to one drive IMO. Either an ssd or HDD for both directories. A hard drive is still quite fast though.  Although that's external for you so it will be slower than an internal one.

[cellard]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

There's also the problem of space obviously, since SSD's are way more expensive.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I was wondering if anyone knows how to do this in Linux. (im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files, so could save a lot of money by not having to store block files inside SSD drives).

You can't easily do this and anyway you did it could potentially cause lag and more database corruption.
It is worthwhile getting a good size ssd if you want to go that way to store everything on it (copying your wallet.dat to other places to try to save it should the ssd need formatting).

Why would it add lag?

As far as I know, chainstate files are the files that get accessed the most by Bitcoin Core. blocks folder files aren't accessed that often. So if this is true then you would be making it a lot faster if the files that get accessed the most are sitting on an SSD.

Im not sure if it's worth investing on a bigger SSD, when im just going to be using it so sync the blockchain. Sure I could re-use it a couple years from now to store data in the future but I dont want to invest unnecessary money.

I mean the blocks and chainstate directory are designed to be in the same parent data directory and unless you can do something fancy with the code or the partition architecture of how everything runs, you could get it working normally. However if not, then it'd either require files to trick Bitcoin Core into putting the chainstate in a different place.

Thus, you'll have to stick to one drive IMO. Either an ssd or HDD for both directories. A hard drive is still quite fast though.  Although that's external for you so it will be slower than an internal one.

Well that is unfortunate if true. I think I read achow saying that (that it would be faster if you put the chainstate files on an SSD).

Why would an external HDD be slower? If it runs at 7200 rpm like most modern HDD, it should be as fast? or maybe it's due the fact that it's not SATA but USB?

I just was looking forward buying one of these Libreboot t400 laptops, I saw some people talking about them as a safe way to run a node away from spyware (see them at minifree dot org)

What do you think? I would pick the 8 GB 1TB HDD configuration. Assuming the blocksize stays at 1MB it would last me for a long time until I need to pick a bigger HDD.

[jackg]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

There's also the problem of space obviously, since SSD's are way more expensive.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I was wondering if anyone knows how to do this in Linux. (im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files, so could save a lot of money by not having to store block files inside SSD drives).

You can't easily do this and anyway you did it could potentially cause lag and more database corruption.
It is worthwhile getting a good size ssd if you want to go that way to store everything on it (copying your wallet.dat to other places to try to save it should the ssd need formatting).

Why would it add lag?

As far as I know, chainstate files are the files that get accessed the most by Bitcoin Core. blocks folder files aren't accessed that often. So if this is true then you would be making it a lot faster if the files that get accessed the most are sitting on an SSD.

Im not sure if it's worth investing on a bigger SSD, when im just going to be using it so sync the blockchain. Sure I could re-use it a couple years from now to store data in the future but I dont want to invest unnecessary money.

I mean the blocks and chainstate directory are designed to be in the same parent data directory and unless you can do something fancy with the code or the partition architecture of how everything runs, you could get it working normally. However if not, then it'd either require files to trick Bitcoin Core into putting the chainstate in a different place.

Thus, you'll have to stick to one drive IMO. Either an ssd or HDD for both directories. A hard drive is still quite fast though.  Although that's external for you so it will be slower than an internal one.

Well that is unfortunate if true. I think I read achow saying that (that it would be faster if you put the chainstate files on an SSD).

Why would an external HDD be slower? If it runs at 7200 rpm like most modern HDD, it should be as fast? or maybe it's due the fact that it's not SATA but USB?

I just was looking forward buying one of these Libreboot t400 laptops, I saw some people talking about them as a safe way to run a node away from spyware (see them at minifree dot org)

What do you think? I would pick the 8 GB 1TB HDD configuration. Assuming the blocksize stays at 1MB it would last me for a long time until I need to pick a bigger HDD.

Those computers seem like a good value for a computer of that spec. I'd suggest the 8GB 1TB HDD will probably be sufficient at the current status of the blockchain. Every year the blockchain increases by ~30GB so you can do calculations from there to see if it is indeed sufficient. Onto operating systems, I'd suggest Debian or Ubuntu if you want Linux as they are the most widely used open source Linux packages with bitcoin core.

Also sata can function at 1GBPS+ whereas a USB is normally limited to 512mbps (hence why internal hard drives are more expensive than external ones, better performance needed).

As an extra note, it is possible to prune the blockchain which was maybe what achow alluded to. This means you keep only the latest blocks after syncing the entire chain once at the start (and obviously whenever data becomes corrupted)./

[LoyceV]

I do not consider SSD as safe when it comes to storing sensitive data (such as your wallet.dat) since in the event that you needed to wipe it out, you couldn't, that I know off. In classic HDD drives, you could delete it easily.

An easy way to wipe it, is simply remove it and then fill the disk with other data. Just copy your music collection until the disk is full, then remove it again.

So my idea was to try to make Bitcoin Core run faster by allocating chainstate files in an SSD, then store wallet.dat and blockchain files in an external HDD.

I have an SSD and HDD in my laptop. My wallet.dat used to be in my home directory, the rest of the Bitcoin Core data is on the hdd. Since the 0.15-upgrade though, Bitcoin Core doesn't allow the wallet.dat to be symlinked outside it's data directory.
I haven't tried to symlink the chainstate yet, if it improves speed I'll go that way. And you just gave me the idea to symlink the blocks-directory, so I can have my data-directory and wallet on my SSD again.
Normally, a program just follows any symlink the file system shows. I disagree with the way Bitcoin Core rejects a symlinked wallet.dat.

(im not sure how fast it will be with this method, but I heard that the important files to have in the SSD are the chainstate files

Normally, I just keep it running, but when it starts, it takes a while indeed. It does make sense faster access to chainstate can improve that.

I wouldn't be so happy storing it on a USB disk though, that risks unplugging and corrupting your data.

Test results
chainstate on hdd
When starting, "Verifying blocks" takes a long time.
Total time to start Bitcoin Core 0.15.1, and close it directly after popping up: 1m20.391s.

chainstate on ssd
When starting, "Verifying blocks" is much faster.
Total time to start Bitcoin Core 0.15.1, and close it directly after popping up: 0m22.246s.

For testing, I made sure my file cache was empty. All data has to come from disk while loading Bitcoin Core.

I'd say you're on to something
It's much faster, and as a bonus I can have my wallets on my ssd again. Thanks for giving me the idea!

The only drawback is that the chainstate directory performs many write actions, adding to the wear level of my SSD.

Basically, from your perspective, I did this:

Code:

cd ~/.bitcoin # my home directory is on the ssd
mv blocks/ /media/hdd/bitcoin/
ln -s /media/hdd/bitcoin/blocks blocks

You want to move your chainstate, so use some variation to this:
In ~/.config/Bitcoin/Bitcoin-Qt.conf , edit the location of your bitcoin data directory to point it at your USB-disk

Code:

strDataDir=/media/usd-disk/bitcoin

You'll need to format the usb disk with a Linux file system to deny other users access to that directory.

Code:

mkdir /home/directory-on-ssd/bitcoin
mv chainstate/ /home/directory-on-ssd/bitcoin/
ln -s /home/directory-on-ssd/bitcoin/chainstate chainstate