Safe to restart Core after out of space crash?

[jnano]

Bitcoin Core (0.18) crashed due to chainstate LevelDB hitting "not enough space".
Freeing up some space and restarting seemed to have worked without a hitch. I think not even a hint in the log file (after the restart, not the previous crash run).

Is it absolutely safe as far as data consistency is concerned, or is it better to continue from a backup?

[1713993302]

1713993302

[BitMaxz]

I just a little confused about your question what actually do you want to do with your wallet?

Do you want to fix the "Not enough space"? or are you planning to use your backup to export private keys and import them to another wallet?

If you want to fix the "not enough space" you will need an extra hard drive that can fit 350GB of blockchain you can just copy the whole bitcoin data to the new hard drive and change the properties of bitcoin core under target by adding this

Code:

-datadir=D:\BitcoinData

Just put it at the end with the right bitcoin data path from your new hard drive. Now it should safe to restart the bitcoin core after you apply this.

If you don't have an extra hard drive then your last hope is to backup and export the private keys and import them into Electrum(This is fast and easy).

[jnano]

I just want to continue using it normally. The backup, if needed, includes the chainstate and block data.

I fixed the out of space issue, restarted Core and let it continue catching up with the blockchain, and it seems to work.
The space issue affected only the chainstate, not the blocks (it runs in pruned mode).

But I do wonder if there may be some inconsistency in the chainstate or block data, despite the fact that it doesn't complain and everything seems to work fine.


By the way, I've had it crash twice due to disk space. The second time the GUI's error message complained about something else, maybe a block hash mismatch, but the log only mentioned LevelDB running out of space.

[ranochigo]

The space issue affected only the chainstate, not the blocks (it runs in pruned mode).

But I do wonder if there may be some inconsistency in the chainstate or block data, despite the fact that it doesn't complain and everything seems to work fine.


By the way, I've had it crash twice due to disk space. The second time the GUI's error message complained about something else, maybe a block hash mismatch, but the log only mentioned LevelDB running out of space.

The last time I checked, Bitcoin Core shuts down gracefully upon running out of disk space so not really a lot of concern for any corruption there. Even if there is, it wouldn't be a big issue, just replace your data.

The leveldb should have CRC32 checksum for the records and would throw an error upon a mismatch.

[jnano]

It crashed with apparent C++ exceptions, though maybe with some cleanup afterwards.
LevelDB might keep the records valid, but what about Bitcoin-level consistency?

The logged error messages are (summarized, some lines were repeated):

Code:

Fatal LevelDB error: IO error: Win32WriteableFile.Append::WriteFile: (...*.ldb): There is not enough space on the disk.
*** System error while flushing: (...)
Error: Error: A fatal internal error occurred, see debug.log for details.
You can use -debug=leveldb to get more complete diagnostic messages
FlushStateToDisk: failed to flush state (...)
 (code 0))
[...] Releasing wallet
shutdown: done

The second crash had less lines, less repetitions, and one new line:

Code:

ERROR: ProcessNewBlock: ActivateBestChain failed (System error while flushing ...)

Although it did follow the above with some thread exit messages, there was no "Releasing wallet" message, but maybe I copied the log file before it fully flushed.

FWIW, later, after it finished syncing the blockchain, I tried a restart and it did not complain about anything.

[NotATether]

It crashed with apparent C++ exceptions, though maybe with some cleanup afterwards.
LevelDB might keep the records valid, but what about Bitcoin-level consistency?

Usually, during an out-of-space crash, only one block or chainstate file gets corrupted by the DB engine not being able to finish writing data to it. Deleting any of those files isn't fatal, it's not like it destroys wallets or anything. It should just re-fetch that information, particularly if you use -reindex.

[jnano]

I didn't run with any special commands, and can't run a full reindex because it's pruned mode. But if it's localized to one or two files, and Core can detect it, then I guess no harm done and the usual startup rewind-rescan can handle it.

BTW, I never understood how chainstate is stored, but it seems every run touches practically all chainstate files, based on file modification times.

[ranochigo]

BTW, I never understood how chainstate is stored, but it seems every run touches practically all chainstate files, based on file modification times.

Looks alright to me. During startup, Core checks the last X blocks for inconsistencies. Specifically, the debug.log prints that they've checked the last X blocks for coin database inconsistencies.

I don't think it does a full check of the entire chainstate, as it'll take too much time. Do CMIIW.

[jnano]

Looks alright to me.

I didn't mean to imply that's abnormal.

Maybe if the whole chainstate is rewritten there's opportunity (not that I know that it does that) to check for complete consistency, or at least no data corruption.

Anyway, I'm going to leave it be. The wallet state is as I expect it to be, and it seems to work. A pre-crash wallet backup should suffice. Worst case I'd have to do a fresh IBD from scratch, which would be a pain, but doable.

It would be more reassuring if, when restarting post-crash, Core would explicitly indicate that it's aware of the previous crash and that it checked or recovered successfully. I don't know, maybe it already does that in newer versions (considering I'm on 0.18).

[DaveF]

It you are really out of space, as in 0 bytes free then there is no way to write the error in the log file.
So, all it knows that there was not a clean shutdown does not know why.
Either way get off of 0.18 and onto something newer.
Any reason why you are still running such an old version?

-Dave

[jnano]

The chainstate directory here is separate from where the log is.

I will update eventually, maybe when v22 is out. It's just that so far there weren't any improvements that are obviously useful to me, and I'm usually going for "leave it be if it's not broken".

[HCP]

If you've corrected the space issue... then continuing to run afterwards should be perfectly fine. Unless you are continuing to get errors when you are starting/running your node, then I wouldn't worry about it.

I will update eventually, maybe when v22 is out. It's just that so far there weren't any improvements that are obviously useful to me, and I'm usually going for "leave it be if it's not broken".

You're not wrong. Sometimes not upgrading is the smart play... if there are not any critical vulnerabilities that have been identified and fixed, the "if it ain't broke, don't fix it" strategy can often be the best course of action

[DaveF]

Odd that there was nothing in the log then.
And this is just me saying it, 0.20 removed OpenSSL and it's issues. Probably not that important if you are just running a desktop wallet for your own stuff. But it's something to think about.

Enjoy the weekend.

-Dave