What is the cleanest way to recover from an abruptly terminated Bitcoin Core?

[NotATether]

Long story short, the server my node is hosted on locked out of the network, so I had to power cycle it to connect to it again. Since it wasn't a software reboot, Core did not exit gracefully. At first when I tried to start Core again I got errors related to lock files .bitcoin/.lock and ./bitcoin/wallets/.walletlock, which I both deleted. I also deleted .bitcoin/bitcoind.pid in case that was also used as a lock file.

So having done all that when I start bitcoind (0.20.1 on Linux), I get a message in standard output telling me that, understandably, the block data is corrupted, and that to fix it I have to run -reindex or -reindex-chainstate again.

Code:

2021-02-13T18:46:38Z init message: Loading block index...
2021-02-13T18:46:38Z Opening LevelDB in /home/redacted/.bitcoin/blocks/index
2021-02-13T18:46:38Z Fatal LevelDB error: IO error: lock /home/redacted/.bitcoin/blocks/index/LOCK: Resource temporarily unavailable
2021-02-13T18:46:38Z You can use -debug=leveldb to get more complete diagnostic messages
2021-02-13T18:46:38Z Fatal LevelDB error: IO error: lock /home/redacted/.bitcoin/blocks/index/LOCK: Resource temporarily unavailable
2021-02-13T18:46:38Z : Error opening block database.
Please restart with -reindex or -reindex-chainstate to recover.

Two questions:

1. Is -reindex-chainstate enough to fix the corrupted block files, or do I have to do a full -reindex? (Bandwidth is not an issue).
2. Is there anything else I need to do to gracefully recover an interrupted bitcoin core?

[DaveF]

....
1. Is -reindex-chainstate enough to fix the corrupted block files, or do I have to do a full -reindex? (Bandwidth is not an issue).
2. Is there anything else I need to do to gracefully recover an interrupted bitcoin core?

You will not know till you try. Depends on how badly everything crapped when it crashed. Others might have a better answer but I have had it both ways depending on what the node was doing at the time. I don't know your server but if you are using some caching controller setup when you hit the rest button who knows what data was actually on the drive.

Not much else you can do to recover, it is what it is, a database has to be checked / rebuilt.

-Dave

[ranochigo]

1. Is -reindex-chainstate enough to fix the corrupted block files, or do I have to do a full -reindex? (Bandwidth is not an issue).

Reindexing the chainstate is faster, Bitcoin Core doesn't have to validate and reindex all the blocks in addition to the chainstate. It took me 2 hours to finish reindexing chainstate as opposed to the 6 hours it took for me to do a full reindex. It would depend on your type of corruption to the files to determine which option you should run. Does it throw an error when reading the chainstate or the blocks?

2. Is there anything else I need to do to gracefully recover an interrupted bitcoin core?

\
Database corruption is the only issue that I have during an unclean shutdown. Nothing else should be affected.

[NotATether]

...if you are using some caching controller setup when you hit the rest button who knows what data was actually on the drive.

I can't remember if I have writeback enabled on my disks (yes disks I run them in RAID0) but I'm positive that the block database was the only thing that could've been written to at the time of reset, everything else is just web servers that became idle as a result of loss of connectivity.

Does it throw an error when reading the chainstate or the blocks?

Unfortunately it's an error related to the blocks database as can be seen in the logs (Error opening block database) meaning I have to do a full -reindex. Shouldn't take longer than a couple hours on my hardware though.