Error opening block database

[gypaete]

Hi all,

My backup app started complaining a few days ago that it could not back up the files that are obviously the blockchain data which seem to be corrupted,
and when starting bitcoin core (bitcoin-qt) it would report "error opening block database" and a rebuild would fail - i.e. the program just exits.

On top of that, it looks like my backups are incompete, while I have an extarnal wallet.dat backup on a usb key.
It might well be that I need to download the complete blockchain.

For the moment I removed and reinstalled bitcoin-qt from the PPA to my Linux 14.01 AMD 64bit box, but the program does of course look at the corrupted database and fails the same way as before.
What can be done as from this point to get bitcoin-qt going again, ether for a reindex (the lighter trouble of the two) or to start downloading the blockchain again?
(I have a metered internet connection, so I would like to avoid this, because it will take several months to have the blockchain back on line)

Thanks in advance for any help

[gypaete]

to add some hopefully useful info: I have 81.7 Gbytes of blknnnnn.dat files in the Blocks directory - maybe still good, or part of them
database directory only contains one log.000001. file now
chainstate directory contains a load of  mostly 2.1Mbyte .ldb files, some are smaller.
wallet dat is still present.
the end of debug.log shows permission probles when trying to open .bitcoin/blocks/index/LOCK - maybe just a secondary result from aborting previously?

I'll delete the LOCK file and see what happens..

[LoyceV]

If the files got corrupted, your hard drive may be on its end.

Let me help you check your blocks.
Do this in your blocks-directory: md5sum *.dat > checksums_for_Loyce.txt
This should be done in an hour. Post the contents here between code-tags. I'll check if any of my blocks is different. This may help you pinpoint the problem without redownloading everything.

Update: it turns out this doesn't work. All blocks on disk are different, starting from the first one.

[gypaete]

thanks for the help - the hard disk looks clean as per the disk tools and also direct readout of the SMARt data in a terminal window, but you can never trust a disk...

While I try the command you gave me, here is the end of the debug log that seems to hit a permission problem on a LOCK file even after I deleted it.

2016-09-25 10:31:13 init message: Verifying wallet...
2016-09-25 10:31:13 CDBEnv::Open: LogDir=/home/(username)/.bitcoin/database ErrorFile=/home/(username)/.bitcoin/db.log
2016-09-25 10:31:13 scheduler thread start
2016-09-25 10:31:13 Bound to [::]:8333
2016-09-25 10:31:13 Bound to 0.0.0.0:8333
2016-09-25 10:31:13 Cache configuration:
2016-09-25 10:31:13 * Using 2.0MiB for block index database
2016-09-25 10:31:13 * Using 8.0MiB for chain state database
2016-09-25 10:31:13 * Using 290.0MiB for in-memory UTXO set
2016-09-25 10:31:13 init message: Loading block index...
2016-09-25 10:31:13 Opening LevelDB in /home/johan/.bitcoin/blocks/index
2016-09-25 10:31:13 IO error: /home/johan/.bitcoin/blocks/index/LOCK: Permission non accordée  --> French for access denied - I'd love it if error messages were NOT translated or doubled with english all the time
2016-09-25 10:31:15 init message: Loading block index...
2016-09-25 10:31:15 Wiping LevelDB in /home/johan/.bitcoin/blocks/index
2016-09-25 10:31:15 Opening LevelDB in /home/johan/.bitcoin/blocks/index
2016-09-25 10:31:15 IO error: /home/johan/.bitcoin/blocks/index/LOCK: Permission non accordée
2016-09-25 10:31:17 scheduler thread interrupt
2016-09-25 10:31:17 Shutdown: In progress...
2016-09-25 10:31:17 StopNode()
2016-09-25 10:31:17 Shutdown: done

[steven0021]

There's no harm in trying to reindex first before re-downloading the whole blockchain, so reindex first. If I remember correctly you get prompted to reindex or not if qt fails to load the block database, but if you don't get it you do that by running bitcoin-qt with -reindex command. Linux users should know how to do it.

If the files got corrupted, your hard drive may be on its end.

Let me help you check your blocks.
Do this in your blocks-directory: md5sum *.dat > checksums_for_Loyce.txt
This should be done in an hour. Post the contents here between code-tags. I'll check if any of my blocks is different. This may help you pinpoint the problem without redownloading everything.

I don't think this is necessary. You'd still end up reindexing after deleting any of the file. Also it's safer to let bitcoin-qt do its thing than deleting/replacing files by your own.

[LoyceV]

I don't think this is necessary. You'd still end up reindexing after deleting any of the file. Also it's safer to let bitcoin-qt do its thing than deleting/replacing files by your own.

It was meant to check if any if the files got corrupted. I'm more familiar with checking files myself than all the ins and outs of Bitcoin Core.
The list is too long to post as a message. If you post yours I can do the comparing for you.

[gypaete]

Actually the history of this fault is that bitcoin-qt would want to reindex every time I opened it to get my blockchain amended, which started about in april this year.
I saw that there wasn't significant internet useage due to that, so I just would let it do it's thing and at the end of the reindex it would download the missing blocks from the net.
That would just take a whole day every time, so I became a bit lazy re-running bitcoin-qt more often than once a month.

I might have started bitcoin-qt with "sudo bitcoin-qt" at least once since that problem appeared, which may have created the permission problem (LOCK file created by a higher level user than my normal user).
Maybe that even affects one of the directories bitcoin-qt uses.

Corrupt block files could still be the root cause of all this - I might need to move them to another drive, which seemingly also has it's culprits as I read elsewhere.

[LoyceV]

I might have started bitcoin-qt with "sudo bitcoin-qt" at least once since that problem appeared, which may have created the permission problem (LOCK file created by a higher level user than my normal user).
Maybe that even affects one of the directories bitcoin-qt uses.

You can easily check and correct that by changing back the permissions of all files to the user that should own them.

[gypaete]

md5sum ran into permission refusals on a number of block files - possibly created by my previous runs of bitcoin-qt in supervisor mode.
Changing the ownership of all the files in the .bitcoin directory tree seems to have solved that problem.

(sudo chown -R (username):users .bitcoin)

Bitcoin-core is now busy verifying blocks. let's see what it says in a couple of many hours..

.. wow, it fell on it's feet already and is now synchronizing, so the blocks on disk should be unharmed. It is some weeks behind on the blockchain, obviously.

Thanks for the help! I'll re-run the md5sum checker and update this thread in case I see the same trouble again.

The backup must also just have muttered on just these root-owned blocks that it wasn't allowed (not unable) to back up.

Bye, and thanks again.

[gypaete]

while the program is synchronizing, I had a look into the debug log and found:

2016-09-25 11:20:21 Verifying last 288 blocks at level 3
2016-09-25 11:20:21 [0%]...[10%]...[20%]...[30%]...[40%]...[50%]...[60%]...[70%]...[80%]...[90%]...[DONE].
2016-09-25 11:23:39 No coin database inconsistencies in last 36 blocks (60715 transactions)

Would that mean that running the md5sum check on all blocks and sending you the output text would still be wise, since bitcoin-qt hasn't really checked them all?