Corrupt Database

[toshiva]

Bitcoin Client Software and Version Number: 0.14.1.0
Operating System: Ubuntu 14.04.5 64-bit
System Hardware Specs: Q9150e, 6GB RAM, 400GB HDD
Description of Problem: Corrupt database causes Core to shutdown repeatedly. Various messages seen in debug.log including 'Corrupt database shutting down', something along the lines of Bad-txnmerklroot (sorry — this is from my memory, may be inexact).
Any Related Addresses:
Any Related Transaction IDs:
Screenshot of the problem:
Log Files from the Bitcoin Client:

I keep running into this issue when trying to sync. Closest I've got so far is just under three years ago.

Is it enough to simply delete the highest number blk***.dat and rev***.dat files, and start bitcoin-qt again? I've been trying this and was making some progress which seems to have stalled, the problem persists. I've also tried both increasing and decreasing the process niceness. Planning to do a memtest. Wondering also if there are other files (chainstate?) that I should also delete?

Any suggestions would be much appreciated!

[achow101]

Can you post the debug.log file?

The latest blk*.dat and rev*.dat files may not be enough. You will need to check the debug.log file to see which files it is actually failing at. You should also delete the chainstate folder so that you fully rebuild the databases.

You may need to resync the entire blockchain from scratch in which case you should just delete the blocks folder.

[toshiva]

Thank you. I'll try that. I have not seen any error messages related to specific blocks. This is the end of the log after shutdown:

2017-06-20 18:03:13 UpdateTip: new best=0000000000000044dc1a31d2d3ed0edd166c0961492a458ca1e16fddae834ab6 height=248483 version=0x00000002 log2_work=70.882806 tx=21122820 date='2013-07-26 03:44:48' progress=0.091470 cache=335.5MiB(553686tx)
2017-06-20 18:03:13 LevelDB read failure: Corruption: block checksum mismatch
2017-06-20 18:03:13 Corruption: block checksum mismatch
2017-06-20 18:05:34 connect() to [2001:0:9d38:6abd:b3:1e28:8cbd:5c34]:8333 failed: Network is unreachable (101)
2017-06-20 18:06:40 connect() to [2003:e6:bce:bc00:cd65:ced0:11cb:3c4b]:8333 failed: Network is unreachable (101)
2017-06-20 18:13:02 connect() to [2001:0:4137:9e76:1021:d149:da84:5dd6]:8333 failed: Network is unreachable (101)
2017-06-20 18:13:10 connect() to [2a02:8109:b140:6a4:2db5:dd00:94d:da56]:8333 failed: Network is unreachable (101)
2017-06-20 18:18:53 connect() to [2001:0:9d38:6abd:2437:3c1a:8343:b59a]:8333 failed: Network is unreachable (101)
2017-06-20 18:22:08 socket sending timeout: 1201s
2017-06-20 18:22:09 socket sending timeout: 1201s
2017-06-20 18:22:14 socket sending timeout: 1201s
2017-06-20 18:22:14 socket sending timeout: 1201s
2017-06-20 18:22:21 socket sending timeout: 1201s
2017-06-20 18:22:28 socket sending timeout: 1201s
2017-06-20 18:22:33 socket sending timeout: 1201s
2017-06-20 18:22:50 socket sending timeout: 1201s
2017-06-20 18:48:19 Error reading from database: Database corrupted

[achow101]

Thank you. I'll try that. I have not seen any error messages related to specific blocks. This is the end of the log after shutdown:

Can you post some of what comes before that? There should be lines saying which numbered file it is currently indexing. Otherwise your best bet would be to just delete the blocks and chainstate folders and resync the blockchain from scratch.

[toshiva]

I ran it again after deleting the chainstate as well. So a different, but same looking (to me) output. As far as I can see in the log file (I guess it only captures the tail end of it), it all seems to be variations of the first line of the above snippet, with the just the date increasing. Would the most recent date allow me to figure out the block that I should be deleting?:

2017-06-20 23:56:11 UpdateTip: new best=00000000000000000781bcb70bb508ec057746cd99c5d1de40b3214a868c2e0d height=316718 version=0x00000002 log2_work=80.287983 tx=45003211 date='2014-08-21 00:55:02' progress=0.194825 cache=313.0MiB(246087tx)
2017-06-20 23:56:11 UpdateTip: new best=00000000000000000cd5fc1fe02bcbb3ef3474d475353eb98e64e48fdc5584e9 height=316719 version=0x00000002 log2_work=80.288083 tx=45004039 date='2014-08-21 01:18:20' progress=0.194828 cache=313.1MiB(246651tx)
2017-06-20 23:56:18 Corruption: block checksum mismatch
2017-06-20 23:56:18 *** System error while flushing: Database corrupted

I've tried running it several times and everytime it aborts. Does this indicate hardware issue?

[achow101]

I ran it again after deleting the chainstate as well. So a different, but same looking (to me) output. As far as I can see in the log file (I guess it only captures the tail end of it), it all seems to be variations of the first line of the above snippet, with the just the date increasing. Would the most recent date allow me to figure out the block that I should be deleting?:

2017-06-20 23:56:11 UpdateTip: new best=00000000000000000781bcb70bb508ec057746cd99c5d1de40b3214a868c2e0d height=316718 version=0x00000002 log2_work=80.287983 tx=45003211 date='2014-08-21 00:55:02' progress=0.194825 cache=313.0MiB(246087tx)
2017-06-20 23:56:11 UpdateTip: new best=00000000000000000cd5fc1fe02bcbb3ef3474d475353eb98e64e48fdc5584e9 height=316719 version=0x00000002 log2_work=80.288083 tx=45004039 date='2014-08-21 01:18:20' progress=0.194828 cache=313.1MiB(246651tx)
2017-06-20 23:56:18 Corruption: block checksum mismatch
2017-06-20 23:56:18 *** System error while flushing: Database corrupted

I've tried running it several times and everytime it aborts. Does this indicate hardware issue?

The issue likely lies with the block data already on disk. Since you are not deleting any of it, the error will keep appearing as it hits the corrupted data repeatedly. Again, try deleting the blocks folder too in order to resync the entire blockchain. If you get the same error after doing that, then you likely have a hardware error.