*** Corrupt block found indicating potential hardware failure; shutting down

[galmok]

My bitcoin core just started to report this error after I shut down my pc, apparently not waiting long enough for BC to shut down.

I waited over 5 minutes, but bitcoin remained unresponsive, but busy working on the harddisk. Couldn't delay the reboot any longer. I hate that bitcoin can't shut down faster than once every 5-10 minutes…

Anyway, what is the cheapest way for me to rescue the database? Is it even possible?

I get some weird entries in the debug log:

2020-06-14T20:40:11Z Bound to 0.0.0.0:8333
2020-06-14T20:40:11Z init message: Loading P2P addresses...
2020-06-14T20:40:11Z *** Corrupt block found indicating potential hardware failure; shutting down
2020-06-14T20:40:11Z Error: A fatal internal error occurred, see debug.log for details
2020-06-14T20:40:11Z Loaded 73275 addresses from peers.dat  262ms
2020-06-14T20:40:11Z init message: Starting network threads...
2020-06-14T20:40:11Z net thread start
2020-06-14T20:40:11Z init message: Done loading
2020-06-14T20:40:11Z addcon thread start
2020-06-14T20:40:11Z opencon thread start
2020-06-14T20:40:11Z dnsseed thread start
2020-06-14T20:40:11Z msghand thread start
2020-06-14T20:40:14Z UPnP: ExternalIPAddress = 85.184.138.108
2020-06-14T20:40:14Z AddLocal(85.184.138.108:8333,3)
2020-06-14T20:40:14Z UPnP Port Mapping successful.
2020-06-14T20:40:22Z Loading addresses from DNS seed seed.bitcoin.sipa.be
2020-06-14T20:40:23Z Loading addresses from DNS seed seed.btc.petertodd.org
2020-06-14T20:40:23Z Loading addresses from DNS seed seed.bitcoin.jonasschnelli.ch
2020-06-14T20:40:34Z Loading addresses from DNS seed seed.bitcoinstats.com
2020-06-14T20:40:34Z Loading addresses from DNS seed dnsseed.bitcoin.dashjr.org
2020-06-14T20:40:34Z Loading addresses from DNS seed dnsseed.emzy.de
2020-06-14T20:40:46Z Loading addresses from DNS seed seed.bitcoin.sprovoost.nl
2020-06-14T20:40:46Z Loading addresses from DNS seed dnsseed.bluematt.me
2020-06-14T20:40:46Z 165 addresses found from DNS seeds
2020-06-14T20:40:46Z dnsseed thread exit
2020-06-14T20:41:44Z ERROR: ConnectTip: ConnectBlock 000000000000000000075751d2ce1e7b550617d3f1b060352dbbaedb1b959cb3 failed, bad-txnmrklroot, hashMerkleRoot mismatch (code 16)
2020-06-14T20:41:44Z Failed to connect best block (bad-txnmrklroot, hashMerkleRoot mismatch (code 16))
2020-06-14T20:41:44Z GUI: Platform customization: "windows"
2020-06-14T20:41:44Z GUI: QObject::connect: Cannot queue arguments of type 'size_t'
(Make sure 'size_t' is registered using qRegisterMetaType().)
2020-06-14T20:41:44Z tor: Thread interrupt
2020-06-14T20:41:44Z addcon thread exit

Not sure what really has gone wrong here, and even less how to fix this (although I know I can delete it all and wait several weeks for the resync)...

[DaveF]

You can try a reindex to see if that fixes it. Depending on the speed of the machine / the speed of the drive / the amount of RAM / what else the machine is doing at the time it can take a while, but you do not have to download everything from scratch.

If you want to wipe and start again and you have enough RAM using a larger DB cache can help speed it up.

Some notes: https://bitcointalk.org/index.php?topic=5246078.msg54478975#msg54478975

-Dave

[galmok]

Hmm reindex… Is that where I delete the chainstate folder and start Bitcoin Core up again?

[galmok]

Well, I did just that and now it is processing 10 years of blocks chains… I'll take a nap. ;-)

[DaveF]

No, you start bitcoin with -reindex  exact command will be determined by OS and GUI or not give a quick google search.
It will go though all the blocks on the disk and only re-download the corrupted ones.
Should fix your problem. If not then you delete and start again.

-Dave

[nc50lc]

2020-06-14T20:41:44Z ERROR: ConnectTip: ConnectBlock 000000000000000000075751d2ce1e7b550617d3f1b060352dbbaedb1b959cb3 failed, bad-txnmrklroot, hashMerkleRoot mismatch (code 16)
2020-06-14T20:41:44Z Failed to connect best block (bad-txnmrklroot, hashMerkleRoot mismatch (code 16))

If you're lucky, this might be the only block that's corrupted by the improper shutdown (height 633,921).
And that's not that old.

If -reindex didn't worked, you can delete the last 'blkxxxxx.dat' and 'revxxxxx.dat' files (highest number) in the "blocks" folder.
Start Core with -reindex again and it should rebuild the database and sync starting from the last block in your disk.

You might want to include some of the log containing the actual corrupted block file, something like: "blk02113.dat checksum mismatch"
so you can be sure what block file(s) to delete (if it's older, you need to delete the block files with higher number).

[galmok]

Before the answers arrived, I did what I wrote, removed the chainstate folder and restarted bitcoin core. It was a fix that worked earlier for me.

It still works on this, but very very slowly, and I am wondering why.

I gave the bitcoin core 12GB of RAM  to work with and it seems to have taken most of that. As a consequence, it almost doesn't use the harddisk (which is good, as it is a rotating one).


On closer look, I think Bitcoin Core has crashed.

Yup, it crashed 20 hours ago (but didn't display any crash box, just a GUI that was frozen). Corrupt block. Probably the same corrupt block as before.

I'll take a closer look at your suggestions and see what I can do.

[DaveF]

That's really odd that it crashed with a corrupted block twice.
You might be looking at a hardware issue. That or just some really bad luck.
What does the debug.log say?
Running Windows? Anything in the event viewer application log?

-Dave

[nc50lc]

Before the answers arrived, I did what I wrote, removed the chainstate folder and restarted bitcoin core. It was a fix that worked earlier for me.
-snip-
On closer look, I think Bitcoin Core has crashed.
Yup, it crashed 20 hours ago (but didn't display any crash box, just a GUI that was frozen). Corrupt block. Probably the same corrupt block as before.

Must be because chainstate folder was deleted not the corrupted block;
Once Bitcoin core reached that height, it exhibits the issue again.

BTW, I would like to rephrase this:

(if it's older, you need to delete the block files with higher number).

If the blk file is older, you also need to delete the blk files with higher number together with it.

[galmok]

That's really odd that it crashed with a corrupted block twice.
You might be looking at a hardware issue. That or just some really bad luck.
What does the debug.log say?
Running Windows? Anything in the event viewer application log?

-Dave

Turns out there was a hardware issue. Some RAM I recently got hold of and added to my pc turned out to have a defect that only Bitcoin Core could consistently hit. I did run memtest, men it found nothing.

With the faulty RAM removed, and with some smaller RAM added, the system appears stable again.

I had to fetch all data again, though, but it turns out to be faster than reindexing (which takes forever).

8GB cache however is not enough to. I exit Bitcoin Core once in a while (takes quite a few minutes) when the speed has dropped a lot and it picks up speed again on the next start. Not sure why this is so, but it works.

From debug.log:

2020-06-21T07:09:46Z FlushStateToDisk: write coins cache to disk (38850948 coins, 5485362kB) started
2020-06-21T07:21:39Z FlushStateToDisk: write coins cache to disk (38850948 coins, 5485362kB) completed (713.00s)

That is almost 12 minutes to exit the program.

[galmok]

Only Bitcoin Core used this drive at that time.

I think it is a 5400 rpms drive, but I agree, 12 minutes is way too long.

[nc50lc]

I think it is a 5400 rpms drive, but I agree, 12 minutes is way too long.

That's 5.49GB worth of cached data, I'd say 12 minutes to write it all in a 5400rpms disk is normal.

8GB cache however is not enough to. I exit Bitcoin Core once in a while (takes quite a few minutes) when the speed has dropped a lot and it picks up speed again on the next start. Not sure why this is so, but it works.

Looks like your dbcache is getting filled by the new UTXOs that your client is discovering, and when you close Core, it dumps it in the disk.

That was a lot of UTXO by the way.

[galmok]

That was a lot of UTXO by the way.

I'll take your word on that. I have no idea what UTXO is. ;-)