bitcoin daemon database error in a block

[darkv0rt3x]

Hello.

I'm having some issues keeping my node running.
I'm using Bitcoin Core version v0.20.99.0-e478b11db under Debain Buster.

I have been being surprised multiple times with my node shut down. When I check the debug.log file I can't see much.
Now I started the daemon again and after it was running normally, I shut it down with 'bitcoin-cli stop' and I can see this:

Code:

2020-06-02T23:06:05Z Writing 0 unbroadcast transactions to disk.
2020-06-02T23:06:05Z Dumped mempool: 0.011839s to copy, 0.49971s to dump
2020-06-02T23:06:05Z WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2020-06-02T23:06:05Z FlushStateToDisk: write coins cache to disk (1089745 coins, 154430kB) started
2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
2020-06-02T23:06:06Z You can use -debug=leveldb to get more complete diagnostic messages
2020-06-02T23:06:06Z FlushStateToDisk: write coins cache to disk (1089745 coins, 154430kB) completed (0.16s)
2020-06-02T23:06:06Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
2020-06-02T23:06:06Z Error: A fatal internal error occurred, see debug.log for details
Error: A fatal internal error occurred, see debug.log for details
2020-06-02T23:06:06Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb)
2020-06-02T23:06:06Z WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2020-06-02T23:06:06Z FlushStateToDisk: write coins cache to disk (705742 coins, 105277kB) started
2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
2020-06-02T23:06:06Z You can use -debug=leveldb to get more complete diagnostic messages
2020-06-02T23:06:06Z FlushStateToDisk: write coins cache to disk (705742 coins, 105277kB) completed (0.15s)
2020-06-02T23:06:06Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
2020-06-02T23:06:06Z Error: A fatal internal error occurred, see debug.log for details
Error: A fatal internal error occurred, see debug.log for details
2020-06-02T23:06:06Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb)
2020-06-02T23:06:06Z [default wallet] Releasing wallet
2020-06-02T23:06:06Z Shutdown: done

What can I do to try to fix this?

[jackg]

It tells you, try starting it up with - debug=leveldb

But it's likely you'll have to redownload the chain.

[nc50lc]

Now I started the daemon again and after it was running normally, I shut it down with 'bitcoin-cli stop' and I can see this:

Code:

2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb)
-snip-
2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb)

Seems like there an issue with file: /media/Storage/Blockchain/chainstate/060607.ldb
Is it still crashing?

If it is: Shutdown Core normally then try to delete the "chainstate" folder in "/media/Storage/Blockchain/".
Start Core and it will rebuild the database (not redownload the blockchain); if it still crash, check the latest log.

[darkv0rt3x]

It tells you, try starting it up with - debug=leveldb

But it's likely you'll have to redownload the chain.

I hope I don't have to. I have been having this problem since last week. And last week I stopped bitcoin daemon normally, tried the '-reindex' option but then I was told to cancel the '-reindex' and just try to restart the daemon normally. So I did and the laptop was almost 24h checking all transactions from the very beginning. Not downloading the entire chain, just re-checking txo's.
Then it finished and was working since then. So I didn't pay much more attention to it until 2 days ago when I noticed the node was not running.

Now I started the daemon again and after it was running normally, I shut it down with 'bitcoin-cli stop' and I can see this:

Code:

2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb)
-snip-
2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb
-snip-
2020-06-02T23:06:06Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb)

Seems like there an issue with file: /media/Storage/Blockchain/chainstate/060607.ldb
Is it still crashing?

If it is: Shutdown Core normally then try to delete the "chainstate" folder in "/media/Storage/Blockchain/".
Start Core and it will rebuild the database (not redownload the blockchain); if it still crash, check the latest log.

It just crashed again. It was working well since yesterday night (when I noticed it was stopped) until 5 minutes ago. It has been working but I can see this block error every now and then, probably when the daemon dumps the cache onto the drive.

Do you think I need to delete the entire 'chainstate' folder? Wouldn't that single file be enough to delete and the daemon re-download it again?

OP, you might want to check and test your HDD/SSD health if the problem persists after following advice by other members. Use tools called smartctl to do so.
Just in case you never use it, here's short command that i usually use

* See general information

Code:

sudo smartctl -a /dev/<your_storage_path>

* Perform short or long test

Code:

sudo smartctl --test=short /dev/<your_storage_path>

Code:

sudo smartctl --test=long /dev/<your_storage_path>

Then use sudo smartctl -a /dev/<your_storage_path> again to see the test result

It tells you, try starting it up with - debug=leveldb

But it's likely you'll have to redownload the chain.

Only if the blockchain files corrupted as well, from the log, looks like only single chainstate file is corrupted.

Thanks for the cheat sheet of commands.



Edited:

Well, I just noticed the error is now on a different file... A file that doesn't even exist in my chainstate folder. It's '061114.ldb' file.
I also noticed something I'm not sure is normal. I have a symlink inside 'chainstate' to... 'chainstate'...

Code:

-rw------- 1 narayan narayan  88K ago 15  2019 104069.ldb
lrwxrwxrwx 1 narayan narayan   10 ago 10  2019 chainstate -> chainstate
-rw------- 1 narayan narayan   16 ago 15  2019 CURRENT
-rw------- 1 narayan narayan    0 ago 10  2019 LOCK
-rw------- 1 narayan narayan 337K ago 15  2019 MANIFEST-102299

[HCP]

It would appear increasingly likely that there is a hardware issue of some description.

If you're getting corrupted files occurring randomly, then the most likely culprits are either your HDD/SSD or possibly bad RAM... there is also a possibility that it's the drive/memory controller and/or chipset on the mobo that is failing.

Diagnosing the exact cause of these faults can be tricky... you might also want to try using Memtest86+ and see if there are any issues with your RAM.

[darkv0rt3x]

It would appear increasingly likely that there is a hardware issue of some description.

If you're getting corrupted files occurring randomly, then the most likely culprits are either your HDD/SSD or possibly bad RAM... there is also a possibility that it's the drive/memory controller and/or chipset on the mobo that is failing.

Diagnosing the exact cause of these faults can be tricky... you might also want to try using Memtest86+ and see if there are any issues with your RAM.

Well, at this point I just facked up this shit.

I accidentally deleted part of my .bitcoin folder.

Now I had to re-write my bitcoin.conf file and I don't remember the exact settings I had before.

Anyways, I wrote a new bitcoin.conf file and I started bitcoin with the switch '-reindex' and pray that this fix the missing data in my .bitcoin folder.

I see millions of messages like this in my terminal right now:

Code:

2020-06-03T22:50:20Z Reindexing block file blk00085.dat...
LoadExternalBlockFile: Processing out of order child 00000000000000199dbc751fccefd9990a69d84ce3b65df2111ad353f142f116 of 000000000000000cb8e8600edcc8d919f844ee84a99d6bf85a67073e15497452

and

Code:

2020-06-03T23:09:22Z Reindexing block file blk00719.dat...
LoadExternalBlockFile: Out of order block 000000000000000002aca1df9327326ae86db6fb26c2a4f7174fb3fd68ee5cbf, parent 000000000000000002016b6e847878e3f4ada7eb3fbe23a19ab069270308f151 not known

I hope this can download the missing data and just fix this once and for all.

[nc50lc]

It just crashed again. It was working well since yesterday night (when I noticed it was stopped) until 5 minutes ago. It has been working but I can see this block error every now and then, probably when the daemon dumps the cache onto the drive.

Do you think I need to delete the entire 'chainstate' folder? Wouldn't that single file be enough to delete and the daemon re-download it again?

It will ask you to rebuild the whole database even if you delete that single file; it might even produce errors.
Rebuilding the database wont take long compared to syncing the whole blockchain.

Anyways, based from the new replies, it looks like you have more corrupted files than the one in the chainstate folder.
Your .blk files seems to have inconsistencies like missing transactions probably because of corrupted data.
You might need to check for hardware issues.

[darkv0rt3x]

Ok, the proccess is now complete.

Later today, after work, I'll run a few hardware tests, just for the sake of it.
Bitcoind seems to be running ok now.

I'll post feedback later.

Thanks

[darkv0rt3x]

Hello again...

Well, my node looks running ok, but when I run:

Code:

bitcoin-cli getblockcount

I get only about 402000 blocks.
What do I need to do to get the missing blocks from the Blockchain?

[HCP]

I get only about 402000 blocks.

For whatever reason, it seems like your node is not syncing... is the number of blocks increasing at all?

What output do you get if you run bitcoin-cli getnetworkinfo? Do you see "networkactive": true and "connections": XX where XX is > 0?

[darkv0rt3x]

I get only about 402000 blocks.

For whatever reason, it seems like your node is not syncing... is the number of blocks increasing at all?

What output do you get if you run bitcoin-cli getnetworkinfo? Do you see "networkactive": true and "connections": XX where XX is > 0?

Well, I decided to restart bitcoind with -reindex-chainstate...

About getnetworkinfo, i get:

Code:

{
"version": 209900,
"subversion": "/Satoshi:0.20.99/",
"protocolversion": 70015,
"localservices": "0000000000000409",
"localservicesnames": [
"NETWORK",
"WITNESS",
"NETWORK_LIMITED"
],
"localrelay": true,
"timeoffset": -3,
"networkactive": true,
"connections": 11,
"networks": [

So, yes, network is active and with 11 connections.
However I can also see that "NETWORK_LIMITED" and I'm not sure what it means in this specific context. I don't know what is usually there when the node is fully running.

[HCP]

Well, it's connected and achieving network connections to peers, so it should be receiving new blocks...

Using the reindex option, you might need to give it some time to finish that process before it starts receiving/validating new blocks... is there anything obvious in the debug.log that might explain why it's not?

[darkv0rt3x]

Well, it's connected and achieving network connections to peers, so it should be receiving new blocks...

Using the reindex option, you might need to give it some time to finish that process before it starts receiving/validating new blocks... is there anything obvious in the debug.log that might explain why it's not?

Well, my debug log is recent because I forgot to set it up after I accidentally deleted part of my folder.
But I set it up yesterday and again I made a mistake. I set it up like:

Code:

debug=net,httprpc,reindex,proxy,leveldb

I missed a coma. Now, I'm not sure what is it logging.
And probably, if I stop bitcoin daemon, fix the debug setup and start -reindex-chainstate,it will restart the process from the very beginning, no?

[darkv0rt3x]

This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks.

Do I have any other way, other than '-reindx-chainstate' to try to fix my database?

My 'blocks' folder size says 298GB and my 'index' folder (inside 'blocks' folder), says only 94MB.
So, I think I have all the blocks files but looks like I might be missing some data from folder 'index', no?
Will this be the only way to fix the problem or maybe some other approach can do the trick once I have all the data in 'blocks' folder?

[nc50lc]

This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks.

I think it's quite normal.
Sync is usually faster at the first few blocks when there're few transactions in them: smaller blocks sizes.
Then considerably slower at the last few blocks because of the verification that was skipped in the old blocks.
Consider adding tweaks to your config file like dbcache="half_of_your_RAM+" if you're using the default size.

The index folder's size seems fine to me.

A new debug.log entries will help a lot.

[darkv0rt3x]

This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks.

I think it's quite normal.
Sync is usually faster at the first few blocks when there're few transactions in them: smaller blocks sizes.
Then considerably slower at the last few blocks because of the verification that was skipped in the old blocks.
Consider adding tweaks to your config file like dbcache="half_of_your_RAM+" if you're using the default size.

The index folder's size seems fine to me.

A new debug.log entries will help a lot.

Yeah, I know about the first blocks being faster and last blocks being significantly slower.

In the meantime I tried to stop bitcoind -reindex-chainstate and restarted bitcoind normally, but using dbcache=12000 (I have 16GB of RAM) and this last night it progressed quite well, considering the last 24h. It's processing around file blk01800.dat.

So, I expect this finishes during today an I hope I have my full data again. I'll then run the smart tools on this drive to check it's sanity.

[Jet Cash]

Have you backed up your wallet and associated files? You should do this before you do any experimenting or attempt recovery.

[darkv0rt3x]

This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks.

I think it's quite normal.
Sync is usually faster at the first few blocks when there're few transactions in them: smaller blocks sizes.
Then considerably slower at the last few blocks because of the verification that was skipped in the old blocks.
Consider adding tweaks to your config file like dbcache="half_of_your_RAM+" if you're using the default size.

The index folder's size seems fine to me.

A new debug.log entries will help a lot.

I fail how it's normal, unless OP's internet is quite slow (200 kb/s or slower) or there's major bottleneck (indicated by 100% CPU or HDD usage almost all the time).

I'll then run the smart tools on this drive to check it's sanity.

If the result is quite bad, you just shorten age of your storage device.

No, my  internet is ok. It's Optics Fiber and it's rated at 200/100 Mbits.

An I don't even know if the internet is playing any role right now, because I just add an internet disconnection and the process just continued!
As I have my folder with the total amount it is supposed to have, I think that whatever it is doing, it is only using data that I already have on my drive.

[darkv0rt3x]

Well, this is in the end but I started getting this messages:

Code:

2020-06-06T16:58:03Z Pre-allocating up to position 0x500000 in rev02102.dat
2020-06-06T16:58:03Z UpdateTip: new best=0000000000000000000386f1b8776307e3e54a92e0a8edfd8dbd97d0d8f253d9 height=633218 version=0x20000000 log2_work=92.006175 tx=536228356 date='2020-06-05T16:02:34Z' progress=0.999447 cache=9803.2MiB(72194509txo)
2020-06-06T16:58:03Z UpdateTip: new best=00000000000000000009013052d1b341bf00db0740e14cf9e97fe5700d4c5c4a height=633219 version=0x20000000 log2_work=92.006192 tx=536230681 date='2020-06-05T16:19:44Z' progress=0.999453 cache=9803.4MiB(72196232txo)
2020-06-06T16:58:04Z UpdateTip: new best=00000000000000000000b078867de04ff83dd8a5d77c30ef92505cbb0d544d05 height=633220 version=0x3fffe000 log2_work=92.006209 tx=536233017 date='2020-06-05T16:21:52Z' progress=0.999454 cache=9803.3MiB(72195112txo)
2020-06-06T16:58:04Z UpdateTip: new best=0000000000000000000455ee424e84e2d2cdd811b35396b5ef1a2cbbf0e7662b height=633221 version=0x20400000 log2_work=92.006226 tx=536235909 date='2020-06-05T16:32:02Z' progress=0.999458 cache=9803.5MiB(72197006txo)
2020-06-06T16:58:05Z UpdateTip: new best=000000000000000000015f4b7af01c7b64b1d0ca47776bad6d4af03fbc361101 height=633222 version=0x20400000 log2_work=92.006243 tx=536238667 date='2020-06-05T16:49:24Z' progress=0.999464 cache=9803.8MiB(72199590txo)
2020-06-06T16:58:05Z UpdateTip: new best=000000000000000000024cfd85e688590ecd8318c296fcb54025700c283c5d42 height=633223 version=0x20400000 log2_work=92.00626 tx=536241630 date='2020-06-05T16:52:13Z' progress=0.999465 cache=9803.8MiB(72199791txo)
2020-06-06T16:58:06Z Leaving InitialBlockDownload (latching to false)
2020-06-06T16:58:07Z UpdateTip: new best=0000000000000000001315d3b171b3bbfb604d2ae84b92ce0271d0613d2bf3ca height=633224 version=0x20000000 log2_work=92.006277 tx=536244335 date='2020-06-05T16:59:28Z' progress=0.999468 cache=9803.9MiB(72200219txo) warning='62 of last 100 blocks have unexpected version'
2020-06-06T16:58:08Z UpdateTip: new best=000000000000000000045a44c0a3193a78726d49e7ceb15330d97452f7b38058 height=633225 version=0x37ffe000 log2_work=92.006294 tx=536246822 date='2020-06-05T17:03:42Z' progress=0.999470 cache=9803.8MiB(72199978txo) warning='63 of last 100 blocks have unexpected version'

Edited:

This is doing the same whole shit again.
My chainstate folder is only about 1000 items.
I stopped the daemon again an I got the following output:

Code:

2020-06-06T17:13:04Z FlushStateToDisk: write coins cache to disk (72150244 coins, 10273727kB) started
2020-06-06T17:15:32Z Error: Out of memory. Terminating.
Aborted (core dumped)

[HCP]

Those first messages are just showing that it was syncing new blocks... you can ignore the "unknown version" messages... while it can be alarming to see, they're 'normal'.

It's a bit tricky to diagnose when you're giving snippets of debug.log... so you got that "Error: Out of memory" AFTER you executed bitcoin-cli stop and the node was shutting down? Did you get the normal shutting down debug like this:

Code:

2020-06-05T06:31:18Z tor: Thread interrupt
2020-06-05T06:31:18Z opencon thread exit
2020-06-05T06:31:18Z addcon thread exit
2020-06-05T06:31:18Z Shutdown: In progress...
2020-06-05T06:31:18Z torcontrol thread exit
2020-06-05T06:31:18Z net thread exit
2020-06-05T06:31:18Z msghand thread exit
2020-06-05T06:31:44Z scheduler thread exit
2020-06-05T06:31:44Z Dumped mempool: 0.000997s to copy, 0.152187s to dump
2020-06-05T06:31:44Z FlushStateToDisk: write coins cache to disk (677775 coins, 94304kB) started

And then it died at this point with the OOM?