All Bitcoind / Bitcoin-qt nodes failing to come up. Workaround inside!

[gmaxwell]

Due to a bug introduced in Bitcoind/-qt introduced in v0.8.0 nodes may fail to come up after restart, reporting:

": *** coin database inconsistencies found"
and/or
"Do you want to rebuild the block database now?"

Reindexing will not solve the problem (so I do not recommend you do so, at it will just waste your time).

You can workaround this issue by adding -checklevel=2 to your command-line arguments or checklevel=2 to your configuration file.

The problem that an inconsistency in the transaction database has been caused by some unusual whos version cannot be represented in the database. This inconsistency is correctly detected by the agressive database sanity-checks on startup.  Because this inconsistency happens to be in a field we currently do not use for anything, it is safe to ignore it for now. Lowering the checklevel skips these and other checks and allows the node to proceed as usual.

The issue may resolve itself within another 200 blocks or so if no more of the trigger transactions are added to the blockchain after the triggering block falls out of the window used for consistency checking, but because it could easily happen again a longer term fix is required and has been submitted for review.

UPDATE: 0.8.5 is now out and fixes this issue.


Walkthrough in OSX:
Open a terminal and type
open /Applications/Bitcoin-Qt.app --args -checklevel=2

Walkthrough in Windows:

In Windows, locate the file bitcoin-qt.exe
Usually is to be found in C:\Program Files (x86)\Bitcoin\ (on 64 bit version) or C:\Program Files\Bitcoin\

Right click on the file bitcoin-qt.exe  and select Create Shortcut. On the new file that appeared (bitcoin-qt.exe - Shortcut or bitcoin-qt.lnk) right click and select Properties.
add to Target line
-checklevel=2

as in this picture

https://i.imgur.com/8P1rnRA.png

Then click Apply, then execute.

[1713976477]

1713976477

[Icon]

Yep that got it back up, Thanks Maxwell!

[TraderTimm]

Appreciate the quick turnaround, thank you for your efforts.

[atomium]

thank you for this, ive been having this issue and i cant seem to fix it.

where do i find the config file? I'm on a mac, in library/bitcoin i have 2 folders, one is blocks and one is chainstate then it has logs and the wallet.dat and peers.dat, where could i find the file to modify?

[KaosMcRage]

thank you for this, ive been having this issue and i cant seem to fix it.

where do i find the config file? I'm on a mac, in library/bitcoin i have 2 folders, one is blocks and one is chainstate then it has logs and the wallet.dat and peers.dat, where could i find the file to modify?

~/Library/Application Support/Bitcoin/bitcoin.conf

[atomium]

i dont have that file inside the bitcoin folder, i just have

blocks (folder)
chainstate (folder)
db.log
debug.log
peers.dat
wallet.dat

[KaosMcRage]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

[atomium]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

it worked thank you so much!! i had such a headache trying to figure out why my client keep crashing.

[ghostshirt]

Shoot! I've been reindexing for the last couple of hours. Should've read this before..

[WikileaksDude]

Thanks for the alert and the workaround.

Happy node here!

[Trader Steve]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

it worked thank you so much!! i had such a headache trying to figure out why my client keep crashing.

Still don't understand how to do this. I have a Mac. How do I create the config file and add this line?

[Trader Steve]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

How exactly? I'm not a techie and that link doesn't make sense to me...(I have a Mac)

[atomium]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

How exactly? I'm not a techie and that link doesn't make sense to me...(I have a Mac)

dont worry about the link just create a new file and add "checklevel=2" inside that file as text and then save as bitcoin.conf.

[Trader Steve]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

How exactly? I'm not a techie and that link doesn't make sense to me...(I have a Mac)

dont worry about the link just create a new file and add "checklevel=2" inside that file as text and then save as bitcoin.conf.

Thanks! That worked perfectly! What a great community!

[atomium]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

How exactly? I'm not a techie and that link doesn't make sense to me...(I have a Mac)

dont worry about the link just create a new file and add "checklevel=2" inside that file as text and then save as bitcoin.conf.

Thanks! That worked perfectly! What a great community!

awesome!

this community is great, just remember to help others that dont know much too so that when you ever need help someone can help you out

[goodbc]

In Windows, locate the file bitcoin-qt.exe
Usually is to be found in C:\Program Files (x86)\Bitcoin\ (on 64 bit version) or C:\Program Files\Bitcoin\

Right click on the file bitcoin-qt.exe  and select Create Shortcut. On the new file that appeared (bitcoin-qt.exe - Shortcut or bitcoin-qt.lnk) right click and select Properties.
add to Target line
-checklevel=2

as in this picture

https://i.imgur.com/8P1rnRA.png

Then click Apply, then execute.

[sr_miner]

You can workaround this issue by adding -checklevel=2 to your command-line arguments or checklevel=2 to your configuration file.

That worked perfectly!

Thanks!

[Cryddit]

Why this resulted in bitcoin failure to come up?  With security model in place should have resulted in a chain fork, yes?

[gmaxwell]

Why this resulted in bitcoin failure to come up?  With security model in place should have resulted in a chain fork, yes?

There is no network rule that does anything with the transaction versions currently, there is no fork because the consensus algorithm doesn't care if its consistent. The startup database validation cares if the database is correct, however.

[Brunt, FCA]

I encountered this problem this morning before I was aware of this thread, but in my case reindexing, although it took all day, was successful.
I have now added the suggested line to my bitcoin.conf file, and when I tried it again, it started up correctly.
Lowering the checklevel can't be good for security, so hopefully we will be informed when the extra line may no longer be needed?

[1PFYcabWEwZFm2Ez5LGTx3ftz]

Sorry for silly question, but would it not be better to release 0.8.5 version with this bug fixed, than to give instructions for workaround? Version 0.8.3 also had just one fix added, so why not do the same?

[gmaxwell]

I encountered this problem this morning before I was aware of this thread, but in my case reindexing, although it took all day, was successful.

Indeed, reindexing will now work if after the reindex you leave the node running long enough to process the trigger block and the next ~100 blocks. However if more of these transactions get mined it will break again.

It's ~mostly harmless to have the check level down, it just changes the checking at startup. ... it means _if_ you suffer disk corruption your node may eventually produce a bad block if mining or get stuck processing the chain in the future.  That said, after this we may rename the checklevel knob in the next version just to turn it back to the normal level for people who've stuck in this workaround.

Sorry for silly question, but would it not be better to release 0.8.5 version with this bug fixed, than to give instructions for workaround? Version 0.8.3 also had just one fix added, so why not do the same?

0.8.4 had three-ish changes, but they also had at least three weeks of testing (more much more for some of them). Partially this was possible because we were able to keep the bug that triggered the release a secret until publishing the fixed version.  Because of the workaround is basically as good as the fix and factors mentioned above getting a fixed version out is slightly less urgent than it was for the first day after the issue happened, and I'm not inclined to rush out a version which isn't thoroughly tested.

The fix for this is now merged in the git development branch. There will be an updated release soon.

[jwzguy]

Is it known who submitted these transactions, and with what client?

[gmaxwell]

Is it known who submitted these transactions, and with what client?

I do. They were created by Genjix and his SX wallet software. Looks like it was due to a failure to initialize the version numbers in transactions. I was able to determine this because they spent funds send to the well known and oft reused libbitcoin donation address two hops back, so I sent Genjix an email to ask and he confirmed and tracked down his bug.
 

[movellan]

Seems there are those taking advantage of the FUD produced by this glitch:

https://www.youtube.com/watch?v=MUGgdov5P6g

[keystroke]

Is this the bug I reported back in March?

https://bitcointalk.org/index.php?topic=155140.0

If so, awesome if fixed!

[BTC100]

Thanks for the post, it's very helpful~

[gmaxwell]

Is this the bug I reported back in March?

No, absolutely not.  Your database was corrupted and needed reindexing.  If you are on a mac, your issue may have been fixed in 0.8.4.

[World]

Is it known who submitted these transactions, and with what client?

SX wallet software

This means that each experiment (SX wallet,mastercoin...etc)touch Tx can corrupted thousand users databases in Bitcoin-Qt?

[gmaxwell]

On my side reindexing fixed the problem. I'm using v0.8.3 Coin Control client on WinXP SP3.

It will, sort of, if the problem transactions are not in the last 100-288 blocks or so and you leave it up after the rescan long enough catch up. But no need to now as 0.8.5 is out which fixes it.

This means that each experiment (SX wallet,mastercoin...etc)touch Tx can corrupted thousand users databases in Bitcoin-Qt?

Thats why unusual transactions are normally filtered out by the network, but permitted on the separate testnet.  The corruption in this case was inconsequential, but tripped up the start-time sanity checking.

This particular issue is fixed now.

[bitcoinchecker]

I didn't install the update the other day when the bug surfaced but after reloading the blockchain (it took 20hours) everything worked fine.

I have used the client to carry out many transactions since then with no problem. But this afternoon it "crashed" again and I am currently having to reload again. A pain in the arse and I will try to do everything correctly next time.

But what I want to know is why does (approx) 96% of the blockchain reload in about an hour and the last 4% take several hours?

[LightRider]

But what I want to know is why does (approx) 96% of the blockchain reload in about an hour and the last 4% take several hours?

The checkpoint.

[OldDutchman777]

Just got this message with Qt: "Corrupted Blockchain Detected do you want to rebuild".  Advice please.  TIA

[Arto]

Just got this message with Qt: "Corrupted Blockchain Detected do you want to rebuild".  Advice please.  TIA

Click "Abort" and upgrade to Bitcoin-Qt 0.8.5 before restarting the app. That sorted the problem for me.

[OldDutchman777]

Just got this message with Qt: "Corrupted Blockchain Detected do you want to rebuild".  Advice please.  TIA

Click "Abort" and upgrade to Bitcoin-Qt 0.8.5 before restarting the app. That sorted the problem for me.

Thanks for the reply and I should have read this thread before posting.

One more question:  When this first occurred, I selected the option to begin re-indexing.  Then decided I'd better research this matter before continuing and aborted the re-index at about 67 weeks remaining.  I have now downloaded and installed 0.8.5.   When I run the new version it is continuing the re-indexing from the abort point.  Any problem, risk with allowing this to continue?  Thank you, I really appreciate your help.

[Arto]

Just got this message with Qt: "Corrupted Blockchain Detected do you want to rebuild".  Advice please.  TIA

Click "Abort" and upgrade to Bitcoin-Qt 0.8.5 before restarting the app. That sorted the problem for me.

Thanks for the reply and I should have read this thread before posting.

One more question:  When this first occurred, I selected the option to begin re-indexing.  Then decided I'd better research this matter before continuing and aborted the re-index at about 67 weeks remaining.  I have now downloaded and installed 0.8.5.   When I run the new version it is continuing the re-indexing from the abort point.  Any problem, risk with allowing this to continue?  Thank you, I really appreciate your help.

I didn't go this route myself, but it sounds fine. It'll just take a good long while until it completes, so the "Abort" option would have been faster. Anyhow, one ought to always keep a backup copy of the wallet.dat file somewhere else, and with that in hand there's not much reason to worry; in the worst case you can always start over.

[OldDutchman777]

Just got this message with Qt: "Corrupted Blockchain Detected do you want to rebuild".  Advice please.  TIA

Click "Abort" and upgrade to Bitcoin-Qt 0.8.5 before restarting the app. That sorted the problem for me.

Thanks for the reply and I should have read this thread before posting.

One more question:  When this first occurred, I selected the option to begin re-indexing.  Then decided I'd better research this matter before continuing and aborted the re-index at about 67 weeks remaining.  I have now downloaded and installed 0.8.5.   When I run the new version it is continuing the re-indexing from the abort point.  Any problem, risk with allowing this to continue?  Thank you, I really appreciate your help.

I didn't go this route myself, but it sounds fine. It'll just take a good long while until it completes, so the "Abort" option would have been faster. Anyhow, one ought to always keep a backup copy of the wallet.dat file somewhere else, and with that in hand there's not much reason to worry; in the worst case you can always start over.

Thanks again for the excellent info.

Best

[Beans]

I upgraded to 0.8.5 and tried to -checklevel=2 method. It's still asking to rebuild.

[gmaxwell]

I upgraded to 0.8.5 and tried to -checklevel=2 method. It's still asking to rebuild.

If your database is actually corrupted then thats an unrelated issue. Database corruption can happen for many reasons unrelated to this.

[Bitcollector]

I upgraded to 0.8.5 and tried to -checklevel=2 method. It's still asking to rebuild.

Mine is not asking to rebuild. Mine is just simply frozen and will not go passed the 23 behind. It says that its synchronizing with network, but its not doing anything.

I had 0.8.1 this morning when I realized it was beginning to reload the whole block chain again. I got scared because I thought some virus screwed up my computer and I stopped the syncing. Then I saw someone suggested that to up grade to 0.8.5. This was a mistake because now nothing is happening and its frozen at 23 weeks behind.

I have done the -checklevel=2 in the conf file. It did nothing. Then I put it in the properties of the start up ... still nothing.

So, my question is ... of course, what do I do? delete everything except the wallet.dat file and reinstall?

[Beans]

I upgraded to 0.8.5 and tried to -checklevel=2 method. It's still asking to rebuild.

If your database is actually corrupted then thats an unrelated issue. Database corruption can happen for many reasons unrelated to this.

Shouldn't it only need to rebuild it once though? What do you suggest? Just delete the program and reinstall?

[gmaxwell]

Mine is not asking to rebuild. Mine is just simply frozen and will not go passed the 23 behind. It says that its synchronizing with network, but its not doing anything.

Try just leaving it alone, sometimes it takes another block to trigger progress. If its not throwing any warnings at you it should continue.

Shouldn't it only need to rebuild it once though? What do you suggest? Just delete the program and reinstall?

Correct, assuming you let it go all the way through and then shut it down cleanly. Is it failing at every start for you even after letting it finish?

[doof]

I had to delete my block chain after installing 8.5 on mac

[Dabs]

What's the harm in a lower checklevel? I tried lower checklevels because I don't keep my QT running all the time. And it grabs the current blockchain from the network anyway and verifies it. It makes QT start up much faster too.

[gmaxwell]

What's the harm in a lower checklevel? I tried lower checklevels because I don't keep my QT running all the time. And it grabs the current blockchain from the network anyway and verifies it. It makes QT start up much faster too.

It means you are more likely to suffer from undetected corruption that could potentially be somewhat harmful to your peers. if you're trying to reduce your startup time, setting checkblocks to 6 would be a much better idea than reducing the checklevel.

[Dabs]

What's the harm in a lower checklevel? I tried lower checklevels because I don't keep my QT running all the time. And it grabs the current blockchain from the network anyway and verifies it. It makes QT start up much faster too.

It means you are more likely to suffer from undetected corruption that could potentially be somewhat harmful to your peers. if you're trying to reduce your startup time, setting checkblocks to 6 would be a much better idea than reducing the checklevel.

I remember now, I set my checkblocks to 1 I think. Maybe I should bump it to 2 or 3. I don't think I need 6, in the same way that I can probably accept most transactions with only 1 confirmation.

[gmaxwell]

I remember now, I set my checkblocks to 1 I think. Maybe I should bump it to 2 or 3. I don't think I need 6, in the same way that I can probably accept most transactions with only 1 confirmation.

I will eat my hat if you can notice the difference in speed. You really should have it turned up a bit so that the data checked will at least cross several erase blocks.

[Dabs]

Okay. I'll set it to 7 then. 7 is lucky. hehe. Or 4. I won 4th place in an online poker tourney just now.

System: Dabs finishes tournament in place #4 and wins 2 chips

[lonsharim]

I would like to report that this problem occurs to me on the 0.8.6 client.



I have used the checklevel param to no avail and reindex my client twice (2 long days). Short of redownloading 17gb worth of block chain, I don't know what else to do.

Any ray of light in this pitch dark cave will be most welcome.

This is my bitcoin.conf
rpcuser=rpc
rpcpassword=xxxx
server=1
daemon=1
txindex=1

[lonsharim]

I ran the wallet again this time with -debug option and checked the debug.log


2014-01-21 13:01:14 ERROR: bool CBlock::ReadFromDisk(const CDiskBlockPos&)() : deserialize or I/O error
2014-01-21 13:01:14 ERROR: VerifyDB() : *** block.ReadFromDisk failed at 281326, hash=00000000000000012149f5c1268c481778811d7e83ce7a496bc478bfc31cd3aa
2014-01-21 13:01:21 Shutdown : In progress...

Is there anything I can do to circumvent this?

Thanks

[TheBomber999]

Sorry but i've same problem with 0.8.6 daemon (on centos).

I've tried to run "./bitcoind -checklevel=2" but problem still remains

How can i fix it?

I've already try with "./bitcoind -checklevel=1" and deamon start but block chain not update (still remain on "300712" )

[PrivacyBTC]

Faced same problem in latest core ver 0.10.2

Solved by Setting Right Time and date.

[tomsw6]

i dont have that file inside the bitcoin folder

That's okay, you can just create it and add the line gmaxwell mentioned.

https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File

it worked thank you so much!! i had such a headache trying to figure out why my client keep crashing.

forgive me for my ignorance but how do i "just create it and add the line gmaxwell mentioned"

thank you in advance

[goodbc]

Create or edit the existing file named bitcoin.conf

Add this line in that file:

checklevel=2