[FIXED] MacOS X LevelDB Corruption Bounty (10.00 BTC + 200.2 LTC)

[moderate]

If the Bitcoin code can be refactored, or is already factored, to support easily changing the database "Backend" then you can try HyperLevelDB or RocksDB instead, they are both based off LevelDB and with identical APIs. This can be a good solution to try if problems re-surface with LevelDB.

I'm not seeing the reasoning here. Are you saying that those other options fixed this bug and other ones but didn't contribute it back to leveldb ? I doubt that is the case.

[1713559872]

1713559872

[1713559872]

1713559872

[1713559872]

1713559872

[1713559872]

1713559872

[1713559872]

1713559872

[1713559872]

1713559872

[goatpig]

Are the instances of DB corruptions on Windows caused by the same problems occuring on the OSX builds?

[koelen3]

Well this is my unofficial explanation.

Due of the mac updates, it changed standard applications.
This happened to me when updating to Maverick

Due of the changes, when having installed VLC, it uses your VLC to open your wallet.dat
On the other side, you would have to unlock your basic documents when installing Bitcoin-QT to have a new transaction being sent.
As this changes your file, MAC OS doesn't allow you to change the files without you unlocking them.

You also need to change a 'hidden' file on your Mac, to allow you to connect from files to the internet as Mac brought some "security" updates.

[jordan314]

I can answer 1) and 2) but don't have the skills to write the code for 3).

1) Go to system preferences and click on Date/Time. Set the date to 2012. Close System preferences and hit save. Open Bitcoin-qt and get the corrupted database message. Click abort.
Set system preferences and click set date and time automatically. Open Bitcoin-qt and the database is fine.
2) The database is appearing corrupted to leveldb because the system clock is incorrect.
3) Can you use an atomic clock API and rely on that instead of the system clock to prevent leveldb from becoming corrupted?

[marcotheminer]

Passed this on to a friend who might be able to help. we'll see!

[cfields]

As discussed on IRC, I don't think my first change is enough.

I've poked around and now I can reproduce the problem at-will by injecting some sleeps in key places, basically magnifying an unlikely race condition. I'll post my findings and some patches tomorrow, once I can make a bit more sense of it.

[melacs]

I can only support the people trying to solve this problem, don't have the capacity to program a solution. (It seems that the recent attention for bitcoin is putting more and more pressure on the developers.

It seems as if the sleep mode or shutdown or internet connection break is interrupting the client while it is downloading blocks. Is this true or does the problem also occur when all blocks have been downloaded before shutting down and new blocks need to be added later?
In the first case, the problem could lie with an unfininished block that is considered finished or a block that is downloaded multiple times. (probably similar to what cfields stated)
If it also occurs after having been shut down with a complete download of blocks, then the problem is in how the complete blockchain or a finished block is ended and restarted.

[sixtyeightmk2]

I posted a reply about this in the newbies forums earlier tonight and on github as well.

Basically there are a few things happening with HFS and resource forks, spotlight indexing data, and ssd caching or having a race condition come up when putting the system to sleep or hibernate quickly and taking it out by turning it back on:

https://bitcointalk.org/index.php?topic=348426.msg3731858#msg3731858

See portion with links to HFS ssd tuning.

If you are running on a FAT partition you don't have these problems. (Can't resource fork FAT)

[Diapolo]

Are the instances of DB corruptions on Windows caused by the same problems occuring on the OSX builds?

About which Windows corruptions are you talking about?

Dia

[goatpig]

Are the instances of DB corruptions on Windows caused by the same problems occuring on the OSX builds?

About which Windows corruptions are you talking about?

Dia

Not sure under which criterion this corruption falls. About a week ago I try to start Bitcoin-Qt to no avail. Had to delete the DB folder to get Qt to start, which resulted in a full redownload and verification of the blockchain.

[Roy Badami]

Apropos of nothing, I'll contribute my experience of this problem:

To the best of my knowledge, I have had precisely two unclean shutdowns of my Mac whilst Bitcoin-QT 0.8.x was running.  In two out of two cases, it triggered a rebuild due to a corrupt database.  So I find it interesting that this problem is actually difficult to reproduce - I guess I'm just unlucky.  (This is 10.6 on a MacBook Air.)

What really surprised me, though, is that on the occasion of the second of those unclean shutdowns (the OS X UI froze and I had to do a force power off) not only was I not actively using Bitcoion-QT at the time, but I am 99% sure that I had no network connection, and hadn't had a connection for several minutes.  So it seems pretty unlikely that it was a pathological case involving an interrupted DB transaction.

None of the above makes any sense to me, and it is, of course, quite possible just a fluke occurance - but on the off chance it gives anyone any ideas I thought I'd post it anyway.

roy

[misterbigg]

Are you saying that those other options fixed this bug and other ones but didn't contribute it back to leveldb ? I doubt that is the case.

No. I'm suggesting that if there are more serious problems with LevelDB down the line that prevent features from being added (either due to performance or architecture limitations), then both HyperLevelDB and RocksDB (which are based on LevelDB) are proven alternatives with performance improvements. If the bitcoin backend database is customizable, then it is easy to experiment with alternatives.

[rrjr777]

Hello All,

I have read through this thread and also researched some of the reports of this issue, I read about bitcoin in the beginning, and had so much going on at the time that I sort of dismissed it and forgot about it. With the recent media coverage, I have been researching bitcoin in general and became very interested, and stumbled across this thread. That being said, I have a comment, a suggestion and my own theory / hypothesis regarding what is going on, however, it is also apparent that there could very well be several unrelated "bugs" throwing the same / similar error and exhibiting similar behavior on the client machine.

First, the comment:
It is apparent, considering the recent media frenzy surrounding bitcoin and positive news regarding the acceptance of it as a viable currency, that any hint of instability / corruption / bugs within the codebase, could be enough to sabotage the efforts to date, and damage the confidence in the currency.

Now the suggestion:
I have read the reports, threads, theories, and I am hoping to contribute to the ultimate solution, however, the most important factor in potentially resolving the issue, is to gather accurate facts regarding the issue that is actually being experienced. Not to say that throwing theories out is counterproductive, it could lead someone along the right path, however, I can say for sure, that the probability of actually fixing the issue, is directly correlated to the quality of data collected. Without the exact problem / circumstances, etc. we would be shooting in the dark.

I have read reports of Mac's, Windows, and other OS's at some point getting a corrupt database, and I suspect based on the sparse descriptions of the event, that the innate behavior of LevelDB is the culprit, specifically the way a write is actually committed, (or not), to media, but under what conditions this occurs is paramount to finding a solution. I wish I had gone with my intuition at the beginning of this Bitcoin project, and became involved, I'm too late to the game of mining, and can't afford to even buy a bitcoin at this point. Its taking so long for me to even download the blockchain, that I decided to start this post in parallel, and hope to make a contribution in some way.

If anyone reading this thread, has a machine that is experiencing the corrupted database, it would be priceless to the community at large to post some basic information, such as:

Computer OS - (version / service pack)
Hard Drive - (Solid State / SATA / IDE?)
Date of first experiencing the issue.
The exact Error Message.
What you or the computer were doing when / immediately before the error occurred.
Whether you have hibernate / sleep / power off options on the Network Adapter configured.
Whether your computer was hard wired or wireless.
If the computer crashed prior to the database corruption.
What version of BitcoinQT you were using at the time the error occurred.
Whether you had a previous version of BitcoinQT installed and later upgraded.
If you have been able to resolve the error, (perhaps re-installing an earlier version), and if so, how.

I believe, if we have enough of the individuals who are actually experiencing the issue, report, to the best of their ability, the answers to the questions above, we as a group, may be able to put together the exact scenario / circumstance to reproduce the issue, and most importantly, begin to understand it, and progress to the path of resolution.

Sorry for the lengthy post, I hope it helps.

- Rich

[cfields]

Though I hate to admit defeat (I had nailed down the issue, but my approach to fix was incorrect), Robert Escriva has submitted the correct fix: https://groups.google.com/forum/#!topic/leveldb/GXhx8YvFiig
It probably didn't hurt that he's a leveldb consultant

His victory speech: http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/

I'd like to thank Robert for diving into the issue. He's got my +1 for the bounty.

[wtogami]

https://bitcointalk.org/index.php?topic=337294.msg3718821#msg3718821
Builds were updated with Robert Escriva's patch.  Please test!

[toffoo]

Robert Escriva has submitted the correct fix: https://groups.google.com/forum/#!topic/leveldb/GXhx8YvFiig

"It works for me" (so far ...)  https://github.com/bitcoin/bitcoin/issues/2770#issuecomment-29479456

[doof]

I have had 2 power trips in 2 days, thanks to my Butterflylabs miner.  Both times have forced me to download the chain.  Relieved to see this thread and its not my mac.

Might take a look at the code tonight.

[doof]

Computer OS - (version / service pack)
Hard Drive - (Solid State / SATA / IDE?)
Date of first experiencing the issue.
The exact Error Message.
What you or the computer were doing when / immediately before the error occurred.
Whether you have hibernate / sleep / power off options on the Network Adapter configured.
Whether your computer was hard wired or wireless.
If the computer crashed prior to the database corruption.
What version of BitcoinQT you were using at the time the error occurred.
Whether you had a previous version of BitcoinQT installed and later upgraded.
If you have been able to resolve the error, (perhaps re-installing an earlier version), and if so, how.

2011 27" iMac with HDD
28-11-2013 & 29-11-2012.  The exact same times.  Each morning I have woken up around 5 and plugged in my BTF Jala miner.  This is cause the power to trip, and turn off my mac and other devices.

I don't have a screen shot of the exact message, but off memory "Blockchain db is corrupt and needs to be re indexed"

This as actually occurred on litecoin-qt.  I haven't opend bitcoin-qt to see if the same has occurred.

Seeing how it has happened twice,  I'm certain I can reproduce.   Both times, the block chain had been entirely synced, and the mac sitting there powered on.

[doof]

https://bitcointalk.org/index.php?topic=337294.msg3718821#msg3718821
Builds were updated with Robert Escriva's patch.  Please test!

I will apply this patch and crash my mac in the same was as the above.  See if it doesn't result in a corrupt db.

[wtogami]

https://bitcointalk.org/index.php?topic=337294.msg3718821#msg3718821
New builds again.  We need feedback from MacOS X users who were previously experiencing corruption.  This is a blocker for the release of Bitcoin 0.8.6.