[FIXED] MacOS X LevelDB Corruption Bounty (10.00 BTC + 200.2 LTC)

[Rampion]

Since 0.8x I have the corrupt db (and thus forced rescan) each time that my computer powers off due to battery/electrical failure.

Wtogami: try to power off your computer without shuttind down Bitcoin-QT, the db will probably be damaged.

[1713530807]

1713530807

[1713530807]

1713530807

[1713530807]

1713530807

[1713530807]

1713530807

[wtogami]

Since 0.8x I have the corrupt db (and thus forced rescan) each time that my computer powers off due to battery/electrical failure.

Wtogami: try to power off your computer without shuttind down Bitcoin-QT, the db will probably be damaged.

0.8.5 OMG3 has three patches that prevent other corruption, although not the particular issue of this bounty.  Also we have never seen corruption on 10.6.8.

[medicine]

I run the latest versions of both bitcoin QT and litcoin QT on my Mac running the Maverick OS.  I don't use time machine and I've never had a problem with either program. 

I won't be using Time machine now either.

[toffoo]

Thank you to Gavin and Warren and everyone for your efforts to help resolve this issue.  I think this bounty is a great idea and I'm very happy to see the Mac client receive some 1st class attention.

2013 has probably been bitcoin's most exciting year, but it's been a rough one for some Mac bitcoin-qt users.  It was way back on December 14, 2012 when the last version of bitcoin-qt (v0.7.2 with the later May 15 workaround) to work reliably on my MacBook Pro was released.  Every one of the many v0.8.x releases since has corrupted for me.  I've grown weary of reindexing the blockchain and testing out new builds.

Some of my experiences and setup are detailed in the original github link:
https://github.com/bitcoin/bitcoin/issues/2770

I'm sure that many new Mac users who have gotten into bitcoin this year have not had the same bullet-proof experience that I enjoyed when I first got started with the earlier versions.  It can be very frustrating, after all the complexity inherent to bitcoin, to have something not function the way it should when you're first getting started.  All financial software needs to be rock solid, especially with the sort of values that are being tossed around these days with our experimental little currency.

So with that in mind, I pleased to support this very worthwhile effort with a contribution of 0.48552571 BTC, so you can now update your bounty fund to an even 6.0 BTC + 200.2 LTC:

https://blockchain.info/tx/dc17a4e4b9ae40fdec9699936f965b57f0fe3c6fce8722033c56c30a11b01900

May bitcoin-qt have a long and illustrious future running on the world's fruitiest operating system, and may this bug meet a swift and permanent demise!!! 

[theymos]

The Bitcoin Forum pledges 4 BTC to this bounty, and we'll probably also put up an ad about this in forum ad round 103. This is more than I really want to spend on non-forum development issues from the forum's treasury, especially when the forum software situation is non-optimal, but this issue seems to be becoming serious despite the efforts of many experts over a long period of time, so I think that increasing the bounty amount is important.

[gmaxwell]

Wow! Theymos, thats awesome!

[jordan.dev]

Corrupted wallet.dat is not the issue of this particular bounty.

ok nm. than sorry I misread and though my issue was related.

[Boussac]

BitcoinQt 0.8.5 works like a charm on OSX 10.7.5 but crashes (with database corrupted message) on OSX 10.8.5 (brand new machine, no filevault involved nor any funky settings).

[cfields]

I have a theory, and a binary to try out for someone who can reproduce. I don't see the issue on my macbook.

Patch is here: https://github.com/theuni/bitcoin/commit/ac0d918bb49e0f8393b1e27662bc4c25113980b2
Binary is here for anyone who trusts me (you can, but of course you shouldn't) https://www.dropbox.com/s/r959wns6rm4xjsu/Bitcoin-Qt.dmg

Theory goes like this:

Currently leveldb's atomics use a gcc-style compiler barrier like:
asm volatile("" : : : "memory");

OSX has a native memory barrier that can be used instead. It's my understanding that on x86/x86_64, a real memory barrier is not needed, but only a flag to tell the compiler not to optimize+reorder store/loads. Looks to me like it's a mistake, and that the OSX function should be chosen instead.

In my tests, the asm hack was often completely optimized away, and loads/stores were reordered anyway. So I believe the barrier in current code is effectively a nop.

Since this is a build-time optim and not run-time, it does not explain why 10.8 is affected while 10.6 is. However, if atomics are busted, timings in other underlying functions (write/fsync/mmap/etc) would have a huge effect, so it seems reasonable that this could be the indirect cause.

[moderate]

I have a theory, and a binary to try out for someone who can reproduce. I don't see the issue on my macbook.

Patch is here: https://github.com/theuni/bitcoin/commit/ac0d918bb49e0f8393b1e27662bc4c25113980b2

Care to explain that your patch isn't just about moving a piece of code 20 lines above where it was previously ? I might be too tired, but that patch does nothing that wasn't being done already.

[cfields]

In the current code, OSX falls into this this case:

Code:

#elif defined(ARCH_CPU_X86_FAMILY) && defined(__GNUC__)

(Note that clang also defines __GNUC__)

I believe that's unintentional. Moving the "#elif defined(OS_MACOSX)" up causes the barrier to be

Code:

OSMemoryBarrier();

rather than

Code:

asm volatile("" : : : "memory");

[moderate]

In the current code, OSX falls into this this case:

Code:

#elif defined(ARCH_CPU_X86_FAMILY) && defined(__GNUC__)

(Note that clang also defines __GNUC__)

I believe that's unintentional. Moving the "#elif defined(OS_MACOSX)" up causes the barrier to be

Code:

OSMemoryBarrier();

rather than

Code:

asm volatile("" : : : "memory");

Oh, alright, that makes sense. You should bring it up to the leveldb people.

[wtogami]

Updated: December 11h, 2013
The following versions contain cfields' memory barrier patch and phantomcircuit's no-mmap patch.  Extensive testing indicates no new corruption happens on MacOS X.

Official Bitcoin 0.8.6
http://bitcoin.org/en/download

Bitcoin 0.8.5-OMG10
http://download1.rpmfusion.org/~warren/bitcoin-0.8.5-OMG10/macosx/
https://github.com/litecoin-project/bitcoinomg/commits/0.8.5-OMG10
This build contains all the same bug fixes as 0.8.6 above.  Includes additional features like Coin Control.

Litecoin 0.8.6.1
https://download.litecoin.org/litecoin-0.8.6.1/
https://litecoin.info/Upgrading_Litecoin#Upgrading_to_0.8.6.1
Please follow these directions to use Litecoin 0.8.6.1.

https://github.com/bitcoin/bitcoin/issues/2770
Read more here.

[cfields]

Upon further investigation, this does indeed seem to be unintended in leveldb.

Their atomics are borrowed from here (see the note in port/atomic_pointer.h): https://code.google.com/p/gperftools/source/browse/src/base/atomicops.h#102

It's clear there that mach/apple takes precedence over x86. The patch above fixes leveldb to match that behavior. Regardless of whether or not this fixes the db corruption for bitcoin, I'll take it upstream.

[Mike Hearn]

Wow. That's bad and would indeed explain the issues (or at least some of them). Apple really should not be claiming in the headers to be GCC compatible and then ignoring the GCC memory barrier idiom. That's not compatible at all. So technically this is an Apple bug, I'd say, but it's one we can work around pretty easily.

Very exciting that this has been discovered, great work cfields, worthy of a bounty award for sure. It's hard to imagine this not causing corruption. And it would also explain the otherwise unexplainable "self-fixing corruption" problem.

[cfields]

Thanks Mike. Though I must point out that my summation above is a theory, and certainly shouldn't be taken as fact. But the premise is basically this: We can't really know if the GCC idiom is supported on Apple's compiler-of-the-week, but it seems we can be sure that OSMemoryBarrier will work as intended, so there should be no harm in switching to it.

[misterbigg]

If the Bitcoin code can be refactored, or is already factored, to support easily changing the database "Backend" then you can try HyperLevelDB or RocksDB instead, they are both based off LevelDB and with identical APIs. This can be a good solution to try if problems re-surface with LevelDB.

[Basjee]

Noob here, how do I know if my "leveldb" is corrupted?

I have both Bitcoin client and Litecoin client running for approx. 10 days now (with some short breaks / macbook sleep time).

Both my wallets are showing the correct value.

2nd question: are mac os x users risking losing their coins by running the current clients?

Thanks :-)

[picobit]

You know it is corrupted by the client complaining that it is so.

And no, you do not lose coins.  The wallet is not affected (although you should always have a backup), the worst that could potentially happen is that your client "fortgets" you have the coins, or think you have too many (forgets you have spent some).  But rebuilding the database will fix that, since your bitcoins are not stored on your computer but in the blockchain.

[Basjee]

Thanks, that's clear!