Alert: chain fork caused by pre-0.8 clients dealing badly with large blocks

[lucb1e]

Apparently from what I read, this was handled extremely well. My respect to the devs.

Imho a "decentralized currency" shouldn't need handling from anyone.

I still don't understand what the issue was, but I'll formulate my question exactly at a later time. I (thought I) understand how the block chain works, it's not that the posts are too technical or anything, but I'll ask later.

[1714528691]

1714528691

[1714528691]

1714528691

[1714528691]

1714528691

[oakpacific]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Well said. Whatever the short-term fallout, this gives me *more* confidence in the long-term robustness, both in terms of system design (in that issues *can* be resolved quickly without much user impact), and also in the community to quickly arrive at consensus and execute the best solution.

Bitcoin will no doubt run into issues in the future, ranging from similar issues to outright attacks, and I like seeing this fire drill handled so effectively.

And the immediate willingness of people to cooperate when it matters.

[whitenight639]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Well said. Whatever the short-term fallout, this gives me *more* confidence in the long-term robustness, both in terms of system design (in that issues *can* be resolved quickly without much user impact), and also in the community to quickly arrive at consensus and execute the best solution.

Bitcoin will no doubt run into issues in the future, ranging from similar issues to outright attacks, and I like seeing this fire drill handled so effectively.

+1

Is this thread sufficiently old and boring that I can now start trolling it with pictures?

[Parazyd]

Thread should be closed.

[becoin]

why the hell is Deepbit only on 0.3.21 and Luke on 0.6.0?

I would like to ask this question as well.

[Ichthyo]

Apparently from what I read, this was handled extremely well. My respect to the devs.

Imho a "decentralized currency" shouldn't need handling from anyone.

The protocol whould have dealt with that situation just fine even if there was no "handling". Just in that case, we had gotten a (temporary) fork, not just some orphaned blocks, and this might have created more confusion. So the system was safe during that incident anyway, but the action taken by some community members helped to resolve it with the least hassle.

[evilpete]

It's interesting to see the two different sides of this debate. Is 0.8 the problem because it doesn't conform to the established rules of the system or are the older clients the problem because they use a crappy database which can't handle large blocks?

From what I can gather it seems to me like the database used by 0.8 is much more superior, so it will be used. But in order to patch 0.8 it seems like an artificial cap will be placed on the block size so that 0.8.1 is compatible with older versions.

The bug is that there has been an unknown BDB limit the in 0.7 and earlier code for quite some time.  It wasn't known about, so therefore measures weren't added to avoid provoking the bug in the older code.

0.8 is at "fault" for not being perfectly bug compatable.
0.7 is actually at fault for being broken..

The <= 0.7 problem is not "fixed".  The mining pool rollback just got everyone on the same blockchain again and bought some time to patch the older code.  The chain won't fork as long as > 51% of the hash pool is on 0.7 but that won't stop smaller operators generating valid blocks that cause trouble in the future.

The bdb bug has to be understood and fixed.

[Minor]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Well said. Whatever the short-term fallout, this gives me *more* confidence in the long-term robustness, both in terms of system design (in that issues *can* be resolved quickly without much user impact), and also in the community to quickly arrive at consensus and execute the best solution.

Bitcoin will no doubt run into issues in the future, ranging from similar issues to outright attacks, and I like seeing this fire drill handled so effectively.

And the immediate willingness of people to cooperate when it matters.

OK, so 25 blocks got orphaned, that's 25 times 25 BTC = 625BTC (or approx $29K at the time of the event) that miners worked hard for, and did not get paid for.
The miners on 0.8 did not have to switch back to 0.7, but they did so anyways, "for the greater good".

Are we going to compensate them in any way?
(Of course not, there is a limit to our willingness to cooperate. But I though it should still be mentioned.)

Disclaimer, I am a (minor) miner, but I only mine a few bit cents per day and mostly on Eligius so if I lost anything, it is not much.

[JoelKatz]

OK, so 25 blocks got orphaned, that's 25 times 25 BTC = 625BTC (or approx $29K at the time of the event) that miners worked hard for, and did not get paid for.

It's actually about half of that. About half of those people shouldn't have found blocks and only did because the person who should have mined the block was working on the other chain.

Think about it this way, for each block in the fork, someone mined the block in the 0.7 chain and someone mined it in the 0.8 chain. About half the time, the right person got it (because the forks merged when the work in them was roughly equal), and they still have it. About half the time, the wrong person got it and the right person lost it in an orphaned block.

[bitfreak!]

The bug is that there has been an unknown BDB limit the in 0.7 and earlier code for quite some time.

Not being aware of a certain property/feature of BDB does not make that feature a bug.

[JoelKatz]

The bug is that there has been an unknown BDB limit the in 0.7 and earlier code for quite some time.

Not being aware of a certain property/feature of BDB does not make that feature a bug.

It makes the code you write using it a bug if it relies on it not having that property/feature!

[Parazyd]

OK, so 25 blocks got orphaned, that's 25 times 25 BTC = 625BTC (or approx $29K at the time of the event) that miners worked hard for, and did not get paid for.
The miners on 0.8 did not have to switch back to 0.7, but they did so anyways, "for the greater good".

Are we going to compensate them in any way?
(Of course not, there is a limit to our willingness to cooperate. But I though it should still be mentioned.)

Disclaimer, I am a (minor) miner, but I only mine a few bit cents per day and mostly on Eligius so if I lost anything, it is not much.

Those 625 BTC seem like just a few people should've got them.
Divide those 625 by, let's say, 500 miners and see. It's actually not a lot.

If you were on Eligius, you didn't lose anything because Eligius is on 0.6.0 mainly.

[bitfreak!]

The bug is that there has been an unknown BDB limit the in 0.7 and earlier code for quite some time.

Not being aware of a certain property/feature of BDB does not make that feature a bug.

It makes the code you write using it a bug if it relies on it not having that property/feature!

Well I find it a bit odd that no one ever knew about this limit before. How can that be when we've always had people saying we need a larger block limit? How can that be when anyone can look at the blockchain and see there is a limit, and that anything past that limit gets rejected? Maybe I'm not understanding the problem correctly but this baffles me... how could the limit be unknown.

[Blowfeld]

To be accurate, it wasn't "the lead developer" who suggested raising the block size limit, it was me. I am a Bitcoin developer, but Gavin is the lead. So you can blame me if you like.

Apologies to "the lead developer".  I have amended my original post.  That portion of my original post was not intended so much as to cast blame on an individual as to point out a fundamental flaw in the process that permitted an "off-the-cuff" remark to be taken seriously (and almost simultaneously) by many miners.  I am a software developer, and I know what it is like to release upgrades to large legacy projects.  [Yuck, usually.]  Assuming you guys follow *some* process, a planned line-item in a scheduled release to increased the blocksize should have led to significant testing of that line-item.

Even if the problem hadn't been found during testing, if miners had gradually rolled out the change to 0.8 (with a built-in bigger block-size limit), then when the problem cropped up, as long as 51% of the mining power hadn't been on the new "big block 0.8 release", there would not have been a hard fork.  (Early adopters of "big block 0.8 release" would have found they were generating blocks that quickly became orphans.  There would have been no panic and no great urgency to solve the problem by fiat.)

[I'm not trying to second-guess the developers.  It's always easy to see with 20/20 hindsight.  But there should be a process.  And the process should be followed.]

There are two lessons here:
1.  Avoid off-the-cuff suggestions unless you *know* the impact is minor.
2.  Where possible, roll out the tested releases gradually -- especially to mining pools, who have become so centralized, a few of them can easily tip the scales to 51%.

[becoin]

Is this issue fixed or do we still have 2 blockchains running?

Fixed? U call 51% attack organized by Bitcoin Foundation "a fix"? :facepalm:

Good question.

[niner]

7-day Bitcoin/USD price chart. The snapshot was taken 12-Mar-2013 09:00 UTC (a few hours after the blockchain fork incident commenced).

Typically you'd need a longer duration to put price volatility into perspective.  In this case 7 days seems to suffice.

[Mike Hearn]

Even if the problem hadn't been found during testing, if miners had gradually rolled out the change to 0.8 (with a built-in bigger block-size limit), then when the problem cropped up, as long as 51% of the mining power hadn't been on the new "big block 0.8 release", there would not have been a hard fork.

The problem that cropped up is a hard fork, so by definition it would have happened. It's clear now that a hard fork is unavoidable. The only question is when does it happen and who will lose out because of it.

Also, I'm afraid it's very easy to say "just test for longer" but the reason we started generating larger blocks is that we ran out of time. We ran out of block space and transactions started stacking up and not confirming (ie, Bitcoin broke for a lot of its users). Rolling back to the smaller blocks simply puts us out of the frying pan and back into the fire.

We will have to roll forward to 0.8 ASAP. There isn't any choice.

[caveden]

OK, so 25 blocks got orphaned, that's 25 times 25 BTC = 625BTC (or approx $29K at the time of the event) that miners worked hard for, and did not get paid for.

It's actually about half of that. About half of those people shouldn't have found blocks and only did because the person who should have mined the block was working on the other chain.

Regardless, he does have a good point in that these miners which were on 0.8 did make a monetary sacrifice, while it wasn't them which were using a buggy software in the first place, but precisely those which were on the older version.

I believe those pools which got their blocks in the 0.7 after the fork should donate perhaps half of the bitcoins they've generated up until the end of the fork to those who got orphaned for mining in 0.8 and then downgraded.
It would be a noble display of gratitude, since those in 0.7 were using "bad software" and those in 0.8 agreed on downgrading just to keep the network as a whole.

[oakpacific]

Anyone can point me to anything about that 2010 fork?

[ItsDom]

Anyone can point me to anything about that 2010 fork?

This one?

https://en.bitcoin.it/wiki/Common_Vulnerabilities_and_Exposures#CVE-2010-5139