Alert: chain fork caused by pre-0.8 clients dealing badly with large blocks

[J.Socal]

what does (orphaned) mean

[Blowfeld]

It was fascinating to watch the higher block chain continue to build.  Was it ignorance by some large miners?  Or was it an intentional attempt to keep the blockchain forked.

No, it was night. I woke up at 3am thanks to one guy who called me over skype. I have a lot of monitoring scripts which trigger SMS alert when something bad happen on pool, but there was almost no chance to catch this, because the bug wasn't in bitcoind 0.8 used by the pool, but in previous (and still widely spread) bitcoind 0.7.

I include that in "ignorance".  You were ignorant of the situation, else you would have responded.     I guess I assumed there would be some sort of "friend net" by which the major mining pools operators would hear of trouble demanding their attention.  Such is the nature of a distributed, volunteer project, eh.

My mining client is configured to print the message that goes to the server when it finds a share.  From the hexadecimal code in the message, I could see the block header's Previous Block field, and I could see my pool was mining the wrong fork.  The next pool I switched to was also mining the wrong fork.  The third pool I tried was mining the correct (now current) fork.

[thefiniteidea]

what does (orphaned) mean

something annoying

[Blowfeld]

It *was* indirectly because of the size of the block.  Even at 166 bytes each (or whatever the minimum size of transaction), a 250K block cannot contain 1700+ transactions.  And a number of transactions that exceeds the BDB configuration is believed to be the root of the problem.  I know hindsight is 20/20, but I will give the developers credit and assume testing all extremes from 1 really big transaction to many really tiny transactions probably would have been in a formal release cycle.  No such testing was done, chiefly because this was an off the cuff suggestion, not a formal release.

The problem is with the old version, and is a previously unknown bug. Are you saying the developers should have thought to test the old version for an unknown bug before releasing the new version?

It is quite normal, when developing a complex system, that a new release must maintain compatibility with some of the "bugs" that were present in the old release.  I believe the developers of bitcoin follow that philosophy.  By definition, a *unexpected* difference in behavior between the old client and the new client is a bug in the new client.

I believe if the developers had decided to raise the blocksize before releasing 0.8, they may have found this bug during their testing.  The problem is that NO FORMAL TESTING was performed before the lead developer's suggestion was taken to heart by several mining pools.  Don't ask your "customers" to do X, unless you have tested that doing X causes no harm.  As someone else already pointed out on this thread, the development team was well aware that changing to a new database was a possible source of trouble.  I will give the developers credit and assume they tested thoroughly.  So I repeat, if the blocksize had been raised *before* the official release of 0.8, at least there would have been a chance of detecting the difference in behavior, and thus a chance of avoiding this fork.

[GideonGono]

How can we prevent this from happening in the future and is there any other way the average user can contribute to the robustness of bitcoin?

[mp420]

We need a scheduled hardfork eventually to solve this one for good, yes?

A suggestion: include some of the wishlisted changes in the upcoming fork. Hard forks should be few and far between (ideally: none), so the chance to have one should be exploited to the max.

[The-Real-Link]

Just synched my 0.7.0 client (to the latest block height) and there is no longer any DB error displayed.  I again see a green bar, the blocks downloaded and the checkmark is good.

All appears to be in order.

[J.Socal]

  all good here also.Coins finally confirmed.

[evilpete]

what does (orphaned) mean

Simplifying a bit:

Bitcoin works at two levels.  There's a peer-to-peer network that floods instant transactions from node to node.  These are sanity checked and redistributed to the other peers that haven't seen it yet.  They're fast but are only kept in memory.

Miners do the number crunching to form blocks.  Blocks form the permanent records.  These blocks are also distributed over the peer-to-peer network and are integrated into the permanent database.  Miners are paid by the network itself for doing this.  Miners do extensive validation and permanently record the transactions above.

If two miners both try to produce a block at the exact same time, only one can survive.  The "fast" transactions are recorded in both blocks so nothing's lost.  However, the losing miner doesn't get paid.

What happened tonight is a minority (but significant) number of miners and end users with <= 0.7 rejected a large but valid block and then went off on their own fork.

There were two solutions to this, either:
1) Everyone must upgrade to 0.8+.  Everyone.  (Or upgrade to a fixed 0.7, 0.6, 0.3 etc and reindex)
2) Move enough of the 0.8 miners to the 0.7 blockchain fork and cause the network to reorganize.  That makes everyone converge on the same chain again.

The upshot is that the miners of the previously winning fork gave up all their miner fees for the good of the community in order to get everyone back on the same page.

Code:

received block 000000000000016924f85069603be8164578eedf113f44d60bf0438cba047c7f
REORGANIZE: Disconnect 25 blocks; 0000000000000366ce98ca28338900094e8cbf445776253181749f782546d006..00000000000000df96f272c3b1e9dd15272b55750966cbd239219b94756c73ec
REORGANIZE: Connect 26 blocks; 0000000000000366ce98ca28338900094e8cbf445776253181749f782546d006..000000000000016924f85069603be8164578eedf113f44d60bf0438cba047c7f

[Nancarrow]

IN 3 hours the price of bitcoins will be around 14 bucks. Mark my words

Why thankyou, that's exactly what I'd like to do to your words at this juncture.

Well. There's still 90 minutes left. But I think you'll probably still be an idiot in 90 minutes.

[crazy_rabbit]

How can we prevent this from happening in the future and is there any other way the average user can contribute to the robustness of bitcoin?

Well the fact that it was solved so fast seems to imply that the system is pretty robust.

[hgmichna]

Atlas warned about database change some time ago. Original discussion here https://bitcointalk.org/index.php?topic=119566.0;all

Remarkable! And see how Atlas was attacked for his posting, which has now turned out to be right on target.

[Mike Hearn]

To be accurate, it wasn't "the lead developer" who suggested raising the block size limit, it was me. I am a Bitcoin developer, but Gavin is the lead. So you can blame me if you like.

Along with raising the size limit, I have also been strongly urging people to upgrade to 0.8 for some time now. In fact, I'm the one who originally benchmarked and wrote the code to use LevelDB - the mysterious problems we've had with bdb being one of my motivations (the other was performance).

Ultimately, this problem is an old bug that has already been resolved through a large effort to get us off bdb. It's unfortunate that so many miners have NOT heeded the call to upgrade and we had to learn another one of bdbs inadequacies the hard way, along with rolling back to keep those slow miners online. Every miner will have to upgrade sooner or later, that is just inevitable.

Atlas's post was not prescient or insightful. We knew when we made the switch to LevelDB that it introduced the risk of a hard fork. This is not something that had to be pointed out by a banned forum poster. However - as you can see - there was no alternative. BDB has serious, fundamental problems that cannot be fixed. Upgrading to LevelDB is the solution.

[Wekkel]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

[Parazyd]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Bitcoin stays protected. Nothing could actually happen. If an attack was made, the algorithm would be changed.
If something like this happens, it fixes.

[crazy_rabbit]

How can we prevent this from happening in the future and is there any other way the average user can contribute to the robustness of bitcoin?

Keep your software updated.

[AngelusWebDesign]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Well said.

Can you imagine how some large bank would have handled this? A bunch of press conferences, perception management, BS, lies, you name it.

Sure, it looks ugly when the gritty details are thrown out there and people get scared, misunderstand what's going on, etc. but that's the price of transparency.

We can handle a few inconveniences (don't use bitcoin for a couple hours, etc.) as long as we are allowed to know what the heck's going on.

It's not much to ask for, really, but if you think about it who besides Bitcoin gives that to us?

[Wekkel]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Well said.

Can you imagine how some large bank would have handled this? A bunch of press conferences, perception management, BS, lies, you name it.

Sure, it looks ugly when the gritty details are thrown out there and people get scared, misunderstand what's going on, etc. but that's the price of transparency.

If we would treat our credit crisis in the regular markets like this, the crisis of 2008 would already have been forgotten (except for the losers).

[bitfreak!]

It's interesting to see the two different sides of this debate. Is 0.8 the problem because it doesn't conform to the established rules of the system or are the older clients the problem because they use a crappy database which can't handle large blocks?

From what I can gather it seems to me like the database used by 0.8 is much more superior, so it will be used. But in order to patch 0.8 it seems like an artificial cap will be placed on the block size so that 0.8.1 is compatible with older versions.

Obviously this was the best solution to apply and I'm glad the hard fork route was not taken. Such a route needs to be properly planned and even then it will be difficult to coordinate a change such as an increase in the max block size.

But do we really need to increase the max size of blocks? I don't know all the technical details behind bitcoin, but it seems to me that if 1 block is being saved into the blockchain every 10 minutes then the max size should be limited to something relatively small.

[Melbustus]

Gentlemen, you just watched finance in the 21st century.

*complete transparency
*community driven
*solution driven
*results

Bitcoin just gained a lot more confidence.

Well said. Whatever the short-term fallout, this gives me *more* confidence in the long-term robustness, both in terms of system design (in that issues *can* be resolved quickly without much user impact), and also in the community to quickly arrive at consensus and execute the best solution.

Bitcoin will no doubt run into issues in the future, ranging from similar issues to outright attacks, and I like seeing this fire drill handled so effectively.