Understanding why the call to rollback to v0.7 was made.

[DeathAndTaxes]

There seems to be some confusion on why a call to rollback to v0.7 was made so hopefully this will dispel some myth.

What happened?
The network hard forked on block 225433.  The block produced by a v0.8 node was rejected by at least some v0.7 nodes.  Contrary to initial claims this was more complex than just "the block was too big".  Blocks right up to the 1MB limit have been created and relayed without issue on testnet.  The BDB used by v0.7 failed to validate the block 225433 due to an inability to handle the number of tx inputs (not to be confused with block size or number of transactions) and rejected it.  This was an undocumented issue with v0.7 ("bug").

At this point a hard fork existed.  The mining nodes running v0.8 built upon the block 225433 and extended their chain.  The nodes running v0.7 rejected the block 225433 and eventually produced an alternative and incompatible block "225433a".   Since the deviation was uni-direction in theory v0.7 could eventually surpass v0.8 chain and re-unify the network however v0.8 chain had a significant majority of hashing power and that never happened.  By the time the issue was being analyzed v0.8 was over 5 blocks ahead and pulling further ahead with time.

Why was the fork time critical?
The users on v0.7 fork would never rejoin the network.  If all those users shutdown their clients and didn't accept any transactions there was no real risk for them.  However in time as hashing power fled the v0.7 fork remaining nodes would become more and more vulnerable to a variety of attacks.  Not just from a potential 51% attack, but also from accepting generated coins which were valid on the v0.8 blocks and even non hashpower related double spends. 

Why not just let "most hashing power wins"?
The "most hashing power wins" concept is used to solve splits/reorgs not hard forks.  It would create a bad precedent.  Essentially miners on v0.8 node would be leaving potential v0.7 users vulnerable to attack to save a few block rewards.  Since non-mining nodes (users) running v0.7 would never accept the #225433 block produced by v0.8 they would have no method to rejoin the longest chain short of upgrading.  v0.8 users (non miners) would accept blocks produced by v0.7.  By having a few miners downgrade to v0.7 it "reunified" the network in the shortest period of time with the smallest number of changes required.

What if developers/pool operators couldn't reach a consensus?  Would this have destroyed Bitcoin?
No.  The v0.8 chain would continue.  v0.7 users would be urged to immediately stop all transactions and upgrade.  Users on v0.7 clients would remain (increasingly) vulnerable to a variety of attacks if actively engaging in transactions until they upgraded.  Eventually all users would upgrade and the network would have continued with the v0.7 branch being an oprhaned sidechain.  The potential threat to active v0.7 users would make this less than optimal but the network as a whole was never in any danger of failure.

Was there a real risk of 51% attack on the major chain?
Not really.  The v0.8 chain had roughly 60% of the hashing power so to double spend that chain would require more hashing power than 60% of the Bitcoin network.  While 60% is less than 100% it is still a staggering amount of hashing power.  Any malicous actor building out a network that large wouldn't stop and wait for the remote chance that 60% would be enough.  Anyone with that amount of time and resources would be planning to outbuild the full network and attack.

What risks were there for users on v0.7 fork?
The largest risk would be from a 51% attack.  Initially the v0.7 fork was relatively well protected but had it been abandoned in favor of extending the v0.8 chain that hash power would have dropped over time.  The longer a user remained active on the v0.7 network the more danger they would be in.  Eventually hashing power would fall to a level where even a small pool could have easily executed double spends.  

Users on the v0.7 would eventually (100 blocks after the fork) be at risk of bring double spent by newly generated coins.  The coins would confirm but once they upgraded to v0.8 that transaction would never exist (as the block producing it also never existed).  v0.7 users who heeded the alert key warning and stopped all transactions were never in any significant danger.  

Lastly a smart attacker with good timing could have executed a "no hashing power" double spend against vulnerable v0.7 users (who continued to engage in active transactions despite the alert key warning).  A simplified version of this attack would work like this.  As hashing power on v0.7 fork fell, the block generation rate would slow and the number of transactions in the memory pool would increase.  An attacker could exploit this differential by creating a large number of low/ no fee transactions and wait for some of them to be accepted into v0.8 blocks.  The Bitcoin network will eventually forget about unconfirmed transactions as they get old enough to fall out of the memory pool.  Normally a client will rebroadcast an unconfirmed transaction periodically but in this case the attacker's client would see the transaction as confirmed.  The attacker could speed up this process but producing a large number of unrelated transactions.  When the tx in v0.8 blocks were dropped from the v0.7 memory pool the attack could double spend victims on the v0.7 network.

[Come-from-Beyond]

Is there any transaction from abandoned 0.8 chain that was not included into legit 0.7 chain?

[DeathAndTaxes]

Is there any transaction from abandoned 0.8 chain that was not included into legit 0.7 chain?

Coinbase transactions (block rewards) from the roughly 25 orphaned blocks in the v0.8 chain.  It was important to resolve this before either chain was more than 100 blocks past the fork as then those coins would be spendable which would have been "chaotic".

[Ivica]

Good post.

[RoadStress]

Thank you for the post.Very insightful!

[nevafuse]

Is there any transaction from abandoned 0.8 chain that was not included into legit 0.7 chain?

https://bitcointalk.org/index.php?topic=152348.0

[Come-from-Beyond]

Is there any transaction from abandoned 0.8 chain that was not included into legit 0.7 chain?

https://bitcointalk.org/index.php?topic=152348.0

That's very bad.

[Ivica]

Is there any transaction from abandoned 0.8 chain that was not included into legit 0.7 chain?

https://bitcointalk.org/index.php?topic=152348.0

That's very bad.

There shouldn't be such transactions, there can be fault at merchant end accepting double-spend transaction during fork time.

[dserrano5]

It was important to resolve this before either chain was more than 100 blocks past the fork as then those coins would be spendable which would have been "chaotic".

Why? After 120 blocks, what makes those coins different from any other unspent output?

[Tesla71]

I think its because you have to wait for 120 confimations on coins that where mined before they add up in your wallet

[Sukrim]

It was important to resolve this before either chain was more than 100 blocks past the fork as then those coins would be spendable which would have been "chaotic".

Why? After 120 blocks, what makes those coins different from any other unspent output?

The limit to spend newly mined coins is actually 100 blocks (120 in the UI) and since they are coming "out of thin air" they would be viewed as nonexistant as in "never been there" in a different chain.

[proudhon]

There seems to be some confusion on why a call to rollback to v0.7 was made so hopefully this will dispel some myth.

What happened?
The network hard forked on block 225433.  The block produced by a v0.8 node was rejected by at least some v0.7 nodes.  Contrary to initial claims this was more complex than just "the block was too big".  Blocks right up to the 1MB limit have been created and relayed without issue on testnet.  The BDB used by v0.7 failed to validate the block 225433 due to an inability to handle the number of tx inputs (not to be confused with block size or number of transactions) and rejected it.  This was an undocumented issue with v0.7 ("bug").

At this point a hard fork existed.  The mining nodes running v0.8 built upon the block 225433 and extended their chain.  The nodes running v0.7 rejected the block 225433 and eventually produced an alternative and incompatible block "225433a".   Since the deviation was uni-direction in theory v0.7 could eventually surpass v0.8 chain and re-unify the network however v0.8 chain had a significant majority of hashing power and that never happened.  By the time the issue was being analyzed v0.8 was over 5 blocks ahead and pulling further ahead with time.

Why was the fork time critical?
The users on v0.7 fork would never rejoin the network.  If all those users shutdown their clients and didn't accept any transactions there was no real risk for them.  However in time as hashing power fled the v0.7 fork remaining nodes would become more and more vulnerable to a variety of attacks.  Not just from a potential 51% attack, but also from accepting generated coins which were valid on the v0.8 blocks and even non hashpower related double spends. 

Why not just let "most hashing power wins"?
The "most hashing power wins" concept is used to solve splits/reorgs not hard forks.  It would create a bad precedent.  Essentially miners on v0.8 node would be leaving potential v0.7 users vulnerable to attack to save a few block rewards.  Since non-mining nodes (users) running v0.7 would never accept the #225433 block produced by v0.8 they would have no method to rejoin the longest chain short of upgrading.  v0.8 users (non miners) would accept blocks produced by v0.7.  By having a few miners downgrade to v0.7 it "reunified" the network in the shortest period of time with the smallest number of changes required.

What if developers/pool operators couldn't reach a consensus?  Would this have destroyed Bitcoin?
No.  The v0.8 chain would continue.  v0.7 users would be urged to immediately stop all transactions and upgrade.  Users on v0.7 clients would remain (increasingly) vulnerable to a variety of attacks if actively engaging in transactions until they upgraded.  Eventually all users would upgrade and the network would have continued with the v0.7 branch being an oprhaned sidechain.  The potential threat to active v0.7 users would make this less than optimal but the network as a whole was never in any danger of failure.

Was there a real risk of 51% attack on the major chain?
Not really.  The v0.8 chain had roughly 60% of the hashing power so to double spend that chain would require more hashing power than 60% of the Bitcoin network.  While 60% is less than 100% it is still a staggering amount of hashing power.  Any malicous actor building out a network that large wouldn't stop and wait for the remote chance that 60% would be enough.  Anyone with that amount of time and resources would be planning to outbuild the full network and attack.

What risks were there for users on v0.7 fork?
The largest risk would be from a 51% attack.  Initially the v0.7 fork was relatively well protected but had it been abandoned in favor of extending the v0.8 chain that hash power would have dropped over time.  The longer a user remained active on the v0.7 network the more danger they would be in.  Eventually hashing power would fall to a level where even a small pool could have easily executed double spends.  

Users on the v0.7 would eventually (100 blocks after the fork) be at risk of bring double spent by newly generated coins.  The coins would confirm but once they upgraded to v0.8 that transaction would never exist (as the block producing it also never existed).  v0.7 users who heeded the alert key warning and stopped all transactions were never in any significant danger.  

Lastly a smart attacker with good timing could have executed a "no hashing power" double spend against vulnerable v0.7 users (who continued to engage in active transactions despite the alert key warning).  A simplified version of this attack would work like this.  As hashing power on v0.7 fork fell, the block generation rate would slow and the number of transactions in the memory pool would increase.  An attacker could exploit this differential by creating a large number of low/ no fee transactions and wait for some of them to be accepted into v0.8 blocks.  The Bitcoin network will eventually forget about unconfirmed transactions as they get old enough to fall out of the memory pool.  Normally a client will rebroadcast an unconfirmed transaction periodically but in this case the attacker's client would see the transaction as confirmed.  The attacker could speed up this process but producing a large number of unrelated transactions.  When the tx in v0.8 blocks were dropped from the v0.7 memory pool the attack could double spend victims on the v0.7 network.

Fantastic post!  Thanks DandT!  Somebody else on Reddit also offered a pretty good non-technical explanation.  This was a really cool event to watch.

[Ekaros]

Since when was this bug in software? So how long did the testnet have time to discover it?

[Littleshop]

Since when was this bug in software? So how long did the testnet have time to discover it?

While a bug is easy to spot in hindsight, this but looks like it would have not easily emerged even in testnet unless testnet was being used quite hard.  Yes, I know complex scripts to can do that and if a series of complex to simulate transactions are not out there, maybe they should be.

[Stephen Gornick]

There shouldn't be such transactions, there can be fault at merchant end accepting double-spend transaction during fork time.

When the fork was first being looked into, at March 12 2013 and at 00:03 AM the main net at the time (mined by v0.8 clients) was at block 225439, yet other clients were stuck on the fork at the time (mined by v0.7 and prior clients) was at 225431.

00:00   sipa   ;;bc,blocks
00:00   gribble   225431

00:01   sipa   but it seems blockexplorer is stuck too...
00:01   sipa   as i'm on 225439

 - http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/03/12

So even before anyone knew for sure that a fork was underway, there could have been transactions with six confirmations on the v0.8 side.  If for whatever reason those transactions didn't also already get included in blocks on the v0.7 side, there was the opportunity to perform a race attack to double spend the transaction that had already been included on the v0.8 side.

We now know that exact scenario is what happened with the transfer to OKPay that is being claimed to have been successfully double spent.  Though in that instance, it was after the alert went out that OKPay still processed the deposit as being valid.  So that specific incident could have been prevented had they halted processing once the alert went out, but there still was a window between when confirmations would occur since the fork started and when the alert eventually went out.

[Stephen Gornick]

This was an undocumented issue with v0.7 ("bug").

And the term v0.7 is being misused here.  It really is a pre v0.8 bug, right?  i.e., it has existed since day one ... a configuration setting for BDB that has been that way since v0.3 at least, if I read correctly.

Since when was this bug in software? So how long did the testnet have time to discover it?

See above.  It was a scenario (a transaction requiring 10,000 locks, or about 5,000 inputs) that hadn't been tested (again, from what conversations I've seen on it).

[Ekaros]

This was an undocumented issue with v0.7 ("bug").

And the term v0.7 is being misused here.  It really is a pre v0.8 bug, right?  i.e., it has existed since day one ... a configuration setting for BDB that has been that way since v0.3 at least, if I read correctly.

Since when was this bug in software? So how long did the testnet have time to discover it?

See above.  It was a scenario (a transaction requiring 10,000 locks, or about 5,000 inputs) that hadn't been tested (again, from what conversations I've seen on it).

Thanks.

Still, if it is that old I wonder how well scenarios with really large scale user base and amount of daily transactions is tested. As some people prophecises about global use. But wouldn't such day also mean millions to billions users and thus very large amount transactions per block... I have to look into this myself when I have time...

[AndyRossy]

There shouldn't be such transactions, there can be fault at merchant end accepting double-spend transaction during fork time.

When the fork was first being looked into, at March 12 2013 and at 00:03 AM the main net at the time (mined by v0.8 clients) was at block 225439, yet other clients were stuck on the fork at the time (mined by v0.7 and prior clients) was at 225431.

00:00   sipa   ;;bc,blocks
00:00   gribble   225431

00:01   sipa   but it seems blockexplorer is stuck too...
00:01   sipa   as i'm on 225439

 - http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/03/12

So even before anyone knew for sure that a fork was underway, there could have been transactions with six confirmations on the v0.8 side.  If for whatever reason those transactions didn't also already get included in blocks on the v0.7 side, there was the opportunity to perform a race attack to double spend the transaction that had already been included on the v0.8 side.

We now know that exact scenario is what happened with the transfer to OKPay that is being claimed to have been successfully double spent.  Though in that instance, it was after the alert went out that OKPay still processed the deposit as being valid.  So that specific incident could have been prevented had they halted processing once the alert went out, but there still was a window between when confirmations would occur since the fork started and when the alert eventually went out.

Not sure if you're really trying to place any blame on the merchant.... errr

[cypherdoc]

See above.  It was a scenario (a transaction requiring 10,000 locks, or about 5,000 inputs) that hadn't been tested (again, from what conversations I've seen on it).

what are locks?

[Stephen Gornick]

Not sure if you're really trying to place any blame on the merchant.... errr

No, I'm not.

It might be common sense / generally accepted protocol that when an alert is received by the bitcoin client to basically halt all payment processing until the problem is understood.

In the case of the March 12th fork, the alert didn't go out until after there had already been six or more confirmations on transactions.  There's nothing an OKPay or other merchant could have done if this transaction had occurred a few blocks earlier (after the fork started but before the alert was released).   The merchant can't know at five o'clock that at six o'clock miners will abandon your chain and instead help a fork become the longest chain.