Alert: chain fork caused by pre-0.8 clients dealing badly with large blocks

[gyverlb]

So, a regular user who isn't mining should simply have waited a couple of hours and the issue of the fork would be resolved by the network... fine, great...

But what about miners? How is this going to be resolved for miners going forward?
Reddit said that BTCGuild and their ilk have changed back to 0.7, that's great for them but what about people using p2pool? p2pool interfaces with your BC client and as such it would mean everyone with p2pool has to downgrade to 0.7...

Are there plans for a 0.8.1 which resolves this or what?

Anyone knows the answer?

Edit your bitcoin.conf:

Code:

#Maximum size, in bytes, of blocks you create, large blocks can be rejected by the standard bitcoin node 0.7 and maybe earlier:
# 250000 is standard and safe for the network, 100000 lowers the latency of getblocktemplate calls which is good for p2pool's efficiency
blockmaxsize=100000

#How many bytes of the block should be dedicated to high-priority transactions,                                                                                                 
#included regardless of the fees they pay                                                                                                                                 
blockprioritysize=20000

#Minimum block size you want to create; block will be filled with free transactions                                                                                       
#until there are no more or the block reaches this size:                                                                                                                  
blockminsize=10000

#Fee-per-kilobyte amount (in BTC) considered the same as "free"                                                                                                                    
#Be careful setting this: if you set it to zero then                                                                                                                      
#a transaction spammer can cheaply fill blocks using                                                                                                                      
#1-satoshi-fee transactions. It should be set above the real                                                                                                              
#cost to you of processing a transaction.                                                                                                                                 
mintxfee=0.0005

Edit: oh, and restart your bitcoind process to apply changes.

[GideonGono]

How can we prevent this from happening in the future and is there any other way the average user can contribute to the robustness of bitcoin?

I remember reading somewhere Gavin Andresen saying that they need more "test nodes" or something like that? How can I set this up and donate my resources to the network?

[taltamir]

So, a regular user who isn't mining should simply have waited a couple of hours and the issue of the fork would be resolved by the network... fine, great...

But what about miners? How is this going to be resolved for miners going forward?
Reddit said that BTCGuild and their ilk have changed back to 0.7, that's great for them but what about people using p2pool? p2pool interfaces with your BC client and as such it would mean everyone with p2pool has to downgrade to 0.7...

Are there plans for a 0.8.1 which resolves this or what?

Anyone knows the answer?

Edit your bitcoin.conf:

Code:

#Maximum size, in bytes, of blocks you create, large blocks can be rejected by the standard bitcoin node 0.7 and maybe earlier:
# 250000 is standard and safe for the network, 100000 lowers the latency of getblocktemplate calls which is good for p2pool's efficiency
blockmaxsize=100000

#How many bytes of the block should be dedicated to high-priority transactions,                                                                                                 
#included regardless of the fees they pay                                                                                                                                 
blockprioritysize=20000

#Minimum block size you want to create; block will be filled with free transactions                                                                                       
#until there are no more or the block reaches this size:                                                                                                                  
blockminsize=10000

#Fee-per-kilobyte amount (in BTC) considered the same as "free"                                                                                                                    
#Be careful setting this: if you set it to zero then                                                                                                                      
#a transaction spammer can cheaply fill blocks using                                                                                                                      
#1-satoshi-fee transactions. It should be set above the real                                                                                                              
#cost to you of processing a transaction.                                                                                                                                 
mintxfee=0.0005

Edit: oh, and restart your bitcoind process to apply changes.

Thank you very much.
This should be stickied

[gyverlb]

I think I'll post a detailed guide about p2pool in the following days in order to promote it's adoption (my configuration is giving me satisfaction and I see many people who don't get as good results as mine).

There are some things hard to find by searching the forums and some urban legend to address too. There are cases where p2pool isn't a good solution that should be clearly reminded (BFL singles, some network setups, ...) so that people don't waste time with it when they'll clearly lose income instead of gaining some vs ordinary pools.

[Come-from-Beyond]

I think I'll post a detailed guide about p2pool in the following days in order to promote it's adoption.

Btw, If most of the miners were on p2pool using 0.8, would it be possible to resolve 0.8->0.7 issue?

[evilpete]

The bug is that in all versions of bitcoin prior to 0.8 the maximum number of transactions that can fit in a block is not universally definable, because the limit varies based on the exact hardware and software configuration a particular node happens to be running.

It's possible for 0.7 nodes to produce valid blocks that other 0.7 nodes can't process.

And yet the current solution is "everyone use 0.7 not 0.8". BTC and all other mining guilds switched back to 0.7 to prevent a hard fork.
How can you migrate off of the buggy version when attempting to migrate off of it causes a fork?

No, not true.

What we need is:
1) end users to get off 0.7/below and need to upgrade.
2) merchants  to get off 0.7/below and need to upgrade.
3) when people run a miner program as part of a 3rd party pool, still need to upgrade.  Their pool runs 0.7, not the person running the miner.
4) a couple of large pools stay on 0.7 to buy time for #1/#2/#3 to happen.

"Upgrade" includes upgrading their bitcoind/bitcoin-qt etc to 0.8+ OR setting the DB_CONFIG entries mentioned above.  The DB_CONFIG changes will fix the problems *that we know about* in 0.7 and earlier. Naturally there's still potentially more problems in 0.7 lurking.

However, those pools running this 0.7 bitcoind cannot "fix" the DB_CONFIG settings - if they did they might as well just run 0.8.

Once the vast majority of nodes are "fixed" (there's only 10,000 of them), then the big pools can go back to 0.8.  (nodes: http://luke.dashjr.org/programs/bitcoin/files/charts/branches.html)

The reason 0.8 can't be taught to reject valid blocks that will break 0.7 and earlier is that there's no exact number.  It's an internal artifact that depends on the run-time state of an individual node.  For example, it has been observed by some folks that simply stopping/starting bitcoind 0.7 will cause it to validate and accept the block their copy had previously crashed on.

The "pools on 0.7" thing is just a tactic to by time for people to upgrade.  It is not a solution, and it won't last.

Prediction: the moment sufficient hashpower switches over, somebody will generate another BDB-buster.  It probably won't be a big pool, but somebody who wants the transaction fees will do it.

[Luke-Jr]

I think I'll post a detailed guide about p2pool in the following days in order to promote it's adoption (my configuration is giving me satisfaction and I see many people who don't get as good results as mine).

There are some things hard to find by searching the forums and some urban legend to address too. There are cases where p2pool isn't a good solution that should be clearly reminded (BFL singles, some network setups, ...) so that people don't waste time with it when they'll clearly lose income instead of gaining some vs ordinary pools.

Don't forget to mention that p2pool was basically broken by all this :p

[marcus_of_augustus]

Naturally there's still potentially more problems in 0.7 lurking.

and in 0.8 .... what's good for the goose .... 0.7 has been running for much longer in production than 0.8.

[taltamir]

The bug is that in all versions of bitcoin prior to 0.8 the maximum number of transactions that can fit in a block is not universally definable, because the limit varies based on the exact hardware and software configuration a particular node happens to be running.

It's possible for 0.7 nodes to produce valid blocks that other 0.7 nodes can't process.

And yet the current solution is "everyone use 0.7 not 0.8". BTC and all other mining guilds switched back to 0.7 to prevent a hard fork.
How can you migrate off of the buggy version when attempting to migrate off of it causes a fork?

No, not true.

What we need is:
1) end users to get off 0.7/below and need to upgrade.
2) merchants  to get off 0.7/below and need to upgrade.
3) when people run a miner program as part of a 3rd party pool, still need to upgrade.  Their pool runs 0.7, not the person running the miner.
4) a couple of large pools stay on 0.7 to buy time for #1/#2/#3 to happen.

"Upgrade" includes upgrading their bitcoind/bitcoin-qt etc to 0.8+ OR setting the DB_CONFIG entries mentioned above.  The DB_CONFIG changes will fix the problems *that we know about* in 0.7 and earlier. Naturally there's still potentially more problems in 0.7 lurking.

However, those pools running this 0.7 bitcoind cannot "fix" the DB_CONFIG settings - if they did they might as well just run 0.8.

Once the vast majority of nodes are "fixed" (there's only 10,000 of them), then the big pools can go back to 0.8.  (nodes: http://luke.dashjr.org/programs/bitcoin/files/charts/branches.html)

The reason 0.8 can't be taught to reject valid blocks that will break 0.7 and earlier is that there's no exact number.  It's an internal artifact that depends on the run-time state of an individual node.  For example, it has been observed by some folks that simply stopping/starting bitcoind 0.7 will cause it to validate and accept the block their copy had previously crashed on.

The "pools on 0.7" thing is just a tactic to by time for people to upgrade.  It is not a solution, and it won't last.

Prediction: the moment sufficient hashpower switches over, somebody will generate another BDB-buster.  It probably won't be a big pool, but somebody who wants the transaction fees will do it.

0.8 doesn't need to be taught to reject anything, the problem is 0.7 and earlier reject valid blocks due to a bug in 0.7 and earlier.
0.7 can actually CREATE blocks that it will itself reject due to that bug.

Demanding EVERYONE switch to 0.8 is not realistic either.

What needs to happen is that 0.8.1 should be released where its the same as 0.8 only it changes the default CONFIGURATION such that it will NOT create 0.7- incompatible blocks.
When over 90% of the network are using 0.8+ this limitation can be safely removed and people using 0.7 or earlier will be forced to upgrade.

Until such a hypothetical 0.8.1 is released you can MANUALLY apply the needed configuration files using the instructions in:
https://bitcointalk.org/index.php?topic=152030.msg1620853#msg1620853

[gyverlb]

I think I'll post a detailed guide about p2pool in the following days in order to promote it's adoption (my configuration is giving me satisfaction and I see many people who don't get as good results as mine).

There are some things hard to find by searching the forums and some urban legend to address too. There are cases where p2pool isn't a good solution that should be clearly reminded (BFL singles, some network setups, ...) so that people don't waste time with it when they'll clearly lose income instead of gaining some vs ordinary pools.

Don't forget to mention that p2pool was basically broken by all this :p

Not as much as other pools. Pools running a 0.8 node were completely broken. As p2pool is decentralized the potentially lost income was proportional to the hashrate behind 0.8 nodes (some of us still run 0.7 and could have found a valid block).

In this particular case we lucked out: a block was found by a 0.8 node and orphaned when the 0.7 chain caught up and none were found by 0.7 node.

[johnyj]

http://cointext.com/the-dust-settles-on-the-bitcoin-fork-incident/

"One scenario is that a takeover could occur where a fork is created with a fake chain faster than the real chain.  The attackers could then hold the users “hostage” and tell them to change to their client (and change the rules) or they will disrupt the system.  Another scenario is that a disagreement will occur over how Bitcoin will be used and a hard fork will be created purposely.  In this case those with the greater computing power will win.   It is speculated that many large players will maintain mining equipment kept in hibernation only to be turned on in the event of such an event."

Wow, conspiracy is fun 

[evilpete]

What needs to happen is that 0.8.1 should be released where its the same as 0.8 only it changes the default CONFIGURATION such that it will NOT create 0.7- incompatible blocks.
When over 90% of the network are using 0.8+ this limitation can be safely removed and people using 0.7 or earlier will be forced to upgrade.

Until such a hypothetical 0.8.1 is released you can MANUALLY apply the needed configuration files using the instructions in:
https://bitcointalk.org/index.php?topic=152030.msg1620853#msg1620853

No, the point of the big mining pools switching to non-upgraded 0.7 was to act as the lowest common denominator so that more than 51% of the hash power will (mostly) reject the same valid blocks that the other non-upgraded 0.7 users and merchants.  As close as it can be done, that is.

Quoting what I said above:  "Upgrade" includes upgrading their bitcoind/bitcoin-qt etc to 0.8+ OR setting the DB_CONFIG entries mentioned above.

Releasing a 0.8.1 won't fix anything.  All that it can do is clamp the value that you can set the max generated block size to.  The problem is the code is still there, and so is 0.8.  If somebody wants to compile a custom 0.8.1 from source with the limit at 1MB, they can.

If a person (or a pool) wanted to cause trouble, they could simply change the code in their 0.8.1 checkout and compile, or use 0.8.  Of course they need to be able to solo-mine their own blocks.  Generating a block that's complex enough to overflow the lock tables of old 0.7 bdb based systems is still valid.  If more than 51% of the hash power applies the DB_CONFIG fixes (even on 0.7) then they'll accept it and build on it.

What actually needs to happen is a new fixed 0.6.x, 0.7.x need to come out, and the people who can't/won't go to 0.8.x need to pick up one of those instead.  Or do the DB_CONFIG fixes.  But more than 51% of the hash power has to stay behind until a good 90%+ of nodes have been upgraded.  Because sooner or later somebody will generate one bdb-breaker on purpose.

To repeat, by "upgraded", I mean either switching to 0.8.x or applying the DB_CONFIG changes, or switching to a new 0.6.x or 0.7.x with the bdb changes.

Releasing 0.8.1 won't fix the vulnerable 0.3 / 0.6 / 0.7 nodes.  The moment somebody uses a a couple of avalons to feed a custom patched 0.8 in order to deliberately make a bdb-breaker block we'll be in the same mess again unless there is > 51% of the hash power on un-fixed 0.7 to cause the bdb-breaker block to be orphaned.

The moment somebody figures out a financial angle to exploit this, it'll happen.  eg: a replay of the OKPAY double spend etc.

Anyway, the pool operators have this covered.  The only reason I said anything at all was somehow end users are getting the idea they need to downgrade.  They should not.
If they're on 0.8, they should stay on 0.8.
If they're on 0.7 or earlier they EITHER need to fix their DB_CONFIG; or get a fixed 0.7.x when it comes out; or upgrade to 0.8.
If they're running mining software as part of a pool then the bitcoind they're running doesn't matter - they still need the fixed one.
Merchants particularly have to pay attention.

The only people that should be downgrading are pool operators or large solo miners.

[niko]

Naturally there's still potentially more problems in 0.7 lurking.

and in 0.8 .... what's good for the goose .... 0.7 has been running for much longer in production than 0.8.

This. Most people here are focusing on the manifestation of the problem, and ignoring the continuing pattern of human behavior that increased the risk of this problem emerging. As long as bitcoin protocol is treated as a black box full of poorly documented code and unexpected behavior that is filled with more of the same with every version, the risk will remain significant.

[taltamir]

Quoting what I said above:  "Upgrade" includes upgrading their bitcoind/bitcoin-qt etc to 0.8+ OR setting the DB_CONFIG entries mentioned above.

AND not OR.
There is no point in setting anything in 0.7-;
0.7 can NOT make a block that 0.8 rejects.
0.7 AND 0.8 can both make blocks that 0.7 will reject due to a bug in 0.7-.
0.8 can be configured to only make blocks that are accepted by everyone.

When everyone was on 0.7, if it made a block that it itself rejected than nothing bad happened. with 0.8 you had the fork risk suddenly come up.

If you are going to be messing with your install then you should do so by switching to 0.8 with the right config not by trying to mess with the bugged 0.7

Releasing a 0.8.1 won't fix anything.  All that it can do is clamp the value that you can set the max generated block size to.  The problem is the code is still there, and so is 0.8.  If somebody wants to compile a custom 0.8.1 from source with the limit at 1MB, they can.

Default configuration is included in the source code and as such anyone who compiles an 0.8.1 without explicitly altering said code would get the new configuration.

If a person (or a pool) wanted to cause trouble, they could simply change the code in their 0.8.1 checkout and compile, or use 0.8.  Of course they need to be able to solo-mine their own blocks.  Generating a block that's complex enough to overflow the lock tables of old 0.7 bdb based systems is still valid.  If more than 51% of the hash power applies the DB_CONFIG fixes (even on 0.7) then they'll accept it and build on it.

This wont cause trouble to anyone except for themselves, they will be generating blocks that are rejected by the chain until and unless 51+% are using 0.8+, at which point those using obsolete bugged version are forced to upgrade.

What actually needs to happen is a new fixed 0.6.x, 0.7.x need to come out

There is already a fixed 0.7, its called 0.8.

[evilpete]

Quoting what I said above:  "Upgrade" includes upgrading their bitcoind/bitcoin-qt etc to 0.8+ OR setting the DB_CONFIG entries mentioned above.

AND not OR.
There is no point in setting anything in 0.7-;
0.7 can NOT make a block that 0.8 rejects.
0.7 AND 0.8 can both make blocks that 0.7 will reject due to a bug in 0.7-.
0.8 can be configured to only make blocks that are accepted by everyone.

.. and 0.8 won't make a block by default that is rejected by 0.7.  Miners had to explicitly raise their -blockmaxsize, which is what happened.  It was suggested on a thread in the miners forum and a few folks tried it out.  And we saw what happened.

Releasing a 0.8.1 won't fix anything.  All that it can do is clamp the value that you can set the max generated block size to.  The problem is the code is still there, and so is 0.8.  If somebody wants to compile a custom 0.8.1 from source with the limit at 1MB, they can.

Default configuration is included in the source code and as such anyone who compiles an 0.8.1 without explicitly altering said code would get the new configuration.

You're missing the point.  I'm not talking about the default configuration, which is "safe" by default everywhere.  0.8 is safe by default. The problem is when somebody maliciously makes a bdb-buster block because they either want to cause trouble, or because they figure out a financial advantage to do so.

If more than 51% of the hash power applies the DB_CONFIG fixes (even on 0.7) then they'll accept it and build on it.

This wont cause trouble to anyone except for themselves, they will be generating blocks that are rejected by the chain until and unless 51+% are using 0.8+, at which point those using obsolete bugged version are forced to upgrade.

Read what I said.  The only thing preventing another bdb-buster fork is that 51% of the hashpower are running a broken 0.7 on purpose.  If that changes, eg: miners apply the fixes to 0.6 or 0.7, or a ton of asic solo miners turn on at 0.8, then we're back in fork territory when somebody wants to cause trouble.  If the miners fix their bdb config they'll behave just like 0.8 next time a killer block comes along.

But the last thing we need is more people downgrading to 0.7.  It just delays fixing this land mine.

[Tyger]

Its like a computer, you have to keep it up to date, hardware and software. if you do not upgrade you wont get allong with the rest of the world. if people refuse to upgrade (there are some verry old clients out there) then its just logic that it will stop working at some point, just like with everything else.

If they dont upgrade its there problem.

[marcus_of_augustus]

evilpete

But more than 51% of the hash power has to stay behind until a good 90%+ of nodes have been upgraded.

This tries to swim against the tide of all individual incentives built into the system.

Why would anyone want to be on a version that the miners are not using? ... you are opening yourself up to the possible risk of ending up on a useless fork.

[jgarzik]

For diagnostic purposes, here is a blockchain dataset built by a 0.7.2 bitcoind w/ db 4.8:

     http://gtf.org/garzik/bitcoin/chain-db48-h225429.tar.bz2

It contains blockchain + index, up to height 225429, making it easy to reproduce an injection of too-large blocks at the precise juncture where the recent chain fork occurred.

Byte size: 5755999214 (5.3G)
MD5: 479e6364352d0ec68ea5cf87200f7f1e
SHA1: 18bc8ff5e572fc4a27828341249477eb8355f012
SHA256: 41787306487da1957717d72407fc1e01ed5965202802f7d18e2930a30fc169a2

[John Tobey]

What if the .7 chain never catches the .8 chain?

Correct me if I'm wrong but if many miners abandon the .8 chain won't the difficulty adjust down, so that even though there are much few miners on the .8 chain they will still be able to stay ahead of the .7 chain due to the decreased difficulty?

This could not have happened:

"Length" is calculated as total combined difficulty of that chain, not number of blocks...

[taltamir]

.. and 0.8 won't make a block by default that is rejected by 0.7.  Miners had to explicitly raise their -blockmaxsize, which is what happened.  It was suggested on a thread in the miners forum and a few folks tried it out.  And we saw what happened.

Now we are getting somewhere. All our disagreement can be distilled into this one single "fact" that the two of us disagreed on and only now that you posted this do I find out what that "fact" is.

So, can anyone point me at evidence for or against that? as far as I knew there was no manual raising by miners and that 0.8 can, by default, generate blocks that 0.7- will reject.