The March 2013 chain fork from the #bitcoin-dev log: 25 block reorg

[windpath]

New Rabbit Hole on LearnBitcoin: The 2013 Chain Fork
https://www.learnbitcoin.com/rabbit-hole/2013-chain-fork

Some of you were here for it, and the threads from that night are still on this forum, so this is the crowd most likely to catch what I got wrong. It follows the 11-12 March 2013 fork from the #bitcoin-dev log with UTC timestamps, checked against block headers and the 0.7.2 and 0.8.1 source.

The parts I'd expect an argument about:

• Length. Almost every summary says 24 blocks and six hours. Two separate 0.8 nodes logged "REORGANIZE: Disconnect 25 blocks" at 06:22, the abandoned chain ran 225,430 to 225,454 inclusive, and it was 7 hours 41 minutes from Slush's block (22:39:09) to the block that overtook it (225,455 at 06:20:04).
• Cause. Not block size. 998 kB and about 1,750 transactions touching a little over 5,000 tx-index entries, against set_lk_max_locks(10000) in db.cpp, with locks taken per page so the ceiling varied from node to node.
• The decision. Gavin opened with "first rule of bitcoin: majority hashpower wins" and "The bug is clearly in 0.7." sipa and Luke-Jr argued a majority is not enough for a hard fork. Eleuthria started rolling BTC Guild back fourteen minutes before the ACKs came.
• Slush. BIP-50 says both pools downgraded. The log has slush unable to get 0.7 synced and rejoining on 0.8 with sipa's emergency blacklist patch.
• BIP-50 itself. Parts of it were rewritten in February 2016, so it is not all 2013 wording.
• The 0.8.1 rule. The consensus rule was at most 4,500 distinct txids per block from 21 March to 15 May. The 500 kB figure was a cap on what 0.8.1 itself would mine.
• One log archive runs an hour behind UTC in its later captures, and a well-known 2015 analysis inherited that. Times in the chapter are from the other archive and match the block headers and mailing list stamps.

If you were in the channel, or you have and will share a debug.log from that night, I'd like to hear where this is off. It gets fixed the same day and credited.

[BlackHatCoiner]

For those who don't understand, here's what happened:

Satoshi chose 1 MB block size limit, but he also (unintentionally?) had written another quite rule: a block was rejected if processing it needed more than 10,000 database locks. Because locks are taken per database page, the real limit depended on how each node's files happened to be arranged on disk, so the same block could pass on one machine and fail on another.

The fix was just removing this weird rule, and this is why it caused a fork between 0.8 and some 0.7 nodes.

[wapwapwendy]

New Rabbit Hole on LearnBitcoin: The 2013 Chain Fork
https://www.learnbitcoin.com/rabbit-hole/2013-chain-fork

Some of you were here for it, and the threads from that night are still on this forum, so this is the crowd most likely to catch what I got wrong. It follows the 11-12 March 2013 fork from the #bitcoin-dev log with UTC timestamps, checked against block headers and the 0.7.2 and 0.8.1 source.

The parts I'd expect an argument about:

• Length. Almost every summary says 24 blocks and six hours. Two separate 0.8 nodes logged "REORGANIZE: Disconnect 25 blocks" at 06:22, the abandoned chain ran 225,430 to 225,454 inclusive, and it was 7 hours 41 minutes from Slush's block (22:39:09) to the block that overtook it (225,455 at 06:20:04).
• Cause. Not block size. 998 kB and about 1,750 transactions touching a little over 5,000 tx-index entries, against set_lk_max_locks(10000) in db.cpp, with locks taken per page so the ceiling varied from node to node.
• The decision. Gavin opened with "first rule of bitcoin: majority hashpower wins" and "The bug is clearly in 0.7." sipa and Luke-Jr argued a majority is not enough for a hard fork. Eleuthria started rolling BTC Guild back fourteen minutes before the ACKs came.
• Slush. BIP-50 says both pools downgraded. The log has slush unable to get 0.7 synced and rejoining on 0.8 with sipa's emergency blacklist patch.
• BIP-50 itself. Parts of it were rewritten in February 2016, so it is not all 2013 wording.
• The 0.8.1 rule. The consensus rule was at most 4,500 distinct txids per block from 21 March to 15 May. The 500 kB figure was a cap on what 0.8.1 itself would mine.
• One log archive runs an hour behind UTC in its later captures, and a well-known 2015 analysis inherited that. Times in the chapter are from the other archive and match the block headers and mailing list stamps.

If you were in the channel, or you have and will share a debug.log from that night, I'd like to hear where this is off. It gets fixed the same day and credited.

The 2013 fork is a good reminder that Bitcoin’s early years were far from smooth. The fact that the developers had to coordinate so quickly across different versions makes the incident especially worth studying today.

[windpath]

Satoshi chose 1 MB block size limit, but he also (unintentionally?) had written another quite rule: a block was rejected if processing it needed more than 10,000 database locks. Because locks are taken per database page, the real limit depended on how each node's files happened to be arranged on disk, so the same block could pass on one machine and fail on another.

That's a better two-sentence version than mine, and yes, unintentionally. The 10,000 lock setting is in the code as far back as v0.1.5 in 2009, so it dates from Satoshi's time, and I can't find anything suggesting anyone thought of it as a limit on blocks until that night. Luke's line in the channel was "it's a bitcoin rule we didn't know about."

One small thing on the order of events. 0.8 didn't remove the rule as a fix, it removed it by accident when it switched databases, and that's what caused the fork. The fix on the night went the other way: the 0.8 miners went back and obeyed the old limit so the chain everyone could follow would win. 0.8.1 then kept obeying it on purpose for two months (the 4,500 txid rule), and only after 15 May was it actually dropped.

The fact that the developers had to coordinate so quickly across different versions makes the incident especially worth studying today.

Agreed. And it wasn't only the developers. The person who made it possible was a pool operator, and he ate the loss himself.