Reflections on the 4 July fork

[jl2012]

I think we can learn a lot from this lesson

1. Good practice of SPV mining?

SPV mining means mining without fully validating the previous block(s). Miners do that to reduce wastage of hashing power. I don't think prohibiting SPV mining is possible as we will never have 95% of miner support. If done properly, SPV mining is not particularly risky and could increase the overall hash rate of the network. Instead of trying to ban SPV mining, I think we should have a good practice guideline for SPV mining.

Miner should always validate the header, including the version, bits, and timestamp. There is no excuse to skip this. The laziness costs them 125BTC.

Building blindly on a valid header is fine, until the block is validated. The first confirmation of the original bogus block (363731) came after 13 minutes. It was more than enough for full validation and there was no excuse for the ignorance.

Miners should never extend an "SPV chain" to more than 2 (or 3 at maximum) blocks, even if that contains their own block. If an SPV chain is too long, they should assume it as invalid until it is fully validated. They should go back to the last known good block. This is to protect SPV wallets.

2. The 5 consecutive empty blocks

In the bogus fork, F2Pool and AntPool had a chain of 5 consecutive empty blocks. Although empty blocks are not uncommon, 5 in a row is suspicious. I guess there is some bugs in their system. For example, their system may insist to mine on their own blocks even if they are invalid (F2Pool and AntPool are alliance). And since the system knew the chain was invalid, it was unable to include any transaction.

3. Softfork voting by version bits

It is suggested that softfork voting should be indicated by bits in version, and allowing reuse of bits after the softfork is accepted or rejected. ( https://gist.github.com/sipa/bf69659f43e763540550 ) . I think this is the right way to go. However, in order to be compatible with SPV mining, the version bit should be kept switched on for 5,000 blocks after enforcement. This will be more than a month and should be enough to squeeze any laggards out. After that, the bit is free for another softfork proposal.

4. Wallet design

Full nodes should report which BIPs are they enforcing. SPV nodes should connect to full nodes with different rule set and compare their chains to identify any fork

SPV nodes should monitor recent forks. They should warn the users if there are long competing forks. They should consider transactions are unconfirmed unless they are included in both forks.

[TierNolan]

I think we can learn a lot from this lesson

1. Good practice of SPV mining?

Miner should always validate the header, including the version, bits, and timestamp. There is no excuse to skip this. The laziness costs them 125BTC.

Absolutely.  Relative to network delay, checking the header chain, in full, is effectively instant.

Miners should never extend an "SPV chain" to more than 2 (or 3 at maximum) blocks, even if that contains their own block.

I think a simple timeout is sufficient.  If a minute passes without a header being validated, switch to "emergency mode" and disable SPV-mining until a validated block is found.

The rule to limit fork size would mean that miners would build on the same (invalid) block over and over again.

A <- B <- c

c is not validated, but the miner finds d

A <- B <- c <- d

c & d now make an invalid unverified fork.

Under the max of 2 unvalidated blocks rule, the miner would start mining d', which builds on c.  This means the miner is trying to orphan itself .

The rule should be to enter emergency mode.  Once d is found, the miner should start mining on B until the non-validated chain is orphaned.

The timeout rule is better though.  A pool just needs to determine what is the maximum latency for validating blocks.  If 99% of blocks are validated within 30 seconds of the header arriving, then using 60 seconds as a timeout will almost never harm the node.

2. The 5 consecutive empty blocks

This is fighting the last war.  This occurred due to the specifics of this particular issue.  Using a timeout would fix the issue anyway.

3. Softfork voting by version bits

It is suggested that softfork voting should be indicated by bits in version, and allowing reuse of bits after the softfork is accepted or rejected. ( https://gist.github.com/sipa/bf69659f43e763540550 ) . I think this is the right way to go. However, in order to be compatible with SPV mining, the version bit should be kept switched on for 5,000 blocks after enforcement. This will be more than a month and should be enough to squeeze any laggards out. After that, the bit is free for another softfork proposal.

I don't think it matters.  What is required is that there is a standard way to track version updates.  SPV clients should be required to keep up to date with any rule changes.

Bitcoinj apparently doesn't do any version field checking. 

I think that SPV wallets should give a warning if they cannot correctly understand the version field.

SPV clients from before the soft fork was released, should give a warning when 75% of the miners were using a version field that could not be interpreted.  This would have given a warning that the clients need to be updated when when the miners hit 750 out of 1000.

If the new bits field is added, then SPV clients should be able to track them.  Generally, SPV clients should be required to fully understand the header chain.  The simplification is that they don't process the entire blocks.

4. Wallet design

Full nodes should report which BIPs are they enforcing. SPV nodes should connect to full nodes with different rule set and compare their chains to identify any fork

SPV nodes should monitor recent forks. They should warn the users if there are long competing forks. They should consider transactions are unconfirmed unless they are included in both forks.

That is a network protocol change and seems reasonable.    It could be an added field to the version message or a list of BIPs that are being enforced.

[johoe]

One problem was also that the effect of the soft fork on the "SPV mining"-hack was never tested.  The developers didn't even know the miners were doing this or if they were aware they wouldn't know the details.  It would be good if the miners also used TestNet regularly under realistic scenarios.  E.g. they could spend a ten-billionth of the hashing power on TestNet.  Keep the hashing power small enough, so that others can still do useful tests on TestNet with a small ASIC usb stick, but use it all the time and try changes there first.  A soft fork can be forced to apply earlier on testnet by just sending some additional hashing power supporting the soft fork.

SPV nodes should monitor recent forks. They should warn the users if there are long competing forks. They should consider transactions are unconfirmed unless they are included in both forks.

this is a good idea.  Most SPV clients already are connected to several nodes, so it shouldn't be too hard to implement.  One probably has to ignore a single node on a shorter fork to avoid false warnings.  In this scenario it would have made the SPV client secure as long as it has enough outgoing connections.

[amaclin]

Miner should always validate the header, including the version, bits, and timestamp. There is no excuse to skip this. The laziness costs them 125BTC.

How much have they saved in past by skipping it for speed purposes? 

[gogxmagog]

the more stuff like this split happens, the more i feel that one day it will be fatal. fortunately it looks like no double spends happened this time, but 3 years in the bitcoin world and ive seen every type of double-dealing and scam happen, even by those you would never expect it.

we got off easy this time, but what if this split had enabled bad actors to pull a massive fraud? that would crash btc to nothing.

[jl2012]

3. Softfork voting by version bits

It is suggested that softfork voting should be indicated by bits in version, and allowing reuse of bits after the softfork is accepted or rejected. ( https://gist.github.com/sipa/bf69659f43e763540550 ) . I think this is the right way to go. However, in order to be compatible with SPV mining, the version bit should be kept switched on for 5,000 blocks after enforcement. This will be more than a month and should be enough to squeeze any laggards out. After that, the bit is free for another softfork proposal.

I don't think it matters.  What is required is that there is a standard way to track version updates.  SPV clients should be required to keep up to date with any rule changes.

Bitcoinj apparently doesn't do any version field checking. 

I think that SPV wallets should give a warning if they cannot correctly understand the version field.

SPV clients from before the soft fork was released, should give a warning when 75% of the miners were using a version field that could not be interpreted.  This would have given a warning that the clients need to be updated when when the miners hit 750 out of 1000.

If the new bits field is added, then SPV clients should be able to track them.  Generally, SPV clients should be required to fully understand the header chain.  The simplification is that they don't process the entire blocks.

If we allow miners to turn off the version immediately after the 95% vote, it is impossible to identify un-upgraded miners with the header only.

Use BIP66 as example. Block with version 2 or below is invalid after enforcement, even if all transactions are DER compliant.

If it allowed switching back to version 2 AFTER the enforcement, the  original bogus block (363731) would have been considered as a valid one even by Bitcoin core 0.10, since it did not contain any non-DER-compliant signature. However, if it did contain any invalid signature, it is impossible to tell just by looking at the header version.

That's why I suggest a version bit must be kept on for 5000 blocks AFTER enforcement, so SPV wallets (and SPV miners!) will know there is something wrong just by looking at the header.

[TierNolan]

If we allow miners to turn off the version immediately after the 95% vote, it is impossible to identify un-upgraded miners with the header only.

Ahh ok. 

Those blocks are unlikely to be built on top of anyway.  It means single (and maybe double) confirmation transactions are safer.

If it allowed switching back to version 2 AFTER the enforcement, the  original bogus block (363731) would have been considered as a valid one even by Bitcoin core 0.10, since it did not contain any non-DER-compliant signature. However, if it did contain any invalid signature, it is impossible to tell just by looking at the header version.

That situation seems ok, since it would be a valid block?  It is literally indistinguishable from a block produced by any upgraded miner. 

The benefit is for invalid block produced by the 4%, you can tell by the header.

On the plus side if the majority (but invalid) fork has some obsolete blocks in its chain, it basically black-lists the entire invalid fork for SPV clients.

This assumes that SPV miners do less SPV checking than SPV clients.

SPV miners need to make sure their SPV header rules are as up to date as possible.  If their SPV client rejected the version 2 blocks after 95% lock-in, then everything would have worked out ok.

Keeping the bits active after enforcement would help SPV-miners too.

[tspacepilot]

I think we can learn a lot from this lesson

SPV mining means mining without fully validating the previous block(s). Miners do that to reduce wastage of hashing power. I don't think prohibiting SPV mining is possible as we will never have 95% of miner support.

DannyHamilton makes an interesting suggestion here about how non SPV-miners might attack spv-miners such that spv mining may no longer be trusted and therefore no longer viable.  If I understood his suggestion correctly and if he's right and if this happens, then perhaps spv mining will no longer be viable anyway.

the more stuff like this split happens, the more i feel that one day it will be fatal. fortunately it looks like no double spends happened this time, but 3 years in the bitcoin world and ive seen every type of double-dealing and scam happen, even by those you would never expect it.

we got off easy this time, but what if this split had enabled bad actors to pull a massive fraud? that would crash btc to nothing.

gogxmagog: I don't think you're correct.  We *want* to see stuff like this because it builds confidence that the checks and balances of the network and core players are working as expected.  In my opinion we want to see little forks and hiccups so that our minds get expanded to the kinds of things that can happen and we can build more and more robust systems.  If everything was always boring, that's when we'd get lulled into complancency and some emergent situation would eventually come along and wipe us out.

[jl2012]

I think we can learn a lot from this lesson

SPV mining means mining without fully validating the previous block(s). Miners do that to reduce wastage of hashing power. I don't think prohibiting SPV mining is possible as we will never have 95% of miner support.

DannyHamilton makes an interesting suggestion here about how non SPV-miners might attack spv-miners such that spv mining may no longer be trusted and therefore no longer viable.  If I understood his suggestion correctly and if he's right and if this happens, then perhaps spv mining will no longer be viable anyway.

I don't think such attack is practical. Attacker has to waste hashing power to mine an invalid block first.

[TierNolan]

I don't think such attack is practical. Attacker has to waste hashing power to mine an invalid block first.

I did a worked example.  The cost to the attackers is more than the slight loss to the SPV-miners and then the SPV-miners are better off after the next difficulty update.

[jl2012]

I don't think such attack is practical. Attacker has to waste hashing power to mine an invalid block first.

I did a worked example.  The cost to the attackers is more than the slight loss to the SPV-miners and then the SPV-miners are better off after the next difficulty update.

I think it's quite obvious without any calculation: an attacker MUST lose a block, while an SPV miner MAY lose a block