"SPV Mining" or mining on invalidated blocks

[BitUsher]

https://bitcoin.org/en/alert/2015-07-04-spv-mining

Thanks to Peter Todd, Luke-Jr, and Gregory Maxwell careful watch we averted a disaster, but it is a little disconcerting this problem was allowed to fork the chain for so long causing a panic that is still not completely resolved.

A few points of consideration:
If you do the math the pool operators that "SPV mine" or building empty blocks on the longest invalidated header chain they are still incentivized to SPV mine as the latency reduction advantages gained from this practice exceed the ~50k lost during this fork in the chain.

We shouldn't leave it up to alerts and  warnings after the fact as the panic itself creates tremendous loss of confidence in our ecosystem.

Have there been any discussions on various solutions to either force or incentivize miners to validate their block headers?

Are there any proposals to implement a warning system that SPV/Web wallets could implement that would pickup the same alerts from Bitcoin Core and warn the users?

[DannyHamilton]

https://bitcoin.org/en/alert/2015-07-04-spv-mining

Thanks to Peter Todd, Luke-Jr, and Gregory Maxwell careful watch we averted a disaster, but it is a little disconcerting this problem was allowed to fork the chain for so long causing a panic that is still not completely resolved.

A few points of consideration:
If you do the math the pool operators that "SPV mine" or building empty blocks on the longest invalidated header chain they are still incentivized to SPV mine as the latency reduction advantages gained from this practice exceed the ~50k lost during this fork in the chain.

We shouldn't leave it up to alerts and  warnings after the fact as the panic itself creates tremendous loss of confidence in our ecosystem.

Have there been any discussions on various solutions to either force or incentivize miners to validate their block headers?

Are there any proposals to implement a warning system that SPV/Web wallets could implement that would pickup the same alerts from Bitcoin Core and warn the users?

Am I mistaken, or is there a significant financial incentive for non-SPV mining pools to get their miners to mine a block that will send SPV mining pools onto a bad fork?

Doing so would cut the hash power of the network nearly in half for a short period of time (either until the non-SPV fork exceeds the length of the SPV fork, or until the SPV pools notice the problem and switch to the correct fork) which should double the chances of the non-SPV pools finding blocks until the SPV mining pools get themselves back on to the correct fork.

If all of the non-SPV pools continuously send the SPV pools off on forks that will get orphaned, then they will all increase their own revenue and destroy the revenue (and incentive) for SPV mining, right?

Furthermore, all the mining participants that are participating in SPV pools will see their personal revenue drop to miniscule levels.  As such, they will have very strong financial incentive to leave the SPV pools and join one of the non-SPV pools. This will increase the number of participants (and the total hash power) of the non-SPV pools and will therefore eventually reduce the length of the orphaned chains.

If non-SPV pools implement this, isn't this SPV mining dis-incentive likely to outweigh the incentives of SPV mining?  As such, wouldn't any pool that wants to remain relevant be forced to switch to non-SPV mining?

[BitUsher]

Am I mistaken, or is there a significant financial incentive for non-SPV mining pools to get their miners to mine a block that will send SPV mining pools onto a bad fork?

Doing so would cut the hash power of the network nearly in half for a short period of time (either until the non-SPV fork exceeds the length of the SPV fork, or until the SPV pools notice the problem and switch to the correct fork) which should double the chances of the non-SPV pools finding blocks until the SPV mining pools get themselves back on to the correct fork.

If all of the non-SPV pools continuously send the SPV pools off on forks that will get orphaned, then they will all increase their own revenue and destroy the revenue (and incentive) for SPV mining, right?

Furthermore, all the mining participants that are participating in SPV pools will see their personal revenue drop to miniscule levels.  As such, they will have very strong financial incentive to leave the SPV pools and join one of the non-SPV pools. This will increase the number of participants (and the total hash power) of the non-SPV pools and will therefore eventually reduce the length of the orphaned chains.

If non-SPV pools implement this, isn't this SPV mining dis-incentive likely to outweigh the incentives of SPV mining?  As such, wouldn't any pool that wants to remain relevant be forced to switch to non-SPV mining?

Very interesting attack vector I had not thought about. From what I understand the practice of SPV mining can reduce orphan rate from ~4% down to near ~0% from F2Pools claims. This would give a very strong incentive to continue this practice despite this risk. Additionally, this form of attack would cause multiple panics like we are seeing now which would undermine our whole confidence in the bitcoin ecosystem which is not something non-SPV mining pools would typically be interested in.

[Carlton Banks]

Doing so would cut the hash power of the network nearly in half for a short period of time (either until the non-SPV fork exceeds the length of the SPV fork, or until the SPV pools notice the problem and switch to the correct fork) which should double the chances of the non-SPV pools finding blocks until the SPV mining pools get themselves back on to the correct fork.

Hmmm, difficulty remains the same though. It could work if a huge miner kept some (otherwise uneconomic) idle hashing power in wait, to be deployed only when a chain fork is detected.

[DannyHamilton]

Very interesting attack vector I had not thought about. From what I understand the practice of SPV mining can reduce orphan rate from ~4% down to near ~0% from F2Pools claims.

So, if I understand correctly, SPV mining when nobody is broadcasting forking blocks reduces the orphan rate trom ~4% to nearly 0%.

However, SPV mining when non-SPV pools and miners ARE broadcasting forking blocks increases the orphan rate from nearly 0% to nearly 100%, right?

This would give a very strong incentive to continue this practice despite this risk.

I guess that depends on how much non-SPV hash power is participating in the attack against the SPV pools?  With 100% participation in the attack, there is 0 incentive to continue SPV mining. With 0% participation there is obviously still an incentive for SPV mining.  I suppose the question that would be interesting to answer is "How much hash power needs to participate in the attack for SPV mining to overcome the current SPV incentive?"

Additionally, this form of attack would cause multiple panics like we are seeing now which would undermine our whole confidence in the bitcoin ecosystem which is not something non-SPV mining pools would typically be interested in.

Perhaps.

Or perhaps people would quickly adjust to trusting non-SPV mining, and SPV mining would die rather quickly.

If a non-SPV pool knows that the fork CAN happen in the future, and that they can profit from causing it themselves instead of just waiting for it to happen, wouldn't they WANT to do this?  Either way it's likely to happen, and the panics will result.  What's the benefit of NOT being the one that does it, especially if the pool thinks that any other pool might already be thinking about doing it?

[DannyHamilton]

Hmmm, difficulty remains the same though. It could work if a huge miner kept some (otherwise uneconomic) idle hashing power in wait, to be deployed only when a chain fork is detected.

Difficulty only remains the same until the next adjustment.  Then it will decrease, since a significant percentage of the network has been continuously participating in orphaned chains.

Meanwhile, the miners that leave the SPV pools (due to inadequate revenue) and join the non-SPV pools will increase the hashing power of the non-SPV pools.  This is the "idle hashing power" you are talking about, isn't it?  Hash power that the non-SPV pools don't have right now, but which they gain as the attack progresses?

[Carlton Banks]

Hmmm, difficulty remains the same though. It could work if a huge miner kept some (otherwise uneconomic) idle hashing power in wait, to be deployed only when a chain fork is detected.

Difficulty only remains the same until the next adjustment.  Then it will decrease, since a significant percentage of the network has been continuously participating in orphaned chains.

Meanwhile, the miners that leave the SPV pools (due to inadequate revenue) and join the non-SPV pools will increase the hashing power of the non-SPV pools.  This is the "idle hashing power" you are talking about, isn't it?  Hash power that the non-SPV pools don't have right now, but which they gain as the attack progresses?

I agree with your point about the independent miners on the non-verifying pools, they would be incentivised to switch to verifying pools.

These forks are highly unlikely to last for a significant proportion of a 2016 block difficulty period. The difference to the readjust that such an event produces in a given difficulty period may not be significant enough to produce the hashrate decrease you're imagining. If attack victims switch relatively quickly, then the effect it has on the hashrate/difficulty is none, as you say.

So if a large majority of miners (both SPV and fully verifying) act rationally, then this invalid block attack would bring everyone back into the verifying fold. I expect they would do so, the range of irrational actions they have to choose from could only be interesting to someone being self destructive, they can't harm the network to any significant extent.

[DannyHamilton]

- snip -
So if a large majority of miners (both SPV and fully verifying) act rationally, then this invalid block attack would bring everyone back into the verifying fold. I expect they would do so
- snip -

I agree.  Which is why I presented it as a response to this thread...

- snip -
Have there been any discussions on various solutions to either force or incentivize miners to validate their block headers?
- snip -

If I ran my own mining pool, I would be giving some serious consideration to implementing this attack right now.

[trout]

To implement this attack, non-SPV-mining pools need to waste enough hash power to create a "bad" block. This is profitable to all of them together, but they have to cooperate.  Cooperation seems more difficult to achieve. THen again, if there is one single big pool, the insentive may be big enough for him to do it alone.

It's not difficult to calculate how big it should be; the main unknown paramter is for how long it sends the SPV-miners off to mind the wrong side of the fork.

[Carlton Banks]

To implement this attack, non-SPV-mining pools need to waste enough hash power to create a "bad" block. This is profitable to all of them together, but they have to cooperate.  Cooperation seems more difficult to achieve. THen again, if there is one single big pool, the insentive may be big enough for him to do it alone.

It's not difficult to calculate how big it should be; the main unknown paramter is for how long it sends the SPV-miners off to mind the wrong side of the fork.

I considered this too, but it depends on your definition of "waste". The progenitor of the attack stands to gain both financially and in esteem (and also builds overall confidence in the network), whether by attracting independent miners from SPV pools, or by infilling with spare hashing capacity. It's true that there will be a critical hashrate below which creating attack/invalid blocks represents too large a proportion of the blocks they do find such that they can write it off against the potential gains, but miners/pools of that size can still gain marginally from another miner conducting the attack, as long as they're also verifying new blocks.

[TierNolan]

So, if I understand correctly, SPV mining when nobody is broadcasting forking blocks reduces the orphan rate trom ~4% to nearly 0%.

However, SPV mining with non-SPV pools and miners ARE broadcasting forking blocks increases the orphan rate from nearly 0% to nearly 100%, right?

It depends on the bad block rate and normal orphan rate.

With a 5% orphan rate, that means that miners are spending 30 seconds per block validating new blocks.  This is 30 seconds of wasted hashing.

If 10% of blocks are invalid and miners switch to non-SPV mode after 1 minute, then

90% of the time: 30 second of mining saved from wastage
10% of the time: 60 seconds wasted building on the invalid block

This means that on net, the miner saves 0.9*30 - 0.1*60 = 21 seconds and they are better off using SPV-mining.

The key to make it work is to have a timeout, so you can handle the (rare) invalid blocks correctly.

The timeout can be made less than 60 seconds, if they actually check the block they are building on when it is received.

If miners do this, then there is no risk to the network.  The miner automatically switches back to non-SPV mode after the 60 second timeout.  A fork can only exist for that long, so probably less than 1 block.

[johoe]

Thanks to Peter Todd, Luke-Jr, and Gregory Maxwell careful watch we averted a disaster, but it is a little disconcerting this problem was allowed to fork the chain for so long causing a panic that is still not completely resolved.

How much hashing power was supporting the fork?  I only saw AntMiner, F2Pool and BTCNugget, which have less than 50 % together.  Wouldn't the fork have stopped automatically when these pools ran out of "luck" and the remaining network caught up?  We'll probably never know who of the smaller pools would have supported the wrong chain.

The 150 BTC reward that these pools lost is not really lost; they will be reflected in a slightly smaller difficulty soon and thus evenly distributed between all pools.  Also, the 100 BTC f2pool lost is probably less than what they gained by their dangerous tactic of mining on the longest chain without verifying it.

However, it's hard to say how much they really gain with their "SPV-mining" strategy.  The orphan rate would be higher, but this would apply for the other pools as well. Sometimes they would produce a block on the old parent that gets orphaned immediately, sometimes they orphan the block of another miner that didn't distribute fast enough.  Since China has most mining power, maybe the big Chinese pools would even profit from this higher orphan rate, when the difficulty is adjusted.

[TierNolan]

How much hashing power was supporting the fork?  I only saw AntMiner, F2Pool and BTCNugget, which have less than 50 % together.  Wouldn't the fork have stopped automatically when these pools ran out of "luck" and the remaining network caught up?  We'll probably never know who of the smaller pools would have supported the wrong chain.

I think they moved 2 blocks ahead by luck (the bad block was built on) and then were able to stay ahead for a while.

With 48% of the hashing power and a 2 block lead, it would take around 50 blocks for the 52% side to catch up, on average.

[trout]

do you guys  think that current ASICs or GPUs could speed up block validation?
Currently (as far as I'm aware) bitcoind  only uses CPUs.

May be including GPU support to bitcoind  it could solve the problem, by
simply making verification time negligible.

[TierNolan]

do you guys  think that current ASICs or GPUs could speed up block validation?
Currently (as far as I'm aware) bitcoind  only uses CPUs.

It isn't entirely validation speed.  It also includes network bandwidth to actually forward the block.

No matter how fast validation is, just validating the block header is going to be much faster.  Checking POW is very fast and sending 80 bytes of data is much faster than sending up to 1MB of data.

[trout]

do you guys  think that current ASICs or GPUs could speed up block validation?
Currently (as far as I'm aware) bitcoind  only uses CPUs.

It isn't entirely validation speed.  It also includes network bandwidth to actually forward the block.

I thought the block transmission time problem was solved by the relay network. The one that
allows miners that found a block  only transmit txs that are not in the mempool.
I'm not sure how widely it's used though.

No matter how fast validation is, just validating the block header is going to be much faster.  Checking POW is very fast and sending 80 bytes of data is much faster than sending up to 1MB of data.

sure, but if the validation time is reduced to milliseconds then it's a negligible expenditure

[tspacepilot]

To implement this attack, non-SPV-mining pools need to waste enough hash power to create a "bad" block. This is profitable to all of them together, but they have to cooperate.  Cooperation seems more difficult to achieve. THen again, if there is one single big pool, the insentive may be big enough for him to do it alone.

It's not difficult to calculate how big it should be; the main unknown paramter is for how long it sends the SPV-miners off to mind the wrong side of the fork.

I think this is the crucial calculation for someone who's running a big, non-spv mining pool.  From what I could tell, Danny H's attack seems like it has just been shown that it would work (we just saw it go down), and so the only real question is how much you'd have to "waste" by finding the bad blocks that you're going to throw at the spv-miners to send them down the wrong path.

[TierNolan]

I think this is the crucial calculation for someone who's running a big, non-spv mining pool.  From what I could tell, Danny H's attack seems like it has just been shown that it would work (we just saw it go down), and so the only real question is how much you'd have to "waste" by finding the bad blocks that you're going to throw at the spv-miners to send them down the wrong path.

With a timeout SPV-mining is actually good for the health of the network, since it drops the orphan rate to almost zero.  Miners should be encouraged to do it not discouraged.

If 50% of miners produce 10% of bad blocks, then the profits are as follows

Non-SPV coalition:
Output: 50% * 90% = 45%

SPV-coalition:
Bad previous block (10%)
1 minute out of 10 of wasted hashing
9 minutes out of 10 of valid hashing
Output: 50% * 90% * 10% = 4.5%

Good previous block (90%)
10 minutes of valid hashing
Output: 50% * 100% * 90% = 45%

Total outpout: 49.5%

The SPV effective miner's hash rate only drops from 50% to 49.5%.  The attacking coalition drops from 50% to 45%.  This means that the SPV-miners get 49.5/(49.5 + 45) = 52.4% of the blocks for 50% of the hashing power.

The attack causes the SPV-miners block rate to drop from 50% to 49.5% and then after the next difficulty adjustment they get 52.4% of the blocks.

The attackers go from 50% of the blocks to 45% of the blocks and then to 47.6% after the next difficulty update.

[unamis76]

I am also of the opinion that SPV mining should me deincentivized... To begin with, this will only result in pools trying to attack themselves, as DannyHamilton referred. Besides that, would we risk having non valid blocks and lots of forks in near future? This one wasn't all that serious, and devs were here to help at the time and immediately took action... But what if no one was monitoring?

And we also don't know about the future. there may be more strict blocks rules from now on and breaking them might make a huge fork... I don't think that's worth risking for more mining profit, or less wasted hashrate.

[TierNolan]

I am also of the opinion that SPV mining should me deincentivized... To begin with, this will only result in pools trying to attack themselves, as DannyHamilton referred. Besides that, would we risk having non valid blocks and lots of forks in near future? This one wasn't all that serious, and devs were here to help at the time and immediately took action... But what if no one was monitoring?

SPV mining is safe if there is a timeout (say 1 minute).

The problem was that the SPV-miners kept building more and more blocks on top of the invalid chain.  If they had a rule that said if the fully validated chain doesn't catch up with the SPV chain within 1 minute switch back to non-SPV mining, then the fork would never have happened.

The SPV-miners would have mined on the bad block for one minute and then switch back to mining on the good fork.