Block malleability

[valiron]

Hi everyone,

This has surely been discussed before, so my apologies.

I want to know what is the freedom of block malleability. More precisely, if we look at the structure of block headers (as described in https://en.bitcoin.it/wiki/Block_hashing_algorithm ), which fields are malleable?

What I mean is what fields can I change arbitrarily, and the solved block will still be accepted by the network. For example, the "Time" is surely malleable and blocks with diverse "Time" fields will be accepted if solved.

Having a validated block with the right hash, in order to be accepted by the network, which fields of the header are checked? For example, can I put anything I like in the "Version" field? Obviously both the hash of the previous block and the Merkel tree of transactions must be verified. Anything else?

[uminatsu]

You can't touch any single bit in a block header (or anywhere else in the block) without invalidating the block hash. So answer is no.

[valiron]

You can't touch any single bit in a block header (or anywhere else in the block) without invalidating the block hash. So answer is no.

This is not what I am asking. Sorry, I didn't make myself clear enough.

I want to build the header and then solve the block. Can I put anything in the fields of the header in order to have the solved block accepted by the network? (provided that it gets solved on time)

Clearly there is some latitude in the "Time" field, and there is no latitude in the field containing the hash of the previous block.


(and, by the way, only the header is double hashed)

[uminatsu]

You could change fields in the block header as long as they satisfy the conditions mandated by the AcceptBlock() function (main.cpp: checks proof of work, timestamp against prev, all transactions are finalized, block version and a bunch other conditions). But doing so offers you no benefit whatsoever.

[gmaxwell]

Hi everyone,

This has surely been discussed before, so my apologies.

I want to know what is the freedom of block malleability. More precisely, if we look at the structure of block headers (as described in https://en.bitcoin.it/wiki/Block_hashing_algorithm ), which fields are malleable?

What I mean is what fields can I change arbitrarily, and the solved block will still be accepted by the network. For example, the "Time" is surely malleable and blocks with diverse "Time" fields will be accepted if solved.

Having a validated block with the right hash, in order to be accepted by the network, which fields of the header are checked? For example, can I put anything I like in the "Version" field? Obviously both the hash of the previous block and the Merkel tree of transactions must be verified. Anything else?

None of the fields are "arbitrary" except for the nonce.

Time has a range of accepted values, though the range has a fuzzy definition and setting it too high increases your probability of being orphaned without a completely sharp cutoff.  You can easily find a list of the block validity criteria.

Version is defined for forward capability. While it only has some constraints today, it is not a free form field: to be useful for forward compatibility future values it must not be constrained, but it also must not be set randomly. You could, if you wanted, begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists. If you do this I will circulate a patch to miners to orphan your blocks, however.

[jl2012]

begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists.

This attack won't work as the future softfork could ignore the version field of earlier blocks

If you do this I will circulate a patch to miners to orphan your blocks,

This is not needed and is very dangerous as the chain may fork

[gmaxwell]

begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists.

This attack won't work as the future softfork could ignore the version field of earlier blocks

I draw your attention to "if the usage persists". The problem is if some application comes to depend on coding crazy data there then it may be difficult to get the usage to cleanly stop in order to set the flag day point for the signaling.  So thats why a threat to start orphaning the things is effective: it doesn't matter if someone does it some, and if they are persistent some orphaning will make them stop before the usage becomes enshrined and breaks forward compatibility.

This is not needed and is very dangerous as the chain may fork

Not may fork, will fork. And thats fine. The chain forks ~every day. Unless the crazy version numbering miner had a (near-)majority of the hashpower the forking will rapidly resolve itself— and if such a miner did have a majority hashpower we'd have worse issues to worry about. It's also possible to 'discourage' a block— reject it until its burried at least two deep, disadvantaging it, but not producing a persistent fork in the case that it were a majority hashpower.

[valiron]

You could change fields in the block header as long as they satisfy the conditions mandated by the AcceptBlock() function (main.cpp: checks proof of work, timestamp against prev, all transactions are finalized, block version and a bunch other conditions). But doing so offers you no benefit whatsoever.

Thanks! This is the answer I was looking for.

Possible benefit? Instead of changing the extra nonce that requires computing again the Merkle tree you could touch the other "free" parameters.

[jl2012]

begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists.

This attack won't work as the future softfork could ignore the version field of earlier blocks

I draw your attention to "if the usage persists". The problem is if some application comes to depend on coding crazy data there then it may be difficult to get the usage to cleanly stop in order to set the flag day point for the signaling.  So thats why a threat to start orphaning the things is effective: it doesn't matter if someone does it some, and if they are persistent some orphaning will make them stop before the usage becomes enshrined and breaks forward compatibility.

This is not needed and is very dangerous as the chain may fork

Not may fork, will fork. And thats fine. The chain forks ~every day. Unless the crazy version numbering miner had a (near-)majority of the hashpower the forking will rapidly resolve itself— and if such a miner did have a majority hashpower we'd have worse issues to worry about. It's also possible to 'discourage' a block— reject it until its burried at least two deep, disadvantaging it, but not producing a persistent fork in the case that it were a majority hashpower.

I still think this approach is risky. Let say we now decide to orphan all blocks with version > 2. Every updated nodes (not just miners) will need to update again to reverse the ban. It's like a softfork.

It's similar to adding the 1MB block size limit, and now we get stuck

[valiron]

Hi everyone,

This has surely been discussed before, so my apologies.

I want to know what is the freedom of block malleability. More precisely, if we look at the structure of block headers (as described in https://en.bitcoin.it/wiki/Block_hashing_algorithm ), which fields are malleable?

What I mean is what fields can I change arbitrarily, and the solved block will still be accepted by the network. For example, the "Time" is surely malleable and blocks with diverse "Time" fields will be accepted if solved.

Having a validated block with the right hash, in order to be accepted by the network, which fields of the header are checked? For example, can I put anything I like in the "Version" field? Obviously both the hash of the previous block and the Merkel tree of transactions must be verified. Anything else?

None of the fields are "arbitrary" except for the nonce.

Time has a range of accepted values, though the range has a fuzzy definition and setting it too high increases your probability of being orphaned without a completely sharp cutoff.  You can easily find a list of the block validity criteria.

Thanks for the answer.

Version is defined for forward capability. While it only has some constraints today, it is not a free form field: to be useful for forward compatibility future values it must not be constrained, but it also must not be set randomly. You could, if you wanted, begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists. If you do this I will circulate a patch to miners to orphan your blocks, however.

Hey! I am not a terrorist! Silly you! You want to orphan my blocks!

I just don't understand why the nonce field is so small that requires using an extranonce in the coinbase transaction...etc... Wouldn't it have been simpler to have a large enough nonce for the purposes of hashing?


Also I don't understand why a full 4 Bytes are needed for Version...seems to me a waste of memory space...

[valiron]

begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists.

This attack won't work as the future softfork could ignore the version field of earlier blocks

I draw your attention to "if the usage persists". The problem is if some application comes to depend on coding crazy data there then it may be difficult to get the usage to cleanly stop in order to set the flag day point for the signaling.  So thats why a threat to start orphaning the things is effective: it doesn't matter if someone does it some, and if they are persistent some orphaning will make them stop before the usage becomes enshrined and breaks forward compatibility.

This is not needed and is very dangerous as the chain may fork

Not may fork, will fork. And thats fine. The chain forks ~every day. Unless the crazy version numbering miner had a (near-)majority of the hashpower the forking will rapidly resolve itself— and if such a miner did have a majority hashpower we'd have worse issues to worry about. It's also possible to 'discourage' a block— reject it until its burried at least two deep, disadvantaging it, but not producing a persistent fork in the case that it were a majority hashpower.

In your scenario, wouldn't it to be disruptive to have a block with weird version number from time to time? Will you fork the blockchain for just one block with bad version number? I doubt it...and once it is buried with several confirmations you won't fork.

[jl2012]

I just don't understand why the nonce field is so small that requires using an extranonce in the coinbase transaction...etc... Wouldn't it have been simpler to have a large enough nonce for the purposes of hashing?

It might encourage miners to regularly add new transactions to the unsolved block, as anyway they have to change to Merkle Root after the nonce is exhausted.

[valiron]

I just don't understand why the nonce field is so small that requires using an extranonce in the coinbase transaction...etc... Wouldn't it have been simpler to have a large enough nonce for the purposes of hashing?

It might encourage miners to regularly add new transactions to the unsolved block, as anyway they have to change to Merkle Root after the nonce is exhausted.

IMHO the encouragement to add new transactions should come from the transaction fee, not from a technical issue.

To change the Merkle root costs computation time. Anyone has computed up to what level of tx fees is no longer economically cost effective to update the merkle tree?

[gmaxwell]

Possible benefit? Instead of changing the extra nonce that requires computing again the Merkle tree you could touch the other "free" parameters.

You already get a factor of 4 billion speedup from just the nonce alone, you can update the extra nonce with no more than 12 compression function calls. It's good to update the block in any case.

[TierNolan]

Version is defined for forward capability. While it only has some constraints today, it is not a free form field: to be useful for forward compatibility future values it must not be constrained, but it also must not be set randomly. You could, if you wanted, begin to attack the network by signaling other values in the version field— disrupting its use for forward compatibility if the usage persists. If you do this I will circulate a patch to miners to orphan your blocks, however.

While large pools have probably modified their clients anyway, it might be worth adding a -max_version command line option.  It would default to INT_MAX.

If it is set, then only blocks with versions of 0 <= version <= max_version would be accepted.  

This would mean that p2pool miners could easily update their nodes if the block version is updated.

If the code could be setup so that it only affects mining, then it wouldn't cause fork problems.

A user who only uses their client for transactions would be unaffected.

I still think this approach is risky. Let say we now decide to orphan all blocks with version > 2. Every updated nodes (not just miners) will need to update again to reverse the ban. It's like a softfork.

It's similar to adding the 1MB block size limit, and now we get stuck

Right, it would be critical that it is a "miner-only" patch, rather than an update to the general public's clients.

Another option would be to include updating the version number in the protocol.

If 750 of the last 1000 nodes include /VER_INC_BOTH/ in the coinbase, then the maximum version for transactions and blocks is increased by 1 from that point on.  (/VER_INV_BLOCK/ and /VER_INV_TX/ could be used to increase one or the other).