Bitcoin Core soft-fork proposal for strict DER enforcing

[gmaxwell]

Greetings, as suggested in the post about the recent OpenSSL incompatibility there is a proposal for strictly enforcing DER encoding for signatures as a block validity rule: http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg06744.html

Bitcoin Core has enforced this restriction for relay/mempool/mining since 0.8 in preparation for eventually making a change like this.

I know its kind of a boring change and boring changes tend to not gather a lot of comments; but it's still important so I'd greatly appreciate it if people would review and comment even if the comment is just "I read this, makes sense, no further comments.".  Ideally you'd comment on bitcoin-development directly, but if you comment here instead I can relay views (you take my editorial judgements at your own risk: best to comment directly).

[Nicolas Dorier]

Is there any cons worth discussing ? I only see the pros.

[gmaxwell]

Is there any cons worth discussing ? I only see the pros.

I believe all the cons we could uncover were hashed out in pre-proposal adjustments; so whats left is any unknowns people discover and just the general cost of doing something at all (which has to be weighed against the rather considerable and clear known costs/risks of doing nothing).

[Nicolas Dorier]

Well, "I read this, makes sense, no further comments.". I am already producing strict DER inside NBitcoin, so it won't affect already deployed users.

[jl2012]

Is there any cons worth discussing ? I only see the pros.

Let's consider this:

A bitcoin early adopter signed a time-locked transaction for his heir before he died. He believed that bitcoin would become really big and didn't want his heir spent it early. The transaction used a non-compliant signature but won't get confirmed until 2020. The private key was lost. Enforcing this rule will burn the locked bitcoin.

A possible solution is to enforce this rule only for new UTXO after a certain block height. For old UTXO, we will still follow the existing rules.

Am I just too paranoid?

This is not a problem because the signature could be fixed by malleability

[amaclin]

The transaction used a non-compliant signature but won't get confirmed until 2020. The private key was lost. Enforcing this rule will burn the locked bitcoin.

1) Isn't it possible to convert non-canonical signature to canonical leaving transaction valid? (Transaction not in block)
(Right now I do not think about the transactions which redeem outputs from the tx with non-canonical signature)

2) https://en.wikipedia.org/wiki/Omnipotence_paradox
We can not create algorithm/rule/law which can not be changed by our children
Just do not think about minority and time-locking transactions. You have no obligations either to early adopter or to his children

[dserrano5]

I'd greatly appreciate it if people would review and comment even if the comment is just "I read this, makes sense, no further comments.".

Well if it's really appreciated, "I read it and makes sense". But then I'm easily fooled .

[johoe]

The proposal looks good for me.

Since, BIP 62 (dealing with malleability) is mentioned in the draft, what is its status?  This also needs a soft fork. Will it be completely abandoned or will it be post-poned (to version 4 blocks) or will it be enabled at the same soft fork?

Provided that an agreement was already reached on BIP 62, I see no problem doing the required soft fork at the same time (except for the additional coding and testing involved).  Support for non-malleable transactions in clients can be added later when the network supports them.

[jl2012]

The transaction used a non-compliant signature but won't get confirmed until 2020. The private key was lost. Enforcing this rule will burn the locked bitcoin.

1) Isn't it possible to convert non-canonical signature to canonical leaving transaction valid? (Transaction not in block)
(Right now I do not think about the transactions which redeem outputs from the tx with non-canonical signature)

[/quote]

I'm not quite aware of the details of the proposed fork. If this is the case I think it's fine.

2) https://en.wikipedia.org/wiki/Omnipotence_paradox
We can not create algorithm/rule/law which can not be changed by our children
Just do not think about minority and time-locking transactions. You have no obligations either to early adopter or to his children

I do think we have obligations to all bitcoin users, to keep the promise in the original bitcoin protocol. Otherwise, in order to pump the price we may just declare all of Satoshi's early coins are invalid, for example.

[amaclin]

I do think we have obligations to all bitcoin users, to keep the promise in the original bitcoin protocol.

OK, you have obligations. I do not have. Who will be majority with 51% - wins the game.

Otherwise, in order to pump the price we may just declare all of Satoshi's early coins are invalid, for example.

We can do it. And we can recover lost coins https://bitcointalk.org/index.php?topic=50206
Just vote with 51% of hashing power for hard-fork.
(BTW, this will not pump price   )

[jl2012]

I do think we have obligations to all bitcoin users, to keep the promise in the original bitcoin protocol.

OK, you have obligations. I do not have. Who will be majority with 51% - wins the game.

Otherwise, in order to pump the price we may just declare all of Satoshi's early coins are invalid, for example.

We can do it. And we can recover lost coins https://bitcointalk.org/index.php?topic=50206
Just vote with 51% of hashing power for hard-fork.
(BTW, this will not pump price   )

You're wrong. To create a hard-fork you don't need 51%, not even 1% of hashing power. All alt-coins are hardforks of bitcoin

[amaclin]

You're wrong. To create a hard-fork you don't need 51%, not even 1% of hashing power. All alt-coins are hardforks of bitcoin

I mean "winning" hard-fork of course. The majority does not need dead hard-forks.

[Realpra]

DER should be removed it makes no sense for Bitcoin since Bitcoin has already defined the lengths and encodings of everything in its protocol.

Once you realize what DER actually is and what it does in the block its ridiculous.

[gmaxwell]

Am I just too paranoid?

Yes. Due to malleability invalid scriptSig are re-encodable as valid for any plausible signature type. This was explicitly considered and it's why BIP62 didn't impose changes on pubkey encoding requirements (e.g. requiring pubkeys be valid, or compressed), since whatever scriptPubKeys there are are set in stone. Height restricting strictder would completely undermine the motivation of the change, since consensus could be broken by anything spending an old coin.

One case that isn't covered is if you authored a chain of private-key-destroyed timelocked transactions using invalid encodings. You can fix-up the signature in the first transaction, but doing so changes its txid and thus invalidates its dependents.  Of course, if you did this you were already likely to get burned even absent any further change: anyone in the network could mutate your transaction (and with the invalid encoding, all the network would already ignore it and take the mutant instead) and invalidate the child equally well. There already appear to be some parties that mutate any invalid encodings they observe to valid ones.

In general, I think long term timelocks like this are suicidally ill advised... but indeed, I do consider it important to not soft-fork out the ability to spend old coins, even those of people who've done generally ill-advised things. Though changes have been made in the past which invalidated previously spendable scriptPubKeys, e.g. OP_RETURN being made invalid closed off some perfectly reasonable scriptPubKeys.

Provided that an agreement was already reached on BIP 62, I see no problem doing the required soft fork at the same time (except for the additional coding and testing involved).  Support for non-malleable transactions in clients can be added later when the network supports them.

Several of people who've been working on BIP 62 feel that its not mature enough yet for rapid deployment, as it is somewhat complicated and we've still found ourselves somewhat recently uncovering more interactions that were somewhat surprising. The strictder part of it is mature and hasn't been yielding any surprises and with that in place we can simplify BIP62 some, which will help get BIP62 deployed. It's still in progress and isn't being abandoned in the slightest.

I mean "winning" hard-fork of course. The majority does not need dead hard-forks.

You may misunderstand Bitcoin. If you are mining in violation of the network rules (e.g. on a hard fork) you are no longer mining as far as the nodes that disagree with you are concerned. 100% of the hashpower visible to a network is always in agreement with the hard fork rules.

Can't really claim to understand this fully so might be off the mark entirely.
Does this mean SSL is here to stay? I thought there where plans to move away from it and maybe even implement an alternative. Being unable to move away from SSL would be my only concern, otherwise sounds like a good move imho, +0.000001

SSL is used nowhere in the Bitcoin protocol. We do use the OpenSSL library to get access to some cryptographic functions and would eventually like to not do so, this change is a necessary step to make along that path. (e.g. if anything, the opposite of your fear).

DER should be removed it makes no sense for Bitcoin since Bitcoin has already defined the lengths and encodings of everything in its protocol.
Once you realize what DER actually is and what it does in the block its ridiculous.

Can't be removed without totally breaking incompatibility, making people's coins unspendable, etc. It's also more or less harmless at least once most of the possible complexity is removed by enforcing strict encoding, it just adds a couple bytes of overhead. Any future CHECKSIG alternatives obviously wouldn't use it; but there are still all the already existing coins left to deal with.

[luv2drnkbr]

One case that isn't covered is if you authored a chain of private-key-destroyed timelocked transactions using invalid encodings. You can fix-up the signature in the first transaction, but doing so changes its txid and thus invalidates its dependents.  Of course, if you did this you were already likely to get burned even absent any further change: anyone in the network could mutate your transaction (and with the invalid encoding, all the network would already ignore it and take the mutant instead) and invalidate the child equally well. There already appear to be some parties that mutate any invalid encodings they observe to valid ones.

Chiming in to say I agree with jl2012's proposal to have the implementation only enforced on UTXOs after a certain block height.  Even though it's not wise, and even though it's suicidal, and even though other people can already mutate the signatures, I very strongly think it's a bad precedent to set to invalidate a tx somebody made that was valid at the time they created it, and we should do everything in our power to keep the ability to spend that nlocked tx.  The core principle of Bitcoin is "a valid tx is a valid tx".  No matter how unlikely or irresponsible it is, and no matter that other people can already mutate it anyway, we should at least TRY to preserve the ability to have a tx that was valid at the time it was made stay valid, and have those child tx's also be valid if possible.  A valid tx is a valid tx.

[gmaxwell]

Chiming in to say I agree with jl2012's proposal to have the implementation only enforced on UTXOs after a certain block height.  Even though it's not wise, and even though it's suicidal, and even though other people can already mutate the signatures, I very strongly think it's a bad precedent to set to invalidate a tx somebody made that was valid at the time they created it, and we should do everything in our power to keep the ability to spend that nlocked tx.  The core principle of Bitcoin is "a valid tx is a valid tx".  No matter how unlikely or irresponsible it is, and no matter that other people can already mutate it anyway, we should at least TRY to preserve the ability to have a tx that was valid at the time it was made stay valid, and have those child tx's also be valid if possible.  A valid tx is a valid tx.

These were never valid, they were always an improper encoding. The existing network will not relay or mine them already. I have no evidence of anyone having created any such nlocked invalidly encoded signature, and if they did it could be fixed by anyone (the signature is not under the signature).  I believe you may have missed the point I made that height gating it would _completely_ _entirely_ and _totally_ defeat the point of the change; rendering it totally useless. It also wouldn't achieve the goal you set out, since someone might have created 'outputs' in the past that are not in the chain yet.

[jl2012]

One case that isn't covered is if you authored a chain of private-key-destroyed timelocked transactions using invalid encodings. You can fix-up the signature in the first transaction, but doing so changes its txid and thus invalidates its dependents.  Of course, if you did this you were already likely to get burned even absent any further change: anyone in the network could mutate your transaction (and with the invalid encoding, all the network would already ignore it and take the mutant instead) and invalidate the child equally well. There already appear to be some parties that mutate any invalid encodings they observe to valid ones.

Chiming in to say I agree with jl2012's proposal to have the implementation only enforced on UTXOs after a certain block height.  Even though it's not wise, and even though it's suicidal, and even though other people can already mutate the signatures, I very strongly think it's a bad precedent to set to invalidate a tx somebody made that was valid at the time they created it, and we should do everything in our power to keep the ability to spend that nlocked tx.  The core principle of Bitcoin is "a valid tx is a valid tx".  No matter how unlikely or irresponsible it is, and no matter that other people can already mutate it anyway, we should at least TRY to preserve the ability to have a tx that was valid at the time it was made stay valid, and have those child tx's also be valid if possible.  A valid tx is a valid tx.

For the reason given by gmaxwell I'd retract my proposal

[theymos]

I agree with the soft fork, though an annoying thing about this is that increasing the block version to 3 will cause everyone to get an "upgrade required" warning even though upgrading is not really required. It might cause systems to automatically shut down (due to alertnotify) or cause people to panic, and it increases the value of bitcoin.org as an attack target because everyone will be upgrading at the same time. I guess it can't be helped in this case, though maybe this warning should be removed or at least softened in future versions.

[gmaxwell]

I agree with the soft fork, though an annoying thing about this is that increasing the block version to 3 will cause everyone to get an "upgrade required" warning even though upgrading is not really required. It might cause systems to automatically shut down (due to alertnotify) or cause people to panic, and it increases the value of bitcoin.org as an attack target because everyone will be upgrading at the same time. I guess it can't be helped in this case, though maybe this warning should be removed or at least softened in future versions.

I think we now know a better way to do soft-forks in the future, but it was premature to deploy a new method for this change. Expect to see a BIP proposal on that.

On the warning front, I think we have something of a split preference here. One of the reasons that Mike Hearn has opposed soft forks in the past is an argument that your full node security could be silently downgraded.  The warning could be made less dire ("Your system may be enforcing a subset of the rules that the network may soon begin enforcing. Upgrading your software may be prudent."), and its timing could be somewhat randomized though.