Semi-soft-fork to decrease the risk of tx malleability

[Cryddit]

Forgive what may be an unusually dumb question, but how exactly does it work, that a third party can change the script but cannot change what the script does?  

EDIT:

Nevermind, I get it.  It's the spend script, not the store script.  If they change what it does then there is no transaction. 

[Realpra]

You're not correct, things are not that simple.

Malleability has big impact on any anything depending on transaction hash of unconfirmed transactions, e.g:

• Refund protocols: https://bitcointalk.org/index.php?topic=303088.msg11149909
• Spending outputs of unconfirmed transactions. Due to third party malleability, a transaction chain could be broken even if all payers are honest

I had not thought of that in this context.

Well your proposed solution of having different TX types where one is flexible and the other is not or something similar seems good to me.

[TierNolan]

I proposed this last year:

(1) The txid will be the hash of the tx with all scriptSig removed.
(currently, txid is the hash of the tx, including all scriptSig)

An alternative is to have both hashes be valid ways to refer to previous transactions.  This means that legacy transactions still work.  This is needed to ensure that you don't invalidate transactions that were created in the past, but have a locktime after the hard-fork happens.

If you are creating a refund transactions, then you can use hash(transaction without sigscripts) to refer to the previous transaction.

It doubles the size of the transaction index though, since it means 2 keys for each transaction.

(2) The first level of merkle root will be hash of (txid-a|size-a|txid-b|size-b), where txid-a and size-a are the txid and size of tx-a respectively
(currently, the first level of merkle root is (txid-a|txid-b))

What is the benefit of this?  It only protects against length changing malleability attacks (which might cover a lot of them).

[jl2012]

I proposed this last year:

(1) The txid will be the hash of the tx with all scriptSig removed.
(currently, txid is the hash of the tx, including all scriptSig)

An alternative is to have both hashes be valid ways to refer to previous transactions.  This means that legacy transactions still work.  This is needed to ensure that you don't invalidate transactions that were created in the past, but have a locktime after the hard-fork happens.

If you are creating a refund transactions, then you can use hash(transaction without sigscripts) to refer to the previous transaction.

It doubles the size of the transaction index though, since it means 2 keys for each transaction.

You only need legacy support for those UTXOs already in the blockchain. For those after the hardfork, only the new txid format will be supported.

(2) The first level of merkle root will be hash of (txid-a|size-a|txid-b|size-b), where txid-a and size-a are the txid and size of tx-a respectively
(currently, the first level of merkle root is (txid-a|txid-b))

What is the benefit of this?  It only protects against length changing malleability attacks (which might cover a lot of them).

We need this to provide a upper bound of the block size. Otherwise it is impossible to determine whether the block size is under 1MB (or other limit)

[funkenstein]

I'm still not so sure tx malleability is a real issue.  One can always check that the receiving address received payment from the sending address.  And what else is really so important?  In the recent proposal of payment channels (lighthouse) there is also talk that malleability could cause a problem.  I haven't understood how that is the case yet.  What am I missing?  

You're correct, TX maleability is only an issue if people use the TX hashes as Ids.

You're not correct, things are not that simple.

Malleability has big impact on any anything depending on transaction hash of unconfirmed transactions, e.g:

• Refund protocols: https://bitcointalk.org/index.php?topic=303088.msg11149909
• Spending outputs of unconfirmed transactions. Due to third party malleability, a transaction chain could be broken even if all payers are honest

Thanks for your reply!  And yours Cryddit.  

I see more clearly how this malleability thing can suck.  

It looks from your link like a partial solution is already proposed by GMaxwell.  An imperfect solution.  

However I see another potential solution: private TX mining.  

The trouble with malleability in the case of the refund situation is that the transaction has been broadcast publicly.  If I were to forego that and submit it privately to one of the large pools, the malleability attacks could never happen.  Sure it wouldn't be mined as quickly, but for these protocols that is probably not an issue.  

I'm sure if there were a demand large pools and/or consortiums of pools would publish pubic keys and accept TXs delivered encrypted.  After all, this means more fees for them even before they charge anything extra.  

In fact I'm a bit surprised nobody is doing that yet.  
  

[TierNolan]

You only need legacy support for those UTXOs already in the blockchain. For those after the hardfork, only the new txid format will be supported.

It depends on how much effort is put into maintaining backwards compatibility.

Someone could have a string of transactions starting with one with a long locktime.  You couldn't spend that if after the hard fork.

The easiest is probably just to have a version number update.  The txid for transactions with version > <some value> ignores the sigScripts.

We need this to provide a upper bound of the block size. Otherwise it is impossible to determine whether the block size is under 1MB (or other limit)

Ahh ok, thought it was related to malleability.  I agree, if there is a hard fork, that should be added.  I would also add the fee sum as a 2nd piece of meta-data.  That protects against inflation.

[TierNolan]

We'll be moving on 62 once 66 is actually deployed (one flaw in the the legacy softfork deployment mechanism is that only one change can be in flight at a time)

Is the plan BIP-66 then BIP-62 and then OP_CHECKLOCKTIMEVERIFY?

[jl2012]

We'll be moving on 62 once 66 is actually deployed (one flaw in the the legacy softfork deployment mechanism is that only one change can be in flight at a time)

Is the plan BIP-66 then BIP-62 and then OP_CHECKLOCKTIMEVERIFY?

Is there any consensus that CHECKLOCKTIMEVERIFY will be implemented??

[gmaxwell]

Is there any consensus that CHECKLOCKTIMEVERIFY will be implemented??

Everyone who has commented has spoken up in favor of it, and plenty of people have. So I think so.

Is the plan BIP-66 then BIP-62 and then OP_CHECKLOCKTIMEVERIFY?

66 is in flight and must finish before another softfork can be undertaken.  I'd like further softforks to switch to the new softforking system that doesn't require serializing things.

If you'd like to see faster progress; please go nag miners to upgrade to 0.10.1.

[TierNolan]

I'd like further softforks to switch to the new softforking system that doesn't require serializing things.

Is there a github branch and/or description of exactly what is proposed for that (or is the plan to just manually set what BIP each bit refers to)?

[Cryddit]

Is there any consensus that CHECKLOCKTIMEVERIFY will be implemented??

Everyone who has commented has spoken up in favor of it, and plenty of people have. So I think so.

OP_CHECKLOCKTIMEVERIFY makes it possible to make txouts that cannot be spent until after some block height.

One thing that's particularly beneficial about that is that it makes it much safer for a future BIP to create nLastTime - (by analogy with nLockTime) transactions that cannot be entered into a block AFTER a given block height.  This would be the "fill-or-kill" trade that a lot of people have asked for. 

If a tx that is not valid until after block X produces only outputs that are not spendable until block x+50, then any subsequent transactions invalidated by the tx becoming invalid in a reorg are transactions that would not be valid until 50 blocks later anyway - which removes the primary problem with nLastTime.

So If nLastTime transactions are required to produce no outputs spendable within 50 blocks after their last valid block, it's "safe" insofar as not invalidating subsequent transactions, barring a 50-block reorg.

[doug_armory]

Pieter has a new proposal which makes major improvements: every softfork is signaled by a single bit, the when the bit crossed threshold it 'latches' and becomes available for reuse again, and there is a deadline by which if it fails to latch its aborted and becomes available for use again.

Just curious, is this proposal documented anywhere? I spent a few minutes checking the usual suspects (mailing list, Reddit, this board, IRC logs, BIPs) and couldn't find anything. Maybe I missed it somehow?

Thanks.

[jl2012]

we have BIP66 but there are still many forms of malleability to be fixed.

BIP66 is not about malleability, really; BIP62 is.

Is this the real reason for BIP66, which you couldn't mention in April?

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html
(Disclosure: consensus bug indirectly solved by BIP66)

[gmaxwell]

we have BIP66 but there are still many forms of malleability to be fixed.

BIP66 is not about malleability, really; BIP62 is.

Is this the real reason for BIP66, which you couldn't mention in April?
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html
(Disclosure: consensus bug indirectly solved by BIP66)

The explicitly stated goal right in the BIP62 document was a reduction in dependence on OpenSSL for consensus; which the truth and nearly the whole truth all that was missing was that the concern was less theoretical than presented.  But yes, the unstated privately known consensus vulnerability was a driving force for clearing up that ugly corner with BIP66 immediately rather than mopping it up as a side effect of BIP62. Especially due to fact that BIP62's progress has been slow-- and had grown over complicated. The fact that BIP66 accomplished one of the necessary steps will make BIP62 or its logical successor easier to move forward.

If you note from the timeline that BIP62 had already been revised to accomplish a BIP66 like goal prior to us knowing about that specific vulnerability:  We were aware of the risk class in principle before having specific knoweldge with enough concern that we thought it was worth closing off.  Knowing that it was _actually_ exposed had only an impact on timing and sequencing; and only after it was clear that BIP62 was not rapidly converging (unfortunately, in spite of much progress on BIP62 we continued to find problems with it).

It's sad, there is a vastly superior-- simpler, more comprehensive, and with secondary benefits like enabling more degrees between full and SPV-- malleability 'fix' in elements alpha but I don't have any way to get it into Bitcoin, BIP62 is disappointing by comparison (brillant ideas welcome).

[sickpig]

we have BIP66 but there are still many forms of malleability to be fixed.

BIP66 is not about malleability, really; BIP62 is.

Is this the real reason for BIP66, which you couldn't mention in April?
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html
(Disclosure: consensus bug indirectly solved by BIP66)

The explicitly stated goal right in the BIP62 document was a reduction in dependence on OpenSSL for consensus; which the truth and nearly the whole truth all that was missing was that the concern was less theoretical than presented.  But yes, the unstated privately known consensus vulnerability was a driving force for clearing up that ugly corner with BIP66 immediately rather than mopping it up as a side effect of BIP62. Especially due to fact that BIP62's progress has been slow-- and had grown over complicated. The fact that BIP66 accomplished one of the necessary steps will make BIP62 or its logical successor easier to move forward.

If you note from the timeline that BIP62 had already been revised to accomplish a BIP66 like goal prior to us knowing about that specific vulnerability:  We were aware of the risk class in principle before having specific knoweldge with enough concern that we thought it was worth closing off.  Knowing that it was _actually_ exposed had only an impact on timing and sequencing; and only after it was clear that BIP62 was not rapidly converging (unfortunately, in spite of much progress on BIP62 we continued to find problems with it).

It's sad, there is a vastly superior-- simpler, more comprehensive, and with secondary benefits like enabling more degrees between full and SPV-- malleability 'fix' in elements alpha but I don't have any way to get it into Bitcoin, BIP62 is disappointing by comparison (brillant ideas welcome).

do you care to elaborate on that better fix? especially on the reason why it cannot be "ported" to bitcoin. thanks in advance.