Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: jl2012 on April 21, 2015, 05:04:11 AM



Title: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 21, 2015, 05:04:11 AM
tx malleability could cause big trouble. I think all devs agree that we have to fix it some point in the future. However, due to the risk involving soft-fork, this could only be done very slowly. Now we have BIP66 but there are still many forms of malleability to be fixed.

I propose to have to a technique called "semi-soft-fork" as a remedy before a real soft-fork is done. Currently, transactions are divided into standard and non-standard. I propose to divide it into 3 types:

  • Standard tx (ST): a valid tx with a strict set of scriptSig and scriptPubKey, with all known malleability issues fixed
  • Type 1 non-standard tx (NST1): any mutated standard tx, otherwise valid
  • Type 2 non-standard tx (NST2): any other valid tx not in the previous categories

A block with only ST is a standard block (SB). A block with at least 1 NST1 non-standard tx is a Type 1 non-standard block (NSB1). A block with no NST1 but with at least 1 NST2 is a Type 2 non-standard block (NSB2). SB, NSB1 and NSB2 are all valid blocks, just with different level of standardness

Miners joining this semi-soft-fork will still try to mine the longest chain, no matter the blocks in it are SB, NSB1, or NSB2. However, if there are forks with same length, the miner will always switch to the fork with least number of NSB1.

If there are enough miners joining this semi-soft-fork, it will provide incentive for the rest of miners to avoid NST1 and NSB1. Therefore, the risk of tx malleability is reduced. And since this is not a consensus rule change, it is easily reversible if anything goes wrong. It will also allow us to test the anti-malleability code before we migrate to a real soft-fork.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: funkenstein on April 21, 2015, 05:16:30 AM
I'm still not so sure tx malleability is a real issue.  One can always check that the receiving address received payment from the sending address.  And what else is really so important?  In the recent proposal of payment channels (lighthouse) there is also talk that malleability could cause a problem.  I haven't understood how that is the case yet.  What am I missing? 


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Amph on April 21, 2015, 06:51:42 AM
wasn't this issued with an update? i remember that was a big excuse used by gox for their loss, and others exchangee has started to take it as an excuse also(minus cryptsy)


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on April 21, 2015, 09:32:54 AM
we have BIP66 but there are still many forms of malleability to be fixed.
BIP66 is not about malleability, really; BIP62 is.

What you're describing has previously been described as "Block discouragement";  here I think it has all the software complexity of a normal soft-fork plus the additional complexity of the discouragement mechanism.

I do not see how this helps much; the reversibility is a selling point, but at a far from zero cost. We'll be moving on 62 once 66 is actually deployed (one flaw in the the legacy softfork deployment mechanism is that only one change can be in flight at a time)


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 21, 2015, 10:14:30 AM
we have BIP66 but there are still many forms of malleability to be fixed.
BIP66 is not about malleability, really; BIP62 is.
But in the BIP66 description it says it "has the added benefit of reducing transaction malleability":

Quote
every non-compliant signature can trivially be converted into a compliant one, so there is no loss of functionality by this requirement. This proposal has the added benefit of reducing transaction malleability (see BIP 62).

And if a non-compliant signature can trivially be converted into a compliant one (I assume private key is not needed for such conversion?), why isn't it a source of malleability?

What you're describing has previously been described as "Block discouragement";  here I think it has all the software complexity of a normal soft-fork plus the additional complexity of the discouragement mechanism.

Most block discouragement rules should eventually become soft-fork rules. So the code should be pretty much reusable. The code for discouragement mechanism is also reusable in the future.

I do not see how this helps much; the reversibility is a selling point, but at a far from zero cost. We'll be moving on 62 once 66 is actually deployed (one flaw in the the legacy softfork deployment mechanism is that only one change can be in flight at a time)

As mining becomes more professional and competitive, miners will try everything to reduce the stale rate. It's not zero cost but most work done are reusable. And the selling point is low risk, not low cost.

Can we deploy 2 (or more) soft-fork concurrently like this: block version 3 is BIP62 only; block version 4 is BIP66 only; block version 5 is both BIP62 and 66?


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on April 21, 2015, 05:17:08 PM
But in the BIP66 description it says it "has the added benefit of reducing transaction malleability":
Sure, its a forward move of one of the BIP62 features; but "reducing" doesn't improve anything; it's not like malleability happens by chance.

Quote
And if a non-compliant signature can trivially be converted into a compliant one (I assume private key is not needed for such conversion?), why isn't it a source of malleability?
Because only the single canonical form is permitted in the blockchain.

Quote
And the selling point is low risk, not low cost.
The amount of code that must be written and confirmed to be correct is the risk, and the risk is the cost.

An ordinary soft-fork is not especially risky so long as the work is done to be confident that its engineered correctly; and virtually all the risk comes from the definition of the restriction itself; there was an inflow of virtually no review or commentary for BIP66.

Quote
Can we deploy 2 (or more) soft-fork concurrently like this: block version 3 is BIP62 only; block version 4 is BIP66 only; block version 5 is both BIP62 and 66?
Not with the current definition for the mechanism used in BIP62. Version 5 would enable BIP 62 on existing nodes.

Pieter has a new proposal which makes major improvements: every softfork is signaled by a single bit, the when the bit crossed threshold it 'latches' and
becomes available for reuse again, and there is a deadline by which if it fails to latch its aborted and becomes available for use again.

We wanted to use that for BIP62 but it would have delayed the process.

Keep in mind, that for refund protocols where the potential attacker is a signer of the transaction no "fix" in the form of signature encoding improvements is possible; their problem is not that transactions are malleable but that the malleability matters. BIP66 is about preventing (we hope) third party malleability.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 21, 2015, 06:08:52 PM

Pieter has a new proposal which makes major improvements: every softfork is signaled by a single bit, the when the bit crossed threshold it 'latches' and
becomes available for reuse again, and there is a deadline by which if it fails to latch its aborted and becomes available for use again.


But there are only 32 bits in the version. It will be exhausted very soon if it is used in this way, isn't it?


Keep in mind, that for refund protocols where the potential attacker is a signer of the transaction no "fix" in the form of signature encoding improvements is possible; their problem is not that transactions are malleable but that the malleability matters. BIP66 is about preventing (we hope) third party malleability.


By third party malleability do you mean the ability of mutating a tx without private key? I think this is the only type of malleability we may fix. Private key holder will always be able to mutate and double spend a tx before it is confirmed.

I'm not sure if I misunderstand you. In the refund protocol, the potential attacker (i.e. the payee) does not need to sign the payment to the escrow address. It should work like this:

1. Merchant generates a new private key, and send the public key to the customer.
2. Customer creates a 2-of-2 P2SH address with his own public key and merchant's public key
3. Customer creates a tx, sending some bitcoin to the P2SH address from step 2, but not publishing the tx
4. Customer asks the merchant to sign a nlocktime refund tx with the outpoint from step 3.
5. Customer publishes the tx from step 3.
6. After the tx from step 3 is confirmed, customer shows the script to the merchant to prove that the bitcoin is under escrow.

Customer may lose money if tx mutation happens between step 5 and 6. However, if we fixed all known third-party malleability, the customer is the only person having to ability to mutate it but he has absolutely no incentive to do it.



Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on April 21, 2015, 06:18:25 PM
It will be exhausted very soon if it is used in this way, isn't it?
No, because the bit becomes available for use again after the feature latches or passes the deadline without latching;  So up to 31 features in flight at once, and no total limit (though we might reduce it to just 15 of the bits being used in this way).

If the soft-fork rules are consistent even for yet undefined future soft-forks then even a non-upgraded client can tell what the state of the softfork is even without knowing its specifics.

Quote
I'm not sure if I misunderstand you. In the refund protocol, the potential attacker (i.e. the payee) does not need to sign the payment to the escrow address. It should work like this:
Depends on what you mean by refund protocol. As soon as your pattern makes multiple transactions-- e.g. the escrow makes change back to the escrow, things are broken again.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 21, 2015, 08:55:35 PM
No, because the bit becomes available for use again after the feature latches or passes the deadline without latching;  So up to 31 features in flight at once, and no total limit (though we might reduce it to just 15 of the bits being used in this way).

So, version becomes non-monotonic?

What about 2 bytes for "count" and 2 bytes for softforks.

If a soft-fork is accepted, the "count" part of the version increases by 1.

If the count is made the top 2 bytes, then the version will increase as each soft fork is accepted.

Quote
If the soft-fork rules are consistent even for yet undefined future soft-forks then even a non-upgraded client can tell what the state of the softfork is even without knowing its specifics.

A warning to update has the risk of everyone updating their client at the same time.

If the consensus lib was based on a virtual CPU with bytecode, the new soft rule could potentially be added as part of the process.  That might make it too easy to add soft forks though.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on April 21, 2015, 10:24:26 PM
So, version becomes non-monotonic?
Correct. And why shouldn't it be? It's just data. An expectation of monotonicity is not very compatible with decentralization in any case.

Quote
What about 2 bytes for "count" and 2 bytes for softforks.
If a soft-fork is accepted, the "count" part of the version increases by 1.
If the count is made the top 2 bytes, then the version will increase as each soft fork is accepted.
What would this gain? You can already count the acceptance from the flags, even if you don't know what they mean.

We'd want to use the remaining bits for extra information that needs to be judged statelessly, without seeing the prior headers. (there are very few cases where this matters, but e.g. additional POW constraints, for soft-forking in an additional POW, would be an example).

Quote
A warning to update has the risk of everyone updating their client at the same time.
Not a warning to update-- but notice that the network is imposing rules that your software doesn't understand, meaning that it might accept a fork the overwhelming majority of the hashpower on the chain you're on would not accept.   Such a notice would only happen around the point the new rules went into effect in any case; e.g. once its already widely deployed.

Quote
If the consensus lib was based on a virtual CPU with bytecode, the new soft rule could potentially be added as part of the process.  That might make it too easy to add soft forks though.

That could only safely work for rules that didn't lock in, since they could only really safely be applied in the context of a single chain.  Keep in mind, a big reason for the rules a node imposes exist to create a reason for miners to behave honestly; the purpose of it is to constrain their behavior, so giving them control over it doesn't make sense.  A soft-fork isn't a vote; its safe coordination for activation of a change that people already (almost certainly) consent to. The whole reason to have it isn't to decide to do it or not, but to make sure that everyone imposes it at the same time.



Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 22, 2015, 10:31:36 AM
Depends on what you mean by refund protocol. As soon as your pattern makes multiple transactions-- e.g. the escrow makes change back to the escrow, things are broken again.

Then just make it be escrow -> customer -> escrow.

Or to fix it permanently, we may have a hardfork (or a softfork in a creative way) to replace txid with normalized-txid. This will ensure that a safe subset of tx (SIGHASH_ALL and pay-to-key-hash/pay-to-pubkey) will be totally free of malleability.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 22, 2015, 04:33:00 PM
Correct. And why shouldn't it be? It's just data. An expectation of monotonicity is not very compatible with decentralization in any case.

It would make it easier to do the soft fork checks in code. 

Code:
// BIP - XYZ - Soft fork check
If (block.version >> 16 >= 7) {
   // Do check
}

Though probably the easiest is to just hard code the hash for a block after the change and set all blocks before that as automatically valid.

Quote
That could only safely work for rules that didn't lock in, since they could only really safely be applied in the context of a single chain.  Keep in mind, a big reason for the rules a node imposes exist to create a reason for miners to behave honestly; the purpose of it is to constrain their behavior, so giving them control over it doesn't make sense.

It would still need 95%+ support for miners.

It would be a way for miners to coordinate more easily though.

The proposal could include some way to actually designate what the new bits actually mean.  For example, the miner could include "magic_number | 0xFF | hash(BIP text)" in the coinbase.  If 10 people have already done that in the last 1000 blocks, then the 10th person can replace 0xFF with the bit index they want to use for the new BIP.  It is like proposing and seconding for consideration.

The rule for the last 1000 blocks could be

< 10%: Bit reverts to unused [*]
> 75%: Activate rule for blocks with the bit set
> 95%: Activate rule for all blocks

[*]: Blocks with multiple bits set could as 1/N blocks per bit (N = number of bits)

A pool (or group) with >10% of the mining power would be able to get a BIP considered, but couldn't lockout all 16.  The division rule could be activated if to many bits were active.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on April 22, 2015, 05:42:02 PM
It would make it easier to do the soft fork checks in code. 
It would make it easier to do them wrong.  You're expecting change A, but the network has adopted B.  You're now confused.  That kind of simplistic check is only safe with a centralized process providing a universal order of changes; and to provide that non-safety it has to use up a fair amount of precious space.

Compared to the rest of the complexity of checking headers I don't see keeping 15 counters a list of completed changes to be a major barrier.

Quote
It would still need 95%+ support for miners.
It's much easier to get 95% hashrate (not that this isn't 95% people or anything like that) for a very profitable blank check.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 22, 2015, 06:48:16 PM
It would make it easier to do them wrong.  You're expecting change A, but the network has adopted B.

I was thinking implementations after the fork has been accepted for a while.  The threshold could be based on block height to give the same benefit in that case though.

The consensus lib would still need the full set of rules.  There does need to be some way to tie bits to particular BIPs though.  A major re-org could mean that the software can't figure out which rules are in effect.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Cryddit on April 22, 2015, 07:09:20 PM
The only problem I see with the per-bit masking of block version is old clients that misinterpret the bits without realizing that the same bit they expect to mean A, now means B.  

One way to avoid this is for the old clients to know at what block height their knowledge of the meaning of the bits was current; then when checking the block chain, if they see more than one on-off cycle during the block chain since that point they are aware that they no longer know the meaning of a bit.



Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 22, 2015, 10:07:21 PM
One way to avoid this is for the old clients to know at what block height their knowledge of the meaning of the bits was current; then when checking the block chain, if they see more than one on-off cycle during the block chain since that point they are aware that they no longer know the meaning of a bit.

The first objective is to allow old/obsolete clients know that a soft fork has occured.  It doesn't matter what the bits are defined as.  If after the last known block, there is 950 out of the last 1000 blocks with a version bit set, then the soft fork flag can be set.

The second objective is to allow multiple soft forks to run at the same time.  The process followed could be that the reference client is updated with the soft fork rule and then is allowed to run with it being part of the reference client for 6 months.  The bit could be recovered by having a max block height rule.  If the soft fork doesn't activate before a certain block height, then it is permenently disabled.  This prevents the bit accidently activating the rule later.  If the rule is not accepted, then it is removed/amended in the reference client for the next release.

The inherent assumption is that major re-orgs don't happen.  Each new soft-fork acts as a checkpoint. 


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Realpra on April 23, 2015, 09:36:46 AM
I'm still not so sure tx malleability is a real issue.  One can always check that the receiving address received payment from the sending address.  And what else is really so important?  In the recent proposal of payment channels (lighthouse) there is also talk that malleability could cause a problem.  I haven't understood how that is the case yet.  What am I missing?  
You're correct, TX malleability is only an issue if people use the TX hashes as Ids.

This is because a script saying 1=1 can be changed to saying 1=1=1 and it will still be correct (gross simplification), but the hash of the entire TX changes.

Obviously scripts cannot sign themselves because it would change the script.


"Fixing" malleability would also mean making the script system less flexible which I am against.

It has even been shown that very few of the losses of MtGox could be blamed on TX malleability - it was a total lie.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 23, 2015, 09:43:08 AM
I'm still not so sure tx malleability is a real issue.  One can always check that the receiving address received payment from the sending address.  And what else is really so important?  In the recent proposal of payment channels (lighthouse) there is also talk that malleability could cause a problem.  I haven't understood how that is the case yet.  What am I missing? 
You're correct, TX maleability is only an issue if people use the TX hashes as Ids.



You're not correct, things are not that simple.

Malleability has big impact on any anything depending on transaction hash of unconfirmed transactions, e.g:
  • Refund protocols: https://bitcointalk.org/index.php?topic=303088.msg11149909
  • Spending outputs of unconfirmed transactions. Due to third party malleability, a transaction chain could be broken even if all payers are honest


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Cryddit on April 23, 2015, 04:09:43 PM
You're correct, TX maleability is only an issue if people use the TX hashes as Ids.

The problem with not using tx hashes as a transaction ID, is that tx hashes are exactly what the block chain itself uses as a tx ID when specifying what transaction's output to use as an input.  If you don't know the transaction hash in advance, you can't make a transaction that spends its output in advance.  And if you can't make transactions that spend outputs before the outputs they spend actually get into the block chain, then a lot of escrow and other protocols don't work. 

We need transaction ID's to be stable.  Can we make the tx hash by exclusively hashing things that nobody outside the original transaction set can change? 


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 23, 2015, 06:00:28 PM
You're correct, TX maleability is only an issue if people use the TX hashes as Ids.

The problem with not using tx hashes as a transaction ID, is that tx hashes are exactly what the block chain itself uses as a tx ID when specifying what transaction's output to use as an input.  If you don't know the transaction hash in advance, you can't make a transaction that spends its output in advance.  And if you can't make transactions that spend outputs before the outputs they spend actually get into the block chain, then a lot of escrow and other protocols don't work. 

We need transaction ID's to be stable.  Can we make the tx hash by exclusively hashing things that nobody outside the original transaction set can change? 

Yes, but that would need a hardfork, or a very complicated softfork.

I proposed this last year:

(1) The txid will be the hash of the tx with all scriptSig removed.
(currently, txid is the hash of the tx, including all scriptSig)

(2) The first level of merkle root will be hash of (txid-a|size-a|txid-b|size-b), where txid-a and size-a are the txid and size of tx-a respectively
(currently, the first level of merkle root is (txid-a|txid-b))

With (1), no third party mutation of the txid is possible. Even when spending n-of-m multi-sig output, the txid is mutable only if n signers agree (currently, any 1 signers may mutate it)

However, due to scriptSig malleability, there is an infinite number of ways to record the same tx. Therefore, we need to put the tx size in the formula of the merkle root. For a tx to be valid, it must be equal or smaller than the size indicated. This guarantees that blocks in different nodes will have an upper size limit. However, the actual content the txs could be different, as long as they are valid.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Cryddit on April 23, 2015, 07:45:16 PM
Forgive what may be an unusually dumb question, but how exactly does it work, that a third party can change the script but cannot change what the script does?  

EDIT:

Nevermind, I get it.  It's the spend script, not the store script.  If they change what it does then there is no transaction. 


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Realpra on April 24, 2015, 10:39:17 AM
You're not correct, things are not that simple.

Malleability has big impact on any anything depending on transaction hash of unconfirmed transactions, e.g:
  • Refund protocols: https://bitcointalk.org/index.php?topic=303088.msg11149909
  • Spending outputs of unconfirmed transactions. Due to third party malleability, a transaction chain could be broken even if all payers are honest
I had not thought of that in this context.

Well your proposed solution of having different TX types where one is flexible and the other is not or something similar seems good to me.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 24, 2015, 11:06:36 AM
I proposed this last year:

(1) The txid will be the hash of the tx with all scriptSig removed.
(currently, txid is the hash of the tx, including all scriptSig)

An alternative is to have both hashes be valid ways to refer to previous transactions.  This means that legacy transactions still work.  This is needed to ensure that you don't invalidate transactions that were created in the past, but have a locktime after the hard-fork happens.

If you are creating a refund transactions, then you can use hash(transaction without sigscripts) to refer to the previous transaction.

It doubles the size of the transaction index though, since it means 2 keys for each transaction.

Quote
(2) The first level of merkle root will be hash of (txid-a|size-a|txid-b|size-b), where txid-a and size-a are the txid and size of tx-a respectively
(currently, the first level of merkle root is (txid-a|txid-b))

What is the benefit of this?  It only protects against length changing malleability attacks (which might cover a lot of them).


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 24, 2015, 01:18:01 PM
I proposed this last year:

(1) The txid will be the hash of the tx with all scriptSig removed.
(currently, txid is the hash of the tx, including all scriptSig)

An alternative is to have both hashes be valid ways to refer to previous transactions.  This means that legacy transactions still work.  This is needed to ensure that you don't invalidate transactions that were created in the past, but have a locktime after the hard-fork happens.

If you are creating a refund transactions, then you can use hash(transaction without sigscripts) to refer to the previous transaction.

It doubles the size of the transaction index though, since it means 2 keys for each transaction.

You only need legacy support for those UTXOs already in the blockchain. For those after the hardfork, only the new txid format will be supported.

Quote
(2) The first level of merkle root will be hash of (txid-a|size-a|txid-b|size-b), where txid-a and size-a are the txid and size of tx-a respectively
(currently, the first level of merkle root is (txid-a|txid-b))

What is the benefit of this?  It only protects against length changing malleability attacks (which might cover a lot of them).

We need this to provide a upper bound of the block size. Otherwise it is impossible to determine whether the block size is under 1MB (or other limit)


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: funkenstein on April 24, 2015, 03:18:39 PM
I'm still not so sure tx malleability is a real issue.  One can always check that the receiving address received payment from the sending address.  And what else is really so important?  In the recent proposal of payment channels (lighthouse) there is also talk that malleability could cause a problem.  I haven't understood how that is the case yet.  What am I missing?  
You're correct, TX maleability is only an issue if people use the TX hashes as Ids.



You're not correct, things are not that simple.

Malleability has big impact on any anything depending on transaction hash of unconfirmed transactions, e.g:
  • Refund protocols: https://bitcointalk.org/index.php?topic=303088.msg11149909
  • Spending outputs of unconfirmed transactions. Due to third party malleability, a transaction chain could be broken even if all payers are honest

Thanks for your reply!  And yours Cryddit.  

I see more clearly how this malleability thing can suck.  

It looks from your link like a partial solution is already proposed by GMaxwell.  An imperfect solution.  

However I see another potential solution: private TX mining.  

The trouble with malleability in the case of the refund situation is that the transaction has been broadcast publicly.  If I were to forego that and submit it privately to one of the large pools, the malleability attacks could never happen.  Sure it wouldn't be mined as quickly, but for these protocols that is probably not an issue.  

I'm sure if there were a demand large pools and/or consortiums of pools would publish pubic keys and accept TXs delivered encrypted.  After all, this means more fees for them even before they charge anything extra.  

In fact I'm a bit surprised nobody is doing that yet.  
  



Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 24, 2015, 06:37:13 PM
You only need legacy support for those UTXOs already in the blockchain. For those after the hardfork, only the new txid format will be supported.

It depends on how much effort is put into maintaining backwards compatibility.

Someone could have a string of transactions starting with one with a long locktime.  You couldn't spend that if after the hard fork.

The easiest is probably just to have a version number update.  The txid for transactions with version > <some value> ignores the sigScripts.

Quote
We need this to provide a upper bound of the block size. Otherwise it is impossible to determine whether the block size is under 1MB (or other limit)

Ahh ok, thought it was related to malleability.  I agree, if there is a hard fork, that should be added.  I would also add the fee sum as a 2nd piece of meta-data.  That protects against inflation.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 28, 2015, 04:10:37 PM
We'll be moving on 62 once 66 is actually deployed (one flaw in the the legacy softfork deployment mechanism is that only one change can be in flight at a time)

Is the plan BIP-66 then BIP-62 and then OP_CHECKLOCKTIMEVERIFY?


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on April 28, 2015, 05:48:26 PM
We'll be moving on 62 once 66 is actually deployed (one flaw in the the legacy softfork deployment mechanism is that only one change can be in flight at a time)

Is the plan BIP-66 then BIP-62 and then OP_CHECKLOCKTIMEVERIFY?

Is there any consensus that CHECKLOCKTIMEVERIFY will be implemented??


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on April 29, 2015, 05:06:43 AM
Is there any consensus that CHECKLOCKTIMEVERIFY will be implemented??
Everyone who has commented has spoken up in favor of it, and plenty of people have. So I think so.

Is the plan BIP-66 then BIP-62 and then OP_CHECKLOCKTIMEVERIFY?
66 is in flight and must finish before another softfork can be undertaken.  I'd like further softforks to switch to the new softforking system that doesn't require serializing things.

If you'd like to see faster progress; please go nag miners to upgrade to 0.10.1.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: TierNolan on April 29, 2015, 08:41:03 PM
I'd like further softforks to switch to the new softforking system that doesn't require serializing things.

Is there a github branch and/or description of exactly what is proposed for that (or is the plan to just manually set what BIP each bit refers to)?


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: Cryddit on April 30, 2015, 09:59:20 PM
Is there any consensus that CHECKLOCKTIMEVERIFY will be implemented??

Everyone who has commented has spoken up in favor of it, and plenty of people have. So I think so.


OP_CHECKLOCKTIMEVERIFY makes it possible to make txouts that cannot be spent until after some block height.

One thing that's particularly beneficial about that is that it makes it much safer for a future BIP to create nLastTime - (by analogy with nLockTime) transactions that cannot be entered into a block AFTER a given block height.  This would be the "fill-or-kill" trade that a lot of people have asked for. 

If a tx that is not valid until after block X produces only outputs that are not spendable until block x+50, then any subsequent transactions invalidated by the tx becoming invalid in a reorg are transactions that would not be valid until 50 blocks later anyway - which removes the primary problem with nLastTime.

So If nLastTime transactions are required to produce no outputs spendable within 50 blocks after their last valid block, it's "safe" insofar as not invalidating subsequent transactions, barring a 50-block reorg.



Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: doug_armory on May 06, 2015, 02:52:50 AM
Pieter has a new proposal which makes major improvements: every softfork is signaled by a single bit, the when the bit crossed threshold it 'latches' and becomes available for reuse again, and there is a deadline by which if it fails to latch its aborted and becomes available for use again.

Just curious, is this proposal documented anywhere? I spent a few minutes checking the usual suspects (mailing list, Reddit, this board, IRC logs, BIPs) and couldn't find anything. Maybe I missed it somehow?

Thanks.


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: jl2012 on July 29, 2015, 04:04:12 AM
we have BIP66 but there are still many forms of malleability to be fixed.
BIP66 is not about malleability, really; BIP62 is.



Is this the real reason for BIP66, which you couldn't mention in April?

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html
(Disclosure: consensus bug indirectly solved by BIP66)



Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: gmaxwell on July 29, 2015, 05:21:54 AM
we have BIP66 but there are still many forms of malleability to be fixed.
BIP66 is not about malleability, really; BIP62 is.
Is this the real reason for BIP66, which you couldn't mention in April?
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html
(Disclosure: consensus bug indirectly solved by BIP66)
The explicitly stated goal right in the BIP62 document (https://github.com/bitcoin/bips/blob/master/bip-0066.mediawiki#Motivation) was a reduction in dependence on OpenSSL for consensus; which the truth and nearly the whole truth all that was missing was that the concern was less theoretical than presented.  But yes, the unstated privately known consensus vulnerability was a driving force for clearing up that ugly corner with BIP66 immediately rather than mopping it up as a side effect of BIP62. Especially due to fact that BIP62's progress has been slow-- and had grown over complicated. The fact that BIP66 accomplished one of the necessary steps will make BIP62 or its logical successor easier to move forward.

If you note from the timeline that BIP62 had already been revised to accomplish a BIP66 like goal prior to us knowing about that specific vulnerability:  We were aware of the risk class in principle before having specific knoweldge with enough concern that we thought it was worth closing off.  Knowing that it was _actually_ exposed had only an impact on timing and sequencing; and only after it was clear that BIP62 was not rapidly converging (unfortunately, in spite of much progress on BIP62 we continued to find problems with it).

It's sad, there is a vastly superior-- simpler, more comprehensive, and with secondary benefits like enabling more degrees between full and SPV-- malleability 'fix' in elements alpha but I don't have any way to get it into Bitcoin, BIP62 is disappointing by comparison (brillant ideas welcome).


Title: Re: Semi-soft-fork to decrease the risk of tx malleability
Post by: sickpig on July 29, 2015, 06:06:18 AM
we have BIP66 but there are still many forms of malleability to be fixed.
BIP66 is not about malleability, really; BIP62 is.
Is this the real reason for BIP66, which you couldn't mention in April?
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html
(Disclosure: consensus bug indirectly solved by BIP66)
The explicitly stated goal right in the BIP62 document (https://github.com/bitcoin/bips/blob/master/bip-0066.mediawiki#Motivation) was a reduction in dependence on OpenSSL for consensus; which the truth and nearly the whole truth all that was missing was that the concern was less theoretical than presented.  But yes, the unstated privately known consensus vulnerability was a driving force for clearing up that ugly corner with BIP66 immediately rather than mopping it up as a side effect of BIP62. Especially due to fact that BIP62's progress has been slow-- and had grown over complicated. The fact that BIP66 accomplished one of the necessary steps will make BIP62 or its logical successor easier to move forward.

If you note from the timeline that BIP62 had already been revised to accomplish a BIP66 like goal prior to us knowing about that specific vulnerability:  We were aware of the risk class in principle before having specific knoweldge with enough concern that we thought it was worth closing off.  Knowing that it was _actually_ exposed had only an impact on timing and sequencing; and only after it was clear that BIP62 was not rapidly converging (unfortunately, in spite of much progress on BIP62 we continued to find problems with it).

It's sad, there is a vastly superior-- simpler, more comprehensive, and with secondary benefits like enabling more degrees between full and SPV-- malleability 'fix' in elements alpha but I don't have any way to get it into Bitcoin, BIP62 is disappointing by comparison (brillant ideas welcome).


do you care to elaborate on that better fix? especially on the reason why it cannot be "ported" to bitcoin. thanks in advance.