SIGHASH_WITHINPUTVALUE: Super-lightweight HW wallets and offline data

[etotheipi]

It is (or should be) uncontroversial, because it adds an optional, extra restriction to tx validity.  

Technically, that makes it a soft fork.  What happens if undefined bits in the sighash field are set to 1?

I meant, uncontroversial if you've already accepted that there will be a soft/hard fork.  I think inducing a fork just to introduce this would be pretty controversial.

By the way, I accidentally glossed over your soft-fork idea.  I like it.  But I think more than changing the validation rules, you would require all nodes to start accepting and using that script type (which would require a new address format, I'm sure).  And if they use it before it's a valid rule, people could inject incorrect values just for fun.  I guess, after the miners all agreed and you reached threshold, you would hardcode the starting height.

One of the nice benefits of the SIGHASH_WITHINPUTVALUE code, is that it doesn't depend at all on the people sending you coins.  You are completely in control of whether you are using it.  It works will all prior UTXOs in the blockchain, and doesn't require changing any block parsers, etc.

[meowmeowbrowncow]

Do it™

[jl2012]

I think there is another way to do this, but still need a soft-fork:

1. Define a new transaction version: valid only if total input amount = total output amount

2. Miner fee is sent to OP_TRUE

3. The rest will follow https://gist.github.com/gavinandresen/2355445

This has 2 bonus effect:

a. People using this type of transaction will not accidentally send a huge amount of transaction fee

b. In multi-parties contracts, people can define fee explicitly

The drawback is that would cost extra block space, so the transaction fee will be higher in long run

Comments?

[etotheipi]

I think there is another way to do this, but still need a soft-fork:

1. Define a new transaction version: valid only if total input amount = total output amount

2. Miner fee is sent to OP_TRUE

3. The rest will follow https://gist.github.com/gavinandresen/2355445

This has 2 bonus effect:

a. People using this type of transaction will not accidentally send a huge amount of transaction fee

b. In multi-parties contracts, people can define fee explicitly

The drawback is that would cost extra block space, so the transaction fee will be higher in long run

Comments?

Although it's a neat idea, it underestimates the blockchain bloat that comes with it.  If this became standard, and we had 1000 tx per block, that's potentially 1,000 extra UTXOs we wouldn't otherwise have.  Adding 1000 UTXOs per block would be pretty rough, and the total blockchain size would increase pretty dramatically in an effort to spend all those. 

It seems that you would end up with much more frequent "double-spends", for the same reason coinbase transactions require 100 blocks to become valid.  Miners could spend those fees immediately, meaning that orphaned blocks could lead to countless "accidental" double-spends.

[TierNolan]

Although it's a neat idea, it underestimates the blockchain bloat that comes with it.  If this became standard, and we had 1000 tx per block, that's potentially 1,000 extra UTXOs we wouldn't otherwise have.  Adding 1000 UTXOs per block would be pretty rough, and the total blockchain size would increase pretty dramatically in an effort to spend all those. 

No, OP_TRUE can be spent by anyone.  It implicitly means "pay to miner".  Ofc, some miners might not detect them and so it becomes pay to the first miner who finds them.

[etotheipi]

Although it's a neat idea, it underestimates the blockchain bloat that comes with it.  If this became standard, and we had 1000 tx per block, that's potentially 1,000 extra UTXOs we wouldn't otherwise have.  Adding 1000 UTXOs per block would be pretty rough, and the total blockchain size would increase pretty dramatically in an effort to spend all those. 

No, OP_TRUE can be spent by anyone.  It implicitly means "pay to miner".  Ofc, some miners might not detect them and so it becomes pay to the first miner who finds them.

Oh, I understood that part, but I forgot about the part where they don't have to provide a large signature in order to spend it to themselves.  And I missed the part that they can mine all of them in a single transaction and produce only one extra output for themself.  Okay, it's kind of cool.

[Peter Todd]

Oh, I understood that part, but I forgot about the part where they don't have to provide a large signature in order to spend it to themselves.  And I missed the part that they can mine all of them in a single transaction and produce only one extra output for themself.  Okay, it's kind of cool.

Basically you're adding 1+1+8=10 bytes for the OP_TRUE txout and 36+4+1=41 bytes to spend it, or 51 extra bytes in total per transaction. That's not great - it's roughly a 25% overhead for typical transactions - but it's not infeasible either.

[TierNolan]

Also, it is only needed for transfers to cold storage addresses. 

A client could have a checkbox to create the appropriate transaction.

[Peter Todd]

Also, it is only needed for transfers to cold storage addresses. 

A client could have a checkbox to create the appropriate transaction.

On the other hand we're probably going make miners create proofs of transaction fees sometime in the future, either as part of the UTXO proof stuff or something separate, so I'm not sure a hacked up mechanism like this is really worth it.

Users aren't going to want to have to check some checkbox due to some strange technical reason that they have no understanding of.

[jl2012]

Although it's a neat idea, it underestimates the blockchain bloat that comes with it.  If this became standard, and we had 1000 tx per block, that's potentially 1,000 extra UTXOs we wouldn't otherwise have.  Adding 1000 UTXOs per block would be pretty rough, and the total blockchain size would increase pretty dramatically in an effort to spend all those.  

No, OP_TRUE can be spent by anyone.  It implicitly means "pay to miner".  Ofc, some miners might not detect them and so it becomes pay to the first miner who finds them.

Oh, I understood that part, but I forgot about the part where they don't have to provide a large signature in order to spend it to themselves.  And I missed the part that they can mine all of them in a single transaction and produce only one extra output for themself.  Okay, it's kind of cool.

After thinking carefully I believe your original proposal is better. It allows the HW wallet to sign a partial transaction (e.g., a user wants to sell smartcoins for ordinary bitcoin. The HW wallet has to know the exact amount of smartcoins being sent.)

With scriptSig Malleability (https://en.bitcoin.it/wiki/Transaction_Malleability), a variation of the original proposal could be done with a soft fork:

First, redefine OP_NOP1 to OP_CHECKVALUE. If the top stake item does not equal to the value of the input, the script fails. If they are equal, OP_CHECKVALUE does nothing.

The scriptSig for a standard send-to-key-hash transaction will look like this: <sig> <pubKey> <value> OP_CHECKVALUE OP_DROP
and the combined script will be <sig> <pubKey> <value> OP_CHECKVALUE OP_DROP OP_DUP OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

For old version clients, OP_CHECKVALUE is simply OP_NOP1, which will do nothing. OP_DROP will then remove <value>. So the design is completely backward compatible.

(This is my first time designing a script. Please comment)

Edit: sorry it doesn't work, as the <sig> does not sign <value>

[jl2012]

I have a ugly but backward compatible way to implement the OP idea:

1. Redefine one of the unassigned OP code (e.g. 0xfc) as OP_CHECKVALUE
2. When signing of input value is required, add an extra output like "<TxPrevHash> <TxPrevOutputIndex> <TxPrevOutputValue> OP_CHECKVALUE" is added, with 0 bitcoin assigned to this output
3. If an output of this format is found when validating a transaction, the engine will check whether the value of the specified output is same as TxPrevOutputValue. If not, the whole transaction is invalid. If yes, the code remove the top 3 stack items.

More rules:
1. There could be multiple OP_CHECKVALUE in the same dummy output.
2. The whole transaction is invalid if the specified output is not found in the input list.

Advantage:
This is transparent to existing clients, making it a perfect soft-fork. For both old and new clients, the dummy output is provably unspendable and will be pruned from the UTXO set.

Disadvantage:
Bloats blockchain, and looks hacky

Any comments? At least I want to know if it works in theory

[etotheipi]

I have a ugly but backward compatible way to implement the OP idea:

1. Redefine one of the unassigned OP code (e.g. 0xfc) as OP_CHECKVALUE
2. When signing of input value is required, add an extra output like "<TxPrevHash> <TxPrevOutputIndex> <TxPrevOutputValue> OP_CHECKVALUE" is added, with 0 bitcoin assigned to this output
3. If an output of this format is found when validating a transaction, the engine will check whether the value of the specified output is same as TxPrevOutputValue. If not, the whole transaction is invalid. If yes, the code remove the top 3 stack items.

More rules:
1. There could be multiple OP_CHECKVALUE in the same dummy output.
2. The whole transaction is invalid if the specified output is not found in the input list.

Advantage:
This is transparent to existing clients, making it a perfect soft-fork. For both old and new clients, the dummy output is provably unspendable and will be pruned from the UTXO set.

Disadvantage:
Bloats blockchain, and looks hacky

Any comments? At least I want to know if it works in theory

Does it have to be an extra output?  Couldn't we just put this in the TxIn script being signed directly?  Right now, a standard TxIn script is:

Code:

PUSH(Signature) PUSH(PubKey)

Couldn't we do:

Code:

PUSH(8bytefee) OP_CHECKFEEVERIFY PUSH(Signature) PUSH(PubKey)

EDIT: Never mind, I just realized that that those op-codes don't get signed, so they can be altered without breaking the signature.

[etotheipi]

I have a ugly but backward compatible way to implement the OP idea:

1. Redefine one of the unassigned OP code (e.g. 0xfc) as OP_CHECKVALUE
2. When signing of input value is required, add an extra output like "<TxPrevHash> <TxPrevOutputIndex> <TxPrevOutputValue> OP_CHECKVALUE" is added, with 0 bitcoin assigned to this output
3. If an output of this format is found when validating a transaction, the engine will check whether the value of the specified output is same as TxPrevOutputValue. If not, the whole transaction is invalid. If yes, the code remove the top 3 stack items.
...
Any comments? At least I want to know if it works in theory

Does it have to be one extra output per input?  Couldn't you just have a single output specifying the fee for the whole tx?  It would essentially do the same thing, and the OP_NOP/OP_CHECKFEEVERIFY would still work with zero output value and be pruned. 

That would make it somewhat complicated to do more-advanced contracts though... where not all inputs and/or outputs are known at the time of signing (like the lighthouse contract).  Just more food for thought.

[jl2012]

I have a ugly but backward compatible way to implement the OP idea:

1. Redefine one of the unassigned OP code (e.g. 0xfc) as OP_CHECKVALUE
2. When signing of input value is required, add an extra output like "<TxPrevHash> <TxPrevOutputIndex> <TxPrevOutputValue> OP_CHECKVALUE" is added, with 0 bitcoin assigned to this output
3. If an output of this format is found when validating a transaction, the engine will check whether the value of the specified output is same as TxPrevOutputValue. If not, the whole transaction is invalid. If yes, the code remove the top 3 stack items.

More rules:
1. There could be multiple OP_CHECKVALUE in the same dummy output.
2. The whole transaction is invalid if the specified output is not found in the input list.

Advantage:
This is transparent to existing clients, making it a perfect soft-fork. For both old and new clients, the dummy output is provably unspendable and will be pruned from the UTXO set.

Disadvantage:
Bloats blockchain, and looks hacky

Any comments? At least I want to know if it works in theory

Does it have to be an extra output?  Couldn't we just put this in the TxIn script being signed directly?  Right now, a standard TxIn script is:

Code:

PUSH(Signature) PUSH(PubKey)

Couldn't we do:

Code:

PUSH(8bytefee) OP_CHECKFEEVERIFY PUSH(Signature) PUSH(PubKey)

EDIT: Never mind, I just realized that that those op-codes don't get signed, so they can be altered without breaking the signature.

I thought something similar (see my crossed out reply above), and found it won't work for the same reason

[jl2012]

Does it have to be one extra output per input? 

As I mentioned, there could be multiple OP_CHECKVALUE in the same dummy output, e.g. <h1><n1><v1> OP_CHECKVALUE <h2><n2><v2> OP_CHECKVALUE ........

Couldn't you just have a single output specifying the fee for the whole tx?  It would essentially do the same thing, and the OP_NOP/OP_CHECKFEEVERIFY would still work with zero output value and be pruned. 

Interesting. This could be more efficient than my previous OP_TRUE solution since the miner won't need to transfer the fee once again. Just redefine an unassigned OP code to OP_CHECKFEEVERIFY (We don't need to consume an OP_NOP because the output is not supposed to be spent)

That would make it somewhat complicated to do more-advanced contracts though... where not all inputs and/or outputs are known at the time of signing (like the lighthouse contract).  Just more food for thought.

Yes, that would require the OP_CHECKVALUE or the OP_TRUE solution.

[jl2012]

I am making a new proposal for colored coin soft-fork: https://bitcointalk.org/index.php?topic=253385.0

To make it works with lightweight HW wallet, OP_CHECKFEEVERIFY is not enough because colored coin is not fungible with normal coin. In this case only OP_CHECKVALUE or OP_TRUE solution would work.

[ShadowOfHarbringer]

Do it™

Yeah, go for it.

[TierNolan]

This thread was pinged from another thread where someone (probably) lost 200 BTC to fees.

It is probably worth creating a BIP and reference implementation so that it is tested and included in any subsequent hard fork.

From here, it looks like only the bottom 5 bits make any difference.

This suggests that you can add extra sighash bits and it won't cause anything to explode.

Sighash decides how the transaction is hashed, so you can't do it in a soft fork kind of way.  Either the bit has no effect (so is inherently backward compatible) or it changes something and so will fail checksig for older clients.

[Peter Todd]

There are much better ways to do this than using up what limited SIGHASH bits we have, and we already have other uses for SIGHASH bits that should be implemented.

In particular it has been proposed to extend the merkle tree of transactions down into the transactions themselves, which would allow you to easily provide compact proofs of the value of the transaction outputs that a transaction is spending.

tl;dr: Strong NACK on SIGHASH_WITHINPUTVALUE

[etotheipi]

There are much better ways to do this than using up what limited SIGHASH bits we have, and we already have other uses for SIGHASH bits that should be implemented.

In particular it has been proposed to extend the merkle tree of transactions down into the transactions themselves, which would allow you to easily provide compact proofs of the value of the transaction outputs that a transaction is spending.

tl;dr: Strong NACK on SIGHASH_WITHINPUTVALUE

You seem to suggest there is consensus about your opinion it should not be done, and that there are seriously-considered proposals to change the meaning of the merkle root.  But, if you read the rest of this thread, there is a quite a bit of high-level support for this modification, as an "add-on" to any future hard forks.  Of course, we're all happy to discuss alternatives, and what other uses there are for the other SIGHASH types, but you haven't mentioned what they are or where I can go read about it.

On the other hand, if we were to implement such a drastic/breaking change in the protocol as you suggest, to actually change what the merkle root means in the header, we'd have no problem simply "fixing" the existing hash types to include the output values, as that would equally break all existing parsing/verification code as changing the meaning of the merkle root.  If your suggestion was implemented, we could remove the "wasted" SIGHASH_WITHINPUTVALUE bit and recover it for these other use cases you mention.

On a related note:  TierNolan, can you link to the thread where this was reported?  I'd like to see how this massive fee occurred.  Was it malicious?  Or just likely a bug in the way the online computer was reporting input values to the offline computer?  If I had to guess, it was someone who tried to manually execute the offline signing procedure, but did not include all the supporting transactions, and ended up mis-reporting input values to the offline computer (which would've been caught if they supplied and checked the supporting tx, or SIGHASH_WITHINPUTVALUE was implemented).