SIGHASH_WITHINPUTVALUE: Super-lightweight HW wallets and offline data

[Peter Todd]

There are much better ways to do this than using up what limited SIGHASH bits we have, and we already have other uses for SIGHASH bits that should be implemented.

In particular it has been proposed to extend the merkle tree of transactions down into the transactions themselves, which would allow you to easily provide compact proofs of the value of the transaction outputs that a transaction is spending.

tl;dr: Strong NACK on SIGHASH_WITHINPUTVALUE

You seem to suggest there is consensus about your opinion it should not be done, and that there are seriously-considered proposals to change the meaning of the merkle root.  But, if you read the rest of this thread, there is a quite a bit of high-level support for this modification, as an "add-on" to any future hard forks.  Of course, we're all happy to discuss alternatives, and what other uses there are for the other SIGHASH types, but you haven't mentioned what they are or where I can go read about it.

The all I see is gmaxwell: "I think the sighash with input value seems pretty inelegant." and Mike saying he likes the idea of doing something, lets talk more at the conference.

The seriously-considered proposals are the UTXO commitment proposals. As for other uses for SIGHASH bits, see IRC discussions about the need to be able to specify sets of txins and txouts being committed to. I also posted a quick idea myself for txout summation.

FWIW one of my main objections to this is that we really need general ways to prove fees paid by transactions for inflation fraud proofing, and short proofs of txout existence are very helpful to SPV nodes who are able to use them to check that transactions are valid - you're focusing on something very specific.

On the other hand, if we were to implement such a drastic/breaking change in the protocol as you suggest, to actually change what the merkle root means in the header, we'd have no problem simply "fixing" the existing hash types to include the output values, as that would equally break all existing parsing/verification code as changing the meaning of the merkle root.  If your suggestion was implemented, we could remove the "wasted" SIGHASH_WITHINPUTVALUE bit and recover it for these other use cases you mention.

Additional merkle roots can be committed to in the coinbase txout; adding them is a soft-fork just like adding additional SIGHASH bits is. The best way to do this is to put the digest of the secondary merkle root in the last txout of the coinbase (with a P2Pool style 0-value OP_RETURN) so the proof can use a SHA256 midstate of the coinbase tx to keep the size independent of the size of the coinbase tx. It's at this point that a "v2" transaction format can be defined to fix all this stuff for once.

I'd be happy to pitch in on a BIP defining that transaction format, and we can do it in a way that leaves room to add the UTXO commitment digest when that is fleshed out as well.

[etotheipi]

There are much better ways to do this than using up what limited SIGHASH bits we have, and we already have other uses for SIGHASH bits that should be implemented.

In particular it has been proposed to extend the merkle tree of transactions down into the transactions themselves, which would allow you to easily provide compact proofs of the value of the transaction outputs that a transaction is spending.

tl;dr: Strong NACK on SIGHASH_WITHINPUTVALUE

You seem to suggest there is consensus about your opinion it should not be done, and that there are seriously-considered proposals to change the meaning of the merkle root.  But, if you read the rest of this thread, there is a quite a bit of high-level support for this modification, as an "add-on" to any future hard forks.  Of course, we're all happy to discuss alternatives, and what other uses there are for the other SIGHASH types, but you haven't mentioned what they are or where I can go read about it.

The all I see is gmaxwell: "I think the sighash with input value seems pretty inelegant." and Mike saying he likes the idea of doing something, lets talk more at the conference.

The seriously-considered proposals are the UTXO commitment proposals. As for other uses for SIGHASH bits, see IRC discussions about the need to be able to specify sets of txins and txouts being committed to. I also posted a quick idea myself for txout summation.

FWIW one of my main objections to this is that we really need general ways to prove fees paid by transactions for inflation fraud proofing, and short proofs of txout existence are very helpful to SPV nodes who are able to use them to check that transactions are valid - you're focusing on something very specific.

On the other hand, if we were to implement such a drastic/breaking change in the protocol as you suggest, to actually change what the merkle root means in the header, we'd have no problem simply "fixing" the existing hash types to include the output values, as that would equally break all existing parsing/verification code as changing the meaning of the merkle root.  If your suggestion was implemented, we could remove the "wasted" SIGHASH_WITHINPUTVALUE bit and recover it for these other use cases you mention.

Additional merkle roots can be committed to in the coinbase txout; adding them is a soft-fork just like adding additional SIGHASH bits is. The best way to do this is to put the digest of the secondary merkle root in the last txout of the coinbase (with a P2Pool style 0-value OP_RETURN) so the proof can use a SHA256 midstate of the coinbase tx to keep the size independent of the size of the coinbase tx. It's at this point that a "v2" transaction format can be defined to fix all this stuff for once.

I'd be happy to pitch in on a BIP defining that transaction format, and we can do it in a way that leaves room to add the UTXO commitment digest when that is fleshed out as well.

I'm all for UTXO commitment proposals.  After all, I have a pretty elaborate one I'd like to see pushed forward.  But that still serves a different purpose than this:  sending the offline computer UTXO proofs/branches is going to be a lot more than 8 bytes per input.  Part of the goal is to minimize the data sent across the online-offline channel, and the complexity of what needs to be done by the offline signing device.   

I'm not sure where the inelegance comes from -- we add 8 bytes to the data to be hashed and we close up a major oversight/inefficiency in the protocol.  If this was some kind of epic, controversial change, I'd agree that it may not be worth it.  I've already said it's not even worth its own hard-fork.  But given how simple-yet-useful the change is, I see little reason not to do it if the timing is right.  Unless it will be will feasibly be solved by some other major upgrade in the near future.

[Peter Todd]

I'm all for UTXO commitment proposals.  After all, I have a pretty elaborate one I'd like to see pushed forward.  But that still serves a different purpose than this:  sending the offline computer UTXO proofs/branches is going to be a lot more than 8 bytes per input.  Part of the goal is to minimize the data sent across the online-offline channel, and the complexity of what needs to be done by the offline signing device.  

I'm not sure where the inelegance comes from -- we add 8 bytes to the data to be hashed and we close up a major oversight/inefficiency in the protocol.  If this was some kind of epic, controversial change, I'd agree that it may not be worth it.  I've already said it's not even worth its own hard-fork.  But given how simple-yet-useful the change is, I see little reason not to do it if the timing is right.  Unless it will be will feasibly be solved by some other major upgrade in the near future.

The big problem is it's specific to OP_CHECKSIG, and in the future it's quite possible that OP_CHECKSIG will be changed/made obsolete/who knows? Doing this via a SIGHASH bit is inelegant because it's so specific to a single use-case.

With a v2 transaction format, with a merkle tree extending into the tx and including a hash of the CTxOut structure being spent your hardware wallets would just take that data and operate only on that. For instance this is helpful if you are signing just a small part of a much bigger combined transaction with funds from multiple parties - there might be 1000 people chipping in to pay a large bounty for instance, and with a v2 tx format and set-based SIGHASH's your hardware wallet only needs to be given the small part of the txouts you are signing for.

Interestingly in that use-case your signature can still cover all the other txouts safely. Your wallet only needs to know that you've contributed xBTC of funds, and the txouts it knows about, via the merkle paths its been provided, are for (x-fee)BTC of funds. Perfectly safe no-matter what malware is on your computer, yet the outside world only knows that you added funds to some huge txout set. This is perfect for CoinJoin tx's.

[etotheipi]

The big problem is it's specific to OP_CHECKSIG, and in the future it's quite possible that OP_CHECKSIG will be changed/made obsolete/who knows? Doing this via a SIGHASH bit is inelegant because it's so specific to a single use-case.

With a v2 transaction format, with a merkle tree extending into the tx and including a hash of the CTxOut structure being spent your hardware wallets would just take that data and operate only on that. For instance this is helpful if you are signing just a small part of a much bigger combined transaction with funds from multiple parties - there might be 1000 people chipping in to pay a large bounty for instance, and with a v2 tx format and set-based SIGHASH's your hardware wallet only needs to be given the small part of the txouts you are signing for.

Interestingly in that use-case your signature can still cover all the other txouts safely. Your wallet only needs to know that you've contributed xBTC of funds, and the txouts it knows about, via the merkle paths its been provided, are for (x-fee)BTC of funds. Perfectly safe no-matter what malware is on your computer, yet the outside world only knows that you added funds to some huge txout set. This is perfect for CoinJoin tx's.

I would not dismiss offline transactions a "meh, just a single use case."   That is the future of any serious Bitcoin users and HW wallets.  Accommodating them is in everyone's best interest.

Also, you are talking about a major protocol upgrade that is far from uncontroversial and could take months/years of debate, testing, etc.  I'm talking about a very simple, modification of a couple lines of code that completely solves a problem that a lot of Bitcoin innovators having today.

[TierNolan]

On a related note:  TierNolan, can you link to the thread where this was reported?  

It was this transaction in the Coinjoin thread.

I don't know if it would have helped, but if the transaction generation engine had to include tx fees or something maybe it would have failed more gracefully (i.e. produced an invalid tx).

[Peter Todd]

I would not dismiss offline transactions a "meh, just a single use case."   That is the future of any serious Bitcoin users and HW wallets.  Accommodating them is in everyone's best interest.

No, I'm dismissing offline transactions with hardware where getting n*100K of data to the device is infeasible. Microcontrollers these days are very fast - the Trezor for instance has a USB interface which is perfectly capable of getting a MB of data (10 txins given the 100K tx limit) over the wire in a second or two. QR codes are an issue, but by then you're talking about a single use-case.

Also, you are talking about a major protocol upgrade that is far from uncontroversial and could take months/years of debate, testing, etc.  I'm talking about a very simple, modification of a couple lines of code that completely solves a problem that a lot of Bitcoin innovators having today.

Any protocol upgrade is controversial and takes a long time to implement - might as well knock off some other use cases.

For instance an alternate approach, as Mike hinted at, is a OP_CHECKSIG2. It'd be extremely valuable to make the SIGHASH flags more general, and in addition it would also be very valuable if you could sign a signature to spend a specific set of CTxOut's - scriptPubKey:value - rather than a specific COutPoint - txid:n - which would make transaction mutability far less of an issue. (also forces people to stop re-using addresses) We can add pubkey recovery at that point too, which makes transactions a fair bit smaller.

Again, you've got support from myself and gmaxwell, and it seems Mike, going down that approach. And again, remember that a set-based CHECKSIG with a merkle tree for the txouts allows you participate in big CoinJoin tx's without a lot of device overhead.

[TierNolan]

The big problem is it's specific to OP_CHECKSIG, and in the future it's quite possible that OP_CHECKSIG will be changed/made obsolete/who knows? Doing this via a SIGHASH bit is inelegant because it's so specific to a single use-case.

A bit could be added for an extended SIGHASH.  Maybe SIGHASH could be a varint or something.

[natb]

etotheipi - your original proposal is just the thing I'm looking for and is quite simple and elegant. I was happily implementing my lightweight HW wallet and blundered across this nasty snag just today. Has there been any further development on this front recently?

[etotheipi]

The consensus seems to be that no one thinks it's worth doing this.  I still think it makes sense to do as one of those latent upgrades that goes in now but isn't active until 12 months from now.   But as you can see from the discussion some people are very focused on overhauling the SIGHASH system entirely, and don't think it's worth the effort to do this small change.  I disagree, but what can I do?

[natb]

Well, that is quite unfortunate. Thanks for the update, guess I have to go find another way to prevent the 'transaction fee' attack on my wallet...

[grau]

The consensus seems to be that no one thinks it's worth doing this.  I still think it makes sense to do as one of those latent upgrades that goes in now but isn't active until 12 months from now.   But as you can see from the discussion some people are very focused on overhauling the SIGHASH system entirely, and don't think it's worth the effort to do this small change.  I disagree, but what can I do?

The consensus unlikely includes those developing wallets. It is the single most dangerous "feature" I came across while implementing Bitcoin that one is not able to validate the fee of a transaction on its own but need to retrieve its referenced predecessors. We should do this.

[natb]

I'll gladly do what is needed to support this proposal, whatever way I can. I wonder what some of the other HW wallet guys are doing (e.g. Trezor)? Probably some form of SPV - but this kind of thing seriously increases the complexity and resource demands of a small HW wallet implementation.

[etotheipi]

The consensus seems to be that no one thinks it's worth doing this.  I still think it makes sense to do as one of those latent upgrades that goes in now but isn't active until 12 months from now.   But as you can see from the discussion some people are very focused on overhauling the SIGHASH system entirely, and don't think it's worth the effort to do this small change.  I disagree, but what can I do?

The consensus unlikely includes those developing wallets. It is the single most dangerous "feature" I came across while implementing Bitcoin that one is not able to validate the fee of a transaction on its own but need to retrieve its referenced predecessors. We should do this.

For reference, I specifically recall retep talking to Gavin about it on IRC shortly after his last post, and Gavin agreed that modifying the SIGHASH system was a bad idea.  I didn't feel like pressing it, at the time, though.

That's not to say that this can't be done or that Gavin and the other devs aren't persuadable.  But it would mean mobilizing support for it and presenting a strong case to those that are against it.  If the system was built with feature from the start, we'd all be happy and no one would have an issue with it.  It's more about convincing folks that the benefit is worth the trouble of making the change.  As you said... the people who most strongly support this are the ones actually developing software that has to deal with the megabytes of supporting transactions just to verify a few bytes.

[Peter Todd]

For reference, I specifically recall retep talking to Gavin about it on IRC shortly after his last post, and Gavin agreed that modifying the SIGHASH system was a bad idea.  I didn't feel like pressing it, at the time, though.

That's not to say that this can't be done or that Gavin and the other devs aren't persuadable.  But it would mean mobilizing support for it and presenting a strong case to those that are against it.  If the system was built with feature from the start, we'd all be happy and no one would have an issue with it.  It's more about convincing folks that the benefit is worth the trouble of making the change.  As you said... the people who most strongly support this are the ones actually developing software that has to deal with the megabytes of supporting transactions just to verify a few bytes.

You'll have more luck pushing for this by developing a multisig wallet implementation and getting people to actually use it first - P2SH required a painful soft-fork yet years later it's still hardly ever used, making people question the value of new features. P2SH was something that everyone was supposed to be using by now because of the "obvious" security need; you're proposed reason for a soft-fork and associated system-wide risk is significantly more niche.

Like it or not the amount of money that can be lost for a single consensus bug is staggering and gets bigger every day.

FWIW Litecoin will need to undergo a soft-fork soon to finally get height-in-coinbase implemented - talk this over with them, specifically Warren Togami.

[justusranvier]

P2SH was something that everyone was supposed to be using by now because of the "obvious" security need; you're proposed reason for a soft-fork and associated system-wide risk is significantly more niche.

It won't be niche when the PC platform as a whole is rendered unusable for secure applications by firmware-level, airgap-crossing malware.

*Secure hardware wallets need to be here before that happens.

*"Secure" means custom silicon specifically hardened against side channel attacks and such, not off the shelf embedded systems containing god-knows-what vulnerabilities.

[etotheipi]

You'll have more luck pushing for this by developing a multisig wallet implementation and getting people to actually use it first - P2SH required a painful soft-fork yet years later it's still hardly ever used, making people question the value of new features. P2SH was something that everyone was supposed to be using by now because of the "obvious" security need; you're proposed reason for a soft-fork and associated system-wide risk is significantly more niche.

Slightly off-topic, but if it wasn't for the rapid blockchain growth and my procrastination in fixing Armory's scalability issues, I'd have Multisig-with-P2SH implemented already.  The problem is not the features being useless or unwanted, it's that there's such a shortage of human capital to make progress in this part of the Bitcoin development world.  There's so few environments where such a feature could be implemented, and so many other priorities of the few people developing those environments.

However, this particular change is trivial to implement, and dramtically simplifies implementation of any kind of offline signing device.  There's no doubt it would be used immediately by Armory and all the HW wallet developers.

[Peter Todd]

P2SH was something that everyone was supposed to be using by now because of the "obvious" security need; you're proposed reason for a soft-fork and associated system-wide risk is significantly more niche.

It won't be niche when the PC platform as a whole is rendered unusable for secure applications by firmware-level, airgap-crossing malware.

*Secure hardware wallets need to be here before that happens.

*"Secure" means custom silicon specifically hardened against side channel attacks and such, not off the shelf embedded systems containing god-knows-what vulnerabilities.

Custom silicon is the problem: how do you know what it's actually doing?

The sweet spot is probably 16 and 32-bit microprocessors: fast enough to handle crypto without pain, small enough that hiding malicious features is very tough for the manufacturer, and cheap enough that the community has a chance of auditing the actual shipped hardware and firmware against the claimed design.

Microprocessors in this class don't have significant problems with txin's provided they have a reasonable interface to the PC, like USB: just hash the supporting transactions incrementally. With well-designed hardware the fact that the device is connected by USB has no security risks and can be easily audited. (FTDI's USB<->high-speed-serial chips are a good option for the truly paranoid)

[Peter Todd]

Slightly off-topic, but if it wasn't for the rapid blockchain growth and my procrastination in fixing Armory's scalability issues, I'd have Multisig-with-P2SH implemented already.  The problem is not the features being useless or unwanted, it's that there's such a shortage of human capital to make progress in this part of the Bitcoin development world.  There's so few environments where such a feature could be implemented, and so many other priorities of the few people developing those environments.

However, this particular change is trivial to implement, and dramtically simplifies implementation of any kind of offline signing device.  There's no doubt it would be used immediately by Armory and all the HW wallet developers.

If it's so trivial to implement then where is your proposed implementation and BIP? It could be re-implemented fifty times through rejections and objections and still be less work that the true work associated with all the testing, wallet implementations, and alt-implementations associated with this idea.

Anyway, it doesn't simplify offline signing all that much, it just makes it possible with more limited hardware. The actual code is basically the same in both cases. Heck, the design effort required for the hardware isn't much different in most cases: moderately fast USB interfaces aren't a big deal these days and come pre-packaged.

This also encourages the design of really limited hardware wallets that don't support the payment protocol: if you don't know who you are paying, all you've done is limited the rate that your funds can be stolen a bit. Heck, on that basis I think I'd actually NACK such a patch myself...

[etotheipi]

If it's so trivial to implement then where is your proposed implementation and BIP? It could be re-implemented fifty times through rejections and objections and still be less work that the true work associated with all the testing, wallet implementations, and alt-implementations associated with this idea.

I was talking about from the wallet design perspective.  Multi-sig & P2SH increases the complexity of the app.  This change is trivial leverage in your software and dramatically simplifies it.  I assure you I understand that any such changes are "not trivial" to implement in the protocol.  But it certainly doesn't hurt that it's so simple in concept...

[justusranvier]

The sweet spot is probably 16 and 32-bit microprocessors: fast enough to handle crypto without pain, small enough that hiding malicious features is very tough for the manufacturer, and cheap enough that the community has a chance of auditing the actual shipped hardware and firmware against the claimed design.

It shouldn't be a microprocessor at all.

Think something like this, but to perform ECDSA instead: https://en.wikipedia.org/wiki/555_timer_IC

Truly secure hardware wallets will need to be built at the level of electrical engineering, not software engineering.