What the "average user" needs to know about Transaction Mutability

[keithers]

Thanks for this OP.  I tried reading it, but to be honest, I don't understand all the computer lingo and code    I wish I did, because I really wanted to understand what was going on today...

[Zarathustra]

What you need to know about Transaction mutability even if you are running the QT (reference) client or another major client.  
(..)

Great posts and structured informations in this thread. Thank you DeathAndTaxes!

[Nagle]

The original post is well written. There's another problem with the fix, though. If you can't spend unconfirmed change arising from confirmed inputs, once you've spent some money, you sometimes can't spend again until a few confirmations have been processed. There's now an option in the official QT client to enforce that rule. 

This creates operational problems for Bitcoin use. Once you've spent some Bitcoins, you may not be able to spend any more Bitcoins for tens of minutes. Whether or not you can spend depends on the denominations of Bitcoins currently in your wallet. You don't have an "account balance", you have a collection of Bitcoin items with varying denominations, and this now matters.  Explaining this to end users is going to be a problem, especially since the current user interface doesn't expose that level of detail to the user. With the "no spending unconfirmed change" rule enforced, it's going to look like this, I think:

User has 1 BTC, 0 unconfirmed, 1.0 available for spending. The 1 BTC is from a single transaction.
User spends 0.5 BTC.
User now has 0.5 BTC, 0.5 unconfirmed, 0 available for spending.
Half an hour later....
User now has 0.5 BTC, 0 unconfirmed, 0.5 available for spending.

This is not going to play well at the mall.

For a big operation with a lot of outgoing transactions, like an exchange, wallet systems may have to have something like active currency management. (This is the "we have too many twenties, and not enough fives right now" problem a retailer sometimes faces).

Anybody thinking about this?

[DeathAndTaxes]

The original post is well written. There's another problem with the fix, though. If you can't spend unconfirmed change arising from confirmed inputs, once you've spent some money, you sometimes can't spend again until a few confirmations have been processed. There's now an option in the official QT client to enforce that rule.  

This creates operational problems for Bitcoin use. Once you've spent some Bitcoins, you may not be able to spend any more Bitcoins for tens of minutes. Whether or not you can spend depends on the denominations of Bitcoins currently in your wallet. You don't have an "account balance", you have a collection of Bitcoin items with varying denominations, and this now matters.  Explaining this to end users is going to be a problem, especially since the current user interface doesn't expose that level of detail to the user. With the "no spending unconfirmed change" rule enforced, it's going to look like this, I think:

User has 1 BTC, 0 unconfirmed, 1.0 available for spending. The 1 BTC is from a single transaction.
User spends 0.5 BTC.
User now has 0.5 BTC, 0.5 unconfirmed, 0 available for spending.
Half an hour later....
User now has 0.5 BTC, 0 unconfirmed, 0.5 available for spending.

This is not going to play well at the mall.

For a big operation with a lot of outgoing transactions, like an exchange, wallet systems may have to have something like active currency management. (This is the "we have too many twenties, and not enough fives right now" problem a retailer sometimes faces).

Anybody thinking about this?

That assumes the user only has a single unspent output available to spend. If a user had multiple unspent outputs they could make multiple txs each involving one or more of them while the new unconfirmed change confirms.  My current wallet has 118 unspent outputs so I wouldn't be affected as much by the more conservative version than someone who literally has received Bitcoins once ever.

The patch isn't ideal but it is better than a situation where txs "unexplainably" simply never confirm.  The long term goal should be immutable transaction ids however this is a significant challenge and it is going to take a long time.  The "fix" proposed would be temporary because doing nothing until the long term goal is reached is not viable.  Make no mistake, it isn't ideal, but it is better than what we have right now.  I also suggested that it be optional so if users want to risk broken transactions they can spend unconfirmed change outputs.  Another optional change would be to have the client create two or more change outputs when the number of discrete unspent outputs (we are talking unique outputs not necessarily value) is "low".  This would give the coin selection algorithm more options and (once confirmed) allowed the user to make multiple back to back transactions before needing to wait for a confirmation.  You can manually do this yourself (i.e. in your 1 BTC example the user could right now send 0.1 BTC to 10 different addresses in his wallet).

[wywoc]

The original post is well written. There's another problem with the fix, though. If you can't spend unconfirmed change arising from confirmed inputs, once you've spent some money, you sometimes can't spend again until a few confirmations have been processed. There's now an option in the official QT client to enforce that rule.  

This creates operational problems for Bitcoin use. Once you've spent some Bitcoins, you may not be able to spend any more Bitcoins for tens of minutes. Whether or not you can spend depends on the denominations of Bitcoins currently in your wallet. You don't have an "account balance", you have a collection of Bitcoin items with varying denominations, and this now matters.  Explaining this to end users is going to be a problem, especially since the current user interface doesn't expose that level of detail to the user. With the "no spending unconfirmed change" rule enforced, it's going to look like this, I think:

User has 1 BTC, 0 unconfirmed, 1.0 available for spending. The 1 BTC is from a single transaction.
User spends 0.5 BTC.
User now has 0.5 BTC, 0.5 unconfirmed, 0 available for spending.
Half an hour later....
User now has 0.5 BTC, 0 unconfirmed, 0.5 available for spending.

This is not going to play well at the mall.

For a big operation with a lot of outgoing transactions, like an exchange, wallet systems may have to have something like active currency management. (This is the "we have too many twenties, and not enough fives right now" problem a retailer sometimes faces).

Anybody thinking about this?

That assumes the user only has a single unspent output available to spend. If a user had multiple unspent outputs they could make multiple txs each involving one or more of them while the new unconfirmed change confirms.  My current wallet has 118 unspent outputs so I wouldn't be affected as much by the more conservative version than someone who literally has received Bitcoins once ever.

The patch isn't ideal but it is better than a situation where txs "unexplainably" simply never confirm.  The long term goal should be immutable transaction ids however this is a significant challenge and it is going to take a long time.  The "fix" proposed would be temporary because doing nothing until the long term goal is reached is not viable.  Make no mistake, it isn't ideal, but it is better than what we have right now.  I also suggested that it be optional so if users want to risk broken transactions they can spend unconfirmed change outputs.  Another optional change would be to have the client create two or more change outputs when the number of discrete unspent outputs (we are talking unique outputs not necessarily value) is "low".  This would give the coin selection algorithm more options and (once confirmed) allowed the user to make multiple back to back transactions before needing to wait for a confirmation.  You can manually do this yourself (i.e. in your 1 BTC example the user could right now send 0.1 BTC to 10 different addresses in his wallet).

Thanks for this OP.  
I tried reading it, but to be honest, I don't understand all the computer lingo and code.

Thanks D&T for comprehensive explanation.
I  tried my best to read it.  To be honest, I can't say get it, but at least understand something.
In the example TX.A the user spends 0.6 and not 0.4?

[Rampion]

This post should be stickied, at least until the transaction malleability issue is solved.

Good job D&T, the clearest and most complete explanation of the issue I've read so far.

[gamecenteruk]

I am a normal user, affected by this issue and lost money, now all bitcoin transactions should be stopped to prevent this happening, or the dead of bitcoin.

[Rampion]

I am a normal user, affected by this issue and lost money, now all bitcoin transactions should be stopped to prevent this happening, or the dead of bitcoin.

How did you lose money? Please elaborate.

[blueberry]

I am a normal user, affected by this issue and lost money, now all bitcoin transactions should be stopped to prevent this happening, or the dead of bitcoin.

How did you lose money? Please elaborate.

It's just another newbie poster spreading FUD about Bitcoin.

Name:    gamecenteruk
Posts:    5
Activity:    4
Position:    Newbie

[crazy987]

I am a normal user, affected by this issue and lost money, now all bitcoin transactions should be stopped to prevent this happening, or the dead of bitcoin.

How did you lose money? Please elaborate.

Well, then someone who has been here longer then yesterday.
I have a payment of 0.10btc pending since yesterday.
I cannot find it in the blockchain so i hope it will correct itself.

[btcash]

Why does Gavin keep denying that the reference implementation can't handle duplicated TXs.

Transactions are always tracked properly by the Bitcoin-Qt/bitcoind software.

Only users who make multiple transactions in a short period of time will be affected.

This is a bold lie (see Duplicated Transaction Visibility explained by DaT).

[eldentyrell]

The more permanent fix is to identify transactions by inputs and outputs, and (correctly) treat the relationship between transactions and txids as one-to-many.

The situation isn't helped by the fact that the blockchain data structure is founded on the assumption of unique txids (each transaction references its inputs by their transaction hash), whereas the wire protocol cannot assume unique txids.

By the way, this is the second time bitcoin has gotten burned by this.  The first time was when a miner managed to get more than one identical coinbase transaction into the chain, breaking this assumption.

[jeliman]

Sensible suggestions D&T, as has been pointed out, the first solution has already been implemented.


Interesting that Mt Gox appeared to be the only affected service at first. It looks like the "attacker" waited until Gox announced on Monday, watched the markets react, then stabilise, then unleashed the attack on the other exchanges on Tuesday.

So, systematic FUD campaign? Seems very like it. And the biggest wave of the whole attack happened just after a few bad news stories, too. Someone's definitely trying to exploit the exchange rate.

like Wall Street (or other) wants to send the price to the earth and make some significant purchase?

[justusranvier]

The situation isn't helped by the fact that the blockchain data structure is founded on the assumption of unique txids (each transaction references its inputs by their transaction hash), whereas the wire protocol cannot assume unique txids.

That's another way of saying that only the blockchain is canonical.

[un_ordinateur]

Nice explanation D&T!

The DDos attacks almost seem like a distraction, or a synergistic attack - but the maleability issue seems to come from an intention to steal coins transaction by transaction.

You guys agree or what?

NO. Malleability does NOT (directly) enable one to steal coins.

In a given transaction, the inputs, outputs and amounts ARE signed by the sender. One CANNOT change that without invalidating the transaction. However one can pad additionnal data to the transaction, which results in an equally valid transaction, but with a different txid. (because the txid is the hash of the transaction). That transaction has the SAME effect that the original transaction: The "mutator" cannot take the coins.

HOWEVER, if the sender ONLY checks for txids to know if a transaction went trough, then it the mutated transaction went trough instead of the original, then it'll falsely believe that the transaction failed, and may send a new transaction. The receiver may then have recieved the money twice.

Thus one should NEVER check for txids, to verify transaction, and instead, check for inputs, outputs and amount to acertain that. Mt.Gox's software only checked for txids, and is now confused by all this mutability issue.

[Nagle]

The original post is well written. There's another problem with the fix, though. If you can't spend unconfirmed change arising from confirmed inputs, once you've spent some money, you sometimes can't spend again until a few confirmations have been processed. There's now an option in the official QT client to enforce that rule.  

This creates operational problems for Bitcoin use....

That assumes the user only has a single unspent output available to spend. If a user had multiple unspent outputs they could make multiple txs each involving one or more of them while the new unconfirmed change confirms.  My current wallet has 118 unspent outputs so I wouldn't be affected as much by the more conservative version than someone who literally has received Bitcoins once ever.

Having many unspent outputs in a wallet helps speed up spends, but transactions get bigger, as they contain more inputs. This may accelerate dust generation, if there's a bias in favor of never consolidating inputs. Wallets may have to become smarter about when to divide and when to consolidate inputs. The optimal strategy depends on the wallet owner's spending pattern.

This is most likely to be a problem for a commercial wallet with high traffic, such as an exchange's hot wallet.  This could impose a limit on outgoing transaction volume. (I almost hate to suggest this, because some inept exchange may use it as an excuse for withdrawal delays.)

[DeathAndTaxes]

Having many unspent outputs in a wallet helps speed up spends, but transactions get bigger, as they contain more inputs. This may accelerate dust generation, if there's a bias in favor of never consolidating inputs. Wallets may have to become smarter about when to divide and when to consolidate inputs.

Not necessarily.  A compliant wallet should not generate dust outputs.  It should use proper coin selection to ensure all outputs (including change) are above the dust threshold.  Any spend consolidates outputs.  Outside of niche applications it doesn't make any sense to consolidate outputs.   If you have a large number of small outputs and need to make a large spend then the tx will be larger.   If you conolidate them to avoid that then the consolidation tx will be large.  More outputs (within reason) shouldn't significantly increase transaction size.  Imagine two hypothetical wallets with 100 BTC in confirmed unspent outputs.

Unspent outputs (100 BTC total)
-----------------------
"A" 100 BTC

vs

Unspent outputs (100 BTC total)
-----------------------
"A" 1 BTC
"B" 3 BTC
"C" 3 BTC
"D" 6 BTC
"E" 7 BTC
"F" 20 BTC
"G" 25 BTC
"H" 35 BTC

If I make a withdraw of 0.8 BTC, 6.7 BTC, or 23 BTC either wallet can handle it with a single input. The second wallet would need a second input if I wanted to withdraw 50 BTC but that will not change the transaction size significantly.  I would always prefer to have the later wallet as it can handle all four hypothetical transactions without either waiting for change output to confirm or taking a risk and spending it while unconfirmed.

For a large exchange "output management" is going to be important.  Having all your funds in a single unspent output is going to kill real time transaction processing.

[John Tobey]

Ongoing work here:

https://github.com/bitcoin/bitcoin/pull/3651

And here, for the other issue (duplicates counted in the balance)

[DeathAndTaxes]

Ongoing work here:

https://github.com/bitcoin/bitcoin/pull/3651

And here, for the other issue (duplicates counted in the balance)

I had no doubts that it would get done.  Reading the comments though shows it is a challenge.  Looks like a bug already exists where if you have sufficient confirmed outputs to cover the amount sent but not the amount of the fee, the fee will be paid using unconfirmed output which breaks the "fix".

Hopefully the OP helps "non programmers" to understand the issue and what to expect.

[etotheipi]

Can someone please elaborate why the unconfirmed-change-spending issue is a real problem? To me, it sounds like it may cause some users to accidentally reverse subsequent zero-confirmation transactions, and that's it.  But the community has been pretty darn vocal about telling people/merchants not to accept zero-conf tx, and there's plenty of ways to intentionally double-spend them without this bug/quirk.  It seems that this particular part of the problem is simply making it (more) painfully obvious why they shouldn't be accepting zero-conf tx.

What am I missing? 

(Of course I know there's bigger problems with transaction mutability than just spending unconf change, but that seems to be the only issue Armory currently has ... if it's a real issue)