Stats on malled transactions

[klondike_bar]

Maybe this is too simple to be right, but...
Would not running rescan in bitcoin-QT from the time "some time before problematic unconfirmed-change-spending transaction" help?

It presumably would, and a few options such as last 100 or last 1000 might be sufficient.

The only concern would be that:
a) this would need to be implemented by each client
b) it *could* create weak points for an evil trojan/virus to replace and then rescan a false short blockchain - seems unlikely though

[ZephramC]

Maybe this is too simple to be right, but...
Would not running rescan in bitcoin-QT from the time "some time before problematic unconfirmed-change-spending transaction" help?

It presumably would, and a few options such as last 100 or last 1000 might be sufficient.

The only concern would be that:
a) this would need to be implemented by each client
b) it *could* create weak points for an evil trojan/virus to replace and then rescan a false short blockchain - seems unlikely though

a) this is already implemented in each bitcoin-QT client. Although it would be time consuming to do it regularly.
b) creating "false blockchain" is almost impossible, it needs forking blockchain somewhere and then constructing new "false blocks". This "construction" means in fact mining new alternative and legitimate blocks from this fork. No single entity has enough computational power to do this. And some "easier blockchain" would not be correctly verified by reference client.

[Jan]

Seems we are back to business as usual.
2014-02-12 393
Updating OP.

[AndersAA]

Seems we are back to business as usual.
2014-02-12 393
Updating OP.

Any reason to believe it won't happen again tomorrow? I think we'd all like to know when Bitstamp and other exc will start processing withdrawals again ;-)

[djeZo]

Really sad that there is no solution to even clean these transactions manually... at least not for official bitcoin-qt client.

[BurtW]

For those that have transactions that will never confirm, what is the procedure on QT to clean up the wallet?

[Peter R]

Here in Vancouver, several brick-and-mortar businesses accept zero-confirm transactions using BitPay.  If you are buying a coffee, no one is going to wait 10 minutes (or longer) for confirmation;  it is critical that zero-confirm transactions are reasonably secure in order for this business model to work.  

So let me see if I understand the transaction malleability problem in relation to brick-and-mortar stores:

I pay for my coffee at Waves for 4mBTC.  Assume that to avoid "address reuse," my phone uses the full balance in 1AddressA, sends 4mBTC to Waves, and the rest to a new change address 1AddressB.  BitPay accepts this transaction and I get my coffee.  As soon as I finish paying, I realize that I also wanted a donut.  So I pay for this using what are now unconfirmed coins sitting in 1AddressB.  BitPay still approves the transaction and I get my donut too.  

But my first transaction was mutated, and the mutated version was accepted into the blockchain.  This means that the transaction I used to pay for my donut is now invalid, and Waves/BitPay won't get the money (and I've already left the store).  

This seems like a messy problem to deal with.  If we disallow spending unconfirmed change outputs, then there will be certain cases when I can't actually purchase my donut [without a long wait], correct?  Ideally, my wallet would try to break up my coins so that there are always plenty of fully-confirmed outputs ready for spending.  But how do most wallets actually work?  The Blockchain.info mobile wallet [that I use] sends the change back to the same address, so I'd expect that it would be very rare to have exhausted all of the confirmed outputs.  But for mobile wallets that avoid address reuse [do these even exist?], would it be less likely that you'd have confirmed coins available?

And should BitPay do anything to protect itself?

[Syke]

But my first transaction was mutated, and the mutated version was accepted into the blockchain.  This means that the transaction I used to pay for my donut is now invalid, and Waves/BitPay won't get the money (and I've already left the store).

Yes, that is possible, but that's nothing new. Accepting with zero-confirm transactions is risky.

[Peter R]

But my first transaction was mutated, and the mutated version was accepted into the blockchain.  This means that the transaction I used to pay for my donut is now invalid, and Waves/BitPay won't get the money (and I've already left the store).

Yes, that is possible, but that's nothing new. Accepting with zero-confirm transactions is risky.

I disagree.  I think the community now sees zero-confirm transactions as more risky than prior to the malleability attack.  I know I do.  Previously, I think most people were under the impression that to invalidate a zero-confirm transaction (e.g., buying my donut at Waves in Vancouver), the customer would be the attacker (and basically had to be in cahoots with a nefarious miner in order to double spend).  The malleability attack has shown that even if both customer and merchant are honest, proper fees are paid, and the transaction propagates across the network, it's still possible (and in some cases probable) that the transaction doesn't go through (and thus D&T's solution to disallow transactions using unconfirmed "change" outputs).  

So I agree with D&T that we need a smarter way to deal with unconfirmed change, until malleability is properly fixed.  My question was if we enforce this new change rule, how often will casual users find themselves unable to pay immediately (such as my donut example)?  How will this impact the experience of say, my mom using her smartphone, who couldn't wrap her head around this?  

[F-bernanke]

I agree, Before this I pretty much trusted each transaction without confirmations if as there was a fee paid. I really hope this Issue wil be fixed soon.

[waxwing]

Here in Vancouver, several brick-and-mortar businesses accept zero-confirm transactions using BitPay.  If you are buying a coffee, no one is going to wait 10 minutes (or longer) for confirmation;  it is critical that zero-confirm transactions are reasonably secure in order for this business model to work.  

So let me see if I understand the transaction malleability problem in relation to brick-and-mortar stores:

I pay for my coffee at Waves for 4mBTC.  Assume that to avoid "address reuse," my phone uses the full balance in 1AddressA, sends 4mBTC to Waves, and the rest to a new change address 1AddressB.  BitPay accepts this transaction and I get my coffee.  As soon as I finish paying, I realize that I also wanted a donut.  So I pay for this using what are now unconfirmed coins sitting in 1AddressB.  BitPay still approves the transaction and I get my donut too.  

But my first transaction was mutated, and the mutated version was accepted into the blockchain.  This means that the transaction I used to pay for my donut is now invalid, and Waves/BitPay won't get the money (and I've already left the store).  

This seems like a messy problem to deal with.  If we disallow spending unconfirmed change outputs, then there will be certain cases when I can't actually purchase my donut [without a long wait], correct?  Ideally, my wallet would try to break up my coins so that there are always plenty of fully-confirmed outputs ready for spending.  But how do most wallets actually work?  The Blockchain.info mobile wallet [that I use] sends the change back to the same address, so I'd expect that it would be very rare to have exhausted all of the confirmed outputs.  But for mobile wallets that avoid address reuse [do these even exist?], would it be less likely that you'd have confirmed coins available?

And should BitPay do anything to protect itself?

Very good example to think about, but: did I miss something? I think it won't happen, or at least doesn't have to. When you try to pay for the donut, assuming your wallet makes use of the change utxo which has been malled and is not yet confirmed, will bitpay accept it? I don't know whether they would today, but isn't it trivial to just check each utxo used as input and look at the blockchain to see if that transaction has any confirms? That way, they can accept unconfirmed spends, but not accept unconfirmed spends of unconfirmed spends, which is practically absolutely fine. Except you don't get your donut

Unless your wallet is coded to not use unconfirmed change...

[DeathAndTaxes]

Here in Vancouver, several brick-and-mortar businesses accept zero-confirm transactions using BitPay.  If you are buying a coffee, no one is going to wait 10 minutes (or longer) for confirmation;  it is critical that zero-confirm transactions are reasonably secure in order for this business model to work. 

So let me see if I understand the transaction malleability problem in relation to brick-and-mortar stores:

I pay for my coffee at Waves for 4mBTC.  Assume that to avoid "address reuse," my phone uses the full balance in 1AddressA, sends 4mBTC to Waves, and the rest to a new change address 1AddressB.  BitPay accepts this transaction and I get my coffee.  As soon as I finish paying, I realize that I also wanted a donut.  So I pay for this using what are now unconfirmed coins sitting in 1AddressB.  BitPay still approves the transaction and I get my donut too. 

But my first transaction was mutated, and the mutated version was accepted into the blockchain.  This means that the transaction I used to pay for my donut is now invalid, and Waves/BitPay won't get the money (and I've already left the store). 

This seems like a messy problem to deal with.  If we disallow spending unconfirmed change outputs, then there will be certain cases when I can't actually purchase my donut [without a long wait], correct?  Ideally, my wallet would try to break up my coins so that there are always plenty of fully-confirmed outputs ready for spending.  But how do most wallets actually work?  The Blockchain.info mobile wallet [that I use] sends the change back to the same address, so I'd expect that it would be very rare to have exhausted all of the confirmed outputs.  But for mobile wallets that avoid address reuse [do these even exist?], would it be less likely that you'd have confirmed coins available?

And should BitPay do anything to protect itself?

Very good example to think about, but: did I miss something? I think it won't happen, or at least doesn't have to. When you try to pay for the donut, assuming your wallet makes use of the change utxo which has been malled and is not yet confirmed, will bitpay accept it? I don't know whether they would today, but isn't it trivial to just check each utxo used as input and look at the blockchain to see if that transaction has any confirms? That way, they can accept unconfirmed spends, but not accept unconfirmed spends of unconfirmed spends, which is practically absolutely fine. Except you don't get your donut

Unless your wallet is coded to not use unconfirmed change...

Today BitPay would accept it.  Unconfirmed change is used rather routinely although probably will be changing soon.  In your scenario though the bad news is not BitPay needs to figure out a way to return your coins.  You can't undo it and eventually it will confirm (unless a duplicate of the prior output is what confirms).  So you are now out the donut and coins.  Of course the shop will have no clue how to get your coins back, and "you" (or a less educated user) might be kinda upset seeing the donut funds deducted from your wallet but not getting the donut and the clueless clerk having no idea how to get your coins back or even where they went.

If this happened enough the store might just not accept Bitcoin in the future.  Now I am not going all doom and gloom and none of this is unsolvable but you can see how all this starts to get really ugly real quick.  Transaction Ids need to be immutable.  Period.   There is no other viable long term option.  However that is going to take some time so things might get a little clunky for service providers before they get better.

[waxwing]

In your scenario though the bad news is not BitPay needs to figure out a way to return your coins.  You can't undo it and eventually it will confirm (unless a duplicate of the prior output is what confirms).  So you are now out the donut and coins. 

Oh yes of course, stupid of me, the customer loses out there. But it still always comes back to the same thing for me: wallets should not attempt to spend unconfirmed change. Yes, I know there are some practical difficulties there with the way wallets are designed, but it doesn't look insurmountable - at worst it just causes a few inconveniences.

[Rampion]

In your scenario though the bad news is not BitPay needs to figure out a way to return your coins.  You can't undo it and eventually it will confirm (unless a duplicate of the prior output is what confirms).  So you are now out the donut and coins.  

Oh yes of course, stupid of me, the customer loses out there. But it still always comes back to the same thing for me: wallets should not attempt to spend unconfirmed change. Yes, I know there are some practical difficulties there with the way wallets are designed, but it doesn't look insurmountable - at worst it just causes a few inconveniences.

Do you realize how confusing and frustrating can be for users to disallow spending of unconfirmed change?

Imagine you received 100 BTC and you spend 1 BTC - suddenly 99 BTC become unspendable for 5,10, 30 minutes... That's very counter intuitive and will confuse a lot many users who do not understand how the protocol works.

Not allowing to spend unconfirmed change is an important steps backwards, and I think that we shouldn't downplay it's effect. I'd probably have it as an optional feature on the client, but if possible I wouldn't make it the default behaviour.

About 0-conf: for small, day by day payments they were absolutely safe. The cost to perform a double spend was simply not worth the cost of a coffee. Now, that changes, 0-conf are not safe even if the buyer is 100% honest and the cost of the product is negligible: again we cannot downplay the effect of that, it's quite serious.

[oakpacific]

In your scenario though the bad news is not BitPay needs to figure out a way to return your coins.  You can't undo it and eventually it will confirm (unless a duplicate of the prior output is what confirms).  So you are now out the donut and coins.  

Oh yes of course, stupid of me, the customer loses out there. But it still always comes back to the same thing for me: wallets should not attempt to spend unconfirmed change. Yes, I know there are some practical difficulties there with the way wallets are designed, but it doesn't look insurmountable - at worst it just causes a few inconveniences.

The client could just try to/prompt to respend if after 10 minutes or so if it fails to see the expected change txid, but some outputs summed to the same amount with another txid getting confirmed in a block, and the sum of all outputs doesn't change.

[waxwing]

In your scenario though the bad news is not BitPay needs to figure out a way to return your coins.  You can't undo it and eventually it will confirm (unless a duplicate of the prior output is what confirms).  So you are now out the donut and coins.  

Oh yes of course, stupid of me, the customer loses out there. But it still always comes back to the same thing for me: wallets should not attempt to spend unconfirmed change. Yes, I know there are some practical difficulties there with the way wallets are designed, but it doesn't look insurmountable - at worst it just causes a few inconveniences.

Do you realize how confusing and frustrating can be for users to disallow spending of unconfirmed change?

Imagine you received 100 BTC and you spend 1 BTC - suddenly 99 BTC become unspendable for 5,10, 30 minutes... That's very counter intuitive and will confuse a lot many users who do not understand how the protocol works.

Not allowing to spend unconfirmed change is an important steps backwards, and I think that we shouldn't downplay it's effect. I'd probably have it as an optional feature on the client, but if possible I wouldn't make it the default behaviour.

About 0-conf: for small, day by day payments they were absolutely safe. The cost to perform a double spend was simply not worth the cost of a coffee. Now, that changes, 0-conf are not safe even if the buyer is 100% honest and the cost of the product is negligible: again we cannot downplay the effect of that, it's quite serious.

Yes, I do appreciate the problem. But I would say it could/should be attacked from three angles: 1, disallow it by default but allow it to be switched on, 2, make people aware that they sometimes have to wait for confirmations before proceeding and 3, some alterations within wallets, e.g. automatic internal transfers to split up into denominations.

For sure, it is much more convenient if we don't have to worry about this - but if I'm reading things correctly, malleability is with us for some time.

Zero conf payments still work - it's just zero conf payments spending existing  zero confirm payments that don't.

[DeathAndTaxes]

Do you realize how confusing and frustrating can be for users to disallow spending of unconfirmed change?

Imagine you received 100 BTC and you spend 1 BTC - suddenly 99 BTC become unspendable for 5,10, 30 minutes... That's very counter intuitive and will confuse a lot many users who do not understand how the protocol works.

Wallets will need to be "smarter".  A wallet knows how many unique unspent outputs it has.

So in the 100 BTC example it could spend it back to itself as 5x 20 BTC outputs.  Then a user could spend 1, have 19 BTC change confirming and still have 80 BTC available to spend.

Wallets also probably need to use simpler language like:  "Total balance: XXXX (available to spend: XXXX)  Why is this different?" with link/popup to simple user level explanation.

A "smart wallet" can engage in output management to minimize the disruption to a user.  There is no reason a tx must have 1 change output (to the protocol change is simply an output no different than the "intended" output).  If anything multiple change outputs would make blockchain analysis slightlymore difficult.  So when creating an outbound tx the wallet could look at its internal confirmed output pool and if it is "low" make 2 or more change outputs.  How many to make would depend a lot of the amount of the change (in BTC) and the number of unspent outputs (in discrete outputs not value).

Instead of

Code:

Input[0]:  Value 100 BTC

Output[0]:  <Receivers PubKey> Value 1 BTC  <- payment
Output[1]:  <Change PubKey> Value 99 BTC <- change

The wallet could instead do this:

Code:

Input[0]:  Value 100 BTC

Output[0]:  <Receivers PubKey> Value 1 BTC  <- payment
Output[1]:  <Change PubKey #1> Value * BTC <- change
Output[2]:  <Change PubKey #2> Value * BTC <- change
...
Output[n]:  <Change PubKey #n> Value * BTC <- change

Where * is a random amount such that the sum of the change addresses is 99 BTC.

About 0-conf: for small, day by day payments they were absolutely safe. The cost to perform a double spend was simply not worth the cost of a coffee. Now, that changes, 0-conf are not safe even if the buyer is 100% honest and the cost of the product is negligible: again we cannot downplay the effect of that, it's quite serious.

It is important now with mass mutation attack going on to distinguish between:

0 confirm tx where all inputs are confirmed
vs
0 confirm tx where one or more inputs are also 0 confirm.

The former requires a race attack and malicious intent of the sender for the receiver to lose funds.  So if you trust your customers, or feel only a small % of your customers would attempt an attack you are relatively safe (especially for low value, low risk purchases).   The later however only requires that SOMEONE on the network mutate the prior unconfirmed transaction and that the mutated version ends up confirming.   The risk profile becomes "do you trust the entire world to not mess with your money even though it is painfully easy?"  In most cases that answer should be no.

In the donut shop example up thread, the customer isn't the one doing the mutating. He could be a regular customer and liked by the shopkeeper however his coffee tx gets mutated by a malicious third party.  Now mutated or not the coffee tx will go through, no risk to the shop there.   However if the duplicate is the one which is confirmed, then the change output used for the doughnut becomes invalid.  Now through no fault of either the shopkeeper or the legit customer the donut payment will never confirm.  It gets worse when you consider how confusing this will be to put the shopkeeper and the customer especially if their are "crypto-nerds".

Honestly until tx ids are immutable the use of unconfirmed change outputs is just going to cause mass chaos.  The long term goal should be immutable tx ids but lets be honest that is probably a six to nine month project.  It will most likely require a hard fork of the protocol and there is simply no fast and easy way to do that.    So in the immediate future transactions are mutable (even by third parties) and that means unconfirmed change can be broken by a malicious third party.  For the record the "DDOS attack" against exchanges involves this aspect of mutable transactions. 

I would also point out how nuanced this conversation is.  Now imagine someone who has a "user level" knowledge of Bitcoin.  Many people have no idea that Bitcoin works on the concepts of inputs and outputs.  Their knowledge is no deeper than the "abstraction" of wallet balances.  Now that is fine, the job of good software is to abstract a very complex system into a simple concept.  It isn't the user's fault that they don't know the inner workings, however you can't even start to explain the problem, until you explain how Bitcoin "really works".  That simply isn't a viable scenario for non-enthusiasts.  Case in point my need for a side note here: https://bitcointalk.org/index.php?topic=460944.0

[phelix]

Seems we are back to business as usual.
2014-02-12 393
Updating OP.

Is this because of the attacker / the exchanges on hold?

Or is it because pools updated their software with the 0.8 fixes?

[DeathAndTaxes]

Seems we are back to business as usual.
2014-02-12 393
Updating OP.

Is this because of the attacker / the exchanges on hold?

Or is it because pools updated their software with the 0.8 fixes?

The attacker stopped.  Why and for long run remains unknown.  A patched client doesn't prevent the duplicates from existing so if tx were still being duplicated they should show up.

[Bogart]

Wouldn't a better fix be to change the client behavior such that there is only one correct way to encode a transaction (with enforcement to begin in some future block), removing the possibility of malleability?