Explain the gox transaction malleability issue like you are five

[jl2012]

for those who own btc with gox, it seems like the issue is going to be the extent to which gox "re-issued" bitcoin deliveries to customers bc the recipients lied and said they had not received it.  if there were many double deliveries, then gox may not have the bitcoin to pay remaining owners all the bitcoin to which they are entitled.  a few issues/questions come to mind:

1. why the other exchanges are not having the same problem
2. would mtgox be able to figure out which customers claimed failed delivery but lied, and were then paid twice.
3. it seems like the gox omnibus acct should net out even at the end of the day.  double deliveries would raise red flags (one would think) when all has been netted

1. Gox uses a customized wallet, which is obviously faulty. Other exchanges either implement it correctly (checking the address balance), or simply use standard bitcoind

2. Yes, if they have kept all the transaction and conversation log.

3. Not sure what you mean. But if gox really double paid some customers, they are the one to absorb the loss (or they will close and run)

[stompix]

for those who own btc with gox, it seems like the issue is going to be the extent to which gox "re-issued" bitcoin deliveries to customers bc the recipients lied and said they had not received it.  if there were many double deliveries, then gox may not have the bitcoin to pay remaining owners all the bitcoin to which they are entitled.  a few issues/questions come to mind:

1. why the other exchanges are not having the same problem
2. would mtgox be able to figure out which customers claimed failed delivery but lied, and were then paid twice.
3. it seems like the gox omnibus acct should net out even at the end of the day.  double deliveries would raise red flags (one would think) when all has been netted

Because other exchangers aren't being run by fairies dancing around the magical pot of bitcoins when the owner is drunk dead in the basement ?

[ManeBjorn]

So basically MTGoX made a mess and they blame it on something else to try and look like a victim instead of lazy with their own system. 
Glad I did not keep any BTC there.

[nakaone]

for those who own btc with gox, it seems like the issue is going to be the extent to which gox "re-issued" bitcoin deliveries to customers bc the recipients lied and said they had not received it.  if there were many double deliveries, then gox may not have the bitcoin to pay remaining owners all the bitcoin to which they are entitled.  a few issues/questions come to mind:

1. why the other exchanges are not having the same problem
2. would mtgox be able to figure out which customers claimed failed delivery but lied, and were then paid twice.
3. it seems like the gox omnibus acct should net out even at the end of the day.  double deliveries would raise red flags (one would think) when all has been netted

1. Gox uses a customized wallet, which is obviously faulty. Other exchanges either implement it correctly (checking the address balance), or simply use standard bitcoind

2. Yes, if they have kept all the transaction and conversation log.

3. Not sure what you mean. But if gox really double paid some customers, they are the one to absorb the loss (or they will close and run)

thanks for the great explanation - are other services outside exchanges also using this customized wallet?

[Nancarrow]

Because other exchangers aren't being run by fairies dancing around the magical pot of bitcoins when the owner is drunk dead in the basement ?

I think you mean "dead drunk", though I like your way better.

[bitbouillion]

Even worse, some customers find the gox's bug and try to exploit it. After they cashed the cleaned cheque, they complain to gox saying that they have not received a cheque. Since gox can't find the cheque in the record of Bitcoin Bank, they credit the bitcoin back to the customer's gox account so the customer doubled his bitcoin at the expense of gox's fund

That could be easily proven by gox, since there will be "double spends" to the same address in the blockchain.

[stompix]

Because other exchangers aren't being run by fairies dancing around the magical pot of bitcoins when the owner is drunk dead in the basement ?

I think you mean "dead drunk", though I like your way better.

I always thought it's first the drinking and then the dying =))))  , so > drunk dead.

But right now ,It  won't be long till it he ends dead with lots of holes in it , depending on how much money people lost with his exchanger.

[JerryHou]

So Gox has to manually compare all previous cheques issued with photos to sort out the mess of their cashbook?

[jl2012]

So Gox has to manually compare all previous cheques issued with photos to sort out the mess of their cashbook?

It is always not a good practice to compare the photos (transaction hash). They should check the balance of all their bitcoin accounts

[Armis]

Good explanation. Thank you.
Now I wonder, why someone doesn't open a bank called 'The Bitcoin Bank' and operate it in a country like Germany, Finland or the other few that actually regulated bitcoin in a positive way?

because at that point they would be a bank first, a bitcoin facilitator second, even if their only "currency" is cryptocurrency.

[nullus]

So Gox has to manually compare all previous cheques issued with photos to sort out the mess of their cashbook?

It is always not a good practice to compare the photos (transaction hash). They should check the balance of all their bitcoin accounts

A cryptographic hash is not a "photo", nor alternate representations comparable to a "dirty cheque".

They must be unique and reproducible identifiers which are unambiguous. If this is untrue of transaction IDs, why bother returning them at all? In that sense it is a protocol flaw.

Minor vulnerabilities can be used to leverage more serious exploits, this needs to be fixed.

[FiatKiller]

FYI, at least 3 of us had issues last night with double transactions. We sent a payment and it appeared a second time, one of them is unconfirmed. Not sure if it is related to this.

[il--ya]

The problem is, there is indefinite ways to alter a cheque without invalidate it. We need to live with transaction malleability and actually it's no big deal

Malleability is a potential and hypothetical issue nuisance, which only became possible to exploit at MtGox for two reasons: because Gox failed to correctly implement Bitcoin specification properly, and also because it failed to implement proper workarounds for this issue. You correctly pointed out second reason, but the first is more important to point out, in my opinion, because this is why other exchanges are much less likely to be affected, if likely at all.
Gox didn't follow the specification, which required tx signature to be encoded with ASN1/DER encoding. This requirement was specified in April 2011: https://en.bitcoin.it/wiki/Protocol_specification#Signatures
Instead they used some sloppy format which was not DER encoding but was still accepted by SSL library and old reference client. When tighter checks were implemented in bitcoin reference client (the main reason for which was actually to prevent malleability issue), their transactions, which violated bitcoin spec, were rejected. Basically, their transactions looked like what hackers would employ to exploit this issue. That allowed hackers to pick these rejected transactions up, malleate them to "fix" the signature format, and re-submit. Ironically, hackers were helping MtGox to propagate their malformed transactions through the network.
If MtGox submitted correct DER-encoded signature in the first place, hackers would have to figure out how to malleate the transaction without breaking the spec so that the transaction is still accepted by the network, and then outrace the originator of the transaction in propagating the malleated message to the miners, or use their own miner, and be lucky enough to mine the block before somebody else mined the original transaction. A lot of trouble - and all that just to modify tx id, which is already a published known issue, and can be easily detected with manual investigation or proper workarounds in place.

It's not malleability per se which is the reason for MtGox failure, but failure to properly implement bitcoin specification + malleabiiliy + failure to implement workaround + lack of attention to abnormal behaviour in their system + inability to react quickly when the issue became glaringly apparent+not testing their system with reference implementation+....
Or, in other words, lack of competence, unacceptable for the only golden member of Bitcoin Foundation.
There was so may ways this could have been alright - and only utter stupidity and incompetence allowed this to go wrong.

[il--ya]

The big question is how long has this been going on and has someone actively exploited it?

This is simply gox's problem, as they shouldn't follow the transaction flow this way in the first place.

It's wrong to think this is just Gox's problem. It's a problem of Gox's customers (large part of bitcoin community), and this is a problem of Bitcoin public image. When "the oldest and at one point the biggest bitcoin exchange" is run by such moron, that taints the whole community.

[il--ya]

3. Not sure what you mean. But if gox really double paid some customers, they are the one to absorb the loss (or they will close and run)

Actually they may try to recover some, if they are able to figure out who did that. But I wouldn't bet much on it.
Also they may go into liquidation in a civilized way. That would be the best option for everybody. Bitcoin foundation should use all their influence to pressure Mark to do that.

[whitenight639]

The problem is, there is indefinite ways to alter a cheque without invalidate it. We need to live with transaction malleability and actually it's no big deal

Gox didn't follow the specification, which required tx signature to be encoded with ASN1/DER encoding. This requirement was specified in April 2011: https://en.bitcoin.it/wiki/Protocol_specification#Signatures
Instead they used some sloppy format which was not DER encoding but was still accepted by SSL library and old reference client. When tighter checks were implemented in bitcoin reference client (the main reason for which was actually to prevent malleability issue), their transactions, which violated bitcoin spec, were rejected. Basically, their transactions looked like what hackers would employ to exploit this issue. That allowed hackers to pick these rejected transactions up, malleate them to "fix" the signature format, and re-submit. Ironically, hackers were helping MtGox to propagate their malformed transactions through the network.

So does that mean if you saw a transaction floating around the network that has a TX signature with junk padding, you could copy it, remove the padding and resend so the benifactor of that transaction would get paid twice? I take it that would have to be someone from Mt Gox resending the payment in order for it to work?

So just to be clear it is not possible to find an Unconfirmed Transaction on the Network with junk padding, copy it and change things like recipient, amount, remove padding and resend?

Jeez you would think Mt Gox would be looking for ways to speed up transactions, so you would think they would know Miners are rejecting their transactions with junk padding on the signatures and amend any script so as to remove the junk padding and speed up confirmation times. Crazy incompetence.   

[il--ya]

So does that mean if you saw a transaction floating around the network that has a TX signature with junk padding, you could copy it, remove the padding and resend so the benifactor of that transaction would get paid twice?

No, double-spending is not possible within bitcoin protocol. MtGox didn't see that original transaction went through and re-issued it (automatically or manually - I don't know) with different inputs.

So just to be clear it is not possible to find an Unconfirmed Transaction on the Network with junk padding, copy it and change things like recipient, amount, remove padding and resend?

You cannot change recepient or amount without breaking the signature and invalidating it. You can remove padding though - thus creating a good-looking transaction. If the inputs used in this transaction were not yet spent, you can get this transaction confirmed (included into a block). So basically you will do what originator of this transaction was intending to do anyway, the only difference is that you modify transaction in a way which may be unexpected by some outdated bitcoin software like MtGox's. And that is a known issue since 2011, so reference client and probably most custom clients don't rely on that and are not affected.

Jeez you would think Mt Gox would be looking for ways to speed up transactions, so you would think they would know Miners are rejecting their transactions with junk padding on the signatures and amend any script so as to remove the junk padding and speed up confirmation times. Crazy incompetence.  

They became too big and important to care. Just like many banks.
This is my pure speculation, but probably their attempts to speed up transactions actually played role here too. They had partner miners, so they were able to send transactions directly to them, bypassing other nodes. "Network doesn't accept our transactions? Fuck the network, we will go straight to the miners". Until miners finally updated their software.

[DeathAndTaxes]

3. it seems like the gox omnibus acct should net out even at the end of the day.  double deliveries would raise red flags (one would think) when all has been netted

You assume MtGox is doing daily reconciliation.  Given all the other things they either didn't do, or did wrong why would you assume that.  It is entirely possible they had absolutely no check's and balances in place and the same attacker "double withdrew" dozens or maybe even hundreds of times.  The issue with MtGox generated bad withdraws has been going on for a month.

[DeathAndTaxes]

So does that mean if you saw a transaction floating around the network that has a TX signature with junk padding, you could copy it, remove the padding and resend so the benifactor of that transaction would get paid twice? I take it that would have to be someone from Mt Gox resending the payment in order for it to work?

Yes and that is what it is assumed that attackers did.  Of course of that original pair only one could be confirmed so yes to get double paid you would also need to trick MtGox into cutting you another payment.   Of course MtGox client is horribly defective and there were thousands and thousands of legitimate reasons why they had to cut new payments so the attackers requests would hide in a sea of requests created by their incompetence.

So just to be clear it is not possible to find an Unconfirmed Transaction on the Network with junk padding, copy it and change things like recipient, amount, remove padding and resend?

Correct, you can not do that.  The inputs, outputs, value of tx, fee paid, recipients, and value to each recipient are immutable.

Jeez you would think Mt Gox would be looking for ways to speed up transactions, so you would think they would know Miners are rejecting their transactions with junk padding on the signatures and amend any script so as to remove the junk padding and speed up confirmation times. Crazy incompetence.   

The OP was a simplified explanation of what MtGox got wrong.  MtGox had a huge laundry list of fails in their client wallet. At minimum (just by observing their "missing" transactions) they
a) tried to spend immature newly mined coins which caused the tx to be dropped (until they matured if Gox was still broadcasting it) by some or all of their peers.
b) tried to make payments which violated the spam rules and thus wouldn't be relayed by some or all of their peers.
c) paid insufficient fees on low priority tx which caused them to not be relayed by some or all of their peers.
d) used non canonical signatures which caused them to not be relayed by some or all of their peers.
e) double spent their own coins which caused the tx to be dropped (correctly) by some or all of their peers.

It is very likely their wallet was deficient in other ways these are just symptoms that I and others have observed by looking at the transactions MtGox created.  As an example, obviously if MtGox is creating a tx which is spending newly mined coins less than 120 blocks from when they were mined we know their wallet is not performing that required check.  How many other deficiencies are in the "Gox Special v0" custom client.  Nobody outside MtGox knows for sure but you should not take this list to be exhaustive.

[justusranvier]

3. it seems like the gox omnibus acct should net out even at the end of the day.  double deliveries would raise red flags (one would think) when all has been netted

You assume MtGox is doing daily reconciliation.

Even daily reconciliation isn't really good enough. It's the 21st century - why shouldn't these kinds of services be expected to reconcile with each new block?