What MtGox didn't say: Their bad code hygiene was the direct cause of problems

[Falkvinge]

Like the rest of the community, I was rather angry at MtGox' attempt to shift blame onto the bitcoin protocol as such yesterday. Something else entirely is at fault, and MtGox didn't mention it in their press release.

Therefore, I did a writeup on it this morning (European time).

The Embarrassing Fact MtGox Left Out Of Their Press Release: Their Bad Code Hygiene Was The Direct Cause Of Problems

Yesterday, the bitcoin exchange MtGox – riddled by problems – issued a press release saying the bitcoin protocol was to blame for its ongoing problems. That statement, which caused the markets to nosedive temporarily, is outright false. The problem is, and was, bad code hygiene in the MtGox exchange itself. Here are the details.

http://falkvinge.net/2014/02/11/the-embarrassing-fact-mtgox-left-out-of-their-press-release/

[Ente]

Oh, wow!
This is.. just wow.
It explains pretty much everything, this seems very plausible for me.
Thank you for your analysis.

Oh, and great to have you here, Rick!

Ente

[Falkvinge]

It explains pretty much everything, this seems very plausible for me.
Thank you for your analysis.

Thank you for the kind words, and for submitting the article to /r/bitcoin!

Cheers,
Rick

[HeliKopterBen]

Looks like mt gox may have been right.  Other exchanges reporting similar problems:

Dear Bitstamp users

Bitstamp’s exchange software is extremely cautious concerning Bitcoin transactions. Currently it has suspended processing Bitcoin withdrawals due to inconsistent results reported by our bitcoind wallet, caused by a denial-of-service attack using transaction malleability to temporarily disrupt balance checking. As such, Bitcoin withdrawal and deposit processing will be suspended temporarily until a software fix is issued.

No funds have been lost and no funds are at risk.

This is a denial-of-service attack made possible by some misunderstandings in Bitcoin wallet implementations. These misunderstandings have simple solutions that are being implemented as we speak, and we're confident everything will be back to normal shortly.

Withdrawals which failed on the 10th and 11th of February will be canceled and the amounts added back to the customer account balances.

We will communicate any further developments regarding this issue.

Thank you for your understanding!

Best regards
Bitstamp team

https://www.bitstamp.net/article/bitcoin-withdraws-suspended/

[alfabitcoin]

Looks like mt gox may have been right.  Other exchanges reporting similar problems:

Dear Bitstamp users

Bitstamp’s exchange software is extremely cautious concerning Bitcoin transactions. Currently it has suspended processing Bitcoin withdrawals due to inconsistent results reported by our bitcoind wallet, caused by a denial-of-service attack using transaction malleability to temporarily disrupt balance checking. As such, Bitcoin withdrawal and deposit processing will be suspended temporarily until a software fix is issued.

No funds have been lost and no funds are at risk.

This is a denial-of-service attack made possible by some misunderstandings in Bitcoin wallet implementations. These misunderstandings have simple solutions that are being implemented as we speak, and we're confident everything will be back to normal shortly.

Withdrawals which failed on the 10th and 11th of February will be canceled and the amounts added back to the customer account balances.

We will communicate any further developments regarding this issue.

Thank you for your understanding!

Best regards
Bitstamp team

https://www.bitstamp.net/article/bitcoin-withdraws-suspended/

That what is going on now is direct consequence of mtgox cover up statemant. I am sure bitstamp will solve the issue at warp speed. Hacker now casing panic and ddos exchange in order to dip the price down.

[HeliKopterBen]

That what is going on now is direct consequence of mtgox cover up statemant. I am sure bitstamp will solve the issue at warp speed. Hacker now casing panic and ddos exchange in order to dip the price down.

Their issue is in part due to malleability just as mtgox's issue.  Both exchanges directly stated this.  Now btc-e users reporting delays:

https://bitcointalk.org/index.php?topic=459464.0

Sure looks like a systemic problem to me.  Probably an easy fix, but nevertheless affecting many users.

[alfabitcoin]

That what is going on now is direct consequence of mtgox cover up statemant. I am sure bitstamp will solve the issue at warp speed. Hacker now casing panic and ddos exchange in order to dip the price down.

Their issue is in part due to malleability just as mtgox's issue.  Both exchanges directly stated this.  Now btc-e users reporting delays:

https://bitcointalk.org/index.php?topic=459464.0

Sure looks like a systemic problem to me.  Probably an easy fix, but nevertheless affecting many users.

Reread it again. Mtgox withdraw problems started because their non updated custom wallet and improper padding. Malleability issue were known long ago. If mtgox fixed that their system bug they wohld never had a problems or now even worst as they published mallsabillity worldwide.
Due that mtgox stateman some malicus people attack ddos exchange and dust spam btc network.

So my point is that mtgox made from publicing maleabillity issue way worst then it is what is now exploitet. Beside, meanibillity are notba reason of their btc withdraw prolems but a two bugs in their custom wallet, what cased huge amount of stuck transaction.

[PirateHatForTea]

Lots of people, including HelikopterBen, seem to be misunderstanding this recent development with the ddos/transaction spamming, that has led to BitStamp and others to suspend withdrawals.

Transaction malleability is *potentially* an issue for everyone in that even if you do proper accounting and don't rely on TxID, if someone rebroadcasts a mutant transaction, for a while there will be two versions of the transaction kicking around the network. This is merely an inconvenience, not a threat, unless you are as incompetent as Gox was. There are at least two reasons why hackers/spambots rebroadcasting mutated transactions is unlikely to be a problem:

• The first and most important is that since we know that TXIDs are malleable and thus tell us nothing about whether BTC was sent or no, we cannot rely on TXID for our accounting. Mt Gox did so, and it allowed people to withdraw their balance multiple times from Gox.
• The other important reason is that the first (ie the true) transaction broadcast has the greatest chance of being accepted into the blockchain as it has a headstart in propogation. So even if you mutate and rebroadcast a transaction, it is the original that will make it into the blockchain, so all that is required to be safe is to wait for 1 confirmation. You can even rely on TXID (but definitely shouldn't!) if your transactions are able to win the propogation race 100% of the time. MtGox managed to fuck this one up as well, because they use non-standard padding that ends up creating transactions that are seens as invalid by the vanilla bitcoind client as of late last year. So [some of] their transactions get rejected by miners, and the mutant transactions are the ones that end up in the blockchain!

It really took this double-whammy of incompetence on the part of Gox to turn this minor issue/design decision in the Bitcoin protocol (that's been known about since 2011 and that every other wallet software has handled) into a big problem.

Stamp delaying withdrawals seems like a fairly responsible move, they may want to double check that their implementation is not affected by tx malleability - it sounded like their bitcoind is being confused by the multiple TXes, but that logic is not the logic for monitoring customer account balances so I doubt they suffer the same problem as gox. Though as Mike Caldwell rightly points out, there is no excuses for halting withdrawals for more than a couple of days, or for paying them any less than daily, as even a manual process would suffice.

[HeliKopterBen]

@PirateHatForTea

I understand the problem and as I said, the problem is due in PART to malleability as both exchanges directly cited.  Only bitstamp cited DDOS as part of the problem.  I am definitely not trying to defend gox as I think they have shady practices, but the problem was severe enough for both exchanges to completely halt withdrawals for a lengthy period of time, so it wasn't just a check on the part of bitstamp. 

Also, I agree with Mike Caldwell, the manner in which the exchanges handle the problem and treat their customers at this time will be very telling as to the character of the exchange.

[Sukrim]

Also bitcoin-qt up to 0.8.6 was/is affected of widespread transaction mutation in the wild...

Just putting the blame on Gox is a bit short sighted, after all a "transaction ID" that is completely random and useless until that transaction is buried a few blocks deep is NOT something implementors actually do expect I guess and while there was some theoretical info on that available, there surely was no big warning like "TXID should never be assumed to stay the same after broadcast!".

All in all, the situation sucks, Gox gets blamed and Bitcoin moves on. Welcome to 2011 2012 2013 2014!

[PirateHatForTea]

Helikopter Ben: Ah ok, glad that you do get it. I'm just getting sick of stooges turning up and posting stuff like 'See! Gox were right, you were too hard on them!'

No, Gox were not 'right'. They SPUN the shit outta that announcement - doing everything they could to hide/gloss over their mistakes (which even if we are gentle are best termed massive fuck-ups) and trying to blame the protocol for a design decision (which yes, has some undesirable, but not fatal, consequences) that has been well-documented since 2011! Particularly disingenuous is that they made no mention of the bad transactions they were sending out due to ignoring the tightened standards on padding which, ironically, were tightened to fix this so-called 'bug'. Without that piece of code bodginess, any attack would have required being better-connected to the network than Gox, which is difficult/expensive.

So yes actually, Gox is the only one to blame for transaction malleability having serious consequences, as you can see from reading my post. As far as we know at least, no other exchange created the circumstances to allow for it to be any more than a minor inconvenience.

I could have held off blaming Gox until that announcement, but the announcement was so irresponsible and cravenly (I mean, seriously, they need to learn to OWN their mistakes) that the blame is now less about the mistakes that they made in their implementation - which is incredibly poor form for such a large/wealthy org - and more about their pissant response.

[Bit_Happy]

Lots of people, including HelikopterBen, seem to be misunderstanding this recent development with the ddos/transaction spamming, that has led to BitStamp and others to suspend withdrawals.

Transaction malleability is *potentially* an issue for everyone in that even if you do proper accounting and don't rely on TxID, if someone rebroadcasts a mutant transaction, for a while there will be two versions of the transaction kicking around the network. This is merely an inconvenience, not a threat, unless you are as incompetent as Gox was. There are at least two reasons why hackers/spambots rebroadcasting mutated transactions is unlikely to be a problem:

• The first and most important is that since we know that TXIDs are malleable and thus tell us nothing about whether BTC was sent or no, we cannot rely on TXID for our accounting. Mt Gox did so, and it allowed people to withdraw their balance multiple times from Gox.
• The other important reason is that the first (ie the true) transaction broadcast has the greatest chance of being accepted into the blockchain as it has a headstart in propogation. So even if you mutate and rebroadcast a transaction, it is the original that will make it into the blockchain, so all that is required to be safe is to wait for 1 confirmation. You can even rely on TXID (but definitely shouldn't!) if your transactions are able to win the propogation race 100% of the time. MtGox managed to fuck this one up as well, because they use non-standard padding that ends up creating transactions that are seens as invalid by the vanilla bitcoind client as of late last year. So [some of] their transactions get rejected by miners, and the mutant transactions are the ones that end up in the blockchain!

It really took this double-whammy of incompetence on the part of Gox to turn this minor issue/design decision in the Bitcoin protocol (that's been known about since 2011 and that every other wallet software has handled) into a big problem.

Stamp delaying withdrawals seems like a fairly responsible move, they may want to double check that their implementation is not affected by tx malleability - it sounded like their bitcoind is being confused by the multiple TXes, but that logic is not the logic for monitoring customer account balances so I doubt they suffer the same problem as gox. Though as Mike Caldwell rightly points out, there is no excuses for halting withdrawals for more than a couple of days, or for paying them any less than daily, as even a manual process would suffice.

Good clear summary, thanks.
It's been an exciting week