Bitcoin Forum

Let's assume we have a bank called "Bitcoin Bank". People can open accounts at the bank, get an account number (bitcoin address), and send money to their account. Money is transferred with cheque.

Gox opened many accounts at the Bitcoin Bank, with many account numbers. They give one account number to one customer. By monitoring these accounts, gox will know which customer has sent money to them, and credit to their gox account

When a customer submits a withdrawal request, gox will sign a cheque for one of its accounts at the bitcoin bank. They take a photo of the cheque, and use it as an evidence of delivery. However, some of the cheques issued by gox have dirt on them. Some customers cleaned the cheque first, then sent to Bitcoin Bank and got paid. The related gox bank account is then emptied.

Unlike a traditional bank, the bitcoin bank will publish the photos of all accepted cheques. Gox compares their photo records with the public records. Since the accepted cheque looks different from the original cheque (dirt is removed), gox can't recognize it and falsely believes that the related bank accounts still have money. Therefore, when another customer requests for withdrawal, they try to sign another cheque with the now emptied bank account. The Bitcoin Bank will reject this double spending cheque, and lead to all those withdrawal issues we have seen.

Even worse, some customers find the gox's bug and try to exploit it. After they cashed the cleaned cheque, they complain to gox saying that they have not received a cheque. Since gox can't find the cheque in the record of Bitcoin Bank, they credit the bitcoin back to the customer's gox account so the customer doubled his bitcoin at the expense of gox's fund (there is NO double-spending at the Bitcoin Bank)

So gox now blames the Bitcoin Bank that it should not accept the altered but yet valid cheque.

Gox also proposes that people should not trace a cheque by comparing photo. Instead, they should trace the unique ID of each cheque, as the ID is non-modifiable. They require the Bitcoin Bank officially endorse this practice before the re-open bitcoin withdraw.

-----------------

So what is the practice of the standard bitcoin client (i.e. bitcoin-qt)? Instead of comparing the photo of cheque, bitcoin-qt actually monitors the account balance. Therefore, whether the cheque is altered is totally irrelevant.

Conclusion: Gox uses a WRONG way to trace transaction, and blame the Bitcoin Bank when everything is fucked up

Great post, thank you so much!

stupid developer does no follow rule. i hungry. maybe soiled :(

I really love your explanation! Thx For it!

still need diapers at five? bit of a retarted kid hm?

Good explanation. Thank you.
Now I wonder, why someone doesn't open a bank called 'The Bitcoin Bank' and operate it in a country like Germany, Finland or the other few that actually regulated bitcoin in a positive way?

Great post. You should post this on reddit and some generous users may tip you a beer

Did it already: http://www.reddit.com/r/Bitcoin/comments/1xiowj/explain_the_gox_transaction_malleability_issue/

nice !

One important point here is that this all was fine, as long as banks accepted both dirty and clean receipts, and dirty receipts issued by gox were actually more likely to reach the bank, because all receipts are sent by post, and it would take significant amount of effort for the customer to intercept those receipts and modify them.

But at some point banks have changed their policy and decided not to accept dirty receipts to prevent exactly this form of of fraud - modification of receipts. There are many ways to produce dirty receipt, but only way to produce clean. So they rejected them and simply throw dirty receipts away. And that's when some customers were able to collect those rejected recepts, clean them and re-submit to make them processed (unnoticed by Gox).

There is still exist a hypothetical problem of malicious banks, but it has very low impact, and will be addressed in the future bitcoin releases.

A cheque with valid signature, no matter dirty or clean, is a valid cheque.

The Bitcoin Bank is a decentralized bank. Anyone with a mining rig could become part of the bank (miner). Some miners do not like dirty cheque (although they are still valid), so customers have to manually clean the cheque before they could cash it. However, this is really up to the policy of each miner.

Valid point, it's not a network rule, although it may become one at some point.
Still it's a good practice accepted nowadays in most of the "banks" and post offices - not to accept dirty receipts, which exposed those MtGox vulnerabilities.

The problem is, there is indefinite ways to alter a cheque without invalidate it. We need to live with transaction malleability and actually it's no big deal

If you are five, you probably will not be able to explain the Bitcoin malleability issue.

The title of this thread is a huge fail.

I don't understand what getting dirt on a check means.  Can you explain it like you're an adult?

It is well known for years that a bitcoin transaction is malleable in many ways. One way is to pad some garbage in the signature. If this is done properly, the transaction is still valid. By malleability, however, you can't change the payer, payee, and the amount paid, so no one could steal others bitcoin in this way. Just like in the real world, spilling some coffee on a cheque won't invalidate it. The rightful payee will still get the money.

In the gox case, they mistakenly padded their transaction with garbage (dirt on a cheque). Although the transaction is still valid, many miners do not like garbage in transaction and refuse to confirm gox's translations. Therefore, some users try to remove the garbage (clean the cheque), and the transaction got confirmed. So the user is happy. However, as the transaction looks different now (without garbage, different hash), gox's stupid customized wallet can't realize that the transaction is already confirmed, and falsely think that the coin is unspent.

is a mathematically true statement.

is mathematically true, and in fact is the same statement.

They've got different hash values though, because hash functions care about binary representations, not mathematical equivalence.

The big question is how long has this been going on and has someone actively exploited it?

This is simply gox's problem, as they shouldn't follow the transaction flow this way in the first place.

for those who own btc with gox, it seems like the issue is going to be the extent to which gox "re-issued" bitcoin deliveries to customers bc the recipients lied and said they had not received it.  if there were many double deliveries, then gox may not have the bitcoin to pay remaining owners all the bitcoin to which they are entitled.  a few issues/questions come to mind:

1. why the other exchanges are not having the same problem
2. would mtgox be able to figure out which customers claimed failed delivery but lied, and were then paid twice.
3. it seems like the gox omnibus acct should net out even at the end of the day.  double deliveries would raise red flags (one would think) when all has been netted

1. Gox uses a customized wallet, which is obviously faulty. Other exchanges either implement it correctly (checking the address balance), or simply use standard bitcoind

2. Yes, if they have kept all the transaction and conversation log.

3. Not sure what you mean. But if gox really double paid some customers, they are the one to absorb the loss (or they will close and run)

Because other exchangers aren't being run by fairies dancing around the magical pot of bitcoins when the owner is drunk dead in the basement ?

So basically MTGoX made a mess and they blame it on something else to try and look like a victim instead of lazy with their own system. 
Glad I did not keep any BTC there.

thanks for the great explanation - are other services outside exchanges also using this customized wallet?

I think you mean "dead drunk", though I like your way better.

That could be easily proven by gox, since there will be "double spends" to the same address in the blockchain.

I always thought it's first the drinking and then the dying =))))  , so > drunk dead.

But right now ,It  won't be long till it he ends dead with lots of holes in it , depending on how much money people lost with his exchanger.

So Gox has to manually compare all previous cheques issued with photos to sort out the mess of their cashbook?

It is always not a good practice to compare the photos (transaction hash). They should check the balance of all their bitcoin accounts

because at that point they would be a bank first, a bitcoin facilitator second, even if their only "currency" is cryptocurrency.

A cryptographic hash is not a "photo", nor alternate representations comparable to a "dirty cheque".

They must be unique and reproducible identifiers which are unambiguous. If this is untrue of transaction IDs, why bother returning them at all? In that sense it is a protocol flaw.

Minor vulnerabilities can be used to leverage more serious exploits, this needs to be fixed.

FYI, at least 3 of us had issues last night with double transactions. We sent a payment and it appeared a second time, one of them is unconfirmed. Not sure if it is related to this.

Malleability is a potential and hypothetical issue nuisance, which only became possible to exploit at MtGox for two reasons: because Gox failed to correctly implement Bitcoin specification properly, and also because it failed to implement proper workarounds for this issue. You correctly pointed out second reason, but the first is more important to point out, in my opinion, because this is why other exchanges are much less likely to be affected, if likely at all.
Gox didn't follow the specification, which required tx signature to be encoded with ASN1/DER encoding. This requirement was specified in April 2011: https://en.bitcoin.it/wiki/Protocol_specification#Signatures
Instead they used some sloppy format which was not DER encoding but was still accepted by SSL library and old reference client. When tighter checks were implemented in bitcoin reference client (the main reason for which was actually to prevent malleability issue), their transactions, which violated bitcoin spec, were rejected. Basically, their transactions looked like what hackers would employ to exploit this issue. That allowed hackers to pick these rejected transactions up, malleate them to "fix" the signature format, and re-submit. Ironically, hackers were helping MtGox to propagate their malformed transactions through the network.
If MtGox submitted correct DER-encoded signature in the first place, hackers would have to figure out how to malleate the transaction without breaking the spec so that the transaction is still accepted by the network, and then outrace the originator of the transaction in propagating the malleated message to the miners, or use their own miner, and be lucky enough to mine the block before somebody else mined the original transaction. A lot of trouble - and all that just to modify tx id, which is already a published known issue, and can be easily detected with manual investigation or proper workarounds in place.

It's not malleability per se which is the reason for MtGox failure, but failure to properly implement bitcoin specification + malleabiiliy + failure to implement workaround + lack of attention to abnormal behaviour in their system + inability to react quickly when the issue became glaringly apparent+not testing their system with reference implementation+....
Or, in other words, lack of competence, unacceptable for the only golden member of Bitcoin Foundation.
There was so may ways this could have been alright - and only utter stupidity and incompetence allowed this to go wrong.

It's wrong to think this is just Gox's problem. It's a problem of Gox's customers (large part of bitcoin community), and this is a problem of Bitcoin public image. When "the oldest and at one point the biggest bitcoin exchange" is run by such moron, that taints the whole community.

Actually they may try to recover some, if they are able to figure out who did that. But I wouldn't bet much on it.
Also they may go into liquidation in a civilized way. That would be the best option for everybody. Bitcoin foundation should use all their influence to pressure Mark to do that.

So does that mean if you saw a transaction floating around the network that has a TX signature with junk padding, you could copy it, remove the padding and resend so the benifactor of that transaction would get paid twice? I take it that would have to be someone from Mt Gox resending the payment in order for it to work?

So just to be clear it is not possible to find an Unconfirmed Transaction on the Network with junk padding, copy it and change things like recipient, amount, remove padding and resend?

Jeez you would think Mt Gox would be looking for ways to speed up transactions, so you would think they would know Miners are rejecting their transactions with junk padding on the signatures and amend any script so as to remove the junk padding and speed up confirmation times. Crazy incompetence.   

No, double-spending is not possible within bitcoin protocol. MtGox didn't see that original transaction went through and re-issued it (automatically or manually - I don't know) with different inputs.

You cannot change recepient or amount without breaking the signature and invalidating it. You can remove padding though - thus creating a good-looking transaction. If the inputs used in this transaction were not yet spent, you can get this transaction confirmed (included into a block). So basically you will do what originator of this transaction was intending to do anyway, the only difference is that you modify transaction in a way which may be unexpected by some outdated bitcoin software like MtGox's. And that is a known issue since 2011, so reference client and probably most custom clients don't rely on that and are not affected.


They became too big and important to care. Just like many banks.
This is my pure speculation, but probably their attempts to speed up transactions actually played role here too. They had partner miners, so they were able to send transactions directly to them, bypassing other nodes. "Network doesn't accept our transactions? Fuck the network, we will go straight to the miners". Until miners finally updated their software.

You assume MtGox is doing daily reconciliation.  Given all the other things they either didn't do, or did wrong why would you assume that.  It is entirely possible they had absolutely no check's and balances in place and the same attacker "double withdrew" dozens or maybe even hundreds of times.  The issue with MtGox generated bad withdraws has been going on for a month.

Yes and that is what it is assumed that attackers did.  Of course of that original pair only one could be confirmed so yes to get double paid you would also need to trick MtGox into cutting you another payment.   Of course MtGox client is horribly defective and there were thousands and thousands of legitimate reasons why they had to cut new payments so the attackers requests would hide in a sea of requests created by their incompetence.


Correct, you can not do that.  The inputs, outputs, value of tx, fee paid, recipients, and value to each recipient are immutable.


The OP was a simplified explanation of what MtGox got wrong.  MtGox had a huge laundry list of fails in their client wallet. At minimum (just by observing their "missing" transactions) they
a) tried to spend immature newly mined coins which caused the tx to be dropped (until they matured if Gox was still broadcasting it) by some or all of their peers.
b) tried to make payments which violated the spam rules and thus wouldn't be relayed by some or all of their peers.
c) paid insufficient fees on low priority tx which caused them to not be relayed by some or all of their peers.
d) used non canonical signatures which caused them to not be relayed by some or all of their peers.
e) double spent their own coins which caused the tx to be dropped (correctly) by some or all of their peers.

It is very likely their wallet was deficient in other ways these are just symptoms that I and others have observed by looking at the transactions MtGox created.  As an example, obviously if MtGox is creating a tx which is spending newly mined coins less than 120 blocks from when they were mined we know their wallet is not performing that required check.  How many other deficiencies are in the "Gox Special v0" custom client.  Nobody outside MtGox knows for sure but you should not take this list to be exhaustive.

Even daily reconciliation isn't really good enough. It's the 21st century - why shouldn't these kinds of services be expected to reconcile with each new block?

Now this is going on.

http://www.coindesk.com/massive-concerted-attack-launched-bitcoin-exchanges/

I have looked up the change logs of the Bitcoin client of the previous year, and I have yet to find any sign that the client switched to more stringent checks. There are some code changes on github that you referred to earlier, but even if those made it into the default client they wouldn't go as far as to fix the problem, because these code changes still leave open lots of room for malleability. But with access to good mining equipment it would be somewhat easy to race the original transaction being transmitted on the bitcoin network with your manipulated version.

I would still like to see an actual instance of the original Mt. Gox transaction and the transaction that got in the blockchain instead and who mined that.

 :D Good one

If you compare the signature on the bitcoin transaction (on an input of it actually, like a transaction is composed of multiple checks.) with a traditional  ::)  wet handwritten signature, then the malleability is like the signature being done in another color of ink. A grafologist would still conclude that the purported sender could have made that signature, if he didn't know that the real sender always used a very specific color and composition of ink.

Good explanation. Thank you.