Mt.Gox technical autopsy

[jamal26]

Hello, this is a topic I've been wondering about.  Please excuse me if my knowledge of bitcoin transaction details are a not up to speed at this point.

The thing that keeps sticking in my mind is how Gox went for 2 years without people noticing and screaming about it.  If Gox balances didn't agree with legit transactions balances, how do that go unnoticed?  First of all there are 2 different situations that are relevant to this issue.

1. Individual depositors transactions appear on the block chain.
2. Gox aggregated individual depositors transactions so they really only have a commitment from Gox that their bitcoins are real.

From what I've heard #2 is the way it works, and that other "exchanges" work the same way.  I find this very much against the bitcoin philosophy of a redundant trust system.  You're trusting Gox, not the redundant block chain, and maybe that's the fundamental problem with it.

But let's look at the idea that a long term attack could have gone unnoticed under these 2 different conditions.

1. Individual depositors would have recognized right away that their bitcoins were missing from their wallets if they saw they didn't appear correctly on the block chain.  Very hard to believe that they wouldn't.

2. Gox reconciliations should have picked up missing bitcoins from the block chain.  Did they not do this simple reconciliation?  This is also very hard to believe, but since it's based on inept management from a dysfunctional organization, it's easier to believe than #1.

My view on this is that the ecosystem is immature at this point and is suffering from it being primarily a fiat investment device.  People want non-sustainable things from it like excessive returns and fiat anonymity which makes people want to use exchanges and become vulnerable to trust problems with the exchange.

Another issue I have is the size of the block chain and the fact that it's not distributed, as fiat money is.  Think about if there were a block chain for dollars or euros.  It would be absolutely unmanageable and a huge breach of privacy.  These are issues that are fundamental to the viability of the bitcoin model if you ask me.

[bitserve]

Very interesting explanations about the malleability "issue" and its limited practical impact on mtgox assets.

The "funny" thing here is that MK has deluded himself into thinking that the "It's gone" will suffice as an explanation without offering any proofs in an environment so full of them.

Sooner or later he will have to provide internal accounting records and then it will be very easy to finally know what (and when) really happenned.

One of most important accounting records he will have to provide is the list of all the cold wallet addresses that they were using at each moment in time. Being the most valuable assets of the company (apart from bank accounts) all of them should be perfectly identified in company reports showing periodical balance.

Then we will have the full picture of what was happenning and since when.... the same full picture that Karpeles had in front of him all the time... the same that would automatically show a discrepance as soon as it started between mtgox internal BTC balances and real ("blockchained") addresses under their control.

After that basic accounting is done then it will be easy to check which explanation fits better into this case... and since when was MTgox knowingly operating in insolvency without reporting it, which I supposse its a criminal act in Japan as much as everywhere else.

Maybe then the malleability "issue" and all the bunch of excuses will be insignificant and no matter anymore even if it had any additional impact on the already criminal operation of MtGOX.

... Or maybe the next twist of this plot will be to execute another "It's gone!" this time to the accounting records? That would be even more funny.

TL;DR: The answer is in the accounting records which he will have to provide to prove the insolvency and how it developed over time. It won't be easy to fake them as they must match and be coherent with the public blockchain.

[wheatstone]

Seriously, this nonsense must stop. Understanding of the transaction malleability issue is completely wrong throughout this thread.

Transaction malleability does not result in two transactions in the blockchain.

This cannot be stressed enough, apparently.

nanonano pretty much had it right with his list.

The claimed explanation for MtGox was:
* MtGox sends a transaction
* attacker modifies the txid
* modified transaction goes in the blockchain, and bitcoin ownership changes
* attacker complains MtGox "I didn't receive my coins"
* MtGox searches for original txid in blockchain, doesn't find it  (<-- this is the bug)
* MtGox thinks they haven't paid, apologizes and sends a second, totally separate transaction of the same value

Nothing here is incorrect, as such, but the conclusions drawn are incorrect.

When mtgox, or anyone else, submits a transaction it does not immediately enter the blockchain. This only happens once a block containing the transaction is mined, resulting in what is referred to as a confirmation.

Transaction malleability relies on the fact that while the inputs and outputs (source and destination) are fixed by cryptographic signatures, the transaction id (txid) is not. In other words, anyone can submit an identical transaction (same source and destination, same amounts etc), just with a different txid.

The way mining works, if a miner already has the "original" transaction in their memory, they will discard the modified transaction. However, the miners' perception of which is the original depends only on which one they see first.

In other words, exploiting transaction malleability relies on submitting modified transactions in the very small window before the original transaction reaches all miners... and then hoping that the modified variant is the one that gets mined first.

Only the transaction that gets mined first gets in the blockchain (with the small caveat that sometimes blocks are "orphaned" if two miners solve a block almost at the same time).

To see which transactions had multiple versions would thus require "live" access to the miners' memory pools. If no logging of these discarded transactions (whichever reached a miner last) happened, there's no way to see it after the fact.

[bitserve]

To see which transactions had multiple versions would thus require "live" access to the miners' memory pools. If no logging of these discarded transactions (whichever reached a miner last) happened, there's no way to see it after the fact.

Thanks for the detailed explanation.

Do you have any idea of how high is the percentage of discarded transations under "normal" conditions? Would it be feasible for some miners to be logging those "incidents"?

Anyway, one thing is that we can't detect those "malleability attacks" afterwards and other very different that mtgox can't reconcile its internal accounting with the blockchain and detect each and all discrepancies.

Please let me know if theres some reason that a process like this would not work:

1- Balances for all accounts with no withdrawals are ok (at least from a "malleability issue" point of view)
2- Balances for all accounts with no failed withdrawal + reissue are also OK
3- Accounts with failed withdrawal transactions lets check like this:
    - For each failed transaction lets search if the transaction went through. Obviously not using the txid, but ammount, input, output. If it did, then flag the account as "abuser" and take note of the "leaked" ammount.

Now you should know exactly how much the malleability "issue" is to blame for the "missing" BTC's and also who were the users behind it, and when/where the leaked money did go to.

Is there any technical reason for this to not be a simple process to run against mtgox records or maybe the only reason they havent made this figure public is because they already know that total sum would just be negligible and that was not the real problem?

P.S.: I am aware that there have been some comments saying that one of the faults of mtgox was that maybe they were doing the reissue using different inputs...

Well, that's what makes difficult/impossible to locate the "dupes" using only the blockchain, but mtgox accounting records should have the missing information, i.e. inputs/output/ammount for each attempted transaction, original or reissued, so that it would be a matter of going through that records rechecking everything against the blockchain.

[jamal26]

In other words, exploiting transaction malleability relies on submitting modified transactions in the very small window before the original transaction reaches all miners... and then hoping that the modified variant is the one that gets mined first.

Are you saying that someone can observe a transaction being submitted to one node and then quickly submit a bogus transaction to other redundant nodes?  If not, how is the original transaction information obtained?  If so, then what happens when the different nodes disagree?

I have an accounting background, so I'm still very confused about why there appears to have been no reconciliations done.

[wheatstone]

Are you saying that someone can observe a transaction being submitted to one node and then quickly submit a bogus transaction to other redundant nodes?

Correct. To do so on any consistent basis, one would need good (low latency) access to multiple nodes (preferably the one mtgox submits to plus another with a lot of hashing power).

If so, then what happens when the different nodes disagree?

Hashing power rules all...

Nodes never "agree" in the sense that their set of transactions is identical. Mining pools are completely free to include whatever transactions they wish and the same for relaying transactions. Certain transactions are not supposed to be relayed, including modified transactions. Assuming everyone behaves (and they pretty much do), exploiting transaction malleability becomes a race to reach the greatest amount of hashing power.

Even if the attacker only reaches 10% of all hashing power (and he could likely reach much more with good connections to the major pools), he would still get the modified transaction into the blockchain about 10% of the time.

There is also another way "nodes" can disagree, but I don't think that is what you were talking about. I alluded to it a few times earlier in my previous reply (and again two sentences ago). The issue here is that if two mining pools find a valid block at very close to the same time, the nodes will relay the version of the blockchain they received first (just like with the modified transactions earlier). The result is that for a short while, the blockchain has actually forked. Each individual mining pool works on the one they received first (or are supposed to, anyway, see "51% attack"). At some point, the next block in one of the chains will be found and all the miners will switch to the longest chain.

[ilpirata79]

You couldn't do that, because only one of the transactions would actually be in the block chain.

That would mean that valid transactions do not appear on blockchain - I consider this rubbish.

Both transactions would be in the blockchain - the initial withdrawal and then the re-withdrawl request several weeks later which mtgox would have allowed.

I agree with this.

One needs to pair those two transactions (find them in the blockchain). There would be a few non-complex criteria for finding such malleable-looking transactions:
- amount of both must match
- date range should not be broad, e.g. the second transaction cannot be older than 1 day / week / month than the first one
- both transactions must be spent from an exchange address (not sure how to find such addresses but my guess is that such addresses generated a huge number of transactions for a short time)
- others

------------------------------

It is a perfect research project for applicants who want grants from TBF.

There is no need for equal transactions (apart from the txid) to be present in the blockchain:

1) I ask for a withdraw of some amount of bitcoins
2) I alter the txid
4) I wait for the altered transaction to get confirmed
3) I ask Mtgox to credit those bitcoins back to my account because the transaction was not successful
4) go to 1

Best regards,
ilpirata79

[wheatstone]

I have an accounting background, so I'm still very confused about why there appears to have been no reconciliations done.

There's no arguing with this sentiment. It is truly baffling if the ledgers were not reconciled on a regular basis.

[wheatstone]

Do you have any idea of how high is the percentage of discarded transations under "normal" conditions?

I'm not sure what "normal" conditions are. If you mean no-one-trying-to-abuse-anything situation, then 0%. Not that every single modified txid would be an attempt to abuse, but there doesn't seem to be any good reason for doing it and certainly none of the major clients support it.

Would it be feasible for some miners to be logging those "incidents"?

Yes. In fact, you wouldn't even need to be a miner. Anyone relaying transactions (e.g. running the full bitcoin client) could, in theory, log these "incidents".

Someone actually did this to an extent after mtgox made their initial announcement. From the results he posted to this forum, probably only "counts" were logged (not each individual transaction). Their count was generally quite low, with a few spikes here and there (and a 24 hour break on Chinese New Year, if I recall correctly).

[bitserve]

I have an accounting background, so I'm still very confused about why there appears to have been no reconciliations done.

There's no arguing with this sentiment. It is truly baffling if the ledgers were not reconciled on a regular basis.

Do you really think that it is really possible that all the BTC's were leaked out of gox by taking advantage of the malleability issue?

And that, even if no periodic balance reconciliation were done (which might be criminal negligence in itself), they woulnt notice something very wrong was happenning each time they had to load another of their deep cold wallet address because the previous ones had all been emptied without any logical reason?

I find easier to believe that all the time they knew they were running a fractional reserve, probably risking the remaining BTCs in an attempt to recover solvency and losing them in unfortunate trades, until they couldnt even fulfill its pending withdrawals... and then blaming some old known issue which may or may not have had an additional but negligible impact in its balance.

[bitserve]

Do you have any idea of how high is the percentage of discarded transations under "normal" conditions?

I'm not sure what "normal" conditions are. If you mean no-one-trying-to-abuse-anything situation, then 0%. Not that every single modified txid would be an attempt to abuse, but there doesn't seem to be any good reason for doing it and certainly none of the major clients support it.

Then there is a reason why such collisions should be considered an "incident" and logged separately (for forensic purposes), especially if the volume is not that big as to make it impractical. Pity it wasn't being already done on a regular basis, at least on some "strategical" points of the network.

Would it be feasible for some miners to be logging those "incidents"?

Yes. In fact, you wouldn't even need to be a miner. Anyone relaying transactions (e.g. running the full bitcoin client) could, in theory, log these "incidents".

Someone actually did this to an extent after mtgox made their initial announcement. From the results he posted to this forum, probably only "counts" were logged (not each individual transaction). Their count was generally quite low, with a few spikes here and there (and a 24 hour break on Chinese New Year, if I recall correctly).

Again a pity it wasn't being logged in full even after the mtgox announcement. But anyways, that information could imply that the "attack" was not so generalised as to justify the ammount of damage mtgox wants everyone to believe. Maybe it even was a smoke screen to give some credibility to the announcement.

Also, being this type of attack some sort of race condition (combined with mtgox negligencies), the volume should have probably been huge in comparison to the sucessful "exploits", and this doesnt seem to have been the case.

[wheatstone]

Do you really think that it is really possible that all the BTC's were leaked out of gox by taking advantage of the malleability issue?

And that, even if no periodic balance reconciliation were done (which might be criminal negligence in itself), they woulnt notice something very wrong was happenning each time they had to load another of their deep cold wallet address because the previous ones had all been emptied without any logical reason?

There seems to be no shortage of speculation on this topic around here without me contributing.

Let me simply state that it would have been quite interesting to have access to a full log from one or more relaying/mining nodes, were one such to exist. Preferably from day 1.

[wheatstone]

Please let me know if theres some reason that a process like this would not work:

1- Balances for all accounts with no withdrawals are ok (at least from a "malleability issue" point of view)
2- Balances for all accounts with no failed withdrawal + reissue are also OK
3- Accounts with failed withdrawal transactions lets check like this:
    - For each failed transaction lets search if the transaction went through. Obviously not using the txid, but ammount, input, output. If it did, then flag the account as "abuser" and take note of the "leaked" ammount.

...

Well, that's what makes difficult/impossible to locate the "dupes" using only the blockchain, but mtgox accounting records should have the missing information, i.e. inputs/output/ammount for each attempted transaction, original or reissued, so that it would be a matter of going through that records rechecking everything against the blockchain.

Any reasonable audit would require access to inside information (i.e. from mtgox).

Every single point you list above requires record keeping on the part of mtgox. Although one might reasonably presume that such records existed, I do not know what records mtgox did in fact keep.

The logic in your points seems sound, however.

As for the inputs used in a given payment, it seems reasonable that the exchange would simply queue up the outgoing transaction using the normal process for doing so (which would result in an entirely new payment, with a (possibly) new set of inputs).

With regards to what tentative analysis could be done on the blockchain, one would need a reasonably exhaustive list of addresses / outputs that belonged to mtgox in order to even get started. Such a list does not exist, publicly at least. Since mtgox used a custom wallet to create these transactions, it is possible that they could in some way be identified if they had some characteristic that was different from other transactions (but I doubt it).

[ilpirata79]

Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

This helps locating the stolen bitcoins. Off course it does not help to locate the thieves because they for sure used false documents to register and an anonymous connection.

Best regards,
ilpirata79

[bitserve]

Thanks for the detailed explanations. Yes, I meant using mtgox internal accounting records, which obviously mtgox should have and so (being this forensic reconciliation possible) could/should do this audit themselves and offer it as proof in the process as defense of the accusation of internal theft (which is also already in the air).

Each attempted transaction should have been logged, not only because of its forensic value, but because one couldnt know it would become a "failed" one, you need to log it to be able to track the outcome. In no way they could justify that failed transactions were simple wiped from the logs afterwards.

It's the same as they should have a detailed log of the order book at any point in time.

But now that I am thinking about the malleability issue, and after reading the explanations, some thing has come to my mind:

Withdrawal transactions are put in queue and then a transaction with many inputs and many outputs (all the withdrawal destinations) is created... so... for each sucessful malleability attack, not only the "attacker" withdrawal would be reissued, but all the ones in the same "failed" transaction would. So many people besides the attacker would have received duplicate bitcoin transactions for each sucessful attack. (Unless I am wrong in my understanding of how that process works)

At least if we come to believe the explanation that it was an AUTOMATED reissue process and not a manual one after opening a ticket asking for the reissue.

The more I think about it, the more it all seems totally BS to me.

[nanonano]

Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

So have you been logging those transactions that never ended up in the block chain? Because I sure haven't.

[wheatstone]

Off course it does not help to locate the thieves because they for sure used false documents to register and an anonymous connection.

No false documents necessary. Karpales has stated that the double payouts were from unverified accounts. Until recently, mtgox did not require verified accounts for btc withdrawal.

[ilpirata79]

Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

So have you been logging those transactions that never ended up in the block chain? Because I sure haven't.

I haven't, but Mtgox should have. As I said we need their authentic logs (authentic in the sense that they are not faked)...

Best regards,
ilpirata79

[nanonano]

Get to know the abusers is not difficult. Just check the transactions that got confirmed with a different txid. For this there is need of the authentic Mtgox logs.

So have you been logging those transactions that never ended up in the block chain? Because I sure haven't.

I haven't, but Mtgox should have. As I said we need their authentic logs (authentic in the sense that they are not faked)...

Ok, then we agree. The only reason I asked is that I thought you said the opposite in the post I quoted: that we would not need MtGox logs for this.

[bigasic]

I know for a fact that karpales (sp) has been investigated by us authorities for months. One federal agent told me that they had a close eye on him and that he was a crook. I was told this late last summer. So, its pretty obvious that this whole mtgox mess was caused by them (mtgox or mk) by either just by plain old theft or trying to manipulate bitcoin someway..

Either way, Im glad mtgox is out of the equation and let the healing begin... Hopefully we will be up to 1k by summer...