MtGox 9/11: a wild speculation about what happened

[molecular]

First: thanks to phantomcircuit and jarpiain for helping me out with #mtgox irc chatlogs!


What happened?

On September 11th,  2011, some weird trades showed up on MtGox' ticker. They seemingly executed way out of spread, as can still be seen here: http://bitcoincharts.com/charts/mtgoxUSD#rg5zig5-minzvztgSzm1g10zm2g25


MtGox' explanation

MtGox' explanation (https://support.mtgox.com/entries/20433652-resolved-outage-11804-unexecuted-trades) talks about possibly compromised accounts in relation to the CosbyCoin-hack on this forum.

As a result of this event, some of the Bitcoin Forum users` accounts may have been compromised.  Subsequently, some of the information have been used to conduct unauthorized orders, resulting in unusually high trade activities. 

The Press Release, if I may call it that, then goes on to talk about these "unusual activities" and says that staff has nullified these trades. It then educates us users about password security and states

Please be advised that trades can now be conducted in full confidence.

This explanation is not satisfactory for me.

So I came up with a highly speculative explanation myself.


What I speculate really happened

I'm largely basing my speculation on things that were said in #mtgox irc channel and quoting from that, not sure about the timezones in the quotes, since the logs are from different sources.

9/11 - 18:32 <MagicalTux> molecular, I blocked ~2000 accounts created most likely for the purpose of killing bitcoin on 9/11

Now let me introduce you to a bug that was found Aug 14th 2011 (short description: orders (can) get temporarily disabled when being partly filled):

01:15 < molecular> weird, the following order did not get filled: 9bd49edb-2073-44e3-8f68-34971a1a4d45  bid    4.835     9.73    - 1 open, although the price just dropped to 9.72 by this trade: 00:14:00    6.93168 for   9.72    ask
01:15 < molecular> that order has existed for a whle
01:17 <@neofutur> an older order could have been filled before
01:17 < molecular> at what price?
01:18 < molecular> price dropped from 9.8 to 9.72 and my order at 9.73 did not get filled
01:19 < molecular> part of it got filled before:  00:10:12    5.165   for   9.73    ask
 
01:24 < deego> The only explanation I could think of is a queuing issue: If your older, though pre-existing to it, was in fact newer to the executing engine - that is, the engine executes them in the order they arrive to it. And, the engine saw a 9.72 first, and your 9.73 arrived later to the engine.
01:25 < molecular> but 9.73 is higher than 9.72, it surely should fill higher bids first, right?
01:25 < molecular> deego, that bid existed for at least 10 minutes
01:26 < deego> I see.

01:27 < molecular> deego, also it was partly filled before:  "00:10:12    5.165   for   9.73    ask"
01:27 < deego> ^ Ah.
01:27 < molecular> maybe... ah!
01:27 < molecular> I think I have an explanation:
01:27 < molecular> maybe when an order is partly filled, a new one is created in "pending" status
01:27 < deego> heh, just what I was thinking
01:28 < molecular> then the other bid at 9.72 got filled while my order was still pending
01:28 < deego> and, it's requeued..
01:28 < molecular> so an order goes to pending when part of it is filled...? that shouldn't be the case and would be a bug, right?
01:28 < deego> shouldn't it ideally retain its position in the que, somehow?
01:29 < molecular> the position in the queue is secondary. it should, however, stay in status "open" alle the time (while I don't know exactly what that means)
01:29 < deego> IIUC, Pending should be equivalent to: "waiting to get queued."
01:30 < molecular> deego, I don't know any details of the trade matching engine... but I think we might've figured out what's happening roughly
01:30 < deego> agreed.
01:36 < deego> I think, in principle, the requeuing should be considered a bug - because then I can, in principle,  negate others' orders - I can move anyone's orders "into the future" by filling 0.001% of them; and I can get my own fill at the currently lower price.

So far for the bug and possible analysis of how it works.

Now deego and me come up with some evil ways to exploit this bug:

01:38 < molecular> if you put your order at the same price, you jump the queue
01:39 < molecular> even worse: you can even buy at a lower price if you time it just right. should be very hard to do, but theoretically possible, because it takes some time to requeue the "disabled" order
01:39 < deego> or ever lower price: If I negate every order at 9.73 (like yours), so that the first thing engine sees is 9.72..
01:39 < deego> exactly.
01:39 < molecular> yeah
01:40 < molecular> wow, didn't think of doing it to multiple orders successively

And this is exactly what I think happened: this bug got exploitet by use of a botnet (or similar) creating 2000 accounts on mtgox and "disabling" orders successively in order to get an order filled way out of spread.

MtGox then hastily nullified these orders and tried to calm people down talking about compromised accounts and CosbyCoin, maybe in order to avoid having to shut down trading to fix the bug.


Why am I publishing these wild speculations?

While this speculation might be accurate to some extent, I don't think it is.

By publishing this, however, I hope to put some more pressure on MtGox to explain what happened on 9/11 in more detail, because I think this should be made transparent.


Why does MtGox not transparently publish more detailed information?

There might be legitimate reasons not to do this at this point. In case there are, I apologize to MtGox for trying to put pressure on them to do so.

Following excerpt might shed some light on this (this was on September 12th):

[09:05:50] <molecular> What the hell? Just read: https://support.mtgox.com/home. no mention of a bug or anything. How can a user with a compromised account make deals much higher/lower than the market? No explanation for that is given, why not?
[09:06:59] <MagicalTux> molecular: it's a known bug, we are still tracking it
[09:07:19] <molecular> ok, but why try to "cover it up" talking about compromized accounts?
[09:07:36] <MagicalTux> because right now to cause this bug to happen, you need to trade unholy amounts of coins
[09:07:58] <phantomcircuit> wat
[09:08:13] <molecular> Hmm, ok. Still: why not explain that in the news-release?
[09:08:20] <MagicalTux> more exactly, you need to have your large trades be disabled in the system
[09:08:57] <molecular> what does that mean? "have large trades disabled"?
[09:08:58] <MagicalTux> molecular: because most people wouldn't understand what this means. Also we cannot put too much info in the public until we finish our declarations to the MET

So maybe the "legitimate reason" is that there are some ongoing investigations and MtGox is not allowed to give us info.

Maybe it's just that he doesn't want to, using "people wouldn't understand" as an excuse.

What do you guys think?

[molecular]

Ok, it seems I didn't think this through all the way, as ezl just pointed out:

<ezl> molecular: if you want to pay 5.50,even i you did fire a bunch of tiny offers to clear out the bids from 5.60 down to 5.50 (where your bid is resting)
<ezl> somebody still has to sell that at 5.50 in order for you to be able to execute outside of the "real" bbo
<ezl> so if someone happens to hit "sell market" in that instant, it'll push through, but oddsare slim

Ezl allowed me to quote what he pmed me right after:

<ezl> basically, you can make bids disappear, but you wo't be able to buy unless there's a seller.
<ezl> and they have to sell in the instant that orders are being "requeued"
<molecular> maybe my speculation is totally bogus after all
<ezl> no, its possible that its still getting requeued, which would be really bad exchange behavior (i don't think this is happening though)
<ezl> but if someone is willing to just burn money to change pricing behavior for some other purpose (just doesn't like mtgox, wants to see bitcoin discredited, etc)
<ezl> then it'd be possible, if this requeueing behavior existed
<ezl> 2 ways to test:
<ezl> 1. get 2 accounts to post a same price bid for 2 contact
<ezl> then create 10 orders of 0.01 contracts that hit the bid.
<ezl> if each bid gets filled for 5 contracts, then your theory is validated
<ezl> or,
<ezl> 2. just create a ton of offers at some arbitrarily low price
<ezl> since you know the size on very rung of the ladder (via mtgox api)
<ezl> if the lowest price you transact during the test != the amount you should have to blow through to get there:
<ezl> mtgox is recreating the orders and they "disappear" in the interim
<ezl> however, i suspect they're safe on this.

So it seems my theory doesn't work out, at least not in the way I imagined.

I still would like to see some more light shed on this, which was the main goal of the post.

[mizerydearia]

In case there are, I apologize to MtGox for trying to put pressure on them to do so.

Huh?  Apologize for pointing out flaws for poorly written proprietary, security through obscurity implementation of bitcoin market with history of lying, censoring/deleting forum posts, market data and other history-rewriting activities?  I, for one, would rather have high expectations of any and all operations and not offer any leniency or acceptancy of poor performers simply due to .... .. actually, I have no idea what reason there would be...  perhaps opportunity for expressing innocence?  incompetence?  other?

Alternatively, instead of thoroughly analyzing/stress-testing mtgox market so as to establish any amount of legitimacy (or lack thereof), why not establish effort towards evaluating other later-established (later than early/mid-last year) exchange markets and to give them opportunity to establish themselves as reputable, reliable, trustworthy, secure, etc. exchange market worthy of establishing as bestest exchange markets to make use of...  What is it about mtgox that is superior to all of the other existing exchanges in that it has more volume?

o/ obnoxious post is obnoxious

[phantomcircuit]

The original explanation makes perfect sense actually.

This appears to be a race condition within the MtGox trading platform.

If trades are indeed being downgraded from 'open' to 'pending' and being placed at the end of the queue when being partially filled an attacker placing a large number of very small bids could and appears to have succeeded in forcing other bids to purchase significantly overpriced asks.

By pricing the small bids directly upto the attackers ask order there is a near guarantee that the attackers over priced ask will be fulfilled.

Evidence of bids being put at the end of a queue for reprocessing can be found in the websocket depth adjustments.

'{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"5.525","price_int":"552500","type":1,"type_str":"ask","volume":"-2","volume_int":"-200000000"},"op":"private","origin":"broadcast","private":"depth"}'
'{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"5.525","price_int":"552500","type":1,"type_str":"ask","volume":"2","volume_int":"200000000"},"op":"private","origin":"broadcast","private":"depth"}'

But what is the significance of this attack?

It's huge, people placing pseudo market bid orders, ie limit orders with substantially over market rates, will end up purchasing the attackers over priced ask instead of all the lower but pending orders.

Note that this attack works just the same substituing bids for asks.

[molecular]

But what is the significance of this attack?

It's huge, people placing pseudo market bid orders, ie limit orders with substantially over market rates, will end up purchasing the attackers over priced ask instead of all the lower but pending orders.

Not only people placing pseudo market bid orders might get screwed. Also peoples bots might "detect a rally", and just buy from the next-best ask (the one placed by the attacker) in order to be in early on the rally that seems to have a lot of velocity.

[indio007]

Could this e the reason we see all these hard momentum reversals. Meaning whenever I see the price take a large jump up it shoots right back down instantly. I thought it was people trading on stale information by idling on the trade page which isn't realtime i.e. has the price at the time the page was loaded. Mtgoxlive looks like a seismograph from the Island of Sumatra.  It doesn't make sense that the prie shoots up to say $10 and a second later some sells into a .50 fall ALL THE WAY DOWN. This happens all day everyday.

Compare MTGOX graphs to mainstream market graphs. They look nothing like this.

http://www.forex-markets.com/charts.htm

[molecular]

Could this e the reason we see all these hard momentum reversals. Meaning whenever I see the price take a large jump up it shoots right back down instantly. I thought it was people trading on stale information by idling on the trade page which isn't realtime i.e. has the price at the time the page was loaded. Mtgoxlive looks like a seismograph from the Island of Sumatra.  It doesn't make sense that the prie shoots up to say $10 and a second later some sells into a .50 fall ALL THE WAY DOWN. This happens all day everyday.

Compare MTGOX graphs to mainstream market graphs. They look nothing like this.

http://www.forex-markets.com/charts.htm

I'm not sure, but I doubt this is still going on. The effect you're describing and the fact that bitcoin market behaves differently from other markets could have other reasons.

[phantomcircuit]

Was there an official response to this?

[molecular]

Was there an official response to this?

This was 9/15 7:22 GMT:

<molecular> MagicalTux, is this the last word on the "out-of spread trades": https://support.mtgox.com/entries/20433652-resolved-outage-11804-unexecuted-trades ?
<molecular> MagicalTux, or will we get more detail at some point?
<MagicalTux> molecular: there will be more details in the future, once we can release more
<molecular> MagicalTux, is the reason you cannot release more some ongoing investigation or what?

It seems we're left with speculating for now.