Gox Lag at 11 minutes WTF

[samson]

Just now at https://mtgox.com/api/1/generic/order/lag

It says this : {"result":"success","return":{"lag":685162131,"lag_secs":685.162131,"lag_text":"11 minutes"}}

I found a good way of cancelling an order you don't want to go through is to simply transfer all your BTC out of the account.

This immediately cancels an order with no lag time whatsoever.

[el_rlee]

{"result":"success","return":{"lag":874191746,"lag_secs":874.191746,"lag_text":"14 minutes"}}

[el_rlee]

{"result":"success","return":{"lag":987460168,"lag_secs":987.460168,"lag_text":"16 minutes"}}

[Enigma81]

{"result":"success","return":{"lag":1052062808,"lag_secs":1052.062808,"lag_text":"17 minutes"}}

[Enigma81]

{"result":"success","return":{"lag":1081055125,"lag_secs":1081.055125,"lag_text":"18 minutes"}}

[samson]

{"result":"success","return":{"lag":1081055125,"lag_secs":1081.055125,"lag_text":"18 minutes"}}

This issue on it's own has the potential to cause a crash.

[SkRRJyTC]

{"result":"success","return":{"lag":1081055125,"lag_secs":1081.055125,"lag_text":"18 minutes"}}

This issue on it's own has the potential to cause a crash.

https://bitcointalk.org/index.php?topic=138679.0

[yuvadm]

Mt.Gox situation over the past few days is a clusterfuck.

I would offer my experience in building and maintaining large-scale real-time web operations, except I wouldn't know who to talk to at Gox. If anyone from Gox is listening, please get in touch. I want to, and can, help.

[Donatus]

I had an interesting conversation today among the other senior software engineers where I work.  We talked about exactly this topic.

I'm interested in your opinion too.  What exactly would you do to avoid what we saw today.   There is no doubt that the infrastructure at gox is failing-by-design, so I'd love to hear your ideas too.

Lets start with the event broker.  The lag is an obvious clue that its doing more work that it really has to.   What aspects of the design of this sort of system would you say are the easiest to change, the "lowest-hanging-fruit" which is probably at the core of the problem.

Even now that things are settling down, we see lag go from zero (idle) to over 20 seconds every few minutes.  This is a big clue to the issues, I think.

--D

[yuvadm]

Anything I can say at this point would be sheer speculation... Hmm, maybe we're still on topic To the best of my judgment Mt. Gox is facing two main issues, one with streaming API availability, and one within the core brokerage service.

The socket.io streaming API issues are not new, they have existed for a long time, but clearly have gotten worse. I would speculate that gox have a flaky stack handling socket.io. WebSockets being unable to connect, or connecting with no data, means the endpoint is not negotiating the connection properly. This isn't a complicated task, but rather trivial. I suspect that gox has a misconfiguration on that stack, since there is no need for lots of hardware for these types of services (proper evented, IO-loop based, servers can easily thousands of clients per instance), just for proper configuration of the real-time server.

Second, the core trading engine. Lots of things can go wrong on this level. Clients should be assumed to be able to send an arbitrarily large amount of orders (and if not, state the limit clearly to users). Orders are probably kept in-memory somewhere, and traversed by the trading engine to match orders, and execute them. If some malicious user is DoSing the trading engine, gox should have a setup in place to mitigate such malicious actions. Otherwise, they simply have never dealt with the scale of trading going on during the past few days and clearly have a bottleneck, either with the trading engine itself (i.e. matching algorithm) or with the supporting infrastructure.

If gox shares more details about their setup, I can try to speculate further about the issues.

[davout]

If gox shares more details about their setup, I can try to speculate further about the issues.

You can speculate as much as you want, even no additional information is disclosed

[crazy_rabbit]

Isn't it possible that someone is flooding the system with orders/cancels? Iirc this is done in Wall St regularly because it offers some sort of advantage to the guy doint it.

If that is the problem, an easy solution would be to put a limit in the amount of orders per minute per account, which I don't think is implemented now.

Interesting. Although it's hard to understand who this would benefit. Although I'm sure someone could construct some sort of scenario.

[Cluster2k]

Isn't it possible that someone is flooding the system with orders/cancels? Iirc this is done in Wall St regularly because it offers some sort of advantage to the guy doint it.

The advantage it gives is to drown competing bots in useless information.  Place and cancel hundreds of orders per second while at the same time strategically placing legitimate orders.  In share market high frequency trading, the apologists for Goldman Sachs, etc say this increases liquidity and is good for everyone.  They claim it means a buyer is always available for a seller.  They miss the entire point that the HFT is designed to solely benefit the operator, and it's a huge cluster fuck when dozens of bots compete with the proven possbility to flash crash markets.

Maybe we're beginning to see this at MtGox.

[ehoffman]

Correct me if I'm wrong, but both Mt. Gox and all the API clients in the world are using the same socket.io server.  So the lag experienced yesterday, due to a probably very large amount of API calls, was probably mainly caused by the non-MtGox API calls overloading the server.

A very first step would be to ensure that when the trading server load reach a certain point, the non-MtGox requests get discarded (not delayed, but discarded) past a certain request queue size to prevent server overload and keep priority to MtGox site trading to take place.  And when network congestion occurs, do similar and give higher QOS (Quality Of Service, a priority scheme in networking protocols) ensuring that requests from MtGox site get absolute priority.

With the above, yes, probably the third-party sites (and user applications) relying on the API will get longer lags, and possibly "can't get result, server overloaded" type of errors, but this ensure that a trade entered at MtGox get executed in a timely fashion, and also ensure that the information displayed on MtGox site is accurate (including their mtgoxlive applet).

After all, having a lag of 5 minutes or not having results is pretty much the same.  At least, this ensure, for 3rd party apps, that when a request pass through (does not get discarded), it do receive a response in a timely fashion.

This is a more advanced networking topic, but pretty much basic concept, and a must to ensure stability.  Think about it, when you call somewhere with your phone, it most certainly now somewhere along the way get transformed to IP traffic (same as with regular Internet data).  And that voice data pass on the same wires/fibers as the regular PC to PC data.  You would not want to hear your interlocutor at the other end burping and lagging as soon as you have network congestion.  So that is treated the same way.  Absolute real-time priority for voice data.  The network could be 100% overloaded, lot of regular data lost, even denial of service, but not a single voice data packet will be lost or even delayed.  The network switches ensure that - 1) the voice data have absolute priority, bypassing any data with lower priority, thus eliminating lag, and - 2) the regular data get discarded if network congestion occurs and not the voice data.

[davout]

Correct me if I'm wrong, but both Mt. Gox and all the API clients in the world are using the same socket.io server.

You're wrong

Setting up a socket.io stack isn't trivial.

It got easier with the last version of nginx, but as far as I recall it has always required some fiddling around.

Socket.io doesn't support SSL natively AFAIK, so their setup probably uses stunnel to run the WS traffic in an SSL tunnel.

And stunnel is not exactly what I'd call stable.

[Zangelbert Bingledack]

If gox shares more details about their setup, I can try to speculate further about the issues.

You may want to bring this to Service Discussion.

[piramida]

Interesting. Although it's hard to understand who this would benefit. Although I'm sure someone could construct some sort of scenario.

I'm sure with the volume we were saying lately, 30% price swing caused by technical inability to process orders can benefit somebody by quite a large pile of cash

[justusranvier]

Although I'm sure someone could construct some sort of scenario.

I can construct a scenario in which Mt Gox benefits from the high trade volume generated by a panic.

They are operating on a fractional reserve of bitcoins due to losses from an undisclosed hack and need the extra fees to cover their losses.

[samson]

Does anyone know of any MtGox status pages or a place where they place important announcements ?

When the whole system gets into this state I think they should suspend all operations for a period of time before resuming.

MtGox need to make some serious announcements addressing the cause of these issues and tell us what they are going to do to prevent it happening again.

For all we know it will happen again in the next few days when we reach another all time high. That's what I'm expecting right now.