The coming change at mtGox and its implications

[molecular]

Last night I finally understood what's with the gox engine lag:

<MagicalTux> sturles: if you need realtime data, use socket.io
<molec> MagicalTux, "realtime" ;-)
<MagicalTux> molec: socket.io feed is actually realtime
<molec> MagicalTux, so it's the engine that can't cope?
<MagicalTux> right now, yep
<MagicalTux> because it needs to do too many checks
<MagicalTux> we are working on changing the way all this works
<MagicalTux> but it needs a *lot* of changes
<molec> MagicalTux, I think you're doing something wrong. I can't see why it's hard... what's the reason?
<molec> MagicalTux, might be my ignorance, but I did quite a few dbs. I might be overlooking some problem,... maybe security-related?
<MagicalTux> molec: we need to change the way users place orders to solve this
<molec> MagicalTux, in what way?
<MagicalTux> disallow users to place orders they can't fill
<molec> MagicalTux, +1. oh...
<molec> MagicalTux, it's dawning on me
<MagicalTux> and disallow users to withdraw balance required for open orders
<molec> MagicalTux, you have to trickle down the whole order list to set invalid/open status
<molec> MagicalTux, go do it!
<MagicalTux> right now each time a user balance change (trade, etc) we need to recheck all his open orders
<molec> MagicalTux, I see. and that gives you concurrency issues like hell
<MagicalTux> we're working on solving this, but changing the way the site works need to be done right
<MagicalTux> so we're doing a lot of tests first
<molec> MagicalTux, I agree. don't haste
<MagicalTux> well
<MagicalTux> we need to haste, because current volumes cannot be handled anymore by current engine
<MagicalTux> we made it faster (no more 30min delays), but it still has troubles with large moves

What this means is: you can't have any order in the book that aren't backed by funds. Currently "un-backed" orders will have status "invalid" and wont show in the public order book, but they can exist and once funds become available to back the order, it will change its status to "open".

What are the implications of this?

Currently, in a uptrend market like we have had for a long time, when price rises and people who have invalid bids get their asks filled, fund become available and (some of) the bids of that user appear automatically in the orderbook. As I see it, this mechanism is rather bullish, because it keeps the bid-side of the book nicely filled up.

Once these invalid order cannot exist any more due to the change, the bid-side will not fill up like that any more. This might be interpreted by the market participant as a weakness of the bulls, especiall if the participants are not yet fully aware of the change.

On the bright side: we can expect the trading engine to perform much better.

[1714504881]

1714504881

[1714504881]

1714504881

[1714504881]

1714504881

[1714504881]

1714504881

[oakpacific]

If I just did a calculation with my ass, it's gonna make the complexity roughly O(m*log(n)) instead of O(log(n)), and m could be easily larger than log(n).

I suspect there are many badly-programmed bots relying on this out there.

[genuise]

Does it also mean that ask side is also working the same way? So this is also bearish?

[oakpacific]

Does this have anything to do with the original requirement of a card trading exchange?

[molecular]

Does it also mean that ask side is also working the same way? So this is also bearish?

The situation I described had bearish result (rising market, no more auto-opening of bids => perceptively less bids). The other way around has bullish result: in a crash or falling market no more auto-opening of asks => ask side looks less impressive.

So if I judge this right, it could dampen the moves (up and down).

But keep in mind: this is mainly by perception. If people judge this correctly, it should not have much impact at all.

EDIT: just thought again: of course the order-book will be thinner, maybe this could mean that rebounds will be even quicker.

[Zomdifros]

With the transition of North American customers to Coinlab, are they still going to use the same trading engine (in Tokyo?)? If all these orders will be done on a new engine in the US, this could drastically reduce the pressure on the current system. If so, when will this transition happen?

[molecular]

With the transition of North American customers to Coinlab, are they still going to use the same trading engine (in Tokyo?)? If all these orders will be done on a new engine in the US, this could drastically reduce the pressure on the current system. If so, when will this transition happen?

I can't imagine them running 2 markets. That doesn't make sense.

[damnek]

Does it also mean that ask side is also working the same way? So this is also bearish?

The situation I described had bearish result (rising market, no more auto-opening of bids => perceptively less bids). The other way around has bullish result: in a crash or falling market no more auto-opening of asks => ask side looks less impressive.

So if I judge this right, it could dampen the moves (up and down).

But keep in mind: this is mainly by perception. If people judge this correctly, it should not have much impact at all.

EDIT: just thought again: of course the order-book will be thinner, maybe this could mean that rebounds will be even quicker.

I sometimes use the current system to actually dampen some moves. For instance last night I set a sell at 92 USD and a bid, while not having enough USD funds, at 88.

[oakpacific]

We need to pay attention especially to a rapid price movement scenario, when people will be trying to issue orders desperately, and may never bother to check on their balances, and Gox's UI irresponsiveness only adds salt to the injury, as people will try to issue more orders when they can't see their orders showing on the list. The more orders people issue, the greater the lag, which would make people more anxious and making more vain efforts, which in turn causes even more lag. This vicious closed-feedback loop causes the ridiculous 10-minutes lag we saw on a weekly basis.

[Hawkix]

Last night I finally understood what's with the gox engine lag:

...
<MagicalTux> molec: we need to change the way users place orders to solve this
<molec> MagicalTux, in what way?
<MagicalTux> disallow users to place orders they can't fill
<molec> MagicalTux, +1. oh...
<molec> MagicalTux, it's dawning on me
<MagicalTux> and disallow users to withdraw balance required for open orders
<molec> MagicalTux, you have to trickle down the whole order list to set invalid/open status
<molec> MagicalTux, go do it!
<MagicalTux> right now each time a user balance change (trade, etc) we need to recheck all his open orders
...

I think that with proper data structures, one can link together the bids and asks of single user and immediately update "reserve" for the bid/ask after sell or buy. Funding and withdrawing are infrequent compared to trades. The structure can be held in memory, with commits to permanent storage. This should handle hundreds of trades per second, not a few, like it is now.

Of course, this requires to avoid using normal relational databases, queries upon them, PHP code in server, etc.

Also, the socket with trade data is sometimes overloaded, too. I am not a TCP/IP expert, but I cant see problem with cloning the data between multiple (in order of thousands) sockets open. This would solve all those disconnects on ClarkMoody, MtGoxLive, etc.

An exchange, which cannot provide reliable and non-lagged data for its users, is .. unusable. Like that graph at the top of mtgox.com. Isn't it ironic that to trade at MtGox, or just watch it, you need to use third party tools?

[damnek]

Last night I finally understood what's with the gox engine lag:

...
<MagicalTux> molec: we need to change the way users place orders to solve this
<molec> MagicalTux, in what way?
<MagicalTux> disallow users to place orders they can't fill
<molec> MagicalTux, +1. oh...
<molec> MagicalTux, it's dawning on me
<MagicalTux> and disallow users to withdraw balance required for open orders
<molec> MagicalTux, you have to trickle down the whole order list to set invalid/open status
<molec> MagicalTux, go do it!
<MagicalTux> right now each time a user balance change (trade, etc) we need to recheck all his open orders
...

I think that with proper data structures, one can link together the bids and asks of single user and immediately update "reserve" for the bid/ask after sell or buy. Funding and withdrawing are infrequent compared to trades. The structure can be held in memory, with commits to permanent storage. This should handle hundreds of trades per second, not a few, like it is now.

Of course, this requires to avoid using normal relational databases, queries upon them, PHP code in server, etc.

Also, the socket with trade data is sometimes overloaded, too. I am not a TCP/IP expert, but I cant see problem with cloning the data between multiple (in order of thousands) sockets open. This would solve all those disconnects on ClarkMoody, MtGoxLive, etc.

An exchange, which cannot provide reliable and non-lagged data for its users, is .. unusable. Like that graph at the top of mtgox.com. Isn't it ironic that to trade at MtGox, or just watch it, you need to use third party tools?

Absolutely agree with you. I've been long enough to know that bitcoin and exchanges are not the same thing but I can't help to think that this market got too big for what the infrastructure really is. It's just not there yet.

[molecular]

We need to pay attention especially to a rapid price movement scenario, when people will be trying to issue orders desperately, and may never bother to check on their balances, and Gox's UI irresponsiveness only adds salt to the injury, as people will try to issue more orders when they can't see their orders showing on the list. The more orders people issue, the greater the lag, which would make people more anxious and making more vain efforts, which in turn causes even more lag. This vicious closed-feedback loop causes the ridiculous 10-minutes lag we saw on a weekly basis.

The idea is that this change will do away with the lag problem. So the UI will hopefully be responsive.

[molecular]

An exchange, which cannot provide reliable and non-lagged data for its users, is .. unusable. Like that graph at the top of mtgox.com. Isn't it ironic that to trade at MtGox, or just watch it, you need to use third party tools?

According to MagicalTux (and I can actually believe what he says there), it's not the socket.io/websocket that lags. It's actually "realtime", it's the engine that can't cope with the amount of trades.

[molecular]

Last night I finally understood what's with the gox engine lag:

...
<MagicalTux> molec: we need to change the way users place orders to solve this
<molec> MagicalTux, in what way?
<MagicalTux> disallow users to place orders they can't fill
<molec> MagicalTux, +1. oh...
<molec> MagicalTux, it's dawning on me
<MagicalTux> and disallow users to withdraw balance required for open orders
<molec> MagicalTux, you have to trickle down the whole order list to set invalid/open status
<molec> MagicalTux, go do it!
<MagicalTux> right now each time a user balance change (trade, etc) we need to recheck all his open orders
...

I think that with proper data structures, one can link together the bids and asks of single user and immediately update "reserve" for the bid/ask after sell or buy. Funding and withdrawing are infrequent compared to trades. The structure can be held in memory, with commits to permanent storage. This should handle hundreds of trades per second, not a few, like it is now.

Of course, this requires to avoid using normal relational databases, queries upon them, PHP code in server, etc.

Also, the socket with trade data is sometimes overloaded, too. I am not a TCP/IP expert, but I cant see problem with cloning the data between multiple (in order of thousands) sockets open. This would solve all those disconnects on ClarkMoody, MtGoxLive, etc.

An exchange, which cannot provide reliable and non-lagged data for its users, is .. unusable. Like that graph at the top of mtgox.com. Isn't it ironic that to trade at MtGox, or just watch it, you need to use third party tools?

Absolutely agree with you. I've been long enough to know that bitcoin and exchanges are not the same thing but I can't help to think that this market got too big for what the infrastructure really is. It's just not there yet.

And that is exactly the reason why this change on the infrastructure is being made.

It's debatable wether or not this could be done with the same order behaviour. mtGox decided to change the order behaviour to make the engine perform better. That's ok with me as long as it actually solves the issue and I know about it beforehand.

[Zomdifros]

With the transition of North American customers to Coinlab, are they still going to use the same trading engine (in Tokyo?)? If all these orders will be done on a new engine in the US, this could drastically reduce the pressure on the current system. If so, when will this transition happen?

I can't imagine them running 2 markets. That doesn't make sense.

I guess not, but when Coinlab customers are able to get their dollars on the trade floor within 30 minutes through Silicon Valley Bank, this is gonna weigh on their engine. Then I can imagine that MagicalTux really has to work on a tight schedule. I wish him all the best pulling this of!

[molecular]

With the transition of North American customers to Coinlab, are they still going to use the same trading engine (in Tokyo?)? If all these orders will be done on a new engine in the US, this could drastically reduce the pressure on the current system. If so, when will this transition happen?

I can't imagine them running 2 markets. That doesn't make sense.

I guess not, but when Coinlab customers are able to get their dollars on the trade floor within 30 minutes through Silicon Valley Bank, this is gonna weigh on their engine. Then I can imagine that MagicalTux really has to work on a tight schedule. I wish him all the best pulling this of!

They can (and should) just maintain a central order book where all orders present are sure do be fully covered (see my previous comment), and have the two "brokers" independantly insert orders into it.

This (and your previous description about maintianing 2 balances) is effectively what this change is about. No need to demand it, Magicaltux is on it. Wether or not he will use your implementation idea is up to him.

[Enigma81]

This whole things is either
A ) Complete bullshit
or
B ) A sign of the level of inept that is MtGox

For God's sake - It's NOT that damn hard.  It's just not.  I promise.  They do not need to limit the ability of their customers to do what they can do now (place currently unfunded orders).  When I traded at Gox, I did it all the time.  If I thought the price might fall, place a sell at -5% and then a buy (unfunded) at -10%.. Or something of that nature.  It's a nice "feature".

Re-Calculating an accounts open-orders after each transaction is just NOT that big of a deal.  If the site was designed by anyone with even the remotest sense of distributed computing, it would be such a non-issue that this discussion wouldn't even be taking place.  A properly designed trading system could handle the current load of bitcoin trading and still twiddle it's electronic fingers 75% of the time.

I'm not a huge Oracle fan.. But jesus Gox.. Call them.. Or IBM.. or one of the other heavy-metal DB-App specialists.. They'll be happy to politely explain how it works.

SOOOOO glad I left Gox.  Yeah, the reduced liquidity is a bit of a bummer.. But when things start rolling, I place a trade and its done... No 19 Minute "LAG" shit..

Enigma

[foo]

This new system is how Bitstamp works already, so you can get a preview of it there.

[bitcoinBull]

Re-Calculating an accounts open-orders after each transaction is just NOT that big of a deal.  If the site was designed by anyone with even the remotest sense of distributed computing, it would be such a non-issue that this discussion wouldn't even be taking place.  A properly designed trading system could handle the current load of bitcoin trading and still twiddle it's electronic fingers 75% of the time.

Scaling a database which needs guaranteed atomic consistency is hard. Nasdaq had a "flash crash" when it couldn't handle the load, and BATS botched their IPO when they debuted a new back-end and their APPL prices went haywire. They had to suspend trading and roll back trades.

I am disappointed that mtgox still lags, but I am glad that MagicalTux is doing more extensive testing. I remember when we couldn't go more than a month without seeing some mismatched rogue trades show up on bitcoincharts outside of the trading range with crazy volume, and not knowing whether they were real trades until they were deleted and removed from the charts. And rememember when trading was "halted" once, for about 12 hours, when someone sent an invalid currency pair to the API? Its more important to prevent those kinds of mishaps than to do realtime processing of peak volume (though i agree, we are paying enough for both).

[oakpacific]

Are there people actively taking advantage of this deficiency? Gox is  lagging even if there are only several orders being placed in total, WTF?

[Le Happy Merchant]

If I just did a calculation with my ass, it's gonna make the complexity roughly O(m*log(n)) instead of O(log(n)), and m could be easily larger than log(n).

I suspect there are many badly-programmed bots relying on this out there.

If m > log(n) > 1 this is bad for the complexity.
If 1 > m > log(n) this is good for the complexity.

What are you trying to say?

[SkRRJyTC]

Are there people actively taking advantage of this deficiency? Gox is  lagging even if there are only several orders being placed in total, WTF?

Would be silly not to take advantage, since it is extremely easy.

[hgmichna]

Not allowing users to place an order if they do not have the funds at the time the order is placed seems wrong to me. The order needs the funds only when it is executed, not before. Consider that Mt.Gox is there to fulfill its users' needs, not the other way around. Provided, of course, that the users pay appropriately for the service.

The user could have a combination of orders that works, even though not all orders are funded beforehand. There could be sufficient funds coming in before an order is executed. I see no need to check funding before an order is actually executed. I would not complain if they did that though. They can do what they like, as long as they don't alienate their users.

I do see a need though for a fixed base fee for transactions, like every other online broker has. If necessary, there could even be a base fee for placing an order or a punishment fee for trying to execute an unfunded order. It seems that Mt.Gox is wasting their limited computer power on lots of tiny nuisance orders. The whole bitcoin system is not designed for large numbers of tiny transactions. There is a block size limit, for example. Bitcoin is not a micro-payment system. This will become ever more obvious as bitcoin trade volume grows.

[molecular]

Not allowing users to place an order if they do not have the funds at the time the order is placed seems wrong to me. The order needs the funds only when it is executed, not before. Consider that Mt.Gox is there to fulfill its users' needs, not the other way around. Provided, of course, that the users pay appropriately for the service.

The user could have a combination of orders that works, even though not all orders are funded beforehand. There could be sufficient funds coming in before an order is executed. I see no need to check funding before an order is actually executed. I would not complain if they did that though. They can do what they like, as long as they don't alienate their users.

I do see a need though for a fixed base fee for transactions, like every other online broker has. If necessary, there could even be a base fee for placing an order or a punishment fee for trying to execute an unfunded order. It seems that Mt.Gox is wasting their limited computer power on lots of tiny nuisance orders. The whole bitcoin system is not designed for large numbers of tiny transactions. There is a block size limit, for example. Bitcoin is not a micro-payment system. This will become ever more obvious as bitcoin trade volume grows.

I see the need to check funding of order before they are displayed in the public order book. Unfunded order should not be in the book because they cannot be fulfilled.

[dancingnancy]

Not allowing users to place an order if they do not have the funds at the time the order is placed seems wrong to me. The order needs the funds only when it is executed, not before. Consider that Mt.Gox is there to fulfill its users' needs, not the other way around. Provided, of course, that the users pay appropriately for the service.

The user could have a combination of orders that works, even though not all orders are funded beforehand. There could be sufficient funds coming in before an order is executed. I see no need to check funding before an order is actually executed. I would not complain if they did that though. They can do what they like, as long as they don't alienate their users.

I do see a need though for a fixed base fee for transactions, like every other online broker has. If necessary, there could even be a base fee for placing an order or a punishment fee for trying to execute an unfunded order. It seems that Mt.Gox is wasting their limited computer power on lots of tiny nuisance orders. The whole bitcoin system is not designed for large numbers of tiny transactions. There is a block size limit, for example. Bitcoin is not a micro-payment system. This will become ever more obvious as bitcoin trade volume grows.

I see the need to check funding of order before they are displayed in the public order book. Unfunded order should not be in the book because they cannot be fulfilled.

True.  This is what bitcoinica was doing as well, IIRC.  You could post anything you wanted to the order book, but it was another thing if you had the money in your account.  Seems entirely messed up, not really sure what to think about all that.

[myself]

well in stock and forex u cant advertise that you want to buy or sell a currency or a stock if you dont freaking owned it first

[molecular]

well in stock and forex u cant advertise that you want to buy or sell a currency or a stock if you dont freaking owned it first

what about naked short selling?

[shmadz]

Not familiar with Gox.

Over at virtex, every bid or ask that you place puts that money into escrow. so if you don't have the money (ex. if I want to buy right now, but all my money is tied up in lower bids) I have to cancel some bids to free up the money.

From the sounds of it, this is not the way it works at Gox?

Is the escrow system just dumb? or is it something that Gox will have to implement to get around this problem?

(was thinking of trying Gox out, once coinlab takes over Can/US payments, since virtex % is ridiculous (they started out at 0.6% - now they're up to 3% for low volume accounts)

[hgmichna]

Not familiar with Gox.

Over at virtex, every bid or ask that you place puts that money into escrow. so if you don't have the money (ex. if I want to buy right now, but all my money is tied up in lower bids) I have to cancel some bids to free up the money.

From the sounds of it, this is not the way it works at Gox?

Is the escrow system just dumb? or is it something that Gox will have to implement to get around this problem? […]

From what I have read here, Mt.Gox allows orders without complete funding. In my view this is not a problem, unless users heavily overuse this feature. But Mt.Gox could protect itself against this easily, as they have the whole account history and could easily put up a simple rule that prevents the overuse of this feature.

As I have mentioned before, their real problem is the large number of tiny nuisance orders. I think they should first demand a small fixed fee for each order or for each transaction to get rid of micro-orders.

[myself]

well in stock and forex u cant advertise that you want to buy or sell a currency or a stock if you dont freaking owned it first

what about naked short selling?

you cant do that no broker allow you to do that

[hgmichna]

well in stock and forex u cant advertise that you want to buy or sell a currency or a stock if you dont freaking owned it first

what about naked short selling?

you cant do that no broker allow you to do that

Outside bitcoin it is normal within some reasonable margin, so I'd expect it to become normal for bitcoin too, as soon as the bitcoin market is a bit more stable and mature.