[BETA] MTGox websocket API, testers wanted

[pescador]

Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817. I do that and it seems to be working fine. I only don't know what to do with trades in other currencies.

I'm already using the trick to use ticker messages to fix the tables when they appear to be broken, but that doesn't happen a lot.

[genuise]

strange I currently use only depth and ticker messages. and did not have problems with orderbook state.

on my opinion there is not need to use trade messages because some trades can be from orders which are not in orderbook but executed before getting into orderbook.

but probably I missed something. It would be good to hear others

[prof7bit]

Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817

I did some further experimenting and debugging and it seems this old posting of mine is wrong. There *will* be a depth message immediately following the trade message, so theoretically it would be enough to use only depth messages to update the orderbook.

But in goxtool/goxapi I still update the orderbook already in the trade message, so a hypothetical goxtool bot that would rely on the *new* orderbook state *after* the trade has been executed would already find the orderbook in the new updated state from inside the slot_trade() slot. Since I only ever use the total_vol of the depth messages and ignore the volume difference in my implementation the depth messages that immediately follow a trade message will simply result in no change to the orderbook at all.

[pescador]

Thanks you both for your answers!

[alpet]

How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.

[prof7bit]

How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

[alpet]

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests

[prof7bit]

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests

You can also trigger your actions in the user_order message for order removal, this message will be fired when the order is completely filled or the last remaining part of it is filled (and therefore the order is removed). Problem here is: it will also fire when you cancel the order. So you need some additional code that knows what was going on before. If your bot is the only thing that ever cancels orders then will know the reason for the remove message but if you also cancel orders manually or through other bots running on the same account then you need somethng more sophisticated, then you need to keep track of the private trade messages too and if a trade message at the same price preceded the the order remove than it is a filled order, otherwise its a cancel or something like that.

The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.

[alpet]

The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.

I Agree, MtGox API need strongly to improve. Current realisation can be declared only as "not real-time". Extreme event lags! Some time required tens seconds, from send order to retrive all trades, what make API not usable for fast trailing stops and many others. Since my trading program cannot normal work with MtGox, I just switch to Icbit.

[yucca]

I'm trying to subscribe to "lag" channel on socketIO:

Tried sending this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}

and without quotes around channel ID like this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}


But whatever I try there is no response, and certainly no subscription success response.

My Socket.IO TX framing is correct, if I break the framing the mtgox server drops connection.

I'm not able to subscribe/unsubscribe any channels dynamically.

To add/remove currencies I have to reset connection with new list of currency codes in the initial HTTP exchange.

I wonder what am I doing wrong?

[genuise]

Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.



It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks

[Clark]

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

Well there was a $1M bid wall that got removed right before the selling started yesterday. I don't believe that large drop was an error.

[genuise]

that is the question,

there was no any corresponding trading at that time.

[molecular]

Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.



It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks

I've been having massive problems keeping order book synced in my client, there just seem to be lost messages. Never got it right.

Last resort: just pull full-depth 5 times per hour (magicaltux allows that frequency) and sync your orderbook

[genuise]

I agree, and wanted to add that this noticed affect is further suspicious becuase it is visible for all 8 currecly orderbooks at the same moment.

Just to remind that we support state of 8 currency orderbooks in the sime time. And this effect can be seen for all of them synchronosouly.

Needless to say that for such currencies like GPB or RUB here are significantly less trading activity which could be account for those simultanous volume decreases.

hm

[Eisenhower34]

Most of you are writing a trading bot, so Im sorry, but why do you all need full-depth? depth/fetch works fine and you can pull the data like every 30sec...

[genuise]

see bitcoin-analytics.com

[prof7bit]

I agree, and wanted to add that this noticed affect is further suspicious becuase it is visible for all 8 currecly orderbooks at the same moment.

MtGox has only one orderbook. Every order you add in one currency will also appear in every other currency (bids will appear 2.5% lower in other currencies, asks will appear 2.5% higher in other currencies).

[genuise]

As long as gox provides separate http requests for full orderbooks for different currencies and allows socket.io subscribtion to multiple currenclies sending depth messages for each currecy in a separate channel this allows us to support orderbook state for each currency separately as if they are independent.

Observed lost depth messages effect is visible not only for one crrency orderbook but for all of them synchronously. We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood. The thing is that no matter is there only one major orderbook or not, the mechanism of depth messages itself is broken and not all depth messages reach the client.

This is suported by the obvious neccessity of fixing invalid orderbook state (when best bid price is greater than best ask price) against ticker messages, but also by observing the effect of accumulated old (not removed) orders which lead to increase in orderbook volume through time and significant descrease immediately after orderbooks state was reinitialized.

[prof7bit]

We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood.

I mentioned this only as an explanation of why the problem is observed synchronously in all currencies. MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

I am observing the same problem in goxtool and have no solution for it. Also fulldepth is cached a few minutes by the server, so simply reloading fulldepth every x minutes will also lead to some sort of inconsistency, especially near the top of the book where most of the action is and where its most important for traders and bots. I have no idea how to properly and cleanly fix this mess client side, probably nothing can be done about it at all.