[BETA] MTGox websocket API, testers wanted

[pescador]

Thanks you both for your answers!

[alpet]

How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.

[prof7bit]

How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

[alpet]

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests

[prof7bit]

There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests

You can also trigger your actions in the user_order message for order removal, this message will be fired when the order is completely filled or the last remaining part of it is filled (and therefore the order is removed). Problem here is: it will also fire when you cancel the order. So you need some additional code that knows what was going on before. If your bot is the only thing that ever cancels orders then will know the reason for the remove message but if you also cancel orders manually or through other bots running on the same account then you need somethng more sophisticated, then you need to keep track of the private trade messages too and if a trade message at the same price preceded the the order remove than it is a filled order, otherwise its a cancel or something like that.

The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.

[alpet]

The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.

I Agree, MtGox API need strongly to improve. Current realisation can be declared only as "not real-time". Extreme event lags! Some time required tens seconds, from send order to retrive all trades, what make API not usable for fast trailing stops and many others. Since my trading program cannot normal work with MtGox, I just switch to Icbit.

[yucca]

I'm trying to subscribe to "lag" channel on socketIO:

Tried sending this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}

and without quotes around channel ID like this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}


But whatever I try there is no response, and certainly no subscription success response.

My Socket.IO TX framing is correct, if I break the framing the mtgox server drops connection.

I'm not able to subscribe/unsubscribe any channels dynamically.

To add/remove currencies I have to reset connection with new list of currency codes in the initial HTTP exchange.

I wonder what am I doing wrong?

[genuise]

Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.



It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks

[Clark]

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

Well there was a $1M bid wall that got removed right before the selling started yesterday. I don't believe that large drop was an error.

[genuise]

that is the question,

there was no any corresponding trading at that time.

[molecular]

Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.



It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks

I've been having massive problems keeping order book synced in my client, there just seem to be lost messages. Never got it right.

Last resort: just pull full-depth 5 times per hour (magicaltux allows that frequency) and sync your orderbook

[genuise]

I agree, and wanted to add that this noticed affect is further suspicious becuase it is visible for all 8 currecly orderbooks at the same moment.

Just to remind that we support state of 8 currency orderbooks in the sime time. And this effect can be seen for all of them synchronosouly.

Needless to say that for such currencies like GPB or RUB here are significantly less trading activity which could be account for those simultanous volume decreases.

hm

[Eisenhower34]

Most of you are writing a trading bot, so Im sorry, but why do you all need full-depth? depth/fetch works fine and you can pull the data like every 30sec...

[genuise]

see bitcoin-analytics.com

[prof7bit]

I agree, and wanted to add that this noticed affect is further suspicious becuase it is visible for all 8 currecly orderbooks at the same moment.

MtGox has only one orderbook. Every order you add in one currency will also appear in every other currency (bids will appear 2.5% lower in other currencies, asks will appear 2.5% higher in other currencies).

[genuise]

As long as gox provides separate http requests for full orderbooks for different currencies and allows socket.io subscribtion to multiple currenclies sending depth messages for each currecy in a separate channel this allows us to support orderbook state for each currency separately as if they are independent.

Observed lost depth messages effect is visible not only for one crrency orderbook but for all of them synchronously. We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood. The thing is that no matter is there only one major orderbook or not, the mechanism of depth messages itself is broken and not all depth messages reach the client.

This is suported by the obvious neccessity of fixing invalid orderbook state (when best bid price is greater than best ask price) against ticker messages, but also by observing the effect of accumulated old (not removed) orders which lead to increase in orderbook volume through time and significant descrease immediately after orderbooks state was reinitialized.

[prof7bit]

We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood.

I mentioned this only as an explanation of why the problem is observed synchronously in all currencies. MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

I am observing the same problem in goxtool and have no solution for it. Also fulldepth is cached a few minutes by the server, so simply reloading fulldepth every x minutes will also lead to some sort of inconsistency, especially near the top of the book where most of the action is and where its most important for traders and bots. I have no idea how to properly and cleanly fix this mess client side, probably nothing can be done about it at all.

[genuise]

... MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

this is interesting

[molecular]

We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood.

I mentioned this only as an explanation of why the problem is observed synchronously in all currencies. MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

I am observing the same problem in goxtool and have no solution for it. Also fulldepth is cached a few minutes by the server, so simply reloading fulldepth every x minutes will also lead to some sort of inconsistency, especially near the top of the book where most of the action is and where its most important for traders and bots. I have no idea how to properly and cleanly fix this mess client side, probably nothing can be done about it at all.

tbh: the solution is to switch exchange. magicaltux will never fix this. I've given up.

no matter what the problem is: magicaltux should either provide us with a solution to keep a synced orderbook or outsource it if he can't do it himself.

[Floomi]

Folks,

I've been reading this thread with interest as I too am writing a client in Python (using websocket-client). I'm connecting over plain old websockets (not socketio) and am mostly working apart from a couple of issues:

1. Once the connection is established I subscribe to the USD trades channel:

Code:

TRADES = "dbf1dee9-4f2e-4a08-8cb7-748919a71b21"
def subscribe( channel ):
	output = { 'op': 'mtgox.subscribe', 'channel': channel }
	return json.dumps( output )

ws.send(subscribe(TRADES))

...but I get the remark spat back "Unknown channel requested or not a public channel". I seem to be subscribed to trades by default, so this isn't a huge problem - but is there anything I'm doing wrong here?

2. My websocket connection keeps dropping, for no reason that I can see. This thread's history has some talk about having to send heartbeat messages, but from what I understand that's only when you're connected to socket.io, not the plain old websockets I keep on using. I find all clients I run drop at the same time, regardless of when I start them, which makes me suspect the problem is on Gox's end. I can easily reconnect to it when it drops, but it's frustrating because it takes about 90s between the connection being dropped and me getting a new trade message after I've reconnected. EDIT: I discovered a settimeout(5) sitting quietly in the code, which was likely wrecking things... removing it and upgrading websocket-client to 0.11 seems to have done the trick.

Any advice would be much appreciated!
-Floomi