[BETA] MTGox websocket API, testers wanted

[genuise]

... MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

this is interesting

[1713882948]

1713882948

[1713882948]

1713882948

[1713882948]

1713882948

[1713882948]

1713882948

[molecular]

We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood.

I mentioned this only as an explanation of why the problem is observed synchronously in all currencies. MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

I am observing the same problem in goxtool and have no solution for it. Also fulldepth is cached a few minutes by the server, so simply reloading fulldepth every x minutes will also lead to some sort of inconsistency, especially near the top of the book where most of the action is and where its most important for traders and bots. I have no idea how to properly and cleanly fix this mess client side, probably nothing can be done about it at all.

tbh: the solution is to switch exchange. magicaltux will never fix this. I've given up.

no matter what the problem is: magicaltux should either provide us with a solution to keep a synced orderbook or outsource it if he can't do it himself.

[Floomi]

Folks,

I've been reading this thread with interest as I too am writing a client in Python (using websocket-client). I'm connecting over plain old websockets (not socketio) and am mostly working apart from a couple of issues:

1. Once the connection is established I subscribe to the USD trades channel:

Code:

TRADES = "dbf1dee9-4f2e-4a08-8cb7-748919a71b21"
def subscribe( channel ):
	output = { 'op': 'mtgox.subscribe', 'channel': channel }
	return json.dumps( output )

ws.send(subscribe(TRADES))

...but I get the remark spat back "Unknown channel requested or not a public channel". I seem to be subscribed to trades by default, so this isn't a huge problem - but is there anything I'm doing wrong here?

2. My websocket connection keeps dropping, for no reason that I can see. This thread's history has some talk about having to send heartbeat messages, but from what I understand that's only when you're connected to socket.io, not the plain old websockets I keep on using. I find all clients I run drop at the same time, regardless of when I start them, which makes me suspect the problem is on Gox's end. I can easily reconnect to it when it drops, but it's frustrating because it takes about 90s between the connection being dropped and me getting a new trade message after I've reconnected. EDIT: I discovered a settimeout(5) sitting quietly in the code, which was likely wrecking things... removing it and upgrading websocket-client to 0.11 seems to have done the trick.

Any advice would be much appreciated!
-Floomi

[prof7bit]

2. My websocket connection keeps dropping, for no reason that I can see. This thread's history has some talk about having to send heartbeat messages,

websocket should never disconnect at all, I have it running here 24/7 and it disconnects only maybe once or twice a day, some days it doesn't disconnect at all (and probably my few disconnects here are only due to my very poor wifi connection).

[Floomi]

Well, it seems that the diagnosis for at least one of these two issues is "I'm an idiot" I've edited my post above with an update.

[prof7bit]

1. Once the connection is established I subscribe to the USD trades channel:

Code:

TRADES = "dbf1dee9-4f2e-4a08-8cb7-748919a71b21"
def subscribe( channel ):
	output = { 'op': 'mtgox.subscribe', 'channel': channel }
	return json.dumps( output )

ws.send(subscribe(TRADES))

...but I get the remark spat back "Unknown channel requested or not a public channel".

Try "mtgox.subscribe" instead of "subscribe" and/or also try "type" instead of "channel" and/or also try using the channel name instead of the GUID.

The documentation seems incomplete or outdated, you need to experiment a bit.

For example this works for me: {"op":"mtgox.subscribe", "type":"trades"}
And also: {"op":"mtgox.subscribe", "channel":"ticker.BTCUSD"}

And for subscribing to the private channel (after receiving my idkey) I'm using the following:
{"op":"mtgox.subscribe", "key":idkey}

[Floomi]

That works for me, thanks! It's really helpful that the documentation is full of lies 

[Floomi]

Thread necro! Is there a limit on the number of times you can call private/idkey? My bot asks for it at startup, which is many times a day during development, and towards the end of the day (perhaps past midnight, if that has anything to do with it?), I start getting "invalid call" back instead of the key.

It's trivial enough for me to store the idkey and the time it was requested, but I wanted to know if anyone else has seen this behaviour, or if it's just Broken On My Machine?

Ta.

[Floomi]

And another one (after this I'll just start shouting Bueller?): I don't seem to be able to subscribe to both depth and trade messages. If I do, I get depth but not trade. I can repro this in <50 lines of python; is this normal behaviour?

[yucca]

And another one (after this I'll just start shouting Bueller?): I don't seem to be able to subscribe to both depth and trade messages. If I do, I get depth but not trade. I can repro this in <50 lines of python; is this normal behaviour?

Normal for us two maybe...

Im trying to get websocket stream working (coded on top of vanilla C socket) and I seem to have same problem?

I am getting lag, depth and ticker but no trades.

One error message comes back, related to the failing trade subscription.

Code:

!!!THANKS TO PROF7BIT FOR SUBCRIPTION INFO IN EARLIER POST!!!

04:25:30 MtGox_Socket::SocketCreate: CONNECTED TO: websocket.mtgox.com:80
04:25:31 MtGox_Socket::DoHandshake_WebSocket: GOOD: NowListening
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "trade.lag"}
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "ticker.BTCUSD"}
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "depth.BTCUSD"}
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "trades.BTCUSD"}

04:25:33 
{
    "message":"Unknown channel requested or not a public channel",
    "op":"remark",
    "success":false
}
.....
Now lag, depth and ticker JSON comes streaming in
.....

[Clark]

Subscribe to trade.BTC 

c.f. http://data.mtgox.com/api/2/stream/list_public

[genuise]

wait a second, now you have to subscribe on your own?
They really changed the default behavior when on connect you were subscribed automatically?

[yucca]

Subscribe to trade.BTC 

c.f. http://data.mtgox.com/api/2/stream/list_public

Awesomeo, thx Clark!

[yucca]

Well Socket.io and Websocket have been down for a while, for all of us it seems.

Except for this guy, older API version or something? Or maybe just lots of HTTP requests?:

http://bitcointicker.co/

EDIT: curiosity got the better of me, wireshark caught this:

GET /socket.io/1/websocket/qRpvu5AQFZvju4nIao6m HTTP/1.1

[yucca]

wait a second, now you have to subscribe on your own?
They really changed the default behavior when on connect you were subscribed automatically?

Yes, it changed for websocket some time ago, or maybe websocket has always been like that I don't know.

And I just noticed that socket.io now also serves a dead stream to start with, you need to subscribe (i think/hope).

For socket.io I was passing all needed currencies as args in initial GET, that doesn't work now.

[yucca]

OK, I got websocket working in my client again.

I'm implementing the websocket protocol myself in C++ on a raw socket so this is probably not relevant to those using javascript in browsers.

It turns out that the MtGox websocket now requires properly framed messages as per RFC6455.

A while ago only the Socket.IO required the RFC6455 framing.

Before I was getting away with just this minimal framing for websocket: [0x00] ....RAW STRING DATA... [0xFF]

Now you need the RFC6455 framing function, I posted C code earlier in this thread.

Now I need to get my socket.io implementation working also

WebSocket + JSON == WASTED_BANDWIDTH

[Fireball]

WebSocket + JSON == WASTED_BANDWIDTH

I plan to investigate the option of using WebSocket + BJSON to prevent bandwidth wasting in ICBIT. We are fine as it is so far, but it's always good to plan for future.

[Clark]

WebSocket + JSON == WASTED_BANDWIDTH

I plan to investigate the option of using WebSocket + BJSON to prevent bandwidth wasting in ICBIT. We are fine as it is so far, but it's always good to plan for future.

Have you had a look at MessagePack? http://msgpack.org/

[savis]

I was using modified C# http://socketio4net.codeplex.com/ library to consume depths updates from https://socketio.mtgox.com/mtgox. It was working fine till the biggining of this month. Now it may work for several hours, then just drop and stop accepting connections (Error: The connection was closed unexpectedly.).

So I tried to switch to https://github.com/sta/websocket-sharp and connect to wss://websocket.mtgox.com/mtgox. The library seems to be recommended on https://en.bitcoin.it/wiki/MtGox/API/Streaming. First I spent several hours before I found I need to specify Origin. Now it works fine expect that server sends close frame every 5-60 seconds.

Any idea what could be the reason? Thanks.

Code:

01:49:02.874|DEBUG|26 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"132.13605","type":2,"type_str":"bid","volume":"-0.10083051","price_int":"13213605","volume_int":"-10083051","item":"BTC","currency":"USD","now":"1379713750305094","total_volume_int":"0"}}
01:49:02.960|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"123.90414","type":2,"type_str":"bid","volume":"0.60778582","price_int":"12390414","volume_int":"60778582","item":"BTC","currency":"USD","now":"1379713750414105","total_volume_int":"60778582"}}
01:49:04.186|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"139.39536","type":1,"type_str":"ask","volume":"1.103871","price_int":"13939536","volume_int":"110387100","item":"BTC","currency":"USD","now":"1379713751538095","total_volume_int":"110387100"}}
01:49:07.415|DEBUG|8  |MtGoxApi: Close Event raised
01:49:08.622|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"135.986","type":1,"type_str":"ask","volume":"5.196","price_int":"13598600","volume_int":"519600000","item":"BTC","currency":"USD","now":"1379713756075743","total_volume_int":"519600000"}}
01:49:08.661|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"133.85221","type":2,"type_str":"bid","volume":"-3.855","price_int":"13385221","volume_int":"-385500000","item":"BTC","currency":"USD","now":"1379713756090538","total_volume_int":"0"}}
01:49:08.676|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00225","type":2,"type_str":"bid","volume":"-10.8","price_int":"13400225","volume_int":"-1080000000","item":"BTC","currency":"USD","now":"1379713756105880","total_volume_int":"540000000"}}
01:49:09.511|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00225","type":2,"type_str":"bid","volume":"-5.4","price_int":"13400225","volume_int":"-540000000","item":"BTC","currency":"USD","now":"1379713756934842","total_volume_int":"0"}}
01:49:09.571|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"131.41101","type":2,"type_str":"bid","volume":"-5.4","price_int":"13141101","volume_int":"-540000000","item":"BTC","currency":"USD","now":"1379713756950113","total_volume_int":"1080000000"}}
01:49:09.607|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"133.27742","type":2,"type_str":"bid","volume":"5.4","price_int":"13327742","volume_int":"540000000","item":"BTC","currency":"USD","now":"1379713757041934","total_volume_int":"1620000000"}}
01:49:13.229|DEBUG|8  |MtGoxApi: Close Event raised
01:49:14.963|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"129.6864","type":2,"type_str":"bid","volume":"0.25","price_int":"12968640","volume_int":"25000000","item":"BTC","currency":"USD","now":"1379713762412059","total_volume_int":"125000000"}}
01:49:15.040|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00226","type":2,"type_str":"bid","volume":"-10.34395506","price_int":"13400226","volume_int":"-1034395506","item":"BTC","currency":"USD","now":"1379713762427277","total_volume_int":"0"}}
01:49:15.101|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00228","type":2,"type_str":"bid","volume":"10.34331649","price_int":"13400228","volume_int":"1034331649","item":"BTC","currency":"USD","now":"1379713762550896","total_volume_int":"1034331649"}}

[alikim]

Does anyone know how to subscribe to "own" private channel?

I start with this message:

{"call":"idkey","id":".....","nonce":...}

Successfully get the key:

{"id":"....","op":"result","result":"GhXJXwx..."}

Then subscribe:

{"op":"mtgox.subscribe","key":"GhXJXwx..."}

... and I don't get any response to the subscribe message and after sending orders or their execution I don't get any messages in the private "own" channel.

What am I doing wrong?

Thank you!