[BETA] MTGox websocket API, testers wanted

[Puppet]

On trade messages you need to update the orderbook yourself, there will be no depth message.

If the trade is type:"bid" then it has filled an ask order, update your asks, if trade is type:"ask" then it has filled a bid order, update your bids.

On *own* trades there will be two trade messages: One public that everybody receives (like above), use that to update your orderbook like above and one private only to notify you about your trade that happened. Also on every trade that affects your *own* orders there will be user_order messages, at least one that sets the volume to the remaining volume (or zero and another one to finally remove it).

thanks a lot. That explains the weirdness I was getting.

[1714115196]

1714115196

[1714115196]

1714115196

[1714115196]

1714115196

[Puppet]

is websocket service down? I can connected but I cant get any response (nor via telnet).
 Clarkmoody and the other sites still seem to work though?

[prof7bit]

is websocket service down? I can connected but I cant get any response (nor via telnet).
 Clarkmoody and the other sites still seem to work though?

Yes, its down again (for almost 3 hours now [this is nothing, sometimes its down for a few days]). And in the 24/7 IRC support channel #mtgox on freenode (btw. why does a commercial company use freenode?) there is currently nobody answering any questions.

Socketio works (clarkmoody is using socketio) but trading via the socketio server is no fun at all because of the 10 seconds lag (on top of the usual goxlag) even during quiet times and becomes totally unusable when there is some more action.

[Puppet]

Thanks for confirming its not just me. I thought socketio was implemented on top of websockets though, so its weird that still works?
Either way, Ill make a HTTP fallback option. Do you know how often you can poll it? The documentation says no more than once per 10 seconds, but Im not sure if that is per request or for all requests (I need trades, depth, and ticker) ?

[prof7bit]

Thanks for confirming its not just me. I thought socketio was implemented on top of websockets though, so its weird that still works?

its a separate server. And on the socketio websocket all messages look like this:

0x00 "4::/mtgox:{a json message}" 0xff
0x00 "4::/mtgox:{another json msg}" 0xff

and occasionally:

0x00 "2::" 0xff     <-- this is a ping, must be answered with "2::"

(I'm not totally sure about the 0xff 0x00 thing, there exist 13 different (no joke!) versions of websocket and there exist also different ways to delimit the messages, crazy stuff like: if bit7=1 then bit 0..6 = message length and when the next byte also has bit7=1 then its even more bits for the message length until a byte comes along that has bit7=0 but if all bits = 0 then its variable length ascii, 0xff terminated.

Fortunately the binary stuff is abstracted away by libraries like websocket.py in Python and other client implementations for other languages, so usually you just have a send() and a recv() method and don't need to care about 0x00 and 0xff etc., only the socketio "1::", "2::" and "4::" must be done manually.

The "/mtgox" part is some kind of "channel" name on the socketio layer but MtGox actually does not make any use of this feature, there is only one channel, the channels you subscribe *through* the websocket actually are an MtGox-internal thing, not part of the socketio layer. So there is some socketio protocol related stuff going on and wrapped around it that is not actually used for anything but its needed to satisfy the protocol rules.


On the plain websocket server you just connect, send the request header and it immediately begins talking to you with no additional handshake or other overhead:

0x00 "{a json message}" 0xff
0x00 "{another json msg}" 0xff

So its not the *same* websocket server (also it has a different hostname and IP address). Socketio is using websocket transport but a totally different handshake and protocol and framing.

Either way, Ill make a HTTP fallback option. Do you know how often you can poll it? The documentation says no more than once per 10 seconds, but Im not sure if that is per request or for all requests (I need trades, depth, and ticker) ?

depth (fulldepth) definitely has a limit and I ran into this once while testing (and it tried to ban me from calling this API for 24 hours [getting a new IP address on my DSL line fixed it]). I don't know about the other APIs.

[Fireball]

its a separate server. And on the socketio websocket all messages look like this:

0x00 "4::/mtgox:{a json message}" 0xff
0x00 "4::/mtgox:{another json msg}" 0xff

and occasionally:

0x00 "2::" 0xff     <-- this is a ping, must be answered with "2::"

...I'm not totally sure ...

There is a specification describing socket.io protocol.

[prof7bit]

...I'm not totally sure ...

There is a specification describing socket.io protocol.

With "I'm not totally sure" I was referring to websocket framing (0xff and friends), not to socketio. For websocket there exists an RFC. Websocket is actually not the problem, client implementations exist for many languages.

I actually just wanted to explain that these two MtGox servers are not the same and after writing down the bytes I realized that I might have forgotten some tiny details and therefore wanted to make clear that this is NOT a protocol specification, I just wrote it down as I remember it and so I wrote "I'm not totally sure" as a warning that it might be not 100% correct and nobody tries to implement something based on this posting.

[Puppet]

This sucks. I spent a day learning about websockets and making goxgame work pretty decently, now I can start over again and it seems got a choice between redoing most of the work while learning (a lot more) javascript and node.js to implement some existing nodejs socket.io implementation and make it talk with my db, or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

Thanks for the long explanation though.

[Fireball]

This sucks. I spent a day learning about websockets and making goxgame work pretty decently, now I can start over again and it seems got a choice between redoing most of the work while learning (a lot more) javascript and node.js to implement some existing nodejs socket.io implementation and make it talk with my db, or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

Thanks for the long explanation though.

Btw, if you're doing node.js+socket.io you can use my example: https://github.com/icbit/trader.nodejs
It's not very well tested and debugged yet, but it supports both Mt.Gox and ICBIT.

[prof7bit]

or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

If you have a working version for plain websocket already then it should be trivial to add the little socketio overhead on top of it without needing a complete socketio library.

basically instead of just connecting to websocket.mtgox.com like you do it now you do the following:

send a http request to socketio.mtgox.com (look for the example somewhere in this thread for the exact URL), parse the result to get the session ID which is part of your personalized websocket url and then you use *that* to do the websocket connect. The rest is adding (or removing) the "1::" and "4::/mtgox:" headers before/after send/receive and the rest stays the same, its effectively only a handful of code you have to add, you don't have to implement everything of socketio, just these few things, its actually simpler than it might seem.

There was a python example near the beginning (or somewhere in the middle) of this thread that helped me a lot, and of course also the official socketio specification.

[Puppet]

Ah, you make it sound easy! And it probably is when you know what you are doing, but I dont
I found the url to get the session thingy. Im not sure what to do next; this is the closest I got so far:

edit: I guess the handshake should be this instead?
GET /socket.io/1/websocket/454534534534 HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio.mtgox.com
Origin: null

Nothing is happening, no response.

edit2.
I debugged the python app linked above, and that made me use this string:

Code:

GET /socket.io/1/websocket/5360180921750645752 HTTP/1.1
Sec-WebSocket-Draft: 2
Host: socketio.mtgox.com
Upgrade: WebSocket
Connection: Upgrade
Origin: socketio.mtgox.com

Still no response, but now I wonder, do i need to set the authentication thing?

[prof7bit]

I debugged the python app linked above, and that made me use this string:

Code:

GET /socket.io/1/websocket/5360180921750645752 HTTP/1.1
Sec-WebSocket-Draft: 2
Host: socketio.mtgox.com
Upgrade: WebSocket
Connection: Upgrade
Origin: socketio.mtgox.com

Still no response, but now I wonder, do i need to set the authentication thing?

Looks good but remember there exist multiple different versions of websocket and their server might refuse to connect if you use the wrong version. The python example uses the websocket.py module https://pypi.python.org/pypi/websocket-client/ to do the actual websocket connect and so does my program to.

I remember before that I experimented with different implementations of websocket and not all of them worked, one of them was disconnected when sending the headers, i guess because it tried to use the wrong websocket version or something else was wrong with its headers (iirc by investigating the reasons it seemed to have to do with the sec-websocket-* headers and the random generated key, but then I just threw the code away and started from scratch with websocket.py). What I now use is urllib2 to do the initial http request, then construct the URL like above (the url you posted looks good) and then I let the websocket.py do the websocket connect and don't care about low level details like request headers

I don't know how this is in javascript/node.js (assuming this is what you are using?) but I guess there should exist some library too that will let you establish a webocket connection that takes care about all the low-level websocket request header stuff.

this is my working python code if this is of any help for you (its basically doing exactly the same as the other code snippets in this thread): (I guess to solve your problem you need to find the equivalent of "websocket.py" for your programming language)

Code:

        use_ssl = self.config.get_bool("gox", "use_ssl")
        wsp = {True: "wss://", False: "ws://"}[use_ssl]
        htp = {True: "https://", False: "http://"}[use_ssl]

        while not self._terminating: #loop 0 (connect, reconnect)
            try:
                url = urlopen(htp + self.SOCKETIO_HOST + "/socket.io/1?Currency=" + self.currency, timeout=20)
                params = url.read()
                url.close()

                ws_id = params.split(":")[0]
                ws_url = wsp + self.SOCKETIO_HOST + "/socket.io/1/websocket/" + ws_id + "?Currency=" + self.currency

                self.socket = websocket.WebSocket()
                self.socket.connect(ws_url)

                self.debug("connected")

                self.socket.send("1::/mtgox")
                self.socket.recv() # '1::'
                self.socket.recv() # '1::/mtgox'

                self.debug("subscribing to channels...")
                self.channel_subscribe()

                self.debug("waiting for data...")
                while not self._terminating: #loop1 (read messages)
                    msg = self.socket.recv()
                    if msg == "2::":
                        self.debug("### ping -> pong")
                        self.socket.send("2::")
                        continue
                    prefix = msg[:10]
                    if prefix == "4::/mtgox:":
                        str_json = msg[10:]
                        if str_json[0] == "{":
                            self.signal_recv(self, (str_json))

            except Exception as exc:
                if not self._terminating:
                    self.debug(exc, "reconnecting in 5 seconds...")
                    if self.socket:
                        self.socket.close()
                    time.sleep(5)

[Puppet]

Im actually using php server side, and there is very little to be found on the web. Only code to create websockets, not to connect to them.

Anyway, I dont mind coding it if I know what to send, but python is mostly like chinese to me, so can you do me one more favor and modify websocket.py
change line 43 to:
_debug = True

Then run your app.
It will spit out the handshakes. If you copy/paste the output  I can try to implement it in php.

[prof7bit]

Im actually using php server side, and there is very little to be found on the web. Only code to create websockets, not to connect to them.

Anyway, I dont mind coding it if I know what to send, but python is mostly like chinese to me, so can you do me one more favor and modify websocket.py
change line 43 to:
_debug = True

Then run your app.
It will spit out the handshakes. If you copy/paste the output  I can try to implement it in php.

You seem to be looking at a different version of websocket.py, there is no _debug variable at line 43 (I'm using version 0.9.0 (this is the latest version, implementing websocket version 13). I have enabled tracing, this is the log until after the channel_subscribe()

Note the messages starting with 0x81, this is the framing to delimit the messages, this is why it is absolutely no fun implementing this by hand if you can get a working implementation from somewhere. Now after looking at this dump myself I remember, my first attempt writing the webocket client myself the first problem I stumbled upon were the Sec-Websocket-Key and the Sec-Websocket-Version headers, they look slightly different for every of the dozen ws versions that exist and MtGox does not like all variants of them. The funny thing is my first implementation worked just fine with their plain websocket server and refused to connect to the socketio. Then i threw it away and used websocket.py.

Code:

2013-03-19 13:48:54,752:DEBUG:SocketIOClient:trying Websocket: wss://socketio.mtgox.com/socket.io/1/websocket/10083243691392078721?Currency=USD ...
2013-03-19 13:48:59,312:DEBUG:--- request header ---
2013-03-19 13:48:59,313:DEBUG:GET /socket.io/1/websocket/10083243691392078721?Currency=USD HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio.mtgox.com:443
Origin: socketio.mtgox.com:443
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13


2013-03-19 13:48:59,313:DEBUG:-----------------------
2013-03-19 13:48:59,313:DEBUG:--- response header ---
2013-03-19 13:48:59,762:DEBUG:HTTP/1.1 101 Switching Protocols
2013-03-19 13:48:59,763:DEBUG:Upgrade: websocket
2013-03-19 13:48:59,763:DEBUG:Connection: Upgrade
2013-03-19 13:48:59,764:DEBUG:Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=
2013-03-19 13:48:59,764:DEBUG:-----------------------
2013-03-19 13:48:59,764:DEBUG:SocketIOClient:connected
2013-03-19 13:48:59,767:DEBUG:send: '\x81\x89\x9f\x99\x10\xa1\xae\xa3*\x8e\xf2\xedw\xce\xe7'
2013-03-19 13:48:59,767:DEBUG:recv: '\x81\x031::'
2013-03-19 13:49:00,303:DEBUG:recv: '\x81\t1::/mtgox'
2013-03-19 13:49:00,304:DEBUG:SocketIOClient:subscribing to channels
2013-03-19 13:49:00,306:DEBUG:send: '\x81\xb4\xe7\xc2\xc8\xda\xd3\xf8\xf2\xf5\x8a\xb6\xaf\xb5\x9f\xf8\xb3\xf8\x93\xbb\xb8\xbf\xc5\xf8\xe8\xf8\x83\xa7\xb8\xae\x8f\xe0\xe4\xfa\xc5\xad\xb8\xf8\xdd\xe2\xea\xb7\x93\xa5\xa7\xa2\xc9\xb1\xbd\xb8\x94\xa1\xba\xb3\x85\xa7\xea\xa7'
2013-03-19 13:49:00,307:DEBUG:send: '\x81\xb5\x8c\xda\xf7\x7f\xb8\xe0\xcdP\xe1\xae\x90\x10\xf4\xe0\x8c]\xf8\xa3\x87\x1a\xae\xe0\xd7]\xf8\xb3\x94\x14\xe9\xa8\xd5S\xac\xf8\x98\x0f\xae\xe0\xd7]\xe1\xae\x90\x10\xf4\xf4\x84\n\xee\xa9\x94\r\xe5\xb8\x92]\xf1'
2013-03-19 13:49:00,307:DEBUG:send: '\x81\xb5w\xf2\x8b\xbeC\xc8\xb1\x91\x1a\x86\xec\xd1\x0f\xc8\xf0\x9c\x03\x8b\xfb\xdbU\xc8\xab\x9c\x03\x80\xea\xda\x12\x81\xa9\x92W\xd0\xe4\xceU\xc8\xab\x9c\x1a\x86\xec\xd1\x0f\xdc\xf8\xcb\x15\x81\xe8\xcc\x1e\x90\xee\x9c\n'

...and so on. Now its subscribed to all channels and should start sending data, 

[Puppet]

thanks for that. Before I try.. what that websocket key? It doesnt look anything like what I get from
https://socketio.mtgox.com/socket.io/1

I get  5360180921750645752

Is that related to your mtgox api key (and if so, is it safe to publish?) and if so, why doesnt it look anything like mine?
Or is it a fixed key and I can just use it?

[prof7bit]

thanks for that. Before I try.. what that websocket key? It doesnt look anything like what I get from
https://socketio.mtgox.com/socket.io/1

I get  5360180921750645752

Is that related to your mtgox api key (and if so, is it safe to publish?) and if so, why doesnt it look anything like mine?
Or is it a fixed key and I can just use it?

The 5360180921750645752 you get from the initial http request is the socketio session id, it belongs to the socketio protocol. It is generated by the server, see the socketio document on github (posted by Fireball a few postings before). its only used to generate the websocket GET request (the websocket url)

The Sec-Websocket-Key belongs to the websocket protocol (see: http://datatracker.ietf.org/doc/rfc6455/?include_text=1) its a random 16 byte you create and then base64 encode (see section 11.3.1 and section 4.1 (page 17, article 7))

_____
PS: The websocket protocol as defined in rfc6455 exists primarily to:
 * ensure that the google engineers who created it have secure jobs until the end of their life
 * ensure that nobody will ever write a *complete* reference implementation of serer and client
 * prove that it is possible to embed utter nonsense that serves no purpose like for example the masking of client-to-server data as specified in RFC6455 in an RFC and nobody recognizing it as what it really is: an april fools joke.
 * [...]
 * to establish a bidirectional connection between A and B

in exactly the above order.

[Puppet]

Cheers mate! With that I think I will make it work. Your help is much appreciated.
In case you are wondering, this is what Im working on:
http://goxgame.sytes.net/
Click legend to show/hide stuff, click and drag to zoom in.
Warning: buggy, obviously incomplete data, cant keep order book synched  etc,  its definitely still WIP.

[writhe]

My 2 Satoshi:

* ensure that nobody will ever write a *complete* reference implementation of serer and client

Really? I find it one of the simpler and better written networking specs out there (assuming you know some HTTP). I wrote a client library from scratch in a few days (~1000 lines of code) that I believe is complete. Don't Firefox and Chrome have complete client implementations with a Javascript API?

I also have a partial server implementation but that's because I don't need a WebSocket server for anything. Most of the work is already done because the framing and handshake code can be shared with the client version.

My only complaints are that it unnecessarily relies on crypto libraries (SHA-1 and base-64 for the handshake is overkill) and that the binary / text message distinction is kind of redundant. Perhaps it makes sense in the context of web browsers and Javascript.

* prove that it is possible to embed utter nonsense that serves no purpose like for example the masking of client-to-server data as specified in RFC6455 in an RFC and nobody recognizing it as what it really is: an april fools joke.

I also did a double-take when I read about the masking. However, I think it might make sense to avoid interference from web proxies. The part of the RFC that justifies masking as a way to avoid 'attacks on infrastructure' is utter bullshit though.

If you're worried about the performance hit then just set the mask to all zeros and have a fast path in the software that doesn't perform the XOR.

Fragmented messages may also be unnecessary but might be useful in low-latency applications. It doesn't add much complexity to handle them.

* to establish a bidirectional connection between A and B

A real bidirectional TCP connection is a pain when you factor in web proxies, firewalls, NAT, etc. WebSocket is a reasonable alternative with relatively little overhead per message.

[genuise]

I can see my socket.io client connects to mtgox and connect event is invoked.

and then no any subscribtion message at all nothing more.

this behavior is already for several hours.

restarting client does not help.

Clarks's client shows the same. Connects and then silence.

Does anybody see the same?

[Clark]

Clarks's client shows the same. Connects and then silence.

Does anybody see the same?

Total MtGox tease. Usually happens during high-traffic times.