[BETA] MTGox websocket API, testers wanted

[genuise]

so the only thing that have to be changed is url? Any changes to socket.io lib version necessary?


are you sure you placed correct url? I tried and getting multiple reconnects and errors from socket io client?

can you check?

[prof7bit]

so the only thing that have to be changed is url? Any changes to socket.io lib version necessary?


are you sure you placed correct url? I tried and getting multiple reconnects and errors from socket io client?

can you check?

This also seems to use different server software and it at the moment it seems it is essential that the websocket connection is done on the same open socket that the initial http request came from. So the client has to fetch the session ID (with the header Connection: keep-alive) and then use that same socket to  upgrade the connection to websocket. Otherwise it will fail with reason "client not handshaken". MT said this is a bug and he intends to fix it. Meanwhile I have tweaked my own socketio client to be able to use the same socket for both requests, so for me it works. I don't know how the javascript implementation handles this.

Code:

<prof7bit> socketio-beta rocks. no disconnect no freeze, thats how it should have been all the time
<MagicalTux> heh
<MagicalTux> prof7bit: except connection will usually fail if you do not use the same socket for the initial request and the websocket
<prof7bit> that is fixed in my client now. it was actually simpler to fix than I expected
<MagicalTux> ya, but that's something /we/ should be actually fixing
<MagicalTux> we'll probably disable everything but websocket and flashsocket, and make socket.io server accept any session id

[genuise]

Does it mean that it is actually not original socket.io client have to be used but rather different client implementation?

Do you have any link to the new socketio-beta project?

or what I did not get?

[yucca]

Using puTTY I tried:

=====================================
websocket.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

Server replied with handshake ACK.

Then I was getting the stream displayed, until server seemed to drop connection. I tried this a few times with success and now cant repeat.

=====================================
socketio-beta.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio-beta.mtgox.com
Origin: null

No reply, Server remains silent?
=====================================

[Puppet]

Im getting this error using the beta socketio server with the javascript library:


DEBUG: Error: Hostname/IP doesn't match certificate's altnames
    at SecurePair.<anonymous> (tls.js:1280:23)
    at SecurePair.EventEmitter.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:883:10)
    at CleartextStream.read [as _read] (tls.js:421:15)
    at CleartextStream.Readable.read (_stream_readable.js:293:10)
    at EncryptedStream.write [as _write] (tls.js:330:25)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)
    at EncryptedStream.Writable.write (_stream_writable.js:172:11)
    at write (_stream_readable.js:547:24)

[yucca]

Im getting this error using the beta socketio server with the javascript library:


DEBUG: Error: Hostname/IP doesn't match certificate's altnames
    at SecurePair.<anonymous> (tls.js:1280:23)
    at SecurePair.EventEmitter.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:883:10)
    at CleartextStream.read [as _read] (tls.js:421:15)
    at CleartextStream.Readable.read (_stream_readable.js:293:10)
    at EncryptedStream.write [as _write] (tls.js:330:25)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)
    at EncryptedStream.Writable.write (_stream_writable.js:172:11)
    at write (_stream_readable.js:547:24)

maybe too much DNS? try using this as the URL:

socketio-lb-1655485032.us-west-1.elb.amazonaws.com

I still cant hook up thru telnet.

[prof7bit]

Using puTTY I tried:

=====================================
socketio-beta.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio-beta.mtgox.com
Origin: null

No reply, Server remains silent?
=====================================

becaue its a socket.io server and not a websocket server:

telnet socketio-beta.mtgox.com 80
Trying 184.169.152.65...
Connected to socketio-lb-1655485032.us-west-1.elb.amazonaws.com.
Escape character is '^]'.
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 23 Mar 2013 20:35:15 GMT
Connection: keep-alive
Transfer-Encoding: chunked

53
W_4my4iKDRSNXtJN89vh:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

then you parse the second line of the response take the ID W_4my4iKDRSNXtJN89vh to form a new URL that look like /socket.io/1/websocket/W_4my4iKDRSNXtJN89vh and then you use the same still open socket (its still open because you sent keep-alive header) and initiate the websocket request:

GET /socket.io/1/websocket/W_4my4iKDRSNXtJN89vh HTTP/1.1
Upgrade: WebSocket
and so on...

If it does not work then make sure that you are sending websocket version 13 headers (see the websocket RFC I posted a few days ago) and not the old version you showed above (that were not version 13 headers), this also means you need to provide Sec-Websocket-Key header and all that. See how goxtool does it (its using https://pypi.python.org/pypi/websocket-client/ with an overridden connect() method to do the initial http for the ID before the actual websocket starts).

You either need to find a complete socket.io implementation for your programming language or a websocket V13 client that you can tweak the same way like I did it with the python client.

[Fireball]

Im getting this error using the beta socketio server with the javascript library:

Yes, that's because this server uses cert issued for socketio.mtgox.com, which obviously does not match the different hostname.

[genuise]

So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

[prof7bit]

here is a complete example as far into the handshake as it is possible with telnet (because after that it switches to binary WS protocol):

--($)-- telnet socketio-beta.mtgox.com 80
Trying 184.169.152.65...
Connected to socketio-lb-1655485032.us-west-1.elb.amazonaws.com.
Escape character is '^]'.
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 23 Mar 2013 21:06:56 GMT
Connection: keep-alive
Transfer-Encoding: chunked

53
Qi-RuYva5rgLXtXuhsrE:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

GET /socket.io/1/websocket/Qi-RuYva5rgLXtXuhsrE HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

�1::�2::

the �1:: and the �2:: are the first two websocket frames it sends (websocket is a binary protocol), now it expects "1::/mtgox" and for every "2::" it sends you must reply "2::", of course you need to properly mask and frame what you send as it is described in the Websocket RFC. This is how socket.io over websocket (or websocket over socket.io?) looks like. Unfortunately from here on its impossible to proceed with telnet because of the binary frames.

[prof7bit]

So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

Maybe not use SSL, use unencrypted on port 80

[genuise]

So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

Maybe not use SSL, use unencrypted on port 80

Tried that already - the same result - errors

[yucca]

@prof7bit,

Thanks so much for your great reply!

I explored thru telnet first just to get my toes wet.

I am working ontop of WindowsMFC CAsyncSocket class and I got the websocket going OK (when server is up).

I'll set about trying to get CAsyncSocket to act as socket.io now and manage the binary frames as they arrive.

I'm not very experienced when it comes to streaming stuff, you just helped me bigtime!

[alpet]

My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?

[prof7bit]

My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?

Did you change your algorithm that creates your nonce? If you did then you might need a fresh API key. Nonce problems were the only problems I have witnessed causing sudden mysterious "Invalid call" problems.

Also if you send a lot of commands immediately after each other they will be processed in parallel on the server and this can mess up the chronological order (basically a race condition on their server) that will cause sporadic "Invalid call" (because the nonce not strictly incrementing anymore if the sequence is messed up). The only way around it is to answer each "Invalid call" by just sending the same command (with a new nonce of course) again... until it succeeds.

[alpet]

prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp. API key and secret are correct. Multiple attempts give same result "Invalid call". So bad that the server does not provide a detailed description of the error.

[yucca]

I've got a little further initiating the beta socketio stream, below is my TRACE.

############SOCKET OPENED BY CLIENT

############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked

############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80

############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

############SOCKET CLOSED BY HOST

The server closes the socket when I dont give it correct Sec-WebSocket-Key.

So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.

Should i do it like mentioned in websocket wiki?:

The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.

[prof7bit]

I've got a little further initiating the beta socketio stream, below is my TRACE.

############SOCKET OPENED BY CLIENT

############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked

############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80

############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

############SOCKET CLOSED BY HOST

The server closes the socket when I dont give it correct Sec-WebSocket-Key.

So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.

Should i do it like mentioned in websocket wiki?:

The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.

There is something wrong. you are sending 3 GET requests instead of 2, the first one looks ok, the second one is wrong.

The sec-websocket-key is a random number (i believe 16 bytes without looking into the RFC) base64 encoded. the server does some simple transformations with it and returns it as sec-websocket-accept. You can basically throw this away, you could use it to detect that there is no caching proxy in between because for every different sec-websocket-key you will get a different sec-websocket-accept. So you could check if the server response really belongs to your request and is not something cached but you could just as well simply ignore it because no sane webcache would ever attempt to cache a websocket connection anyways.

Once you are past this initial handshake the real fun (or no fun at all) will begin: You will have to implement the framing and masking as defined in RFC 6455. This initial handshake will look like childs play compared to what is awaiting you in this RFC.

[prof7bit]

prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp.

There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)

Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.

[genuise]

Hey, guys, I am sorry but can you advice?

is it possible or will it be possible to use just plain nodjs socket.io client to connect to socketio-beta?

I use socket.io-client on server to fetch mtgox socketio stream. I do not have any http requests - client is connecting on its own.

But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?

thanks