[BETA] MTGox websocket API, testers wanted

[genuise]

Clarks's client shows the same. Connects and then silence.

Does anybody see the same?

Total MtGox tease. Usually happens during high-traffic times.

Thank you. Just wanted to be sure that it is not my local problem.

By the way I wanted to thank you for your hint about orderbook state synchronization.

full book stamp and 60 secs depth channel buffer + ticker fixing made my mtgox stream api client work very well
Thank you once more

[Puppet]

i seem to be able to connect to websocket occasionally; but socket.io seems completely non responsive now?

[prof7bit]

i seem to be able to connect to websocket occasionally; but socket.io seems completely non responsive now?

Yesterday in the #mtgox channel on freenode MT has finally announced the hostname of the new beta server: socketio-beta.mtgox.com

This new server seems much improved, I have not yet seen it disconnecting or acting strange a single time since yesterday (maybe partially also because only few are using it yet).

goxtool now also supports this new server: pull the latest release and then:

Code:

./goxtool.py --protocol=socketio-beta

[genuise]

so the only thing that have to be changed is url? Any changes to socket.io lib version necessary?


are you sure you placed correct url? I tried and getting multiple reconnects and errors from socket io client?

can you check?

[prof7bit]

so the only thing that have to be changed is url? Any changes to socket.io lib version necessary?


are you sure you placed correct url? I tried and getting multiple reconnects and errors from socket io client?

can you check?

This also seems to use different server software and it at the moment it seems it is essential that the websocket connection is done on the same open socket that the initial http request came from. So the client has to fetch the session ID (with the header Connection: keep-alive) and then use that same socket to  upgrade the connection to websocket. Otherwise it will fail with reason "client not handshaken". MT said this is a bug and he intends to fix it. Meanwhile I have tweaked my own socketio client to be able to use the same socket for both requests, so for me it works. I don't know how the javascript implementation handles this.

Code:

<prof7bit> socketio-beta rocks. no disconnect no freeze, thats how it should have been all the time
<MagicalTux> heh
<MagicalTux> prof7bit: except connection will usually fail if you do not use the same socket for the initial request and the websocket
<prof7bit> that is fixed in my client now. it was actually simpler to fix than I expected
<MagicalTux> ya, but that's something /we/ should be actually fixing
<MagicalTux> we'll probably disable everything but websocket and flashsocket, and make socket.io server accept any session id

[genuise]

Does it mean that it is actually not original socket.io client have to be used but rather different client implementation?

Do you have any link to the new socketio-beta project?

or what I did not get?

[yucca]

Using puTTY I tried:

=====================================
websocket.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

Server replied with handshake ACK.

Then I was getting the stream displayed, until server seemed to drop connection. I tried this a few times with success and now cant repeat.

=====================================
socketio-beta.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio-beta.mtgox.com
Origin: null

No reply, Server remains silent?
=====================================

[Puppet]

Im getting this error using the beta socketio server with the javascript library:


DEBUG: Error: Hostname/IP doesn't match certificate's altnames
    at SecurePair.<anonymous> (tls.js:1280:23)
    at SecurePair.EventEmitter.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:883:10)
    at CleartextStream.read [as _read] (tls.js:421:15)
    at CleartextStream.Readable.read (_stream_readable.js:293:10)
    at EncryptedStream.write [as _write] (tls.js:330:25)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)
    at EncryptedStream.Writable.write (_stream_writable.js:172:11)
    at write (_stream_readable.js:547:24)

[yucca]

Im getting this error using the beta socketio server with the javascript library:


DEBUG: Error: Hostname/IP doesn't match certificate's altnames
    at SecurePair.<anonymous> (tls.js:1280:23)
    at SecurePair.EventEmitter.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:883:10)
    at CleartextStream.read [as _read] (tls.js:421:15)
    at CleartextStream.Readable.read (_stream_readable.js:293:10)
    at EncryptedStream.write [as _write] (tls.js:330:25)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)
    at EncryptedStream.Writable.write (_stream_writable.js:172:11)
    at write (_stream_readable.js:547:24)

maybe too much DNS? try using this as the URL:

socketio-lb-1655485032.us-west-1.elb.amazonaws.com

I still cant hook up thru telnet.

[prof7bit]

Using puTTY I tried:

=====================================
socketio-beta.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio-beta.mtgox.com
Origin: null

No reply, Server remains silent?
=====================================

becaue its a socket.io server and not a websocket server:

telnet socketio-beta.mtgox.com 80
Trying 184.169.152.65...
Connected to socketio-lb-1655485032.us-west-1.elb.amazonaws.com.
Escape character is '^]'.
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 23 Mar 2013 20:35:15 GMT
Connection: keep-alive
Transfer-Encoding: chunked

53
W_4my4iKDRSNXtJN89vh:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

then you parse the second line of the response take the ID W_4my4iKDRSNXtJN89vh to form a new URL that look like /socket.io/1/websocket/W_4my4iKDRSNXtJN89vh and then you use the same still open socket (its still open because you sent keep-alive header) and initiate the websocket request:

GET /socket.io/1/websocket/W_4my4iKDRSNXtJN89vh HTTP/1.1
Upgrade: WebSocket
and so on...

If it does not work then make sure that you are sending websocket version 13 headers (see the websocket RFC I posted a few days ago) and not the old version you showed above (that were not version 13 headers), this also means you need to provide Sec-Websocket-Key header and all that. See how goxtool does it (its using https://pypi.python.org/pypi/websocket-client/ with an overridden connect() method to do the initial http for the ID before the actual websocket starts).

You either need to find a complete socket.io implementation for your programming language or a websocket V13 client that you can tweak the same way like I did it with the python client.

[Fireball]

Im getting this error using the beta socketio server with the javascript library:

Yes, that's because this server uses cert issued for socketio.mtgox.com, which obviously does not match the different hostname.

[genuise]

So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

[prof7bit]

here is a complete example as far into the handshake as it is possible with telnet (because after that it switches to binary WS protocol):

--($)-- telnet socketio-beta.mtgox.com 80
Trying 184.169.152.65...
Connected to socketio-lb-1655485032.us-west-1.elb.amazonaws.com.
Escape character is '^]'.
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 23 Mar 2013 21:06:56 GMT
Connection: keep-alive
Transfer-Encoding: chunked

53
Qi-RuYva5rgLXtXuhsrE:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

GET /socket.io/1/websocket/Qi-RuYva5rgLXtXuhsrE HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

�1::�2::

the �1:: and the �2:: are the first two websocket frames it sends (websocket is a binary protocol), now it expects "1::/mtgox" and for every "2::" it sends you must reply "2::", of course you need to properly mask and frame what you send as it is described in the Websocket RFC. This is how socket.io over websocket (or websocket over socket.io?) looks like. Unfortunately from here on its impossible to proceed with telnet because of the binary frames.

[prof7bit]

So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

Maybe not use SSL, use unencrypted on port 80

[genuise]

So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

Maybe not use SSL, use unencrypted on port 80

Tried that already - the same result - errors

[yucca]

@prof7bit,

Thanks so much for your great reply!

I explored thru telnet first just to get my toes wet.

I am working ontop of WindowsMFC CAsyncSocket class and I got the websocket going OK (when server is up).

I'll set about trying to get CAsyncSocket to act as socket.io now and manage the binary frames as they arrive.

I'm not very experienced when it comes to streaming stuff, you just helped me bigtime!

[alpet]

My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?

[prof7bit]

My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?

Did you change your algorithm that creates your nonce? If you did then you might need a fresh API key. Nonce problems were the only problems I have witnessed causing sudden mysterious "Invalid call" problems.

Also if you send a lot of commands immediately after each other they will be processed in parallel on the server and this can mess up the chronological order (basically a race condition on their server) that will cause sporadic "Invalid call" (because the nonce not strictly incrementing anymore if the sequence is messed up). The only way around it is to answer each "Invalid call" by just sending the same command (with a new nonce of course) again... until it succeeds.

[alpet]

prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp. API key and secret are correct. Multiple attempts give same result "Invalid call". So bad that the server does not provide a detailed description of the error.

[yucca]

I've got a little further initiating the beta socketio stream, below is my TRACE.

############SOCKET OPENED BY CLIENT

############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked

############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80

############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

############SOCKET CLOSED BY HOST

The server closes the socket when I dont give it correct Sec-WebSocket-Key.

So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.

Should i do it like mentioned in websocket wiki?:

The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.