[BETA] MTGox websocket API, testers wanted

[prof7bit]

I've got a little further initiating the beta socketio stream, below is my TRACE.

############SOCKET OPENED BY CLIENT

############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked

############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80

############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

############SOCKET CLOSED BY HOST

The server closes the socket when I dont give it correct Sec-WebSocket-Key.

So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.

Should i do it like mentioned in websocket wiki?:

The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.

There is something wrong. you are sending 3 GET requests instead of 2, the first one looks ok, the second one is wrong.

The sec-websocket-key is a random number (i believe 16 bytes without looking into the RFC) base64 encoded. the server does some simple transformations with it and returns it as sec-websocket-accept. You can basically throw this away, you could use it to detect that there is no caching proxy in between because for every different sec-websocket-key you will get a different sec-websocket-accept. So you could check if the server response really belongs to your request and is not something cached but you could just as well simply ignore it because no sane webcache would ever attempt to cache a websocket connection anyways.

Once you are past this initial handshake the real fun (or no fun at all) will begin: You will have to implement the framing and masking as defined in RFC 6455. This initial handshake will look like childs play compared to what is awaiting you in this RFC.

[1714788803]

1714788803

[1714788803]

1714788803

[1714788803]

1714788803

[prof7bit]

prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp.

There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)

Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.

[genuise]

Hey, guys, I am sorry but can you advice?

is it possible or will it be possible to use just plain nodjs socket.io client to connect to socketio-beta?

I use socket.io-client on server to fetch mtgox socketio stream. I do not have any http requests - client is connecting on its own.

But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?

thanks

[prof7bit]

But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?

There seems a bug in the server, MT has promised to fix it. Therefore its only "beta" and he hasn't replaced the old server yet. I have no idea when this will happen. Some client implementations can handle this bug (the bug is that both requests MUST come over the same socket), some other clients cant (until he fixes this bug). The ones who happen to have a working client (like me) are lucky because this server seems much more reliable and has had 0% downtime so far.

The GET requests and basically the entire last few pages of this thread with all their technical details about socket.io and GET requests and frames are only of interest if one must write his own socket.io/websocket client from scratch because there exist no socket.io libraries for many (most) programming languages except JavaScript (and this is because socket.io is not an internet protocol and not a standard of any kind, socket.io is just a private hobby project from some guys who hacked some code together as a non-solution for a non-problem and uploaded it to github and unfortunately MagicalTux somehow accidentally found this hobby-protocol and thought it was a good idea to use it for MtGox-API and now we all have to suffer from the consequences).

[alpet]

There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)

Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.

Yes, now it works again. Big Thanks!

[prof7bit]

Yes, now it works again. Big Thanks!

It works with md5? Then the bug is not fixed! It must NOT work with md5! Please show a json dump of one of your api calls, let me see how it looks like.

[prof7bit]

We can stop worrying about socketio-beta, MT has taken it live. socketio is now the new server and socketio-beta is obsolete and the old server is now socketio-old.mtgox.com

[genuise]

I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

[alpet]

It works with md5?

No, I just use large numbers calculated from system time.

[prof7bit]

I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.

[genuise]

I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.

thank you for clarification

[yucca]

For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.

In this table the least significant bit is on the right, makes it easier to mask and shift.

I cannot say this has no error, I have not tested, so verify by eye against original table first:

EDIT:BELOW TABLE IS WRONG! SCROLL DOWN PAGE FOR FIXED VERSION


   /*-------------------------------------------------------------*\
   | LSB ON RIGHT IN THIS TABLE                                    |
   |  3                   2                   1                    |
   |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
   +-------------------------------+-------------+-+-------+-+-+-+-+
   |    Extended payload length    | Payload len |M| opcode|R|R|R|F|
   |             (16/64)           |     (7)     |A|  (4)  |S|S|S|I|
   |   (if payload len==126/127)   |             |S|       |V|V|V|N|
   |                               |             |K|       |3|2|1| |
   +-------------------------------+-------------+-+-------+-+-+-+-+
   |     Extended payload length continued, if payload len == 127  |
   +-------------------------------+-------------------------------+
   |Masking-key, if MASK set to 1  |                               |
   +-------------------------------+-------------------------------+
   |          Payload Data         | Masking-key (continued)       |
   +-------------------------------+-------------------------------+
   :                     Payload Data continued ...                :
   +---------------------------------------------------------------+
   |                     Payload Data continued ...                |
   +--------------------------------------------------------------*/

      ORIGINAL TABLE: LSB ON LEFT.
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+

[writhe]

For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.

In this table the least significant bit is on the right, makes it easier to mask and shift.

I cannot say this has no error, I have not tested, so verify by eye against original table first:

Your table is incorrect. The original version in RFC 6455 is drawn with the most significant on the left. It's a bizarre numbering scheme used in most RFCs called MSB-0.

It's easier to read the RFC tables a byte at a time. The protocols assume the granularity of data transfer is 8-bit bytes so they don't really have the concept of a multi-bit word. When they do need multi-bit word they usually encode it as multiple 8-bit bytes using big-endian ordering.

For example, the lower 4 bits of the first byte are the opcode and the most significant bit is the FIN flag. Similarly, for the second byte the payload length is the lower 7 bits and the mask flag is the most significant bit.

Assuming little-endian with least significant bit (0) on the right, the first 2 bytes should look like this:

Code:

 1
 5    14 - 8     7 6 5 4  3 - 0
+-+-------------+-+-+-+-+--------+
|M| Payload len |F|R|R|R| opcode |
|A|      (7)    |I|S|S|S|  (4)   |
|S|             |N|V|V|V|        |
|K|             | |1|2|3|        |
+-+-------------+-+-+-+-+--------+

 |-------------| |-------------|
   second byte     first byte

[yucca]

Oh my... thanks muchly writhe!  Slowly but surely, I'll get this stream established.

Here's an ammended table for little endian:

   /*-------------------------------------------------------------*\
   |         LITTLE ENDIAN FORMAT WHERE LSB == 0x00000001          |
   |  3                   2                   1                   0|
   |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
   +-------------------------------+-+-------------+-+-+-+-+-------+
   |    Extended payload length    |M| Payload len |F|R|R|R| opcode|
   |             (16/64)           |A|     (7)     |I|S|S|S|  (4)  |
   |   (if payload len==126/127)   |S|             |N|V|V|V|       |
   |                               |K|             | |1|2|3|       |
   +-------------------------------+-+-------------+-+-+-+-+-------+
   |     Extended payload length continued, if payload len == 127  |
   +-------------------------------+-------------------------------+
   |Masking-key, if MASK set to 1  |                               |
   +-------------------------------+-------------------------------+
   |          Payload Data         | Masking-key (continued)       |
   +-------------------------------+-------------------------------+
   :                     Payload Data continued ...                :
   +---------------------------------------------------------------+
   |                     Payload Data continued ...                |
   \*-------------------------------------------------------------*/

[yucca]

I got the socket.io ticker stream coming in... weeeeee!

Thanks to prof7bit, few satoshis coming your way!!!!

I used a hardcoded packet for first send after socket upgrade ala prof7bit's python debug output.

Code:

	unsigned char pBuffer[15];
	unsigned char* pBuild = pBuffer;
	*pBuild++ = 0x81;
	*pBuild++ = 0x89;
	*pBuild++ = 0x9f;
	*pBuild++ = 0x99;
	*pBuild++ = 0x10;
	*pBuild++ = 0xa1;
	*pBuild++ = 0xae;
	*pBuild++ = 0xa3;
	*pBuild++ =  '*';
	*pBuild++ = 0x8e;
	*pBuild++ = 0xf2;
	*pBuild++ = 0xed;
	*pBuild++ =  'w';
	*pBuild++ = 0xce;
	*pBuild++ = 0xe7;
	Transmit_Raw(pBuffer, 15, false, true);

I cant reply to its echo requests yet because my frame/mask func has gremlins. But the data is coming in well until echo request.

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

[writhe]

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

There are some examples in the RFC that might be useful to check against.

Also, using all zeros for the masking key is valid (you still need to set the masking bit though). This should make it easier to debug as the payload won't be modified by the XOR operation.

[yucca]

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

There are some examples in the RFC that might be useful to check against.

Also, using all zeros for the masking key is valid (you still need to set the masking bit though). This should make it easier to debug as the payload won't be modified by the XOR operation.

Thanks, i missed that, (note to self: READ specs before coding!).

So now i have realised that the payload length bytes are dynamic and are only used if payload size needs them. ahhh so thats why the table has dotted lines for some cell walls. doh!

edit: and thx for the zero mask tip, sure will make things simpler to get going!

[yucca]

here is a C mask/frame function that is tested and working, again thanks guys for the help:

edit: See a few posts on for the func....

[genuise]

sorry, can you please explain what you are actually developing?

[writhe]

here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:

• payload length is < 126 bytes,
• payload length is < 65536 bytes,
• payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).