[BETA] MTGox websocket API, testers wanted

[prof7bit]

But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?

There seems a bug in the server, MT has promised to fix it. Therefore its only "beta" and he hasn't replaced the old server yet. I have no idea when this will happen. Some client implementations can handle this bug (the bug is that both requests MUST come over the same socket), some other clients cant (until he fixes this bug). The ones who happen to have a working client (like me) are lucky because this server seems much more reliable and has had 0% downtime so far.

The GET requests and basically the entire last few pages of this thread with all their technical details about socket.io and GET requests and frames are only of interest if one must write his own socket.io/websocket client from scratch because there exist no socket.io libraries for many (most) programming languages except JavaScript (and this is because socket.io is not an internet protocol and not a standard of any kind, socket.io is just a private hobby project from some guys who hacked some code together as a non-solution for a non-problem and uploaded it to github and unfortunately MagicalTux somehow accidentally found this hobby-protocol and thought it was a good idea to use it for MtGox-API and now we all have to suffer from the consequences).

[alpet]

There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)

Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.

Yes, now it works again. Big Thanks!

[prof7bit]

Yes, now it works again. Big Thanks!

It works with md5? Then the bug is not fixed! It must NOT work with md5! Please show a json dump of one of your api calls, let me see how it looks like.

[prof7bit]

We can stop worrying about socketio-beta, MT has taken it live. socketio is now the new server and socketio-beta is obsolete and the old server is now socketio-old.mtgox.com

[genuise]

I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

[alpet]

It works with md5?

No, I just use large numbers calculated from system time.

[prof7bit]

I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.

[genuise]

I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.

thank you for clarification

[yucca]

For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.

In this table the least significant bit is on the right, makes it easier to mask and shift.

I cannot say this has no error, I have not tested, so verify by eye against original table first:

EDIT:BELOW TABLE IS WRONG! SCROLL DOWN PAGE FOR FIXED VERSION


   /*-------------------------------------------------------------*\
   | LSB ON RIGHT IN THIS TABLE                                    |
   |  3                   2                   1                    |
   |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
   +-------------------------------+-------------+-+-------+-+-+-+-+
   |    Extended payload length    | Payload len |M| opcode|R|R|R|F|
   |             (16/64)           |     (7)     |A|  (4)  |S|S|S|I|
   |   (if payload len==126/127)   |             |S|       |V|V|V|N|
   |                               |             |K|       |3|2|1| |
   +-------------------------------+-------------+-+-------+-+-+-+-+
   |     Extended payload length continued, if payload len == 127  |
   +-------------------------------+-------------------------------+
   |Masking-key, if MASK set to 1  |                               |
   +-------------------------------+-------------------------------+
   |          Payload Data         | Masking-key (continued)       |
   +-------------------------------+-------------------------------+
   :                     Payload Data continued ...                :
   +---------------------------------------------------------------+
   |                     Payload Data continued ...                |
   +--------------------------------------------------------------*/

      ORIGINAL TABLE: LSB ON LEFT.
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+

[writhe]

For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.

In this table the least significant bit is on the right, makes it easier to mask and shift.

I cannot say this has no error, I have not tested, so verify by eye against original table first:

Your table is incorrect. The original version in RFC 6455 is drawn with the most significant on the left. It's a bizarre numbering scheme used in most RFCs called MSB-0.

It's easier to read the RFC tables a byte at a time. The protocols assume the granularity of data transfer is 8-bit bytes so they don't really have the concept of a multi-bit word. When they do need multi-bit word they usually encode it as multiple 8-bit bytes using big-endian ordering.

For example, the lower 4 bits of the first byte are the opcode and the most significant bit is the FIN flag. Similarly, for the second byte the payload length is the lower 7 bits and the mask flag is the most significant bit.

Assuming little-endian with least significant bit (0) on the right, the first 2 bytes should look like this:

Code:

 1
 5    14 - 8     7 6 5 4  3 - 0
+-+-------------+-+-+-+-+--------+
|M| Payload len |F|R|R|R| opcode |
|A|      (7)    |I|S|S|S|  (4)   |
|S|             |N|V|V|V|        |
|K|             | |1|2|3|        |
+-+-------------+-+-+-+-+--------+

 |-------------| |-------------|
   second byte     first byte

[yucca]

Oh my... thanks muchly writhe!  Slowly but surely, I'll get this stream established.

Here's an ammended table for little endian:

   /*-------------------------------------------------------------*\
   |         LITTLE ENDIAN FORMAT WHERE LSB == 0x00000001          |
   |  3                   2                   1                   0|
   |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
   +-------------------------------+-+-------------+-+-+-+-+-------+
   |    Extended payload length    |M| Payload len |F|R|R|R| opcode|
   |             (16/64)           |A|     (7)     |I|S|S|S|  (4)  |
   |   (if payload len==126/127)   |S|             |N|V|V|V|       |
   |                               |K|             | |1|2|3|       |
   +-------------------------------+-+-------------+-+-+-+-+-------+
   |     Extended payload length continued, if payload len == 127  |
   +-------------------------------+-------------------------------+
   |Masking-key, if MASK set to 1  |                               |
   +-------------------------------+-------------------------------+
   |          Payload Data         | Masking-key (continued)       |
   +-------------------------------+-------------------------------+
   :                     Payload Data continued ...                :
   +---------------------------------------------------------------+
   |                     Payload Data continued ...                |
   \*-------------------------------------------------------------*/

[yucca]

I got the socket.io ticker stream coming in... weeeeee!

Thanks to prof7bit, few satoshis coming your way!!!!

I used a hardcoded packet for first send after socket upgrade ala prof7bit's python debug output.

Code:

	unsigned char pBuffer[15];
	unsigned char* pBuild = pBuffer;
	*pBuild++ = 0x81;
	*pBuild++ = 0x89;
	*pBuild++ = 0x9f;
	*pBuild++ = 0x99;
	*pBuild++ = 0x10;
	*pBuild++ = 0xa1;
	*pBuild++ = 0xae;
	*pBuild++ = 0xa3;
	*pBuild++ =  '*';
	*pBuild++ = 0x8e;
	*pBuild++ = 0xf2;
	*pBuild++ = 0xed;
	*pBuild++ =  'w';
	*pBuild++ = 0xce;
	*pBuild++ = 0xe7;
	Transmit_Raw(pBuffer, 15, false, true);

I cant reply to its echo requests yet because my frame/mask func has gremlins. But the data is coming in well until echo request.

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

[writhe]

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

There are some examples in the RFC that might be useful to check against.

Also, using all zeros for the masking key is valid (you still need to set the masking bit though). This should make it easier to debug as the payload won't be modified by the XOR operation.

[yucca]

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

There are some examples in the RFC that might be useful to check against.

Also, using all zeros for the masking key is valid (you still need to set the masking bit though). This should make it easier to debug as the payload won't be modified by the XOR operation.

Thanks, i missed that, (note to self: READ specs before coding!).

So now i have realised that the payload length bytes are dynamic and are only used if payload size needs them. ahhh so thats why the table has dotted lines for some cell walls. doh!

edit: and thx for the zero mask tip, sure will make things simpler to get going!

[yucca]

here is a C mask/frame function that is tested and working, again thanks guys for the help:

edit: See a few posts on for the func....

[genuise]

sorry, can you please explain what you are actually developing?

[writhe]

here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:

• payload length is < 126 bytes,
• payload length is < 65536 bytes,
• payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).

[yucca]

here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:

• payload length is < 126 bytes,
• payload length is < 65536 bytes,
• payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).

Woops! I only tested with short message so only tested the 7bits version. Will fix that and post code below.

[yucca]

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

[yucca]

Here is the corrected rfc6455 framing/masking func in C.

Corrected, as @writhe said to do payload length correctly:

edit: Corrected AGAIN, thx @writhe!!

Code:

///////////////////////////////////////////////////////////////////////////////////////////////////
//	CREATE A MASKED DATA FRAME FROM THE PASSED BYTES.
//
//	NOTES:
//		This implementation has max payload size of 0xFFFFFFFF == 4 GB
//		v.pucFrameData is a pointer used to read the data from outside this func.
//		v.uiFrameBytes is a variable used to read frame size from outside this func.
//		REF: (http://tools.ietf.org/html/rfc6455#section-5.3)
///////////////////////////////////////////////////////////////////////////////////////////////////
void THR_MtGox::CreateFrameData(unsigned char* pucInput, unsigned int uiByteLen, bool bText)
{
	const bool B_MASK_ENABLED = true;
	const int  MAX_HEADER_BYTES = 14;
	
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	ALLOC NEW MEMORY FOR DERIVED FRAME
	if(v.pucFrameData)
		delete v.pucFrameData;
	v.pucFrameData = new unsigned char[MAX_HEADER_BYTES + v.uiFrameBytes];
	unsigned char* pucOutput = v.pucFrameData;//temporary pointer, can increment to build the frame
	
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	FIRST BYTE FLAGS ("FIN" BIT ALWAYS SET BECAUSE THIS FUNC SENDS A COMPLETE MESSAGE)
	if(bText)				//TEXTUAL PAYLOAD 10000001
		*pucOutput++ = 0x81;
	else					//BINARY PAYLOAD  10000010
		*pucOutput++ = 0x82;

	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	STORE ADDRESS OF BYTE THAT HOLDS MASK BIT
	unsigned char* pucMaskByte = pucOutput;
	
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	PAYLOAD BYTE SIZE:
	//	[1]	payload length is < 126 bytes,
	//		the length can be encoded entirely within the 7 bits of the payload length field.
	if(uiByteLen < 126)
	{
		*pucOutput++ = (unsigned char)uiByteLen;
	}
	//	[2]	payload length is < 65536 bytes,
	//		Payload length field is set to 126, next 2 bytes are payload length (big-endian).
	else if(uiByteLen < 65536)
	{
		*pucOutput++ = 126;
		*pucOutput++ = (unsigned char)(uiByteLen>>8);
		*pucOutput++ = (unsigned char)(uiByteLen   );
	}
	//	[3]	payload length is upto pow(2,64)-1 bytes.
	//		payload length field is set to 127 and the next 8 bytes are the payload length (big-endian).
	else
	{
		*pucOutput++ = 127;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)(uiByteLen>>24);
		*pucOutput++ = (unsigned char)(uiByteLen>>16);
		*pucOutput++ = (unsigned char)(uiByteLen>> 8);
		*pucOutput++ = (unsigned char)(uiByteLen    );
	}
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	MASK ENABLE BIT
	if(B_MASK_ENABLED)
		*pucMaskByte |= 0x80;
	else
		*pucMaskByte &=~0x80;

	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	MASKING KEY
	unsigned int uiMaskingKey = (unsigned int)rand();
	if(B_MASK_ENABLED)
	{
		unsigned int uiMask = uiMaskingKey;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
		uiMask >>= 8;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
		uiMask >>= 8;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
		uiMask >>= 8;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
	}
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	ADD PAYLOAD AND XOR WITH MASK IF NEEDED
	if(B_MASK_ENABLED)
	{
		for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
		{
			int iMaskOctet = uiByte%4;
			unsigned char ucMaskOctet = (unsigned char)(uiMaskingKey >>(iMaskOctet*8) & 0xFF);
			*pucOutput = *pucInput ^ ucMaskOctet;
			pucOutput++;
			pucInput++;
		}
	}
	else
	{
		for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
		{
			*pucOutput = *pucInput;
			pucOutput++;
			pucInput++;
		}
	}
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	CALCULATE RESULTING FRAME LENGTH BYTES
	v.uiFrameBytes = pucOutput - v.pucFrameData;
}