[BETA] MTGox websocket API, testers wanted

[yucca]

here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:

• payload length is < 126 bytes,
• payload length is < 65536 bytes,
• payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).

Woops! I only tested with short message so only tested the 7bits version. Will fix that and post code below.

[1715168247]

1715168247

[1715168247]

1715168247

[1715168247]

1715168247

[1715168247]

1715168247

[yucca]

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

[yucca]

Here is the corrected rfc6455 framing/masking func in C.

Corrected, as @writhe said to do payload length correctly:

edit: Corrected AGAIN, thx @writhe!!

Code:

///////////////////////////////////////////////////////////////////////////////////////////////////
//	CREATE A MASKED DATA FRAME FROM THE PASSED BYTES.
//
//	NOTES:
//		This implementation has max payload size of 0xFFFFFFFF == 4 GB
//		v.pucFrameData is a pointer used to read the data from outside this func.
//		v.uiFrameBytes is a variable used to read frame size from outside this func.
//		REF: (http://tools.ietf.org/html/rfc6455#section-5.3)
///////////////////////////////////////////////////////////////////////////////////////////////////
void THR_MtGox::CreateFrameData(unsigned char* pucInput, unsigned int uiByteLen, bool bText)
{
	const bool B_MASK_ENABLED = true;
	const int  MAX_HEADER_BYTES = 14;
	
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	ALLOC NEW MEMORY FOR DERIVED FRAME
	if(v.pucFrameData)
		delete v.pucFrameData;
	v.pucFrameData = new unsigned char[MAX_HEADER_BYTES + v.uiFrameBytes];
	unsigned char* pucOutput = v.pucFrameData;//temporary pointer, can increment to build the frame
	
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	FIRST BYTE FLAGS ("FIN" BIT ALWAYS SET BECAUSE THIS FUNC SENDS A COMPLETE MESSAGE)
	if(bText)				//TEXTUAL PAYLOAD 10000001
		*pucOutput++ = 0x81;
	else					//BINARY PAYLOAD  10000010
		*pucOutput++ = 0x82;

	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	STORE ADDRESS OF BYTE THAT HOLDS MASK BIT
	unsigned char* pucMaskByte = pucOutput;
	
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	PAYLOAD BYTE SIZE:
	//	[1]	payload length is < 126 bytes,
	//		the length can be encoded entirely within the 7 bits of the payload length field.
	if(uiByteLen < 126)
	{
		*pucOutput++ = (unsigned char)uiByteLen;
	}
	//	[2]	payload length is < 65536 bytes,
	//		Payload length field is set to 126, next 2 bytes are payload length (big-endian).
	else if(uiByteLen < 65536)
	{
		*pucOutput++ = 126;
		*pucOutput++ = (unsigned char)(uiByteLen>>8);
		*pucOutput++ = (unsigned char)(uiByteLen   );
	}
	//	[3]	payload length is upto pow(2,64)-1 bytes.
	//		payload length field is set to 127 and the next 8 bytes are the payload length (big-endian).
	else
	{
		*pucOutput++ = 127;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)(uiByteLen>>24);
		*pucOutput++ = (unsigned char)(uiByteLen>>16);
		*pucOutput++ = (unsigned char)(uiByteLen>> 8);
		*pucOutput++ = (unsigned char)(uiByteLen    );
	}
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	MASK ENABLE BIT
	if(B_MASK_ENABLED)
		*pucMaskByte |= 0x80;
	else
		*pucMaskByte &=~0x80;

	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	MASKING KEY
	unsigned int uiMaskingKey = (unsigned int)rand();
	if(B_MASK_ENABLED)
	{
		unsigned int uiMask = uiMaskingKey;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
		uiMask >>= 8;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
		uiMask >>= 8;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
		uiMask >>= 8;
		*pucOutput++ = (unsigned char)(uiMask & 0xFF);
	}
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	ADD PAYLOAD AND XOR WITH MASK IF NEEDED
	if(B_MASK_ENABLED)
	{
		for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
		{
			int iMaskOctet = uiByte%4;
			unsigned char ucMaskOctet = (unsigned char)(uiMaskingKey >>(iMaskOctet*8) & 0xFF);
			*pucOutput = *pucInput ^ ucMaskOctet;
			pucOutput++;
			pucInput++;
		}
	}
	else
	{
		for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
		{
			*pucOutput = *pucInput;
			pucOutput++;
			pucInput++;
		}
	}
	///////////////////////////////////////////////////////////////////////////////////////////////////
	//	CALCULATE RESULTING FRAME LENGTH BYTES
	v.uiFrameBytes = pucOutput - v.pucFrameData;
}

[writhe]

I think you mean:

Code:

		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)(uiByteLen>>24);
		*pucOutput++ = (unsigned char)(uiByteLen>>16);
		*pucOutput++ = (unsigned char)(uiByteLen>> 8);
		*pucOutput++ = (unsigned char)(uiByteLen    );

instead of:

Code:

		*pucOutput++ = (unsigned char)(uiByteLen>>24);
		*pucOutput++ = (unsigned char)(uiByteLen>>16);
		*pucOutput++ = (unsigned char)(uiByteLen>> 8);
		*pucOutput++ = (unsigned char)(uiByteLen    );
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;

 

[yucca]

I think you mean:

Code:

		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)(uiByteLen>>24);
		*pucOutput++ = (unsigned char)(uiByteLen>>16);
		*pucOutput++ = (unsigned char)(uiByteLen>> 8);
		*pucOutput++ = (unsigned char)(uiByteLen    );

instead of:

Code:

		*pucOutput++ = (unsigned char)(uiByteLen>>24);
		*pucOutput++ = (unsigned char)(uiByteLen>>16);
		*pucOutput++ = (unsigned char)(uiByteLen>> 8);
		*pucOutput++ = (unsigned char)(uiByteLen    );
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;
		*pucOutput++ = (unsigned char)0x00;

 

you know... i think you're right! 

corrected above codepaste (again ).

[Clark]

Here is the corrected rfc6455 framing/masking func in C.

C code! Things got serious in here... 

[genuise]

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

So you implement socket.io-client in C? And there was no any existing implementation already in the net?

[yucca]

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

So you implement socket.io-client in C? And there was no any existing implementation already in the net?

There seems no existing implementation in C or C++. Closest is objective C. But I chose only to reference only RFC6455 for frame/mask.

I referenced prof7bits python code for the handshake.

I already had socket working in my codebase for mtgox websocket, but as you know its not a reliable data stream.

I just had to add a frame/mask func per RFC6455, this was the only really tricky bit, and I was happy to receive help here with that.

I was using an asynchronous socket with message callbacks, things became alot simpler when I switched to blocking socket. I over-ride base class so it also has timeout function, then I can reset connection if line goes quiet for 1 minute.

I've also coded a JSON parser, not to full spec, it just finds a val string by braching into braces according to passed key path.

[genuise]

great work I think!

any plans to publish your socket.io-client implementation on github for the benefit the public?

[yucca]

great work I think!

any palns to publish your socket.io-client implementation on github for the benefit the public?

thanks. with any vanilla socket you just need to use the posted framing func for sending data after upgrade. that's the trickiest part.

that func could even be used to make an arduino into a socket.io client with good performance.

i have a standalone visual studio project (threaded, well encapsulated, no lib dependencies, just MFC) that has just the socket.io logging the json stream to a window if anyones interested.

[yucca]

Here is the corrected rfc6455 framing/masking func in C.

C code! Things got serious in here... 

lol, now someone do an optimised X86 assemler version.

Great site you made by the way Clark!

[yucca]

I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

both "tid" and "date" entries have this wrong time?

[prof7bit]

I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

How is the bitcoin price next week? Is it above 100 already?
I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh 

[genuise]

I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

How is the bitcoin price next week? Is it above 100 already?
I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh 

It is just hilarious

[yucca]

LOL, i've routed into a temporal rift, i will keep this a secret, future prices are good!!!

But seriously guys the JSON timestamps are wrong, or i'm just not understanding what that time is for???

Nothing wrong at my RX end as server sends non masked frames, just plain text blocks between delimiters.

TRADE and DEPTH json stream:


{"channel":"dbf1dee9-f2e-a08-8cb7-78919a71b21","op":"private","origin":"broadcast","private":
"trade","trade":{"type":"trade","date":1366631,"amount":2,"price":91.00021,"tid":"136663155159",
"amount_int":"200000000","price_int":"9100021","item":"BTC","price_currency":"USD","trade_type":"bid",
"primary":"Y","properties":"market"}}�~
4::/mtgox:{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe",
"op":"private","origin":"broadcast","private":"depth",
"depth":{"price":"91.00021","type":1,"type_str":"ask","volume":"0","price_int":"9100021","volume_int":"0",
"item":"BTC","currency":"USD","now":"136663158281","total_volume_int":"3016823508"}}�

{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe","op":"private","origin":"broadcast","private":
"depth","depth":{"price":"87.5001","type":2,"type_str":"bid","volume":"0","price_int":"8750010",
"volume_int":"0","item":"BTC","currency":"USD","now":"136663161999","total_volume_int":"136753855"}}
�


trade has "now":"136663158281"  === Mon, 22 Apr 2013 11:53:02 GMT

depth has "now":"136663161999" === Mon, 22 Apr 2013 11:53:39 GMT

if you dont trust my unixtime to datetime conversion then plug them into an online converter (using the 10 leftmost bolded characters):
http://www.onlineconversion.com/unix_time.htm

so what gives, does nobody else use this data, do I have to discard timestamps and stamp them myself using local clock? that's not so good!

[prof7bit]

This is strange.

My timestamps as received here are correct. Below are some messages I have received here from socketio.mtgox.com a few minutes not long ago, the timestamps look normal.

Also the trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart. I could not see any mismatch between the timestamps of "money/trades" and new incoming trade messages.

The first thing that catches the eye is that in your trade message the date field is only 7 digits long while mine is a complete timestamp (in seconds, 10 digits) and also your microseconds timestamps seem too short (12 digits instead of 16), how can this happen? Something eating characters and always the *same* sequence of characters? Btw I cannot see the character "4" appearing anywhere in your dump (except for the 4::/mtgox (!)) and also some of the GUID strings are too short!

Below are the mesages as received by me (and pretty-printed/formatted by json.dumps())

trade

Code:

{
     "origin": "broadcast", 
     "trade": {
          "price_currency": "USD", 
          "trade_type": "bid", 
          "price_int": "9120000", 
          "item": "BTC", 
          "price": 91.2, 
          "primary": "Y", 
          "tid": "1364662801201914", 
          "amount": 1.41528958, 
          "amount_int": "141528958", 
          "date": 1364662801, 
          "type": "trade", 
          "properties": "limit"
     }, 
     "private": "trade", 
     "channel": "dbf1dee9-4f2e-4a08-8cb7-748919a71b21", 
     "op": "private"
}

Depth

Code:

{
     "origin": "broadcast", 
     "depth": {
          "volume_int": "0", 
          "price_int": "9120000", 
          "currency": "USD", 
          "price": "91.2", 
          "volume": "0", 
          "item": "BTC", 
          "type_str": "ask", 
          "total_volume_int": "9828724331", 
          "now": "1364662801229415", 
          "type": 1
     }, 
     "private": "depth", 
     "channel": "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe", 
     "op": "private"
}

Ticker

Code:

{
     "origin": "broadcast", 
     "ticker": {
          "sell": {
               "value_int": "9120000", 
               "currency": "USD", 
               "display_short": "$91.20", 
               "display": "$91.20000", 
               "value": "91.20000"
          }, 
          "last_orig": {
               "value_int": "9120000", 
               "currency": "USD", 
               "display_short": "$91.20", 
               "display": "$91.20000", 
               "value": "91.20000"
          }, 
          "now": "1364662937327085", 
          "buy": {
               "value_int": "9119999", 
               "currency": "USD", 
               "display_short": "$91.20", 
               "display": "$91.19999", 
               "value": "91.19999"
          }, 
          "last": {
               "value_int": "9120000", 
               "currency": "USD", 
               "display_short": "$91.20", 
               "display": "$91.20000", 
               "value": "91.20000"
          }, 
          "vol": {
               "value_int": "5095833923444", 
               "currency": "BTC", 
               "display_short": "50,958.34\u00a0BTC", 
               "display": "50,958.33923444\u00a0BTC", 
               "value": "50958.33923444"
          }, 
          "last_local": {
               "value_int": "9120000", 
               "currency": "USD", 
               "display_short": "$91.20", 
               "display": "$91.20000", 
               "value": "91.20000"
          }, 
          "last_all": {
               "value_int": "9120000", 
               "currency": "USD", 
               "display_short": "$91.20", 
               "display": "$91.20000", 
               "value": "91.20000"
          }, 
          "vwap": {
               "value_int": "9064567", 
               "currency": "USD", 
               "display_short": "$90.65", 
               "display": "$90.64567", 
               "value": "90.64567"
          }, 
          "high": {
               "value_int": "9498000", 
               "currency": "USD", 
               "display_short": "$94.98", 
               "display": "$94.98000", 
               "value": "94.98000"
          }, 
          "low": {
               "value_int": "8550000", 
               "currency": "USD", 
               "display_short": "$85.50", 
               "display": "$85.50000", 
               "value": "85.50000"
          }, 
          "avg": {
               "value_int": "9037305", 
               "currency": "USD", 
               "display_short": "$90.37", 
               "display": "$90.37305", 
               "value": "90.37305"
          }
     }, 
     "private": "ticker", 
     "channel": "d5f06780-30a8-4a48-a2f8-7ed181b4a13f", 
     "op": "private"
}

Lag

Code:

{
     "origin": "broadcast", 
     "lag": {
          "stamp": "1364663115576119", 
          "qid": "74655212-1bc0-4b69-95da-79e40f4f771f", 
          "age": 329545
     }, 
     "private": "lag", 
     "channel": "85174711-be64-4de1-b783-0628995d7914", 
     "op": "private"
}

[yucca]

thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

Then I will subscribe to the new streams, so i will try subscribing to new streams anyway and see how it goes, i guess i just send a masked/framed subscription request as per the mtgox stream api wiki.

So perhaps i will just have to ignore the junk stream, it will still have to flow thru my parser though, i dislike wasted clock cycles, plus it loads the server.

[prof7bit]

thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

No, I am using them too.

But all your "4" are missing! Are you trying to parse the "4::/mtgox:" by any chance and have an error there in your parser? In your previous post the "4::/mtgox:" is appearing between the json objects and is the only thing that has a "4" in it, all other "4" seem to have gone missing!

[yucca]

you are correct prof7bit, my parsing state machine is buggy causing 4s to get dropped .

I'm doing it with inc pointer on the raw incoming data buffer for speed and screwed up. I will rework it now. thanks man!

[yucca]

...trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart....

cool, similar to what i'm doing.

It would be nice to have a mtgox call similar to bitcoincharts, so one could specify to get all trades since X unixtime.

or even better binned trades in choice of bin size, M15, M30 etc. this would really save some bandwidth for mtgox!

P.S.
are you plotting volume also? if so do you find the live numbers sum closely to the history?