Bitcoin Forum
December 08, 2016, 02:19:57 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [17] 18 19 20 21 »  All
  Print  
Author Topic: [BETA] MTGox websocket API, testers wanted  (Read 72248 times)
yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 27, 2013, 10:34:48 PM
 #321

here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:
  • payload length is < 126 bytes,
  • payload length is < 65536 bytes,
  • payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).

Woops! I only tested with short message so only tested the 7bits version. Will fix that and post code below.

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481206797
Hero Member
*
Offline Offline

Posts: 1481206797

View Profile Personal Message (Offline)

Ignore
1481206797
Reply with quote  #2

1481206797
Report to moderator
1481206797
Hero Member
*
Offline Offline

Posts: 1481206797

View Profile Personal Message (Offline)

Ignore
1481206797
Reply with quote  #2

1481206797
Report to moderator
yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 27, 2013, 10:41:25 PM
 #322

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 28, 2013, 12:09:03 AM
 #323

Here is the corrected rfc6455 framing/masking func in C.

Corrected, as @writhe said to do payload length correctly:

edit: Corrected AGAIN, thx @writhe!!

Code:
///////////////////////////////////////////////////////////////////////////////////////////////////
// CREATE A MASKED DATA FRAME FROM THE PASSED BYTES.
//
// NOTES:
// This implementation has max payload size of 0xFFFFFFFF == 4 GB
// v.pucFrameData is a pointer used to read the data from outside this func.
// v.uiFrameBytes is a variable used to read frame size from outside this func.
// REF: (http://tools.ietf.org/html/rfc6455#section-5.3)
///////////////////////////////////////////////////////////////////////////////////////////////////
void THR_MtGox::CreateFrameData(unsigned char* pucInput, unsigned int uiByteLen, bool bText)
{
const bool B_MASK_ENABLED = true;
const int  MAX_HEADER_BYTES = 14;

///////////////////////////////////////////////////////////////////////////////////////////////////
// ALLOC NEW MEMORY FOR DERIVED FRAME
if(v.pucFrameData)
delete v.pucFrameData;
v.pucFrameData = new unsigned char[MAX_HEADER_BYTES + v.uiFrameBytes];
unsigned char* pucOutput = v.pucFrameData;//temporary pointer, can increment to build the frame

///////////////////////////////////////////////////////////////////////////////////////////////////
// FIRST BYTE FLAGS ("FIN" BIT ALWAYS SET BECAUSE THIS FUNC SENDS A COMPLETE MESSAGE)
if(bText) //TEXTUAL PAYLOAD 10000001
*pucOutput++ = 0x81;
else //BINARY PAYLOAD  10000010
*pucOutput++ = 0x82;

///////////////////////////////////////////////////////////////////////////////////////////////////
// STORE ADDRESS OF BYTE THAT HOLDS MASK BIT
unsigned char* pucMaskByte = pucOutput;

///////////////////////////////////////////////////////////////////////////////////////////////////
// PAYLOAD BYTE SIZE:
// [1] payload length is < 126 bytes,
// the length can be encoded entirely within the 7 bits of the payload length field.
if(uiByteLen < 126)
{
*pucOutput++ = (unsigned char)uiByteLen;
}
// [2] payload length is < 65536 bytes,
// Payload length field is set to 126, next 2 bytes are payload length (big-endian).
else if(uiByteLen < 65536)
{
*pucOutput++ = 126;
*pucOutput++ = (unsigned char)(uiByteLen>>8);
*pucOutput++ = (unsigned char)(uiByteLen   );
}
// [3] payload length is upto pow(2,64)-1 bytes.
// payload length field is set to 127 and the next 8 bytes are the payload length (big-endian).
else
{
*pucOutput++ = 127;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// MASK ENABLE BIT
if(B_MASK_ENABLED)
*pucMaskByte |= 0x80;
else
*pucMaskByte &=~0x80;

///////////////////////////////////////////////////////////////////////////////////////////////////
// MASKING KEY
unsigned int uiMaskingKey = (unsigned int)rand();
if(B_MASK_ENABLED)
{
unsigned int uiMask = uiMaskingKey;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
uiMask >>= 8;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
uiMask >>= 8;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
uiMask >>= 8;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// ADD PAYLOAD AND XOR WITH MASK IF NEEDED
if(B_MASK_ENABLED)
{
for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
{
int iMaskOctet = uiByte%4;
unsigned char ucMaskOctet = (unsigned char)(uiMaskingKey >>(iMaskOctet*8) & 0xFF);
*pucOutput = *pucInput ^ ucMaskOctet;
pucOutput++;
pucInput++;
}
}
else
{
for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
{
*pucOutput = *pucInput;
pucOutput++;
pucInput++;
}
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// CALCULATE RESULTING FRAME LENGTH BYTES
v.uiFrameBytes = pucOutput - v.pucFrameData;
}

writhe
Newbie
*
Offline Offline

Activity: 17


View Profile
March 28, 2013, 12:12:08 AM
 #324

I think you mean:
Code:
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
instead of:
Code:
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;

 Grin
yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 28, 2013, 12:18:55 AM
 #325

I think you mean:
Code:
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
instead of:
Code:
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;

 Grin

you know... i think you're right!  Embarrassed Cheesy

corrected above codepaste (again Roll Eyes ).

Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
March 28, 2013, 04:32:46 AM
 #326

Here is the corrected rfc6455 framing/masking func in C.

C code! Things got serious in here...  Grin

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
genuise
Sr. Member
****
Offline Offline

Activity: 375


View Profile WWW
March 28, 2013, 05:38:36 AM
 #327

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

So you implement socket.io-client in C? And there was no any existing implementation already in the net?

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 28, 2013, 12:57:39 PM
 #328

sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

So you implement socket.io-client in C? And there was no any existing implementation already in the net?

There seems no existing implementation in C or C++. Closest is objective C. But I chose only to reference only RFC6455 for frame/mask.

I referenced prof7bits python code for the handshake.

I already had socket working in my codebase for mtgox websocket, but as you know its not a reliable data stream.

I just had to add a frame/mask func per RFC6455, this was the only really tricky bit, and I was happy to receive help here with that.

I was using an asynchronous socket with message callbacks, things became alot simpler when I switched to blocking socket. I over-ride base class so it also has timeout function, then I can reset connection if line goes quiet for 1 minute.

I've also coded a JSON parser, not to full spec, it just finds a val string by braching into braces according to passed key path.

genuise
Sr. Member
****
Offline Offline

Activity: 375


View Profile WWW
March 28, 2013, 01:00:34 PM
 #329

great work I think!

any plans to publish your socket.io-client implementation on github for the benefit the public?

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 28, 2013, 04:25:43 PM
 #330

great work I think!

any palns to publish your socket.io-client implementation on github for the benefit the public?

thanks. with any vanilla socket you just need to use the posted framing func for sending data after upgrade. that's the trickiest part.

that func could even be used to make an arduino into a socket.io client with good performance.

i have a standalone visual studio project (threaded, well encapsulated, no lib dependencies, just MFC) that has just the socket.io logging the json stream to a window if anyones interested.

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 28, 2013, 04:44:32 PM
 #331

Here is the corrected rfc6455 framing/masking func in C.

C code! Things got serious in here...  Grin

lol, now someone do an optimised X86 assemler version.

Great site you made by the way Clark! Smiley

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 29, 2013, 01:57:22 AM
 #332

I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

both "tid" and "date" entries have this wrong time?

prof7bit
Hero Member
*****
Offline Offline

Activity: 672



View Profile WWW
March 29, 2013, 05:12:16 PM
 #333

I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

How is the bitcoin price next week? Is it above 100 already?
I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh Wink

Old Bitcoins recycling and disposal: 1N7ZcHoe77JncRYAg27RVqJZbrjqUQ77WE
Blockchain.info -> Multibit converter (and much more) https://github.com/prof7bit/wallet-key-tool
genuise
Sr. Member
****
Offline Offline

Activity: 375


View Profile WWW
March 29, 2013, 07:41:27 PM
 #334

I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

How is the bitcoin price next week? Is it above 100 already?
I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh Wink

It is just hilarious Smiley

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 30, 2013, 12:38:36 PM
 #335

LOL, i've routed into a temporal rift, i will keep this a secret, future prices are good!!! Grin

But seriously guys the JSON timestamps are wrong, or i'm just not understanding what that time is for???

Nothing wrong at my RX end as server sends non masked frames, just plain text blocks between delimiters.

TRADE and DEPTH json stream:


{"channel":"dbf1dee9-f2e-a08-8cb7-78919a71b21","op":"private","origin":"broadcast","private":
"trade","trade":{"type":"trade","date":1366631,"amount":2,"price":91.00021,"tid":"136663155159",
"amount_int":"200000000","price_int":"9100021","item":"BTC","price_currency":"USD","trade_type":"bid",
"primary":"Y","properties":"market"}}~4::/mtgox:{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe",
"op":"private","origin":"broadcast","private":"depth",
"depth":{"price":"91.00021","type":1,"type_str":"ask","volume":"0","price_int":"9100021","volume_int":"0",
"item":"BTC","currency":"USD","now":"136663158281","total_volume_int":"3016823508"}}

{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe","op":"private","origin":"broadcast","private":
"depth","depth":{"price":"87.5001","type":2,"type_str":"bid","volume":"0","price_int":"8750010",
"volume_int":"0","item":"BTC","currency":"USD","now":"136663161999","total_volume_int":"136753855"}}



trade has "now":"136663158281"  === Mon, 22 Apr 2013 11:53:02 GMT

depth has "now":"136663161999" === Mon, 22 Apr 2013 11:53:39 GMT

if you dont trust my unixtime to datetime conversion then plug them into an online converter (using the 10 leftmost bolded characters):
http://www.onlineconversion.com/unix_time.htm

so what gives, does nobody else use this data, do I have to discard timestamps and stamp them myself using local clock? that's not so good!

prof7bit
Hero Member
*****
Offline Offline

Activity: 672



View Profile WWW
March 30, 2013, 05:22:04 PM
 #336

This is strange.

My timestamps as received here are correct. Below are some messages I have received here from socketio.mtgox.com a few minutes not long ago, the timestamps look normal.

Also the trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart. I could not see any mismatch between the timestamps of "money/trades" and new incoming trade messages.

The first thing that catches the eye is that in your trade message the date field is only 7 digits long while mine is a complete timestamp (in seconds, 10 digits) and also your microseconds timestamps seem too short (12 digits instead of 16), how can this happen? Something eating characters and always the *same* sequence of characters? Btw I cannot see the character "4" appearing anywhere in your dump (except for the 4::/mtgox (!)) and also some of the GUID strings are too short!

Below are the mesages as received by me (and pretty-printed/formatted by json.dumps())

trade
Code:
{
     "origin": "broadcast",
     "trade": {
          "price_currency": "USD",
          "trade_type": "bid",
          "price_int": "9120000",
          "item": "BTC",
          "price": 91.2,
          "primary": "Y",
          "tid": "1364662801201914",
          "amount": 1.41528958,
          "amount_int": "141528958",
          "date": 1364662801,
          "type": "trade",
          "properties": "limit"
     },
     "private": "trade",
     "channel": "dbf1dee9-4f2e-4a08-8cb7-748919a71b21",
     "op": "private"
}

Depth
Code:
{
     "origin": "broadcast",
     "depth": {
          "volume_int": "0",
          "price_int": "9120000",
          "currency": "USD",
          "price": "91.2",
          "volume": "0",
          "item": "BTC",
          "type_str": "ask",
          "total_volume_int": "9828724331",
          "now": "1364662801229415",
          "type": 1
     },
     "private": "depth",
     "channel": "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe",
     "op": "private"
}



Ticker
Code:
{
     "origin": "broadcast",
     "ticker": {
          "sell": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "last_orig": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "now": "1364662937327085",
          "buy": {
               "value_int": "9119999",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.19999",
               "value": "91.19999"
          },
          "last": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "vol": {
               "value_int": "5095833923444",
               "currency": "BTC",
               "display_short": "50,958.34\u00a0BTC",
               "display": "50,958.33923444\u00a0BTC",
               "value": "50958.33923444"
          },
          "last_local": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "last_all": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "vwap": {
               "value_int": "9064567",
               "currency": "USD",
               "display_short": "$90.65",
               "display": "$90.64567",
               "value": "90.64567"
          },
          "high": {
               "value_int": "9498000",
               "currency": "USD",
               "display_short": "$94.98",
               "display": "$94.98000",
               "value": "94.98000"
          },
          "low": {
               "value_int": "8550000",
               "currency": "USD",
               "display_short": "$85.50",
               "display": "$85.50000",
               "value": "85.50000"
          },
          "avg": {
               "value_int": "9037305",
               "currency": "USD",
               "display_short": "$90.37",
               "display": "$90.37305",
               "value": "90.37305"
          }
     },
     "private": "ticker",
     "channel": "d5f06780-30a8-4a48-a2f8-7ed181b4a13f",
     "op": "private"
}

Lag
Code:
{
     "origin": "broadcast",
     "lag": {
          "stamp": "1364663115576119",
          "qid": "74655212-1bc0-4b69-95da-79e40f4f771f",
          "age": 329545
     },
     "private": "lag",
     "channel": "85174711-be64-4de1-b783-0628995d7914",
     "op": "private"
}

Old Bitcoins recycling and disposal: 1N7ZcHoe77JncRYAg27RVqJZbrjqUQ77WE
Blockchain.info -> Multibit converter (and much more) https://github.com/prof7bit/wallet-key-tool
yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 30, 2013, 06:05:25 PM
 #337

thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

Then I will subscribe to the new streams, so i will try subscribing to new streams anyway and see how it goes, i guess i just send a masked/framed subscription request as per the mtgox stream api wiki.

So perhaps i will just have to ignore the junk stream, it will still have to flow thru my parser though, i dislike wasted clock cycles, plus it loads the server.

prof7bit
Hero Member
*****
Offline Offline

Activity: 672



View Profile WWW
March 30, 2013, 06:12:00 PM
 #338

thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

No, I am using them too.

But all your "4" are missing! Are you trying to parse the "4::/mtgox:" by any chance and have an error there in your parser? In your previous post the "4::/mtgox:" is appearing between the json objects and is the only thing that has a "4" in it, all other "4" seem to have gone missing!

Old Bitcoins recycling and disposal: 1N7ZcHoe77JncRYAg27RVqJZbrjqUQ77WE
Blockchain.info -> Multibit converter (and much more) https://github.com/prof7bit/wallet-key-tool
yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 30, 2013, 09:06:03 PM
 #339

you are correct prof7bit, my parsing state machine is buggy causing 4s to get dropped Embarrassed.

I'm doing it with inc pointer on the raw incoming data buffer for speed and screwed up. I will rework it now. thanks man!

yucca
Full Member
***
Offline Offline

Activity: 217



View Profile WWW
March 31, 2013, 02:52:09 AM
 #340

...trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart....

cool, similar to what i'm doing.

It would be nice to have a mtgox call similar to bitcoincharts, so one could specify to get all trades since X unixtime.

or even better binned trades in choice of bin size, M15, M30 etc. this would really save some bandwidth for mtgox!

P.S.
are you plotting volume also? if so do you find the live numbers sum closely to the history?

Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [17] 18 19 20 21 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!