writhe
Newbie
 Offline
Activity: 17
Merit: 0
|
 |
March 28, 2013, 12:12:08 AM |
|
I think you mean: *pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen );
instead of: *pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen );
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
 |
|
|
|
|
|
yucca
|
|
March 28, 2013, 12:18:55 AM |
|
I think you mean: *pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen );
instead of: *pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen );
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
you know... i think you're right!   corrected above codepaste (again  ). |
|
|
|
|
Clark
|
|
March 28, 2013, 04:32:46 AM |
|
Here is the corrected rfc6455 framing/masking func in C.
C code! Things got serious in here... |
|
|
|
|
genuise
|
|
March 28, 2013, 05:38:36 AM |
|
So you implement socket.io-client in C? And there was no any existing implementation already in the net? |
|
|
|
|
yucca
|
|
March 28, 2013, 12:57:39 PM |
|
So you implement socket.io-client in C? And there was no any existing implementation already in the net? There seems no existing implementation in C or C++. Closest is objective C. But I chose only to reference only RFC6455 for frame/mask. I referenced prof7bits python code for the handshake. I already had socket working in my codebase for mtgox websocket, but as you know its not a reliable data stream. I just had to add a frame/mask func per RFC6455, this was the only really tricky bit, and I was happy to receive help here with that. I was using an asynchronous socket with message callbacks, things became alot simpler when I switched to blocking socket. I over-ride base class so it also has timeout function, then I can reset connection if line goes quiet for 1 minute. I've also coded a JSON parser, not to full spec, it just finds a val string by braching into braces according to passed key path. |
|
|
|
|
genuise
|
|
March 28, 2013, 01:00:34 PM
Last edit: March 28, 2013, 06:48:47 PM by genuise |
|
great work I think!
any plans to publish your socket.io-client implementation on github for the benefit the public?
|
|
|
|
|
yucca
|
|
March 28, 2013, 04:25:43 PM
Last edit: March 28, 2013, 04:42:01 PM by yucca |
|
great work I think!
any palns to publish your socket.io-client implementation on github for the benefit the public?
thanks. with any vanilla socket you just need to use the posted framing func for sending data after upgrade. that's the trickiest part. that func could even be used to make an arduino into a socket.io client with good performance. i have a standalone visual studio project (threaded, well encapsulated, no lib dependencies, just MFC) that has just the socket.io logging the json stream to a window if anyones interested. |
|
|
|
|
yucca
|
|
March 28, 2013, 04:44:32 PM |
|
Here is the corrected rfc6455 framing/masking func in C.
C code! Things got serious in here... lol, now someone do an optimised X86 assemler version. Great site you made by the way Clark!  |
|
|
|
|
yucca
|
|
March 29, 2013, 01:57:22 AM
Last edit: March 29, 2013, 02:26:26 AM by yucca |
|
I seem to be receiving trade tickers with following unix timestamp:
1365219272
which is a future date:
2013-APRIL-06 04:34:32 UTC
both "tid" and "date" entries have this wrong time?
|
|
|
|
|
prof7bit
|
|
March 29, 2013, 05:12:16 PM |
|
I seem to be receiving trade tickers with following unix timestamp:
1365219272
which is a future date:
2013-APRIL-06 04:34:32 UTC
How is the bitcoin price next week? Is it above 100 already? I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh  |
|
|
|
|
genuise
|
|
March 29, 2013, 07:41:27 PM |
|
I seem to be receiving trade tickers with following unix timestamp:
1365219272
which is a future date:
2013-APRIL-06 04:34:32 UTC
How is the bitcoin price next week? Is it above 100 already? I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh It is just hilarious |
|
|
|
|
yucca
|
|
March 30, 2013, 12:38:36 PM |
|
LOL, i've routed into a temporal rift, i will keep this a secret, future prices are good!!! But seriously guys the JSON timestamps are wrong, or i'm just not understanding what that time is for??? Nothing wrong at my RX end as server sends non masked frames, just plain text blocks between delimiters. TRADE and DEPTH json stream:
{"channel":"dbf1dee9-f2e-a08-8cb7-78919a71b21","op":"private","origin":"broadcast","private":
"trade","trade":{"type":"trade","date":1366631,"amount":2,"price":91.00021,"tid":"136663155159",
"amount_int":"200000000","price_int":"9100021","item":"BTC","price_currency":"USD","trade_type":"bid",
"primary":"Y","properties":"market"}}~4::/mtgox:{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe",
"op":"private","origin":"broadcast","private":"depth",
"depth":{"price":"91.00021","type":1,"type_str":"ask","volume":"0","price_int":"9100021","volume_int":"0",
"item":"BTC","currency":"USD","now":"136663158281","total_volume_int":"3016823508"}}
{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe","op":"private","origin":"broadcast","private":
"depth","depth":{"price":"87.5001","type":2,"type_str":"bid","volume":"0","price_int":"8750010",
"volume_int":"0","item":"BTC","currency":"USD","now":"136663161999","total_volume_int":"136753855"}}
trade has "now":" 136663158281" === Mon, 22 Apr 2013 11:53:02 GMT depth has "now":" 136663161999" === Mon, 22 Apr 2013 11:53:39 GMT if you dont trust my unixtime to datetime conversion then plug them into an online converter (using the 10 leftmost bolded characters): http://www.onlineconversion.com/unix_time.htmso what gives, does nobody else use this data, do I have to discard timestamps and stamp them myself using local clock? that's not so good! |
|
|
|
|
prof7bit
|
|
March 30, 2013, 05:22:04 PM
Last edit: March 30, 2013, 06:02:05 PM by prof7bit |
|
This is strange. My timestamps as received here are correct. Below are some messages I have received here from socketio.mtgox.com a few minutes not long ago, the timestamps look normal. Also the trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart. I could not see any mismatch between the timestamps of "money/trades" and new incoming trade messages. The first thing that catches the eye is that in your trade message the date field is only 7 digits long while mine is a complete timestamp (in seconds, 10 digits) and also your microseconds timestamps seem too short (12 digits instead of 16), how can this happen? Something eating characters and always the *same* sequence of characters? Btw I cannot see the character "4" appearing anywhere in your dump (except for the 4::/mtgox (!)) and also some of the GUID strings are too short! Below are the mesages as received by me (and pretty-printed/formatted by json.dumps()) trade {
"origin": "broadcast",
"trade": {
"price_currency": "USD",
"trade_type": "bid",
"price_int": "9120000",
"item": "BTC",
"price": 91.2,
"primary": "Y",
"tid": "1364662801201914",
"amount": 1.41528958,
"amount_int": "141528958",
"date": 1364662801,
"type": "trade",
"properties": "limit"
},
"private": "trade",
"channel": "dbf1dee9-4f2e-4a08-8cb7-748919a71b21",
"op": "private"
}
Depth {
"origin": "broadcast",
"depth": {
"volume_int": "0",
"price_int": "9120000",
"currency": "USD",
"price": "91.2",
"volume": "0",
"item": "BTC",
"type_str": "ask",
"total_volume_int": "9828724331",
"now": "1364662801229415",
"type": 1
},
"private": "depth",
"channel": "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe",
"op": "private"
}
Ticker {
"origin": "broadcast",
"ticker": {
"sell": {
"value_int": "9120000",
"currency": "USD",
"display_short": "$91.20",
"display": "$91.20000",
"value": "91.20000"
},
"last_orig": {
"value_int": "9120000",
"currency": "USD",
"display_short": "$91.20",
"display": "$91.20000",
"value": "91.20000"
},
"now": "1364662937327085",
"buy": {
"value_int": "9119999",
"currency": "USD",
"display_short": "$91.20",
"display": "$91.19999",
"value": "91.19999"
},
"last": {
"value_int": "9120000",
"currency": "USD",
"display_short": "$91.20",
"display": "$91.20000",
"value": "91.20000"
},
"vol": {
"value_int": "5095833923444",
"currency": "BTC",
"display_short": "50,958.34\u00a0BTC",
"display": "50,958.33923444\u00a0BTC",
"value": "50958.33923444"
},
"last_local": {
"value_int": "9120000",
"currency": "USD",
"display_short": "$91.20",
"display": "$91.20000",
"value": "91.20000"
},
"last_all": {
"value_int": "9120000",
"currency": "USD",
"display_short": "$91.20",
"display": "$91.20000",
"value": "91.20000"
},
"vwap": {
"value_int": "9064567",
"currency": "USD",
"display_short": "$90.65",
"display": "$90.64567",
"value": "90.64567"
},
"high": {
"value_int": "9498000",
"currency": "USD",
"display_short": "$94.98",
"display": "$94.98000",
"value": "94.98000"
},
"low": {
"value_int": "8550000",
"currency": "USD",
"display_short": "$85.50",
"display": "$85.50000",
"value": "85.50000"
},
"avg": {
"value_int": "9037305",
"currency": "USD",
"display_short": "$90.37",
"display": "$90.37305",
"value": "90.37305"
}
},
"private": "ticker",
"channel": "d5f06780-30a8-4a48-a2f8-7ed181b4a13f",
"op": "private"
}
Lag {
"origin": "broadcast",
"lag": {
"stamp": "1364663115576119",
"qid": "74655212-1bc0-4b69-95da-79e40f4f771f",
"age": 329545
},
"private": "lag",
"channel": "85174711-be64-4de1-b783-0628995d7914",
"op": "private"
}
|
|
|
|
|
yucca
|
|
March 30, 2013, 06:05:25 PM |
|
thanks for checking it out!
your trade UUID (top) vs the one I get (bottom)
dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21
the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?
Then I will subscribe to the new streams, so i will try subscribing to new streams anyway and see how it goes, i guess i just send a masked/framed subscription request as per the mtgox stream api wiki.
So perhaps i will just have to ignore the junk stream, it will still have to flow thru my parser though, i dislike wasted clock cycles, plus it loads the server.
|
|
|
|
|
prof7bit
|
|
March 30, 2013, 06:12:00 PM
Last edit: March 30, 2013, 06:29:03 PM by prof7bit |
|
thanks for checking it out!
your trade UUID (top) vs the one I get (bottom)
dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21
the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?
No, I am using them too. But all your "4" are missing! Are you trying to parse the "4::/mtgox:" by any chance and have an error there in your parser? In your previous post the "4::/mtgox:" is appearing between the json objects and is the only thing that has a "4" in it, all other "4" seem to have gone missing! |
|
|
|
|
yucca
|
|
March 30, 2013, 09:06:03 PM |
|
you are correct prof7bit, my parsing state machine is buggy causing 4s to get dropped . I'm doing it with inc pointer on the raw incoming data buffer for speed and screwed up. I will rework it now. thanks man! |
|
|
|
|
yucca
|
|
March 31, 2013, 02:52:09 AM |
|
...trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart.... cool, similar to what i'm doing. It would be nice to have a mtgox call similar to bitcoincharts, so one could specify to get all trades since X unixtime. or even better binned trades in choice of bin size, M15, M30 etc. this would really save some bandwidth for mtgox! P.S. are you plotting volume also? if so do you find the live numbers sum closely to the history? |
|
|
|
|
prof7bit
|
|
March 31, 2013, 09:26:09 AM |
|
are you plotting volume also? if so do you find the live numbers sum closely to the history?
Not yet. I'm only plotting a quick & dirty price chart of the last 24 hours so I can quickly see from 5 meters distance if it has gone up or down during the night. I was too lazy until now to also write code to plot volume bars (I also don't have enough room in the terminal window to plot too many details in ascii-art with ncurses) |
|
|
|
|
yucca
|
|
April 08, 2013, 01:02:58 AM |
|
I think I remember reading in this thread that MagicalTux was going to change the order API so that people cant place more asks than BTC they have on account. Watching the live depth stream I dont think that this is implemented yet. On the GBP stream below we can see a bot went crazy and started placing many asks of the 2 same values resulting in 2 long horizontal lines. This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.  |
|
|
|
|
prof7bit
|
|
April 08, 2013, 10:30:30 AM |
|
This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.
No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else. |
|
|
|
|
