[BETA] MTGox websocket API, testers wanted

[prof7bit]

are you plotting volume also? if so do you find the live numbers sum closely to the history?

Not yet. I'm only plotting a quick & dirty price chart of the last 24 hours so I can quickly see from 5 meters distance if it has gone up or down during the night. I was too lazy until now to also write code to plot volume bars (I also don't have enough room in the terminal window to plot too many details in ascii-art with ncurses)

[yucca]

I think I remember reading in this thread that MagicalTux was going to change the order API so that people cant place more asks than BTC they have on account.

Watching the live depth stream I dont think that this is implemented yet.

On the GBP stream below we can see a bot went crazy and started placing many asks of the 2 same values resulting in 2 long horizontal lines.

This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.

[prof7bit]

This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.

No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.

[yucca]

No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.

Wow well there are some large players then!

Well I now have stable multicurrency stream.io coming into my app. I still notice the odd hang during heavy times (maybe the ddos is not helping?).

I look for consecutive zero receive bytes or receive timout, if I see this I close the socket, destroy it, recreate and reconnect after a short delay.

One note to others (has been mentioned before), the channel UUIDs are not worth much from the client end. if you want multicurrency stream then you should filter on the human readable channel type and currency type. All UUIDs are not published in MtGox API docs, I suppose if you want to use them you would have to discover them yourself in the RX log.

Now how long this will stay good until I have to revisit this part of my code I dont know, hopefully for a long long time. Thankyou to all who helped me stumble to this point  

[genuise]

This is the list of channel ids I descovered till the moment

{ 'USD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'ticker' : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
                              'depth' : '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe'
                           },
                           'EUR' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '057bdc6b-9f9c-44e4-bc1a-363e4443ce87',
                              'ticker' : '0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15'
                           },
                           'GBP' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c',
                              'ticker': '7b842b7d-d1f9-46fa-a49c-c12f1ad5a533'
                           },
                           'CNY' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '0d1ecad8-e20f-459e-8bed-0bdcf927820f',
                              'ticker' : 'c251ec35-56f9-40ab-a4f6-13325c349de4'
                           },
                           'RUB' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'bd04f720-3c70-4dce-ae71-2422ab862c65',
                              'ticker' : 'd6412ca0-b686-464c-891a-d1ba3943f3c6'
                           },
                           'CAD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '10720792-084d-45ba-92e3-cf44d9477775',
                              'ticker' : '5b234cc3-a7c1-47ce-854f-27aee4cdbda5'
                           },
                           'PLN' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01',
                              'ticker' : 'e4ff055a-f8bf-407e-af76-676cad319a21'
                           },
                           'AUD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'eb6aaa11-99d0-4f64-9e8c-1140872a423d',
                              'ticker' : '296ee352-dd5d-46f3-9bea-5e39dede2005'
                           }

[Clark]

This is the list of channel ids I descovered till the moment

Well since we're playing that game 

Code:

            {
                'AUD': {'eb6aaa11-99d0-4f64-9e8c-1140872a423d': 'ticker',
                        '296ee352-dd5d-46f3-9bea-5e39dede2005': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CAD': {'10720792-084d-45ba-92e3-cf44d9477775': 'ticker',
                        '5b234cc3-a7c1-47ce-854f-27aee4cdbda5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CHF': {'2644c164-3db7-4475-8b45-c7042efe3413': 'ticker',
                        '113fec5f-294d-4929-86eb-8ca4c3fd1bed': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CNY': {'c251ec35-56f9-40ab-a4f6-13325c349de4': 'ticker',
                        '0d1ecad8-e20f-459e-8bed-0bdcf927820f': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'DKK': {'e5ce0604-574a-4059-9493-80af46c776b3': 'ticker',
                        '9219abb0-b50c-4007-b4d2-51d1711ab19c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'EUR': {'0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15': 'ticker',
                        '057bdc6b-9f9c-44e4-bc1a-363e4443ce87': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'GBP': {'7b842b7d-d1f9-46fa-a49c-c12f1ad5a533': 'ticker',
                        '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'HKD': {'d3ae78dd-01dd-4074-88a7-b8aa03cd28dd': 'ticker',
                        '049f65dc-3af3-4ffd-85a5-aac102b2a579': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'JPY': {'a39ae532-6a3c-4835-af8c-dda54cb4874e': 'ticker',
                        '94483e07-d797-4dd4-bc72-dc98f1fd39e3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NOK': {'7532e866-3a03-4514-a4b1-6f86e3a8dc11': 'ticker',
                        '66da7fb4-6b0c-4a10-9cb7-e2944e046eb5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NZD': {'5ddd27ca-2466-4d1a-8961-615dedb68bf1': 'ticker',
                        'cedf8730-bce6-4278-b6fe-9bee42930e95': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'PLN': {'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01': 'ticker',
                        'e4ff055a-f8bf-407e-af76-676cad319a21': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'RUB': {'bd04f720-3c70-4dce-ae71-2422ab862c65': 'ticker',
                        'd6412ca0-b686-464c-891a-d1ba3943f3c6': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SEK': {'6caf1244-655b-460f-beaf-5c56d1f4bea7': 'ticker',
                        '8f1fefaa-7c55-4420-ada0-4de15c1c38f3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SGD': {'2cb73ed1-07f4-45e0-8918-bcbfda658912': 'ticker',
                        '41e5c243-3d44-4fad-b690-f39e1dbb86a8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'THB': {'d58e3b69-9560-4b9e-8c58-b5c0f3fda5e1': 'ticker',
                        '67879668-532f-41f9-8eb0-55e7593a5ab8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'USD': {'d5f06780-30a8-4a48-a2f8-7ed181b4a13f': 'ticker',
                        '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'}
            }

[genuise]

Well Done !!! Clark

[Clark]

Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

[genuise]

Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

wow! cool

[yucca]

Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

wow! cool

+1! very useful!

[puax]

Is it me or the websocket has been down for days? Even my most basic implementation is silent.

[yucca]

Is it me or the websocket has been down for days? Even my most basic implementation is silent.

My socket.io implementation gets as far as upgrade and then I just receive zero-byte returns, I had previously coded zero-byte return to trigger connection reset, I took that trigger out and the receive function just finishes instantly with zero-bytes received.

It's been like this since MtGox went down, and has not recovered even after MtGox opened for trading via the user-account web interface again.

So I'm not sure what the problem is. Maybe we just wait?


13:34:43
=============================================================
THR_MtGox::SocketCreate: CONNECTED TO: socketio.mtgox.com:80
=============================================================

13:34:43 TRANSMIT>>>>>>>>
GET /socket.io/1/?Currency=EUR HTTP/1.1
Host: socketio.mtgox.com/mtgox?Currency=EUR
Connection: keep-alive


13:34:44 RECEIVE>>>>>>>>>
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 13 Apr 2013 11:34:43 GMT
Connection: keep-alive
Transfer-Encoding: chunked

30
b-LJ3EKgBF4kkuvdLP1d:60:60:websocket,flashsocket
0


13:34:44 TRANSMIT>>>>>>>>
GET /socket.io/1/websocket/b-LJ3EKgBF4kkuvdLP1d?Currency=EUR HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio.mtgox.com/mtgox?Currency=EUR
Origin: socketio.mtgox.com/mtgox?Currency=EUR
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13


13:34:45 RECEIVE>>>>>>>>>
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

�1::
13:34:45 TRANSMIT>>>>>>>>
TX_STR_FRAMED -> "1::/mtgox"

...ZERO-BYTE RX FROM NOW ON...

[genuise]

Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.

[yucca]

Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.

So you try socket.io and if it fails you fall back to http get with extended connection timeout. Kudos for working around it!

What kind of delay do you have?... Approx how old is the latest book entry that you receive? Do you get depth and trades?

[genuise]

Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.

[yucca]

Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.

Thanks for sharing your insight, sounds like your going the extra mile to mantain as much order as possible on such a shaky foundation.

I'm not sure I have the drive to start wrtiting/debugging such a workaround, I would end up getting kicked off for >5 tries an hour, I'd reset my gateway and try again and probably end up being block banned.

At the moment I have no users reliant on my software so do not have the stressful job of trying to get a fix/workaround out. I will sit tight for a few days and work on some GUI stuff.

Perhaps when MtGox stream re-establishes properly it will run like a dream, let us hope so. ( until it breaks again: ).

Best of luck with your commendable efforts.

[pescador]

I'm experimenting with the MtGox websocket API. I'm using Python with the websocket module from https://pypi.python.org/pypi/websocket-client. See example code below

Code:

import threading
import websocket
import json

class mtgox( threading.Thread ):

    def run( self ):
        websocket.enableTrace( True )
        url = 'ws://websocket.mtgox.com/mtgox?Currency=USD'
        self.socket = websocket.WebSocketApp( url, on_open = self.on_open )
        self.socket.run_forever( )

    def subscribe( self, channel ):
        output = { 'op': 'mtgox.subscribe', 'type': channel }
        output = json.dumps( output )
        self.socket.send( output )

    def on_open( self, socket ):
        self.subscribe( 'depth' )
        self.subscribe( 'lag' )        
        self.subscribe( 'ticker' )
        self.subscribe( 'trades' )

if __name__ == '__main__':
    mtgox = mtgox( )
    mtgox.start( )

When I run this code I receive ticker and depth messages, but no lag or trade messages. Also I do not get any replies to my mtgox.subscribe commands, that the documentation seems to promise. However when I send a mtgox.subscribe command with a wrong type parameter, I get an error message "Unknown mtgox message type", so it seems my subscribe commands are received and accepted.

Could somebody please tell me why I'm not receiving trade and lag messages?

[pescador]

To answer my own question above. It seems to be working now. Now I also get trade and lag messages. Apparently it was something at MtGox's side.

[pescador]

I've downloaded the full bid / ask tables in USD with 'depth/full' and try to update them as depth and trade messages come in. From https://support.mtgox.com/entries/20800336-Multi-Currency-Trading I understand that the tables in USD also include the bids / asks in other currencies, that are converted to USD at the daily rate of the European Central Bank minus / plus a fee of 2.5 %.

How do I update my tables when a primary trade comes in with a price_currency other than USD?

[genuise]

You should not update orderbooks using trades channel. You need to listen to depth channel to update orderbooks. And also it would be convinient to fix ordebooks with ticker channel messages if neccesary (in case not all depth messages were arrived and ordebook state is broken)