[BETA] MTGox websocket API, testers wanted

[yucca]

No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.

Wow well there are some large players then!

Well I now have stable multicurrency stream.io coming into my app. I still notice the odd hang during heavy times (maybe the ddos is not helping?).

I look for consecutive zero receive bytes or receive timout, if I see this I close the socket, destroy it, recreate and reconnect after a short delay.

One note to others (has been mentioned before), the channel UUIDs are not worth much from the client end. if you want multicurrency stream then you should filter on the human readable channel type and currency type. All UUIDs are not published in MtGox API docs, I suppose if you want to use them you would have to discover them yourself in the RX log.

Now how long this will stay good until I have to revisit this part of my code I dont know, hopefully for a long long time. Thankyou to all who helped me stumble to this point  

[genuise]

This is the list of channel ids I descovered till the moment

{ 'USD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'ticker' : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
                              'depth' : '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe'
                           },
                           'EUR' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '057bdc6b-9f9c-44e4-bc1a-363e4443ce87',
                              'ticker' : '0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15'
                           },
                           'GBP' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c',
                              'ticker': '7b842b7d-d1f9-46fa-a49c-c12f1ad5a533'
                           },
                           'CNY' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '0d1ecad8-e20f-459e-8bed-0bdcf927820f',
                              'ticker' : 'c251ec35-56f9-40ab-a4f6-13325c349de4'
                           },
                           'RUB' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'bd04f720-3c70-4dce-ae71-2422ab862c65',
                              'ticker' : 'd6412ca0-b686-464c-891a-d1ba3943f3c6'
                           },
                           'CAD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '10720792-084d-45ba-92e3-cf44d9477775',
                              'ticker' : '5b234cc3-a7c1-47ce-854f-27aee4cdbda5'
                           },
                           'PLN' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01',
                              'ticker' : 'e4ff055a-f8bf-407e-af76-676cad319a21'
                           },
                           'AUD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'eb6aaa11-99d0-4f64-9e8c-1140872a423d',
                              'ticker' : '296ee352-dd5d-46f3-9bea-5e39dede2005'
                           }

[Clark]

This is the list of channel ids I descovered till the moment

Well since we're playing that game 

Code:

            {
                'AUD': {'eb6aaa11-99d0-4f64-9e8c-1140872a423d': 'ticker',
                        '296ee352-dd5d-46f3-9bea-5e39dede2005': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CAD': {'10720792-084d-45ba-92e3-cf44d9477775': 'ticker',
                        '5b234cc3-a7c1-47ce-854f-27aee4cdbda5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CHF': {'2644c164-3db7-4475-8b45-c7042efe3413': 'ticker',
                        '113fec5f-294d-4929-86eb-8ca4c3fd1bed': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CNY': {'c251ec35-56f9-40ab-a4f6-13325c349de4': 'ticker',
                        '0d1ecad8-e20f-459e-8bed-0bdcf927820f': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'DKK': {'e5ce0604-574a-4059-9493-80af46c776b3': 'ticker',
                        '9219abb0-b50c-4007-b4d2-51d1711ab19c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'EUR': {'0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15': 'ticker',
                        '057bdc6b-9f9c-44e4-bc1a-363e4443ce87': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'GBP': {'7b842b7d-d1f9-46fa-a49c-c12f1ad5a533': 'ticker',
                        '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'HKD': {'d3ae78dd-01dd-4074-88a7-b8aa03cd28dd': 'ticker',
                        '049f65dc-3af3-4ffd-85a5-aac102b2a579': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'JPY': {'a39ae532-6a3c-4835-af8c-dda54cb4874e': 'ticker',
                        '94483e07-d797-4dd4-bc72-dc98f1fd39e3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NOK': {'7532e866-3a03-4514-a4b1-6f86e3a8dc11': 'ticker',
                        '66da7fb4-6b0c-4a10-9cb7-e2944e046eb5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NZD': {'5ddd27ca-2466-4d1a-8961-615dedb68bf1': 'ticker',
                        'cedf8730-bce6-4278-b6fe-9bee42930e95': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'PLN': {'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01': 'ticker',
                        'e4ff055a-f8bf-407e-af76-676cad319a21': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'RUB': {'bd04f720-3c70-4dce-ae71-2422ab862c65': 'ticker',
                        'd6412ca0-b686-464c-891a-d1ba3943f3c6': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SEK': {'6caf1244-655b-460f-beaf-5c56d1f4bea7': 'ticker',
                        '8f1fefaa-7c55-4420-ada0-4de15c1c38f3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SGD': {'2cb73ed1-07f4-45e0-8918-bcbfda658912': 'ticker',
                        '41e5c243-3d44-4fad-b690-f39e1dbb86a8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'THB': {'d58e3b69-9560-4b9e-8c58-b5c0f3fda5e1': 'ticker',
                        '67879668-532f-41f9-8eb0-55e7593a5ab8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'USD': {'d5f06780-30a8-4a48-a2f8-7ed181b4a13f': 'ticker',
                        '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'}
            }

[genuise]

Well Done !!! Clark

[Clark]

Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

[genuise]

Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

wow! cool

[yucca]

Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

wow! cool

+1! very useful!

[puax]

Is it me or the websocket has been down for days? Even my most basic implementation is silent.

[yucca]

Is it me or the websocket has been down for days? Even my most basic implementation is silent.

My socket.io implementation gets as far as upgrade and then I just receive zero-byte returns, I had previously coded zero-byte return to trigger connection reset, I took that trigger out and the receive function just finishes instantly with zero-bytes received.

It's been like this since MtGox went down, and has not recovered even after MtGox opened for trading via the user-account web interface again.

So I'm not sure what the problem is. Maybe we just wait?


13:34:43
=============================================================
THR_MtGox::SocketCreate: CONNECTED TO: socketio.mtgox.com:80
=============================================================

13:34:43 TRANSMIT>>>>>>>>
GET /socket.io/1/?Currency=EUR HTTP/1.1
Host: socketio.mtgox.com/mtgox?Currency=EUR
Connection: keep-alive


13:34:44 RECEIVE>>>>>>>>>
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 13 Apr 2013 11:34:43 GMT
Connection: keep-alive
Transfer-Encoding: chunked

30
b-LJ3EKgBF4kkuvdLP1d:60:60:websocket,flashsocket
0


13:34:44 TRANSMIT>>>>>>>>
GET /socket.io/1/websocket/b-LJ3EKgBF4kkuvdLP1d?Currency=EUR HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio.mtgox.com/mtgox?Currency=EUR
Origin: socketio.mtgox.com/mtgox?Currency=EUR
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13


13:34:45 RECEIVE>>>>>>>>>
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

�1::
13:34:45 TRANSMIT>>>>>>>>
TX_STR_FRAMED -> "1::/mtgox"

...ZERO-BYTE RX FROM NOW ON...

[genuise]

Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.

[yucca]

Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.

So you try socket.io and if it fails you fall back to http get with extended connection timeout. Kudos for working around it!

What kind of delay do you have?... Approx how old is the latest book entry that you receive? Do you get depth and trades?

[genuise]

Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.

[yucca]

Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.

Thanks for sharing your insight, sounds like your going the extra mile to mantain as much order as possible on such a shaky foundation.

I'm not sure I have the drive to start wrtiting/debugging such a workaround, I would end up getting kicked off for >5 tries an hour, I'd reset my gateway and try again and probably end up being block banned.

At the moment I have no users reliant on my software so do not have the stressful job of trying to get a fix/workaround out. I will sit tight for a few days and work on some GUI stuff.

Perhaps when MtGox stream re-establishes properly it will run like a dream, let us hope so. ( until it breaks again: ).

Best of luck with your commendable efforts.

[pescador]

I'm experimenting with the MtGox websocket API. I'm using Python with the websocket module from https://pypi.python.org/pypi/websocket-client. See example code below

Code:

import threading
import websocket
import json

class mtgox( threading.Thread ):

    def run( self ):
        websocket.enableTrace( True )
        url = 'ws://websocket.mtgox.com/mtgox?Currency=USD'
        self.socket = websocket.WebSocketApp( url, on_open = self.on_open )
        self.socket.run_forever( )

    def subscribe( self, channel ):
        output = { 'op': 'mtgox.subscribe', 'type': channel }
        output = json.dumps( output )
        self.socket.send( output )

    def on_open( self, socket ):
        self.subscribe( 'depth' )
        self.subscribe( 'lag' )        
        self.subscribe( 'ticker' )
        self.subscribe( 'trades' )

if __name__ == '__main__':
    mtgox = mtgox( )
    mtgox.start( )

When I run this code I receive ticker and depth messages, but no lag or trade messages. Also I do not get any replies to my mtgox.subscribe commands, that the documentation seems to promise. However when I send a mtgox.subscribe command with a wrong type parameter, I get an error message "Unknown mtgox message type", so it seems my subscribe commands are received and accepted.

Could somebody please tell me why I'm not receiving trade and lag messages?

[pescador]

To answer my own question above. It seems to be working now. Now I also get trade and lag messages. Apparently it was something at MtGox's side.

[pescador]

I've downloaded the full bid / ask tables in USD with 'depth/full' and try to update them as depth and trade messages come in. From https://support.mtgox.com/entries/20800336-Multi-Currency-Trading I understand that the tables in USD also include the bids / asks in other currencies, that are converted to USD at the daily rate of the European Central Bank minus / plus a fee of 2.5 %.

How do I update my tables when a primary trade comes in with a price_currency other than USD?

[genuise]

You should not update orderbooks using trades channel. You need to listen to depth channel to update orderbooks. And also it would be convinient to fix ordebooks with ticker channel messages if neccesary (in case not all depth messages were arrived and ordebook state is broken)

[pescador]

Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817. I do that and it seems to be working fine. I only don't know what to do with trades in other currencies.

I'm already using the trick to use ticker messages to fix the tables when they appear to be broken, but that doesn't happen a lot.

[genuise]

strange I currently use only depth and ticker messages. and did not have problems with orderbook state.

on my opinion there is not need to use trade messages because some trades can be from orders which are not in orderbook but executed before getting into orderbook.

but probably I missed something. It would be good to hear others

[prof7bit]

Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817

I did some further experimenting and debugging and it seems this old posting of mine is wrong. There *will* be a depth message immediately following the trade message, so theoretically it would be enough to use only depth messages to update the orderbook.

But in goxtool/goxapi I still update the orderbook already in the trade message, so a hypothetical goxtool bot that would rely on the *new* orderbook state *after* the trade has been executed would already find the orderbook in the new updated state from inside the slot_trade() slot. Since I only ever use the total_vol of the depth messages and ignore the volume difference in my implementation the depth messages that immediately follow a trade message will simply result in no change to the orderbook at all.