[BETA] MTGox websocket API, testers wanted

[MagicalTux]

Hi,

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

You can connect via: ws://websocket.mtgox.com/mtgox

The websocket will subscribe you to some channels automatically:

• dbf1dee9-4f2e-4a08-8cb7-748919a71b21: trades (each time a trade happens, you get something here)
• d5f06780-30a8-4a48-a2f8-7ed181b4a13f: the mtgox ticker (lots of updates, with often the same data)
• 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe: depth information in realtime (price + amount + type... type=1=Ask, type=2=Bid)

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Each message is a JSON-encoded object, with at least "op" element. The "op" element contains the operation to be done (for outgoing messages), or the type of message (for incoming messages).

Possible outgoing commands:

• unsubscribe Stop receiving messages from a channel (parameter "channel")

Example incoming data:

Ticker

Code:

{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":0.9515,"high":1,"low":0.91,"sell":0.9697,"vol":34349}}

Trade

Code:

{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade","trade":{"amount":2.71,"amount_int":"271000000","date":1310279340,"item":"BTC","price":14.43,"price_currency":"USD","price_int":"1443000","tid":"1310279340877902","trade_type":"bid","type":"trade"}}

Contains:

• amount: the traded amount in item (BTC), float, deprecated
• amount_int: same as amount, but in smallest unit
• date: unix timestamp of trade
• item: What was this trade about
• price: price per unit, float, deprecated
• price_int: price in smallest unit as integer (5 decimals of USD, 3 in case of JPY)
• price_currency: currency in which trade was completed
• tid: Trade id (big integer, which is in fact trade timestamp in microseconds)
• trade_type: Did this trade result from the execution of a bid or a ask?

Depth update

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"14.43","price_int":"1443000","type":1,"type_str":"ask","volume":"-2.71","volume_int":"-271000000"},"op":"private","origin":"broadcast","private":"depth"}

Contains:

• currency: the currency affected
• item: the item (BTC)
• price: price as a float, deprecated
• price_int: the price at which volume change happened
• type: 1=ask 2=bid. Deprecated, see type_str
• type_str: type of order at this depth, either "ask" or "bid"
• volume: the volume change as float, deprecated
• volume_int: volume change in smallest unit

(priv) Order update

Code:

{"channel":"(partial key)","op":"private","order_upd":{"amount":1000,"darkStatus":0,"date":1302836027,"oid":"(oid)","price":0.9899,"status":1},"origin":"broadcast","private":"order_upd"}

This is still under tests, and any element of this websocket API may change anytime.
Ideas/comments are welcome

[demonofelru]

Nice keep the updates coming Tux much appreciated.

[Clark]

This will help a ton! Thank you very much.

[Stephen Gornick]

Hoping newsham (or anyone else) will update the MtGox chart to accommodate the new Websocket data feed:
   http://www.thenewsh.com/~newsham/x/mtgox/

[Keefe]

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

[MagicalTux]

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

Should be possible, I'll do something

[memvola]

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?

[MagicalTux]

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?

The draft-75 version of the protocol is really easy to implement from an application.

You send a simple HTTP request:

Code:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

(remember to add an empty line at the end)

And you get the following reply:

Code:

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: null
WebSocket-Location: ws://websocket.mtgox.com/mtgox

Once you got this, the websocket server will send you frames in the following format: a NUL byte (00), some JSON-encoded data, and a FF byte. You can then decode each json packet as they come and handle them.

You can also send packets in the same way, by sending a NUL byte, some json-encoded data, and a FF byte (for example to send an unsubscribe request, or subscribe to a mtgox key).

Example with telnet: http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

[slush]

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field 

[mndrix]

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

[Clark]

Hello, thanks for the API. Few questions/comments:

b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell...

Yes, from reading the protocol specification, it looks like I have to send all sorts of different bytes at different times to the server. Some of them translate into ASCII, while others are non-printing. I get confused here, because in the examples, it looks like we're just sending plain lines of text. Do we need byte strings? Do we need /r/n newlines?

And server timestamps would be a huge help! There are multiple options here, really. First, you could just timestamp very accurately, so the client can recreate the actual order flow precisely. Alternatively (and probably easier), just create a long integer ascending order number that goes along with the trades and orders. That way, the server can stay on whole seconds for timestamps, but the client can arrange trades and orders accurately using the id numbers. There should be separate numbering lists for orders and trades, of course.

[grondilu]

Damn it...

So many things to learn, and only one life

[slush]

Is websocket api down?

Code:

$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

[memvola]

Is websocket api down?

It's down for me for at least the last hour.

And thanks for the nice explanation, MagicalTux. I wrote a client in haskell, I'll post it when the subscription part is complete.

[jed]

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

[MagicalTux]

Is websocket api down?

Code:

$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

Yep was down, found the reason, fixed the code and restarted it (division by zero when someone sent badly formatted draft-00 headers).

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field 

a) The ticker is the current stats about bitcoin. It's sent once each time it have a chance to be changed, which includes when orders are posted, removed or completed.
b) Here's an example:

Code:

{"op":"unsubscribe","channel":"xxx"}

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

Not at this point.

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

Because it was easier since I intend to use this system for many other sites, without having to do something to isolate users of a specific system. Also there are user-specific channels and many other stuff which makes use of uuid a bit easier (for me).

[memvola]

Here's the haskell code which connects to the server, subscribes to the private channel and unsubscribes from the ticker. No json parsing or error handling.

Code:

module Main (main) where

import Network
import System.IO
import Control.Monad

clientHandshake :: String
clientHandshake = "GET /mtgox HTTP/1.1\r\n\
                  \Upgrade: WebSocket\r\n\
                  \Connection: Upgrade\r\n\
                  \Host: websocket.mtgox.com\r\n\
                  \Origin: null\r\n\
                  \\r\n"

-- Expected server response
serverHandshake :: [String]
serverHandshake = ["HTTP/1.1 101 Web Socket Protocol Handshake",
                   "Upgrade: WebSocket","Connection: Upgrade",
                   "WebSocket-Origin: null",
                   "WebSocket-Location: ws://websocket.mtgox.com/mtgox",
                   "WebSocket-Protocol: *"]

-- https://mtgox.com/code/getKey.php
key :: String
key = "\"key\":\"00000000-0000-0000-0000-000000000000:0000000000000000000000000000\""

hGetHeader :: Handle -> IO [String]
hGetHeader h = do s <- hGetLine h
                  if s == "\r"
                    then return []
                    else do s' <- hGetHeader h
                            return (init s : s')

hGetFrame :: Handle -> IO String
hGetFrame h = do c <- hGetChar h
                 if c == '\xff'
                   then return "\n"
                   else do c' <- hGetFrame h
                           return $ if c == '\x00' then c' else (c:c')

hPutFrame :: Handle -> String -> IO ()
hPutFrame h s = hPutStr h $ '\x00' : s ++ "\xff"

main :: IO ()
main = do h <- connectTo "websocket.mtgox.com" (PortNumber 80)
          hSetBuffering h NoBuffering
          hPutStr h clientHandshake
          hdr <- hGetHeader h
          when (head hdr /= head serverHandshake) .
               error $ "Invalid server handshake:\n" ++ show hdr

          hPutFrame h "{\"op\":\"unsubscribe\",\"channel\":\"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\"}"
          hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

          forever $ hGetFrame h >>= putStr

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

[MagicalTux]

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this

[memvola]

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this

Well, anyway, I was actually hoping to be able to keep market depth up to date without polling, like I can do with recent trades using the trades channel, but it seems there's currently no way to do the same with orders. Ticker data doesn't report new or deleted orders, or at least I failed to identify them. I don't want to ask for updates like crazy. Are you planning to introduce a new channel for this?

EDIT: Oh, Keefe already asked for this but I missed it. Nevermind then...

[Clark]

So I've answered some of my own questions:

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

I see there is a new market depth channel as well.

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending

Code:

{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

and

Code:

{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Here's an idea for an initial documentation page:
List the possible fields contained in the JSON return objects, starting with op values and going from there. This is what I have so far, from seeing different things come across the socket:

Code:

op:remark
  message:<message text>
  success:<success boolean>
op:subscribe
  channel:<channel uuid>
op:unsubscribe
  channel:<channel uuid>
op:private
  channel:<channel uuid>
  origin:broadcast
  private:depth
    depth:{
      volume:<volume>
      price:<price>
      type:<order type>
    }
  private:ticker
    ticker:{
      high:<high price>
      low:<low price>
      vol:<volume>
      buy:<buy price>
      sell:<sell price>
    }
  private:trade
    trade:{
      date:<int time>
      amount:<amount float>
      type:trade
      price:<trade price>
   }
  private:order_add
    order_add:{
      oid:<order id int>
      price:<limit price>
      date:<int time>
      amount:<amount>
      status:<status int>
      darkStatus:<0 or 1>
    }
  private:order_rem
    order_rem:{
      oid:<order id int>
    }