Bitcoin Forum
December 07, 2016, 11:01:36 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 »  All
  Print  
Author Topic: [BETA] MTGox websocket API, testers wanted  (Read 72231 times)
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
April 15, 2011, 02:48:21 AM
 #1

Hi,

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

You can connect via: ws://websocket.mtgox.com/mtgox

The websocket will subscribe you to some channels automatically:
  • dbf1dee9-4f2e-4a08-8cb7-748919a71b21: trades (each time a trade happens, you get something here)
  • d5f06780-30a8-4a48-a2f8-7ed181b4a13f: the mtgox ticker (lots of updates, with often the same data)
  • 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe: depth information in realtime (price + amount + type... type=1=Ask, type=2=Bid)

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Each message is a JSON-encoded object, with at least "op" element. The "op" element contains the operation to be done (for outgoing messages), or the type of message (for incoming messages).

Possible outgoing commands:

  • unsubscribe Stop receiving messages from a channel (parameter "channel")

Example incoming data:

Ticker
Code:
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":0.9515,"high":1,"low":0.91,"sell":0.9697,"vol":34349}}
Trade
Code:
{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade","trade":{"amount":2.71,"amount_int":"271000000","date":1310279340,"item":"BTC","price":14.43,"price_currency":"USD","price_int":"1443000","tid":"1310279340877902","trade_type":"bid","type":"trade"}}

Contains:
  • amount: the traded amount in item (BTC), float, deprecated
  • amount_int: same as amount, but in smallest unit
  • date: unix timestamp of trade
  • item: What was this trade about
  • price: price per unit, float, deprecated
  • price_int: price in smallest unit as integer (5 decimals of USD, 3 in case of JPY)
  • price_currency: currency in which trade was completed
  • tid: Trade id (big integer, which is in fact trade timestamp in microseconds)
  • trade_type: Did this trade result from the execution of a bid or a ask?

Depth update
Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"14.43","price_int":"1443000","type":1,"type_str":"ask","volume":"-2.71","volume_int":"-271000000"},"op":"private","origin":"broadcast","private":"depth"}

Contains:
  • currency: the currency affected
  • item: the item (BTC)
  • price: price as a float, deprecated
  • price_int: the price at which volume change happened
  • type: 1=ask 2=bid. Deprecated, see type_str
  • type_str: type of order at this depth, either "ask" or "bid"
  • volume: the volume change as float, deprecated
  • volume_int: volume change in smallest unit

(priv) Order update
Code:
{"channel":"(partial key)","op":"private","order_upd":{"amount":1000,"darkStatus":0,"date":1302836027,"oid":"(oid)","price":0.9899,"status":1},"origin":"broadcast","private":"order_upd"}


This is still under tests, and any element of this websocket API may change anytime.
Ideas/comments are welcome Smiley

1481108496
Hero Member
*
Offline Offline

Posts: 1481108496

View Profile Personal Message (Offline)

Ignore
1481108496
Reply with quote  #2

1481108496
Report to moderator
1481108496
Hero Member
*
Offline Offline

Posts: 1481108496

View Profile Personal Message (Offline)

Ignore
1481108496
Reply with quote  #2

1481108496
Report to moderator
1481108496
Hero Member
*
Offline Offline

Posts: 1481108496

View Profile Personal Message (Offline)

Ignore
1481108496
Reply with quote  #2

1481108496
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481108496
Hero Member
*
Offline Offline

Posts: 1481108496

View Profile Personal Message (Offline)

Ignore
1481108496
Reply with quote  #2

1481108496
Report to moderator
1481108496
Hero Member
*
Offline Offline

Posts: 1481108496

View Profile Personal Message (Offline)

Ignore
1481108496
Reply with quote  #2

1481108496
Report to moderator
demonofelru
Full Member
***
Offline Offline

Activity: 238



View Profile
April 15, 2011, 02:54:51 AM
 #2

Nice keep the updates coming Tux much appreciated.

Names do not matter; however, if you insist...id...
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 15, 2011, 03:58:34 AM
 #3

This will help a ton! Thank you very much.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
Stephen Gornick
Legendary
*
Offline Offline

Activity: 2002



View Profile
April 15, 2011, 04:11:16 AM
 #4

Hoping newsham (or anyone else) will update the MtGox chart to accommodate the new Websocket data feed:
   http://www.thenewsh.com/~newsham/x/mtgox/

Keefe
Hero Member
*****
Offline Offline

Activity: 681


View Profile
April 15, 2011, 08:02:01 AM
 #5

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
April 15, 2011, 08:06:02 AM
 #6

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

Should be possible, I'll do something Smiley

memvola
Hero Member
*****
Offline Offline

Activity: 896


View Profile
April 15, 2011, 10:34:25 AM
 #7

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
April 15, 2011, 11:42:37 AM
 #8

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?


The draft-75 version of the protocol is really easy to implement from an application.

You send a simple HTTP request:

Code:
GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

(remember to add an empty line at the end)

And you get the following reply:

Code:
HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: null
WebSocket-Location: ws://websocket.mtgox.com/mtgox

Once you got this, the websocket server will send you frames in the following format: a NUL byte (00), some JSON-encoded data, and a FF byte. You can then decode each json packet as they come and handle them.

You can also send packets in the same way, by sending a NUL byte, some json-encoded data, and a FF byte (for example to send an unsubscribe request, or subscribe to a mtgox key).

Example with telnet: http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
April 15, 2011, 03:15:10 PM
 #9

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field  Roll Eyes

mndrix
Michael Hendricks
VIP
Sr. Member
*
Offline Offline

Activity: 447


View Profile
April 15, 2011, 03:29:20 PM
 #10

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 15, 2011, 03:46:43 PM
 #11

Hello, thanks for the API. Few questions/comments:

b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell...


Yes, from reading the protocol specification, it looks like I have to send all sorts of different bytes at different times to the server. Some of them translate into ASCII, while others are non-printing. I get confused here, because in the examples, it looks like we're just sending plain lines of text. Do we need byte strings? Do we need /r/n newlines?

And server timestamps would be a huge help! There are multiple options here, really. First, you could just timestamp very accurately, so the client can recreate the actual order flow precisely. Alternatively (and probably easier), just create a long integer ascending order number that goes along with the trades and orders. That way, the server can stay on whole seconds for timestamps, but the client can arrange trades and orders accurately using the id numbers. There should be separate numbering lists for orders and trades, of course.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
April 15, 2011, 04:15:16 PM
 #12

Damn it...

So many things to learn, and only one life Sad
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
April 15, 2011, 05:44:23 PM
 #13

Is websocket api down?

Code:
$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

memvola
Hero Member
*****
Offline Offline

Activity: 896


View Profile
April 15, 2011, 06:24:25 PM
 #14

Is websocket api down?
It's down for me for at least the last hour.

And thanks for the nice explanation, MagicalTux. I wrote a client in haskell, I'll post it when the subscription part is complete.
jed
Full Member
***
Offline Offline

Activity: 165

Jed McCaleb


View Profile WWW
April 15, 2011, 06:24:56 PM
 #15

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

stellar.org   |    twitter
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
April 16, 2011, 02:50:37 AM
 #16

Is websocket api down?

Code:
$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

Yep was down, found the reason, fixed the code and restarted it (division by zero when someone sent badly formatted draft-00 headers).

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field  Roll Eyes


a) The ticker is the current stats about bitcoin. It's sent once each time it have a chance to be changed, which includes when orders are posted, removed or completed.
b) Here's an example:
Code:
{"op":"unsubscribe","channel":"xxx"}

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

Not at this point.

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

Because it was easier since I intend to use this system for many other sites, without having to do something to isolate users of a specific system. Also there are user-specific channels and many other stuff which makes use of uuid a bit easier (for me).

memvola
Hero Member
*****
Offline Offline

Activity: 896


View Profile
April 16, 2011, 09:56:15 AM
 #17

Here's the haskell code which connects to the server, subscribes to the private channel and unsubscribes from the ticker. No json parsing or error handling.

Code:
module Main (main) where

import Network
import System.IO
import Control.Monad

clientHandshake :: String
clientHandshake = "GET /mtgox HTTP/1.1\r\n\
                  \Upgrade: WebSocket\r\n\
                  \Connection: Upgrade\r\n\
                  \Host: websocket.mtgox.com\r\n\
                  \Origin: null\r\n\
                  \\r\n"

-- Expected server response
serverHandshake :: [String]
serverHandshake = ["HTTP/1.1 101 Web Socket Protocol Handshake",
                   "Upgrade: WebSocket","Connection: Upgrade",
                   "WebSocket-Origin: null",
                   "WebSocket-Location: ws://websocket.mtgox.com/mtgox",
                   "WebSocket-Protocol: *"]

-- https://mtgox.com/code/getKey.php
key :: String
key = "\"key\":\"00000000-0000-0000-0000-000000000000:0000000000000000000000000000\""

hGetHeader :: Handle -> IO [String]
hGetHeader h = do s <- hGetLine h
                  if s == "\r"
                    then return []
                    else do s' <- hGetHeader h
                            return (init s : s')

hGetFrame :: Handle -> IO String
hGetFrame h = do c <- hGetChar h
                 if c == '\xff'
                   then return "\n"
                   else do c' <- hGetFrame h
                           return $ if c == '\x00' then c' else (c:c')

hPutFrame :: Handle -> String -> IO ()
hPutFrame h s = hPutStr h $ '\x00' : s ++ "\xff"

main :: IO ()
main = do h <- connectTo "websocket.mtgox.com" (PortNumber 80)
          hSetBuffering h NoBuffering
          hPutStr h clientHandshake
          hdr <- hGetHeader h
          when (head hdr /= head serverHandshake) .
               error $ "Invalid server handshake:\n" ++ show hdr

          hPutFrame h "{\"op\":\"unsubscribe\",\"channel\":\"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\"}"
          hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

          forever $ hGetFrame h >>= putStr

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 617


Working on new MtGox features


View Profile WWW
April 16, 2011, 10:00:45 AM
 #18

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this Smiley

memvola
Hero Member
*****
Offline Offline

Activity: 896


View Profile
April 17, 2011, 04:21:11 PM
 #19

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this Smiley
Well, anyway, I was actually hoping to be able to keep market depth up to date without polling, like I can do with recent trades using the trades channel, but it seems there's currently no way to do the same with orders. Ticker data doesn't report new or deleted orders, or at least I failed to identify them. I don't want to ask for updates like crazy. Smiley Are you planning to introduce a new channel for this?

EDIT: Oh, Keefe already asked for this but I missed it. Nevermind then...
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 17, 2011, 11:39:23 PM
 #20

So I've answered some of my own questions:

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

I see there is a new market depth channel as well.

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending
Code:
{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}
and
Code:
{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Here's an idea for an initial documentation page:
List the possible fields contained in the JSON return objects, starting with op values and going from there. This is what I have so far, from seeing different things come across the socket:
Code:
op:remark
  message:<message text>
  success:<success boolean>
op:subscribe
  channel:<channel uuid>
op:unsubscribe
  channel:<channel uuid>
op:private
  channel:<channel uuid>
  origin:broadcast
  private:depth
    depth:{
      volume:<volume>
      price:<price>
      type:<order type>
    }
  private:ticker
    ticker:{
      high:<high price>
      low:<low price>
      vol:<volume>
      buy:<buy price>
      sell:<sell price>
    }
  private:trade
    trade:{
      date:<int time>
      amount:<amount float>
      type:trade
      price:<trade price>
   }
  private:order_add
    order_add:{
      oid:<order id int>
      price:<limit price>
      date:<int time>
      amount:<amount>
      status:<status int>
      darkStatus:<0 or 1>
    }
  private:order_rem
    order_rem:{
      oid:<order id int>
    }

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
Pages: [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!