[BETA] MTGox websocket API, testers wanted

[MagicalTux]

Hi,

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

You can connect via: ws://websocket.mtgox.com/mtgox

The websocket will subscribe you to some channels automatically:

• dbf1dee9-4f2e-4a08-8cb7-748919a71b21: trades (each time a trade happens, you get something here)
• d5f06780-30a8-4a48-a2f8-7ed181b4a13f: the mtgox ticker (lots of updates, with often the same data)
• 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe: depth information in realtime (price + amount + type... type=1=Ask, type=2=Bid)

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Each message is a JSON-encoded object, with at least "op" element. The "op" element contains the operation to be done (for outgoing messages), or the type of message (for incoming messages).

Possible outgoing commands:

• unsubscribe Stop receiving messages from a channel (parameter "channel")

Example incoming data:

Ticker

Code:

{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":0.9515,"high":1,"low":0.91,"sell":0.9697,"vol":34349}}

Trade

Code:

{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade","trade":{"amount":2.71,"amount_int":"271000000","date":1310279340,"item":"BTC","price":14.43,"price_currency":"USD","price_int":"1443000","tid":"1310279340877902","trade_type":"bid","type":"trade"}}

Contains:

• amount: the traded amount in item (BTC), float, deprecated
• amount_int: same as amount, but in smallest unit
• date: unix timestamp of trade
• item: What was this trade about
• price: price per unit, float, deprecated
• price_int: price in smallest unit as integer (5 decimals of USD, 3 in case of JPY)
• price_currency: currency in which trade was completed
• tid: Trade id (big integer, which is in fact trade timestamp in microseconds)
• trade_type: Did this trade result from the execution of a bid or a ask?

Depth update

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"14.43","price_int":"1443000","type":1,"type_str":"ask","volume":"-2.71","volume_int":"-271000000"},"op":"private","origin":"broadcast","private":"depth"}

Contains:

• currency: the currency affected
• item: the item (BTC)
• price: price as a float, deprecated
• price_int: the price at which volume change happened
• type: 1=ask 2=bid. Deprecated, see type_str
• type_str: type of order at this depth, either "ask" or "bid"
• volume: the volume change as float, deprecated
• volume_int: volume change in smallest unit

(priv) Order update

Code:

{"channel":"(partial key)","op":"private","order_upd":{"amount":1000,"darkStatus":0,"date":1302836027,"oid":"(oid)","price":0.9899,"status":1},"origin":"broadcast","private":"order_upd"}

This is still under tests, and any element of this websocket API may change anytime.
Ideas/comments are welcome

[demonofelru]

Nice keep the updates coming Tux much appreciated.

[Clark]

This will help a ton! Thank you very much.

[Stephen Gornick]

Hoping newsham (or anyone else) will update the MtGox chart to accommodate the new Websocket data feed:
   http://www.thenewsh.com/~newsham/x/mtgox/

[Keefe]

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

[MagicalTux]

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

Should be possible, I'll do something

[memvola]

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?

[MagicalTux]

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?

The draft-75 version of the protocol is really easy to implement from an application.

You send a simple HTTP request:

Code:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

(remember to add an empty line at the end)

And you get the following reply:

Code:

HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: null
WebSocket-Location: ws://websocket.mtgox.com/mtgox

Once you got this, the websocket server will send you frames in the following format: a NUL byte (00), some JSON-encoded data, and a FF byte. You can then decode each json packet as they come and handle them.

You can also send packets in the same way, by sending a NUL byte, some json-encoded data, and a FF byte (for example to send an unsubscribe request, or subscribe to a mtgox key).

Example with telnet: http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

[slush]

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field 

[mndrix]

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

[Clark]

Hello, thanks for the API. Few questions/comments:

b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell...

Yes, from reading the protocol specification, it looks like I have to send all sorts of different bytes at different times to the server. Some of them translate into ASCII, while others are non-printing. I get confused here, because in the examples, it looks like we're just sending plain lines of text. Do we need byte strings? Do we need /r/n newlines?

And server timestamps would be a huge help! There are multiple options here, really. First, you could just timestamp very accurately, so the client can recreate the actual order flow precisely. Alternatively (and probably easier), just create a long integer ascending order number that goes along with the trades and orders. That way, the server can stay on whole seconds for timestamps, but the client can arrange trades and orders accurately using the id numbers. There should be separate numbering lists for orders and trades, of course.

[grondilu]

Damn it...

So many things to learn, and only one life

[slush]

Is websocket api down?

Code:

$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

[memvola]

Is websocket api down?

It's down for me for at least the last hour.

And thanks for the nice explanation, MagicalTux. I wrote a client in haskell, I'll post it when the subscription part is complete.

[jed]

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

[MagicalTux]

Is websocket api down?

Code:

$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

Yep was down, found the reason, fixed the code and restarted it (division by zero when someone sent badly formatted draft-00 headers).

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field 

a) The ticker is the current stats about bitcoin. It's sent once each time it have a chance to be changed, which includes when orders are posted, removed or completed.
b) Here's an example:

Code:

{"op":"unsubscribe","channel":"xxx"}

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

Not at this point.

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

Because it was easier since I intend to use this system for many other sites, without having to do something to isolate users of a specific system. Also there are user-specific channels and many other stuff which makes use of uuid a bit easier (for me).

[memvola]

Here's the haskell code which connects to the server, subscribes to the private channel and unsubscribes from the ticker. No json parsing or error handling.

Code:

module Main (main) where

import Network
import System.IO
import Control.Monad

clientHandshake :: String
clientHandshake = "GET /mtgox HTTP/1.1\r\n\
                  \Upgrade: WebSocket\r\n\
                  \Connection: Upgrade\r\n\
                  \Host: websocket.mtgox.com\r\n\
                  \Origin: null\r\n\
                  \\r\n"

-- Expected server response
serverHandshake :: [String]
serverHandshake = ["HTTP/1.1 101 Web Socket Protocol Handshake",
                   "Upgrade: WebSocket","Connection: Upgrade",
                   "WebSocket-Origin: null",
                   "WebSocket-Location: ws://websocket.mtgox.com/mtgox",
                   "WebSocket-Protocol: *"]

-- https://mtgox.com/code/getKey.php
key :: String
key = "\"key\":\"00000000-0000-0000-0000-000000000000:0000000000000000000000000000\""

hGetHeader :: Handle -> IO [String]
hGetHeader h = do s <- hGetLine h
                  if s == "\r"
                    then return []
                    else do s' <- hGetHeader h
                            return (init s : s')

hGetFrame :: Handle -> IO String
hGetFrame h = do c <- hGetChar h
                 if c == '\xff'
                   then return "\n"
                   else do c' <- hGetFrame h
                           return $ if c == '\x00' then c' else (c:c')

hPutFrame :: Handle -> String -> IO ()
hPutFrame h s = hPutStr h $ '\x00' : s ++ "\xff"

main :: IO ()
main = do h <- connectTo "websocket.mtgox.com" (PortNumber 80)
          hSetBuffering h NoBuffering
          hPutStr h clientHandshake
          hdr <- hGetHeader h
          when (head hdr /= head serverHandshake) .
               error $ "Invalid server handshake:\n" ++ show hdr

          hPutFrame h "{\"op\":\"unsubscribe\",\"channel\":\"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\"}"
          hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

          forever $ hGetFrame h >>= putStr

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

[MagicalTux]

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this

[memvola]

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this

Well, anyway, I was actually hoping to be able to keep market depth up to date without polling, like I can do with recent trades using the trades channel, but it seems there's currently no way to do the same with orders. Ticker data doesn't report new or deleted orders, or at least I failed to identify them. I don't want to ask for updates like crazy. Are you planning to introduce a new channel for this?

EDIT: Oh, Keefe already asked for this but I missed it. Nevermind then...

[Clark]

So I've answered some of my own questions:

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

I see there is a new market depth channel as well.

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending

Code:

{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

and

Code:

{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Here's an idea for an initial documentation page:
List the possible fields contained in the JSON return objects, starting with op values and going from there. This is what I have so far, from seeing different things come across the socket:

Code:

op:remark
  message:<message text>
  success:<success boolean>
op:subscribe
  channel:<channel uuid>
op:unsubscribe
  channel:<channel uuid>
op:private
  channel:<channel uuid>
  origin:broadcast
  private:depth
    depth:{
      volume:<volume>
      price:<price>
      type:<order type>
    }
  private:ticker
    ticker:{
      high:<high price>
      low:<low price>
      vol:<volume>
      buy:<buy price>
      sell:<sell price>
    }
  private:trade
    trade:{
      date:<int time>
      amount:<amount float>
      type:trade
      price:<trade price>
   }
  private:order_add
    order_add:{
      oid:<order id int>
      price:<limit price>
      date:<int time>
      amount:<amount>
      status:<status int>
      darkStatus:<0 or 1>
    }
  private:order_rem
    order_rem:{
      oid:<order id int>
    }

[MagicalTux]

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending

Code:

{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

and

Code:

{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Just added:

Code:

{"op":"mtgox.subscribe","type":"trades"}

Possible types: trades, ticker, depth

[slush]

Example usage of this API from Python: MtGox->Sierrachart feed

[0x6763]

Occasionally the websocket sends a depth message to remove an ask or bid (has a negative volume) with a "null" price.  I've only noticed this right before some (not all) trade messages.  I haven't yet seen it happen at volumes unrelated to a trade.

Could it maybe be a race condition in your code where most of the time it gets the depth information before the trade clears it, but sometimes it doesn't?

[0x6763]

Now that I've looked at more instances of it, the volumes are not always obviously related to the subsequent trades, though I think they're related at least most of the time, since these nulls only seem to occur right before a trade or throughout a series of trades.

Here are some examples of the "null" depth message with some surrounding depth/trade info (I removed some messages coming in around these same times that did not seem relevant):

Code:

REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-56},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -56 @ nil
TRADE 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-46},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -46 @ nil
TRADE 17 @ 1.1827
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-29},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -29 @ nil
TRADE 23.216 @ 1.18151

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-5.784},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -5.784 @ nil
TRADE 5.784 @ 1.1621

Code:

ASK 0.026 @ 1.1875
 ASK 9.974 @ 1.1875
 ASK 0.03 @ 1.1875
REMOVE ASK -9.97 @ 1.1875
 BID 1 @ 1.18751
 ASK 0.026 @ 1.1875
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-9.974},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -9.974 @ nil
TRADE 1 @ 1.18751

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-4},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -4 @ nil
TRADE 4 @ 1.18751

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-19.966},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -19.966 @ nil
TRADE 19.966 @ 1.18751

[MagicalTux]

Hi,

I have located the source of the null and fixed it


Mark

[0x6763]

@MagicalTux

When do you think you'll be able to stop broadcasting dark orders and orders with insufficient funds on the websocket?

[jed]

This depth channel is broadcasting the changes to the depth. It would be nice if you could send something in the beginning to get the current depth table in the same format. Then you can use this feed to change the original depth table as time goes on.
I know I can get it with the old API but it would be nice if the messages were in the same format.
Also is there a point to all these extra fields: "op":"private","origin":"broadcast","private":"depth" ?

[MagicalTux]

Hi,

Broadcasting of depth changes fixed, I think (mostly).

As for the extra fields, they are there mainly to avoid mistaking the messages for something else, however you can safely ignore them

[0x6763]

It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!

[MagicalTux]

It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!

Fixed notices for orders without enough funds.

As for the trade price, it'll be fixed in a few days when we switch to a new backend~

[jed]

I noticed that sometimes the price is sent in quotes:

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

[MagicalTux]

I noticed that sometimes the price is sent in quotes:

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

When the price comes from mysql, it is as a string. I could cast it to float to make sure it's a float, but I guess it doesn't change much for anyone (shouldn't, anyway)

[jed]

Shouldn't the order_add message tell you if it was a buy or sell order?

[jed]

Anybody else occasionally miss trade messages?

[molecular]

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

Hmm, I also tried this. Can't seem to succeed.

What port do I have to use? I tried like this:

Code:

from pywsc.websocket import WebSocket
ws = WebSocket('ws://websocket.mtgox.com:80/mtgox')

This bails with "error receiving" in receiver.py. I inserted debugging output to see what it's receiving. It seems to receive the <html/> page you get when you request http://websocket.mtgox.com/.

I'm confused and obviously doing something wrong. Can someone help?

[Clark]

Well, I had to change my WebSocket._send function to do this:

Code:

self._sendRaw('%s%s%s' % ('\x00', data, '\xff'))

because it was breaking the socket trying to send a non-string.

My receive code is identical except that I added a Threading.Event for that loop instead of just while True.

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

[molecular]

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

SOLVED! Thanks for your help.

Stupid me forgot about that transparent squid proxy.

Am receiving messages, now to the fun stuff!

[Clark]

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

[toffoo]

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

Fantastic news.  Please let us know when you have something ready to test out.

[molecular]

ok, I'm receiving messages, nice

Now I want to update the market depth info, which I'm initially getting through http://mtgox.com/code/data/getDepth.php.

so I'm listening on channel "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe" for messages that have op="private", and am getting depth updates like this:

Code:

{u'volume': -10, u'price': 5.4000000000000004, u'type': 2}

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Above message is an example of a message that caused this... I didn't have an entry for price "5.4000" (I'm using 4 decimal places for the index in my depth "dictionary", but that should do no harm, right), so the result is a new entry with negative volume.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

EDIT:

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

If my problem actually stems from the two timepoints (initial depth request, start of depth updates) not being equal, I would consider this to be not a nice-to-have but a must-have.

You could maybe simply (from client point of view) send the complete depth table as a burst of updates after websocket connect?

[Shrapnel]

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

I'm also wondering the same thing. I think you can't unless the API send the complete data first.
I'm trying to do a realtime order book but it's currently very hard due to dark pool and sync issues.

[molecular]

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

[jed]

I think it doesn't send every update to the order book. It seems like some random ones get dropped.

[0x6763]

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

Mtgox has some bugs.  The websocket often entirely skips sending some depth messages, sends some at a slightly different price than what they should be, and the APIs (or site in general) has rounding inconsistencies.  I've only partially worked around these issues by rounding all depth message prices to the 5th decimal place and volumes to the 3rd decimal place, and creating a button in my program that I can click to manually refresh the entire depth tables when I start noticing odd stuff.

An example is where a bot will download it's own open orders, see that one of it's orders is something like 6.253, but it wants it to be 6.252999, so it removes the order for 6.253 and creates a new one at 6.252999, and both of those come over the websocket just like that.  And the bot does this over and over again because the API for getting your own open orders rounds to the 4th decimal place, but the websocket will do at least 6 decimal places for prices and more than that for volume, so your table ends up removing the order at 6.253 over and over again, resulting in a negative number, while it keeps adding orders to 6.252999 over and over again, making some huge number that's not accurate.  The bugs and inconsistencies are pretty annoying.

[Keefe]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

[molecular]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

MagicalTux, can you give us an estimate when you'll get around to implementing a solution to these problems or talk to us about a solution?

These problems are really not acceptable: we can't have the client side depth data out of sync or even worse stuff happening (missing data?)

I'm here to help with testing.

[jaybny]

my program is listening to the WebSocket and publishing to twitter once an hour, on new highs or lows.

im using c# and am not  using JSON schema, just parsing directly... so if anything changes - I will break..

http://twitter.com/newberg_feed

[Ycros]

Herp, disregard what I posted before, I found my client library was sending an extraneous \r\n\r\n on the end there.

[molecular]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

I'm seeing a slight problem for MagilTux to solve the "keep client depth tables in sync" problem, I hadn't thought about:

(Firstly, wether a client gets the initial depth data through the "classic API" or through an initial burst of depth-change messages doesn't make a difference for my thinking here.)

Problem is: the depth data a client receives initially is chopped off at the ends at a certain distance to the current last trade value.

Now whenever a trade happens, this window moves and some trades exit the window on the one end while others become "visible" on the other end.

There's 2 ways (at least) to solve this:

• Screw that window completely, give the complete depth data to clients
  
• On trade, determine entering/leaving orders and send them as new/cancelled over the channel
  

I'd go for the first option in combination with initially bursting change-messages instead of using the "classic API" to initially fill client depth table.

[toffoo]

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

[mindtheft]

A quick js chrome extension hack for WebSocket -

https://github.com/mindtheft/mtgox-Chrome-Extension

You can use the crx file to install the extension, or better yet check the source code and pack your own crx file to make sure there's no foul smells from mine.

[0x6763]

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

Here's a naive Mt. Gox websocket client I wrote in python: http://pastebin.com/FH6eWhFm
By default it just prints out trades as they happen, but you can modify it to do whatever you want.  If someone really wants an event-based version or something to make it a little easier to work with, you might be able to convince me (with a few bitcoins ) to put something together with pywsc or something.  I may eventually release an executable jar (clojure/java) file for those willing to buy a copy that will give you a GUI to view live trades and depth changes (though mtgox's websocket is still quite buggy in respect to depth changes...there would be a button to let you do manual refreshes as well).

[martin]

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

[0x6763]

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

Mtgox currently has multiple bugs in the websocket's depth channel...your tables are behaving exactly the same as mine are.  Mtgox is moving to a new system in june which will hopefully have all of these problems taken care of.

[martin]

Ah excellent, any information on the form of this new system?

[zelyony]

I wrote some code in C# for MonoDevelop or Visual Studio
MtGox & IMtGoxHandler & TestMtGox - for making orders
  FetchCash
  FetchOrders
  Place/CancelBuyOrder
  Place/CancelSellOrder
  .Handler - interface for responses & errors
WebSocket & Program - for read trades events
  .OnData - delegate received string (in JSON format) for any channel

https://rapidshare.com/files/3646016978/MtGoxApp.zip

[zelyony]

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

technically:
JSON changes to this: add new attr to "depth": "abs"=X: where X = "1" for absolute volume; "0" or absent attr - relative volume as now
That data can be cashed on server for all new clients. Depth changes updates cache. Need to generate JSON (for sent) on request only - more expansive operation than just update some structs in cache.

PS
Sorry for my Russian English

[zelyony]

maybe
needs some authorization vs WebSockets' flood from bag guys (direction: from client to server)

something like this:
- some cookie received from Trade API for websocket connection to MtGox. this cookie valid only next hour (5-10-30min)
- on new connection to websocket client must send this cookie to server. if cookie is valid automatic subscribing will do..
cons: more difficult process for connection

or:
- using SSL/TLS encrypted WebSocekts.
pro: can refuse polling Trade API and do all orders and receive statuses over WS
cons: need implementations for different languages

[zelyony]

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?
for price enough.. 2 digits for cents.. and.. ok, let be 4digits after dot
for volume.. hmm.. similar numbers less than 0.001 are considered as zero.. = so 3 digits after dot for volumes

[martin]

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?

We need as many digits as people can specify, which as far as I know is an unlimited amount.

[zelyony]

I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.

good variant (1)
but people dont want to calculate depth state. they often will request the one. more traffic, slow server, bad trading.

variant (1.5)
every depth message contains all state. without unnessary attrs - one number is price, next in pair is volume with right sort order: "depth":{"asks":[X.XXXX, V.VVV,...],"bids":[...]}

another variant (2):
- new channel for depth-20(or some N) values asks & bids every one minute (only for checking purpose or as starting point)
- clear description of algorithm how calculate depth state over trade & depth channels

one more variant (2.5):
- depth values is absolute forever
- new attr "pos" with values { +N, N, -N } where
     +N - new row inserted at N pos, rows [N..end] shifted down
       N - updates values ar row N
      -N - row N deleted. other rows shifted to close gap (to up)
- "trade" values ​​must match the changes in "depth" values (a priori)
- but still need starting point - current depth state - use variant (2) or something
pros: simple calcs, asks & bids always sorted by server(that must be), traffic +1attr, CPU calcs almost same(as now)

"make your choice!"

We need as many digits as people can specify, which as far as I know is an unlimited amount.

now: values with volume less 0.001 remains in my depth state after process depth & trades from corresponding channels. but next ticker does not account for their. so, I delete them from calculated depth state.

[leepfrog]

This might be a stupid question but I dont fully understand the meaning of the ticker.php api.
It returns something like that:

{"ticker":{"high":7.51,"low":7,"vol":33895,"buy":7.257,"sell":7.3499,"last":7.2591}}

What exactly is high and low? the rest seems pretty clear but I am not sure about those two

[Clark]

The ticker high and low are probably those respective prices for the current trading session, which I'm guessing is a full day. You should watch the ticker values for when the volume resets to zero, as that would be the beginning of the trading session.

[leepfrog]

Makes sense - I'll watch it. Thanks!

[Clark]

I'm not sure it's been mentioned before, but the data type of the getDepth.php return does not seem to be application/json, so it is not possible to do a cross-domain call to MtGox for the depth of market data.

This would be a really easy and quick fix in the getDepth.php code...

[robottwo]

Hi,
    The websockets API is *great*. I have a suggestions for the protocol:


The size in the depth messages should publish the new absolute size, not the relative size. Subscribers can still figure out the relative size change by subtracting from the previous update at that size, but this provides better error recovery on the client side.

[zelyony]

hint (from my bugs)
see MSDN Math.Round( decimal d, int decimals ), which I use
MtGox rounds numbers NOT SAME:
If the value of the first digit in d to the right of the decimals decimal position is 5, the digit in the decimals position is rounded up if it is odd, or left unchanged if it is even. If the precision of d is less than decimals, d is returned unchanged. The behavior of this method follows IEEE Standard 754, section 4. This kind of rounding is sometimes called rounding to nearest, or banker's rounding. It minimizes rounding errors that result from consistently rounding a midpoint value in a single direction.

some datas:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":30.22425,"type":1,"volume":1},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":30,"high":31.89,"last":30,"low":18.6,"sell":30.2243,"vol":98781}}

[zelyony]

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

Which min volume of order required for trading (min step size)?
I saw volume=0.007 in depth channel, but ticker channel ignores that values (so values lessOrEqual 0.007 I ignore)

[jaybny]

is websockets down? any ETA?

thanks

[af12345]

I think the API shows a lot of promise, but it's seems to be missing one key detail --- without publishing an order-id on the depth message, and the constituent order-ids on the trade message, it's not really possible to ever get in sync.

The reason is as follows: if you start your connection and receive an order @ 25 for volume 10, and then there is a trade @ 25 for volume 5, you can't tell if the trade was for part of the order you just received, or an earlier order (pre your connection time).

If, however, you were to provide an order-id on the depth message, and the pair of order ids on the trades, then all ambiguity is removed.

[Clark]

Here are some things I've noticed after building some web applications using the WebScokets API:

• Switching to the Secure WebSocket (wss://) would dramatically improve compatibility, especially for those behind a proxy
• Security would also allow users to send trades via the WebSocket, which would simplify trading to one connection instead of polling+WS.
• Send the prices and volumes as integers! Bitcoins are stored as integers in the client but displayed as floats by dividing by 1e8. Some of the prices and volumes quoted over the WebSocket are floating point numbers with much more precision than the 8th decimal, which doesn't make sense in the framework of Bitcoin overall.
• The ability to request the depth of market table entries through the WS
• More consistency in depth of market updates - I have to poll the data every so often in order to eliminate discrepancies, which is not very robust and could be misleading to people using my service.

[zelyony]

it will be good create IRC-channel

• for talks
• for "websockets OFF for hour"
• for bans in websockets (if exists)

at now
opened socket, sent handshake, received nothing - conn reset by peer
and I dont know what happened and how long it will last

[kseistrup]

• Switching to the Secure WebSocket (wss://) would dramatically improve compatibility, especially for those behind a proxy
• …

+1 for wss://

Cheers,

[matt.collier]

Hello all, developing in Java here.

Aiming at creating an interface similar to Trade Station's Matrix: http://www.tradestation.com/strategy_testing/market_depth.shtm

I'm using Open Office Calc as the front end.  I've got the real-time data flowing into calc nicely, but I've run into the same sync issues everyone else is having.

As others have done, I'm bootstrapping my depth info with http://mtgox.com/code/data/getDepth.php, but before long, none of the volume information makes any sense.

I'll be happy to share my source with any interested parties.  For now I'm going to put this on the back burner until a solution to the sync issue has been implemented.

I'll put my vote in for a channel that publishes absolute volume information in addition to some effective bootstrapping method.

[bitoption]

I'm getting deafening silence from my python websockets client today. This is sad, since I use it to give pricing cues on bitoption.org; I've noticed some websockets-gox stuff is working right now, some is not.

Any indications as to what's what and why?

[Clark]

It would be nice to hear from MagicalTux about the progress of the back end rewrite and deployment. Also, a formal documentation of the WebSockets API is most welcome.

I'm still seeing the feed at the moment, but it stopped a little earlier today. Very odd.

[netxshare]

I am also having issues, my current connection to the websocket is showing data but any new connectionss fail to get anything more then a response header.

[zelyony]

found site with fast check websockets (and wss too)
http://websocket.org/echo.html
browser must support websockets (or enabled it)

for now MtGox-WebSockets requires Sec-WebSocket-Key1 and Sec-WebSocket-Key2 and 8bytes-data

at now (UTC 19:27 13-jun) WS dont responds with 16bytes of hash

and at now (UTC 20:37 13-jun) no connection

at now (UTC 20:35 14-jun) System.Net.Sockets.SocketException: Connection refused

[molecular]

I'm getting deafening silence from my python websockets client today. This is sad, since I use it to give pricing cues on bitoption.org; I've noticed some websockets-gox stuff is working right now, some is not.

Any indications as to what's what and why?

There have been 2 periods where no trades happened and the websocket was dead, apparently some mysql troubles. Maybe you stumbled into one of these.

[zelyony]

today I saw MtGox Live for Android

and here is screen of my iPhone app (not in AppStore yet) - depth state
http://img27.imageshack.us/img27/1115/screenshot20110616at223.png

in first version will be depth state & trades only, no wallet & orders

[kseistrup]

today I saw MtGox Live for Android

I have a feeling that the app is polling Mt. Gox every 5 seconds, rather than using the websocket API.

Something that would be immensly cool is an android app with the same graphs as on MtGoxLive, using websockets.  AFAIK there are no browsers for android that supports websokcets, so I can't just load the web page.

Cheers,

[ezl]

recommendation regarding the depth feed -- , send updates instead of diffs.

 the bandwidth is ~same, but by broadcasting updates, the penalty for missing a message only lasts as long as the interval until the next update for that price level. clients can compute diffs if they want.

it also makes it so that the listener can theoretically construct the full book just by listening to the updates instead of repolling the old api and hoping the messages are in sync

[dev^]

Hi,

is it possible for upcoming version to rename the field "private" in anything other, which isn't a protected keyword in Java?  

Currently it isn't possible (at least for me) to read this field with any common JSON parser library, e.g. Google's Gson.
(of course, it would be possible to set this field's content via text parsing, but this isn't the right way to do so...)

> Something that would be immensly cool is an android app with the same graphs as on MtGoxLive, using websockets.

If someone can develop any nice looking GUI for Android devices, I could provide the complete Java part regarding the WebSockets - it seems to work quite fine.  

Best regards
dev^

[zelyony]

at now
for determine connection lost (iPhone hasn't keepalive ops for socket) I do next:
if App dont recv any datas on channels over 6seconds it send:

"{"op":"ping"}"

if connection is ok than server responds to App:

{"message":"Unknown command","op":"remark","success":false}

if no any responds in next 6seconds (include other datas on channels) App decides that current connection lost and try to create new connection (with close old)

NB: "ping" dont send every 6seconds! only if at last 6seconds no datas (including "Unknown command")
otherwise App works but nothing shows over hour on broken connection (turnoff WAN at router). I dont known default values in iPhone keep-alive logic.

and my suggestion:
maybe add "good" response:

{"op":"pong"}

it dont requires any logic or DB-access. simple response for current connection
although its works well now

[dds]

Looks like MtGox WebSockets broken after reopening - connection can be established but no data received.
Mark, are you going to fix it?

[molecular]

Looks like MtGox WebSockets broken after reopening - connection can be established but no data received.
Mark, are you going to fix it?

+1

also: I can't seem to be able to use the classic Trade API, either. Says something about a "bad token". Anyone know about that?

[kseistrup]

I can't seem to be able to use the classic Trade API, either. Says something about a "bad token". Anyone know about that?

Look at the Trade API page on your account — it says that it will be enabled soonish.

Cheers,

[zelyony]

good! websocket works for now!

some questions:

{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private", "origin":"broadcast","private":"ticker", "ticker":{"avg":15.469053268,"buy":15.63,"high":15.689,"last":15.63001,"low":15.311,"sell":15.63001,"vol":15679}}

"avg" is average volume by trade?

{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade", "trade":{"amount":2,"amount_int":"200000000","date":1309710218,"item":"BTC","price":15.632,"price_currency":"USD","price_int":"1563200","tid":"1309710218345711","trade_type":"ask","type":"trade"}}

what mean "trade_type":"ask"? trade raised by ask order?

and where is depth channel? no datas for it. will it appear in the future?

[dev^]

good! websocket works for now!

Yay! :-)

"avg" is average volume by trade?

I think it's the average price. (As I guess by it's value...)

what mean "trade_type":"ask"? trade raised by ask order?

and where is depth channel? no datas for it. will it appear in the future?

Looks like this is "the new" depth-channel...

Best regards,
dev^

[zelyony]

and where is depth channel? no datas for it. will it appear in the future?

Looks like this is "the new" depth-channel...

my English is not so good. where is "this "the new" depth-channel" ?
EDIT: on #mtgox they say me that "should come soon"


and I try to get key for personal channel via https://mtgox.com/code/getKey.php (as say in post 0). it returns

{"error":"Bad token"}

EDIT: maybe same (come soon)

[kseistrup]

My ws client has been trying to connect for an hour now, is the websocket down for me only?

[dev^]

My ws client has been trying to connect for an hour now, is the websocket down for me only?

It's down for everyone.

Code:

java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
(...)

MtGox should really think about some QA or at least a 2nd developer in it's team...

[molecular]

there's a wiki now: https://en.bitcoin.it/wiki/MtGox/API#Websocket_API

[vector76]

I guess since there is not a good way to synchronize the market depth websocket with the getDepth.php, I could fetch getDepth multiple times and watch the websocket for deltas and try to correlate them.

I'd have to agree with ezl's suggestion of sending the total volume instead of the deltas.  Then there would be no synchronization problem.

Are people still seeing problems with the depth websocket or have those issues been fixed?  I guess I'll fetch getDepth every 5 or ten minutes just in case..

[divebubble]

Reading the different channels, updating data all works perfect... but -

how can i connect my account to get my trading information?

sending
{"op":"mtgox.subscribe","key":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"}

will always result in

{"message":"Invalid MTGOX key","op":"remark","success":false}


XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX is the API Key from the https://mtgox.com/users/settings page.
Or are i'm compleatly wrong?


Thanks.

[throughput]

Websocket API is cool for receiving trade updates! Thanks!

But is it possible to receive updates on my orders anyhow?
If that is implemented, then I don't get the meaning of the "(partial key)" in

Code:

{
  "channel":"(partial key)",
  "op":"private",
  "order_upd":{
    "amount":1000,
    "darkStatus":0,
    "date":1302836027,
    "oid":"(oid)",
    "price":0.9899,"status":1
  },
  "origin":"broadcast",
  "private":"order_upd"
}

(example from https://en.bitcoin.it/wiki/MtGox/API#Websocket_API )

I don't understand how should I implement the authentication to subscribe to my private channel.

Please, can anybody provide more details on:

 1. Does it work at the moment?
 2. How the authentication is implemented?

Also, it would be nice to be able to request the full depth to receive them as depth messages.

[Fireball]

You have a small bug in the reply header:

Code:

HTTP/1.1 101 WebSocket Protocol Handshake

should be the first line, and in your case it's

Code:

HTTP/1.1 101 Web Socket Protocol Handshake

(yes, the additional space between Web and Socket). Nothing major, however if some software validates the header, it won't be recognized as a valid one. Example of such real-world failing app: websocket-sharp. Other are probably just failing silently too.

[piotr_n]

It happens very often that there are consecutive "depth" messages, where one comes immediately after another and they both have the same amount/price and the opposite sign. So effectively they do nothing, except causing confusion and an unnecessary traffic.

For instance:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"9.15417","price_int":"915417","type":1,"type_str":"ask","volume":"-0.09283","volume_int":"-9283000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"9.15417","price_int":"915417","type":1,"type_str":"ask","volume":"0.09283","volume_int":"9283000"},"op":"private","origin":"broadcast","private":"depth"}
...
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"-0.2","volume_int":"-20000000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"0.2","volume_int":"20000000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"-0.2","volume_int":"-20000000"},"op":"private","origin":"broadcast","private":"depth"}
...
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"-0.79287883","volume_int":"-79287883"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"0.79287883","volume_int":"79287883"},"op":"private","origin":"broadcast","private":"depth"}

It seems like some glitch and it would be nice if could be fixed one day.

[zelyony]

It happens very often that there are consecutive "depth" messages, where one comes immediately after another and they both have the same amount/price and the opposite sign. So effectively they do nothing, except causing confusion and an unnecessary traffic.

For instance:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"-0.79287883","volume_int":"-79287883"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"0.79287883","volume_int":"79287883"},"op":"private","origin":"broadcast","private":"depth"}

It seems like some glitch and it would be nice if could be fixed one day.

imho, all changes must be. human's or robot's..

[piotr_n]

imho, all changes must be. human's or robot's..

agree. but these dont look like robot changes, do they? why would a robot do something like that? the only thing i can think of: to DoS the server

[piotr_n]

The WebSocket API does not work with the latest Chrome.

Chrome dev claims that the problem is at mtgox side.

It seems that websocket.mtgox.com is sending out a frame in old format (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00). Chrome 14 uses new version WebSocket protocol (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-08). We've dropped old protocol and are asking servers to adopt new one. mtgox is replying new version handshake correctly. Only new frame support looks broken.
http://code.google.com/p/chromium/issues/detail?id=97026

Please fix.

[AV]

Hi, MagicalTux !

The WebSocket API really does not work with the latest (14.0.835.163) Chrome.


For example http://bitcoin.clarkmoody.com/
WebSocket Closed after 2 second.

Help ! Help ! Help !

Thanks.

[speeder]

Yay! Correct thread to complain about that!

Indeed, me too want clarkmoody on Chrome 14!

(thankfully at work Chrome is still stuck on 13)

[bzzard]

yeah, this should work?
I know that ws://mtgoxlive.com:3456/channel works

MtGox! Anyone there?

[paraipan]

sup MagicalTux, any plans to update websockets to draft-10 any time soon ? Thanks

[Ente]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

How about these and the wss://?
Its been months that Mark didnt post here now..

And, general question:
I chose this API as a starting project to learn ruby.  Maybe a bit (too) ambitious, but would at least be a useable project, if it ever gets finished..
Are there real advantages of this "new" websocket API over the old HTTP API except that I dont have to poll any more and have instant updates (instead of every 10 secs)? I would at least not have the syncing issues?

With the probably still unsolved issues I hesitate to invest much more time into this - being a novice I still am trying to actually post to the websocket.. *rolleyes*

Thanks,

Ente

[Clark]

I found the part of the draft where Chrome 14 is failing:

http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10#section-4.2

Apparently, MtGox is setting the RSV2 bit in the responses to the client. Thus, according to the spec, Chrome is required to FAIL the connection.

Using a packet sniffer, I can see the subscribed responses from the WebSocket after connecting, but the message never gets through and Chrome fails and closes the connection.

This is such a simple fix...

[BkkCoins]

Does the "own" channel work at all?

I can't figure out how to get data about my own trades/orders. I've dumped all the data streamed and there is no data not matching the default channels. I tried subscribing using a key of my own account. That doesn't work either. It seems like this just doesn't work yet.

It would be really useful to see when your own orders/trades occur. If this does work then is there a web page somewhere that says more than the one line on the wiki?

*** Edit:

Another thing I noticed now is multi-currency trades coming thru. I don't quite understand these:

19:07:57 BUY  22.81 @ $2.63192 GBP
19:07:57 BUY  22.81 @ $25.6 CNY

I see British Pounds and Chinese Yuan. But why twice and how does this work - is someone buying with Pounds and settling in Yuan or something? I realize that this must be the same trade since the  GBPCNY exchange rate is now about 10.33. Just not sure how I can handle this as info if the USD isn't indicated.

Thx.

[qxzn]

I have not been able to get the "own" channel to work either. If anyone has, please share how!

[Cluster2k]

Subscribing to my own channel doesn't work either.  I'm using VB and tried sending ANSI strings to MtGox in various formats, but as soon as I do I lose the TCP connection.  Sending the handshake right after connecting exactly the same way works fine however.

I have tried:

{'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'op': 'mtgox.subscribe'}
{'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'op': 'subscribe'}
{'op': 'mtgox.subscribe', 'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'}
{'op': 'subscribe', 'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'}

Sending nonsensical data "eg. 'testing 123' to the connection causes a disconnection as well.  Is this standard behaviour if MtGox doesn't recognise the message?

I assume the key is my key listed in my MtGox API section.  I have successfully used the key and secret to retrieve account data using a sample PHP script.

EDIT: Just tried using plain old telnet to connect to MtGox.  Works fine, just like this example:
http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

But as soon as I try to send any subsequent data to the connection I lose it.  The wiki entry at https://en.bitcoin.it/wiki/MtGox/API is non descriptive. 

"Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades). "

There is no description on how someone can subscribe to their own channel.

[qxzn]

You should not be getting a disconnect just for sending an invalid message type. You should be getting a json message back that says something about how your command was unrecognized. I would double check the bytes you're sending. As a test, you can try to unsubscribe to one of the feeds. That works, so you should be able to get it to work.

[Cluster2k]

You should not be getting a disconnect just for sending an invalid message type. You should be getting a json message back that says something about how your command was unrecognized. I would double check the bytes you're sending. As a test, you can try to unsubscribe to one of the feeds. That works, so you should be able to get it to work.

It seems I can't send anything to MtGox after the initial handshake without getting disconnected.

I've tried in telnet too (MS Windows telnet client).  Surely it should work?

telnet websocket.mtgox.com 80
Sent the following text:
GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

After pressing enter the second time I start to get the stream of data.

But when I try to send something like "{'op': 'mtgox.unsubscribe', 'key': 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe}" the connection is lost.  If I send anything at all the connection dies.

I don't get it.  

EDIT: Just tried it again with Putty on Win7.  Type of connection: raw.  Same result.  I can get the streaming data, but as soon as I try to send anything back to the server on the same connection it disconnects.

[qxzn]

You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

[Cluster2k]

You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

Thanks for that.  I'll try putting \x00 before my message and \xFF after and see how that goes. 

[Cluster2k]

You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

That works fine.  I wasn't sending the \x00 and \xFF at the beginning and end of each message.

I can successfully unsubscribe from the depth feed using:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"unsubscribe"}

However when I try to resubscribe I get the following error:
{"message":"Unknown command","op":"remark","success":false}

I have tried all sorts of combinations of subscribe messages, but none seems to work.  I also can't subscribe to my own channel.

The following all fail:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"subscribe"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"mtgox.subscribe"}
{"op":"subscribe","channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"mtgox.subscribe","channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"subscribe","24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"mtgox.subscribe","24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}

I looked at the haskell code and tried that subscription message first, then tried various other combinations.

hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

[Cluster2k]

I have just been informed on #mtgox IRC that websockets are read-only.  Have been for a number of weeks.  Subscriptions don't work and it's not possible to view your own executed trades.

It's possible get around this limitation, but it really makes the system slow and a bit cumbersome to use.  It would be so much more convenient to get instant confirmation of your own trades.

[Clark]

I would like to see depth messages for all of the currencies broadcast out over the WebSocket, as I do not think it's happening right now.

Can anyone confirm that there are only USD depth messages coming across the socket?

[molecular]

I would like to see depth messages for all of the currencies broadcast out over the WebSocket, as I do not think it's happening right now.

Can anyone confirm that there are only USD depth messages coming across the socket?

Looks like it's true. I am filtering for "depth_msg['currency'] == 'USD'", but put a debug log line for the other case. It doesn't trigger.

[Ente]

I am having problems with Websockets and Ruby in general, maybe someone of you is kind enough to help out?
I am beginner with Ruby and chose this as a learning-by-doing project.

Using examples from the web, I am still trying to connect to the websocket (works) and sending strings to it (handshake and channel subscription/exiting):

Code:

require 'rubygems' 'eventmachine' 'em-http-request'
 
EventMachine.run {
  verbindung = EventMachine::HttpRequest.new("http://websocket.mtgox.com").get :timeout => 0

  verbindung.errback { puts "oops" }
  verbindung.callback   {
    puts "WebSocket connected!"
                 
verbindung.send("GET /mtgox HTTP/1.1\n", 0)
  }
}

Error, my Ruby doesnt recognize "GET /mtgox HTTP/1.1\n" as a string to be "send" :

Code:

websocket.rb:19:in `send': undefined method `GET /mtgox HTTP/1.1 (NoMethodError)
' for #<EventMachine::HttpClient:0x7f9f6d22e8b8>

I tried "print", "send", "write", "<<", still didnt get it to work. I am lost, grateful for any hint!

Ente

[runeks]

Anyone else getting timeouts connecting to ws://websocket.mtgox.com/mtgox? I am right now...

[piotr_n]

the server has been dead since yesterday.

[Cluster2k]

Websockets have been dead for about a week.  Nothing on MtGox's support page about it.  Does MtGox even know, or do they care?  It seems not.

[molecular]

Websockets have been dead for about a week.  Nothing on MtGox's support page about it.  Does MtGox even know, or do they care?  It seems not.

MagicalTux knows, has been saying something routes not being up about 3 or 4 days ago.

Does he care? I'm not sure.

I think the websocket stuff is pretty important.

[piotr_n]

http://mtgoxlive.com/ is also suffering from this failure.
The data is shows doesn't seem to be in a real time anymore.
And this service had supposedly been acquired by MtGox.
So he should care

[BTCurious]

http://mtgoxlive.com/ is also suffering from this failure.
The data is shows doesn't seem to be in a real time anymore.
And this service had supposedly been acquired by MtGox.
So he should care

I'm not sure that's true. If I compare it to the fastest monitor bot I know, #bitcoin-RT in freenode, then the number seems to be behind ~10 seconds. But I don't remember this ever being any different.

[piotr_n]

Before you could see new transactions popping up on the right side, in a real time.
And the green price line was also updated in a real time - now the last point of it is always a couple of minutes behind.

[BTCurious]

Before you could see new transactions popping up on the right side, in a real time.
And the green price line was also updated in a real time - now the last point of it is always a couple of minutes behind.

Oh hey, you could be right. It did work via the websocket before. Maybe it works via the http API now?

Edit: Websocket is back. And yeah, the transactions in real time too.

[runeks]

Yay, websocket is back. Thanks Mt. Gox.

[piotr_n]

Yay, websocket is back. Thanks Mt. Gox.

I saw it back for a moment... but it doesn't work now, does it?

[molecular]

Yay, websocket is back. Thanks Mt. Gox.

I saw it back for a moment... but it doesn't work now, does it?

It's working for me in my own client. Thanks MagicalTux!

[piotr_n]

It's working for me in my own client.

OK - thanks for the info.
This made me thinking... and it seems that I upgraded my websocket libarary to a new version.
It's OK when I go back to the old version. Its probably the same problem as with a new Chrome.

[molecular]

It's working for me in my own client.

OK - thanks for the info.
This made me thinking... and it seems that I upgraded my websocket libarary to a new version.
It's OK when I go back to the old version. Its probably the same problem as with a new Chrome.

Thanks for that hint. I'll keep that in mind, cause I'll probably switch websocket impl at some point (currently using some hacked-up thingy someone gave me).

[runeks]

Yay, websocket is back. Thanks Mt. Gox.

I saw it back for a moment... but it doesn't work now, does it?

Works for me right now as well.

[slush]

Anyone succeed with new MtGox's socket.io interface? I was succesfull with socket.io handshake, server returned "12906784121696250629:15:25:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling". But then I open WS connection to ws://socketio.mtgox.com:80/socket.io/1/websocket/12906784121696250629 and nothing happen, client don't receive any data. Am I doing anything wrong?

[molecular]

Anyone succeed with new MtGox's socket.io interface? I was succesfull with socket.io handshake, server returned "12906784121696250629:15:25:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling". But then I open WS connection to ws://socketio.mtgox.com:80/mtgox/socket.io/1/websocket/12906784121696250629 and nothing happen, client don't receive any data. Am I doing anything wrong?

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.

[slush]

Thanks for info. Is following websocket URL correct?

ws://socketio.mtgox.com:80/socket.io/1/websocket/1290678412169625062

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.

[jothan]

Thanks for info. Is following websocket URL correct?

ws://socketio.mtgox.com:80/socket.io/1/websocket/1290678412169625062

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.

With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

[slush]

With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Yes, you received socket.io handshake response, it's fine. socket.io != websocket.

Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Does it work for you? I dropped idea of get it working after playing with it for hours. Lacking documentation, multiple versions of specifications, total mess.

Then I implemented bitcoincharts socket interface in 15 minutes...

[jothan]

With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Yes, you received socket.io handshake response, it's fine. socket.io != websocket.

Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Does it work for you? I dropped idea of get it working after playing with it for hours. Lacking documentation, multiple versions of specifications, total mess.

Then I implemented bitcoincharts socket interface in 15 minutes...

I can now connect fine, but I cannot subscribe to any channel, I get "invalid request" or "invalid command", so I cannot do anything useful yet. I'm gonna try interfacing bitcoincharts tonight.

[jothan]

Could someone comment on the following points ?

• Is the stream API service currently in a usable state ? (I cannot determine this since I cannot subscribe to any channel).
• What is the proper way to subscribe to a channel ?

Given answers to this, I would be willing to update the wiki with the up-to-date information to clear up the usage of this API.

Heck, i'll give 1 BTC to whomever can enable me to stream prices directly from Mt. Gox with this API.

[slush]

Same status here, lack of any documentation is pretty irritating.

[molecular]

Same status here, lack of any documentation is pretty irritating.

magicaltux has some php sample code that was helpfull for me to code socket.io handshake in python.

(it's still got some debugging output and heartbeat lost detection is not implemented, so on connection drop it doesn't auto-reconnect). oh, and it only does websocket, no other transport.

from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket import WebSocket


class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    data = urllib.urlencode({})
    req = urllib2.Request('https://' + S.url + "/1", data)
    response = urllib2.urlopen(req)
    r = response.read().split(':')
    S.heartbeat_interval = int(r[1])
    print 'heartbeat: ', S.heartbeat_interval
    if 'websocket' in r[3].split(','):
      print "good: transport 'websocket' supported by socket.io server ", S.url
      S.id = r[0]
      print "id: ", S.id

    S.thread = Thread(target = S.thread_func)
    S.thread.setDaemon(True)
    S.thread.start()

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
      
  def keepalive_func(S):
    while S.run:
      time.sleep(S.heartbeat_interval)
      S.ws.send('2::');
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if True:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)

I got that websocket implementation from giel (on irc) ("from websocket import WebSocket"), here it is (put in file websocket/__init__.py):

  http://pastebin.com/jrxachSF

You can start using

  #> python socketio.py

it should dump the messages to stdout

[slush]

    S.ws.send('1::/mtgox')

*headdesk*

* slush bashing all developers writing incomplete documentations and expecting that some things are so obvious they don't need to be in doc.

molecular, you're my savior. What's your donation address?

[molecular]

   S.ws.send('1::/mtgox')

*headdesk*

* slush bashing all developers writing incomplete documentations and expecting that some things are so obvious they don't need to be in doc.

molecular, you're my savior. What's your donation address?

cool to have been of help, especiall to you, slush.

almost didn't post, but so many complained so I did... instant karma

1JX3YkScnu2RBHjUBZTLB6qBgmGeUWxANL

EDIT: slush, are you coming to Prague? I'm there on saturday and maybe sunday, maybe I recognize you

[slush]

No, I'm not coming to Prague...
(...dramatic drums...)
...because I already am in Prague!.

Would be nice to meet you, however I'm myself curious how I'll be able to communicate with my "fluent" english :-).

[molecular]

No, I'm not coming to Prague...
(...dramatic drums...)
...because I already am in Prague!.

Would be nice to meet you, however I'm myself curious how I'll be able to communicate with my "fluent" english :-).

Nice. Hoping to meet you. Thanks for the generous token of appreciation you sent me

[jothan]

   S.ws.send('1::/mtgox')

almost didn't post, but so many complained so I did... instant karma

1JX3YkScnu2RBHjUBZTLB6qBgmGeUWxANL

EDIT: slush, are you coming to Prague? I'm there on saturday and maybe sunday, maybe I recognize you

+1 BTC, Thanks !

[zelyony]

I created project for .NET socket.io + WebSockets at codeplex.com
http://socketiowebsockets.codeplex.com/
for now it can hold ssl-connection with mtgox (no parsing json and etc)

my old ideas (in this topic) died and raised again

[Ente]

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Instead of using socket.io (which I cant get to work) I could as well just connect to the underlying websocket just as before, not? I will not have fallback to longpoll etc in case the websocket dies and the socket.io still works. But besides that, my ticker should happily work with just websockets for the future, not?

Is there a new websocket daemon running now, for socket.io, with a different adress?

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?

Ente

[piotr_n]

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Even though it's "deprecated" it's still much more reliable than the socket.io interface.
So please don't turn it off!

The only problem is that it doesn't work with the latest Chrome and other late WebSocket libs... and nobody wants to fix it.
That's probably why they labeled it "deprecated".

[molecular]

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Instead of using socket.io (which I cant get to work) I could as well just connect to the underlying websocket just as before, not? I will not have fallback to longpoll etc in case the websocket dies and the socket.io still works. But besides that, my ticker should happily work with just websockets for the future, not?

Is there a new websocket daemon running now, for socket.io, with a different adress?

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?

Ente

the way it works: you connect to socket.io server and it will give you a unique ID which will be part of the url you can subsequently use to connect to websocket. socket.io is just a "negotiation protocol".

look at my python code here: https://bitcointalk.org/index.php?topic=5855.msg629600#msg629600

[molecular]

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Even though it's "deprecated" it's still much more reliable than the socket.io interface.
So please don't turn it off!

The only problem is that it doesn't work with the latest Chrome and other late WebSocket libs... and nobody wants to fix it.
That's probably why they labeled it "deprecated".

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.

check out sample implementation: https://bitcointalk.org/index.php?topic=5855.msg629600#msg629600

[piotr_n]

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.

So you see the new "indirect connection" working reliably?

[molecular]

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.

So you see the new "indirect connection" working reliably?

no, not reliable since roughly a week, lots of outages on the websocket part, pretty stable many weeks before that. havent seen the socket.io part fail yet.

[bzzard]

havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

[molecular]

havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

socket.io is just a tool to negotiate a transport layer and keepalive. one of the possible transports is websocket. this means you usually use socket.io PLUS websocket (at least that's how I do it).

I'm guessing a connected websocket with no data flowing means some problem in magicaltux' backend.

any insights appreciated.

[zelyony]

request to "https://socketio.mtgox.com/socket.io/1" doesnt respond (net::ERR_CONNECTION_REFUSED)
Is URL valid?
something happened.. but WHAT happened?..

UPD1:
P.S.
I wrote to Mt.Gox support issue: open some technical twitter channel with info: what? when? how long?
IRC-channel is just flame - no info, no logs, no history, no search..

UPD2 (2:28 AM):
its working now

[molecular]

request to "https://socketio.mtgox.com/socket.io/1" doesnt respond (net::ERR_CONNECTION_REFUSED)
Is URL valid?
something happened.. but WHAT happened?..

UPD1:
P.S.
I wrote to Mt.Gox support issue: open some technical twitter channel with info: what? when? how long?
IRC-channel is just flame - no info, no logs, no history, no search..

UPD2 (2:28 AM):
its working now

yeah, I see this, too from time to time. also a lot of times the connection setup works fine, but there's just no data flow. mtgox datastreaming is on and off on most levels as far as I can tell.

supposedly socket.io is some typical bay-area shit and there's a big rewrite pull request pending for a while now that could fix the issue mtgox is suffering from.

my latest "kind-of-official" info is that the socket.io server is now restarted every 2 hours to mitigate effects of this (and potentially other) issue(s).

I for one am preparing to survive using api requests (switched form api/0 to api/1)

[zelyony]

I see data

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"private","origin":"broadcast","private":"depth","depth":{"price":"5.80025","type":2,"type_str":"bid","volume":"0","price_int":"580025","volume_int":"0","item":"BTC","currency":"USD","now":"1326266472912980","total_volume_int":"19811387"}}    

why exists messages with volume=0 ?
is this rounding errors in MtGox's guts or users posts such orders to exchange ?

+++ existence of "total_volume_int" - is wonderful

[rwcp97]

havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

I have the same problem using the websocket from my C++/Qt-bot - sometimes, usually after some hours working correctly, there is no more data coming in through the socket (but the connection isn't closed by the remote host).
So I check for incoming data and if there isn't anything for 3 minutes, I close the connection and reconnect. Afterwards it's working without problems again. Strange anyway 

[notme]

I'm trying to connect via the SocketIO Ruby gem.  When I try to subscribe to the trades channel the following string is passed to the server: ""\x003:::{\"channel\":\"dbf1dee9-4f2e-4a08-8cb7-748919a71b21\",\"op\":\"subscribe\"}\xFF".  The server response with a json message: {"op":"remark","success":false,"message":"Unknown command","debug":{"op":"client","uuid":"5a1f3b8f-1e03-462b-bd38-e463bb39ff2e","data":{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"subscribe"}}}

The "data" field seems to be the message I intended to submit, but the subscribe fails with "Unknown command".  Any help would be appreciated.

[zelyony]

The "data" field seems to be the message I intended to submit, but the subscribe fails with "Unknown command".  Any help would be appreciated.

my subscribtions (raw socket.io messages over websocket - no explicit frames borders) are:
   "4:::{"op":"mtgox.subscribe","type":"trades"}"
   "4:::{"op":"mtgox.subscribe","type":"ticker"}"
   "4:::{"op":"mtgox.subscribe","type":"depth"}"

[notme]

Thanks... I changed channel to type and subscribe to mtgox.subscribe and now it works.

[Ente]

Is it possible to query my open trades and/or cancel an order or at least all orders? With the websocket API, that is?
Didnt see any hint that this is possible..

But then I dont remember to have ever seen the commands to post orders via websocket neither?

Is the websocket for "reading" only, like making graphs from the data?

edit:
So, get data from the websocket, and exectue/cancel orders via "API version 1" then?

Ente

[rwcp97]

Is it possible to query my open trades and/or cancel an order or at least all orders? With the websocket API, that is?
Didnt see any hint that this is possible..

But then I dont remember to have ever seen the commands to post orders via websocket neither?

Is the websocket for "reading" only, like making graphs from the data?

edit:
So, get data from the websocket, and exectue/cancel orders via "API version 1" then?

Ente

As far as I know there is no possibility to do orders over the websocket... Would be pretty dangerous too, as it isn't SSL encrypted.
But you can subscribe to your "private channel" to be informed about your orders. Use getKey.php to get its name.

[Fireball]

As far as I know there is no possibility to do orders over the websocket... Would be pretty dangerous too, as it isn't SSL encrypted.
But you can subscribe to your "private channel" to be informed about your orders. Use getKey.php to get its name.

socket.io supports secure connections (https, wss transports). That's what I use in ICBIT trading platform, so I would be quite surprised if that's dangerous

[rwcp97]

socket.io supports secure connections (https, wss transports). That's what I use in ICBIT trading platform, so I would be quite surprised if that's dangerous

ok, I wasn't aware of that - I just connect to websocket.mtgox.com at port 80.

[Ente]

Thank you for the hints, rwcp and Fireball!

Ente

[osmosis]

 #> python socketio.py
it should dump the messages to stdout

Thanks molecular.

[is4tomj]

I'm using the streaming feed on my website.  If anyone needs a working example feel free to take a look. (http://BitcoinTitan.com)

[molecular]

Let me say here that the socket.io/websocket feed has been rather stable in late times. I'm hoping this has nothing to do with the fact that trading volume is declining, but is due to Magicaltux making things better. In that case: thanks, Magicaltux and/or mtgox team.

[piotr_n]

I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

[osmosis]

I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Works for me. Sounds like an issue on your side.

[piotr_n]

I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Works for me. Sounds like an issue on your side.

But which language / websocket library do you use?

I tried with latest Chrome and this extension: https://chrome.google.com/webstore/detail/pfdhoblngboilpfeibdedpjgfnlcodoo
.... and also tried with websocket package from Go - and they both disconnect me immediately after sending "7:::1+0"

[piotr_n]

Or just try this page: http://websocket.org/echo.html
... using i.e.  ws://socketio.mtgox.com:80/socket.io/1/websocket/14001291971574386363
and tell me that I'm crazy

[zelyony]

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://

[piotr_n]

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://

shit man.
thank you!!!
that was indeed the problem.

[osmosis]

 #> python socketio.py
it should dump the messages to stdout

Thanks molecular.

For the websocket in python, anyone find a way to deal with the disconnect errors?

File "/home/osmosis/proj_venv/proj_proj/websocket_client.py", line 161, in recv
SSLError: [Errno 1] _ssl.c:1325: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1325: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1325: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1325: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 8] _ssl.c:1325: EOF occurred in violation of protocol
ConnectionClosedException

[Ente]

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?

Some days ago I connected to the old, deprecated websocket. Connecting works, but no data comes in any more.

Now thats an incentive to figure out socket.io :-)

edit:

As far as I know there is no possibility to do orders over the websocket...

Seems like now you can indeed do orders over socket.io:

Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the "version 1" requests from the HTTP API can be called.

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..

Any hints?

Ente

[zelyony]

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..
Any hints?

in the past (3+ weeks ago):
by default u dont subscribed to channels
u may send simple strings over internal (socket.io's) websocket
for "2::" u must send "2::" (or send "2::" every 15secs)
for subscribe u must send post #162
datas stopped in socket after 3-15min.. u can open & sync 2 socket.io..
for now:
post #162 looks no more working - no datas except "1::" & "2::"

[Ente]

OH! Thank you a lot for your help, zelyony!

I was already wondering about the heartbeat. I now answer 1:: and 2:: accordingly, now they keep flowing in constantly.
Well, the socket *must* be working, since the websites using it still work. There would be more outcry in the forum if it was dead altogether..

I am glad I'm not the only one here!

Ente

[Ente]

*bangs head on desk*
I cant get it to work.

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

Here is my (totally stripped) python code:

Code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from websocket import create_connection
import urllib
import re
import time

try:
    import json
except ImportError:
    import simplejson as json


wsurl = "wss://socketio.mtgox.com/socket.io/"
iourl = "https://socketio.mtgox.com/socket.io/"

############

print "Connecting to socket.io.."
url = iourl + "1"
f = urllib.urlopen(url)
output = f.read()
f.close()
ausgabe = re.search('[0-9]+', output, 0)
sessionid = ausgabe.group()

print "New websocket url:"
wssurl = wsurl + "1" + "/websocket/" + sessionid
print wssurl

############

print "Connecting to websocket..\n"

socket = create_connection(wssurl)

while 1:
  echo =  socket.recv()
  if "1::" in echo:
    print "Heartbeat 1"
    socket.send("1::")
  elif "2::" in echo:
    print "Heartbeat 2"
    socket.send("2::")
  else:
    print "Message: ",echo
  time.sleep(2)
      

The websocket client I use here is from http://pypi.python.org/pypi/websocket-client/0.4.
I have a rudimentary heartbeat added here.
The whole script dies eventually, from having stripped all error-handling.
It worked some weeks ago. If you like to see the whole script, let me know.

Bedtime here, will check back in, like, 9 hours..

*frustrated and tired*

Ente

[Ente]

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

I made a new thread for this error-solving bounty:
https://bitcointalk.org/index.php?topic=63969.msg749747#msg749747

Ente

[zelyony]

right now datas arrived (my format after string -> json -> object -> string):
JSON: Packet={ Depth={ price=4,67000 USD,       vol=0,05300000,         dir=bid,        when=2012:02:15_10:03:56.735844 } }
JSON: Packet={ Ticker={ bid=$4.68100,   ask=$4.70075,   lo=$4.21211,    hi=$4.97000,    last=$4.70075,  vol=256,165.42727403??BTC } }

I dont know Python, but:
- on connected (or after recv "1::" - that's mean u connected now) u must send strings like:
4:::{"op":"mtgox.subscribe","type":"trades"}
- on "2::" u must response "2::" (not "1::" as in ur sample)

[Ente]

Thank you for checking, zelyony!
You are right, thats a typo in the snippet. I sent '1::' and '2::' accordingly in my full script, typo corrected.

I sent '4:::{"op":"mtgox.subscribe","type":"trades"}' over the websocket, no change. No response whatsoever, no trade-messages coming in.

Ente

[zelyony]

and right now I dont recv any useful datas  

UPD:
it's working right now (13:44 UTC)

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

To MtGox:
maybe it's sabotage.. Slow Read DoS attack.. bad guys overflow send pool with slow connections and.. voila!
will be better for all clients drop connections that cannot read string, i.e. send to client not finished for.. hmm.. 0.5sec! +ban by IP 10min (epoll, select or other asynchronous mechanism). I took digits from air, I dont know % loading for servers machines and number of connections.
p.s. ping ("2::") works always

[runeks]

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

[Ente]

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

Same here. Sometimes it takes ages to receive any data, sometimes less than a minute. Totally random, and totally annoying. The 1:: and 2:: pings work straight from the beginning.
As soon as I get a response to my channel-connect, data from this channel comes in too.

Ente

[molecular]

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

maybe this helps, my socketio.py:

from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket_client import create_connection
import traceback

class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    try:
      data = urllib.urlencode({})
      req = urllib2.Request('https://' + S.url + "/1", data)
      print 'https://' + S.url + "/1"
      response = urllib2.urlopen(req)
      r = response.read().split(':')
      S.heartbeat_interval = int(r[1])
      #print 'heartbeat: ', S.heartbeat_interval
      if 'websocket' in r[3].split(','):
        print "good: transport 'websocket' supported by socket.io server ", S.url
        S.id = r[0]
        print "id: ", S.id
      else:
        print "error: transport 'websocket' NOT supported by socket.io server ", S.url
        
      S.thread = Thread(target = S.thread_func)
      S.thread.setDaemon(True)
      S.thread.start()
    except:
      traceback.print_exc()      

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = create_connection(my_url)
    
    #S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
    S.ws.close()
      
  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect()
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if False:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)

EDIT: the websocket_client imported I got from here: http://pypi.python.org/pypi/websocket-client/0.4

EDIT2: I've been using this implementation for months and it works well for me. If you spot any problems with this implementation of fix errors or otherwise improve it, please let me know.

[runeks]

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:

  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

[runeks]

Anyone else getting errors like these with molecular's script?

Code:

SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

[molecular]

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:

  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

Thanks, I put that in just in case I need it some day

Anyone else getting errors like these with molecular's script?

Code:

SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

I have no idea, never seen this. But maybe you could try with the websocket_client.py I use (change that import back from "websocket" to "websocket_client" and the get file websocket_client.py from pastebin: http://pastebin.com/CTGj735u). I used to have a lot of troubles and tried at least 3 different websocket implemenations until I had a properly working one.

I _have_ seen "EOF occurred in violation of protocol", but I guess that's just a nasty disconnect...

[runeks]

Cool. Thanks. I'm trying it out now as we speak. Will report back how it works.

[runeks]

I'm still experiencing these errors with websocket_client unfortunately

Code:

08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

[molecular]

I'm still experiencing these errors with websocket_client unfortunately

Code:

08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

I'm experiencing this, too
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:

Code:

08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)

That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.

[bzzard]

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.

I've tested this topic in details and all I can say that lately (~2 weeks) depth data from socketio is useless. Don't trust it.

[runeks]

I'm experiencing this, too
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!

One thing that springs to mind is that upon error, we need to make sure that the existing threads exit, before we start new ones (through calling S.connect() in keepalive_func).
As it is now, keepalive_func will keep running after it has tried re-connecting, and as far as I can see, thread_func will keep running as well. So for each reconnect, we have one extra thread of both keepalive and thread_func running, sending and receiving data on the same socket:

Code:

  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect() #this thread will keep running after this. connect creates a new thread_func, which creates a new keepalive_func
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)

Code:

  def thread_func(S):
    print 'SocketIO: websocket thread started'
   
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
   
    S.ws = create_connection(my_url)

    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
   
    msg = S.ws.recv()
    while msg is not None and S.run:
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      msg = S.ws.recv()
    S.ws.close()

As far as I can tell, this can be fixed by breaking out of the while loop in both threads in case of error, replacing the relevant parts like so:

Code:

   def keepalive_func(S):
      while S.run:
         try:
            S.ws.send('2::');
         except:
            if S.run:
               print 'error sending keepalive socket.io, closing existing connection...'
               S.ws.close()
               print '\ttrying reconnect...'
               S.connect()
               raise
               break
            else:
               print 'exiting socket.io keepalive thread'
         time.sleep(S.heartbeat_interval)

and in thread_func:

Code:

         try:
            msg = S.ws.recv()
         except:
            raise
            break
      S.ws.close()

I will see how this runs and report back.

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:

Code:

08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)

That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

Indeed I was. I had just accidentally switched off the ignore-non-USD-trades option. Fixed now.

[runeks]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

[molecular]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

[Ente]

I tried several hours to create signed commands to send over the socket instead of the old api.
No luck at all. Lastly, I tried to "translate" the php example to python from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands which doesnt look too complicated..

Code:

def get_nonce():
    return int(time.time()*100000)

def encoder(raw):
  dec = json.dumps(raw)
  return dec

def sign_data(secret, data):
    return base64.b64encode(str(HMAC(secret, data, sha512).digest()))


call = "1/info.php"
params = "testparams"
item = "BTC"
currency = "BTC"
id = "1234567890"

Code:

query1 = {'call':call, 'params':params, 'item':item, 'currency':currency, 'id':id, 'nonce':get_nonce()}

{'nonce': 133259594451109, 'item': 'BTC', 'currency': 'BTC', 'params': 'testparams', 'call': '1/info.php', 'id': '1234567890'}

Code:

query2 = encoder(query1)

{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}

Code:

sign = sign_data(secret, query2)

xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==

Code:

query3 = sign, query2

('xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==', '{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}')

Code:

call = {'op':'call', 'call':base64.b64encode(str(query3)), 'id':id, 'context':'mtgox.com'}

{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU 5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMD ksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiL CAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}

Finally I take this last message and paste it in my socket.send:

Code:

socket.send("4:::{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMDksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiLCAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}")

Non-descriptive error-reply:

{u'debug': {u'data': False, u'uuid': u'249bcb53-ccb0-43f1-bac6-e5ae0748e2e1', u'op': u'client'}, u'message': u'Unknown command', u'success': False, u'op': u'remark'}

If someone has enough mercy to have a closer look I would be really happy!
Since this probably takes more than just a few moments, dont forget to quote your bitcoin-address too ;-)

Ente

/who now goes doing some woodwork to clear his head from this

[molecular]

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.

[Ente]

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.

Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

[molecular]

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.

Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

"socket-only"? I know of the "new" and "old" (1, 2) http apis. And I know of socket.io interface for depth/trades (no auth necessary). Care to explain?

[Clark]

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

[Ente]

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

Yep:

Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called.

https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

..with more details and a php example too

Ente

[runeks]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

[molecular]

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:

raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py

Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

I remember while coding it it didn't feel right to detect for reconnect (only) in the keepalive thread.

Thanks for fixing that, please pastebin your version if you consider it robust.

My one is running stably for days now. I might just not care about the 15 seconds.

[runeks]

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

[molecular]

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

[sirk390]

Hi,

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

[Clark]

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

[sirk390]

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

Thanks but what subscribe URL do you mean ?
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .
It sends a request to "https://socketio.mtgox.com/socket.io/1" and then opens a websocket to "wss://socketio.mtgox.com/socket.io/1/1/websocket/{id}"

[Clark]

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

[sirk390]

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

Thanks you! It works now although i'm not using exactly the same URL.
I'm using https://socketio.mtgox.com/socket.io/1/?Currency=EUR .

 

[molecular]

I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .

a newer version of which (with improved thread handling, much more stable concerning reconnects) can be found here https://github.com/molecular/traidor/blob/master/socketio.py

osmosis79 seems to have copied it about a month ago (no objections)

[runeks]

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

Right you are. I think I added it to be able to unsubscribe when a message arrives. But as you say, a global variable works better for this. Fixed in the newest revision.

[knite]

runeks and molecular:

I've reviewed and begun using a combination of your codebases. I have a few questions and comments.

- Why the different approaches to your keepalive threads? Molecular's version is more concise. Is it an issue with the ws.recv() delay on disconnect?
- Is there a reason for using the urllib2 module? The requests module is much nicer.
- Why aren't you using the logging module instead of print statements?
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.
- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

[knite]

Threads are still melting my brain. Can I get the best of both worlds by doing something like this?

Code:

def start_thread(function):
t = Thread(target=function)
t.setDaemon(True)
t.start()
return t

def connect(S):
#setup
start_thread(thread_func)

def reconnect(S):
S.ws.close()
S.connect()

def thread_func(S):
S.ws = create_connection(S.ws_url)
S.run = True
start_thread(keepalive_func)

msg = S.ws.recv()
while msg is not None and S.run:
S.process(msg)
try:
msg = S.ws.recv()
except:
if S.run:
S.reconnect()
break
else:
if run:
S.reconnect()

def keepalive_func():
while S.run:
try:
S.ws.send('2::')
except:
S.ws.close()
break
sleep(S.heartbeat)

The keepalive thread does nothing except send heartbeats and close the connection in the event of an exception. Thread_func reconnects if it hits an exception or exits the while loop, if run is True. Thoughts?

[molecular]

Hi knite, nice to see the code being used / looked at and thanks for your comments. I'll comment on some of them...

- Is there a reason for using the urllib2 module? The requests module is much nicer.

No, not from my side. This (coding traidor) was the first time I did any requesting in python. Tried urllib (didn't work for some reason), the tried urllib2 (worked) so I just stuck with that.

- Why aren't you using the logging module instead of print statements?

Forgive my ignorance of the logging module, will take a look next time I need to dump some info.

On a more general note: there are never reasons for not doing something, there can only be reasons for doing something . Why do you not believe in the flying spaghetti monster?

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience. Let me point you to Goncalo Pinheira (goncalopp, don't know if on forum, he mailed me, will give you his email upon request in pm), who has been asking me about this and if he could use my code (been asking for lgpl) and who seems to want to do exactly that (exchange abstraction)

- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

done

[Seal]

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.

What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.

[daybyter]

Java? I wrote some code for the HTTP API. Maybe we could collaborate for the socket stuff?

[molecular]

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.

What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?

[Seal]

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?

molecular - I currently use a scheduler to execute a PHP call every minute using the http api, however I'm interested in having a live feed for the best prices as they are added to the orderbook. To my knowledge, PHP isnt brilliant at handling streaming feeds and I'm yet to find a way of doing this server-side.

I could stand corrected though, I'm just waiting for a script-kiddy to tell me otherwise.

Whats java like at handling streaming data? How would you run a script like that on a server?

[daybyter]

I would simply run it as a daemon on the server. It would poll every x seconds/minutes . I'll drop you a mail with more details and some code (cannot attach anything here).

[molecular]

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?

molecular - I currently use a scheduler to execute a PHP call every minute using the http api, however I'm interested in having a live feed for the best prices as they are added to the orderbook. To my knowledge, PHP isnt brilliant at handling streaming feeds and I'm yet to find a way of doing this server-side.

I could stand corrected though, I'm just waiting for a script-kiddy to tell me otherwise.

Whats java like at handling streaming data?

Are you talking about streaming from your server to some clients or about receiving mtgox socket.io/websocket stream?

If the latter: http://stackoverflow.com/questions/5783086/java-socket-io-client
If the former: http://code.google.com/p/socketio-java/

I never tried any of these, so I don't know how well they handle things.

How would you run a script like that on a server?

If it's also serving web-pages I'd probably run it in the context of a servlet container like tomcat or jetty.

If not, I'd probably write an init-script that just runs "java -jar myjar.jar" and do scheduling within the app. To be honest, I might just start it in a screen (gnu screen) session if I'm the only one depending on it

[daybyter]

Perfect. We are thinking exactly in the same lines.

I started to write the Java app in a way, that I could run in client or server mode. In client mode, a GUI comes up. In server mode, there's no GUI, so it could be started in an init.d script.

Drop me a PM, if you want to discuss more details.