Bitcoin Forum

Economy => Marketplace => Topic started by: MagicalTux on April 15, 2011, 02:48:21 AM



Title: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 15, 2011, 02:48:21 AM
Hi,

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

You can connect via: ws://websocket.mtgox.com/mtgox

The websocket will subscribe you to some channels automatically:
  • dbf1dee9-4f2e-4a08-8cb7-748919a71b21: trades (each time a trade happens, you get something here)
  • d5f06780-30a8-4a48-a2f8-7ed181b4a13f: the mtgox ticker (lots of updates, with often the same data)
  • 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe: depth information in realtime (price + amount + type... type=1=Ask, type=2=Bid)

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Each message is a JSON-encoded object, with at least "op" element. The "op" element contains the operation to be done (for outgoing messages), or the type of message (for incoming messages).

Possible outgoing commands:

  • unsubscribe Stop receiving messages from a channel (parameter "channel")

Example incoming data:

Ticker
Code:
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":0.9515,"high":1,"low":0.91,"sell":0.9697,"vol":34349}}
Trade
Code:
{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade","trade":{"amount":2.71,"amount_int":"271000000","date":1310279340,"item":"BTC","price":14.43,"price_currency":"USD","price_int":"1443000","tid":"1310279340877902","trade_type":"bid","type":"trade"}}

Contains:
  • amount: the traded amount in item (BTC), float, deprecated
  • amount_int: same as amount, but in smallest unit
  • date: unix timestamp of trade
  • item: What was this trade about
  • price: price per unit, float, deprecated
  • price_int: price in smallest unit as integer (5 decimals of USD, 3 in case of JPY)
  • price_currency: currency in which trade was completed
  • tid: Trade id (big integer, which is in fact trade timestamp in microseconds)
  • trade_type: Did this trade result from the execution of a bid or a ask?

Depth update
Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"14.43","price_int":"1443000","type":1,"type_str":"ask","volume":"-2.71","volume_int":"-271000000"},"op":"private","origin":"broadcast","private":"depth"}

Contains:
  • currency: the currency affected
  • item: the item (BTC)
  • price: price as a float, deprecated
  • price_int: the price at which volume change happened
  • type: 1=ask 2=bid. Deprecated, see type_str
  • type_str: type of order at this depth, either "ask" or "bid"
  • volume: the volume change as float, deprecated
  • volume_int: volume change in smallest unit

(priv) Order update
Code:
{"channel":"(partial key)","op":"private","order_upd":{"amount":1000,"darkStatus":0,"date":1302836027,"oid":"(oid)","price":0.9899,"status":1},"origin":"broadcast","private":"order_upd"}


This is still under tests, and any element of this websocket API may change anytime.
Ideas/comments are welcome :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: demonofelru on April 15, 2011, 02:54:51 AM
Nice keep the updates coming Tux much appreciated.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 15, 2011, 03:58:34 AM
This will help a ton! Thank you very much.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Stephen Gornick on April 15, 2011, 04:11:16 AM
Hoping newsham (or anyone else) will update the MtGox chart to accommodate the new Websocket data feed:
   http://www.thenewsh.com/~newsham/x/mtgox/


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Keefe on April 15, 2011, 08:02:01 AM
I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 15, 2011, 08:06:02 AM
I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

Should be possible, I'll do something :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: memvola on April 15, 2011, 10:34:25 AM
I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 15, 2011, 11:42:37 AM
I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?


The draft-75 (http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75) version of the protocol is really easy to implement from an application.

You send a simple HTTP request:

Code:
GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

(remember to add an empty line at the end)

And you get the following reply:

Code:
HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: null
WebSocket-Location: ws://websocket.mtgox.com/mtgox

Once you got this, the websocket server will send you frames in the following format: a NUL byte (00), some JSON-encoded data, and a FF byte. You can then decode each json packet as they come and handle them.

You can also send packets in the same way, by sending a NUL byte, some json-encoded data, and a FF byte (for example to send an unsubscribe request, or subscribe to a mtgox key).

Example with telnet: http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on April 15, 2011, 03:15:10 PM
Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field  ::)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: mndrix on April 15, 2011, 03:29:20 PM
Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 15, 2011, 03:46:43 PM
Hello, thanks for the API. Few questions/comments:

b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell...


Yes, from reading the protocol specification (http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75#section-4), it looks like I have to send all sorts of different bytes at different times to the server. Some of them translate into ASCII, while others are non-printing. I get confused here, because in the examples, it looks like we're just sending plain lines of text. Do we need byte strings? Do we need /r/n newlines?

And server timestamps would be a huge help! There are multiple options here, really. First, you could just timestamp very accurately, so the client can recreate the actual order flow precisely. Alternatively (and probably easier), just create a long integer ascending order number that goes along with the trades and orders. That way, the server can stay on whole seconds for timestamps, but the client can arrange trades and orders accurately using the id numbers. There should be separate numbering lists for orders and trades, of course.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: grondilu on April 15, 2011, 04:15:16 PM
Damn it...

So many things to learn, and only one life :(


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on April 15, 2011, 05:44:23 PM
Is websocket api down?

Code:
$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: memvola on April 15, 2011, 06:24:25 PM
Is websocket api down?
It's down for me for at least the last hour.

And thanks for the nice explanation, MagicalTux. I wrote a client in haskell, I'll post it when the subscription part is complete.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jed on April 15, 2011, 06:24:56 PM
why are you using UUID's to name the channels rather than just something human readable like: "trades" ?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 16, 2011, 02:50:37 AM
Is websocket api down?

Code:
$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

Yep was down, found the reason, fixed the code and restarted it (division by zero when someone sent badly formatted draft-00 headers).

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field  ::)


a) The ticker is the current stats about bitcoin. It's sent once each time it have a chance to be changed, which includes when orders are posted, removed or completed.
b) Here's an example:
Code:
{"op":"unsubscribe","channel":"xxx"}

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

Not at this point.

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

Because it was easier since I intend to use this system for many other sites, without having to do something to isolate users of a specific system. Also there are user-specific channels and many other stuff which makes use of uuid a bit easier (for me).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: memvola on April 16, 2011, 09:56:15 AM
Here's the haskell code which connects to the server, subscribes to the private channel and unsubscribes from the ticker. No json parsing or error handling.

Code:
module Main (main) where

import Network
import System.IO
import Control.Monad

clientHandshake :: String
clientHandshake = "GET /mtgox HTTP/1.1\r\n\
                  \Upgrade: WebSocket\r\n\
                  \Connection: Upgrade\r\n\
                  \Host: websocket.mtgox.com\r\n\
                  \Origin: null\r\n\
                  \\r\n"

-- Expected server response
serverHandshake :: [String]
serverHandshake = ["HTTP/1.1 101 Web Socket Protocol Handshake",
                   "Upgrade: WebSocket","Connection: Upgrade",
                   "WebSocket-Origin: null",
                   "WebSocket-Location: ws://websocket.mtgox.com/mtgox",
                   "WebSocket-Protocol: *"]

-- https://mtgox.com/code/getKey.php
key :: String
key = "\"key\":\"00000000-0000-0000-0000-000000000000:0000000000000000000000000000\""

hGetHeader :: Handle -> IO [String]
hGetHeader h = do s <- hGetLine h
                  if s == "\r"
                    then return []
                    else do s' <- hGetHeader h
                            return (init s : s')

hGetFrame :: Handle -> IO String
hGetFrame h = do c <- hGetChar h
                 if c == '\xff'
                   then return "\n"
                   else do c' <- hGetFrame h
                           return $ if c == '\x00' then c' else (c:c')

hPutFrame :: Handle -> String -> IO ()
hPutFrame h s = hPutStr h $ '\x00' : s ++ "\xff"

main :: IO ()
main = do h <- connectTo "websocket.mtgox.com" (PortNumber 80)
          hSetBuffering h NoBuffering
          hPutStr h clientHandshake
          hdr <- hGetHeader h
          when (head hdr /= head serverHandshake) .
               error $ "Invalid server handshake:\n" ++ show hdr

          hPutFrame h "{\"op\":\"unsubscribe\",\"channel\":\"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\"}"
          hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

          forever $ hGetFrame h >>= putStr

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 16, 2011, 10:00:45 AM
Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: memvola on April 17, 2011, 04:21:11 PM
Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this :)
Well, anyway, I was actually hoping to be able to keep market depth up to date without polling, like I can do with recent trades using the trades channel, but it seems there's currently no way to do the same with orders. Ticker data doesn't report new or deleted orders, or at least I failed to identify them. I don't want to ask for updates like crazy. :) Are you planning to introduce a new channel for this?

EDIT: Oh, Keefe already asked for this but I missed it. Nevermind then...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 17, 2011, 11:39:23 PM
So I've answered some of my own questions:

I grabbed a copy of the pywsc (http://code.google.com/p/pywsc/) Websockets library for Python and am receiving trades, orders, and ticker events.

I see there is a new market depth channel as well.

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending
Code:
{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}
and
Code:
{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Here's an idea for an initial documentation page:
List the possible fields contained in the JSON return objects, starting with op values and going from there. This is what I have so far, from seeing different things come across the socket:
Code:
op:remark
  message:<message text>
  success:<success boolean>
op:subscribe
  channel:<channel uuid>
op:unsubscribe
  channel:<channel uuid>
op:private
  channel:<channel uuid>
  origin:broadcast
  private:depth
    depth:{
      volume:<volume>
      price:<price>
      type:<order type>
    }
  private:ticker
    ticker:{
      high:<high price>
      low:<low price>
      vol:<volume>
      buy:<buy price>
      sell:<sell price>
    }
  private:trade
    trade:{
      date:<int time>
      amount:<amount float>
      type:trade
      price:<trade price>
   }
  private:order_add
    order_add:{
      oid:<order id int>
      price:<limit price>
      date:<int time>
      amount:<amount>
      status:<status int>
      darkStatus:<0 or 1>
    }
  private:order_rem
    order_rem:{
      oid:<order id int>
    }


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 18, 2011, 12:01:17 AM
Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending
Code:
{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}
and
Code:
{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Just added:

Code:
{"op":"mtgox.subscribe","type":"trades"}

Possible types: trades, ticker, depth


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on April 18, 2011, 02:04:29 AM
Example usage of this API from Python: MtGox->Sierrachart feed (http://bitcointalk.org/index.php?topic=6019)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on April 19, 2011, 03:35:16 PM
Occasionally the websocket sends a depth message to remove an ask or bid (has a negative volume) with a "null" price.  I've only noticed this right before some (not all) trade messages.  I haven't yet seen it happen at volumes unrelated to a trade.

Could it maybe be a race condition in your code where most of the time it gets the depth information before the trade clears it, but sometimes it doesn't?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on April 19, 2011, 05:06:29 PM
Now that I've looked at more instances of it, the volumes are not always obviously related to the subsequent trades, though I think they're related at least most of the time, since these nulls only seem to occur right before a trade or throughout a series of trades.

Here are some examples of the "null" depth message with some surrounding depth/trade info (I removed some messages coming in around these same times that did not seem relevant):

Code:
REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-56},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -56 @ nil
TRADE 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-46},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -46 @ nil
TRADE 17 @ 1.1827
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-29},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -29 @ nil
TRADE 23.216 @ 1.18151

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-5.784},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -5.784 @ nil
TRADE 5.784 @ 1.1621

Code:
 ASK 0.026 @ 1.1875
 ASK 9.974 @ 1.1875
 ASK 0.03 @ 1.1875
REMOVE ASK -9.97 @ 1.1875
 BID 1 @ 1.18751
 ASK 0.026 @ 1.1875
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-9.974},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -9.974 @ nil
TRADE 1 @ 1.18751

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-4},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -4 @ nil
TRADE 4 @ 1.18751

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-19.966},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -19.966 @ nil
TRADE 19.966 @ 1.18751


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 19, 2011, 10:40:12 PM
Hi,

I have located the source of the null and fixed it :)


Mark


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on April 24, 2011, 11:49:41 PM
@MagicalTux

When do you think you'll be able to stop broadcasting dark orders and orders with insufficient funds on the websocket?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jed on April 28, 2011, 08:46:18 PM
This depth channel is broadcasting the changes to the depth. It would be nice if you could send something in the beginning to get the current depth table in the same format. Then you can use this feed to change the original depth table as time goes on.
I know I can get it with the old API but it would be nice if the messages were in the same format.
Also is there a point to all these extra fields: "op":"private","origin":"broadcast","private":"depth" ?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 29, 2011, 02:03:48 AM
Hi,

Broadcasting of depth changes fixed, I think (mostly).

As for the extra fields, they are there mainly to avoid mistaking the messages for something else, however you can safely ignore them :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on April 29, 2011, 05:35:27 AM
It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on April 29, 2011, 06:48:56 AM
It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!

Fixed notices for orders without enough funds.

As for the trade price, it'll be fixed in a few days when we switch to a new backend~


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jed on May 04, 2011, 02:29:52 PM
I noticed that sometimes the price is sent in quotes:

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: MagicalTux on May 04, 2011, 02:45:48 PM
I noticed that sometimes the price is sent in quotes:

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

When the price comes from mysql, it is as a string. I could cast it to float to make sure it's a float, but I guess it doesn't change much for anyone (shouldn't, anyway)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jed on May 04, 2011, 04:33:10 PM
Shouldn't the order_add message tell you if it was a buy or sell order?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jed on May 06, 2011, 05:25:24 PM
Anybody else occasionally miss trade messages?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 09, 2011, 11:59:10 AM
I grabbed a copy of the pywsc (http://code.google.com/p/pywsc/) Websockets library for Python and am receiving trades, orders, and ticker events.
Hmm, I also tried this. Can't seem to succeed.

What port do I have to use? I tried like this:
Code:
from pywsc.websocket import WebSocket
ws = WebSocket('ws://websocket.mtgox.com:80/mtgox')
This bails with "error receiving" in receiver.py. I inserted debugging output to see what it's receiving. It seems to receive the <html/> page you get when you request http://websocket.mtgox.com/.

I'm confused and obviously doing something wrong. Can someone help?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on May 11, 2011, 06:19:31 AM
Well, I had to change my WebSocket._send function to do this:
Code:
self._sendRaw('%s%s%s' % ('\x00', data, '\xff'))
because it was breaking the socket trying to send a non-string.

My receive code is identical except that I added a Threading.Event for that loop instead of just while True.

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 11, 2011, 04:01:22 PM
I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

SOLVED! Thanks for your help.

Stupid me forgot about that transparent squid proxy.

Am receiving messages, now to the fun stuff!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on May 11, 2011, 05:03:16 PM
Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: toffoo on May 11, 2011, 07:12:52 PM
Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

Fantastic news.  Please let us know when you have something ready to test out.


Title: depth data desync?
Post by: molecular on May 12, 2011, 10:07:11 AM
ok, I'm receiving messages, nice ;)

Now I want to update the market depth info, which I'm initially getting through http://mtgox.com/code/data/getDepth.php.

so I'm listening on channel "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe" for messages that have op="private", and am getting depth updates like this:

Code:
{u'volume': -10, u'price': 5.4000000000000004, u'type': 2}

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Above message is an example of a message that caused this... I didn't have an entry for price "5.4000" (I'm using 4 decimal places for the index in my depth "dictionary", but that should do no harm, right), so the result is a new entry with negative volume.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

EDIT:

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

If my problem actually stems from the two timepoints (initial depth request, start of depth updates) not being equal, I would consider this to be not a nice-to-have but a must-have.

You could maybe simply (from client point of view) send the complete depth table as a burst of updates after websocket connect?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Shrapnel on May 12, 2011, 10:19:27 AM
It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Quote
Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?
I'm also wondering the same thing. I think you can't unless the API send the complete data first.
I'm trying to do a realtime order book but it's currently very hard due to dark pool and sync issues.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 12, 2011, 09:47:32 PM
It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.




Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jed on May 12, 2011, 10:14:12 PM
I think it doesn't send every update to the order book. It seems like some random ones get dropped.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on May 13, 2011, 02:42:32 AM
It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

Mtgox has some bugs.  The websocket often entirely skips sending some depth messages, sends some at a slightly different price than what they should be, and the APIs (or site in general) has rounding inconsistencies.  I've only partially worked around these issues by rounding all depth message prices to the 5th decimal place and volumes to the 3rd decimal place, and creating a button in my program that I can click to manually refresh the entire depth tables when I start noticing odd stuff.

An example is where a bot will download it's own open orders, see that one of it's orders is something like 6.253, but it wants it to be 6.252999, so it removes the order for 6.253 and creates a new one at 6.252999, and both of those come over the websocket just like that.  And the bot does this over and over again because the API for getting your own open orders rounds to the 4th decimal place, but the websocket will do at least 6 decimal places for prices and more than that for volume, so your table ends up removing the order at 6.253 over and over again, resulting in a negative number, while it keeps adding orders to 6.252999 over and over again, making some huge number that's not accurate.  The bugs and inconsistencies are pretty annoying.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Keefe on May 13, 2011, 03:48:53 AM
I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 13, 2011, 11:32:29 PM
I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

MagicalTux, can you give us an estimate when you'll get around to implementing a solution to these problems or talk to us about a solution?

These problems are really not acceptable: we can't have the client side depth data out of sync or even worse stuff happening (missing data?)

I'm here to help with testing.


Title: @newberg_test
Post by: jaybny on May 14, 2011, 12:49:05 AM
my program is listening to the WebSocket and publishing to twitter once an hour, on new highs or lows.

im using c# and am not  using JSON schema, just parsing directly... so if anything changes - I will break..

http://twitter.com/newberg_feed



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ycros on May 14, 2011, 08:10:11 AM
Herp, disregard what I posted before, I found my client library was sending an extraneous \r\n\r\n on the end there.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 14, 2011, 09:10:44 AM
I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

I'm seeing a slight problem for MagilTux to solve the "keep client depth tables in sync" problem, I hadn't thought about:

(Firstly, wether a client gets the initial depth data through the "classic API" or through an initial burst of depth-change messages doesn't make a difference for my thinking here.)

Problem is: the depth data a client receives initially is chopped off at the ends at a certain distance to the current last trade value.

Now whenever a trade happens, this window moves and some trades exit the window on the one end while others become "visible" on the other end.

There's 2 ways (at least) to solve this:

  • Screw that window completely, give the complete depth data to clients
  • On trade, determine entering/leaving orders and send them as new/cancelled over the channel

I'd go for the first option in combination with initially bursting change-messages instead of using the "classic API" to initially fill client depth table.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: toffoo on May 17, 2011, 04:40:02 AM
Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: mindtheft on May 17, 2011, 10:22:23 PM
A quick js chrome extension hack for WebSocket -

https://github.com/mindtheft/mtgox-Chrome-Extension

You can use the crx file to install the extension, or better yet check the source code and pack your own crx file to make sure there's no foul smells from mine.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on May 18, 2011, 06:16:43 PM
Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!


Here's a naive Mt. Gox websocket client I wrote in python: http://pastebin.com/FH6eWhFm
By default it just prints out trades as they happen, but you can modify it to do whatever you want.  If someone really wants an event-based version or something to make it a little easier to work with, you might be able to convince me (with a few bitcoins ;)) to put something together with pywsc or something.  I may eventually release an executable jar (clojure/java) file for those willing to buy a copy that will give you a GUI to view live trades and depth changes (though mtgox's websocket is still quite buggy in respect to depth changes...there would be a button to let you do manual refreshes as well).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: martin on May 18, 2011, 09:56:47 PM
I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 0x6763 on May 19, 2011, 03:06:47 AM
I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

Mtgox currently has multiple bugs in the websocket's depth channel...your tables are behaving exactly the same as mine are.  Mtgox is moving to a new system in june which will hopefully have all of these problems taken care of.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: martin on May 19, 2011, 03:45:30 PM
Ah excellent, any information on the form of this new system?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on May 22, 2011, 01:25:33 PM
I wrote some code in C# for MonoDevelop or Visual Studio
MtGox & IMtGoxHandler & TestMtGox - for making orders
  FetchCash
  FetchOrders
  Place/CancelBuyOrder
  Place/CancelSellOrder
  .Handler - interface for responses & errors
WebSocket & Program - for read trades events
  .OnData - delegate received string (in JSON format) for any channel

https://rapidshare.com/files/3646016978/MtGoxApp.zip


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on May 22, 2011, 04:38:11 PM
There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

technically:
JSON changes to this: add new attr to "depth": "abs"=X: where X = "1" for absolute volume; "0" or absent attr - relative volume as now
That data can be cashed on server for all new clients. Depth changes updates cache. Need to generate JSON (for sent) on request only - more expansive operation than just update some structs in cache.

PS
Sorry for my Russian English :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on May 22, 2011, 05:09:17 PM
maybe
needs some authorization vs WebSockets' flood from bag guys (direction: from client to server)

something like this:
- some cookie received from Trade API for websocket connection to MtGox. this cookie valid only next hour (5-10-30min)
- on new connection to websocket client must send this cookie to server. if cookie is valid automatic subscribing will do..
cons: more difficult process for connection

or:
- using SSL/TLS encrypted WebSocekts.
pro: can refuse polling Trade API and do all orders and receive statuses over WS
cons: need implementations for different languages


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on May 22, 2011, 09:27:32 PM
volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?
for price enough.. 2 digits for cents.. and.. ok, let be 4digits after dot
for volume.. hmm.. similar numbers less than 0.001 are considered as zero.. = so 3 digits after dot for volumes


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: martin on May 23, 2011, 02:08:23 PM
There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?

We need as many digits as people can specify, which as far as I know is an unlimited amount.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on May 23, 2011, 05:02:07 PM
I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.
good variant (1)
but people dont want to calculate depth state. they often will request the one. more traffic, slow server, bad trading.

variant (1.5)
every depth message contains all state. without unnessary attrs - one number is price, next in pair is volume with right sort order: "depth":{"asks":[X.XXXX, V.VVV,...],"bids":[...]}

another variant (2):
- new channel for depth-20(or some N) values asks & bids every one minute (only for checking purpose or as starting point)
- clear description of algorithm how calculate depth state over trade & depth channels

one more variant (2.5):
- depth values is absolute forever
- new attr "pos" with values { +N, N, -N } where
     +N - new row inserted at N pos, rows [N..end] shifted down
       N - updates values ar row N
      -N - row N deleted. other rows shifted to close gap (to up)
- "trade" values ​​must match the changes in "depth" values (a priori)
- but still need starting point - current depth state - use variant (2) or something
pros: simple calcs, asks & bids always sorted by server(that must be), traffic +1attr, CPU calcs almost same(as now)

"make your choice!" :)

We need as many digits as people can specify, which as far as I know is an unlimited amount.
now: values with volume less 0.001 remains in my depth state after process depth & trades from corresponding channels. but next ticker does not account for their. so, I delete them from calculated depth state.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: leepfrog on May 25, 2011, 05:52:36 PM
This might be a stupid question but I dont fully understand the meaning of the ticker.php api.
It returns something like that:

{"ticker":{"high":7.51,"low":7,"vol":33895,"buy":7.257,"sell":7.3499,"last":7.2591}}

What exactly is high and low? the rest seems pretty clear but I am not sure about those two


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on May 25, 2011, 06:03:12 PM
The ticker high and low are probably those respective prices for the current trading session, which I'm guessing is a full day. You should watch the ticker values for when the volume resets to zero, as that would be the beginning of the trading session.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: leepfrog on May 25, 2011, 06:12:55 PM
Makes sense - I'll watch it. Thanks!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on May 25, 2011, 08:14:04 PM
I'm not sure it's been mentioned before, but the data type of the getDepth.php return does not seem to be application/json, so it is not possible to do a cross-domain call to MtGox for the depth of market data.

This would be a really easy and quick fix in the getDepth.php code...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: robottwo on June 07, 2011, 03:09:51 AM
Hi,
    The websockets API is *great*. I have a suggestions for the protocol:


The size in the depth messages should publish the new absolute size, not the relative size. Subscribers can still figure out the relative size change by subtracting from the previous update at that size, but this provides better error recovery on the client side.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on June 08, 2011, 03:15:33 PM
hint (from my bugs)
see MSDN Math.Round( decimal d, int decimals ), which I use
MtGox rounds numbers NOT SAME:
If the value of the first digit in d to the right of the decimals decimal position is 5, the digit in the decimals position is rounded up if it is odd, or left unchanged if it is even. If the precision of d is less than decimals, d is returned unchanged. The behavior of this method follows IEEE Standard 754, section 4. This kind of rounding is sometimes called rounding to nearest, or banker's rounding. It minimizes rounding errors that result from consistently rounding a midpoint value in a single direction.

some datas:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":30.22425,"type":1,"volume":1},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":30,"high":31.89,"last":30,"low":18.6,"sell":30.2243,"vol":98781}}


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on June 08, 2011, 05:03:45 PM
There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

Which min volume of order required for trading (min step size)?
I saw volume=0.007 in depth channel, but ticker channel ignores that values (so values lessOrEqual 0.007 I ignore)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jaybny on June 10, 2011, 06:06:16 PM
is websockets down? any ETA?

thanks


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: af12345 on June 11, 2011, 02:45:56 PM
I think the API shows a lot of promise, but it's seems to be missing one key detail --- without publishing an order-id on the depth message, and the constituent order-ids on the trade message, it's not really possible to ever get in sync.

The reason is as follows: if you start your connection and receive an order @ 25 for volume 10, and then there is a trade @ 25 for volume 5, you can't tell if the trade was for part of the order you just received, or an earlier order (pre your connection time).

If, however, you were to provide an order-id on the depth message, and the pair of order ids on the trades, then all ambiguity is removed.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on June 11, 2011, 03:27:26 PM
Here are some things I've noticed after building some web applications using the WebScokets API:

  • Switching to the Secure WebSocket (wss://) would dramatically improve compatibility, especially for those behind a proxy
  • Security would also allow users to send trades via the WebSocket, which would simplify trading to one connection instead of polling+WS.
  • Send the prices and volumes as integers! Bitcoins are stored as integers in the client but displayed as floats by dividing by 1e8. Some of the prices and volumes quoted over the WebSocket are floating point numbers with much more precision than the 8th decimal, which doesn't make sense in the framework of Bitcoin overall.
  • The ability to request the depth of market table entries through the WS
  • More consistency in depth of market updates - I have to poll the data every so often in order to eliminate discrepancies, which is not very robust and could be misleading to people using my service.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on June 12, 2011, 05:27:35 PM
it will be good create IRC-channel
  • for talks
  • for "websockets OFF for hour"
  • for bans in websockets (if exists)

at now
opened socket, sent handshake, received nothing - conn reset by peer
and I dont know what happened and how long it will last


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: kseistrup on June 12, 2011, 05:30:14 PM

  • Switching to the Secure WebSocket (wss://) would dramatically improve compatibility, especially for those behind a proxy

+1 for wss://

Cheers,


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: matt.collier on June 12, 2011, 10:32:56 PM
Hello all, developing in Java here.

Aiming at creating an interface similar to Trade Station's Matrix: http://www.tradestation.com/strategy_testing/market_depth.shtm

I'm using Open Office Calc as the front end.  I've got the real-time data flowing into calc nicely, but I've run into the same sync issues everyone else is having.

As others have done, I'm bootstrapping my depth info with http://mtgox.com/code/data/getDepth.php, but before long, none of the volume information makes any sense.

I'll be happy to share my source with any interested parties.  For now I'm going to put this on the back burner until a solution to the sync issue has been implemented.

I'll put my vote in for a channel that publishes absolute volume information in addition to some effective bootstrapping method.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: bitoption on June 13, 2011, 05:17:55 AM
I'm getting deafening silence from my python websockets client today. This is sad, since I use it to give pricing cues on bitoption.org; I've noticed some websockets-gox stuff is working right now, some is not.

Any indications as to what's what and why?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on June 13, 2011, 05:39:32 AM
It would be nice to hear from MagicalTux about the progress of the back end rewrite and deployment. Also, a formal documentation of the WebSockets API is most welcome.

I'm still seeing the feed at the moment, but it stopped a little earlier today. Very odd.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: netxshare on June 13, 2011, 06:24:58 AM
I am also having issues, my current connection to the websocket is showing data but any new connectionss fail to get anything more then a response header.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on June 13, 2011, 07:24:25 AM
found site with fast check websockets (and wss too)
http://websocket.org/echo.html
browser must support websockets (or enabled it)

for now MtGox-WebSockets requires Sec-WebSocket-Key1 and Sec-WebSocket-Key2 and 8bytes-data

at now (UTC 19:27 13-jun) WS dont responds with 16bytes of hash

and at now (UTC 20:37 13-jun) no connection

at now (UTC 20:35 14-jun) System.Net.Sockets.SocketException: Connection refused


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on June 13, 2011, 07:55:22 PM
I'm getting deafening silence from my python websockets client today. This is sad, since I use it to give pricing cues on bitoption.org; I've noticed some websockets-gox stuff is working right now, some is not.

Any indications as to what's what and why?

There have been 2 periods where no trades happened and the websocket was dead, apparently some mysql troubles. Maybe you stumbled into one of these.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on June 16, 2011, 06:43:52 PM
today I saw MtGox Live for Android

and here is screen of my iPhone app (not in AppStore yet) - depth state
http://img27.imageshack.us/img27/1115/screenshot20110616at223.png

in first version will be depth state & trades only, no wallet & orders


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: kseistrup on June 16, 2011, 07:15:52 PM

today I saw MtGox Live for Android

I have a feeling that the app is polling Mt. Gox every 5 seconds, rather than using the websocket API.

Something that would be immensly cool is an android app with the same graphs as on MtGoxLive (http://mtgoxlive.com/), using websockets.  AFAIK there are no browsers for android that supports websokcets, so I can't just load the web page.

Cheers,


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: ezl on June 17, 2011, 01:31:41 PM
recommendation regarding the depth feed -- , send updates instead of diffs.

 the bandwidth is ~same, but by broadcasting updates, the penalty for missing a message only lasts as long as the interval until the next update for that price level. clients can compute diffs if they want.

it also makes it so that the listener can theoretically construct the full book just by listening to the updates instead of repolling the old api and hoping the messages are in sync


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: dev^ on June 18, 2011, 01:49:12 AM
Hi,

is it possible for upcoming version to rename the field "private" in anything other, which isn't a protected keyword in Java?  :)

Currently it isn't possible (at least for me) to read this field with any common JSON parser library, e.g. Google's Gson.
(of course, it would be possible to set this field's content via text parsing, but this isn't the right way to do so...)

> Something that would be immensly cool is an android app with the same graphs as on MtGoxLive, using websockets.

If someone can develop any nice looking GUI for Android devices, I could provide the complete Java part regarding the WebSockets - it seems to work quite fine.  :D

Best regards
dev^


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on June 19, 2011, 06:46:31 PM
at now
for determine connection lost (iPhone hasn't keepalive ops for socket) I do next:
if App dont recv any datas on channels over 6seconds it send:
Quote
"{"op":"ping"}"
if connection is ok than server responds to App:
Quote
{"message":"Unknown command","op":"remark","success":false}
if no any responds in next 6seconds (include other datas on channels) App decides that current connection lost and try to create new connection (with close old)

NB: "ping" dont send every 6seconds! only if at last 6seconds no datas (including "Unknown command")
otherwise App works but nothing shows over hour on broken connection (turnoff WAN at router). I dont known default values in iPhone keep-alive logic.

and my suggestion:
maybe add "good" response:
Quote
{"op":"pong"}
it dont requires any logic or DB-access. simple response for current connection
although its works well now


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: dds on June 28, 2011, 07:49:59 AM
Looks like MtGox WebSockets broken after reopening - connection can be established but no data received.
Mark, are you going to fix it?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on June 28, 2011, 08:47:39 AM
Looks like MtGox WebSockets broken after reopening - connection can be established but no data received.
Mark, are you going to fix it?

+1

also: I can't seem to be able to use the classic Trade API, either. Says something about a "bad token". Anyone know about that?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: kseistrup on June 28, 2011, 08:59:20 AM

I can't seem to be able to use the classic Trade API, either. Says something about a "bad token". Anyone know about that?

Look at the Trade API page on your account — it says that it will be enabled soonish.

Cheers,


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on July 03, 2011, 04:33:39 PM
good! websocket works for now!

some questions:

Quote
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private", "origin":"broadcast","private":"ticker", "ticker":{"avg":15.469053268,"buy":15.63,"high":15.689,"last":15.63001,"low":15.311,"sell":15.63001,"vol":15679}}
"avg" is average volume by trade?

Quote
{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade", "trade":{"amount":2,"amount_int":"200000000","date":1309710218,"item":"BTC","price":15.632,"price_currency":"USD","price_int":"1563200","tid":"1309710218345711","trade_type":"ask","type":"trade"}}
what mean "trade_type":"ask"? trade raised by ask order?

and where is depth channel? no datas for it. will it appear in the future?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: dev^ on July 03, 2011, 04:36:29 PM
good! websocket works for now!

Yay! :-)

"avg" is average volume by trade?

I think it's the average price. (As I guess by it's value...)

what mean "trade_type":"ask"? trade raised by ask order?
and where is depth channel? no datas for it. will it appear in the future?
Looks like this is "the new" depth-channel...

Best regards,
dev^


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on July 03, 2011, 05:13:31 PM
and where is depth channel? no datas for it. will it appear in the future?
Looks like this is "the new" depth-channel...
my English is not so good. where is "this "the new" depth-channel" ?
EDIT: on #mtgox they say me that "should come soon"


and I try to get key for personal channel via https://mtgox.com/code/getKey.php (https://mtgox.com/code/getKey.php) (as say in post 0). it returns
Quote
{"error":"Bad token"}
???
EDIT: maybe same (come soon)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: kseistrup on July 05, 2011, 06:13:27 AM
My ws client has been trying to connect for an hour now, is the websocket down for me only?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: dev^ on July 05, 2011, 06:57:53 AM
My ws client has been trying to connect for an hour now, is the websocket down for me only?

It's down for everyone.

Code:
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
(...)

MtGox should really think about some QA or at least a 2nd developer in it's team...


Title: wiki
Post by: molecular on July 10, 2011, 10:08:40 AM
there's a wiki now: https://en.bitcoin.it/wiki/MtGox/API#Websocket_API


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: vector76 on July 27, 2011, 12:51:51 AM
I guess since there is not a good way to synchronize the market depth websocket with the getDepth.php, I could fetch getDepth multiple times and watch the websocket for deltas and try to correlate them.

I'd have to agree with ezl's suggestion of sending the total volume instead of the deltas.  Then there would be no synchronization problem.

Are people still seeing problems with the depth websocket or have those issues been fixed?  I guess I'll fetch getDepth every 5 or ten minutes just in case..


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: divebubble on July 28, 2011, 10:30:59 PM

Reading the different channels, updating data all works perfect... but -

how can i connect my account to get my trading information?

sending
{"op":"mtgox.subscribe","key":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"}

will always result in

{"message":"Invalid MTGOX key","op":"remark","success":false}


XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX is the API Key from the https://mtgox.com/users/settings page.
Or are i'm compleatly wrong?


Thanks.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: throughput on August 03, 2011, 09:02:02 AM
Websocket API is cool for receiving trade updates! Thanks!

But is it possible to receive updates on my orders anyhow?
If that is implemented, then I don't get the meaning of the "(partial key)" in
Code:
{
  "channel":"(partial key)",
  "op":"private",
  "order_upd":{
    "amount":1000,
    "darkStatus":0,
    "date":1302836027,
    "oid":"(oid)",
    "price":0.9899,"status":1
  },
  "origin":"broadcast",
  "private":"order_upd"
}
(example from https://en.bitcoin.it/wiki/MtGox/API#Websocket_API )

I don't understand how should I implement the authentication to subscribe to my private channel.

Please, can anybody provide more details on:

 1. Does it work at the moment?
 2. How the authentication is implemented?

Also, it would be nice to be able to request the full depth to receive them as depth messages.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on August 06, 2011, 06:18:26 PM
You have a small bug in the reply header:
Code:
HTTP/1.1 101 WebSocket Protocol Handshake
should be the first line, and in your case it's
Code:
HTTP/1.1 101 Web Socket Protocol Handshake
(yes, the additional space between Web and Socket). Nothing major, however if some software validates the header, it won't be recognized as a valid one. Example of such real-world failing app: websocket-sharp. Other are probably just failing silently too.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on August 27, 2011, 02:13:50 PM
It happens very often that there are consecutive "depth" messages, where one comes immediately after another and they both have the same amount/price and the opposite sign. So effectively they do nothing, except causing confusion and an unnecessary traffic.

For instance:
Quote
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"9.15417","price_int":"915417","type":1,"type_str":"ask","volume":"-0.09283","volume_int":"-9283000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"9.15417","price_int":"915417","type":1,"type_str":"ask","volume":"0.09283","volume_int":"9283000"},"op":"private","origin":"broadcast","private":"depth"}
...
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"-0.2","volume_int":"-20000000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"0.2","volume_int":"20000000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"-0.2","volume_int":"-20000000"},"op":"private","origin":"broadcast","private":"depth"}
...
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"-0.79287883","volume_int":"-79287883"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"0.79287883","volume_int":"79287883"},"op":"private","origin":"broadcast","private":"depth"}

It seems like some glitch and it would be nice if could be fixed one day.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on August 30, 2011, 09:06:20 PM
It happens very often that there are consecutive "depth" messages, where one comes immediately after another and they both have the same amount/price and the opposite sign. So effectively they do nothing, except causing confusion and an unnecessary traffic.

For instance:
Quote
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"-0.79287883","volume_int":"-79287883"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"0.79287883","volume_int":"79287883"},"op":"private","origin":"broadcast","private":"depth"}

It seems like some glitch and it would be nice if could be fixed one day.

imho, all changes must be. human's or robot's..


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on August 30, 2011, 09:14:07 PM
imho, all changes must be. human's or robot's..
agree. but these dont look like robot changes, do they? why would a robot do something like that? the only thing i can think of: to DoS the server :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on September 20, 2011, 09:52:43 AM
The WebSocket API does not work with the latest Chrome. :(

Chrome dev claims that the problem is at mtgox side.
Quote
It seems that websocket.mtgox.com is sending out a frame in old format (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00). Chrome 14 uses new version WebSocket protocol (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-08). We've dropped old protocol and are asking servers to adopt new one. mtgox is replying new version handshake correctly. Only new frame support looks broken.
http://code.google.com/p/chromium/issues/detail?id=97026

Please fix.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: AV on September 20, 2011, 12:56:57 PM
Hi, MagicalTux !

The WebSocket API really does not work with the latest (14.0.835.163) Chrome.
http://www.smayly.ru/gallery/anime/Pig/4.gif

For example http://bitcoin.clarkmoody.com/
WebSocket Closed after 2 second.

Help ! Help ! Help !

Thanks.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: speeder on September 20, 2011, 12:58:38 PM
Yay! Correct thread to complain about that!

Indeed, me too want clarkmoody on Chrome 14!

(thankfully at work Chrome is still stuck on 13)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: bzzard on September 22, 2011, 11:42:50 PM
yeah, this should work?
I know that ws://mtgoxlive.com:3456/channel works :P

MtGox! Anyone there?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: paraipan on September 24, 2011, 05:50:28 PM
sup MagicalTux, any plans to update websockets to draft-10 any time soon ? Thanks


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on September 25, 2011, 04:28:12 PM
I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

How about these and the wss://?
Its been months that Mark didnt post here now..

And, general question:
I chose this API as a starting project to learn ruby.  Maybe a bit (too) ambitious, but would at least be a useable project, if it ever gets finished..
Are there real advantages of this "new" websocket API over the old HTTP API except that I dont have to poll any more and have instant updates (instead of every 10 secs)? I would at least not have the syncing issues?

With the probably still unsolved issues I hesitate to invest much more time into this - being a novice I still am trying to actually post to the websocket.. *rolleyes*

Thanks,

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on October 10, 2011, 07:49:55 PM
I found the part of the draft where Chrome 14 is failing:

http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10#section-4.2

Apparently, MtGox is setting the RSV2 bit in the responses to the client. Thus, according to the spec, Chrome is required to FAIL the connection.

Using a packet sniffer, I can see the subscribed responses from the WebSocket after connecting, but the message never gets through and Chrome fails and closes the connection.

This is such a simple fix...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: BkkCoins on October 12, 2011, 06:43:21 AM
Does the "own" channel work at all?

I can't figure out how to get data about my own trades/orders. I've dumped all the data streamed and there is no data not matching the default channels. I tried subscribing using a key of my own account. That doesn't work either. It seems like this just doesn't work yet.

It would be really useful to see when your own orders/trades occur. If this does work then is there a web page somewhere that says more than the one line on the wiki?

*** Edit:

Another thing I noticed now is multi-currency trades coming thru. I don't quite understand these:

19:07:57 BUY  22.81 @ $2.63192 GBP
19:07:57 BUY  22.81 @ $25.6 CNY

I see British Pounds and Chinese Yuan. But why twice and how does this work - is someone buying with Pounds and settling in Yuan or something? I realize that this must be the same trade since the  GBPCNY exchange rate is now about 10.33. Just not sure how I can handle this as info if the USD isn't indicated.

Thx.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: qxzn on October 12, 2011, 05:06:44 PM
I have not been able to get the "own" channel to work either. If anyone has, please share how!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Cluster2k on October 16, 2011, 12:41:20 PM
Subscribing to my own channel doesn't work either.  I'm using VB and tried sending ANSI strings to MtGox in various formats, but as soon as I do I lose the TCP connection.  Sending the handshake right after connecting exactly the same way works fine however.

I have tried:

{'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'op': 'mtgox.subscribe'}
{'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'op': 'subscribe'}
{'op': 'mtgox.subscribe', 'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'}
{'op': 'subscribe', 'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'}

Sending nonsensical data "eg. 'testing 123' to the connection causes a disconnection as well.  Is this standard behaviour if MtGox doesn't recognise the message?

I assume the key is my key listed in my MtGox API section.  I have successfully used the key and secret to retrieve account data using a sample PHP script.

EDIT: Just tried using plain old telnet to connect to MtGox.  Works fine, just like this example:
http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

But as soon as I try to send any subsequent data to the connection I lose it.  The wiki entry at https://en.bitcoin.it/wiki/MtGox/API is non descriptive. 

"Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades). "

There is no description on how someone can subscribe to their own channel.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: qxzn on October 16, 2011, 05:39:19 PM
You should not be getting a disconnect just for sending an invalid message type. You should be getting a json message back that says something about how your command was unrecognized. I would double check the bytes you're sending. As a test, you can try to unsubscribe to one of the feeds. That works, so you should be able to get it to work.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Cluster2k on October 16, 2011, 10:56:54 PM
You should not be getting a disconnect just for sending an invalid message type. You should be getting a json message back that says something about how your command was unrecognized. I would double check the bytes you're sending. As a test, you can try to unsubscribe to one of the feeds. That works, so you should be able to get it to work.

It seems I can't send anything to MtGox after the initial handshake without getting disconnected.

I've tried in telnet too (MS Windows telnet client).  Surely it should work?

telnet websocket.mtgox.com 80
Sent the following text:
GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

After pressing enter the second time I start to get the stream of data.

But when I try to send something like "{'op': 'mtgox.unsubscribe', 'key': 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe}" the connection is lost.  If I send anything at all the connection dies.

I don't get it.  

EDIT: Just tried it again with Putty on Win7.  Type of connection: raw.  Same result.  I can get the streaming data, but as soon as I try to send anything back to the server on the same connection it disconnects.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: qxzn on October 17, 2011, 01:12:46 AM
You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Cluster2k on October 17, 2011, 01:26:20 AM
You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

Thanks for that.  I'll try putting \x00 before my message and \xFF after and see how that goes. 


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Cluster2k on October 17, 2011, 03:38:46 AM
You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

That works fine.  I wasn't sending the \x00 and \xFF at the beginning and end of each message.

I can successfully unsubscribe from the depth feed using:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"unsubscribe"}

However when I try to resubscribe I get the following error:
{"message":"Unknown command","op":"remark","success":false}

I have tried all sorts of combinations of subscribe messages, but none seems to work.  I also can't subscribe to my own channel.

The following all fail:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"subscribe"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"mtgox.subscribe"}
{"op":"subscribe","channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"mtgox.subscribe","channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"subscribe","24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"mtgox.subscribe","24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}

I looked at the haskell code and tried that subscription message first, then tried various other combinations.

hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Cluster2k on October 17, 2011, 07:33:09 AM
I have just been informed on #mtgox IRC that websockets are read-only.  Have been for a number of weeks.  Subscriptions don't work and it's not possible to view your own executed trades.

It's possible get around this limitation, but it really makes the system slow and a bit cumbersome to use.  It would be so much more convenient to get instant confirmation of your own trades.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on October 20, 2011, 04:12:46 PM
I would like to see depth messages for all of the currencies broadcast out over the WebSocket, as I do not think it's happening right now.

Can anyone confirm that there are only USD depth messages coming across the socket?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on October 21, 2011, 01:23:06 PM
I would like to see depth messages for all of the currencies broadcast out over the WebSocket, as I do not think it's happening right now.

Can anyone confirm that there are only USD depth messages coming across the socket?

Looks like it's true. I am filtering for "depth_msg['currency'] == 'USD'", but put a debug log line for the other case. It doesn't trigger.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on October 26, 2011, 03:29:34 PM
I am having problems with Websockets and Ruby in general, maybe someone of you is kind enough to help out?
I am beginner with Ruby and chose this as a learning-by-doing project.

Using examples from the web, I am still trying to connect to the websocket (works) and sending strings to it (handshake and channel subscription/exiting):

Code:
require 'rubygems' 'eventmachine' 'em-http-request'
 
EventMachine.run {
  verbindung = EventMachine::HttpRequest.new("http://websocket.mtgox.com").get :timeout => 0

  verbindung.errback { puts "oops" }
  verbindung.callback   {
    puts "WebSocket connected!"
                 
verbindung.send("GET /mtgox HTTP/1.1\n", 0)
  }
}

Error, my Ruby doesnt recognize "GET /mtgox HTTP/1.1\n" as a string to be "send" :

Code:
websocket.rb:19:in `send': undefined method `GET /mtgox HTTP/1.1 (NoMethodError)
' for #<EventMachine::HttpClient:0x7f9f6d22e8b8>

I tried "print", "send", "write", "<<", still didnt get it to work. I am lost, grateful for any hint!

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on October 28, 2011, 09:13:22 PM
Anyone else getting timeouts connecting to ws://websocket.mtgox.com/mtgox? I am right now...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on October 28, 2011, 09:14:13 PM
the server has been dead since yesterday.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Cluster2k on October 30, 2011, 11:00:19 PM
Websockets have been dead for about a week.  Nothing on MtGox's support page about it.  Does MtGox even know, or do they care?  It seems not.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on October 31, 2011, 01:10:13 AM
Websockets have been dead for about a week.  Nothing on MtGox's support page about it.  Does MtGox even know, or do they care?  It seems not.

MagicalTux knows, has been saying something routes not being up about 3 or 4 days ago.

Does he care? I'm not sure.

I think the websocket stuff is pretty important.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on October 31, 2011, 04:29:49 PM
http://mtgoxlive.com/ is also suffering from this failure.
The data is shows doesn't seem to be in a real time anymore.
And this service had supposedly been acquired by MtGox.
So he should care :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: BTCurious on October 31, 2011, 04:44:15 PM
http://mtgoxlive.com/ is also suffering from this failure.
The data is shows doesn't seem to be in a real time anymore.
And this service had supposedly been acquired by MtGox.
So he should care :)
I'm not sure that's true. If I compare it to the fastest monitor bot I know, #bitcoin-RT in freenode, then the number seems to be behind ~10 seconds. But I don't remember this ever being any different.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on October 31, 2011, 04:55:37 PM
Before you could see new transactions popping up on the right side, in a real time.
And the green price line was also updated in a real time - now the last point of it is always a couple of minutes behind.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: BTCurious on October 31, 2011, 05:04:29 PM
Before you could see new transactions popping up on the right side, in a real time.
And the green price line was also updated in a real time - now the last point of it is always a couple of minutes behind.
Oh hey, you could be right. It did work via the websocket before. Maybe it works via the http API now?

Edit: Websocket is back. And yeah, the transactions in real time too.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on October 31, 2011, 11:42:57 PM
Yay, websocket is back. Thanks Mt. Gox.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on November 01, 2011, 11:46:07 AM
Yay, websocket is back. Thanks Mt. Gox.
I saw it back for a moment... but it doesn't work now, does it?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on November 01, 2011, 01:40:23 PM
Yay, websocket is back. Thanks Mt. Gox.
I saw it back for a moment... but it doesn't work now, does it?

It's working for me in my own client. Thanks MagicalTux!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on November 01, 2011, 01:46:48 PM
It's working for me in my own client.
OK - thanks for the info.
This made me thinking... and it seems that I upgraded my websocket libarary to a new version.
It's OK when I go back to the old version. Its probably the same problem as with a new Chrome.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on November 01, 2011, 01:48:18 PM
It's working for me in my own client.
OK - thanks for the info.
This made me thinking... and it seems that I upgraded my websocket libarary to a new version.
It's OK when I go back to the old version. Its probably the same problem as with a new Chrome.

Thanks for that hint. I'll keep that in mind, cause I'll probably switch websocket impl at some point (currently using some hacked-up thingy someone gave me).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on November 01, 2011, 02:07:52 PM
Yay, websocket is back. Thanks Mt. Gox.
I saw it back for a moment... but it doesn't work now, does it?
Works for me right now as well.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on November 19, 2011, 12:40:08 AM
Anyone succeed with new MtGox's socket.io interface? I was succesfull with socket.io handshake, server returned "12906784121696250629:15:25:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling". But then I open WS connection to ws://socketio.mtgox.com:80/socket.io/1/websocket/12906784121696250629 and nothing happen, client don't receive any data. Am I doing anything wrong?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on November 20, 2011, 12:52:37 AM
Anyone succeed with new MtGox's socket.io interface? I was succesfull with socket.io handshake, server returned "12906784121696250629:15:25:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling". But then I open WS connection to ws://socketio.mtgox.com:80/mtgox/socket.io/1/websocket/12906784121696250629 and nothing happen, client don't receive any data. Am I doing anything wrong?

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on November 20, 2011, 01:07:25 AM
Thanks for info. Is following websocket URL correct?

ws://socketio.mtgox.com:80/socket.io/1/websocket/1290678412169625062

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jothan on November 23, 2011, 04:37:50 AM
Thanks for info. Is following websocket URL correct?

ws://socketio.mtgox.com:80/socket.io/1/websocket/1290678412169625062

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.


With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on November 23, 2011, 03:17:15 PM
With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Yes, you received socket.io handshake response, it's fine. socket.io != websocket.

Quote
Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Does it work for you? I dropped idea of get it working after playing with it for hours. Lacking documentation, multiple versions of specifications, total mess.

Then I implemented bitcoincharts socket interface in 15 minutes...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jothan on November 23, 2011, 03:36:28 PM
With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Yes, you received socket.io handshake response, it's fine. socket.io != websocket.

Quote
Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Does it work for you? I dropped idea of get it working after playing with it for hours. Lacking documentation, multiple versions of specifications, total mess.

Then I implemented bitcoincharts socket interface in 15 minutes...

I can now connect fine, but I cannot subscribe to any channel, I get "invalid request" or "invalid command", so I cannot do anything useful yet. I'm gonna try interfacing bitcoincharts tonight.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jothan on November 23, 2011, 10:55:13 PM
Could someone comment on the following points ?

  • Is the stream API service currently in a usable state ? (I cannot determine this since I cannot subscribe to any channel).
  • What is the proper way to subscribe to a channel ?

Given answers to this, I would be willing to update the wiki with the up-to-date information to clear up the usage of this API.

Heck, i'll give 1 BTC to whomever can enable me to stream prices directly from Mt. Gox with this API.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on November 23, 2011, 11:02:13 PM
Same status here, lack of any documentation is pretty irritating.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on November 24, 2011, 02:58:37 PM
Same status here, lack of any documentation is pretty irritating.

magicaltux has some php sample code that was helpfull for me to code socket.io handshake in python.

(it's still got some debugging output and heartbeat lost detection is not implemented, so on connection drop it doesn't auto-reconnect). oh, and it only does websocket, no other transport.

Quote from: socketio.py
from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket import WebSocket


class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    data = urllib.urlencode({})
    req = urllib2.Request('https://' + S.url + "/1", data)
    response = urllib2.urlopen(req)
    r = response.read().split(':')
    S.heartbeat_interval = int(r[1])
    print 'heartbeat: ', S.heartbeat_interval
    if 'websocket' in r[3].split(','):
      print "good: transport 'websocket' supported by socket.io server ", S.url
      S.id = r[0]
      print "id: ", S.id

    S.thread = Thread(target = S.thread_func)
    S.thread.setDaemon(True)
    S.thread.start()

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
      
  def keepalive_func(S):
    while S.run:
      time.sleep(S.heartbeat_interval)
      S.ws.send('2::');
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if True:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)


I got that websocket implementation from giel (on irc) ("from websocket import WebSocket"), here it is (put in file websocket/__init__.py):

  http://pastebin.com/jrxachSF

You can start using

  #> python socketio.py

it should dump the messages to stdout


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on November 24, 2011, 03:04:42 PM
Quote
    S.ws.send('1::/mtgox')

*headdesk*
/me bashing all developers writing incomplete documentations and expecting that some things are so obvious they don't need to be in doc.

molecular, you're my savior. What's your donation address?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on November 24, 2011, 03:07:21 PM
Quote
   S.ws.send('1::/mtgox')

*headdesk*
/me bashing all developers writing incomplete documentations and expecting that some things are so obvious they don't need to be in doc.

molecular, you're my savior. What's your donation address?

cool to have been of help, especiall to you, slush.

almost didn't post, but so many complained so I did... instant karma ;)

1JX3YkScnu2RBHjUBZTLB6qBgmGeUWxANL

EDIT: slush, are you coming to Prague? I'm there on saturday and maybe sunday, maybe I recognize you ;)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: slush on November 24, 2011, 03:35:47 PM
No, I'm not coming to Prague...
(...dramatic drums...)
...because I already am in Prague!.

Would be nice to meet you, however I'm myself curious how I'll be able to communicate with my "fluent" english :-).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on November 24, 2011, 03:41:29 PM
No, I'm not coming to Prague...
(...dramatic drums...)
...because I already am in Prague!.

Would be nice to meet you, however I'm myself curious how I'll be able to communicate with my "fluent" english :-).

Nice. Hoping to meet you. Thanks for the generous token of appreciation you sent me ;)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: jothan on November 24, 2011, 05:03:43 PM
Quote
   S.ws.send('1::/mtgox')

almost didn't post, but so many complained so I did... instant karma ;)

1JX3YkScnu2RBHjUBZTLB6qBgmGeUWxANL

EDIT: slush, are you coming to Prague? I'm there on saturday and maybe sunday, maybe I recognize you ;)

+1 BTC, Thanks !


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on December 11, 2011, 05:33:59 PM
I created project for .NET socket.io + WebSockets at codeplex.com
http://socketiowebsockets.codeplex.com/ (http://socketiowebsockets.codeplex.com/)
for now it can hold ssl-connection with mtgox (no parsing json and etc)

my old ideas (in this topic) died and raised again :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on January 07, 2012, 01:03:58 PM
..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Instead of using socket.io (which I cant get to work) I could as well just connect to the underlying websocket just as before, not? I will not have fallback to longpoll etc in case the websocket dies and the socket.io still works. But besides that, my ticker should happily work with just websockets for the future, not?

Is there a new websocket daemon running now, for socket.io, with a different adress?

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?


Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on January 07, 2012, 01:08:29 PM
..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".
Even though it's "deprecated" it's still much more reliable than the socket.io interface.
So please don't turn it off!

The only problem is that it doesn't work with the latest Chrome and other late WebSocket libs... and nobody wants to fix it. :(
That's probably why they labeled it "deprecated".


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on January 08, 2012, 01:26:34 PM
..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Instead of using socket.io (which I cant get to work) I could as well just connect to the underlying websocket just as before, not? I will not have fallback to longpoll etc in case the websocket dies and the socket.io still works. But besides that, my ticker should happily work with just websockets for the future, not?

Is there a new websocket daemon running now, for socket.io, with a different adress?

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?


Ente

the way it works: you connect to socket.io server and it will give you a unique ID which will be part of the url you can subsequently use to connect to websocket. socket.io is just a "negotiation protocol".

look at my python code here: https://bitcointalk.org/index.php?topic=5855.msg629600#msg629600


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on January 08, 2012, 01:28:14 PM
..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".
Even though it's "deprecated" it's still much more reliable than the socket.io interface.
So please don't turn it off!

The only problem is that it doesn't work with the latest Chrome and other late WebSocket libs... and nobody wants to fix it. :(
That's probably why they labeled it "deprecated".

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.

check out sample implementation: https://bitcointalk.org/index.php?topic=5855.msg629600#msg629600


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on January 08, 2012, 01:37:07 PM
"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.
So you see the new "indirect connection" working reliably?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on January 08, 2012, 04:02:31 PM
"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.
So you see the new "indirect connection" working reliably?


no, not reliable since roughly a week, lots of outages on the websocket part, pretty stable many weeks before that. havent seen the socket.io part fail yet.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: bzzard on January 08, 2012, 04:09:51 PM
havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on January 08, 2012, 04:12:45 PM
havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

socket.io is just a tool to negotiate a transport layer and keepalive. one of the possible transports is websocket. this means you usually use socket.io PLUS websocket (at least that's how I do it).

I'm guessing a connected websocket with no data flowing means some problem in magicaltux' backend.

any insights appreciated.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on January 09, 2012, 07:56:18 PM
request to "https://socketio.mtgox.com/socket.io/1" doesnt respond (net::ERR_CONNECTION_REFUSED)
Is URL valid?
something happened.. but WHAT happened?..

UPD1:
P.S.
I wrote to Mt.Gox support issue: open some technical twitter channel with info: what? when? how long?
IRC-channel is just flame - no info, no logs, no history, no search..

UPD2 (2:28 AM):
its working now


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on January 11, 2012, 03:06:49 AM
request to "https://socketio.mtgox.com/socket.io/1" doesnt respond (net::ERR_CONNECTION_REFUSED)
Is URL valid?
something happened.. but WHAT happened?..

UPD1:
P.S.
I wrote to Mt.Gox support issue: open some technical twitter channel with info: what? when? how long?
IRC-channel is just flame - no info, no logs, no history, no search..

UPD2 (2:28 AM):
its working now

yeah, I see this, too from time to time. also a lot of times the connection setup works fine, but there's just no data flow. mtgox datastreaming is on and off on most levels as far as I can tell.

supposedly socket.io is some typical bay-area shit and there's a big rewrite pull request pending for a while now that could fix the issue mtgox is suffering from.

my latest "kind-of-official" info is that the socket.io server is now restarted every 2 hours to mitigate effects of this (and potentially other) issue(s).

I for one am preparing to survive using api requests (switched form api/0 to api/1)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on January 11, 2012, 09:53:33 AM
I see data
Quote
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"private","origin":"broadcast","private":"depth","depth":{"price":"5.80025","type":2,"type_str":"bid","volume":"0","price_int":"580025","volume_int":"0","item":"BTC","currency":"USD","now":"1326266472912980","total_volume_int":"19811387"}}    
why exists messages with volume=0 ?
is this rounding errors in MtGox's guts or users posts such orders to exchange ?

+++ existence of "total_volume_int" - is wonderful


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: rwcp97 on January 11, 2012, 10:20:27 AM
havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

I have the same problem using the websocket from my C++/Qt-bot - sometimes, usually after some hours working correctly, there is no more data coming in through the socket (but the connection isn't closed by the remote host).
So I check for incoming data and if there isn't anything for 3 minutes, I close the connection and reconnect. Afterwards it's working without problems again. Strange anyway  ???


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: notme on January 13, 2012, 01:54:06 AM
I'm trying to connect via the SocketIO Ruby gem.  When I try to subscribe to the trades channel the following string is passed to the server: ""\x003:::{\"channel\":\"dbf1dee9-4f2e-4a08-8cb7-748919a71b21\",\"op\":\"subscribe\"}\xFF".  The server response with a json message: {"op":"remark","success":false,"message":"Unknown command","debug":{"op":"client","uuid":"5a1f3b8f-1e03-462b-bd38-e463bb39ff2e","data":{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"subscribe"}}}

The "data" field seems to be the message I intended to submit, but the subscribe fails with "Unknown command".  Any help would be appreciated.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on January 13, 2012, 09:37:48 PM
The "data" field seems to be the message I intended to submit, but the subscribe fails with "Unknown command".  Any help would be appreciated.
my subscribtions (raw socket.io messages over websocket - no explicit frames borders) are:
   "4:::{"op":"mtgox.subscribe","type":"trades"}"
   "4:::{"op":"mtgox.subscribe","type":"ticker"}"
   "4:::{"op":"mtgox.subscribe","type":"depth"}"


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: notme on January 13, 2012, 11:43:05 PM
Thanks... I changed channel to type and subscribe to mtgox.subscribe and now it works.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on January 20, 2012, 04:44:26 PM
Is it possible to query my open trades and/or cancel an order or at least all orders? With the websocket API, that is?
Didnt see any hint that this is possible..

But then I dont remember to have ever seen the commands to post orders via websocket neither?

Is the websocket for "reading" only, like making graphs from the data?

edit:
So, get data from the websocket, and exectue/cancel orders via "API version 1" then?

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: rwcp97 on January 21, 2012, 08:57:16 AM
Is it possible to query my open trades and/or cancel an order or at least all orders? With the websocket API, that is?
Didnt see any hint that this is possible..

But then I dont remember to have ever seen the commands to post orders via websocket neither?

Is the websocket for "reading" only, like making graphs from the data?

edit:
So, get data from the websocket, and exectue/cancel orders via "API version 1" then?

Ente

As far as I know there is no possibility to do orders over the websocket... Would be pretty dangerous too, as it isn't SSL encrypted.
But you can subscribe to your "private channel" to be informed about your orders. Use getKey.php to get its name.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on January 21, 2012, 09:53:14 AM
As far as I know there is no possibility to do orders over the websocket... Would be pretty dangerous too, as it isn't SSL encrypted.
But you can subscribe to your "private channel" to be informed about your orders. Use getKey.php to get its name.

socket.io supports secure connections (https, wss transports). That's what I use in ICBIT trading platform, so I would be quite surprised if that's dangerous ;)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: rwcp97 on January 21, 2012, 10:07:08 AM
socket.io supports secure connections (https, wss transports). That's what I use in ICBIT trading platform, so I would be quite surprised if that's dangerous ;)

ok, I wasn't aware of that - I just connect to websocket.mtgox.com at port 80.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on January 21, 2012, 10:10:12 AM
Thank you for the hints, rwcp and Fireball!

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: osmosis on January 25, 2012, 02:14:38 AM
 #> python socketio.py
it should dump the messages to stdout

Thanks molecular.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: is4tomj on January 25, 2012, 03:04:54 AM
I'm using the streaming feed on my website.  If anyone needs a working example feel free to take a look. (http://BitcoinTitan.com)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on January 25, 2012, 10:03:34 AM
Let me say here that the socket.io/websocket feed has been rather stable in late times. I'm hoping this has nothing to do with the fact that trading volume is declining, but is due to Magicaltux making things better. In that case: thanks, Magicaltux and/or mtgox team.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on January 25, 2012, 01:14:54 PM
I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: osmosis on January 25, 2012, 01:32:01 PM
I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Works for me. Sounds like an issue on your side.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on January 25, 2012, 01:34:50 PM
I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Works for me. Sounds like an issue on your side.
But which language / websocket library do you use?

I tried with latest Chrome and this extension: https://chrome.google.com/webstore/detail/pfdhoblngboilpfeibdedpjgfnlcodoo
.... and also tried with websocket package from Go - and they both disconnect me immediately after sending "7:::1+0"


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on January 25, 2012, 01:45:21 PM
Or just try this page: http://websocket.org/echo.html
... using i.e.  ws://socketio.mtgox.com:80/socket.io/1/websocket/14001291971574386363
and tell me that I'm crazy :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on January 26, 2012, 03:11:16 PM
I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.
If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: piotr_n on January 26, 2012, 03:12:54 PM
I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.
If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://
shit man.
thank you!!!
that was indeed the problem.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: osmosis on January 27, 2012, 09:35:00 PM
 #> python socketio.py
it should dump the messages to stdout

Thanks molecular.

For the websocket in python, anyone find a way to deal with the disconnect errors?

File "/home/osmosis/proj_venv/proj_proj/websocket_client.py", line 161, in recv
SSLError: [Errno 1] _ssl.c:1325: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1325: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1325: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1325: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 8] _ssl.c:1325: EOF occurred in violation of protocol
ConnectionClosedException


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on February 13, 2012, 08:32:28 AM
TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?


Some days ago I connected to the old, deprecated websocket. Connecting works, but no data comes in any more.

Now thats an incentive to figure out socket.io :-)

edit:

As far as I know there is no possibility to do orders over the websocket...

Seems like now you can indeed do orders over socket.io:

Quote from: Bitcoin Wiki link=https://en.bitcoin.it/wiki/MtGox/API#Streaming_API
Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the "version 1" requests from the HTTP API can be called.

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..

Any hints?

Ente



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on February 14, 2012, 05:57:55 PM
So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..
Any hints?
in the past (3+ weeks ago):
by default u dont subscribed to channels
u may send simple strings over internal (socket.io's) websocket
for "2::" u must send "2::" (or send "2::" every 15secs)
for subscribe u must send post #162 (https://bitcointalk.org/index.php?PHPSESSID=de60841ac9ca67c95a5ff7b992a5fb22&topic=5855.msg690244#msg690244)
datas stopped in socket after 3-15min.. u can open & sync 2 socket.io..
for now:
post #162 looks no more working - no datas except "1::" & "2::"
???


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on February 14, 2012, 07:39:05 PM
OH! Thank you a lot for your help, zelyony!

I was already wondering about the heartbeat. I now answer 1:: and 2:: accordingly, now they keep flowing in constantly.
Well, the socket *must* be working, since the websites using it still work. There would be more outcry in the forum if it was dead altogether..

I am glad I'm not the only one here!

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on February 14, 2012, 10:29:19 PM
*bangs head on desk*
I cant get it to work.

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

Here is my (totally stripped) python code:

Code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from websocket import create_connection
import urllib
import re
import time

try:
    import json
except ImportError:
    import simplejson as json


wsurl = "wss://socketio.mtgox.com/socket.io/"
iourl = "https://socketio.mtgox.com/socket.io/"

############

print "Connecting to socket.io.."
url = iourl + "1"
f = urllib.urlopen(url)
output = f.read()
f.close()
ausgabe = re.search('[0-9]+', output, 0)
sessionid = ausgabe.group()

print "New websocket url:"
wssurl = wsurl + "1" + "/websocket/" + sessionid
print wssurl

############

print "Connecting to websocket..\n"

socket = create_connection(wssurl)

while 1:
  echo =  socket.recv()
  if "1::" in echo:
    print "Heartbeat 1"
    socket.send("1::")
  elif "2::" in echo:
    print "Heartbeat 2"
    socket.send("2::")
  else:
    print "Message: ",echo
  time.sleep(2)
      

The websocket client I use here is from http://pypi.python.org/pypi/websocket-client/0.4 (http://pypi.python.org/pypi/websocket-client/0.4).
I have a rudimentary heartbeat added here.
The whole script dies eventually, from having stripped all error-handling.
It worked some weeks ago. If you like to see the whole script, let me know.

Bedtime here, will check back in, like, 9 hours..

*frustrated and tired*

Ente



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on February 15, 2012, 08:49:30 AM
1 BTC bounty for the first person who helps me receive regular data again over the websocket!

I made a new thread for this error-solving bounty:
https://bitcointalk.org/index.php?topic=63969.msg749747#msg749747 (https://bitcointalk.org/index.php?topic=63969.msg749747#msg749747)

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on February 15, 2012, 10:08:59 AM
right now datas arrived (my format after string -> json -> object -> string):
JSON: Packet={ Depth={ price=4,67000 USD,       vol=0,05300000,         dir=bid,        when=2012:02:15_10:03:56.735844 } }
JSON: Packet={ Ticker={ bid=$4.68100,   ask=$4.70075,   lo=$4.21211,    hi=$4.97000,    last=$4.70075,  vol=256,165.42727403??BTC } }

I dont know Python, but:
- on connected (or after recv "1::" - that's mean u connected now) u must send strings like:
4:::{"op":"mtgox.subscribe","type":"trades"}
- on "2::" u must response "2::" (not "1::" as in ur sample)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on February 15, 2012, 10:19:43 AM
Thank you for checking, zelyony!
You are right, thats a typo in the snippet. I sent '1::' and '2::' accordingly in my full script, typo corrected.

I sent '4:::{"op":"mtgox.subscribe","type":"trades"}' over the websocket, no change. No response whatsoever, no trade-messages coming in.

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: zelyony on February 15, 2012, 10:39:41 AM
and right now I dont recv any useful datas  ???

UPD:
it's working right now (13:44 UTC)

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

To MtGox:
maybe it's sabotage.. Slow Read DoS attack.. bad guys overflow send pool with slow connections and.. voila!
will be better for all clients drop connections that cannot read string, i.e. send to client not finished for.. hmm.. 0.5sec! +ban by IP 10min (epoll, select or other asynchronous mechanism). I took digits from air, I dont know % loading for servers machines and number of connections.
p.s. ping ("2::") works always


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on February 22, 2012, 03:39:23 PM
I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on February 22, 2012, 04:13:57 PM
UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

Same here. Sometimes it takes ages to receive any data, sometimes less than a minute. Totally random, and totally annoying. The 1:: and 2:: pings work straight from the beginning.
As soon as I get a response to my channel-connect, data from this channel comes in too.

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on February 27, 2012, 08:36:15 AM
I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

maybe this helps, my socketio.py:

Quote
from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket_client import create_connection
import traceback

class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    try:
      data = urllib.urlencode({})
      req = urllib2.Request('https://' + S.url + "/1", data)
      print 'https://' + S.url + "/1"
      response = urllib2.urlopen(req)
      r = response.read().split(':')
      S.heartbeat_interval = int(r[1])
      #print 'heartbeat: ', S.heartbeat_interval
      if 'websocket' in r[3].split(','):
        print "good: transport 'websocket' supported by socket.io server ", S.url
        S.id = r[0]
        print "id: ", S.id
      else:
        print "error: transport 'websocket' NOT supported by socket.io server ", S.url
        
      S.thread = Thread(target = S.thread_func)
      S.thread.setDaemon(True)
      S.thread.start()
    except:
      traceback.print_exc()      

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = create_connection(my_url)
    
    #S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
    S.ws.close()
      
  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect()
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if False:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)

EDIT: the websocket_client imported I got from here: http://pypi.python.org/pypi/websocket-client/0.4

EDIT2: I've been using this implementation for months and it works well for me. If you spot any problems with this implementation of fix errors or otherwise improve it, please let me know.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on February 27, 2012, 05:27:52 PM
You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:
  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 04, 2012, 05:26:24 PM
Anyone else getting errors like these with molecular's script?

Code:
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 06, 2012, 11:41:24 PM
You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:
  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

Thanks, I put that in just in case I need it some day ;)

Anyone else getting errors like these with molecular's script?

Code:
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

I have no idea, never seen this. But maybe you could try with the websocket_client.py I use (change that import back from "websocket" to "websocket_client" and the get file websocket_client.py from pastebin: http://pastebin.com/CTGj735u). I used to have a lot of troubles and tried at least 3 different websocket implemenations until I had a properly working one.

I _have_ seen "EOF occurred in violation of protocol", but I guess that's just a nasty disconnect...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 07, 2012, 04:33:44 PM
Cool. Thanks. I'm trying it out now as we speak. Will report back how it works.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 20, 2012, 07:37:38 AM
I'm still experiencing these errors with websocket_client unfortunately :-\
Code:
08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

So, not sure what to do about this... seems something isn't right somewhere (duh!).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 20, 2012, 08:09:32 AM
I'm still experiencing these errors with websocket_client unfortunately :-\
Code:
08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

I'm experiencing this, too :(
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!


It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:
Code:
08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)
That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: bzzard on March 20, 2012, 08:20:26 AM
I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.
I've tested this topic in details and all I can say that lately (~2 weeks) depth data from socketio is useless. Don't trust it.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 20, 2012, 09:21:48 AM
I'm experiencing this, too :(
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!
One thing that springs to mind is that upon error, we need to make sure that the existing threads exit, before we start new ones (through calling S.connect() in keepalive_func).
As it is now, keepalive_func will keep running after it has tried re-connecting, and as far as I can see, thread_func will keep running as well. So for each reconnect, we have one extra thread of both keepalive and thread_func running, sending and receiving data on the same socket:
Code:
  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect() #this thread will keep running after this. connect creates a new thread_func, which creates a new keepalive_func
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)
Code:
  def thread_func(S):
    print 'SocketIO: websocket thread started'
   
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
   
    S.ws = create_connection(my_url)

    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
   
    msg = S.ws.recv()
    while msg is not None and S.run:
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      msg = S.ws.recv()
    S.ws.close()
As far as I can tell, this can be fixed by breaking out of the while loop in both threads in case of error, replacing the relevant parts like so:
Code:
   def keepalive_func(S):
      while S.run:
         try:
            S.ws.send('2::');
         except:
            if S.run:
               print 'error sending keepalive socket.io, closing existing connection...'
               S.ws.close()
               print '\ttrying reconnect...'
               S.connect()
               raise
               break
            else:
               print 'exiting socket.io keepalive thread'
         time.sleep(S.heartbeat_interval)
and in thread_func:
Code:
         try:
            msg = S.ws.recv()
         except:
            raise
            break
      S.ws.close()

I will see how this runs and report back.

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:
Code:
08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)
That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).
You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?
Indeed I was. I had just accidentally switched off the ignore-non-USD-trades option. Fixed now.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 20, 2012, 11:09:30 AM
The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 20, 2012, 02:37:59 PM
The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py




Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on March 24, 2012, 01:47:40 PM
I tried several hours to create signed commands to send over the socket instead of the old api.
No luck at all. Lastly, I tried to "translate" the php example to python from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands which doesnt look too complicated..




Code:

def get_nonce():
    return int(time.time()*100000)

def encoder(raw):
  dec = json.dumps(raw)
  return dec

def sign_data(secret, data):
    return base64.b64encode(str(HMAC(secret, data, sha512).digest()))


call = "1/info.php"
params = "testparams"
item = "BTC"
currency = "BTC"
id = "1234567890"

Code:
query1 = {'call':call, 'params':params, 'item':item, 'currency':currency, 'id':id, 'nonce':get_nonce()}
Quote
{'nonce': 133259594451109, 'item': 'BTC', 'currency': 'BTC', 'params': 'testparams', 'call': '1/info.php', 'id': '1234567890'}

Code:
query2 = encoder(query1)
Quote
{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}

Code:
sign = sign_data(secret, query2)
Quote
xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==

Code:
query3 = sign, query2
Quote
('xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==', '{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}')

Code:
call = {'op':'call', 'call':base64.b64encode(str(query3)), 'id':id, 'context':'mtgox.com'}
Quote
{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU 5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMD ksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiL CAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}


Finally I take this last message and paste it in my socket.send:
Code:
socket.send("4:::{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMDksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiLCAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}")

Non-descriptive error-reply:

Quote
{u'debug': {u'data': False, u'uuid': u'249bcb53-ccb0-43f1-bac6-e5ae0748e2e1', u'op': u'client'}, u'message': u'Unknown command', u'success': False, u'op': u'remark'}


If someone has enough mercy to have a closer look I would be really happy!
Since this probably takes more than just a few moments, dont forget to quote your bitcoin-address too ;-)

Ente

/who now goes doing some woodwork to clear his head from this


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 24, 2012, 02:17:24 PM
I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on March 24, 2012, 02:35:44 PM
I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 24, 2012, 03:03:17 PM
I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

"socket-only"? I know of the "new" and "old" (1, 2) http apis. And I know of socket.io interface for depth/trades (no auth necessary). Care to explain?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on March 24, 2012, 03:14:01 PM
According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Ente on March 24, 2012, 03:28:05 PM
According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

Yep:

Quote
Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called.
https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

..with more details and a php example too

Ente


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 25, 2012, 08:38:58 PM
The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 26, 2012, 07:38:30 AM
The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

I remember while coding it it didn't feel right to detect for reconnect (only) in the keepalive thread.

Thanks for fixing that, please pastebin your version if you consider it robust.

My one is running stably for days now. I might just not care about the 15 seconds.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on March 27, 2012, 03:18:58 PM
Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 27, 2012, 10:12:20 PM
Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote
def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: sirk390 on April 01, 2012, 03:55:33 PM
Hi,

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 01, 2012, 04:33:18 PM
Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com (http://bitcoin.clarkmoody.com)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: sirk390 on April 01, 2012, 08:41:54 PM
Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com (http://bitcoin.clarkmoody.com)
Thanks but what subscribe URL do you mean ?
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .
It sends a request to "https://socketio.mtgox.com/socket.io/1" and then opens a websocket to "wss://socketio.mtgox.com/socket.io/1/1/websocket/{id}"



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 01, 2012, 08:53:18 PM
Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: sirk390 on April 01, 2012, 11:01:43 PM
Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

Thanks you! It works now although i'm not using exactly the same URL.
I'm using https://socketio.mtgox.com/socket.io/1/?Currency=EUR .

 


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on April 01, 2012, 11:09:38 PM
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .

a newer version of which (with improved thread handling, much more stable concerning reconnects) can be found here https://github.com/molecular/traidor/blob/master/socketio.py

osmosis79 seems to have copied it about a month ago (no objections)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on April 04, 2012, 12:55:16 AM
Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote
def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

Right you are. I think I added it to be able to unsubscribe when a message arrives. But as you say, a global variable works better for this. Fixed in the newest revision.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: knite on April 07, 2012, 02:20:53 PM
runeks and molecular:

I've reviewed and begun using a combination of your codebases. I have a few questions and comments.

- Why the different approaches to your keepalive threads? Molecular's version is more concise. Is it an issue with the ws.recv() delay on disconnect?
- Is there a reason for using the urllib2 module? The requests module is much nicer.
- Why aren't you using the logging module instead of print statements?
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.
- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: knite on April 20, 2012, 07:07:36 PM
Threads are still melting my brain. Can I get the best of both worlds by doing something like this?

Code:
def start_thread(function):
t = Thread(target=function)
t.setDaemon(True)
t.start()
return t

def connect(S):
#setup
start_thread(thread_func)

def reconnect(S):
S.ws.close()
S.connect()

def thread_func(S):
S.ws = create_connection(S.ws_url)
S.run = True
start_thread(keepalive_func)

msg = S.ws.recv()
while msg is not None and S.run:
S.process(msg)
try:
msg = S.ws.recv()
except:
if S.run:
S.reconnect()
break
else:
if run:
S.reconnect()

def keepalive_func():
while S.run:
try:
S.ws.send('2::')
except:
S.ws.close()
break
sleep(S.heartbeat)

The keepalive thread does nothing except send heartbeats and close the connection in the event of an exception. Thread_func reconnects if it hits an exception or exits the while loop, if run is True. Thoughts?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 14, 2012, 06:04:04 PM
Hi knite, nice to see the code being used / looked at and thanks for your comments. I'll comment on some of them...

- Is there a reason for using the urllib2 module? The requests module is much nicer.

No, not from my side. This (coding traidor) was the first time I did any requesting in python. Tried urllib (didn't work for some reason), the tried urllib2 (worked) so I just stuck with that.

- Why aren't you using the logging module instead of print statements?

Forgive my ignorance of the logging module, will take a look next time I need to dump some info.

On a more general note: there are never reasons for not doing something, there can only be reasons for doing something ;). Why do you not believe in the flying spaghetti monster?

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience. Let me point you to Goncalo Pinheira (goncalopp, don't know if on forum, he mailed me, will give you his email upon request in pm), who has been asking me about this and if he could use my code (been asking for lgpl) and who seems to want to do exactly that (exchange abstraction)

- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

done



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Seal on May 23, 2012, 06:55:21 AM
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.


What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: daybyter on May 23, 2012, 03:41:51 PM
Java? I wrote some code for the HTTP API. Maybe we could collaborate for the socket stuff?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 23, 2012, 04:05:29 PM
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience.


What languages do you recommend for best integration with SocketIO and streaming data?

I've been looking into scripting pages with PHP but it seems a little tough to integrate as its a language which generates pages on call.

Also I'd like to write a server-side script to run at home which will accept a socketio feed and place trades automatically. I currently do this again with the json calls API however because its written in PHP, it requires a cron call every x minutes. I'm looking for a better solution.

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Seal on May 23, 2012, 11:39:31 PM

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?


molecular - I currently use a scheduler to execute a PHP call every minute using the http api, however I'm interested in having a live feed for the best prices as they are added to the orderbook. To my knowledge, PHP isnt brilliant at handling streaming feeds and I'm yet to find a way of doing this server-side.

I could stand corrected though, I'm just waiting for a script-kiddy to tell me otherwise.

Whats java like at handling streaming data? How would you run a script like that on a server?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: daybyter on May 24, 2012, 10:26:53 AM
I would simply run it as a daemon on the server. It would poll every x seconds/minutes . I'll drop you a mail with more details and some code (cannot attach anything here).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 24, 2012, 10:48:09 AM

java, node.js, write timer loop yourself in php or use some php scheduling stuff? I don't know. For the kind of stuff we're looking at here, there's good APIs in every language, so it really boils down to personal language preference, I guess. I'd probably use java because I'm most comfortable with it. Unfortunately I cannot recommend my "traidor" code, because the orderbook still gets out of sync.

One remark: I don't know about the stability of the socket.io feed nowadays, but maybe you should implement a fallback to http api?


molecular - I currently use a scheduler to execute a PHP call every minute using the http api, however I'm interested in having a live feed for the best prices as they are added to the orderbook. To my knowledge, PHP isnt brilliant at handling streaming feeds and I'm yet to find a way of doing this server-side.

I could stand corrected though, I'm just waiting for a script-kiddy to tell me otherwise.

Whats java like at handling streaming data?

Are you talking about streaming from your server to some clients or about receiving mtgox socket.io/websocket stream?

If the latter: http://stackoverflow.com/questions/5783086/java-socket-io-client
If the former: http://code.google.com/p/socketio-java/

I never tried any of these, so I don't know how well they handle things.

How would you run a script like that on a server?

If it's also serving web-pages I'd probably run it in the context of a servlet container like tomcat or jetty.

If not, I'd probably write an init-script that just runs "java -jar myjar.jar" and do scheduling within the app. To be honest, I might just start it in a screen (gnu screen) session if I'm the only one depending on it ;)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: daybyter on May 24, 2012, 10:51:45 AM
Perfect. We are thinking exactly in the same lines. :)

I started to write the Java app in a way, that I could run in client or server mode. In client mode, a GUI comes up. In server mode, there's no GUI, so it could be started in an init.d script.

Drop me a PM, if you want to discuss more details.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 24, 2012, 12:26:52 PM
Perfect. We are thinking exactly in the same lines. :)

I started to write the Java app in a way, that I could run in client or server mode. In client mode, a GUI comes up. In server mode, there's no GUI, so it could be started in an init.d script.

Drop me a PM, if you want to discuss more details.


In case you're talking to me: maybe better talk to Seal, I was merely giving ideas to Seal and have currently nothing of the sort planned (except maybe (but unlikely (unless someone coughs up coins)) at some point trading/bot infrastructure and exchange abstraction layer in java)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: daybyter on May 24, 2012, 06:48:06 PM
Scary....maybe you are my long lost twin brother, or so...

I already send a PM to Seal and waiting for some reply with an email, so I can send him some code...

Part of my code is a TradeSite interface (just started), that I already implement for MtGox and BtcE. So you can loop over the sites, combine all trades, compare fees etc.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Seal on May 25, 2012, 12:00:40 AM
I'm here! Yes, I'll reply to your PM now daybyter.

Thanks for the advice molecular. I've been developing trading scripts for a while, mostly in PHP, however I've got to admit, socket.io just confuses me.

It most likely won't be serving client pages. However the script I'd like to write will be based on a server and receiving the mtgox stream, whereby it could then decide on trades itself.


If not, I'd probably write an init-script that just runs "java -jar myjar.jar" and do scheduling within the app. To be honest, I might just start it in a screen (gnu screen) session if I'm the only one depending on it ;)


I dont think my current host offers terminal sessions so ideally it'd run in a standard LAMP build environment.

... either that or I'll have to purchase some sort of virtual box.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 25, 2012, 08:54:03 AM
I dont think my current host offers terminal sessions so ideally it'd run in a standard LAMP build environment.

... either that or I'll have to purchase some sort of virtual box.

oh, ok. hmm. I'm very content with one of these which I pay in BTC: http://www.cinfu.com/vps/


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on May 25, 2012, 08:54:28 AM
Scary....maybe you are my long lost twin brother, or so...

I already send a PM to Seal and waiting for some reply with an email, so I can send him some code...

Part of my code is a TradeSite interface (just started), that I already implement for MtGox and BtcE. So you can loop over the sites, combine all trades, compare fees etc.

be sure to tell this thread (or me) when you have some shareable code. I'm interested ;)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on August 20, 2012, 09:24:23 PM
Is the websocket API still working for everyone else? It worked for me fine until today. Now this is the first message I receive after connecting:

Code:
{'suggest': 'retry', 'error': 'command failed', 'op': 'error'}

This is when I connect to https://socketio.mtgox.com/socket.io as I've always been doing. When I connect to https://socketio.mtgox.com/mtgox the mtgox_io script fails because it can't read the heartbeat interval from the first piece of data received when connecting to the server.
The data received in mtgox_io's connect() function at the line "r = response.read().split(':')" from https://socketio.mtgox.com/socket.io looks like this:

Code:
['9861138011216185358', '60', '60', 'websocket,htmlfile,xhr-polling,jsonp-polling']

While, when connecting to https://socketio.mtgox.com/mtgox, it's:

Code:
['Welcome to Tibanne Websocket']

A welcoming the script doesn't seem to appreciate.

Is it not working for anyone else, or is it just my corner of the internet that's broken?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on August 20, 2012, 10:15:26 PM
Is it not working for anyone else, or is it just my corner of the internet that's broken?

Take a glance at the console on the bottom of bitcoin.clarkmoody.com (http://bitcoin.clarkmoody.com), and you will see the same thing. Streaming data has been disabled (by Gox) for most of the afternoon on my website.

So you're not alone.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: runeks on August 20, 2012, 11:23:09 PM
I'm seeing:

Code:
01:21:39 > Got 3864 asks and 5085 bids.

01:21:38 > Synchronizing the order book.

01:21:16 > Subscribed to depth

01:21:16 > Subscribed to ticker

01:21:16 > Subscribed to trades

01:21:16 > MtGox: Connected.

01:21:16 > Attempting connection over websocket

Looks to me like it's working. What am I missing?

Has Mt. Gox gone public about why it's closed down, as you say?

EDIT: It's working for your site because it seems to be up now: works for me too! Yay :D

Oh live feed, how I've missed you.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: 1455 on August 22, 2012, 09:15:34 AM
I tried for a few hours and crawled the forum for a solution but I'm still not able to fetch websocket depth for EUR.

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com (http://bitcoin.clarkmoody.com)

It works fine for USD with (Ruby-code):

Code:
 s = SocketIO.connect("https://socketio.mtgox.com/mtgox", {:sync => true, :reconnect => :true}) do
  before_start do
    puts "here we are"
    on_json_message do |m|
      m = JSON.parse m
      q << m["depth"] if m["op"] == "private" and m["private"] == "depth"
    end
    on_connect do
      send_message({"op" => "mtgox.subscribe", "type" => "depth"}.to_json)
    end
  end
end

but I wasn't able to get EUR depth with the mentioned solutions.

These are not working:
Code:
s = SocketIO.connect("https://socketio.mtgox.com/mtgox?Currency=EUR", {:sync => true, :reconnect => :true})
Code:
s = SocketIO.connect("https://socketio.mtgox.com/socket.io/1?Currency=EUR", {:sync => true, :reconnect => :true})

Could someone please help me with that? Thanks alot!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: whitslack on October 12, 2012, 09:17:38 PM
Is it just me, or have the event feeds come completely unglued today?

EDIT: Appears to be working again.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on January 14, 2013, 09:58:56 PM
Hi,

my task is to use one socket-io client and subscribe to multiple currencies depth channels.

When I connect with url https://socketio.mtgox.com/mtgox
then I automatically is subscribed to depth, tickers and trades channels.

Depth channel is only for USD messages.

When I connect with https://socketio.mtgox.com/mtgox?Currency=EUR, then automatic subscribtion is done for EUR depth channel
Code:
057bdc6b-9f9c-44e4-bc1a-363e4443ce87
then at the same time I managed also to subscribe mannualy to USD channel using command
Code:
{
"op":"mtgox.subscribe",
"type":"depth"
}

so currently I have both EUR and USD messages comming at the same time.

The question is how can I subscribe to more than two currency depth channels?

My current solution does look more like a hack and I cannot understand how to subscriibe for third currecny GBP and fourth CNY
in order to have mdepth messages for 4 currencies.

please advice


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on February 26, 2013, 01:22:34 PM
Hi!

I'm programmer with expirience making trading software for russian market (RTS/FORTS), and want to offer a few improvements to current MtGox API.
1. Optional filtering for depth messages, by price range and minimal volume, aggregating depth messages by price rounding to 0.001 BTC  (this help to significant reduce traffic).
2. Broadcating with depth and ticker messages some additional fields:  ticker.last*.timestamp, depth.timestamp  as double in milliseconds from midnight GMT.

Now I adopts a program TradeStudio to work with the MtGox:
http://s017.radikal.ru/i413/1302/ca/779a67d6f443t.jpg (http://s017.radikal.ru/i413/1302/ca/779a67d6f443.png)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on February 26, 2013, 01:27:31 PM
Hi,

my task is to use one socket-io client and subscribe to multiple currencies depth channels.

When I connect with url https://socketio.mtgox.com/mtgox
then I automatically is subscribed to depth, tickers and trades channels.

Depth channel is only for USD messages.

When I connect with https://socketio.mtgox.com/mtgox?Currency=EUR, then automatic subscribtion is done for EUR depth channel
Code:
057bdc6b-9f9c-44e4-bc1a-363e4443ce87
then at the same time I managed also to subscribe mannualy to USD channel using command
Code:
{
"op":"mtgox.subscribe",
"type":"depth"
}

so currently I have both EUR and USD messages comming at the same time.

The question is how can I subscribe to more than two currency depth channels?

My current solution does look more like a hack and I cannot understand how to subscriibe for third currecny GBP and fourth CNY
in order to have mdepth messages for 4 currencies.

please advice


for those who are interested there is  a possibility to pass several currencies in the url parameter to subscribe for multiple currencies channels

Code:
https://socketio.mtgox.com/mtgox?Currency=EUR,USD,CNY,EUR


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 04, 2013, 05:07:39 PM
Problems I have found:

* sometimes a signed call like for example "private/info" or "private/orders" will fail for no apparent reason. At the beginning after connection succeeded and I subscribed to all channels (depth, trade, ticker) I send 3 signed calls ("private/idkey", "private/orders" and "private/info"), most of the time this works perfectly but sometimes one or two of them will be answered with "invalid call". In an attempt to work around this I just send them again when this error occurs, as often a needed until it finally succeeds. I don't know if I am missing something here (the API is poorly documented), I suspect a bug on the server side. "invalid call" just is not the correct answer when the server is busy or whatever might be causing this.

* I have not found any way to request fulldepth through the websocket API, again there is documentation missing, is this possible at all?

* I have managed to send trading commands like order/add and order/cancel through the streaming API, most of the time it works but sometimes they just go missing, maybe similar to the other calls mentioned above. This shoud not happen, there should be a queue on the server and a valid order should always go through. The HTTP API seems more reliable for trading but it would be nice if I could use the streaming API for that also.

* It would be nice if there were a way to request OHLCV data of certain timeframes from the server. I can request the trade history and build them myself but this is not practical if I want for example a daily chart of the last 365 days, this wold be a mega-download, possibly hitting the rate limit and banning me. Currently I have no idea where to get OHLCV data of arbitrary timeframes from, please implement this, it will save you a lot of bandwidth.

* The connection is lost very often, sometimes its even impossible to reconnect for periods of many minutes, please fix this. As a first step to remove load from your servers I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.

* Please improve the documentation. There are a lot of things that can only be found out by experimenting, I would have edited the wiki myself and added lots of useful information that I collected through experimenting and reverse engineering other snippets of code but the Bitcoin-Wiki authorities don't give me write access and I am NOT willing to pay money for being allowed to work on the wiki, the admins seem to have totally lost the connection to reality.

Maybe a multi-million dollar company like MtGox should be able to publish an *official* API documentation on the *official* MtGox website and not (ab)use some 3rd party wiki for it. Also I would expect a multi-million dollar company like MtGox to be able to afford servers that won't crash or freeze every 10 Minutes.


***

For those who are looking for yet another example: Here is my python client implementation: https://github.com/prof7bit/goxtool  it implements both protocols: plain old websocket and the bizarre socket.io-monstrosity that is always lagging behind the websocket by at least 5 seconds (you can configure which one to use)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on March 04, 2013, 09:09:00 PM
...I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.
...
For those who are looking for yet another example: Here is my python client implementation: https://github.com/prof7bit/goxtool  it implements both protocols: plain old websocket and the bizarre socket.io-monstrosity that is always lagging behind the websocket by at least 5 seconds (you can configure which one to use)

I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

At ICBIT, I provide the API fully via socket.io not only for trading bots, but for the web trading too. As far as I can tell (and our users can confirm that), it works without this kind of glitches, very smooth, and indeed compatible with many different platforms.

I am not that skilled in Python, alas, however I could help you debug/test the problem by using ICBIT's socket.io connection and see if your problems appear there.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 05, 2013, 07:48:27 AM
...I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.
...
For those who are looking for yet another example: Here is my python client implementation: https://github.com/prof7bit/goxtool  it implements both protocols: plain old websocket and the bizarre socket.io-monstrosity that is always lagging behind the websocket by at least 5 seconds (you can configure which one to use)

I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

At ICBIT, I provide the API fully via socket.io not only for trading bots, but for the web trading too. As far as I can tell (and our users can confirm that), it works without this kind of glitches, very smooth, and indeed compatible with many different platforms.

I am not that skilled in Python, alas, however I could help you debug/test the problem by using ICBIT's socket.io connection and see if your problems appear there.

I can confirm the connection drops (especiall when a lot of shit happens).

I can assure you the problems have already existed before the switch from websocket to socket.io.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 05, 2013, 03:59:50 PM
I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

yes, its a websocket connection too, but before connecting you first do a http request to get your own session ID and then use that to get your very own websocket stream. This puts a lot of unnecessary load on the server during connection for *nothing*, especially when one needs to reconnect often because it drops the connection often. The first step of allocating this personalized websocket URL alone takes 30 seconds or longer sometimes, one can almost hear the server groaning on the other side while it is lifting heavy stuff around to allocate me a personalized socket.io session before I can even connect to it.

Plain websocket most of the time connects instantly.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

No, there is no bug on my side, I'm logging the messaes to the console as they arrive, its lagging behind severely. Just try it yourself, run a minimalistic websocket client that does nothing except printing the messages to the console and at the same time run a minimalistic socket.io client in another console window and print the same type of messages and then compare how the messages that you have seen scrolling by in the websocket client will appear 10 or more seconds delayed in the socket.io client.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on March 07, 2013, 07:19:09 AM
For last days, automaticaly subscription for data (ticker,trades,depth) is not performed after connection to socketio.mtgox.com in most attempts. Some times is performed with late, after 10-30 minutes.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 07, 2013, 09:23:33 AM
is plain websocket still considered deprecated? On https://en.bitcoin.it/wiki/MtGox/API/Streaming I cannot find a word mentioning this (anymore). Did they finally get back some common sense? Now the next step would be to fire the PHP-kiddies and hire some experienced Erlang programmers instead to build a brand new matching engine from scratch (or buy one) *before* it all collapses under the sheer weight of unbelievable 5 orders per second.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on March 07, 2013, 01:37:28 PM
is plain websocket still considered deprecated? On https://en.bitcoin.it/wiki/MtGox/API/Streaming I cannot find a word mentioning this (anymore). Did they finally get back some common sense? Now the next step would be to fire the PHP-kiddies and hire some experienced Erlang programmers instead to build a brand new matching engine from scratch (or buy one) *before* it all collapses under the sheer weight of unbelievable 5 orders per second.


I don't know, but I heard MagicalTux is planning some updates. He's getting bashed pretty badly in the "wall thread" about the trade engine performance. Maybe he's getting rid of the socket.io (it could be a bottleneck for some reason).

I don't know what it is but there must be some severe problems when you look at the machine specs: https://mtgox.com/img/pdf/20120831/Transparency.008.jpg and compare that to estimates of the order throughput that has to be digested and resulting lag that happens.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 08, 2013, 09:03:56 PM
@MagicalTux:

When will the streaming API leave the beta phase so that it can be used for financial applications?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Scrat Acorns on March 08, 2013, 09:52:43 PM
No, there is no bug on my side, I'm logging the messaes to the console as they arrive, its lagging behind severely. Just try it yourself, run a minimalistic websocket client that does nothing except printing the messages to the console and at the same time run a minimalistic socket.io client in another console window and print the same type of messages and then compare how the messages that you have seen scrolling by in the websocket client will appear 10 or more seconds delayed in the socket.io client.

The problem is (and has always been) the fallback transports. They are also the reason that socket.io exists. In a world where every browser has RFC 6455 websocket support socket.io has no reason for being. Channels? Meh. Broadcast? Meh. Socket.io with a websocket transport also has significant overhead versus a pure websocket implementation.

Add to that that 0.9 is currently unmaintained (there are over 300 open issues on github). The only way to scale socket.io right now is to throw more machines at it and use RedisStore.

The polling transports (XHR/JSONP) are performance killers but if you disable them you lose IE6-9. They will both leak connections over long periods of time resulting in GC having to manage gigs of useless heap space slowing it down even more.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 09, 2013, 02:17:18 AM
* I have not found any way to request fulldepth through the websocket API, again there is documentation missing, is this possible at all?

Yes, you can do this via call operations with the BTC{currency}/fulldepth endpoint.

Note that this is rate-limited and the rate is pretty low. I've gone over it quite easily during testing. Unless you need the entire depth information it's probably worth sticking with partial depth information via BTC{currency}/depth.

* The connection is lost very often, sometimes its even impossible to reconnect for periods of many minutes, please fix this. As a first step to remove load from your servers I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.

I think the server will deliberately disconnect clients after an hour (at least, this is what it looks like from my logs).

I've only had problems with WebSocket connections in the past few days and I can't connect at all at the moment. It looks as though the TCP connection is established but the server won't sent any responses when the TLS session is being established (or even send any ACK replies…). Ditto for unencrypted connections.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on March 09, 2013, 04:20:32 PM
Socket.io with a websocket transport also has significant overhead versus a pure websocket implementation.

Add to that that 0.9 is currently unmaintained (there are over 300 open issues on github). The only way to scale socket.io right now is to throw more machines at it and use RedisStore.

Could you please elaborate about significant overhead?

https://github.com/LearnBoost/socket.io/blob/master/History.md - recent releases history. I don't see any signs of it being abandoned.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 09, 2013, 05:20:21 PM
Could you please elaborate about significant overhead?

Every layer on top of another layer adds overhead. The overhead in this case (it might also be the way MtGox is using it or how they hooked it up to their other software) must be enormous, otherwise it would not lag behind the plain websocket server by at least 10 seconds during idle times (up to a few minutes at busy times), silently drop API-commands (or not ack them for 2 or more minutes, etc), while at the same time the old websocket server (when it is running) will instantly ACK (op:result) all my commands (for example give me the order-IDs for new pending orders within fractions of a second even when order-lag is high).  On the socketio stream I can sometimes wait 2 minutes for my order-ID while the order lag reading says "idle" all the time.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Scrat Acorns on March 11, 2013, 11:17:16 PM
Could you please elaborate about significant overhead?

Javascript, among other things. Don't get me wrong I love node but it just can't handle memory copying / allocating data structures as fast as a compiled language, which is understandable as it is using a JIT compiler with GC. In any case the difference with native C implementations isn't that big but it's there. I do realise that I'm comparing apples with oranges here but it's a valid comparison since the only stable socket.io server implementation is in javascript.

I forgot to add that the big advantage of socket.io is its (almost) seamless horizontal scalability as long as your redis box is beefy.


https://github.com/LearnBoost/socket.io/blob/master/History.md - recent releases history. I don't see any signs of it being abandoned.

https://github.com/LearnBoost/socket.io/issues?page=1&sort=comments&state=open (https://github.com/LearnBoost/socket.io/issues?page=1&sort=comments&state=open)

Its creator is busy working on 1.0 which is a complete rewrite (uses engine.io as the abstraction for the underlying transports).


it might also be the way MtGox is using it or how they hooked it up to their other software

They are doing it wrong.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 15, 2013, 12:34:34 AM

"Method not found"

is the answer I get from the server for every signed API request (such as private/idkey or private/info or private/orders or order/add, order/cancel etc).

It started happening exactly at 2013-03-14 23:53:48 (central european time), it did not close the connection, it just suddenly stopped accepting any API commands.

Both servers (socketio and websocket) behave this way. I still get ticker, trade and depth but cannot send any trading commands or other commands anymore.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 15, 2013, 12:42:24 AM
"Method not found"

is the answer I get from the server for every signed API request (such as private/idkey or private/info or private/orders or order/add, order/cancel etc).

It started happening exactly at 2013-03-14 23:53:48 (central european time), it did not close the connection, it just suddenly stopped accepting any API commands.

Both servers (socketio and websocket) behave this way. I still get ticker, trade and depth but cannot send any trading commands or other commands anymore.

Yeah, I'm seeing this to. Also seeing "invalid call".

By the way, is it possible to connect to socketio.mtgox.com via WebSocket? That is, a regular RFC 6455 WebSocket and not via the Socket.IO library (which I believe is meant to be able to fall back to WebSocket). I've never had any success with this, I've only ever been able to make WebSocket connections to websocket.mtgox.com on port 80 / 443.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 15, 2013, 01:21:21 AM
fixed within 5 minutes after quick chat in #mtgox on freenode ;-)
Now I know where to go when there are problems


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 15, 2013, 01:30:42 AM

Yeah, I'm seeing this to. Also seeing "invalid call".

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make an unsigned 64 bit integer and not one invalid calls anymore since the day I made that change, before i used something like time.time()*1E6 and saw a lot of "invalid call". One day I'm going to investigate what exactly it was that was wrong with my microseconds and why it works now.

The socketio server only accepts the socketio protocol (http request, then get a session ID, then use that ID to have a new URL to connect to a websocket service but this is not the same websocket as the plain old websocket, all messages are wrapped into some socketio protocol ("1::/mtgox" headers and the like). This is a whole new complicated layer on top of websocket.

According to the person I just talked to the plain websocket server is NOT (I repeat: NOT) deprecated, we can use it! Its also 10 seconds faster (no annoying lag, no sluggishness), reacts instantly to all commands.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 15, 2013, 02:31:56 AM
Yeah, I'm seeing this to. Also seeing "invalid call".

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make a 64 bit integer and no invalid calls anymore, before i used something like time.Time()*1E6 and saaw a lot of "invalid call"

Thanks, I'll try this out.

The documentation and PHP example code led me to believe that the nonce needs to increment for each message. Currently I'm setting the nonce of the first request to the Unix time in microseconds and then incrementing by 1 in each additional request.

It's weird that this would cause problems if a random value works fine. I'll see if I can get some clarification from MtGox.

The socketio server only accepts the socketio protocol (http request, then get a session ID, then use that ID to have a new URL to connect to a websocket service but this is not the same websocket as the plain old websocket, all messages are wrapped into some socketio protocol ("1::/mtgox" headers and the like). This is a whole new complicated layer on top of websocket.

Sounds heinous.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 15, 2013, 03:10:04 AM
Yeah, I'm seeing this to. Also seeing "invalid call".

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make a 64 bit integer and no invalid calls anymore, before i used something like time.Time()*1E6 and saaw a lot of "invalid call"

Thanks, I'll try this out.

The documentation and PHP example code led me to believe that the nonce needs to increment for each message.

STOP

forget everything I just wrote about the nonce, I had a bug in my program (I used struct.unpack() which creates a tuple and put that tuple with the random number into my json instead of only the nonce). And the mtgox-server also has a bug so that it *accepts* this without complaining!

It will *not* accept it if the nonce is a plain random number and not incrementing. And it will never forget, you need a new API key after the change.

I have just spent another hour with my code and now my code look like this, I also requested a brand new virgin API key and now it works:

Code:
    def send_signed_call(self, api_endpoint, params, reqid):
        """send a signed (authenticated) API call over the socket.io.
        This method will only succeed if the secret key is available,
        otherwise it will just log a warning and do nothing."""
        if (not self.secret) or (not self.secret.know_secret()):
            self.debug("### don't know secret, cannot call %s" % api_endpoint)
            return

        key = self.secret.key
        sec = self.secret.secret

        nonce = str(int(time.time() * 1E6))

        call = json.dumps({
            "id"       : reqid,
            "call"     : api_endpoint,
            "nonce"    : nonce,
            "params"   : params,
            "currency" : self.currency,
            "item"     : "BTC"
        })

        # pylint: disable=E1101
        sign = hmac.new(base64.b64decode(sec), call, hashlib.sha512).digest()
        signedcall = key.replace("-", "").decode("hex") + sign + call

        self.debug("### calling %s" % api_endpoint)
        self.send(json.dumps({
            "op"      : "call",
            "call"    : base64.b64encode(signedcall),
            "id"      : reqid,
            "context" : "mtgox.com"
        }))


you find it at github: prof7bit/goxtool


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 18, 2013, 07:52:44 AM
One thing isnt clear to me. when a trade occurs. does the API also send a depth message that reflects this trade? Or do I need to subscribe to the trade channel as well and then  match the trade with the orderbook to keep the orderbook up to date?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 18, 2013, 08:55:09 AM
One thing isnt clear to me. when a trade occurs. does the API also send a depth message that reflects this trade? Or do I need to subscribe to the trade channel as well and then  match the trade with the orderbook to keep the orderbook up to date?

On trade messages you need to update the orderbook yourself, there will be no depth message. [Edit: this is wrong, there is a depth message following every trade message but it can't do any damage if you always only use the total_volume and never use the volume difference]

If the trade is type:"bid" then it has filled an ask order, update your asks, if trade is type:"ask" then it has filled a bid order, update your bids.

On *own* trades there will be two trade messages: One public that everybody receives (like above), use that to update your orderbook like above and one private only to notify you about your trade that happened. Also on every trade that affects your *own* orders there will be user_order messages, at least one that sets the volume to the remaining volume (or zero and another one to finally remove it).

[Edit: the code below is flawed (maybe even outright wrong: You should not use the volume difference in the depth message, you should use the total volume]

WRONG WRONG WRONG
Code:
def on_trade(typ, price, volume, own):
  typ_filled = {"ask":"bid", "bid":"ask"}[typ]
  if own:
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe notify your bot about this event, or notify the user.
    pass
  else:
    update_orderbook(typ_filled, price, -volume)

def on_depth(typ, price, volume):
  update_orderbook(typ, price, volume)

def on_user_order(oid, price, new_volume, new_status):
  update_own_orders(oid, price, new_volume, new_status)
/WRONG /WRONG /WRONG


Edit: Here is a corrected version:
you would have two methods updating the orderbook, one that accepts a delta and one that accepts the new total volume.

Code:
def on_trade(typ, price, trade_volume, own):
  typ_filled = {"ask":"bid", "bid":"ask"}[typ]
  if own:
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe notify your bot about this event, or notify the user.
    pass
  else:
    update_orderbook_with_delta(typ_filled, price, -trade_volume)

def on_depth(typ, price, total_volumel):
  update_orderbook_with_absolute(typ, price, total_volume)

def on_user_order(oid, price, new_volume, new_status):
  update_own_orders(oid, price, new_volume, new_status)

def update_orderbook_with_delta(typ, price, delta_volume):
  # do whatever is needed to update a price level,
  # remove it if it is <= 0 after the update
  [...]

def update_orderbook_with_absolute(typ, price, total_volume);
  # do whatever is needed to update a price level to the new total_volume
  # remove the level if total_volume == 0
  [...]

And here is a simplified version that just waits for the inevitable depth message and completely ignores the trade message:
Code:
def on_trade(typ, price, trade_volume, own):
  if own:
    # an own order has been filled (this is only received in the private channel)
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe notify your bot about this event, or notify the user.
    # the same message will be fired again with vol=False in the public
    # channel like all other trade message too.
    pass
  else:
    # a trade has happened on mtgox (this is the public message)
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe log the message or notify your bot if it needs it
    pass

def on_depth(typ, price, total_volumel):
  update_orderbook_with_absolute(typ, price, total_volume)

def on_user_order(oid, price, new_volume, new_status):
  update_own_orders(oid, price, new_volume, new_status)

def update_orderbook_with_absolute(typ, price, total_volume);
  # do whatever is needed to update a price level to the new total_volume
  # remove the level if total_volume == 0
  [...]


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 18, 2013, 12:01:18 PM
On trade messages you need to update the orderbook yourself, there will be no depth message.

If the trade is type:"bid" then it has filled an ask order, update your asks, if trade is type:"ask" then it has filled a bid order, update your bids.

On *own* trades there will be two trade messages: One public that everybody receives (like above), use that to update your orderbook like above and one private only to notify you about your trade that happened. Also on every trade that affects your *own* orders there will be user_order messages, at least one that sets the volume to the remaining volume (or zero and another one to finally remove it).

thanks a lot. That explains the weirdness I was getting.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 18, 2013, 04:21:50 PM
is websocket service down? I can connected but I cant get any response (nor via telnet).
 Clarkmoody and the other sites still seem to work though?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 18, 2013, 05:46:31 PM
is websocket service down? I can connected but I cant get any response (nor via telnet).
 Clarkmoody and the other sites still seem to work though?

Yes, its down again (for almost 3 hours now [this is nothing, sometimes its down for a few days]). And in the 24/7 IRC support channel #mtgox on freenode (btw. why does a commercial company use freenode?) there is currently nobody answering any questions.

Socketio works (clarkmoody is using socketio) but trading via the socketio server is no fun at all because of the 10 seconds lag (on top of the usual goxlag) even during quiet times and becomes totally unusable when there is some more action.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 18, 2013, 06:25:33 PM
Thanks for confirming its not just me. I thought socketio was implemented on top of websockets though, so its weird that still works?
Either way, Ill make a HTTP fallback option. Do you know how often you can poll it? The documentation says no more than once per 10 seconds, but Im not sure if that is per request or for all requests (I need trades, depth, and ticker) ?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 18, 2013, 07:39:12 PM
Thanks for confirming its not just me. I thought socketio was implemented on top of websockets though, so its weird that still works?

its a separate server. And on the socketio websocket all messages look like this:

0x00 "4::/mtgox:{a json message}" 0xff
0x00 "4::/mtgox:{another json msg}" 0xff

and occasionally:

0x00 "2::" 0xff     <-- this is a ping, must be answered with "2::"

(I'm not totally sure about the 0xff 0x00 thing, there exist 13 different (no joke!) versions of websocket and there exist also different ways to delimit the messages, crazy stuff like: if bit7=1 then bit 0..6 = message length and when the next byte also has bit7=1 then its even more bits for the message length until a byte comes along that has bit7=0 but if all bits = 0 then its variable length ascii, 0xff terminated.

Fortunately the binary stuff is abstracted away by libraries like websocket.py in Python and other client implementations for other languages, so usually you just have a send() and a recv() method and don't need to care about 0x00 and 0xff etc., only the socketio "1::", "2::" and "4::" must be done manually.

The "/mtgox" part is some kind of "channel" name on the socketio layer but MtGox actually does not make any use of this feature, there is only one channel, the channels you subscribe *through* the websocket actually are an MtGox-internal thing, not part of the socketio layer. So there is some socketio protocol related stuff going on and wrapped around it that is not actually used for anything but its needed to satisfy the protocol rules.


On the plain websocket server you just connect, send the request header and it immediately begins talking to you with no additional handshake or other overhead:

0x00 "{a json message}" 0xff
0x00 "{another json msg}" 0xff

So its not the *same* websocket server (also it has a different hostname and IP address). Socketio is using websocket transport but a totally different handshake and protocol and framing.


Either way, Ill make a HTTP fallback option. Do you know how often you can poll it? The documentation says no more than once per 10 seconds, but Im not sure if that is per request or for all requests (I need trades, depth, and ticker) ?

depth (fulldepth) definitely has a limit and I ran into this once while testing (and it tried to ban me from calling this API for 24 hours [getting a new IP address on my DSL line fixed it]). I don't know about the other APIs.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on March 18, 2013, 07:54:06 PM
its a separate server. And on the socketio websocket all messages look like this:

0x00 "4::/mtgox:{a json message}" 0xff
0x00 "4::/mtgox:{another json msg}" 0xff

and occasionally:

0x00 "2::" 0xff     <-- this is a ping, must be answered with "2::"

...I'm not totally sure ...

There is a specification (https://github.com/LearnBoost/socket.io-spec) describing socket.io protocol.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 18, 2013, 08:03:04 PM
Quote
...I'm not totally sure ...
There is a specification (https://github.com/LearnBoost/socket.io-spec) describing socket.io protocol.

With "I'm not totally sure" I was referring to websocket framing (0xff and friends), not to socketio. For websocket there exists an RFC. Websocket is actually not the problem, client implementations exist for many languages.

I actually just wanted to explain that these two MtGox servers are not the same and after writing down the bytes I realized that I might have forgotten some tiny details and therefore wanted to make clear that this is NOT a protocol specification, I just wrote it down as I remember it and so I wrote "I'm not totally sure" as a warning that it might be not 100% correct and nobody tries to implement something based on this posting.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 18, 2013, 08:12:40 PM
This sucks. I spent a day learning about websockets and making goxgame work pretty decently, now I can start over again and it seems got a choice between redoing most of the work while learning (a lot more) javascript and node.js to implement some existing nodejs socket.io implementation and make it talk with my db, or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

Thanks for the long explanation though.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on March 18, 2013, 08:25:19 PM
This sucks. I spent a day learning about websockets and making goxgame work pretty decently, now I can start over again and it seems got a choice between redoing most of the work while learning (a lot more) javascript and node.js to implement some existing nodejs socket.io implementation and make it talk with my db, or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

Thanks for the long explanation though.

Btw, if you're doing node.js+socket.io you can use my example: https://github.com/icbit/trader.nodejs
It's not very well tested and debugged yet, but it supports both Mt.Gox and ICBIT.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 18, 2013, 08:28:34 PM
or  redoing most of the work trying to reverse engineering the socketio server to use websockets. Bah!

If you have a working version for plain websocket already then it should be trivial to add the little socketio overhead on top of it without needing a complete socketio library.

basically instead of just connecting to websocket.mtgox.com like you do it now you do the following:

send a http request to socketio.mtgox.com (look for the example somewhere in this thread for the exact URL), parse the result to get the session ID which is part of your personalized websocket url and then you use *that* to do the websocket connect. The rest is adding (or removing) the "1::" and "4::/mtgox:" headers before/after send/receive and the rest stays the same, its effectively only a handful of code you have to add, you don't have to implement everything of socketio, just these few things, its actually simpler than it might seem.

There was a python example near the beginning (or somewhere in the middle) of this thread that helped me a lot, and of course also the official socketio specification.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 18, 2013, 10:03:54 PM
Ah, you make it sound easy! And it probably is when you know what you are doing, but I dont :)
I found the url to get the session thingy. Im not sure what to do next; this is the closest I got so far:

edit: I guess the handshake should be this instead?
GET /socket.io/1/websocket/454534534534 HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio.mtgox.com
Origin: null

Nothing is happening, no response.

edit2.
I debugged the python app linked above, and that made me use this string:
Code:
GET /socket.io/1/websocket/5360180921750645752 HTTP/1.1
Sec-WebSocket-Draft: 2
Host: socketio.mtgox.com
Upgrade: WebSocket
Connection: Upgrade
Origin: socketio.mtgox.com

Still no response, but now I wonder, do i need to set the authentication thing?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 19, 2013, 12:18:00 PM
I debugged the python app linked above, and that made me use this string:
Code:
GET /socket.io/1/websocket/5360180921750645752 HTTP/1.1
Sec-WebSocket-Draft: 2
Host: socketio.mtgox.com
Upgrade: WebSocket
Connection: Upgrade
Origin: socketio.mtgox.com

Still no response, but now I wonder, do i need to set the authentication thing?

Looks good but remember there exist multiple different versions of websocket and their server might refuse to connect if you use the wrong version. The python example uses the websocket.py module https://pypi.python.org/pypi/websocket-client/ to do the actual websocket connect and so does my program to.

I remember before that I experimented with different implementations of websocket and not all of them worked, one of them was disconnected when sending the headers, i guess because it tried to use the wrong websocket version or something else was wrong with its headers (iirc by investigating the reasons it seemed to have to do with the sec-websocket-* headers and the random generated key, but then I just threw the code away and started from scratch with websocket.py). What I now use is urllib2 to do the initial http request, then construct the URL like above (the url you posted looks good) and then I let the websocket.py do the websocket connect and don't care about low level details like request headers

I don't know how this is in javascript/node.js (assuming this is what you are using?) but I guess there should exist some library too that will let you establish a webocket connection that takes care about all the low-level websocket request header stuff.

this is my working python code if this is of any help for you (its basically doing exactly the same as the other code snippets in this thread): (I guess to solve your problem you need to find the equivalent of "websocket.py" for your programming language)
Code:

        use_ssl = self.config.get_bool("gox", "use_ssl")
        wsp = {True: "wss://", False: "ws://"}[use_ssl]
        htp = {True: "https://", False: "http://"}[use_ssl]

        while not self._terminating: #loop 0 (connect, reconnect)
            try:
                url = urlopen(htp + self.SOCKETIO_HOST + "/socket.io/1?Currency=" + self.currency, timeout=20)
                params = url.read()
                url.close()

                ws_id = params.split(":")[0]
                ws_url = wsp + self.SOCKETIO_HOST + "/socket.io/1/websocket/" + ws_id + "?Currency=" + self.currency

                self.socket = websocket.WebSocket()
                self.socket.connect(ws_url)

                self.debug("connected")

                self.socket.send("1::/mtgox")
                self.socket.recv() # '1::'
                self.socket.recv() # '1::/mtgox'

                self.debug("subscribing to channels...")
                self.channel_subscribe()

                self.debug("waiting for data...")
                while not self._terminating: #loop1 (read messages)
                    msg = self.socket.recv()
                    if msg == "2::":
                        self.debug("### ping -> pong")
                        self.socket.send("2::")
                        continue
                    prefix = msg[:10]
                    if prefix == "4::/mtgox:":
                        str_json = msg[10:]
                        if str_json[0] == "{":
                            self.signal_recv(self, (str_json))

            except Exception as exc:
                if not self._terminating:
                    self.debug(exc, "reconnecting in 5 seconds...")
                    if self.socket:
                        self.socket.close()
                    time.sleep(5)



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 19, 2013, 12:34:33 PM
Im actually using php server side, and there is very little to be found on the web. Only code to create websockets, not to connect to them.

Anyway, I dont mind coding it if I know what to send, but python is mostly like chinese to me, so can you do me one more favor and modify websocket.py
change line 43 to:
_debug = True

Then run your app.
It will spit out the handshakes. If you copy/paste the output  I can try to implement it in php.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 19, 2013, 01:07:05 PM
Im actually using php server side, and there is very little to be found on the web. Only code to create websockets, not to connect to them.

Anyway, I dont mind coding it if I know what to send, but python is mostly like chinese to me, so can you do me one more favor and modify websocket.py
change line 43 to:
_debug = True

Then run your app.
It will spit out the handshakes. If you copy/paste the output  I can try to implement it in php.

You seem to be looking at a different version of websocket.py, there is no _debug variable at line 43 (I'm using version 0.9.0 (this is the latest version, implementing websocket version 13). I have enabled tracing, this is the log until after the channel_subscribe()

Note the messages starting with 0x81, this is the framing to delimit the messages, this is why it is absolutely no fun implementing this by hand if you can get a working implementation from somewhere. Now after looking at this dump myself I remember, my first attempt writing the webocket client myself the first problem I stumbled upon were the Sec-Websocket-Key and the Sec-Websocket-Version headers, they look slightly different for every of the dozen ws versions that exist and MtGox does not like all variants of them. The funny thing is my first implementation worked just fine with their plain websocket server and refused to connect to the socketio. Then i threw it away and used websocket.py.

Code:
2013-03-19 13:48:54,752:DEBUG:SocketIOClient:trying Websocket: wss://socketio.mtgox.com/socket.io/1/websocket/10083243691392078721?Currency=USD ...
2013-03-19 13:48:59,312:DEBUG:--- request header ---
2013-03-19 13:48:59,313:DEBUG:GET /socket.io/1/websocket/10083243691392078721?Currency=USD HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio.mtgox.com:443
Origin: socketio.mtgox.com:443
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13


2013-03-19 13:48:59,313:DEBUG:-----------------------
2013-03-19 13:48:59,313:DEBUG:--- response header ---
2013-03-19 13:48:59,762:DEBUG:HTTP/1.1 101 Switching Protocols
2013-03-19 13:48:59,763:DEBUG:Upgrade: websocket
2013-03-19 13:48:59,763:DEBUG:Connection: Upgrade
2013-03-19 13:48:59,764:DEBUG:Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=
2013-03-19 13:48:59,764:DEBUG:-----------------------
2013-03-19 13:48:59,764:DEBUG:SocketIOClient:connected
2013-03-19 13:48:59,767:DEBUG:send: '\x81\x89\x9f\x99\x10\xa1\xae\xa3*\x8e\xf2\xedw\xce\xe7'
2013-03-19 13:48:59,767:DEBUG:recv: '\x81\x031::'
2013-03-19 13:49:00,303:DEBUG:recv: '\x81\t1::/mtgox'
2013-03-19 13:49:00,304:DEBUG:SocketIOClient:subscribing to channels
2013-03-19 13:49:00,306:DEBUG:send: '\x81\xb4\xe7\xc2\xc8\xda\xd3\xf8\xf2\xf5\x8a\xb6\xaf\xb5\x9f\xf8\xb3\xf8\x93\xbb\xb8\xbf\xc5\xf8\xe8\xf8\x83\xa7\xb8\xae\x8f\xe0\xe4\xfa\xc5\xad\xb8\xf8\xdd\xe2\xea\xb7\x93\xa5\xa7\xa2\xc9\xb1\xbd\xb8\x94\xa1\xba\xb3\x85\xa7\xea\xa7'
2013-03-19 13:49:00,307:DEBUG:send: '\x81\xb5\x8c\xda\xf7\x7f\xb8\xe0\xcdP\xe1\xae\x90\x10\xf4\xe0\x8c]\xf8\xa3\x87\x1a\xae\xe0\xd7]\xf8\xb3\x94\x14\xe9\xa8\xd5S\xac\xf8\x98\x0f\xae\xe0\xd7]\xe1\xae\x90\x10\xf4\xf4\x84\n\xee\xa9\x94\r\xe5\xb8\x92]\xf1'
2013-03-19 13:49:00,307:DEBUG:send: '\x81\xb5w\xf2\x8b\xbeC\xc8\xb1\x91\x1a\x86\xec\xd1\x0f\xc8\xf0\x9c\x03\x8b\xfb\xdbU\xc8\xab\x9c\x03\x80\xea\xda\x12\x81\xa9\x92W\xd0\xe4\xceU\xc8\xab\x9c\x1a\x86\xec\xd1\x0f\xdc\xf8\xcb\x15\x81\xe8\xcc\x1e\x90\xee\x9c\n'

...and so on. Now its subscribed to all channels and should start sending data,



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 19, 2013, 01:31:10 PM
thanks for that. Before I try.. what that websocket key? It doesnt look anything like what I get from
https://socketio.mtgox.com/socket.io/1

I get  5360180921750645752

Is that related to your mtgox api key (and if so, is it safe to publish?) and if so, why doesnt it look anything like mine?
Or is it a fixed key and I can just use it?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 19, 2013, 02:37:14 PM
thanks for that. Before I try.. what that websocket key? It doesnt look anything like what I get from
https://socketio.mtgox.com/socket.io/1

I get  5360180921750645752

Is that related to your mtgox api key (and if so, is it safe to publish?) and if so, why doesnt it look anything like mine?
Or is it a fixed key and I can just use it?

The 5360180921750645752 you get from the initial http request is the socketio session id, it belongs to the socketio protocol. It is generated by the server, see the socketio document on github (posted by Fireball a few postings before). its only used to generate the websocket GET request (the websocket url)

The Sec-Websocket-Key belongs to the websocket protocol (see: http://datatracker.ietf.org/doc/rfc6455/?include_text=1 (http://datatracker.ietf.org/doc/rfc6455/?include_text=1)) its a random 16 byte you create and then base64 encode (see section 11.3.1 and section 4.1 (page 17, article 7))

_____
PS: The websocket protocol as defined in rfc6455 exists primarily to:
 * ensure that the google engineers who created it have secure jobs until the end of their life
 * ensure that nobody will ever write a *complete* reference implementation of serer and client
 * prove that it is possible to embed utter nonsense that serves no purpose like for example the masking of client-to-server data as specified in RFC6455 in an RFC and nobody recognizing it as what it really is: an april fools joke.
 * [...]
 * to establish a bidirectional connection between A and B

in exactly the above order.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 19, 2013, 02:52:40 PM
Cheers mate! With that I think I will make it work. Your help is much appreciated.
In case you are wondering, this is what Im working on:
http://goxgame.sytes.net/
Click legend to show/hide stuff, click and drag to zoom in.
Warning: buggy, obviously incomplete data, cant keep order book synched  etc,  its definitely still WIP.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 20, 2013, 11:08:55 PM
My 2 Satoshi:

* ensure that nobody will ever write a *complete* reference implementation of serer and client
Really? I find it one of the simpler and better written networking specs out there (assuming you know some HTTP). I wrote a client library from scratch in a few days (~1000 lines of code) that I believe is complete. Don't Firefox and Chrome have complete client implementations with a Javascript API?

I also have a partial server implementation but that's because I don't need a WebSocket server for anything. Most of the work is already done because the framing and handshake code can be shared with the client version.

My only complaints are that it unnecessarily relies on crypto libraries (SHA-1 and base-64 for the handshake is overkill) and that the binary / text message distinction is kind of redundant. Perhaps it makes sense in the context of web browsers and Javascript.

* prove that it is possible to embed utter nonsense that serves no purpose like for example the masking of client-to-server data as specified in RFC6455 in an RFC and nobody recognizing it as what it really is: an april fools joke.
I also did a double-take when I read about the masking. However, I think it might make sense to avoid interference from web proxies. The part of the RFC that justifies masking as a way to avoid 'attacks on infrastructure' is utter bullshit though.

If you're worried about the performance hit then just set the mask to all zeros and have a fast path in the software that doesn't perform the XOR.

Fragmented messages may also be unnecessary but might be useful in low-latency applications. It doesn't add much complexity to handle them.

* to establish a bidirectional connection between A and B
A real bidirectional TCP connection is a pain when you factor in web proxies, firewalls, NAT, etc. WebSocket is a reasonable alternative with relatively little overhead per message.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 21, 2013, 09:26:50 PM
I can see my socket.io client connects to mtgox and connect event is invoked.

and then no any subscribtion message at all nothing more.

this behavior is already for several hours.

restarting client does not help.

Clarks's client shows the same. Connects and then silence.

Does anybody see the same?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on March 21, 2013, 09:30:24 PM
Clarks's client shows the same. Connects and then silence.

Does anybody see the same?

Total MtGox tease. Usually happens during high-traffic times.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 21, 2013, 09:33:53 PM
Clarks's client shows the same. Connects and then silence.

Does anybody see the same?

Total MtGox tease. Usually happens during high-traffic times.

Thank you. Just wanted to be sure that it is not my local problem.

By the way I wanted to thank you for your hint about orderbook state synchronization.

full book stamp and 60 secs depth channel buffer + ticker fixing made my mtgox stream api client work very well :)
Thank you once more :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 22, 2013, 07:46:06 PM
i seem to be able to connect to websocket occasionally; but socket.io seems completely non responsive now?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 23, 2013, 03:57:55 PM
i seem to be able to connect to websocket occasionally; but socket.io seems completely non responsive now?

Yesterday in the #mtgox channel on freenode MT has finally announced the hostname of the new beta server: socketio-beta.mtgox.com

This new server seems much improved, I have not yet seen it disconnecting or acting strange a single time since yesterday (maybe partially also because only few are using it yet).

goxtool now also supports this new server: pull the latest release and then:

Code:
./goxtool.py --protocol=socketio-beta


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 23, 2013, 04:04:48 PM
so the only thing that have to be changed is url? Any changes to socket.io lib version necessary?


are you sure you placed correct url? I tried and getting multiple reconnects and errors from socket io client?

can you check?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 23, 2013, 04:33:25 PM
so the only thing that have to be changed is url? Any changes to socket.io lib version necessary?


are you sure you placed correct url? I tried and getting multiple reconnects and errors from socket io client?

can you check?

This also seems to use different server software and it at the moment it seems it is essential that the websocket connection is done on the same open socket that the initial http request came from. So the client has to fetch the session ID (with the header Connection: keep-alive) and then use that same socket to  upgrade the connection to websocket. Otherwise it will fail with reason "client not handshaken". MT said this is a bug and he intends to fix it. Meanwhile I have tweaked my own socketio client to be able to use the same socket for both requests, so for me it works. I don't know how the javascript implementation handles this.

Code:
<prof7bit> socketio-beta rocks. no disconnect no freeze, thats how it should have been all the time
<MagicalTux> heh
<MagicalTux> prof7bit: except connection will usually fail if you do not use the same socket for the initial request and the websocket
<prof7bit> that is fixed in my client now. it was actually simpler to fix than I expected
<MagicalTux> ya, but that's something /we/ should be actually fixing
<MagicalTux> we'll probably disable everything but websocket and flashsocket, and make socket.io server accept any session id


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 23, 2013, 05:08:20 PM
Does it mean that it is actually not original socket.io client have to be used but rather different client implementation?

Do you have any link to the new socketio-beta project?

or what I did not get?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 23, 2013, 07:53:50 PM
Using puTTY I tried:

=====================================
websocket.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null


Server replied with handshake ACK.

Then I was getting the stream displayed, until server seemed to drop connection. I tried this a few times with success and now cant repeat.

=====================================
socketio-beta.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio-beta.mtgox.com
Origin: null


No reply, Server remains silent?
=====================================


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Puppet on March 23, 2013, 08:38:45 PM
Im getting this error using the beta socketio server with the javascript library:


DEBUG: Error: Hostname/IP doesn't match certificate's altnames
    at SecurePair.<anonymous> (tls.js:1280:23)
    at SecurePair.EventEmitter.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:883:10)
    at CleartextStream.read [as _read] (tls.js:421:15)
    at CleartextStream.Readable.read (_stream_readable.js:293:10)
    at EncryptedStream.write [as _write] (tls.js:330:25)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)
    at EncryptedStream.Writable.write (_stream_writable.js:172:11)
    at write (_stream_readable.js:547:24)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 23, 2013, 08:47:32 PM
Im getting this error using the beta socketio server with the javascript library:


DEBUG: Error: Hostname/IP doesn't match certificate's altnames
    at SecurePair.<anonymous> (tls.js:1280:23)
    at SecurePair.EventEmitter.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:883:10)
    at CleartextStream.read [as _read] (tls.js:421:15)
    at CleartextStream.Readable.read (_stream_readable.js:293:10)
    at EncryptedStream.write [as _write] (tls.js:330:25)
    at doWrite (_stream_writable.js:211:10)
    at writeOrBuffer (_stream_writable.js:201:5)
    at EncryptedStream.Writable.write (_stream_writable.js:172:11)
    at write (_stream_readable.js:547:24)


maybe too much DNS? try using this as the URL:

socketio-lb-1655485032.us-west-1.elb.amazonaws.com

I still cant hook up thru telnet.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 23, 2013, 08:51:45 PM
Using puTTY I tried:

=====================================
socketio-beta.mtgox.com, port 80, RAW

Connection is established, I then type:

GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: socketio-beta.mtgox.com
Origin: null


No reply, Server remains silent?
=====================================

becaue its a socket.io server and not a websocket server:

telnet socketio-beta.mtgox.com 80
Trying 184.169.152.65...
Connected to socketio-lb-1655485032.us-west-1.elb.amazonaws.com.
Escape character is '^]'.
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 23 Mar 2013 20:35:15 GMT
Connection: keep-alive
Transfer-Encoding: chunked

53
W_4my4iKDRSNXtJN89vh:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

then you parse the second line of the response take the ID W_4my4iKDRSNXtJN89vh to form a new URL that look like /socket.io/1/websocket/W_4my4iKDRSNXtJN89vh and then you use the same still open socket (its still open because you sent keep-alive header) and initiate the websocket request:

GET /socket.io/1/websocket/W_4my4iKDRSNXtJN89vh HTTP/1.1
Upgrade: WebSocket
and so on...

If it does not work then make sure that you are sending websocket version 13 headers (see the websocket RFC I posted a few days ago) and not the old version you showed above (that were not version 13 headers), this also means you need to provide Sec-Websocket-Key header and all that. See how goxtool does it (its using https://pypi.python.org/pypi/websocket-client/ with an overridden connect() method to do the initial http for the ID before the actual websocket starts).

You either need to find a complete socket.io implementation for your programming language or a websocket V13 client that you can tweak the same way like I did it with the python client.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on March 23, 2013, 08:53:09 PM
Im getting this error using the beta socketio server with the javascript library:

Yes, that's because this server uses cert issued for socketio.mtgox.com, which obviously does not match the different hostname.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 23, 2013, 09:04:10 PM
So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 23, 2013, 09:18:26 PM
here is a complete example as far into the handshake as it is possible with telnet (because after that it switches to binary WS protocol):

--($)-- telnet socketio-beta.mtgox.com 80
Trying 184.169.152.65...
Connected to socketio-lb-1655485032.us-west-1.elb.amazonaws.com.
Escape character is '^]'.
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 23 Mar 2013 21:06:56 GMT
Connection: keep-alive
Transfer-Encoding: chunked

53
Qi-RuYva5rgLXtXuhsrE:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

GET /socket.io/1/websocket/Qi-RuYva5rgLXtXuhsrE HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

�1::�2::

the �1:: and the �2:: are the first two websocket frames it sends (websocket is a binary protocol), now it expects "1::/mtgox" and for every "2::" it sends you must reply "2::", of course you need to properly mask and frame what you send as it is described in the Websocket RFC. This is how socket.io over websocket (or websocket over socket.io?) looks like. Unfortunately from here on its impossible to proceed with telnet because of the binary frames.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 23, 2013, 09:20:36 PM
So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

Maybe not use SSL, use unencrypted on port 80


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 23, 2013, 09:23:07 PM
So this is a problem of certificates and not of socket.io-client version?

Or which version of socket.io-client should I use?

Maybe not use SSL, use unencrypted on port 80


Tried that already - the same result - errors


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 23, 2013, 11:05:03 PM
@prof7bit,

Thanks so much for your great reply!

I explored thru telnet first just to get my toes wet.

I am working ontop of WindowsMFC CAsyncSocket class and I got the websocket going OK (when server is up).

I'll set about trying to get CAsyncSocket to act as socket.io now and manage the binary frames as they arrive.

I'm not very experienced when it comes to streaming stuff, you just helped me bigtime! :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on March 25, 2013, 08:58:37 AM
My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 25, 2013, 09:34:23 AM
My bot received trades with timestamps from future (~ 5 seconds). But my system clock it is precisely synchronized with atomic time (~0.5 ms) through several NTP servers.
Also now all calls (order/lag or private/info) always rejected with message "Invalid call". Can I retrieve more info for this error or API have limited verbosity?

Did you change your algorithm that creates your nonce? If you did then you might need a fresh API key. Nonce problems were the only problems I have witnessed causing sudden mysterious "Invalid call" problems.

Also if you send a lot of commands immediately after each other they will be processed in parallel on the server and this can mess up the chronological order (basically a race condition on their server) that will cause sporadic "Invalid call" (because the nonce not strictly incrementing anymore if the sequence is messed up). The only way around it is to answer each "Invalid call" by just sending the same command (with a new nonce of course) again... until it succeeds.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on March 25, 2013, 10:25:12 AM
prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp. API key and secret are correct. Multiple attempts give same result "Invalid call". So bad that the server does not provide a detailed description of the error.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 25, 2013, 01:25:42 PM
I've got a little further initiating the beta socketio stream, below is my TRACE.

############SOCKET OPENED BY CLIENT

############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked

############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80

############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

############SOCKET CLOSED BY HOST

The server closes the socket when I dont give it correct Sec-WebSocket-Key.

So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.

Should i do it like mentioned in websocket wiki?:

Quote
The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 25, 2013, 06:06:33 PM
I've got a little further initiating the beta socketio stream, below is my TRACE.

############SOCKET OPENED BY CLIENT

############TX:
GET /socket.io/1 HTTP/1.1
Host: socketio-beta.mtgox.com:80
Connection: keep-alive

############RX:
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Mon, 25 Mar 2013 03:48:51 GMT
Connection: keep-alive
Transfer-Encoding: chunked

############TX:
GET /socket.io/1/websocket/ HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80

############RX:
53
GEyZU7Bw-ivuvI6CZon3:60:60:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling
0

############TX:
GET /socket.io/1/websocket/GEyZU7Bw-ivuvI6CZon3 HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio-beta.mtgox.com:80
Origin: socketio-beta.mtgox.com:80
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13

############SOCKET CLOSED BY HOST

The server closes the socket when I dont give it correct Sec-WebSocket-Key.

So I'm guessing in my last TX above I need to derive Sec-WebSocket-Key somehow.

Should i do it like mentioned in websocket wiki?:

Quote
The client sends a Sec-WebSocket-Key which is base64 encoded. To form a response, the magic string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 is appended to this (undecoded) key. The resulting string is then hashed with SHA-1, then base64 encoded. Finally, the resulting reply occurs in the header Sec-WebSocket-Accept.

There is something wrong. you are sending 3 GET requests instead of 2, the first one looks ok, the second one is wrong.

The sec-websocket-key is a random number (i believe 16 bytes without looking into the RFC) base64 encoded. the server does some simple transformations with it and returns it as sec-websocket-accept. You can basically throw this away, you could use it to detect that there is no caching proxy in between because for every different sec-websocket-key you will get a different sec-websocket-accept. So you could check if the server response really belongs to your request and is not something cached but you could just as well simply ignore it because no sane webcache would ever attempt to cache a websocket connection anyways.

Once you are past this initial handshake the real fun (or no fun at all) will begin: You will have to implement the framing and masking as defined in RFC 6455. This initial handshake will look like childs play compared to what is awaiting you in this RFC.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 25, 2013, 06:18:31 PM
prof7bit
Thanks! I have not changed anything in the message signing function, my nonce is MD5 of timestamp.

There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)

Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 25, 2013, 06:23:46 PM
Hey, guys, I am sorry but can you advice?

is it possible or will it be possible to use just plain nodjs socket.io client to connect to socketio-beta?

I use socket.io-client on server to fetch mtgox socketio stream. I do not have any http requests - client is connecting on its own.

But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?

thanks


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 25, 2013, 07:02:22 PM
But what is the difference& between sockeio and socketio-beta?
Why do I need to use GET request and all those things you describe?

There seems a bug in the server, MT has promised to fix it. Therefore its only "beta" and he hasn't replaced the old server yet. I have no idea when this will happen. Some client implementations can handle this bug (the bug is that both requests MUST come over the same socket), some other clients cant (until he fixes this bug). The ones who happen to have a working client (like me) are lucky because this server seems much more reliable and has had 0% downtime so far.

The GET requests and basically the entire last few pages of this thread with all their technical details about socket.io and GET requests and frames are only of interest if one must write his own socket.io/websocket client from scratch because there exist no socket.io libraries for many (most) programming languages except JavaScript (and this is because socket.io is not an internet protocol and not a standard of any kind, socket.io is just a private hobby project from some guys who hacked some code together as a non-solution for a non-problem and uploaded it to github and unfortunately MagicalTux somehow accidentally found this hobby-protocol and thought it was a good idea to use it for MtGox-API and now we all have to suffer from the consequences).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on March 26, 2013, 07:15:14 AM
There we have the reason why it is not working. It *MUST* be a numeric value and it MUST be strictly incrementing. You MUST NEVER send the same or lower number ever again or it will answer with "Invalid call". It will remember the value of the last nonce you sent indefinitely on a per-key basis (to reset this to 0 again you need a new key)

Last week I found by accident that it accepted non-numeric values without complaint, I told MagicalTux about this security bug and I assume he has fixed it now.

Yes, now it works again. Big Thanks!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 26, 2013, 07:55:36 AM
Yes, now it works again. Big Thanks!

It works with md5? Then the bug is not fixed! It must NOT work with md5! Please show a json dump of one of your api calls, let me see how it looks like.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 26, 2013, 07:57:34 AM
We can stop worrying about socketio-beta, MT has taken it live. socketio is now the new server and socketio-beta is obsolete and the old server is now socketio-old.mtgox.com


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 26, 2013, 11:17:13 AM
I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on March 26, 2013, 11:28:57 AM
It works with md5?
No, I just use large numbers calculated from system time.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 26, 2013, 11:40:34 AM
I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 26, 2013, 12:28:59 PM
I noticed that my server socket.io-client is connected for more 10 hours already without any reconnects whille my local dev server reconnects often but reconnects stably.

Can it be gox fixed the new socket.io server you were taling about to be able to connect without any patches?

Yes, this socketio-beta from yesterday is now reachable via the main hostname socketio (apperently he fixed the bug so it works with all clients) and the hostname socketio-beta does no longer exist. And the old server is still reachable with socketio-old (and its lightening fast now because it has only few users now) but I don't know when he will switch it off.

thank you for clarification


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 02:32:10 AM
For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.

In this table the least significant bit is on the right, makes it easier to mask and shift.

I cannot say this has no error, I have not tested, so verify by eye against original table first:

EDIT:BELOW TABLE IS WRONG! SCROLL DOWN PAGE FOR FIXED VERSION


   /*-------------------------------------------------------------*\
   | LSB ON RIGHT IN THIS TABLE                                    |
   |  3                   2                   1                    |
   |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
   +-------------------------------+-------------+-+-------+-+-+-+-+
   |    Extended payload length    | Payload len |M| opcode|R|R|R|F|
   |             (16/64)           |     (7)     |A|  (4)  |S|S|S|I|
   |   (if payload len==126/127)   |             |S|       |V|V|V|N|
   |                               |             |K|       |3|2|1| |
   +-------------------------------+-------------+-+-------+-+-+-+-+
   |     Extended payload length continued, if payload len == 127  |
   +-------------------------------+-------------------------------+
   |Masking-key, if MASK set to 1  |                               |
   +-------------------------------+-------------------------------+
   |          Payload Data         | Masking-key (continued)       |
   +-------------------------------+-------------------------------+
   :                     Payload Data continued ...                :
   +---------------------------------------------------------------+
   |                     Payload Data continued ...                |
   +--------------------------------------------------------------*/

      ORIGINAL TABLE: LSB ON LEFT.
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 27, 2013, 10:24:10 AM
For anyone else getting down and dirty at bit level, heres a redrawn table for RFC 6455 header.

In this table the least significant bit is on the right, makes it easier to mask and shift.

I cannot say this has no error, I have not tested, so verify by eye against original table first:

Your table is incorrect. The original version in RFC 6455 is drawn with the most significant on the left. It's a bizarre numbering scheme used in most RFCs called MSB-0 (http://en.wikipedia.org/wiki/Bit_numbering#MSB_0_bit_numbering).

It's easier to read the RFC tables a byte at a time. The protocols assume the granularity of data transfer is 8-bit bytes so they don't really have the concept of a multi-bit word. When they do need multi-bit word they usually encode it as multiple 8-bit bytes using big-endian ordering.

For example, the lower 4 bits of the first byte are the opcode and the most significant bit is the FIN flag. Similarly, for the second byte the payload length is the lower 7 bits and the mask flag is the most significant bit.

Assuming little-endian with least significant bit (0) on the right, the first 2 bytes should look like this:
Code:
 1
 5    14 - 8     7 6 5 4  3 - 0
+-+-------------+-+-+-+-+--------+
|M| Payload len |F|R|R|R| opcode |
|A|      (7)    |I|S|S|S|  (4)   |
|S|             |N|V|V|V|        |
|K|             | |1|2|3|        |
+-+-------------+-+-+-+-+--------+

 |-------------| |-------------|
   second byte     first byte


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 12:57:06 PM
Oh my... thanks muchly writhe!  Slowly but surely, I'll get this stream established.

Here's an ammended table for little endian:

   /*-------------------------------------------------------------*\
   |         LITTLE ENDIAN FORMAT WHERE LSB == 0x00000001          |
   |  3                   2                   1                   0|
   |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0|
   +-------------------------------+-+-------------+-+-+-+-+-------+
   |    Extended payload length    |M| Payload len |F|R|R|R| opcode|
   |             (16/64)           |A|     (7)     |I|S|S|S|  (4)  |
   |   (if payload len==126/127)   |S|             |N|V|V|V|       |
   |                               |K|             | |1|2|3|       |
   +-------------------------------+-+-------------+-+-+-+-+-------+
   |     Extended payload length continued, if payload len == 127  |
   +-------------------------------+-------------------------------+
   |Masking-key, if MASK set to 1  |                               |
   +-------------------------------+-------------------------------+
   |          Payload Data         | Masking-key (continued)       |
   +-------------------------------+-------------------------------+
   :                     Payload Data continued ...                :
   +---------------------------------------------------------------+
   |                     Payload Data continued ...                |
   \*-------------------------------------------------------------*/


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 02:07:20 PM
I got the socket.io ticker stream coming in... weeeeee! :D

Thanks to prof7bit, few satoshis coming your way!!!!

I used a hardcoded packet for first send after socket upgrade ala prof7bit's python debug output.

Code:
	unsigned char pBuffer[15];
unsigned char* pBuild = pBuffer;
*pBuild++ = 0x81;
*pBuild++ = 0x89;
*pBuild++ = 0x9f;
*pBuild++ = 0x99;
*pBuild++ = 0x10;
*pBuild++ = 0xa1;
*pBuild++ = 0xae;
*pBuild++ = 0xa3;
*pBuild++ =  '*';
*pBuild++ = 0x8e;
*pBuild++ = 0xf2;
*pBuild++ = 0xed;
*pBuild++ =  'w';
*pBuild++ = 0xce;
*pBuild++ = 0xe7;
Transmit_Raw(pBuffer, 15, false, true);

I cant reply to its echo requests yet because my frame/mask func has gremlins. But the data is coming in well until echo request.

Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 27, 2013, 08:37:05 PM
Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

There are some examples (http://tools.ietf.org/html/rfc6455#section-5.7) in the RFC that might be useful to check against.

Also, using all zeros for the masking key is valid (you still need to set the masking bit though). This should make it easier to debug as the payload won't be modified by the XOR operation.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 08:52:24 PM
Now I can set about debugging my framing/masking function as i have a working reference frame to verify against.

There are some examples (http://tools.ietf.org/html/rfc6455#section-5.7) in the RFC that might be useful to check against.

Also, using all zeros for the masking key is valid (you still need to set the masking bit though). This should make it easier to debug as the payload won't be modified by the XOR operation.

Thanks, i missed that, (note to self: READ specs before coding!).

So now i have realised that the payload length bytes are dynamic and are only used if payload size needs them. ahhh so thats why the table has dotted lines for some cell walls. doh!

edit: and thx for the zero mask tip, sure will make things simpler to get going!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 09:51:48 PM
here is a C mask/frame function that is tested and working, again thanks guys for the help:

edit: See a few posts on for the func....


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 27, 2013, 09:54:49 PM
sorry, can you please explain what you are actually developing?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 27, 2013, 10:01:26 PM
here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:
  • payload length is < 126 bytes,
  • payload length is < 65536 bytes,
  • payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 10:34:48 PM
here is a C mask/frame function that is tested and working, again thanks guys for the help:

The way that you're encoding the payload length doesn't look right to me. There are basically 3 cases:
  • payload length is < 126 bytes,
  • payload length is < 65536 bytes,
  • payload length is upto pow(2,64)-1 bytes.

In the first case, the length can be encoded entirely within the 7 bits of the payload length field.

In the second case, the payload length field is set to 126 and the next 2 bytes are the payload length (using big-endian ordering).

In the third case, the payload length field is set to 127 and the next 8 bytes are the payload length (again, in big-endian).

Woops! I only tested with short message so only tested the 7bits version. Will fix that and post code below.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 27, 2013, 10:41:25 PM
sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 28, 2013, 12:09:03 AM
Here is the corrected rfc6455 framing/masking func in C.

Corrected, as @writhe said to do payload length correctly:

edit: Corrected AGAIN, thx @writhe!!

Code:
///////////////////////////////////////////////////////////////////////////////////////////////////
// CREATE A MASKED DATA FRAME FROM THE PASSED BYTES.
//
// NOTES:
// This implementation has max payload size of 0xFFFFFFFF == 4 GB
// v.pucFrameData is a pointer used to read the data from outside this func.
// v.uiFrameBytes is a variable used to read frame size from outside this func.
// REF: (http://tools.ietf.org/html/rfc6455#section-5.3)
///////////////////////////////////////////////////////////////////////////////////////////////////
void THR_MtGox::CreateFrameData(unsigned char* pucInput, unsigned int uiByteLen, bool bText)
{
const bool B_MASK_ENABLED = true;
const int  MAX_HEADER_BYTES = 14;

///////////////////////////////////////////////////////////////////////////////////////////////////
// ALLOC NEW MEMORY FOR DERIVED FRAME
if(v.pucFrameData)
delete v.pucFrameData;
v.pucFrameData = new unsigned char[MAX_HEADER_BYTES + v.uiFrameBytes];
unsigned char* pucOutput = v.pucFrameData;//temporary pointer, can increment to build the frame

///////////////////////////////////////////////////////////////////////////////////////////////////
// FIRST BYTE FLAGS ("FIN" BIT ALWAYS SET BECAUSE THIS FUNC SENDS A COMPLETE MESSAGE)
if(bText) //TEXTUAL PAYLOAD 10000001
*pucOutput++ = 0x81;
else //BINARY PAYLOAD  10000010
*pucOutput++ = 0x82;

///////////////////////////////////////////////////////////////////////////////////////////////////
// STORE ADDRESS OF BYTE THAT HOLDS MASK BIT
unsigned char* pucMaskByte = pucOutput;

///////////////////////////////////////////////////////////////////////////////////////////////////
// PAYLOAD BYTE SIZE:
// [1] payload length is < 126 bytes,
// the length can be encoded entirely within the 7 bits of the payload length field.
if(uiByteLen < 126)
{
*pucOutput++ = (unsigned char)uiByteLen;
}
// [2] payload length is < 65536 bytes,
// Payload length field is set to 126, next 2 bytes are payload length (big-endian).
else if(uiByteLen < 65536)
{
*pucOutput++ = 126;
*pucOutput++ = (unsigned char)(uiByteLen>>8);
*pucOutput++ = (unsigned char)(uiByteLen   );
}
// [3] payload length is upto pow(2,64)-1 bytes.
// payload length field is set to 127 and the next 8 bytes are the payload length (big-endian).
else
{
*pucOutput++ = 127;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// MASK ENABLE BIT
if(B_MASK_ENABLED)
*pucMaskByte |= 0x80;
else
*pucMaskByte &=~0x80;

///////////////////////////////////////////////////////////////////////////////////////////////////
// MASKING KEY
unsigned int uiMaskingKey = (unsigned int)rand();
if(B_MASK_ENABLED)
{
unsigned int uiMask = uiMaskingKey;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
uiMask >>= 8;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
uiMask >>= 8;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
uiMask >>= 8;
*pucOutput++ = (unsigned char)(uiMask & 0xFF);
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// ADD PAYLOAD AND XOR WITH MASK IF NEEDED
if(B_MASK_ENABLED)
{
for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
{
int iMaskOctet = uiByte%4;
unsigned char ucMaskOctet = (unsigned char)(uiMaskingKey >>(iMaskOctet*8) & 0xFF);
*pucOutput = *pucInput ^ ucMaskOctet;
pucOutput++;
pucInput++;
}
}
else
{
for(unsigned int uiByte = 0; uiByte < uiByteLen; uiByte++)
{
*pucOutput = *pucInput;
pucOutput++;
pucInput++;
}
}
///////////////////////////////////////////////////////////////////////////////////////////////////
// CALCULATE RESULTING FRAME LENGTH BYTES
v.uiFrameBytes = pucOutput - v.pucFrameData;
}


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: writhe on March 28, 2013, 12:12:08 AM
I think you mean:
Code:
		*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
instead of:
Code:
		*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;

 ;D


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 28, 2013, 12:18:55 AM
I think you mean:
Code:
		*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
instead of:
Code:
		*pucOutput++ = (unsigned char)(uiByteLen>>24);
*pucOutput++ = (unsigned char)(uiByteLen>>16);
*pucOutput++ = (unsigned char)(uiByteLen>> 8);
*pucOutput++ = (unsigned char)(uiByteLen    );
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;
*pucOutput++ = (unsigned char)0x00;

 ;D

you know... i think you're right!  :-[ :D

corrected above codepaste (again ::) ).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on March 28, 2013, 04:32:46 AM
Here is the corrected rfc6455 framing/masking func in C.

C code! Things got serious in here...  ;D


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 28, 2013, 05:38:36 AM
sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

So you implement socket.io-client in C? And there was no any existing implementation already in the net?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 28, 2013, 12:57:39 PM
sorry, can you please explain what you are actually developing?

A realtime charting/trading platform with neural networks for windoze: http://www.cortex7.net/forum/showthread.php?tid=87

So you implement socket.io-client in C? And there was no any existing implementation already in the net?

There seems no existing implementation in C or C++. Closest is objective C. But I chose only to reference only RFC6455 for frame/mask.

I referenced prof7bits python code for the handshake.

I already had socket working in my codebase for mtgox websocket, but as you know its not a reliable data stream.

I just had to add a frame/mask func per RFC6455, this was the only really tricky bit, and I was happy to receive help here with that.

I was using an asynchronous socket with message callbacks, things became alot simpler when I switched to blocking socket. I over-ride base class so it also has timeout function, then I can reset connection if line goes quiet for 1 minute.

I've also coded a JSON parser, not to full spec, it just finds a val string by braching into braces according to passed key path.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 28, 2013, 01:00:34 PM
great work I think!

any plans to publish your socket.io-client implementation on github for the benefit the public?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 28, 2013, 04:25:43 PM
great work I think!

any palns to publish your socket.io-client implementation on github for the benefit the public?

thanks. with any vanilla socket you just need to use the posted framing func for sending data after upgrade. that's the trickiest part.

that func could even be used to make an arduino into a socket.io client with good performance.

i have a standalone visual studio project (threaded, well encapsulated, no lib dependencies, just MFC) that has just the socket.io logging the json stream to a window if anyones interested.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 28, 2013, 04:44:32 PM
Here is the corrected rfc6455 framing/masking func in C.

C code! Things got serious in here...  ;D

lol, now someone do an optimised X86 assemler version.

Great site you made by the way Clark! :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 29, 2013, 01:57:22 AM
I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

both "tid" and "date" entries have this wrong time?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 29, 2013, 05:12:16 PM
I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

How is the bitcoin price next week? Is it above 100 already?
I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh ;)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on March 29, 2013, 07:41:27 PM
I seem to be receiving trade tickers with following unix timestamp:

1365219272

which is a future date:

2013-APRIL-06 04:34:32 UTC

How is the bitcoin price next week? Is it above 100 already?
I knew that C++ code can be quite fast but this seems to be a remarkable breakthrogh ;)

It is just hilarious :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 30, 2013, 12:38:36 PM
LOL, i've routed into a temporal rift, i will keep this a secret, future prices are good!!! ;D

But seriously guys the JSON timestamps are wrong, or i'm just not understanding what that time is for???

Nothing wrong at my RX end as server sends non masked frames, just plain text blocks between delimiters.

TRADE and DEPTH json stream:


{"channel":"dbf1dee9-f2e-a08-8cb7-78919a71b21","op":"private","origin":"broadcast","private":
"trade","trade":{"type":"trade","date":1366631,"amount":2,"price":91.00021,"tid":"136663155159",
"amount_int":"200000000","price_int":"9100021","item":"BTC","price_currency":"USD","trade_type":"bid",
"primary":"Y","properties":"market"}}~4::/mtgox:{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe",
"op":"private","origin":"broadcast","private":"depth",
"depth":{"price":"91.00021","type":1,"type_str":"ask","volume":"0","price_int":"9100021","volume_int":"0",
"item":"BTC","currency":"USD","now":"136663158281","total_volume_int":"3016823508"}}

{"channel":"2e67e0d-1cad-cc0-9e7a-f8523ef60fe","op":"private","origin":"broadcast","private":
"depth","depth":{"price":"87.5001","type":2,"type_str":"bid","volume":"0","price_int":"8750010",
"volume_int":"0","item":"BTC","currency":"USD","now":"136663161999","total_volume_int":"136753855"}}



trade has "now":"136663158281"  === Mon, 22 Apr 2013 11:53:02 GMT

depth has "now":"136663161999" === Mon, 22 Apr 2013 11:53:39 GMT

if you dont trust my unixtime to datetime conversion then plug them into an online converter (using the 10 leftmost bolded characters):
http://www.onlineconversion.com/unix_time.htm

so what gives, does nobody else use this data, do I have to discard timestamps and stamp them myself using local clock? that's not so good!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 30, 2013, 05:22:04 PM
This is strange.

My timestamps as received here are correct. Below are some messages I have received here from socketio.mtgox.com a few minutes not long ago, the timestamps look normal.

Also the trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart. I could not see any mismatch between the timestamps of "money/trades" and new incoming trade messages.

The first thing that catches the eye is that in your trade message the date field is only 7 digits long while mine is a complete timestamp (in seconds, 10 digits) and also your microseconds timestamps seem too short (12 digits instead of 16), how can this happen? Something eating characters and always the *same* sequence of characters? Btw I cannot see the character "4" appearing anywhere in your dump (except for the 4::/mtgox (!)) and also some of the GUID strings are too short!

Below are the mesages as received by me (and pretty-printed/formatted by json.dumps())

trade
Code:
{
     "origin": "broadcast",
     "trade": {
          "price_currency": "USD",
          "trade_type": "bid",
          "price_int": "9120000",
          "item": "BTC",
          "price": 91.2,
          "primary": "Y",
          "tid": "1364662801201914",
          "amount": 1.41528958,
          "amount_int": "141528958",
          "date": 1364662801,
          "type": "trade",
          "properties": "limit"
     },
     "private": "trade",
     "channel": "dbf1dee9-4f2e-4a08-8cb7-748919a71b21",
     "op": "private"
}

Depth
Code:
{
     "origin": "broadcast",
     "depth": {
          "volume_int": "0",
          "price_int": "9120000",
          "currency": "USD",
          "price": "91.2",
          "volume": "0",
          "item": "BTC",
          "type_str": "ask",
          "total_volume_int": "9828724331",
          "now": "1364662801229415",
          "type": 1
     },
     "private": "depth",
     "channel": "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe",
     "op": "private"
}



Ticker
Code:
{
     "origin": "broadcast",
     "ticker": {
          "sell": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "last_orig": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "now": "1364662937327085",
          "buy": {
               "value_int": "9119999",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.19999",
               "value": "91.19999"
          },
          "last": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "vol": {
               "value_int": "5095833923444",
               "currency": "BTC",
               "display_short": "50,958.34\u00a0BTC",
               "display": "50,958.33923444\u00a0BTC",
               "value": "50958.33923444"
          },
          "last_local": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "last_all": {
               "value_int": "9120000",
               "currency": "USD",
               "display_short": "$91.20",
               "display": "$91.20000",
               "value": "91.20000"
          },
          "vwap": {
               "value_int": "9064567",
               "currency": "USD",
               "display_short": "$90.65",
               "display": "$90.64567",
               "value": "90.64567"
          },
          "high": {
               "value_int": "9498000",
               "currency": "USD",
               "display_short": "$94.98",
               "display": "$94.98000",
               "value": "94.98000"
          },
          "low": {
               "value_int": "8550000",
               "currency": "USD",
               "display_short": "$85.50",
               "display": "$85.50000",
               "value": "85.50000"
          },
          "avg": {
               "value_int": "9037305",
               "currency": "USD",
               "display_short": "$90.37",
               "display": "$90.37305",
               "value": "90.37305"
          }
     },
     "private": "ticker",
     "channel": "d5f06780-30a8-4a48-a2f8-7ed181b4a13f",
     "op": "private"
}

Lag
Code:
{
     "origin": "broadcast",
     "lag": {
          "stamp": "1364663115576119",
          "qid": "74655212-1bc0-4b69-95da-79e40f4f771f",
          "age": 329545
     },
     "private": "lag",
     "channel": "85174711-be64-4de1-b783-0628995d7914",
     "op": "private"
}


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 30, 2013, 06:05:25 PM
thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

Then I will subscribe to the new streams, so i will try subscribing to new streams anyway and see how it goes, i guess i just send a masked/framed subscription request as per the mtgox stream api wiki.

So perhaps i will just have to ignore the junk stream, it will still have to flow thru my parser though, i dislike wasted clock cycles, plus it loads the server.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 30, 2013, 06:12:00 PM
thanks for checking it out!

your trade UUID (top) vs the one I get (bottom)

dbf1dee9-4f2e-4a08-8cb7-748919a71b21
dbf1dee9-f2e-a08-8cb7-78919a71b21


the stream I am parsing is the default stream after first connecting, this is probably the problem. I would like to unsubscribe from them but have not been able to yet. It seems odd that the default stream contains junk, why bother including it?

No, I am using them too.

But all your "4" are missing! Are you trying to parse the "4::/mtgox:" by any chance and have an error there in your parser? In your previous post the "4::/mtgox:" is appearing between the json objects and is the only thing that has a "4" in it, all other "4" seem to have gone missing!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 30, 2013, 09:06:03 PM
you are correct prof7bit, my parsing state machine is buggy causing 4s to get dropped :-[.

I'm doing it with inc pointer on the raw incoming data buffer for speed and screwed up. I will rework it now. thanks man!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on March 31, 2013, 02:52:09 AM
...trade history from https://data.mtgox.com/api/2/BTCUSD/money/trades gives me correct timestamps from today and I'm using them to plot an M15 candle chart and then use the date from the streaming trade messages to continue this chart (update the last candle until time_trade > time_candle_open + 15 minutes) and apend new candles to the chart....

cool, similar to what i'm doing.

It would be nice to have a mtgox call similar to bitcoincharts, so one could specify to get all trades since X unixtime.

or even better binned trades in choice of bin size, M15, M30 etc. this would really save some bandwidth for mtgox!

P.S.
are you plotting volume also? if so do you find the live numbers sum closely to the history?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on March 31, 2013, 09:26:09 AM
are you plotting volume also? if so do you find the live numbers sum closely to the history?

Not yet. I'm only plotting a quick & dirty price chart of the last 24 hours so I can quickly see from 5 meters distance if it has gone up or down during the night. I was too lazy until now to also write code to plot volume bars (I also don't have enough room in the terminal window to plot too many details in ascii-art with ncurses)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on April 08, 2013, 01:02:58 AM
I think I remember reading in this thread that MagicalTux was going to change the order API so that people cant place more asks than BTC they have on account.

Watching the live depth stream I dont think that this is implemented yet.

On the GBP stream below we can see a bot went crazy and started placing many asks of the 2 same values resulting in 2 long horizontal lines.

This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.

http://www.cortex7.net/forum/attachment.php?aid=98


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on April 08, 2013, 10:30:30 AM
This was clearly exceeding the holdings of its owner. The stream window below is 5 minutes.

No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on April 08, 2013, 02:47:23 PM
No, it cannot exceed the holdings of the owner, otherwise it would not show up in the order book. It would stay in "invalid" state until funds become available and only be visible to the owner himself but not for anyone else.
Wow well there are some large players then!

Well I now have stable multicurrency stream.io coming into my app. I still notice the odd hang during heavy times (maybe the ddos is not helping?).

I look for consecutive zero receive bytes or receive timout, if I see this I close the socket, destroy it, recreate and reconnect after a short delay.

One note to others (has been mentioned before), the channel UUIDs are not worth much from the client end. if you want multicurrency stream then you should filter on the human readable channel type and currency type. All UUIDs are not published in MtGox API docs, I suppose if you want to use them you would have to discover them yourself in the RX log.

Now how long this will stay good until I have to revisit this part of my code I dont know, hopefully for a long long time. Thankyou to all who helped me stumble to this point  ;D


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 08, 2013, 03:24:51 PM
This is the list of channel ids I descovered till the moment

Quote
{ 'USD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'ticker' : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
                              'depth' : '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe'
                           },
                           'EUR' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '057bdc6b-9f9c-44e4-bc1a-363e4443ce87',
                              'ticker' : '0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15'
                           },
                           'GBP' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c',
                              'ticker': '7b842b7d-d1f9-46fa-a49c-c12f1ad5a533'
                           },
                           'CNY' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '0d1ecad8-e20f-459e-8bed-0bdcf927820f',
                              'ticker' : 'c251ec35-56f9-40ab-a4f6-13325c349de4'
                           },
                           'RUB' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'bd04f720-3c70-4dce-ae71-2422ab862c65',
                              'ticker' : 'd6412ca0-b686-464c-891a-d1ba3943f3c6'
                           },
                           'CAD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : '10720792-084d-45ba-92e3-cf44d9477775',
                              'ticker' : '5b234cc3-a7c1-47ce-854f-27aee4cdbda5'
                           },
                           'PLN' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01',
                              'ticker' : 'e4ff055a-f8bf-407e-af76-676cad319a21'
                           },
                           'AUD' : {
                              'trades' : 'dbf1dee9-4f2e-4a08-8cb7-748919a71b21',
                              'depth' : 'eb6aaa11-99d0-4f64-9e8c-1140872a423d',
                              'ticker' : '296ee352-dd5d-46f3-9bea-5e39dede2005'
                           }


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 08, 2013, 04:40:00 PM
This is the list of channel ids I descovered till the moment

Well since we're playing that game  :)

Code:
            {
                'AUD': {'eb6aaa11-99d0-4f64-9e8c-1140872a423d': 'ticker',
                        '296ee352-dd5d-46f3-9bea-5e39dede2005': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CAD': {'10720792-084d-45ba-92e3-cf44d9477775': 'ticker',
                        '5b234cc3-a7c1-47ce-854f-27aee4cdbda5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CHF': {'2644c164-3db7-4475-8b45-c7042efe3413': 'ticker',
                        '113fec5f-294d-4929-86eb-8ca4c3fd1bed': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'CNY': {'c251ec35-56f9-40ab-a4f6-13325c349de4': 'ticker',
                        '0d1ecad8-e20f-459e-8bed-0bdcf927820f': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'DKK': {'e5ce0604-574a-4059-9493-80af46c776b3': 'ticker',
                        '9219abb0-b50c-4007-b4d2-51d1711ab19c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'EUR': {'0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15': 'ticker',
                        '057bdc6b-9f9c-44e4-bc1a-363e4443ce87': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'GBP': {'7b842b7d-d1f9-46fa-a49c-c12f1ad5a533': 'ticker',
                        '60c3af1b-5d40-4d0e-b9fc-ccab433d2e9c': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'HKD': {'d3ae78dd-01dd-4074-88a7-b8aa03cd28dd': 'ticker',
                        '049f65dc-3af3-4ffd-85a5-aac102b2a579': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'JPY': {'a39ae532-6a3c-4835-af8c-dda54cb4874e': 'ticker',
                        '94483e07-d797-4dd4-bc72-dc98f1fd39e3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NOK': {'7532e866-3a03-4514-a4b1-6f86e3a8dc11': 'ticker',
                        '66da7fb4-6b0c-4a10-9cb7-e2944e046eb5': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'NZD': {'5ddd27ca-2466-4d1a-8961-615dedb68bf1': 'ticker',
                        'cedf8730-bce6-4278-b6fe-9bee42930e95': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'PLN': {'b4a02cb3-2e2d-4a88-aeea-3c66cb604d01': 'ticker',
                        'e4ff055a-f8bf-407e-af76-676cad319a21': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'RUB': {'bd04f720-3c70-4dce-ae71-2422ab862c65': 'ticker',
                        'd6412ca0-b686-464c-891a-d1ba3943f3c6': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SEK': {'6caf1244-655b-460f-beaf-5c56d1f4bea7': 'ticker',
                        '8f1fefaa-7c55-4420-ada0-4de15c1c38f3': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'SGD': {'2cb73ed1-07f4-45e0-8918-bcbfda658912': 'ticker',
                        '41e5c243-3d44-4fad-b690-f39e1dbb86a8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'THB': {'d58e3b69-9560-4b9e-8c58-b5c0f3fda5e1': 'ticker',
                        '67879668-532f-41f9-8eb0-55e7593a5ab8': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'},
                'USD': {'d5f06780-30a8-4a48-a2f8-7ed181b4a13f': 'ticker',
                        '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe': 'depth',
                        'dbf1dee9-4f2e-4a08-8cb7-748919a71b21': 'trades'}
            }


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 08, 2013, 05:22:04 PM
Well Done !!! Clark :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on April 08, 2013, 06:00:58 PM
Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 08, 2013, 06:02:42 PM
Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

wow! cool :)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on April 08, 2013, 06:48:18 PM
Pro tip: http://data.mtgox.com/api/1/generic/currency?currency=USD

wow! cool :)

+1! very useful!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: puax on April 13, 2013, 11:02:09 AM
Is it me or the websocket has been down for days? Even my most basic implementation is silent.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on April 13, 2013, 11:40:48 AM
Is it me or the websocket has been down for days? Even my most basic implementation is silent.

My socket.io implementation gets as far as upgrade and then I just receive zero-byte returns, I had previously coded zero-byte return to trigger connection reset, I took that trigger out and the receive function just finishes instantly with zero-bytes received.

It's been like this since MtGox went down, and has not recovered even after MtGox opened for trading via the user-account web interface again.

So I'm not sure what the problem is. Maybe we just wait? :(


13:34:43
=============================================================
THR_MtGox::SocketCreate: CONNECTED TO: socketio.mtgox.com:80
=============================================================

13:34:43 TRANSMIT>>>>>>>>
GET /socket.io/1/?Currency=EUR HTTP/1.1
Host: socketio.mtgox.com/mtgox?Currency=EUR
Connection: keep-alive


13:34:44 RECEIVE>>>>>>>>>
HTTP/1.1 200 OK
Content-Type: text/plain
Date: Sat, 13 Apr 2013 11:34:43 GMT
Connection: keep-alive
Transfer-Encoding: chunked

30
b-LJ3EKgBF4kkuvdLP1d:60:60:websocket,flashsocket
0


13:34:44 TRANSMIT>>>>>>>>
GET /socket.io/1/websocket/b-LJ3EKgBF4kkuvdLP1d?Currency=EUR HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: socketio.mtgox.com/mtgox?Currency=EUR
Origin: socketio.mtgox.com/mtgox?Currency=EUR
Sec-WebSocket-Key: UWaphFPiSqq3f2gOmaD5Sg==
Sec-WebSocket-Version: 13


13:34:45 RECEIVE>>>>>>>>>
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: Bzz8qKJPEMNNOgC4hOZd3iZNb5o=

1::
13:34:45 TRANSMIT>>>>>>>>
TX_STR_FRAMED -> "1::/mtgox"

...ZERO-BYTE RX FROM NOW ON...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 13, 2013, 11:44:38 AM
Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on April 13, 2013, 11:53:16 AM
Since mtgox trading halt socket.io api does not work, or works sparadically.

I hade to upgrade our client to be able to work in hybrid mode basically request full orderbook via http without socket.io connect.

and additional timeout logic not to get blocked by mtgox blocker.

So you try socket.io and if it fails you fall back to http get with extended connection timeout. Kudos for working around it!

What kind of delay do you have?... Approx how old is the latest book entry that you receive? Do you get depth and trades?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 13, 2013, 12:10:17 PM
Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on April 13, 2013, 02:41:37 PM
Actually when socket.io works I start with http request full orderbook then when it is initalized socket sends update messages.

when socket.io reconnects the whole initializaton starts again with requesting full order book via http.

the problem is that you if you request full orderbook more then 5 times per hour as stated in mtgox stream api documentation.

So I had to add 5 times limitating logic.

and the whole system worked well until recently.

so now every 15 minutes I trigger connection check and if it is not connected request system to reinitialize orderbook.

this additional logic is implemented as upper layer over original client system.

currently I listen to depth and ticker for  quick fixes of broken orderbook without unnecessary reinitialization.

for trades we use bitcoincharts stream it contains not only mtgox trades.

We plan to advance the system to be able to get trades from mtgox channel too but we want to integrate it with bitcoincharts stream for robustness.


Thanks for sharing your insight, sounds like your going the extra mile to mantain as much order as possible on such a shaky foundation.

I'm not sure I have the drive to start wrtiting/debugging such a workaround, I would end up getting kicked off for >5 tries an hour, I'd reset my gateway and try again and probably end up being block banned.

At the moment I have no users reliant on my software so do not have the stressful job of trying to get a fix/workaround out. I will sit tight for a few days and work on some GUI stuff.

Perhaps when MtGox stream re-establishes properly it will run like a dream, let us hope so. ( until it breaks again::) ).

Best of luck with your commendable efforts.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: pescador on April 22, 2013, 05:51:04 PM
I'm experimenting with the MtGox websocket API. I'm using Python with the websocket module from https://pypi.python.org/pypi/websocket-client (https://pypi.python.org/pypi/websocket-client). See example code below

Code:
import threading
import websocket
import json

class mtgox( threading.Thread ):

    def run( self ):
        websocket.enableTrace( True )
        url = 'ws://websocket.mtgox.com/mtgox?Currency=USD'
        self.socket = websocket.WebSocketApp( url, on_open = self.on_open )
        self.socket.run_forever( )

    def subscribe( self, channel ):
        output = { 'op': 'mtgox.subscribe', 'type': channel }
        output = json.dumps( output )
        self.socket.send( output )

    def on_open( self, socket ):
        self.subscribe( 'depth' )
        self.subscribe( 'lag' )        
        self.subscribe( 'ticker' )
        self.subscribe( 'trades' )

if __name__ == '__main__':
    mtgox = mtgox( )
    mtgox.start( )

When I run this code I receive ticker and depth messages, but no lag or trade messages. Also I do not get any replies to my mtgox.subscribe commands, that the documentation seems to promise. However when I send a mtgox.subscribe command with a wrong type parameter, I get an error message "Unknown mtgox message type", so it seems my subscribe commands are received and accepted.

Could somebody please tell me why I'm not receiving trade and lag messages?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: pescador on April 27, 2013, 08:52:40 AM
To answer my own question above. It seems to be working now. Now I also get trade and lag messages. Apparently it was something at MtGox's side.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: pescador on April 28, 2013, 02:42:57 PM
I've downloaded the full bid / ask tables in USD with 'depth/full' and try to update them as depth and trade messages come in. From https://support.mtgox.com/entries/20800336-Multi-Currency-Trading (https://support.mtgox.com/entries/20800336-Multi-Currency-Trading) I understand that the tables in USD also include the bids / asks in other currencies, that are converted to USD at the daily rate of the European Central Bank minus / plus a fee of 2.5 %.

How do I update my tables when a primary trade comes in with a price_currency other than USD?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 28, 2013, 08:00:40 PM
You should not update orderbooks using trades channel. You need to listen to depth channel to update orderbooks. And also it would be convinient to fix ordebooks with ticker channel messages if neccesary (in case not all depth messages were arrived and ordebook state is broken)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: pescador on April 28, 2013, 09:02:40 PM
Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817 (https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817). I do that and it seems to be working fine. I only don't know what to do with trades in other currencies.

I'm already using the trick to use ticker messages to fix the tables when they appear to be broken, but that doesn't happen a lot.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on April 29, 2013, 05:49:31 AM
strange I currently use only depth and ticker messages. and did not have problems with orderbook state.

on my opinion there is not need to use trade messages because some trades can be from orders which are not in orderbook but executed before getting into orderbook.

but probably I missed something. It would be good to hear others


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on April 29, 2013, 09:15:44 AM
Are you talking about trades in other currencies or about all trades? Because I'm pretty sure that you have to update your depth tables when a trade message comes in as is explained here https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817 (https://bitcointalk.org/index.php?topic=5855.msg1636817#msg1636817)
I did some further experimenting and debugging and it seems this old posting of mine is wrong. There *will* be a depth message immediately following the trade message, so theoretically it would be enough to use only depth messages to update the orderbook.

But in goxtool/goxapi I still update the orderbook already in the trade message, so a hypothetical goxtool bot that would rely on the *new* orderbook state *after* the trade has been executed would already find the orderbook in the new updated state from inside the slot_trade() slot. Since I only ever use the total_vol of the depth messages and ignore the volume difference in my implementation the depth messages that immediately follow a trade message will simply result in no change to the orderbook at all.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: pescador on April 29, 2013, 12:04:02 PM
Thanks you both for your answers!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on May 04, 2013, 12:34:43 PM
How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on May 04, 2013, 03:11:35 PM
How can make a subscription to my bot trades? Requesting for every order private/order/result is slow bad solution. I need recieve trades with tid, oid, price and volume fields, as soon as possible after order matched partally or full.
There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on May 05, 2013, 11:42:34 AM
There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests :(


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on May 05, 2013, 06:06:11 PM
There will be a trade message on the private channel (see streaming API docs) but I'm not sure if it has the order ID.

It is anonymous data stream, without orders association. So bad, protocol provoke for many additional requests :(

You can also trigger your actions in the user_order message for order removal, this message will be fired when the order is completely filled or the last remaining part of it is filled (and therefore the order is removed). Problem here is: it will also fire when you cancel the order. So you need some additional code that knows what was going on before. If your bot is the only thing that ever cancels orders then will know the reason for the remove message but if you also cancel orders manually or through other bots running on the same account then you need somethng more sophisticated, then you need to keep track of the private trade messages too and if a trade message at the same price preceded the the order remove than it is a filled order, otherwise its a cancel or something like that.

The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alpet on May 06, 2013, 06:41:27 AM
The API clearly was not designed by someone who has ever written a trading bot before, its missing needed information in a lot of places and forces you to do cumbersome workarounds to detect certain events or keep track of the information you need, and at the same time all the json messages are bloated with totally useless redundant data that nobody ever uses.

I Agree, MtGox API need strongly to improve. Current realisation can be declared only as "not real-time". Extreme event lags! Some time required tens seconds, from send order to retrive all trades, what make API not usable for fast trailing stops and many others. Since my trading program cannot normal work with MtGox, I just switch to Icbit.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on May 27, 2013, 09:18:59 PM
I'm trying to subscribe to "lag" channel on socketIO:

Tried sending this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}


and without quotes around channel ID like this:

{
    "channel":"85174711-be64-4de1-b783-0628995d7914",
    "op":"subscribe"
}


But whatever I try there is no response, and certainly no subscription success response.

My Socket.IO TX framing is correct, if I break the framing the mtgox server drops connection.

I'm not able to subscribe/unsubscribe any channels dynamically.

To add/remove currencies I have to reset connection with new list of currency codes in the initial HTTP exchange.

I wonder what am I doing wrong?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on June 01, 2013, 04:09:10 PM
Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.

http://i40.tinypic.com/vas6.png

It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on June 03, 2013, 04:51:56 PM

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?


Well there was a $1M bid wall that got removed right before the selling started yesterday. I don't believe that large drop was an error.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on June 03, 2013, 05:35:38 PM
that is the question,

there was no any corresponding trading at that time.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on June 03, 2013, 09:54:50 PM
Hi, all!

We at bitcoin-analytics.com support local state of the orderbook and update it listening depth messages and fix obvious invalid state of orderbook ( when ask price is less than bid price) using ticker messages.

While fixing with ticker messages helps, it seems that it is not enough. Invalid orderbook state occures because some depth mesages does not get to the client from server. It is not so obvious if only you do not maintain history of the orderbook volume which we do on bitcoin-analytics.com.

What we noticed a repeting pattern like on the screenhot.

http://i40.tinypic.com/vas6.png

It looks like the volume increased steadily during several hours and then abruptly reduced. This happend exact at the moment when orderbook was just reinitialized.

The problem as we see it is the following: during the long period of time some depth messages does not come to client but do not invoke fixing procedure cause those lost messages does not break the validity of the orderbook state. But those lost messages did not remove the orders which they had to remove or change. During the period of time new orders were added to the orderbook. Thus the volume was increased while errors of lost messages were accumulated.

Did anyone noticed something similar. And does anyone has any ideas about this problem? How is it possible to detect those lost messages orders and fix them?

thanks

I've been having massive problems keeping order book synced in my client, there just seem to be lost messages. Never got it right.

Last resort: just pull full-depth 5 times per hour (magicaltux allows that frequency) and sync your orderbook


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on June 04, 2013, 08:43:46 AM
I agree, and wanted to add that this noticed affect is further suspicious becuase it is visible for all 8 currecly orderbooks at the same moment.

Just to remind that we support state of 8 currency orderbooks in the sime time. And this effect can be seen for all of them synchronosouly.

Needless to say that for such currencies like GPB or RUB here are significantly less trading activity which could be account for those simultanous volume decreases.

hm


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Eisenhower34 on June 04, 2013, 09:08:07 AM
Most of you are writing a trading bot, so Im sorry, but why do you all need full-depth? depth/fetch works fine and you can pull the data like every 30sec...


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on June 04, 2013, 09:09:23 AM
see bitcoin-analytics.com (http://bitcoin-analytics.com)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on June 04, 2013, 01:09:33 PM
I agree, and wanted to add that this noticed affect is further suspicious becuase it is visible for all 8 currecly orderbooks at the same moment.

MtGox has only one orderbook. Every order you add in one currency will also appear in every other currency (bids will appear 2.5% lower in other currencies, asks will appear 2.5% higher in other currencies).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on June 04, 2013, 02:19:14 PM
As long as gox provides separate http requests for full orderbooks for different currencies and allows socket.io subscribtion to multiple currenclies sending depth messages for each currecy in a separate channel this allows us to support orderbook state for each currency separately as if they are independent.

Observed lost depth messages effect is visible not only for one crrency orderbook but for all of them synchronously. We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood. The thing is that no matter is there only one major orderbook or not, the mechanism of depth messages itself is broken and not all depth messages reach the client.

This is suported by the obvious neccessity of fixing invalid orderbook state (when best bid price is greater than best ask price) against ticker messages, but also by observing the effect of accumulated old (not removed) orders which lead to increase in orderbook volume through time and significant descrease immediately after orderbooks state was reinitialized.



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on June 04, 2013, 07:15:25 PM
We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood.

I mentioned this only as an explanation of why the problem is observed synchronously in all currencies. MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

I am observing the same problem in goxtool and have no solution for it. Also fulldepth is cached a few minutes by the server, so simply reloading fulldepth every x minutes will also lead to some sort of inconsistency, especially near the top of the book where most of the action is and where its most important for traders and bots. I have no idea how to properly and cleanly fix this mess client side, probably nothing can be done about it at all.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on June 04, 2013, 07:34:28 PM

... MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).


this is interesting


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: molecular on June 04, 2013, 09:12:51 PM
We do not know what is the cause of the problem. That fact that all separate currency orderbooks are produced from the one majore orderbook is understood.

I mentioned this only as an explanation of why the problem is observed synchronously in all currencies. MT once mentioned in #mtgox that it might be due to occasional database rollbacks that can happen *after* the depth message has been sent out already (its sending out the depth mesages before the database commit happens and no inverse depth mesage is sent when it must be rolled back for whatever reason).

I am observing the same problem in goxtool and have no solution for it. Also fulldepth is cached a few minutes by the server, so simply reloading fulldepth every x minutes will also lead to some sort of inconsistency, especially near the top of the book where most of the action is and where its most important for traders and bots. I have no idea how to properly and cleanly fix this mess client side, probably nothing can be done about it at all.


tbh: the solution is to switch exchange. magicaltux will never fix this. I've given up.

no matter what the problem is: magicaltux should either provide us with a solution to keep a synced orderbook or outsource it if he can't do it himself.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Floomi on July 09, 2013, 09:05:35 PM
Folks,

I've been reading this thread with interest as I too am writing a client in Python (using websocket-client). I'm connecting over plain old websockets (not socketio) and am mostly working apart from a couple of issues:

1. Once the connection is established I subscribe to the USD trades channel:
Code:
TRADES = "dbf1dee9-4f2e-4a08-8cb7-748919a71b21"
def subscribe( channel ):
output = { 'op': 'mtgox.subscribe', 'channel': channel }
return json.dumps( output )

ws.send(subscribe(TRADES))
...but I get the remark spat back "Unknown channel requested or not a public channel". I seem to be subscribed to trades by default, so this isn't a huge problem - but is there anything I'm doing wrong here?

2. My websocket connection keeps dropping, for no reason that I can see. This thread's history has some talk about having to send heartbeat messages, but from what I understand that's only when you're connected to socket.io, not the plain old websockets I keep on using. I find all clients I run drop at the same time, regardless of when I start them, which makes me suspect the problem is on Gox's end. I can easily reconnect to it when it drops, but it's frustrating because it takes about 90s between the connection being dropped and me getting a new trade message after I've reconnected. EDIT: I discovered a settimeout(5) sitting quietly in the code, which was likely wrecking things... removing it and upgrading websocket-client to 0.11 seems to have done the trick.

Any advice would be much appreciated!
-Floomi


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on July 09, 2013, 09:13:34 PM
2. My websocket connection keeps dropping, for no reason that I can see. This thread's history has some talk about having to send heartbeat messages,

websocket should never disconnect at all, I have it running here 24/7 and it disconnects only maybe once or twice a day, some days it doesn't disconnect at all (and probably my few disconnects here are only due to my very poor wifi connection).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Floomi on July 09, 2013, 10:23:52 PM
Well, it seems that the diagnosis for at least one of these two issues is "I'm an idiot" :) I've edited my post above with an update.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on July 10, 2013, 07:04:53 AM
1. Once the connection is established I subscribe to the USD trades channel:
Code:
TRADES = "dbf1dee9-4f2e-4a08-8cb7-748919a71b21"
def subscribe( channel ):
output = { 'op': 'mtgox.subscribe', 'channel': channel }
return json.dumps( output )

ws.send(subscribe(TRADES))
...but I get the remark spat back "Unknown channel requested or not a public channel".

Try "mtgox.subscribe" instead of "subscribe" and/or also try "type" instead of "channel" and/or also try using the channel name instead of the GUID.

The documentation seems incomplete or outdated, you need to experiment a bit.

For example this works for me: {"op":"mtgox.subscribe", "type":"trades"}
And also: {"op":"mtgox.subscribe", "channel":"ticker.BTCUSD"}

And for subscribing to the private channel (after receiving my idkey) I'm using the following:
{"op":"mtgox.subscribe", "key":idkey}


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Floomi on July 10, 2013, 04:43:32 PM
That works for me, thanks! It's really helpful that the documentation is full of lies  ::)


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Floomi on August 03, 2013, 11:35:12 PM
Thread necro! Is there a limit on the number of times you can call private/idkey? My bot asks for it at startup, which is many times a day during development, and towards the end of the day (perhaps past midnight, if that has anything to do with it?), I start getting "invalid call" back instead of the key.

It's trivial enough for me to store the idkey and the time it was requested, but I wanted to know if anyone else has seen this behaviour, or if it's just Broken On My Machine?

Ta.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Floomi on August 06, 2013, 07:10:06 PM
And another one (after this I'll just start shouting Bueller?): I don't seem to be able to subscribe to both depth and trade messages. If I do, I get depth but not trade. I can repro this in <50 lines of python; is this normal behaviour?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on August 20, 2013, 02:34:41 AM
And another one (after this I'll just start shouting Bueller?): I don't seem to be able to subscribe to both depth and trade messages. If I do, I get depth but not trade. I can repro this in <50 lines of python; is this normal behaviour?

Normal for us two maybe...

Im trying to get websocket stream working (coded on top of vanilla C socket) and I seem to have same problem?

I am getting lag, depth and ticker but no trades. ???

One error message comes back, related to the failing trade subscription.

Code:
!!!THANKS TO PROF7BIT FOR SUBCRIPTION INFO IN EARLIER POST!!!

04:25:30 MtGox_Socket::SocketCreate: CONNECTED TO: websocket.mtgox.com:80
04:25:31 MtGox_Socket::DoHandshake_WebSocket: GOOD: NowListening
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "trade.lag"}
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "ticker.BTCUSD"}
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "depth.BTCUSD"}
04:25:31 TX_STR_FRAMED -> {"op": "mtgox.subscribe", "channel": "trades.BTCUSD"}

04:25:33
{
    "message":"Unknown channel requested or not a public channel",
    "op":"remark",
    "success":false
}
.....
Now lag, depth and ticker JSON comes streaming in
.....


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on August 20, 2013, 03:37:04 AM
Subscribe to trade.BTC  :)

c.f. http://data.mtgox.com/api/2/stream/list_public


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: genuise on August 20, 2013, 06:01:30 AM
wait a second, now you have to subscribe on your own?
They really changed the default behavior when on connect you were subscribed automatically?



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on August 20, 2013, 12:30:45 PM
Subscribe to trade.BTC  :)

c.f. http://data.mtgox.com/api/2/stream/list_public

Awesomeo, thx Clark!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on September 13, 2013, 08:38:45 PM
Well Socket.io and Websocket have been down for a while, for all of us it seems.

Except for this guy, older API version or something? Or maybe just lots of HTTP requests?:

http://bitcointicker.co/

EDIT: curiosity got the better of me, wireshark caught this:

GET /socket.io/1/websocket/qRpvu5AQFZvju4nIao6m HTTP/1.1


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on September 13, 2013, 09:08:52 PM
wait a second, now you have to subscribe on your own?
They really changed the default behavior when on connect you were subscribed automatically?

Yes, it changed for websocket some time ago, or maybe websocket has always been like that I don't know.

And I just noticed that socket.io now also serves a dead stream to start with, you need to subscribe (i think/hope).

For socket.io I was passing all needed currencies as args in initial GET, that doesn't work now.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: yucca on September 14, 2013, 12:36:04 AM
OK, I got websocket working in my client again.

I'm implementing the websocket protocol myself in C++ on a raw socket so this is probably not relevant to those using javascript in browsers.

It turns out that the MtGox websocket now requires properly framed messages as per RFC6455.

A while ago only the Socket.IO required the RFC6455 framing.

Before I was getting away with just this minimal framing for websocket: [0x00] ....RAW STRING DATA... [0xFF]

Now you need the RFC6455 framing function, I posted C code earlier in this thread.

Now I need to get my socket.io implementation working also :P

WebSocket + JSON == WASTED_BANDWIDTH



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Fireball on September 14, 2013, 03:14:56 PM
WebSocket + JSON == WASTED_BANDWIDTH

I plan to investigate the option of using WebSocket + BJSON (http://bsonspec.org/) to prevent bandwidth wasting in ICBIT. We are fine as it is so far, but it's always good to plan for future.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on September 14, 2013, 04:10:08 PM
WebSocket + JSON == WASTED_BANDWIDTH

I plan to investigate the option of using WebSocket + BJSON (http://bsonspec.org/) to prevent bandwidth wasting in ICBIT. We are fine as it is so far, but it's always good to plan for future.

Have you had a look at MessagePack? http://msgpack.org/


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: savis on September 20, 2013, 10:26:21 PM
I was using modified C# http://socketio4net.codeplex.com/ library to consume depths updates from https://socketio.mtgox.com/mtgox. It was working fine till the biggining of this month. Now it may work for several hours, then just drop and stop accepting connections (Error: The connection was closed unexpectedly.).

So I tried to switch to https://github.com/sta/websocket-sharp and connect to wss://websocket.mtgox.com/mtgox. The library seems to be recommended on https://en.bitcoin.it/wiki/MtGox/API/Streaming. First I spent several hours before I found I need to specify Origin. Now it works fine expect that server sends close frame every 5-60 seconds.

Any idea what could be the reason? Thanks.
Code:
01:49:02.874|DEBUG|26 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"132.13605","type":2,"type_str":"bid","volume":"-0.10083051","price_int":"13213605","volume_int":"-10083051","item":"BTC","currency":"USD","now":"1379713750305094","total_volume_int":"0"}}
01:49:02.960|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"123.90414","type":2,"type_str":"bid","volume":"0.60778582","price_int":"12390414","volume_int":"60778582","item":"BTC","currency":"USD","now":"1379713750414105","total_volume_int":"60778582"}}
01:49:04.186|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"139.39536","type":1,"type_str":"ask","volume":"1.103871","price_int":"13939536","volume_int":"110387100","item":"BTC","currency":"USD","now":"1379713751538095","total_volume_int":"110387100"}}
01:49:07.415|DEBUG|8  |MtGoxApi: Close Event raised
01:49:08.622|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"135.986","type":1,"type_str":"ask","volume":"5.196","price_int":"13598600","volume_int":"519600000","item":"BTC","currency":"USD","now":"1379713756075743","total_volume_int":"519600000"}}
01:49:08.661|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"133.85221","type":2,"type_str":"bid","volume":"-3.855","price_int":"13385221","volume_int":"-385500000","item":"BTC","currency":"USD","now":"1379713756090538","total_volume_int":"0"}}
01:49:08.676|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00225","type":2,"type_str":"bid","volume":"-10.8","price_int":"13400225","volume_int":"-1080000000","item":"BTC","currency":"USD","now":"1379713756105880","total_volume_int":"540000000"}}
01:49:09.511|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00225","type":2,"type_str":"bid","volume":"-5.4","price_int":"13400225","volume_int":"-540000000","item":"BTC","currency":"USD","now":"1379713756934842","total_volume_int":"0"}}
01:49:09.571|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"131.41101","type":2,"type_str":"bid","volume":"-5.4","price_int":"13141101","volume_int":"-540000000","item":"BTC","currency":"USD","now":"1379713756950113","total_volume_int":"1080000000"}}
01:49:09.607|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"133.27742","type":2,"type_str":"bid","volume":"5.4","price_int":"13327742","volume_int":"540000000","item":"BTC","currency":"USD","now":"1379713757041934","total_volume_int":"1620000000"}}
01:49:13.229|DEBUG|8  |MtGoxApi: Close Event raised
01:49:14.963|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"129.6864","type":2,"type_str":"bid","volume":"0.25","price_int":"12968640","volume_int":"25000000","item":"BTC","currency":"USD","now":"1379713762412059","total_volume_int":"125000000"}}
01:49:15.040|DEBUG|31 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00226","type":2,"type_str":"bid","volume":"-10.34395506","price_int":"13400226","volume_int":"-1034395506","item":"BTC","currency":"USD","now":"1379713762427277","total_volume_int":"0"}}
01:49:15.101|DEBUG|30 |MtGoxApi: {"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","channel_name":"depth.BTCUSD","op":"private","origin":"broadcast","private":"depth","depth":{"price":"134.00228","type":2,"type_str":"bid","volume":"10.34331649","price_int":"13400228","volume_int":"1034331649","item":"BTC","currency":"USD","now":"1379713762550896","total_volume_int":"1034331649"}}


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alikim on October 21, 2013, 02:32:17 PM
Does anyone know how to subscribe to "own" private channel?

I start with this message:

{"call":"idkey","id":".....","nonce":...}

Successfully get the key:

{"id":"....","op":"result","result":"GhXJXwx..."}

Then subscribe:

{"op":"mtgox.subscribe","key":"GhXJXwx..."}

... and I don't get any response to the subscribe message and after sending orders or their execution I don't get any messages in the private "own" channel.

What am I doing wrong?

Thank you!


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on October 21, 2013, 09:28:28 PM
Then subscribe:

{"op":"mtgox.subscribe","key":"GhXJXwx..."}

... and I don't get any response to the subscribe message and after sending orders or their execution I don't get any messages in the private "own" channel.

What am I doing wrong?

Thank you!

Mt. Gox has been sending a message that my key is not valid. Are you seeing a "remark" message coming in after you attempt to subscribe?

My guess: Either the API has changed, or the private-channel server is down.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alikim on October 22, 2013, 02:44:04 AM
Then subscribe:

{"op":"mtgox.subscribe","key":"GhXJXwx..."}

... and I don't get any response to the subscribe message and after sending orders or their execution I don't get any messages in the private "own" channel.

What am I doing wrong?

Thank you!

Mt. Gox has been sending a message that my key is not valid. Are you seeing a "remark" message coming in after you attempt to subscribe?

My guess: Either the API has changed, or the private-channel server is down.

Yes, if I artificially add another symbol at the end of the key in subscribe call, I get that message that the key is not valid.

If I subscribe with the key as it is, I get no message, no error neither success.

Do I understand it right that you do not sign the subscribe message?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on October 22, 2013, 03:07:42 AM
Yes, if I artificially add another symbol at the end of the key in subscribe call, I get that message that the key is not valid.

If I subscribe with the key as it is, I get no message, no error neither success.

Do I understand it right that you do not sign the subscribe message?

Interesting. I am sending the key as-is and getting an 'invalid key' message.

I am not signing the message, simply sending under mtgox.subscribe.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alikim on October 22, 2013, 01:13:01 PM
Yes, if I artificially add another symbol at the end of the key in subscribe call, I get that message that the key is not valid.

If I subscribe with the key as it is, I get no message, no error neither success.

Do I understand it right that you do not sign the subscribe message?

Interesting. I am sending the key as-is and getting an 'invalid key' message.

I am not signing the message, simply sending under mtgox.subscribe.

All is good now, you were right, the private channel server must have been down.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Clark on October 23, 2013, 08:09:49 PM
All is good now, you were right, the private channel server must have been down.

I am continuing to experience the same problem.

Steps I am doing:

1. Grab the idkey from either /1/generic/private/idkey or /2/money/idkey.
2. Over WebSocket, send {"op":"mtgox.subscribe","key":"idkey_as_from_API_call"}
3. Get response over WebSocket socket: {"message":"Invalid key provided","op":"remark","success":false}

@alikim You say it works for you. Can you verify the message you are sending over the WebSocket to subscribe to the private channel.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on October 24, 2013, 02:30:43 PM
This is what I do (python code) and it works (doesn't seem to have ever changed, I never had to change my code):

Code:
self.client.send(json.dumps({"op":"mtgox.subscribe", "key":result}))

result contains the idkey exactly as it resulted from the idkey request. The above code is from goxapi.py around line 1497 in the on_op_result() method.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alikim on October 27, 2013, 07:10:37 AM
All is good now, you were right, the private channel server must have been down.

I am continuing to experience the same problem.

Steps I am doing:

1. Grab the idkey from either /1/generic/private/idkey or /2/money/idkey.
2. Over WebSocket, send {"op":"mtgox.subscribe","key":"idkey_as_from_API_call"}
3. Get response over WebSocket socket: {"message":"Invalid key provided","op":"remark","success":false}

@alikim You say it works for you. Can you verify the message you are sending over the WebSocket to subscribe to the private channel.


This is what I do over the websocket, the first message is a signed encrypted one:

{"call":"idkey","id":"2a4b88398c872f7150ea4f3a6f.....","nonce":13828......}

The response:

{"id":"2a4b88398c872f7150ea4f.....","op":"result","result":"GhXJXwx......."}

Subscribe command, goes out as plain text:

{"op":"mtgox.subscribe","key":"GhXJXwx......."}

After that I don't receive anything, but the private channel - when /if it's up - will send it's proper messages, for example when you place an order.

I believe if the channel is not working it just won't send you messages but you can still get silently subscribed.

Maybe it's working over the websocket but not working over http API?

Programming and maintenance of various gox servers and services is very sloppy to say the least, these people don't seem to care much...


P.S. On second thought, do you grab the key from http API and then try to use it in websocket API?
Maybe all three steps you've mentioned should go through the websocket, including step one (not using  /1/generic/private/idkey or /2/money/idkey); the websocket will accept the key issued only through the websocket message?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: alikim on October 27, 2013, 07:37:41 AM
In case any developer is following this thread I added a suggestion about the websocket API here:

https://bitcointalk.org/index.php?topic=319546.0


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: frito on December 27, 2013, 07:44:11 PM
Hi websocket uber pros,

I have been testing GOX websocket stream using Xchange java framework.

There are horrible data gaps(sometimes 90% of quotes missed) and lags all the time, both the book and the ticker.

Is the gox websocket crap? Or is the problem with the Xchange implementation or the websocket java lib?
Any idea suggestion or wisdom is welcomed.

thx
f


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on December 27, 2013, 08:35:33 PM
Is the gox websocket crap? Or is the problem with the Xchange implementation or the websocket java lib?
Any idea suggestion or wisdom is welcomed.

MtGox websocket is excellent, 99.9% uptime and <200ms delay. Try goxtool (see my sig) for a proper implementation and reference.


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Praeconium on January 02, 2014, 11:25:14 PM
This is what I do (python code) and it works (doesn't seem to have ever changed, I never had to change my code):

Code:
self.client.send(json.dumps({"op":"mtgox.subscribe", "key":result}))

result contains the idkey exactly as it resulted from the idkey request. The above code is from goxapi.py around line 1497 in the on_op_result() method.

Is it anyhow possible to add a module which would collect info and construct historical database of orderbook and trades?



Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: Zicore47 on January 21, 2014, 09:54:26 AM
I subscribed to the market depth channel via the websocket and I'm parsing the data with Json.net (c#), but there is a problem when I receive a volume in scientific notation.

The volume is treated as string whenever it is in scientific notation and this leads to problems with parsing.

Code:
"volume":"1.0E-8"

In this case Json.Net is unable to parse the string to decimal by default.

To get around this, I currently use this construct:

Code:
        public decimal Volume
        {
            get { return _volume; }
            set { _volume = value; }
        }

        [JsonProperty("volume", Required = Required.Always)]
        public String VolumeString
        {
            get { return Volume.ToString(CultureInfo.InvariantCulture); }
            set { Volume = Decimal.Parse(value, NumberStyles.Any, CultureInfo.InvariantCulture); }
        }

Any suggestions ?


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on January 21, 2014, 10:16:39 AM
Any suggestions ?
Use the integer values for everything, they are contained in every json object that carries any numbers. For BTC (order volume) they represent integer number of satoshi, for fiat currency (quote prices) they represent either 1e-5 or 1e-3 units of the quote currency (depends on what currency it is, USD and most others is 1e-5, JPY and SEK its 1e-3).


Title: Re: [BETA] MTGox websocket API, testers wanted
Post by: prof7bit on February 18, 2014, 04:44:34 PM
i doubt it, btw thx
Doubt what? Just have look at the json that is sent by gox and see for yourself what it contains!