Bitcoin Forum
October 18, 2017, 05:00:37 PM *
News: Latest stable version of Bitcoin Core: 0.15.0.1  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 [All]
  Print  
Author Topic: [BETA] MTGox websocket API, testers wanted  (Read 75873 times)
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 15, 2011, 02:48:21 AM
#1

Hi,

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

You can connect via: ws://websocket.mtgox.com/mtgox

The websocket will subscribe you to some channels automatically:
  • dbf1dee9-4f2e-4a08-8cb7-748919a71b21: trades (each time a trade happens, you get something here)
  • d5f06780-30a8-4a48-a2f8-7ed181b4a13f: the mtgox ticker (lots of updates, with often the same data)
  • 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe: depth information in realtime (price + amount + type... type=1=Ask, type=2=Bid)

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Each message is a JSON-encoded object, with at least "op" element. The "op" element contains the operation to be done (for outgoing messages), or the type of message (for incoming messages).

Possible outgoing commands:

  • unsubscribe Stop receiving messages from a channel (parameter "channel")

Example incoming data:

Ticker
Code:
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":0.9515,"high":1,"low":0.91,"sell":0.9697,"vol":34349}}
Trade
Code:
{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade","trade":{"amount":2.71,"amount_int":"271000000","date":1310279340,"item":"BTC","price":14.43,"price_currency":"USD","price_int":"1443000","tid":"1310279340877902","trade_type":"bid","type":"trade"}}

Contains:
  • amount: the traded amount in item (BTC), float, deprecated
  • amount_int: same as amount, but in smallest unit
  • date: unix timestamp of trade
  • item: What was this trade about
  • price: price per unit, float, deprecated
  • price_int: price in smallest unit as integer (5 decimals of USD, 3 in case of JPY)
  • price_currency: currency in which trade was completed
  • tid: Trade id (big integer, which is in fact trade timestamp in microseconds)
  • trade_type: Did this trade result from the execution of a bid or a ask?

Depth update
Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"14.43","price_int":"1443000","type":1,"type_str":"ask","volume":"-2.71","volume_int":"-271000000"},"op":"private","origin":"broadcast","private":"depth"}

Contains:
  • currency: the currency affected
  • item: the item (BTC)
  • price: price as a float, deprecated
  • price_int: the price at which volume change happened
  • type: 1=ask 2=bid. Deprecated, see type_str
  • type_str: type of order at this depth, either "ask" or "bid"
  • volume: the volume change as float, deprecated
  • volume_int: volume change in smallest unit

(priv) Order update
Code:
{"channel":"(partial key)","op":"private","order_upd":{"amount":1000,"darkStatus":0,"date":1302836027,"oid":"(oid)","price":0.9899,"status":1},"origin":"broadcast","private":"order_upd"}


This is still under tests, and any element of this websocket API may change anytime.
Ideas/comments are welcome Smiley

1508346037
Hero Member
*
Offline Offline

Posts: 1508346037

View Profile Personal Message (Offline)

Ignore
1508346037
Reply with quote  #2

1508346037
Report to moderator
1508346037
Hero Member
*
Offline Offline

Posts: 1508346037

View Profile Personal Message (Offline)

Ignore
1508346037
Reply with quote  #2

1508346037
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1508346037
Hero Member
*
Offline Offline

Posts: 1508346037

View Profile Personal Message (Offline)

Ignore
1508346037
Reply with quote  #2

1508346037
Report to moderator
1508346037
Hero Member
*
Offline Offline

Posts: 1508346037

View Profile Personal Message (Offline)

Ignore
1508346037
Reply with quote  #2

1508346037
Report to moderator
1508346037
Hero Member
*
Offline Offline

Posts: 1508346037

View Profile Personal Message (Offline)

Ignore
1508346037
Reply with quote  #2

1508346037
Report to moderator
demonofelru
Full Member
***
Offline Offline

Activity: 238



View Profile
April 15, 2011, 02:54:51 AM
#2

Nice keep the updates coming Tux much appreciated.

Names do not matter; however, if you insist...id...
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 15, 2011, 03:58:34 AM
#3

This will help a ton! Thank you very much.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
Stephen Gornick
Legendary
*
Offline Offline

Activity: 2282



View Profile
April 15, 2011, 04:11:16 AM
#4

Hoping newsham (or anyone else) will update the MtGox chart to accommodate the new Websocket data feed:
   http://www.thenewsh.com/~newsham/x/mtgox/

Keefe
Hero Member
*****
Offline Offline

Activity: 681


View Profile
April 15, 2011, 08:02:01 AM
#5

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 15, 2011, 08:06:02 AM
#6

I look forward to a feed of bids and asks, so I can restore newsham's page to it's former glory. I don't need the whole order book to be sent every time there's a change; just a stream of updates would be good. For example:
add 1001 bid 23.45 1.043
add 1002 ask 34.56 1.07
remove 1001
fill 1002 14.56
fill 1002 20.00
...

Should be possible, I'll do something Smiley

memvola
Hero Member
*****
Offline Offline

Activity: 938


View Profile
April 15, 2011, 10:34:25 AM
#7

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 15, 2011, 11:42:37 AM
#8

I'm hoping to move my automated trading system from poll-based to event-based. It's first I heard about websockets, looks like it's invented for browser based systems. Is it worth writing a standalone program that makes use of this (in which case I'll start right ahead) or do you have a more basic socket interface in your plans? Or am I talking nonsense (haven't slept for a while)?


The draft-75 version of the protocol is really easy to implement from an application.

You send a simple HTTP request:

Code:
GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

(remember to add an empty line at the end)

And you get the following reply:

Code:
HTTP/1.1 101 Web Socket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
WebSocket-Origin: null
WebSocket-Location: ws://websocket.mtgox.com/mtgox

Once you got this, the websocket server will send you frames in the following format: a NUL byte (00), some JSON-encoded data, and a FF byte. You can then decode each json packet as they come and handle them.

You can also send packets in the same way, by sending a NUL byte, some json-encoded data, and a FF byte (for example to send an unsubscribe request, or subscribe to a mtgox key).

Example with telnet: http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
April 15, 2011, 03:15:10 PM
#9

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field  Roll Eyes

mndrix
Michael Hendricks
VIP
Sr. Member
*
Offline Offline

Activity: 447


View Profile
April 15, 2011, 03:29:20 PM
#10

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 15, 2011, 03:46:43 PM
#11

Hello, thanks for the API. Few questions/comments:

b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell...


Yes, from reading the protocol specification, it looks like I have to send all sorts of different bytes at different times to the server. Some of them translate into ASCII, while others are non-printing. I get confused here, because in the examples, it looks like we're just sending plain lines of text. Do we need byte strings? Do we need /r/n newlines?

And server timestamps would be a huge help! There are multiple options here, really. First, you could just timestamp very accurately, so the client can recreate the actual order flow precisely. Alternatively (and probably easier), just create a long integer ascending order number that goes along with the trades and orders. That way, the server can stay on whole seconds for timestamps, but the client can arrange trades and orders accurately using the id numbers. There should be separate numbering lists for orders and trades, of course.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
grondilu
Legendary
*
Offline Offline

Activity: 1134


View Profile
April 15, 2011, 04:15:16 PM
#12

Damn it...

So many things to learn, and only one life Sad
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
April 15, 2011, 05:44:23 PM
#13

Is websocket api down?

Code:
$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

memvola
Hero Member
*****
Offline Offline

Activity: 938


View Profile
April 15, 2011, 06:24:25 PM
#14

Is websocket api down?
It's down for me for at least the last hour.

And thanks for the nice explanation, MagicalTux. I wrote a client in haskell, I'll post it when the subscription part is complete.
jed
Full Member
***
Offline Offline

Activity: 183

Jed McCaleb


View Profile WWW
April 15, 2011, 06:24:56 PM
#15

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

stellar.org   |    twitter
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 16, 2011, 02:50:37 AM
#16

Is websocket api down?

Code:
$ telnet websocket.mtgox.com 80
Trying 69.64.54.38...
telnet: Unable to connect to remote host: Connection refused

Yep was down, found the reason, fixed the code and restarted it (division by zero when someone sent badly formatted draft-00 headers).

Hello, thanks for the API. Few questions/comments:

a) What exactly is the ticker? Why server broadcast so many packets without any changed data? I see few updates per second without any change...
b) Can you please post example of subscribe/unsubscribe command? It's not clear for me how to unsubscribe from ticker.
c) Can you include (official) server timestamps to the broadcasts? Client timestamps in trading platforms are hell... I'm sorry, I miss 'date' field  Roll Eyes


a) The ticker is the current stats about bitcoin. It's sent once each time it have a chance to be changed, which includes when orders are posted, removed or completed.
b) Here's an example:
Code:
{"op":"unsubscribe","channel":"xxx"}

Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades).

Does a user's "own" channel include information about when BTC/USD funds post to his account?

Not at this point.

why are you using UUID's to name the channels rather than just something human readable like: "trades" ?

Because it was easier since I intend to use this system for many other sites, without having to do something to isolate users of a specific system. Also there are user-specific channels and many other stuff which makes use of uuid a bit easier (for me).

memvola
Hero Member
*****
Offline Offline

Activity: 938


View Profile
April 16, 2011, 09:56:15 AM
#17

Here's the haskell code which connects to the server, subscribes to the private channel and unsubscribes from the ticker. No json parsing or error handling.

Code:
module Main (main) where

import Network
import System.IO
import Control.Monad

clientHandshake :: String
clientHandshake = "GET /mtgox HTTP/1.1\r\n\
                  \Upgrade: WebSocket\r\n\
                  \Connection: Upgrade\r\n\
                  \Host: websocket.mtgox.com\r\n\
                  \Origin: null\r\n\
                  \\r\n"

-- Expected server response
serverHandshake :: [String]
serverHandshake = ["HTTP/1.1 101 Web Socket Protocol Handshake",
                   "Upgrade: WebSocket","Connection: Upgrade",
                   "WebSocket-Origin: null",
                   "WebSocket-Location: ws://websocket.mtgox.com/mtgox",
                   "WebSocket-Protocol: *"]

-- https://mtgox.com/code/getKey.php
key :: String
key = "\"key\":\"00000000-0000-0000-0000-000000000000:0000000000000000000000000000\""

hGetHeader :: Handle -> IO [String]
hGetHeader h = do s <- hGetLine h
                  if s == "\r"
                    then return []
                    else do s' <- hGetHeader h
                            return (init s : s')

hGetFrame :: Handle -> IO String
hGetFrame h = do c <- hGetChar h
                 if c == '\xff'
                   then return "\n"
                   else do c' <- hGetFrame h
                           return $ if c == '\x00' then c' else (c:c')

hPutFrame :: Handle -> String -> IO ()
hPutFrame h s = hPutStr h $ '\x00' : s ++ "\xff"

main :: IO ()
main = do h <- connectTo "websocket.mtgox.com" (PortNumber 80)
          hSetBuffering h NoBuffering
          hPutStr h clientHandshake
          hdr <- hGetHeader h
          when (head hdr /= head serverHandshake) .
               error $ "Invalid server handshake:\n" ++ show hdr

          hPutFrame h "{\"op\":\"unsubscribe\",\"channel\":\"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\"}"
          hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

          forever $ hGetFrame h >>= putStr

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 16, 2011, 10:00:45 AM
#18

Everything works as expected. Looks like I get a lot of identical ticker frames when I enter an order, I'm guessing this is because there are a lot of bots adjusting orders in response?

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this Smiley

memvola
Hero Member
*****
Offline Offline

Activity: 938


View Profile
April 17, 2011, 04:21:11 PM
#19

Might just have been "by chance". There are a lot of ticker events, no need to think too hard about this Smiley
Well, anyway, I was actually hoping to be able to keep market depth up to date without polling, like I can do with recent trades using the trades channel, but it seems there's currently no way to do the same with orders. Ticker data doesn't report new or deleted orders, or at least I failed to identify them. I don't want to ask for updates like crazy. Smiley Are you planning to introduce a new channel for this?

EDIT: Oh, Keefe already asked for this but I missed it. Nevermind then...
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 17, 2011, 11:39:23 PM
#20

So I've answered some of my own questions:

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

I see there is a new market depth channel as well.

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending
Code:
{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}
and
Code:
{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Here's an idea for an initial documentation page:
List the possible fields contained in the JSON return objects, starting with op values and going from there. This is what I have so far, from seeing different things come across the socket:
Code:
op:remark
  message:<message text>
  success:<success boolean>
op:subscribe
  channel:<channel uuid>
op:unsubscribe
  channel:<channel uuid>
op:private
  channel:<channel uuid>
  origin:broadcast
  private:depth
    depth:{
      volume:<volume>
      price:<price>
      type:<order type>
    }
  private:ticker
    ticker:{
      high:<high price>
      low:<low price>
      vol:<volume>
      buy:<buy price>
      sell:<sell price>
    }
  private:trade
    trade:{
      date:<int time>
      amount:<amount float>
      type:trade
      price:<trade price>
   }
  private:order_add
    order_add:{
      oid:<order id int>
      price:<limit price>
      date:<int time>
      amount:<amount>
      status:<status int>
      darkStatus:<0 or 1>
    }
  private:order_rem
    order_rem:{
      oid:<order id int>
    }

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 18, 2011, 12:01:17 AM
#21

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending
Code:
{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}
and
Code:
{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Just added:

Code:
{"op":"mtgox.subscribe","type":"trades"}

Possible types: trades, ticker, depth

slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
April 18, 2011, 02:04:29 AM
#22

Example usage of this API from Python: MtGox->Sierrachart feed

0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
April 19, 2011, 03:35:16 PM
#23

Occasionally the websocket sends a depth message to remove an ask or bid (has a negative volume) with a "null" price.  I've only noticed this right before some (not all) trade messages.  I haven't yet seen it happen at volumes unrelated to a trade.

Could it maybe be a race condition in your code where most of the time it gets the depth information before the trade clears it, but sometimes it doesn't?
0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
April 19, 2011, 05:06:29 PM
#24

Now that I've looked at more instances of it, the volumes are not always obviously related to the subsequent trades, though I think they're related at least most of the time, since these nulls only seem to occur right before a trade or throughout a series of trades.

Here are some examples of the "null" depth message with some surrounding depth/trade info (I removed some messages coming in around these same times that did not seem relevant):

Code:
REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-56},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -56 @ nil
TRADE 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-46},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -46 @ nil
TRADE 17 @ 1.1827
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-29},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -29 @ nil
TRADE 23.216 @ 1.18151

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-5.784},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -5.784 @ nil
TRADE 5.784 @ 1.1621

Code:
ASK 0.026 @ 1.1875
 ASK 9.974 @ 1.1875
 ASK 0.03 @ 1.1875
REMOVE ASK -9.97 @ 1.1875
 BID 1 @ 1.18751
 ASK 0.026 @ 1.1875
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-9.974},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -9.974 @ nil
TRADE 1 @ 1.18751

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-4},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -4 @ nil
TRADE 4 @ 1.18751

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-19.966},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -19.966 @ nil
TRADE 19.966 @ 1.18751
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 19, 2011, 10:40:12 PM
#25

Hi,

I have located the source of the null and fixed it Smiley


Mark

0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
April 24, 2011, 11:49:41 PM
#26

@MagicalTux

When do you think you'll be able to stop broadcasting dark orders and orders with insufficient funds on the websocket?
jed
Full Member
***
Offline Offline

Activity: 183

Jed McCaleb


View Profile WWW
April 28, 2011, 08:46:18 PM
#27

This depth channel is broadcasting the changes to the depth. It would be nice if you could send something in the beginning to get the current depth table in the same format. Then you can use this feed to change the original depth table as time goes on.
I know I can get it with the old API but it would be nice if the messages were in the same format.
Also is there a point to all these extra fields: "op":"private","origin":"broadcast","private":"depth" ?

stellar.org   |    twitter
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 29, 2011, 02:03:48 AM
#28

Hi,

Broadcasting of depth changes fixed, I think (mostly).

As for the extra fields, they are there mainly to avoid mistaking the messages for something else, however you can safely ignore them Smiley

0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
April 29, 2011, 05:35:27 AM
#29

It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
April 29, 2011, 06:48:56 AM
#30

It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!

Fixed notices for orders without enough funds.

As for the trade price, it'll be fixed in a few days when we switch to a new backend~

jed
Full Member
***
Offline Offline

Activity: 183

Jed McCaleb


View Profile WWW
May 04, 2011, 02:29:52 PM
#31

I noticed that sometimes the price is sent in quotes:

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

stellar.org   |    twitter
MagicalTux
VIP
Hero Member
*
Offline Offline

Activity: 608


Working on new MtGox features


View Profile WWW
May 04, 2011, 02:45:48 PM
#32

I noticed that sometimes the price is sent in quotes:

Code:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

When the price comes from mysql, it is as a string. I could cast it to float to make sure it's a float, but I guess it doesn't change much for anyone (shouldn't, anyway)

jed
Full Member
***
Offline Offline

Activity: 183

Jed McCaleb


View Profile WWW
May 04, 2011, 04:33:10 PM
#33

Shouldn't the order_add message tell you if it was a buy or sell order?

stellar.org   |    twitter
jed
Full Member
***
Offline Offline

Activity: 183

Jed McCaleb


View Profile WWW
May 06, 2011, 05:25:24 PM
#34

Anybody else occasionally miss trade messages?

stellar.org   |    twitter
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 09, 2011, 11:59:10 AM
#35

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.
Hmm, I also tried this. Can't seem to succeed.

What port do I have to use? I tried like this:
Code:
from pywsc.websocket import WebSocket
ws = WebSocket('ws://websocket.mtgox.com:80/mtgox')
This bails with "error receiving" in receiver.py. I inserted debugging output to see what it's receiving. It seems to receive the <html/> page you get when you request http://websocket.mtgox.com/.

I'm confused and obviously doing something wrong. Can someone help?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
May 11, 2011, 06:19:31 AM
#36

Well, I had to change my WebSocket._send function to do this:
Code:
self._sendRaw('%s%s%s' % ('\x00', data, '\xff'))
because it was breaking the socket trying to send a non-string.

My receive code is identical except that I added a Threading.Event for that loop instead of just while True.

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 11, 2011, 04:01:22 PM
#37

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

SOLVED! Thanks for your help.

Stupid me forgot about that transparent squid proxy.

Am receiving messages, now to the fun stuff!

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
May 11, 2011, 05:03:16 PM
#38

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
toffoo
Sr. Member
****
Offline Offline

Activity: 402



View Profile
May 11, 2011, 07:12:52 PM
#39

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

Fantastic news.  Please let us know when you have something ready to test out.

molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 12, 2011, 10:07:11 AM
#40

ok, I'm receiving messages, nice Wink

Now I want to update the market depth info, which I'm initially getting through http://mtgox.com/code/data/getDepth.php.

so I'm listening on channel "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe" for messages that have op="private", and am getting depth updates like this:

Code:
{u'volume': -10, u'price': 5.4000000000000004, u'type': 2}

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Above message is an example of a message that caused this... I didn't have an entry for price "5.4000" (I'm using 4 decimal places for the index in my depth "dictionary", but that should do no harm, right), so the result is a new entry with negative volume.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

EDIT:

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

If my problem actually stems from the two timepoints (initial depth request, start of depth updates) not being equal, I would consider this to be not a nice-to-have but a must-have.

You could maybe simply (from client point of view) send the complete depth table as a burst of updates after websocket connect?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Shrapnel
Newbie
*
Offline Offline

Activity: 9


View Profile
May 12, 2011, 10:19:27 AM
#41

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Quote
Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?
I'm also wondering the same thing. I think you can't unless the API send the complete data first.
I'm trying to do a realtime order book but it's currently very hard due to dark pool and sync issues.

My BTC OSX Dashboard Widget http://bitcointalk.org/index.php?topic=6690.0 (http://bitcointalk.org/index.php?topic=6690.0)
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 12, 2011, 09:47:32 PM
#42

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.



PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
jed
Full Member
***
Offline Offline

Activity: 183

Jed McCaleb


View Profile WWW
May 12, 2011, 10:14:12 PM
#43

I think it doesn't send every update to the order book. It seems like some random ones get dropped.

stellar.org   |    twitter
0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
May 13, 2011, 02:42:32 AM
#44

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

Mtgox has some bugs.  The websocket often entirely skips sending some depth messages, sends some at a slightly different price than what they should be, and the APIs (or site in general) has rounding inconsistencies.  I've only partially worked around these issues by rounding all depth message prices to the 5th decimal place and volumes to the 3rd decimal place, and creating a button in my program that I can click to manually refresh the entire depth tables when I start noticing odd stuff.

An example is where a bot will download it's own open orders, see that one of it's orders is something like 6.253, but it wants it to be 6.252999, so it removes the order for 6.253 and creates a new one at 6.252999, and both of those come over the websocket just like that.  And the bot does this over and over again because the API for getting your own open orders rounds to the 4th decimal place, but the websocket will do at least 6 decimal places for prices and more than that for volume, so your table ends up removing the order at 6.253 over and over again, resulting in a negative number, while it keeps adding orders to 6.252999 over and over again, making some huge number that's not accurate.  The bugs and inconsistencies are pretty annoying.
Keefe
Hero Member
*****
Offline Offline

Activity: 681


View Profile
May 13, 2011, 03:48:53 AM
#45

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 13, 2011, 11:32:29 PM
#46

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

MagicalTux, can you give us an estimate when you'll get around to implementing a solution to these problems or talk to us about a solution?

These problems are really not acceptable: we can't have the client side depth data out of sync or even worse stuff happening (missing data?)

I'm here to help with testing.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
jaybny
Sr. Member
****
Offline Offline

Activity: 392


Fantasybits - Proof-of-Skill


View Profile WWW
May 14, 2011, 12:49:05 AM
#47

my program is listening to the WebSocket and publishing to twitter once an hour, on new highs or lows.

im using c# and am not  using JSON schema, just parsing directly... so if anything changes - I will break..

http://twitter.com/newberg_feed


Ycros
Newbie
*
Offline Offline

Activity: 1


View Profile
May 14, 2011, 08:10:11 AM
#48

Herp, disregard what I posted before, I found my client library was sending an extraneous \r\n\r\n on the end there.

1BondRG1ZaHecQARxAtMcnT4g4uEH2pRrM
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 14, 2011, 09:10:44 AM
#49

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

I'm seeing a slight problem for MagilTux to solve the "keep client depth tables in sync" problem, I hadn't thought about:

(Firstly, wether a client gets the initial depth data through the "classic API" or through an initial burst of depth-change messages doesn't make a difference for my thinking here.)

Problem is: the depth data a client receives initially is chopped off at the ends at a certain distance to the current last trade value.

Now whenever a trade happens, this window moves and some trades exit the window on the one end while others become "visible" on the other end.

There's 2 ways (at least) to solve this:

  • Screw that window completely, give the complete depth data to clients
  • On trade, determine entering/leaving orders and send them as new/cancelled over the channel

I'd go for the first option in combination with initially bursting change-messages instead of using the "classic API" to initially fill client depth table.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
toffoo
Sr. Member
****
Offline Offline

Activity: 402



View Profile
May 17, 2011, 04:40:02 AM
#50

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

mindtheft
Newbie
*
Offline Offline

Activity: 11


View Profile
May 17, 2011, 10:22:23 PM
#51

A quick js chrome extension hack for WebSocket -

https://github.com/mindtheft/mtgox-Chrome-Extension

You can use the crx file to install the extension, or better yet check the source code and pack your own crx file to make sure there's no foul smells from mine.

0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
May 18, 2011, 06:16:43 PM
#52

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!


Here's a naive Mt. Gox websocket client I wrote in python: http://pastebin.com/FH6eWhFm
By default it just prints out trades as they happen, but you can modify it to do whatever you want.  If someone really wants an event-based version or something to make it a little easier to work with, you might be able to convince me (with a few bitcoins Wink) to put something together with pywsc or something.  I may eventually release an executable jar (clojure/java) file for those willing to buy a copy that will give you a GUI to view live trades and depth changes (though mtgox's websocket is still quite buggy in respect to depth changes...there would be a button to let you do manual refreshes as well).
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
May 18, 2011, 09:56:47 PM
#53

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

0x6763
Newbie
*
Offline Offline

Activity: 24


View Profile
May 19, 2011, 03:06:47 AM
#54

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

Mtgox currently has multiple bugs in the websocket's depth channel...your tables are behaving exactly the same as mine are.  Mtgox is moving to a new system in june which will hopefully have all of these problems taken care of.
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
May 19, 2011, 03:45:30 PM
#55

Ah excellent, any information on the form of this new system?

zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
May 22, 2011, 01:25:33 PM
#56

I wrote some code in C# for MonoDevelop or Visual Studio
MtGox & IMtGoxHandler & TestMtGox - for making orders
  FetchCash
  FetchOrders
  Place/CancelBuyOrder
  Place/CancelSellOrder
  .Handler - interface for responses & errors
WebSocket & Program - for read trades events
  .OnData - delegate received string (in JSON format) for any channel

https://rapidshare.com/files/3646016978/MtGoxApp.zip
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
May 22, 2011, 04:38:11 PM
#57

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

technically:
JSON changes to this: add new attr to "depth": "abs"=X: where X = "1" for absolute volume; "0" or absent attr - relative volume as now
That data can be cashed on server for all new clients. Depth changes updates cache. Need to generate JSON (for sent) on request only - more expansive operation than just update some structs in cache.

PS
Sorry for my Russian English Smiley
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
May 22, 2011, 05:09:17 PM
#58

maybe
needs some authorization vs WebSockets' flood from bag guys (direction: from client to server)

something like this:
- some cookie received from Trade API for websocket connection to MtGox. this cookie valid only next hour (5-10-30min)
- on new connection to websocket client must send this cookie to server. if cookie is valid automatic subscribing will do..
cons: more difficult process for connection

or:
- using SSL/TLS encrypted WebSocekts.
pro: can refuse polling Trade API and do all orders and receive statuses over WS
cons: need implementations for different languages
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
May 22, 2011, 09:27:32 PM
#59

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?
for price enough.. 2 digits for cents.. and.. ok, let be 4digits after dot
for volume.. hmm.. similar numbers less than 0.001 are considered as zero.. = so 3 digits after dot for volumes
martin
Full Member
***
Offline Offline

Activity: 150



View Profile WWW
May 23, 2011, 02:08:23 PM
#60

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?

We need as many digits as people can specify, which as far as I know is an unlimited amount.

zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
May 23, 2011, 05:02:07 PM
#61

I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.
good variant (1)
but people dont want to calculate depth state. they often will request the one. more traffic, slow server, bad trading.

variant (1.5)
every depth message contains all state. without unnessary attrs - one number is price, next in pair is volume with right sort order: "depth":{"asks":[X.XXXX, V.VVV,...],"bids":[...]}

another variant (2):
- new channel for depth-20(or some N) values asks & bids every one minute (only for checking purpose or as starting point)
- clear description of algorithm how calculate depth state over trade & depth channels

one more variant (2.5):
- depth values is absolute forever
- new attr "pos" with values { +N, N, -N } where
     +N - new row inserted at N pos, rows [N..end] shifted down
       N - updates values ar row N
      -N - row N deleted. other rows shifted to close gap (to up)
- "trade" values ​​must match the changes in "depth" values (a priori)
- but still need starting point - current depth state - use variant (2) or something
pros: simple calcs, asks & bids always sorted by server(that must be), traffic +1attr, CPU calcs almost same(as now)

"make your choice!" Smiley

We need as many digits as people can specify, which as far as I know is an unlimited amount.
now: values with volume less 0.001 remains in my depth state after process depth & trades from corresponding channels. but next ticker does not account for their. so, I delete them from calculated depth state.
leepfrog
Full Member
***
Offline Offline

Activity: 238

Hopity Hopity Hop


View Profile
May 25, 2011, 05:52:36 PM
#62

This might be a stupid question but I dont fully understand the meaning of the ticker.php api.
It returns something like that:

{"ticker":{"high":7.51,"low":7,"vol":33895,"buy":7.257,"sell":7.3499,"last":7.2591}}

What exactly is high and low? the rest seems pretty clear but I am not sure about those two

Unpaid signature whore.
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
May 25, 2011, 06:03:12 PM
#63

The ticker high and low are probably those respective prices for the current trading session, which I'm guessing is a full day. You should watch the ticker values for when the volume resets to zero, as that would be the beginning of the trading session.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
leepfrog
Full Member
***
Offline Offline

Activity: 238

Hopity Hopity Hop


View Profile
May 25, 2011, 06:12:55 PM
#64

Makes sense - I'll watch it. Thanks!

Unpaid signature whore.
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
May 25, 2011, 08:14:04 PM
#65

I'm not sure it's been mentioned before, but the data type of the getDepth.php return does not seem to be application/json, so it is not possible to do a cross-domain call to MtGox for the depth of market data.

This would be a really easy and quick fix in the getDepth.php code...

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
robottwo
Newbie
*
Offline Offline

Activity: 6


View Profile
June 07, 2011, 03:09:51 AM
#66

Hi,
    The websockets API is *great*. I have a suggestions for the protocol:


The size in the depth messages should publish the new absolute size, not the relative size. Subscribers can still figure out the relative size change by subtracting from the previous update at that size, but this provides better error recovery on the client side.
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
June 08, 2011, 03:15:33 PM
#67

hint (from my bugs)
see MSDN Math.Round( decimal d, int decimals ), which I use
MtGox rounds numbers NOT SAME:
If the value of the first digit in d to the right of the decimals decimal position is 5, the digit in the decimals position is rounded up if it is odd, or left unchanged if it is even. If the precision of d is less than decimals, d is returned unchanged. The behavior of this method follows IEEE Standard 754, section 4. This kind of rounding is sometimes called rounding to nearest, or banker's rounding. It minimizes rounding errors that result from consistently rounding a midpoint value in a single direction.

some datas:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":30.22425,"type":1,"volume":1},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private","origin":"broadcast","private":"ticker","ticker":{"buy":30,"high":31.89,"last":30,"low":18.6,"sell":30.2243,"vol":98781}}
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
June 08, 2011, 05:03:45 PM
#68

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.

Which min volume of order required for trading (min step size)?
I saw volume=0.007 in depth channel, but ticker channel ignores that values (so values lessOrEqual 0.007 I ignore)
jaybny
Sr. Member
****
Offline Offline

Activity: 392


Fantasybits - Proof-of-Skill


View Profile WWW
June 10, 2011, 06:06:16 PM
#69

is websockets down? any ETA?

thanks

af12345
Newbie
*
Offline Offline

Activity: 1


View Profile
June 11, 2011, 02:45:56 PM
#70

I think the API shows a lot of promise, but it's seems to be missing one key detail --- without publishing an order-id on the depth message, and the constituent order-ids on the trade message, it's not really possible to ever get in sync.

The reason is as follows: if you start your connection and receive an order @ 25 for volume 10, and then there is a trade @ 25 for volume 5, you can't tell if the trade was for part of the order you just received, or an earlier order (pre your connection time).

If, however, you were to provide an order-id on the depth message, and the pair of order ids on the trades, then all ambiguity is removed.
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
June 11, 2011, 03:27:26 PM
#71

Here are some things I've noticed after building some web applications using the WebScokets API:

  • Switching to the Secure WebSocket (wss://) would dramatically improve compatibility, especially for those behind a proxy
  • Security would also allow users to send trades via the WebSocket, which would simplify trading to one connection instead of polling+WS.
  • Send the prices and volumes as integers! Bitcoins are stored as integers in the client but displayed as floats by dividing by 1e8. Some of the prices and volumes quoted over the WebSocket are floating point numbers with much more precision than the 8th decimal, which doesn't make sense in the framework of Bitcoin overall.
  • The ability to request the depth of market table entries through the WS
  • More consistency in depth of market updates - I have to poll the data every so often in order to eliminate discrepancies, which is not very robust and could be misleading to people using my service.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
June 12, 2011, 05:27:35 PM
#72

it will be good create IRC-channel
  • for talks
  • for "websockets OFF for hour"
  • for bans in websockets (if exists)

at now
opened socket, sent handshake, received nothing - conn reset by peer
and I dont know what happened and how long it will last
kseistrup
Hero Member
*****
Offline Offline

Activity: 565


Unselfish actions pay back better


View Profile WWW
June 12, 2011, 05:30:14 PM
#73


  • Switching to the Secure WebSocket (wss://) would dramatically improve compatibility, especially for those behind a proxy

+1 for wss://

Cheers,

Klaus Alexander Seistrup
http://about.me/kseistrup
matt.collier
Member
**
Offline Offline

Activity: 105



View Profile
June 12, 2011, 10:32:56 PM
#74

Hello all, developing in Java here.

Aiming at creating an interface similar to Trade Station's Matrix: http://www.tradestation.com/strategy_testing/market_depth.shtm

I'm using Open Office Calc as the front end.  I've got the real-time data flowing into calc nicely, but I've run into the same sync issues everyone else is having.

As others have done, I'm bootstrapping my depth info with http://mtgox.com/code/data/getDepth.php, but before long, none of the volume information makes any sense.

I'll be happy to share my source with any interested parties.  For now I'm going to put this on the back burner until a solution to the sync issue has been implemented.

I'll put my vote in for a channel that publishes absolute volume information in addition to some effective bootstrapping method.
bitoption
Jr. Member
*
Offline Offline

Activity: 56


View Profile WWW
June 13, 2011, 05:17:55 AM
#75

I'm getting deafening silence from my python websockets client today. This is sad, since I use it to give pricing cues on bitoption.org; I've noticed some websockets-gox stuff is working right now, some is not.

Any indications as to what's what and why?

----** In Beta: The First Bitcoin Options Market ----**

Explanation and discussion: http://forum.bitcoin.org/index.php?topic=9611.0

API Developer Thread:
http://forum.bitcoin.org/index.php?topic=14194.0

-----------------------------------------------------------
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
June 13, 2011, 05:39:32 AM
#76

It would be nice to hear from MagicalTux about the progress of the back end rewrite and deployment. Also, a formal documentation of the WebSockets API is most welcome.

I'm still seeing the feed at the moment, but it stopped a little earlier today. Very odd.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
netxshare
Full Member
***
Offline Offline

Activity: 120


View Profile
June 13, 2011, 06:24:58 AM
#77

I am also having issues, my current connection to the websocket is showing data but any new connectionss fail to get anything more then a response header.

donations: 1CWddfhXagPoWgzUCs7oVHFjwBmqbAoJRr
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
June 13, 2011, 07:24:25 AM
#78

found site with fast check websockets (and wss too)
http://websocket.org/echo.html
browser must support websockets (or enabled it)

for now MtGox-WebSockets requires Sec-WebSocket-Key1 and Sec-WebSocket-Key2 and 8bytes-data

at now (UTC 19:27 13-jun) WS dont responds with 16bytes of hash

and at now (UTC 20:37 13-jun) no connection

at now (UTC 20:35 14-jun) System.Net.Sockets.SocketException: Connection refused
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
June 13, 2011, 07:55:22 PM
#79

I'm getting deafening silence from my python websockets client today. This is sad, since I use it to give pricing cues on bitoption.org; I've noticed some websockets-gox stuff is working right now, some is not.

Any indications as to what's what and why?

There have been 2 periods where no trades happened and the websocket was dead, apparently some mysql troubles. Maybe you stumbled into one of these.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
June 16, 2011, 06:43:52 PM
#80

today I saw MtGox Live for Android

and here is screen of my iPhone app (not in AppStore yet) - depth state
http://img27.imageshack.us/img27/1115/screenshot20110616at223.png

in first version will be depth state & trades only, no wallet & orders
kseistrup
Hero Member
*****
Offline Offline

Activity: 565


Unselfish actions pay back better


View Profile WWW
June 16, 2011, 07:15:52 PM
#81


today I saw MtGox Live for Android

I have a feeling that the app is polling Mt. Gox every 5 seconds, rather than using the websocket API.

Something that would be immensly cool is an android app with the same graphs as on MtGoxLive, using websockets.  AFAIK there are no browsers for android that supports websokcets, so I can't just load the web page.

Cheers,

Klaus Alexander Seistrup
http://about.me/kseistrup
ezl
Newbie
*
Offline Offline

Activity: 13


View Profile
June 17, 2011, 01:31:41 PM
#82

recommendation regarding the depth feed -- , send updates instead of diffs.

 the bandwidth is ~same, but by broadcasting updates, the penalty for missing a message only lasts as long as the interval until the next update for that price level. clients can compute diffs if they want.

it also makes it so that the listener can theoretically construct the full book just by listening to the updates instead of repolling the old api and hoping the messages are in sync
dev^
Newbie
*
Offline Offline

Activity: 28


View Profile
June 18, 2011, 01:49:12 AM
#83

Hi,

is it possible for upcoming version to rename the field "private" in anything other, which isn't a protected keyword in Java?  Smiley

Currently it isn't possible (at least for me) to read this field with any common JSON parser library, e.g. Google's Gson.
(of course, it would be possible to set this field's content via text parsing, but this isn't the right way to do so...)

> Something that would be immensly cool is an android app with the same graphs as on MtGoxLive, using websockets.

If someone can develop any nice looking GUI for Android devices, I could provide the complete Java part regarding the WebSockets - it seems to work quite fine.  Cheesy

Best regards
dev^
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
June 19, 2011, 06:46:31 PM
#84

at now
for determine connection lost (iPhone hasn't keepalive ops for socket) I do next:
if App dont recv any datas on channels over 6seconds it send:
Quote
"{"op":"ping"}"
if connection is ok than server responds to App:
Quote
{"message":"Unknown command","op":"remark","success":false}
if no any responds in next 6seconds (include other datas on channels) App decides that current connection lost and try to create new connection (with close old)

NB: "ping" dont send every 6seconds! only if at last 6seconds no datas (including "Unknown command")
otherwise App works but nothing shows over hour on broken connection (turnoff WAN at router). I dont known default values in iPhone keep-alive logic.

and my suggestion:
maybe add "good" response:
Quote
{"op":"pong"}
it dont requires any logic or DB-access. simple response for current connection
although its works well now
dds
Newbie
*
Offline Offline

Activity: 7



View Profile
June 28, 2011, 07:49:59 AM
#85

Looks like MtGox WebSockets broken after reopening - connection can be established but no data received.
Mark, are you going to fix it?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
June 28, 2011, 08:47:39 AM
#86

Looks like MtGox WebSockets broken after reopening - connection can be established but no data received.
Mark, are you going to fix it?

+1

also: I can't seem to be able to use the classic Trade API, either. Says something about a "bad token". Anyone know about that?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
kseistrup
Hero Member
*****
Offline Offline

Activity: 565


Unselfish actions pay back better


View Profile WWW
June 28, 2011, 08:59:20 AM
#87


I can't seem to be able to use the classic Trade API, either. Says something about a "bad token". Anyone know about that?

Look at the Trade API page on your account it says that it will be enabled soonish.

Cheers,

Klaus Alexander Seistrup
http://about.me/kseistrup
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
July 03, 2011, 04:33:39 PM
#88

good! websocket works for now!

some questions:

Quote
{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"private", "origin":"broadcast","private":"ticker", "ticker":{"avg":15.469053268,"buy":15.63,"high":15.689,"last":15.63001,"low":15.311,"sell":15.63001,"vol":15679}}
"avg" is average volume by trade?

Quote
{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"private","origin":"broadcast","private":"trade", "trade":{"amount":2,"amount_int":"200000000","date":1309710218,"item":"BTC","price":15.632,"price_currency":"USD","price_int":"1563200","tid":"1309710218345711","trade_type":"ask","type":"trade"}}
what mean "trade_type":"ask"? trade raised by ask order?

and where is depth channel? no datas for it. will it appear in the future?
dev^
Newbie
*
Offline Offline

Activity: 28


View Profile
July 03, 2011, 04:36:29 PM
#89

good! websocket works for now!

Yay! :-)

"avg" is average volume by trade?

I think it's the average price. (As I guess by it's value...)

what mean "trade_type":"ask"? trade raised by ask order?
and where is depth channel? no datas for it. will it appear in the future?
Looks like this is "the new" depth-channel...

Best regards,
dev^
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
July 03, 2011, 05:13:31 PM
#90

and where is depth channel? no datas for it. will it appear in the future?
Looks like this is "the new" depth-channel...
my English is not so good. where is "this "the new" depth-channel" ?
EDIT: on #mtgox they say me that "should come soon"


and I try to get key for personal channel via https://mtgox.com/code/getKey.php (as say in post 0). it returns
Quote
{"error":"Bad token"}
Huh
EDIT: maybe same (come soon)
kseistrup
Hero Member
*****
Offline Offline

Activity: 565


Unselfish actions pay back better


View Profile WWW
July 05, 2011, 06:13:27 AM
#91

My ws client has been trying to connect for an hour now, is the websocket down for me only?

Klaus Alexander Seistrup
http://about.me/kseistrup
dev^
Newbie
*
Offline Offline

Activity: 28


View Profile
July 05, 2011, 06:57:53 AM
#92

My ws client has been trying to connect for an hour now, is the websocket down for me only?

It's down for everyone.

Code:
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
(...)

MtGox should really think about some QA or at least a 2nd developer in it's team...
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
July 10, 2011, 10:08:40 AM
#93

there's a wiki now: https://en.bitcoin.it/wiki/MtGox/API#Websocket_API

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
vector76
Member
**
Offline Offline

Activity: 70


View Profile
July 27, 2011, 12:51:51 AM
#94

I guess since there is not a good way to synchronize the market depth websocket with the getDepth.php, I could fetch getDepth multiple times and watch the websocket for deltas and try to correlate them.

I'd have to agree with ezl's suggestion of sending the total volume instead of the deltas.  Then there would be no synchronization problem.

Are people still seeing problems with the depth websocket or have those issues been fixed?  I guess I'll fetch getDepth every 5 or ten minutes just in case..
divebubble
Newbie
*
Offline Offline

Activity: 7


View Profile
July 28, 2011, 10:30:59 PM
#95


Reading the different channels, updating data all works perfect... but -

how can i connect my account to get my trading information?

sending
{"op":"mtgox.subscribe","key":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"}

will always result in

{"message":"Invalid MTGOX key","op":"remark","success":false}


XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX is the API Key from the https://mtgox.com/users/settings page.
Or are i'm compleatly wrong?


Thanks.
throughput
Full Member
***
Offline Offline

Activity: 158


View Profile
August 03, 2011, 09:02:02 AM
#96

Websocket API is cool for receiving trade updates! Thanks!

But is it possible to receive updates on my orders anyhow?
If that is implemented, then I don't get the meaning of the "(partial key)" in
Code:
{
  "channel":"(partial key)",
  "op":"private",
  "order_upd":{
    "amount":1000,
    "darkStatus":0,
    "date":1302836027,
    "oid":"(oid)",
    "price":0.9899,"status":1
  },
  "origin":"broadcast",
  "private":"order_upd"
}
(example from https://en.bitcoin.it/wiki/MtGox/API#Websocket_API )

I don't understand how should I implement the authentication to subscribe to my private channel.

Please, can anybody provide more details on:

 1. Does it work at the moment?
 2. How the authentication is implemented?

Also, it would be nice to be able to request the full depth to receive them as depth messages.
Fireball
Hero Member
*****
Offline Offline

Activity: 673


View Profile WWW
August 06, 2011, 06:18:26 PM
#97

You have a small bug in the reply header:
Code:
HTTP/1.1 101 WebSocket Protocol Handshake
should be the first line, and in your case it's
Code:
HTTP/1.1 101 Web Socket Protocol Handshake
(yes, the additional space between Web and Socket). Nothing major, however if some software validates the header, it won't be recognized as a valid one. Example of such real-world failing app: websocket-sharp. Other are probably just failing silently too.

Margin trading platform OrderBook.net (ICBIT): https://orderbook.net
Follow us in Twitter: https://twitter.com/orderbooknet
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
August 27, 2011, 02:13:50 PM
#98

It happens very often that there are consecutive "depth" messages, where one comes immediately after another and they both have the same amount/price and the opposite sign. So effectively they do nothing, except causing confusion and an unnecessary traffic.

For instance:
Quote
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"9.15417","price_int":"915417","type":1,"type_str":"ask","volume":"-0.09283","volume_int":"-9283000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"9.15417","price_int":"915417","type":1,"type_str":"ask","volume":"0.09283","volume_int":"9283000"},"op":"private","origin":"broadcast","private":"depth"}
...
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"-0.2","volume_int":"-20000000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"0.2","volume_int":"20000000"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86","price_int":"886000","type":2,"type_str":"bid","volume":"-0.2","volume_int":"-20000000"},"op":"private","origin":"broadcast","private":"depth"}
...
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"-0.79287883","volume_int":"-79287883"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"0.79287883","volume_int":"79287883"},"op":"private","origin":"broadcast","private":"depth"}

It seems like some glitch and it would be nice if could be fixed one day.

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
August 30, 2011, 09:06:20 PM
#99

It happens very often that there are consecutive "depth" messages, where one comes immediately after another and they both have the same amount/price and the opposite sign. So effectively they do nothing, except causing confusion and an unnecessary traffic.

For instance:
Quote
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"-0.79287883","volume_int":"-79287883"},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"currency":"USD","item":"BTC","price":"8.86852","price_int":"886852","type":2,"type_str":"bid","volume":"0.79287883","volume_int":"79287883"},"op":"private","origin":"broadcast","private":"depth"}

It seems like some glitch and it would be nice if could be fixed one day.

imho, all changes must be. human's or robot's..
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
August 30, 2011, 09:14:07 PM

imho, all changes must be. human's or robot's..
agree. but these dont look like robot changes, do they? why would a robot do something like that? the only thing i can think of: to DoS the server Smiley

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
September 20, 2011, 09:52:43 AM

The WebSocket API does not work with the latest Chrome. Sad

Chrome dev claims that the problem is at mtgox side.
Quote
It seems that websocket.mtgox.com is sending out a frame in old format (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-00). Chrome 14 uses new version WebSocket protocol (http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-08). We've dropped old protocol and are asking servers to adopt new one. mtgox is replying new version handshake correctly. Only new frame support looks broken.
http://code.google.com/p/chromium/issues/detail?id=97026

Please fix.

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
AV
Hero Member
*****
Offline Offline

Activity: 910



View Profile
September 20, 2011, 12:56:57 PM

Hi, MagicalTux !

The WebSocket API really does not work with the latest (14.0.835.163) Chrome.


For example http://bitcoin.clarkmoody.com/
WebSocket Closed after 2 second.

Help ! Help ! Help !

Thanks.
speeder
Hero Member
*****
Offline Offline

Activity: 630



View Profile
September 20, 2011, 12:58:38 PM

Yay! Correct thread to complain about that!

Indeed, me too want clarkmoody on Chrome 14!

(thankfully at work Chrome is still stuck on 13)

               ▄▄▄██
           ▄███████▀
▄▄▄██████▄████████▀          ▄▄▄▄▄▄▄
 ▀███████████████         ▄███████████▄▄
   ▀▀███████████       ▄███▀▀       ▀▀███▄
       ▀▀██████       ▄██▀ ▄███▄ ▄███▄ ▀███▄
           ▀████▄   ▄██▀  █████████████  ███▄
            ▀████▄▄████   ▀███████████▀   ███
             ██████████     ▀███████▀     ███
             ██████████       ▀███▀      ████
              ▀█████████▄             ▄▄███▀
                ▀▀▀█████████▄▄▄▄▄▄▄▄█████▀
                      ▀▀▀▀██████████▀▀▀
[]
bzzard
Sr. Member
****
Offline Offline

Activity: 398


View Profile WWW
September 22, 2011, 11:42:50 PM

yeah, this should work?
I know that ws://mtgoxlive.com:3456/channel works Tongue

MtGox! Anyone there?

BTCCharts.com - still for free!
paraipan
Legendary
*
Offline Offline

Activity: 924


Firstbits: 1pirata


View Profile WWW
September 24, 2011, 05:50:28 PM

sup MagicalTux, any plans to update websockets to draft-10 any time soon ? Thanks

BTCitcoin: An Idea Worth Saving - Q&A with bitcoins on rugatu.com - Check my rep
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
September 25, 2011, 04:28:12 PM

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

How about these and the wss://?
Its been months that Mark didnt post here now..

And, general question:
I chose this API as a starting project to learn ruby.  Maybe a bit (too) ambitious, but would at least be a useable project, if it ever gets finished..
Are there real advantages of this "new" websocket API over the old HTTP API except that I dont have to poll any more and have instant updates (instead of every 10 secs)? I would at least not have the syncing issues?

With the probably still unsolved issues I hesitate to invest much more time into this - being a novice I still am trying to actually post to the websocket.. *rolleyes*

Thanks,

Ente
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
October 10, 2011, 07:49:55 PM

I found the part of the draft where Chrome 14 is failing:

http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10#section-4.2

Apparently, MtGox is setting the RSV2 bit in the responses to the client. Thus, according to the spec, Chrome is required to FAIL the connection.

Using a packet sniffer, I can see the subscribed responses from the WebSocket after connecting, but the message never gets through and Chrome fails and closes the connection.

This is such a simple fix...

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
BkkCoins
Hero Member
*****
Offline Offline

Activity: 784


firstbits:1MinerQ


View Profile WWW
October 12, 2011, 06:43:21 AM

Does the "own" channel work at all?

I can't figure out how to get data about my own trades/orders. I've dumped all the data streamed and there is no data not matching the default channels. I tried subscribing using a key of my own account. That doesn't work either. It seems like this just doesn't work yet.

It would be really useful to see when your own orders/trades occur. If this does work then is there a web page somewhere that says more than the one line on the wiki?

*** Edit:

Another thing I noticed now is multi-currency trades coming thru. I don't quite understand these:

19:07:57 BUY  22.81 @ $2.63192 GBP
19:07:57 BUY  22.81 @ $25.6 CNY

I see British Pounds and Chinese Yuan. But why twice and how does this work - is someone buying with Pounds and settling in Yuan or something? I realize that this must be the same trade since the  GBPCNY exchange rate is now about 10.33. Just not sure how I can handle this as info if the USD isn't indicated.

Thx.

qxzn
Hero Member
*****
Offline Offline

Activity: 603



View Profile
October 12, 2011, 05:06:44 PM

I have not been able to get the "own" channel to work either. If anyone has, please share how!
Cluster2k
Legendary
*
Offline Offline

Activity: 1559


View Profile
October 16, 2011, 12:41:20 PM

Subscribing to my own channel doesn't work either.  I'm using VB and tried sending ANSI strings to MtGox in various formats, but as soon as I do I lose the TCP connection.  Sending the handshake right after connecting exactly the same way works fine however.

I have tried:

{'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'op': 'mtgox.subscribe'}
{'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', 'op': 'subscribe'}
{'op': 'mtgox.subscribe', 'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'}
{'op': 'subscribe', 'key': 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'}

Sending nonsensical data "eg. 'testing 123' to the connection causes a disconnection as well.  Is this standard behaviour if MtGox doesn't recognise the message?

I assume the key is my key listed in my MtGox API section.  I have successfully used the key and secret to retrieve account data using a sample PHP script.

EDIT: Just tried using plain old telnet to connect to MtGox.  Works fine, just like this example:
http://dl.dropbox.com/u/24900303/screenshot/2011/04/20110415_websocket_telnet_example.png

But as soon as I try to send any subsequent data to the connection I lose it.  The wiki entry at https://en.bitcoin.it/wiki/MtGox/API is non descriptive. 

"Additionally each user has a "own" channel which streams informations about orders (new order, deleted order, etc) and trades only the user's trades). "

There is no description on how someone can subscribe to their own channel.

Do not send bitcoins to me: 16b8s7pBJ9rUmsExNW25qD5VUqVqRPZuXu
100% solar powered bitcoin generation
qxzn
Hero Member
*****
Offline Offline

Activity: 603



View Profile
October 16, 2011, 05:39:19 PM

You should not be getting a disconnect just for sending an invalid message type. You should be getting a json message back that says something about how your command was unrecognized. I would double check the bytes you're sending. As a test, you can try to unsubscribe to one of the feeds. That works, so you should be able to get it to work.
Cluster2k
Legendary
*
Offline Offline

Activity: 1559


View Profile
October 16, 2011, 10:56:54 PM

You should not be getting a disconnect just for sending an invalid message type. You should be getting a json message back that says something about how your command was unrecognized. I would double check the bytes you're sending. As a test, you can try to unsubscribe to one of the feeds. That works, so you should be able to get it to work.

It seems I can't send anything to MtGox after the initial handshake without getting disconnected.

I've tried in telnet too (MS Windows telnet client).  Surely it should work?

telnet websocket.mtgox.com 80
Sent the following text:
GET /mtgox HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: websocket.mtgox.com
Origin: null

After pressing enter the second time I start to get the stream of data.

But when I try to send something like "{'op': 'mtgox.unsubscribe', 'key': 24e67e0d-1cad-4cc0-9e7a-f8523ef460fe}" the connection is lost.  If I send anything at all the connection dies.

I don't get it.  

EDIT: Just tried it again with Putty on Win7.  Type of connection: raw.  Same result.  I can get the streaming data, but as soon as I try to send anything back to the server on the same connection it disconnects.

Do not send bitcoins to me: 16b8s7pBJ9rUmsExNW25qD5VUqVqRPZuXu
100% solar powered bitcoin generation
qxzn
Hero Member
*****
Offline Offline

Activity: 603



View Profile
October 17, 2011, 01:12:46 AM

You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198
Cluster2k
Legendary
*
Offline Offline

Activity: 1559


View Profile
October 17, 2011, 01:26:20 AM

You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

Thanks for that.  I'll try putting \x00 before my message and \xFF after and see how that goes. 

Do not send bitcoins to me: 16b8s7pBJ9rUmsExNW25qD5VUqVqRPZuXu
100% solar powered bitcoin generation
Cluster2k
Legendary
*
Offline Offline

Activity: 1559


View Profile
October 17, 2011, 03:38:46 AM

You need to include certain bytes before and after every json message. I think no newlines. This would be hard to do in telnet. I suggest taking a hard look at this post which has haskell code that I think would still work:

https://bitcointalk.org/index.php?topic=5855.msg87198#msg87198

That works fine.  I wasn't sending the \x00 and \xFF at the beginning and end of each message.

I can successfully unsubscribe from the depth feed using:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"unsubscribe"}

However when I try to resubscribe I get the following error:
{"message":"Unknown command","op":"remark","success":false}

I have tried all sorts of combinations of subscribe messages, but none seems to work.  I also can't subscribe to my own channel.

The following all fail:
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"subscribe"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"mtgox.subscribe"}
{"op":"subscribe","channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"mtgox.subscribe","channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"subscribe","24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}
{"op":"mtgox.subscribe","24e67e0d-1cad-4cc0-9e7a-f8523ef460fe"}

I looked at the haskell code and tried that subscription message first, then tried various other combinations.

hPutFrame h $ "{\"op\":\"mtgox.subscribe\"," ++ key ++ "}"

Do not send bitcoins to me: 16b8s7pBJ9rUmsExNW25qD5VUqVqRPZuXu
100% solar powered bitcoin generation
Cluster2k
Legendary
*
Offline Offline

Activity: 1559


View Profile
October 17, 2011, 07:33:09 AM

I have just been informed on #mtgox IRC that websockets are read-only.  Have been for a number of weeks.  Subscriptions don't work and it's not possible to view your own executed trades.

It's possible get around this limitation, but it really makes the system slow and a bit cumbersome to use.  It would be so much more convenient to get instant confirmation of your own trades.

Do not send bitcoins to me: 16b8s7pBJ9rUmsExNW25qD5VUqVqRPZuXu
100% solar powered bitcoin generation
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
October 20, 2011, 04:12:46 PM

I would like to see depth messages for all of the currencies broadcast out over the WebSocket, as I do not think it's happening right now.

Can anyone confirm that there are only USD depth messages coming across the socket?

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
October 21, 2011, 01:23:06 PM

I would like to see depth messages for all of the currencies broadcast out over the WebSocket, as I do not think it's happening right now.

Can anyone confirm that there are only USD depth messages coming across the socket?

Looks like it's true. I am filtering for "depth_msg['currency'] == 'USD'", but put a debug log line for the other case. It doesn't trigger.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
October 26, 2011, 03:29:34 PM

I am having problems with Websockets and Ruby in general, maybe someone of you is kind enough to help out?
I am beginner with Ruby and chose this as a learning-by-doing project.

Using examples from the web, I am still trying to connect to the websocket (works) and sending strings to it (handshake and channel subscription/exiting):

Code:
require 'rubygems' 'eventmachine' 'em-http-request'
 
EventMachine.run {
  verbindung = EventMachine::HttpRequest.new("http://websocket.mtgox.com").get :timeout => 0

  verbindung.errback { puts "oops" }
  verbindung.callback   {
    puts "WebSocket connected!"
                 
verbindung.send("GET /mtgox HTTP/1.1\n", 0)
  }
}

Error, my Ruby doesnt recognize "GET /mtgox HTTP/1.1\n" as a string to be "send" :

Code:
websocket.rb:19:in `send': undefined method `GET /mtgox HTTP/1.1 (NoMethodError)
' for #<EventMachine::HttpClient:0x7f9f6d22e8b8>

I tried "print", "send", "write", "<<", still didnt get it to work. I am lost, grateful for any hint!

Ente
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
October 28, 2011, 09:13:22 PM

Anyone else getting timeouts connecting to ws://websocket.mtgox.com/mtgox? I am right now...
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
October 28, 2011, 09:14:13 PM

the server has been dead since yesterday.

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
Cluster2k
Legendary
*
Offline Offline

Activity: 1559


View Profile
October 30, 2011, 11:00:19 PM

Websockets have been dead for about a week.  Nothing on MtGox's support page about it.  Does MtGox even know, or do they care?  It seems not.

Do not send bitcoins to me: 16b8s7pBJ9rUmsExNW25qD5VUqVqRPZuXu
100% solar powered bitcoin generation
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
October 31, 2011, 01:10:13 AM

Websockets have been dead for about a week.  Nothing on MtGox's support page about it.  Does MtGox even know, or do they care?  It seems not.

MagicalTux knows, has been saying something routes not being up about 3 or 4 days ago.

Does he care? I'm not sure.

I think the websocket stuff is pretty important.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
October 31, 2011, 04:29:49 PM

http://mtgoxlive.com/ is also suffering from this failure.
The data is shows doesn't seem to be in a real time anymore.
And this service had supposedly been acquired by MtGox.
So he should care Smiley

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
BTCurious
Hero Member
*****
Offline Offline

Activity: 714


^SEM img of Si wafer edge, scanned 2012-3-12.


View Profile
October 31, 2011, 04:44:15 PM

http://mtgoxlive.com/ is also suffering from this failure.
The data is shows doesn't seem to be in a real time anymore.
And this service had supposedly been acquired by MtGox.
So he should care Smiley
I'm not sure that's true. If I compare it to the fastest monitor bot I know, #bitcoin-RT in freenode, then the number seems to be behind ~10 seconds. But I don't remember this ever being any different.

piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
October 31, 2011, 04:55:37 PM

Before you could see new transactions popping up on the right side, in a real time.
And the green price line was also updated in a real time - now the last point of it is always a couple of minutes behind.

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
BTCurious
Hero Member
*****
Offline Offline

Activity: 714


^SEM img of Si wafer edge, scanned 2012-3-12.


View Profile
October 31, 2011, 05:04:29 PM

Before you could see new transactions popping up on the right side, in a real time.
And the green price line was also updated in a real time - now the last point of it is always a couple of minutes behind.
Oh hey, you could be right. It did work via the websocket before. Maybe it works via the http API now?

Edit: Websocket is back. And yeah, the transactions in real time too.

runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
October 31, 2011, 11:42:57 PM

Yay, websocket is back. Thanks Mt. Gox.
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
November 01, 2011, 11:46:07 AM

Yay, websocket is back. Thanks Mt. Gox.
I saw it back for a moment... but it doesn't work now, does it?

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
November 01, 2011, 01:40:23 PM

Yay, websocket is back. Thanks Mt. Gox.
I saw it back for a moment... but it doesn't work now, does it?

It's working for me in my own client. Thanks MagicalTux!

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
November 01, 2011, 01:46:48 PM

It's working for me in my own client.
OK - thanks for the info.
This made me thinking... and it seems that I upgraded my websocket libarary to a new version.
It's OK when I go back to the old version. Its probably the same problem as with a new Chrome.

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
November 01, 2011, 01:48:18 PM

It's working for me in my own client.
OK - thanks for the info.
This made me thinking... and it seems that I upgraded my websocket libarary to a new version.
It's OK when I go back to the old version. Its probably the same problem as with a new Chrome.

Thanks for that hint. I'll keep that in mind, cause I'll probably switch websocket impl at some point (currently using some hacked-up thingy someone gave me).

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
November 01, 2011, 02:07:52 PM

Yay, websocket is back. Thanks Mt. Gox.
I saw it back for a moment... but it doesn't work now, does it?
Works for me right now as well.
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
November 19, 2011, 12:40:08 AM

Anyone succeed with new MtGox's socket.io interface? I was succesfull with socket.io handshake, server returned "12906784121696250629:15:25:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling". But then I open WS connection to ws://socketio.mtgox.com:80/socket.io/1/websocket/12906784121696250629 and nothing happen, client don't receive any data. Am I doing anything wrong?

molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
November 20, 2011, 12:52:37 AM

Anyone succeed with new MtGox's socket.io interface? I was succesfull with socket.io handshake, server returned "12906784121696250629:15:25:websocket,flashsocket,htmlfile,xhr-polling,jsonp-polling". But then I open WS connection to ws://socketio.mtgox.com:80/mtgox/socket.io/1/websocket/12906784121696250629 and nothing happen, client don't receive any data. Am I doing anything wrong?

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
November 20, 2011, 01:07:25 AM

Thanks for info. Is following websocket URL correct?

ws://socketio.mtgox.com:80/socket.io/1/websocket/1290678412169625062

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.


jothan
Full Member
***
Offline Offline

Activity: 184


Feel the coffee, be the coffee.


View Profile
November 23, 2011, 04:37:50 AM

Thanks for info. Is following websocket URL correct?

ws://socketio.mtgox.com:80/socket.io/1/websocket/1290678412169625062

I did succeed implementing mtgox socket.io client in my trading client. I used the websocket implementation I used before, coded the handshake.

I remember one time where the socket.io would work, websocket would even connect, but no data being delivered. Maybe this problem just reappeared.

It's working as expected for me now.


With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Bitcoin: the only currency you can store directly into your brain.

What this planet needs is a good 0.0005 BTC US nickel.
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
November 23, 2011, 03:17:15 PM

With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Yes, you received socket.io handshake response, it's fine. socket.io != websocket.

Quote
Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Does it work for you? I dropped idea of get it working after playing with it for hours. Lacking documentation, multiple versions of specifications, total mess.

Then I implemented bitcoincharts socket interface in 15 minutes...

jothan
Full Member
***
Offline Offline

Activity: 184


Feel the coffee, be the coffee.


View Profile
November 23, 2011, 03:36:28 PM

With the URL above, I get a weird reply from the server "7:::1+0", using the supposed official URL, wss://socketio.mtgox.com/mtgox, the server just closes the connection after the SSL handshake.

Very weird.

Yes, you received socket.io handshake response, it's fine. socket.io != websocket.

Quote
Edit: I just figured out that using a WebSockets library is not enough, I have just figured out that socket.io was a whole other layer on top of WebSockets.

Does it work for you? I dropped idea of get it working after playing with it for hours. Lacking documentation, multiple versions of specifications, total mess.

Then I implemented bitcoincharts socket interface in 15 minutes...

I can now connect fine, but I cannot subscribe to any channel, I get "invalid request" or "invalid command", so I cannot do anything useful yet. I'm gonna try interfacing bitcoincharts tonight.

Bitcoin: the only currency you can store directly into your brain.

What this planet needs is a good 0.0005 BTC US nickel.
jothan
Full Member
***
Offline Offline

Activity: 184


Feel the coffee, be the coffee.


View Profile
November 23, 2011, 10:55:13 PM

Could someone comment on the following points ?

  • Is the stream API service currently in a usable state ? (I cannot determine this since I cannot subscribe to any channel).
  • What is the proper way to subscribe to a channel ?

Given answers to this, I would be willing to update the wiki with the up-to-date information to clear up the usage of this API.

Heck, i'll give 1 BTC to whomever can enable me to stream prices directly from Mt. Gox with this API.

Bitcoin: the only currency you can store directly into your brain.

What this planet needs is a good 0.0005 BTC US nickel.
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
November 23, 2011, 11:02:13 PM

Same status here, lack of any documentation is pretty irritating.

molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
November 24, 2011, 02:58:37 PM

Same status here, lack of any documentation is pretty irritating.

magicaltux has some php sample code that was helpfull for me to code socket.io handshake in python.

(it's still got some debugging output and heartbeat lost detection is not implemented, so on connection drop it doesn't auto-reconnect). oh, and it only does websocket, no other transport.

Quote from: socketio.py
from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket import WebSocket


class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    data = urllib.urlencode({})
    req = urllib2.Request('https://' + S.url + "/1", data)
    response = urllib2.urlopen(req)
    r = response.read().split(':')
    S.heartbeat_interval = int(r[1])
    print 'heartbeat: ', S.heartbeat_interval
    if 'websocket' in r[3].split(','):
      print "good: transport 'websocket' supported by socket.io server ", S.url
      S.id = r[0]
      print "id: ", S.id

    S.thread = Thread(target = S.thread_func)
    S.thread.setDaemon(True)
    S.thread.start()

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
      
  def keepalive_func(S):
    while S.run:
      time.sleep(S.heartbeat_interval)
      S.ws.send('2::');
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if True:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)


I got that websocket implementation from giel (on irc) ("from websocket import WebSocket"), here it is (put in file websocket/__init__.py):

  http://pastebin.com/jrxachSF

You can start using

  #> python socketio.py

it should dump the messages to stdout

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
November 24, 2011, 03:04:42 PM

Quote
    S.ws.send('1::/mtgox')

*headdesk*
* slush bashing all developers writing incomplete documentations and expecting that some things are so obvious they don't need to be in doc.

molecular, you're my savior. What's your donation address?

molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
November 24, 2011, 03:07:21 PM

Quote
   S.ws.send('1::/mtgox')

*headdesk*
* slush bashing all developers writing incomplete documentations and expecting that some things are so obvious they don't need to be in doc.

molecular, you're my savior. What's your donation address?

cool to have been of help, especiall to you, slush.

almost didn't post, but so many complained so I did... instant karma Wink

1JX3YkScnu2RBHjUBZTLB6qBgmGeUWxANL

EDIT: slush, are you coming to Prague? I'm there on saturday and maybe sunday, maybe I recognize you Wink

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
slush
Legendary
*
Offline Offline

Activity: 1372



View Profile WWW
November 24, 2011, 03:35:47 PM

No, I'm not coming to Prague...
(...dramatic drums...)
...because I already am in Prague!.

Would be nice to meet you, however I'm myself curious how I'll be able to communicate with my "fluent" english :-).

molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
November 24, 2011, 03:41:29 PM

No, I'm not coming to Prague...
(...dramatic drums...)
...because I already am in Prague!.

Would be nice to meet you, however I'm myself curious how I'll be able to communicate with my "fluent" english :-).

Nice. Hoping to meet you. Thanks for the generous token of appreciation you sent me Wink

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
jothan
Full Member
***
Offline Offline

Activity: 184


Feel the coffee, be the coffee.


View Profile
November 24, 2011, 05:03:43 PM

Quote
   S.ws.send('1::/mtgox')

almost didn't post, but so many complained so I did... instant karma Wink

1JX3YkScnu2RBHjUBZTLB6qBgmGeUWxANL

EDIT: slush, are you coming to Prague? I'm there on saturday and maybe sunday, maybe I recognize you Wink

+1 BTC, Thanks !

Bitcoin: the only currency you can store directly into your brain.

What this planet needs is a good 0.0005 BTC US nickel.
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
December 11, 2011, 05:33:59 PM

I created project for .NET socket.io + WebSockets at codeplex.com
http://socketiowebsockets.codeplex.com/
for now it can hold ssl-connection with mtgox (no parsing json and etc)

my old ideas (in this topic) died and raised again Smiley
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
January 07, 2012, 01:03:58 PM

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Instead of using socket.io (which I cant get to work) I could as well just connect to the underlying websocket just as before, not? I will not have fallback to longpoll etc in case the websocket dies and the socket.io still works. But besides that, my ticker should happily work with just websockets for the future, not?

Is there a new websocket daemon running now, for socket.io, with a different adress?

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?


Ente
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
January 07, 2012, 01:08:29 PM

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".
Even though it's "deprecated" it's still much more reliable than the socket.io interface.
So please don't turn it off!

The only problem is that it doesn't work with the latest Chrome and other late WebSocket libs... and nobody wants to fix it. Sad
That's probably why they labeled it "deprecated".

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
January 08, 2012, 01:26:34 PM

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".

Instead of using socket.io (which I cant get to work) I could as well just connect to the underlying websocket just as before, not? I will not have fallback to longpoll etc in case the websocket dies and the socket.io still works. But besides that, my ticker should happily work with just websockets for the future, not?

Is there a new websocket daemon running now, for socket.io, with a different adress?

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?


Ente

the way it works: you connect to socket.io server and it will give you a unique ID which will be part of the url you can subsequently use to connect to websocket. socket.io is just a "negotiation protocol".

look at my python code here: https://bitcointalk.org/index.php?topic=5855.msg629600#msg629600

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
January 08, 2012, 01:28:14 PM

..I try to get a simple python client running for days now. When I finally managed to connect via websockets, the first thing I read was "deprecated, use socket.io instead".
Even though it's "deprecated" it's still much more reliable than the socket.io interface.
So please don't turn it off!

The only problem is that it doesn't work with the latest Chrome and other late WebSocket libs... and nobody wants to fix it. Sad
That's probably why they labeled it "deprecated".

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.

check out sample implementation: https://bitcointalk.org/index.php?topic=5855.msg629600#msg629600

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
January 08, 2012, 01:37:07 PM

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.
So you see the new "indirect connection" working reliably?

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
January 08, 2012, 04:02:31 PM

"direct connection" to old websocket url is deprecated. you should now use socket.io to get an ID which will be part of a websocket url you can use.
So you see the new "indirect connection" working reliably?


no, not reliable since roughly a week, lots of outages on the websocket part, pretty stable many weeks before that. havent seen the socket.io part fail yet.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
bzzard
Sr. Member
****
Offline Offline

Activity: 398


View Profile WWW
January 08, 2012, 04:09:51 PM

havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

BTCCharts.com - still for free!
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
January 08, 2012, 04:12:45 PM

havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

socket.io is just a tool to negotiate a transport layer and keepalive. one of the possible transports is websocket. this means you usually use socket.io PLUS websocket (at least that's how I do it).

I'm guessing a connected websocket with no data flowing means some problem in magicaltux' backend.

any insights appreciated.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
January 09, 2012, 07:56:18 PM

request to "https://socketio.mtgox.com/socket.io/1" doesnt respond (net::ERR_CONNECTION_REFUSED)
Is URL valid?
something happened.. but WHAT happened?..

UPD1:
P.S.
I wrote to Mt.Gox support issue: open some technical twitter channel with info: what? when? how long?
IRC-channel is just flame - no info, no logs, no history, no search..

UPD2 (2:28 AM):
its working now
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
January 11, 2012, 03:06:49 AM

request to "https://socketio.mtgox.com/socket.io/1" doesnt respond (net::ERR_CONNECTION_REFUSED)
Is URL valid?
something happened.. but WHAT happened?..

UPD1:
P.S.
I wrote to Mt.Gox support issue: open some technical twitter channel with info: what? when? how long?
IRC-channel is just flame - no info, no logs, no history, no search..

UPD2 (2:28 AM):
its working now

yeah, I see this, too from time to time. also a lot of times the connection setup works fine, but there's just no data flow. mtgox datastreaming is on and off on most levels as far as I can tell.

supposedly socket.io is some typical bay-area shit and there's a big rewrite pull request pending for a while now that could fix the issue mtgox is suffering from.

my latest "kind-of-official" info is that the socket.io server is now restarted every 2 hours to mitigate effects of this (and potentially other) issue(s).

I for one am preparing to survive using api requests (switched form api/0 to api/1)

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
January 11, 2012, 09:53:33 AM

I see data
Quote
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","op":"private","origin":"broadcast","private":"depth","depth":{"price":"5.80025","type":2,"type_str":"bid","volume":"0","price_int":"580025","volume_int":"0","item":"BTC","currency":"USD","now":"1326266472912980","total_volume_int":"19811387"}}    
why exists messages with volume=0 ?
is this rounding errors in MtGox's guts or users posts such orders to exchange ?

+++ existence of "total_volume_int" - is wonderful
rwcp97
Newbie
*
Offline Offline

Activity: 13


View Profile
January 11, 2012, 10:20:27 AM

havent seen the socket.io part fail yet.

How come you say such thing when such sites as:
http://www.btccharts.com/
http://bitcoin.clarkmoody.com/

have a connection but there's no data received, any explanation for this?

I have the same problem using the websocket from my C++/Qt-bot - sometimes, usually after some hours working correctly, there is no more data coming in through the socket (but the connection isn't closed by the remote host).
So I check for incoming data and if there isn't anything for 3 minutes, I close the connection and reconnect. Afterwards it's working without problems again. Strange anyway  Huh
notme
Legendary
*
Offline Offline

Activity: 1820


View Profile
January 13, 2012, 01:54:06 AM

I'm trying to connect via the SocketIO Ruby gem.  When I try to subscribe to the trades channel the following string is passed to the server: ""\x003:::{\"channel\":\"dbf1dee9-4f2e-4a08-8cb7-748919a71b21\",\"op\":\"subscribe\"}\xFF".  The server response with a json message: {"op":"remark","success":false,"message":"Unknown command","debug":{"op":"client","uuid":"5a1f3b8f-1e03-462b-bd38-e463bb39ff2e","data":{"channel":"dbf1dee9-4f2e-4a08-8cb7-748919a71b21","op":"subscribe"}}}

The "data" field seems to be the message I intended to submit, but the subscribe fails with "Unknown command".  Any help would be appreciated.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
January 13, 2012, 09:37:48 PM

The "data" field seems to be the message I intended to submit, but the subscribe fails with "Unknown command".  Any help would be appreciated.
my subscribtions (raw socket.io messages over websocket - no explicit frames borders) are:
   "4:::{"op":"mtgox.subscribe","type":"trades"}"
   "4:::{"op":"mtgox.subscribe","type":"ticker"}"
   "4:::{"op":"mtgox.subscribe","type":"depth"}"
notme
Legendary
*
Offline Offline

Activity: 1820


View Profile
January 13, 2012, 11:43:05 PM

Thanks... I changed channel to type and subscribe to mtgox.subscribe and now it works.

https://www.bitcoin.org/bitcoin.pdf
While no idea is perfect, some ideas are useful.
12jh3odyAAaR2XedPKZNCR4X4sebuotQzN
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
January 20, 2012, 04:44:26 PM

Is it possible to query my open trades and/or cancel an order or at least all orders? With the websocket API, that is?
Didnt see any hint that this is possible..

But then I dont remember to have ever seen the commands to post orders via websocket neither?

Is the websocket for "reading" only, like making graphs from the data?

edit:
So, get data from the websocket, and exectue/cancel orders via "API version 1" then?

Ente
rwcp97
Newbie
*
Offline Offline

Activity: 13


View Profile
January 21, 2012, 08:57:16 AM

Is it possible to query my open trades and/or cancel an order or at least all orders? With the websocket API, that is?
Didnt see any hint that this is possible..

But then I dont remember to have ever seen the commands to post orders via websocket neither?

Is the websocket for "reading" only, like making graphs from the data?

edit:
So, get data from the websocket, and exectue/cancel orders via "API version 1" then?

Ente

As far as I know there is no possibility to do orders over the websocket... Would be pretty dangerous too, as it isn't SSL encrypted.
But you can subscribe to your "private channel" to be informed about your orders. Use getKey.php to get its name.
Fireball
Hero Member
*****
Offline Offline

Activity: 673


View Profile WWW
January 21, 2012, 09:53:14 AM

As far as I know there is no possibility to do orders over the websocket... Would be pretty dangerous too, as it isn't SSL encrypted.
But you can subscribe to your "private channel" to be informed about your orders. Use getKey.php to get its name.

socket.io supports secure connections (https, wss transports). That's what I use in ICBIT trading platform, so I would be quite surprised if that's dangerous Wink

Margin trading platform OrderBook.net (ICBIT): https://orderbook.net
Follow us in Twitter: https://twitter.com/orderbooknet
rwcp97
Newbie
*
Offline Offline

Activity: 13


View Profile
January 21, 2012, 10:07:08 AM

socket.io supports secure connections (https, wss transports). That's what I use in ICBIT trading platform, so I would be quite surprised if that's dangerous Wink

ok, I wasn't aware of that - I just connect to websocket.mtgox.com at port 80.
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
January 21, 2012, 10:10:12 AM

Thank you for the hints, rwcp and Fireball!

Ente
osmosis
Sr. Member
****
Offline Offline

Activity: 301



View Profile
January 25, 2012, 02:14:38 AM

 #> python socketio.py
it should dump the messages to stdout

Thanks molecular.
is4tomj
Jr. Member
*
Offline Offline

Activity: 48



View Profile WWW
January 25, 2012, 03:04:54 AM

I'm using the streaming feed on my website.  If anyone needs a working example feel free to take a look. (http://BitcoinTitan.com)

http://BitcoinTitan.com - Visualizations of the Bitcoin Markets and Exchanges
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
January 25, 2012, 10:03:34 AM

Let me say here that the socket.io/websocket feed has been rather stable in late times. I'm hoping this has nothing to do with the fact that trading volume is declining, but is due to Magicaltux making things better. In that case: thanks, Magicaltux and/or mtgox team.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
January 25, 2012, 01:14:54 PM

I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
osmosis
Sr. Member
****
Offline Offline

Activity: 301



View Profile
January 25, 2012, 01:32:01 PM

I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Works for me. Sounds like an issue on your side.
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
January 25, 2012, 01:34:50 PM

I don't know guys how you can be using this new socket.io via the websocket interface...

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.

Am I the only one who has this problem?

Works for me. Sounds like an issue on your side.
But which language / websocket library do you use?

I tried with latest Chrome and this extension: https://chrome.google.com/webstore/detail/pfdhoblngboilpfeibdedpjgfnlcodoo
.... and also tried with websocket package from Go - and they both disconnect me immediately after sending "7:::1+0"

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
January 25, 2012, 01:45:21 PM

Or just try this page: http://websocket.org/echo.html
... using i.e.  ws://socketio.mtgox.com:80/socket.io/1/websocket/14001291971574386363
and tell me that I'm crazy Smiley

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
January 26, 2012, 03:11:16 PM

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.
If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://
piotr_n
Legendary
*
Offline Offline

Activity: 1750


aka tonikt


View Profile WWW
January 26, 2012, 03:12:54 PM

I get my ID nicely from https://socketio.mtgox.com/socket.io/1/ but then after I connect my websocket lib to ws://socketio.mtgox.com:80/socket.io/1/websocket/XXXXX - it just sends me "7:::1+0" and disconnects immediately after that.
If u got ID from https:// than u must connect to wss:// with this ID (SSL/TLS - secure connection)
and if http:// than ws://
shit man.
thank you!!!
that was indeed the problem.

Check out gocoin - my original project of full bitcoin node & cold wallet written in Go.
PGP fingerprint: AB9E A551 E262 A87A 13BB  9059 1BE7 B545 CDF3 FD0E
osmosis
Sr. Member
****
Offline Offline

Activity: 301



View Profile
January 27, 2012, 09:35:00 PM

 #> python socketio.py
it should dump the messages to stdout

Thanks molecular.

For the websocket in python, anyone find a way to deal with the disconnect errors?

File "/home/osmosis/proj_venv/proj_proj/websocket_client.py", line 161, in recv
SSLError: [Errno 1] _ssl.c:1325: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1325: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1325: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1325: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 8] _ssl.c:1325: EOF occurred in violation of protocol
ConnectionClosedException
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
February 13, 2012, 08:32:28 AM

TL;DR:
Will ws://websocket.mtgox.com/mtgox continue to work?


Some days ago I connected to the old, deprecated websocket. Connecting works, but no data comes in any more.

Now thats an incentive to figure out socket.io :-)

edit:

As far as I know there is no possibility to do orders over the websocket...

Seems like now you can indeed do orders over socket.io:

Quote from: Bitcoin Wiki link=https://en.bitcoin.it/wiki/MtGox/API#Streaming_API
Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the "version 1" requests from the HTTP API can be called.

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..

Any hints?

Ente

zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
February 14, 2012, 05:57:55 PM

So I got the handshake coded, getting me my connection-ID. I then can connect to the websocket just fine.
However, I dont receive any data, only two messages: '1::' and '2::'. Both come once in the first few seconds of being connected, then total silence. I am not being disconnected neither, I think.
Do I have to join a channel? Or am I connected to the default channels just like before?
I dont have "sending messages" finished yet to test..
Any hints?
in the past (3+ weeks ago):
by default u dont subscribed to channels
u may send simple strings over internal (socket.io's) websocket
for "2::" u must send "2::" (or send "2::" every 15secs)
for subscribe u must send post #162
datas stopped in socket after 3-15min.. u can open & sync 2 socket.io..
for now:
post #162 looks no more working - no datas except "1::" & "2::"
Huh
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
February 14, 2012, 07:39:05 PM

OH! Thank you a lot for your help, zelyony!

I was already wondering about the heartbeat. I now answer 1:: and 2:: accordingly, now they keep flowing in constantly.
Well, the socket *must* be working, since the websites using it still work. There would be more outcry in the forum if it was dead altogether..

I am glad I'm not the only one here!

Ente
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
February 14, 2012, 10:29:19 PM

*bangs head on desk*
I cant get it to work.

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

Here is my (totally stripped) python code:

Code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from websocket import create_connection
import urllib
import re
import time

try:
    import json
except ImportError:
    import simplejson as json


wsurl = "wss://socketio.mtgox.com/socket.io/"
iourl = "https://socketio.mtgox.com/socket.io/"

############

print "Connecting to socket.io.."
url = iourl + "1"
f = urllib.urlopen(url)
output = f.read()
f.close()
ausgabe = re.search('[0-9]+', output, 0)
sessionid = ausgabe.group()

print "New websocket url:"
wssurl = wsurl + "1" + "/websocket/" + sessionid
print wssurl

############

print "Connecting to websocket..\n"

socket = create_connection(wssurl)

while 1:
  echo =  socket.recv()
  if "1::" in echo:
    print "Heartbeat 1"
    socket.send("1::")
  elif "2::" in echo:
    print "Heartbeat 2"
    socket.send("2::")
  else:
    print "Message: ",echo
  time.sleep(2)
      

The websocket client I use here is from http://pypi.python.org/pypi/websocket-client/0.4.
I have a rudimentary heartbeat added here.
The whole script dies eventually, from having stripped all error-handling.
It worked some weeks ago. If you like to see the whole script, let me know.

Bedtime here, will check back in, like, 9 hours..

*frustrated and tired*

Ente

Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
February 15, 2012, 08:49:30 AM

1 BTC bounty for the first person who helps me receive regular data again over the websocket!

I made a new thread for this error-solving bounty:
https://bitcointalk.org/index.php?topic=63969.msg749747#msg749747

Ente
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
February 15, 2012, 10:08:59 AM

right now datas arrived (my format after string -> json -> object -> string):
JSON: Packet={ Depth={ price=4,67000 USD,       vol=0,05300000,         dir=bid,        when=2012:02:15_10:03:56.735844 } }
JSON: Packet={ Ticker={ bid=$4.68100,   ask=$4.70075,   lo=$4.21211,    hi=$4.97000,    last=$4.70075,  vol=256,165.42727403??BTC } }

I dont know Python, but:
- on connected (or after recv "1::" - that's mean u connected now) u must send strings like:
4:::{"op":"mtgox.subscribe","type":"trades"}
- on "2::" u must response "2::" (not "1::" as in ur sample)
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
February 15, 2012, 10:19:43 AM

Thank you for checking, zelyony!
You are right, thats a typo in the snippet. I sent '1::' and '2::' accordingly in my full script, typo corrected.

I sent '4:::{"op":"mtgox.subscribe","type":"trades"}' over the websocket, no change. No response whatsoever, no trade-messages coming in.

Ente
zelyony
Newbie
*
Offline Offline

Activity: 23



View Profile
February 15, 2012, 10:39:41 AM

and right now I dont recv any useful datas  Huh

UPD:
it's working right now (13:44 UTC)

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

To MtGox:
maybe it's sabotage.. Slow Read DoS attack.. bad guys overflow send pool with slow connections and.. voila!
will be better for all clients drop connections that cannot read string, i.e. send to client not finished for.. hmm.. 0.5sec! +ban by IP 10min (epoll, select or other asynchronous mechanism). I took digits from air, I dont know % loading for servers machines and number of connections.
p.s. ping ("2::") works always
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
February 22, 2012, 03:39:23 PM

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
February 22, 2012, 04:13:57 PM

UPD2:
it's working if wait connected sometimes 20 minutes and sometimes 0 minutes and sometimes "never". very strange and no good.
can someone's debug session stops sending messages to other users?.. who knows implementation of socket.io server?

Same here. Sometimes it takes ages to receive any data, sometimes less than a minute. Totally random, and totally annoying. The 1:: and 2:: pings work straight from the beginning.
As soon as I get a response to my channel-connect, data from this channel comes in too.

Ente
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
February 27, 2012, 08:36:15 AM

I have a hard time following all the tips and tricks in this thread on getting the socket.io interface to work with Python.

Has anyone gotten it to work properly? As far as I can tell there are no socket.io client implementations for Python right, right?

maybe this helps, my socketio.py:

Quote
from threading import *
import urllib2, urllib
import simplejson as json
import ssl, socket
import time
from websocket_client import create_connection
import traceback

class SocketIO:
  def __init__(S, url, callback):
    S.url = url
    S.callback = callback
    
  def connect(S):
    try:
      data = urllib.urlencode({})
      req = urllib2.Request('https://' + S.url + "/1", data)
      print 'https://' + S.url + "/1"
      response = urllib2.urlopen(req)
      r = response.read().split(':')
      S.heartbeat_interval = int(r[1])
      #print 'heartbeat: ', S.heartbeat_interval
      if 'websocket' in r[3].split(','):
        print "good: transport 'websocket' supported by socket.io server ", S.url
        S.id = r[0]
        print "id: ", S.id
      else:
        print "error: transport 'websocket' NOT supported by socket.io server ", S.url
        
      S.thread = Thread(target = S.thread_func)
      S.thread.setDaemon(True)
      S.thread.start()
    except:
      traceback.print_exc()      

  def stop(S):
    S.run = False
    S.thread.join(timeout=1)
    S.keepalive_thread.join(timeout=1)


  def thread_func(S):
    print 'SocketIO: websocket thread started'
    
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
    
    S.ws = create_connection(my_url)
    
    #S.ws = WebSocket(my_url, version=0)
    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
    
    msg = S.ws.recv()
    while msg is not None and S.run:
      #print 'SocketIO msg: ', msg
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      #elif msg[:3] == "2::":
      #  True
      #else:
      #  print "SocketIO: dont know how to handle msg: ", msg
      msg = S.ws.recv()
    S.ws.close()
      
  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect()
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)
      
def test_callback(msg):
  print 'msg: ', msg

# testcase
if False:
  sio = SocketIO('socketio.mtgox.com/socket.io', test_callback)
  sio.connect()
  time.sleep(100)

EDIT: the websocket_client imported I got from here: http://pypi.python.org/pypi/websocket-client/0.4

EDIT2: I've been using this implementation for months and it works well for me. If you spot any problems with this implementation of fix errors or otherwise improve it, please let me know.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
February 27, 2012, 05:27:52 PM

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:
  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 04, 2012, 05:26:24 PM

Anyone else getting errors like these with molecular's script?

Code:
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 06, 2012, 11:41:24 PM

You da man molecular! Works like a charm. I'm not sure if Mt. Gox had problems or if the script you posted earlier in this thread missed something, but the script above works for me.

Thanks a lot!

BTW, had to change websocket_client to just websocket to get the script to run.

Also, I added a simple function to the SocketIO class that allows one to unsubscribe from a given channel:

Code:
  def unsubscribe(S, channel_id):
    S.ws.send('4::/mtgox:{"op":"unsubscribe","channel":"%s"}' % channel_id)

Thanks, I put that in just in case I need it some day Wink

Anyone else getting errors like these with molecular's script?

Code:
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
SSLError: [Errno 1] _ssl.c:1350: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:1350: error:14095044:SSL routines:SSL3_READ_N:internal error
SSLError: [Errno 1] _ssl.c:1350: error:140D2081:SSL routines:TLS1_ENC:block cipher pad is wrong
SSLError: [Errno 8] _ssl.c:1350: EOF occurred in violation of protocol
Segmentation fault
WebSocketException: Invalid frame type

Full log: http://pastebin.com/m4LF3tYs

Also, when these errors start occuring, I notice that bandwidth usage of my ticker script goes up. To like 20 KB/s IIRC. That seems like a lot for a trade every second or so. I figure it might be SSL certificates and the like going back and forth between mtgox.com and my computer. I find it hard to believe that this way of delivering trade information uses less resources than the original websocket API if everyone experiences these errors.

I have no idea, never seen this. But maybe you could try with the websocket_client.py I use (change that import back from "websocket" to "websocket_client" and the get file websocket_client.py from pastebin: http://pastebin.com/CTGj735u). I used to have a lot of troubles and tried at least 3 different websocket implemenations until I had a properly working one.

I _have_ seen "EOF occurred in violation of protocol", but I guess that's just a nasty disconnect...

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 07, 2012, 04:33:44 PM

Cool. Thanks. I'm trying it out now as we speak. Will report back how it works.
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 20, 2012, 07:37:38 AM

I'm still experiencing these errors with websocket_client unfortunately Undecided
Code:
08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

So, not sure what to do about this... seems something isn't right somewhere (duh!).
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 20, 2012, 08:09:32 AM

I'm still experiencing these errors with websocket_client unfortunately Undecided
Code:
08:06:59 4.81192 1.0 (4.81)
08:07:15 4.81174 0.55301041 (4.81)
08:07:25 4.81925 0.0616 (4.81)
08:07:26 4.81925 0.48723349 (4.81)
08:08:58 4.81174 16.57171689 (4.81)
trade['price_currency'] != "USD"
08:08:58 4.81146 16.0 (4.81)
08:10:26 4.8 10.0 (4.81)
08:11:01 4.7999 3.0 (4.81)
08:11:25 4.79992 0.54994211 (4.81)
trade['price_currency'] != "USD"
08:11:25 4.79992 5.0 (4.81)
08:11:30 4.78 0.27 (4.81)
08:11:30 4.78 2.0 (4.81)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
08:13:00 4.7998 3.0 (4.81)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
trade['price_currency'] != "USD"
08:14:27 4.7999 2.0 (4.81)
08:15:52 4.7998 3.0 (4.81)
08:16:06 4.8 30.0 (4.80)
08:17:13 4.7998 4.0 (4.80)
08:19:12 4.79906 0.03358625 (4.80)
trade['price_currency'] != "USD"
08:20:01 4.79001 1.48066947 (4.80)
08:20:37 4.79899 0.03190743 (4.80)
08:21:07 4.79894 1.22928907 (4.80)
08:21:38 4.79895 3.0 (4.80)
08:22:37 4.79895 2.80261836 (4.80)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 413, in _recv
    raise ConnectionClosedException()
ConnectionClosedException

error sending keepalive socket.io, trying reconnect
error sending keepalive socket.io, trying reconnect
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/rune/Programming/scripts/bitcoin/mtgox_io.py", line 78, in thread_func
    msg = S.ws.recv()
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 349, in recv
    b = self._recv(1)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 411, in _recv
    bytes = self.io_sock.recv(bufsize)
  File "/home/rune/Programming/scripts/bitcoin/websocket_client.py", line 161, in recv
    return self.ssl.read(bufsize)
SSLError: [Errno 1] _ssl.c:1350: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

error sending keepalive socket.io, trying reconnect
08:24:08 4.79001 3.8303 (4.80)

First, a ConnectionClosedException exception is caught at the line "msg = S.ws.recv()" in thread_func. I assume this is because this call just hangs until either an error occurs or a message is received.
Then the keepalive_func thread will detect it when trying to send a message, and it will try to reconnect. This succeeds (seemingly), but lasts little time before the connection is lost again, and the process starts over (as you can see from the log). At some point, the errors go from the ConnectionClosedException exception to SSL exceptions like "error:1408F096:SSL routines:SSL3_GET_RECORD:encrypted length too long", but these SSL errors seem to vary a lot. I would guess that the SSL Python module expects the wrong data, but Mt. Gox' server sends the right data, though I'm not sure about that.

I'm experiencing this, too Sad
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!


It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:
Code:
08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)
That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).

You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
bzzard
Sr. Member
****
Offline Offline

Activity: 398


View Profile WWW
March 20, 2012, 08:20:26 AM

I havent's seen weird stuff in the trade channel for a long time myself (filtering for USD). My depth stuff is still not syncing correctly and my depth table just gets cluttered with obviously old entries that are not removed after a while. I'm not sure wether this is due to websocket implementation or maybe I'm doing something wrong.

Again, any insight? Please post.
I've tested this topic in details and all I can say that lately (~2 weeks) depth data from socketio is useless. Don't trust it.

BTCCharts.com - still for free!
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 20, 2012, 09:21:48 AM

I'm experiencing this, too Sad
I also see problems with threads. I don't know if my thread creation/burial works corretly. When the thread_func() exits that thread should be dead and recycled or whatever, right? Or do I need to still join it?

I tried to make reconnect more robust (at some point I get some "cannot create thread" exception and program exit), but so far failed.

I don't know wether the ConnectionClosed and "SLL" errors are client side fault.

Any insight welcome!
One thing that springs to mind is that upon error, we need to make sure that the existing threads exit, before we start new ones (through calling S.connect() in keepalive_func).
As it is now, keepalive_func will keep running after it has tried re-connecting, and as far as I can see, thread_func will keep running as well. So for each reconnect, we have one extra thread of both keepalive and thread_func running, sending and receiving data on the same socket:
Code:
  def keepalive_func(S):
    while S.run:
      try:
        S.ws.send('2::');
      except:
        if S.run:
          print 'error sending keepalive socket.io, trying reconnect'
          S.connect() #this thread will keep running after this. connect creates a new thread_func, which creates a new keepalive_func
        else:
          print 'exiting socket.io keepalive thread'
      time.sleep(S.heartbeat_interval)
Code:
  def thread_func(S):
    print 'SocketIO: websocket thread started'
   
    my_url = 'wss://' + S.url + "/1/websocket/" + S.id
   
    S.ws = create_connection(my_url)

    S.run = True
    S.ws.send('1::/mtgox')

    # start keepalive thread
    S.keepalive_thread = Thread(target = S.keepalive_func)
    S.keepalive_thread.setDaemon(True)
    S.keepalive_thread.start()
   
    msg = S.ws.recv()
    while msg is not None and S.run:
      if msg[:10] == "4::/mtgox:":
        S.callback(msg[10:])
      msg = S.ws.recv()
    S.ws.close()
As far as I can tell, this can be fixed by breaking out of the while loop in both threads in case of error, replacing the relevant parts like so:
Code:
   def keepalive_func(S):
      while S.run:
         try:
            S.ws.send('2::');
         except:
            if S.run:
               print 'error sending keepalive socket.io, closing existing connection...'
               S.ws.close()
               print '\ttrying reconnect...'
               S.connect()
               raise
               break
            else:
               print 'exiting socket.io keepalive thread'
         time.sleep(S.heartbeat_interval)
and in thread_func:
Code:
         try:
            msg = S.ws.recv()
         except:
            raise
            break
      S.ws.close()

I will see how this runs and report back.

It also looks like I'm receiving corrupted/incorrect data on the socket sometimes:
Code:
08:54:02 4.79 10.0 (*10.0)
08:54:08 4.79 9.9243215 (*19.9)
08:54:11 4.78991 96.0 (4.79)
08:54:12 4.78941 27.0 (4.79)
08:54:12 3.09251 17.0 (4.50)
08:54:12 4.78902 17.0 (4.50)
08:55:16 3.09251 16.65379914 (4.22)
08:55:16 4.78902 16.65379914 (4.22)
08:55:16 4.78272 1.086e-05 (4.22)
That's two trades at $3.09251/BTC happening in between trades at $4.78902/BTC... That shouldn't be possible.

So, not sure what to do about this... seems something isn't right somewhere (duh!).
You might be looking at differenct currencies? 3.09 could be EUR or GBP or something?
Indeed I was. I had just accidentally switched off the ignore-non-USD-trades option. Fixed now.
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 20, 2012, 11:09:30 AM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 20, 2012, 02:37:59 PM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
March 24, 2012, 01:47:40 PM

I tried several hours to create signed commands to send over the socket instead of the old api.
No luck at all. Lastly, I tried to "translate" the php example to python from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands which doesnt look too complicated..




Code:

def get_nonce():
    return int(time.time()*100000)

def encoder(raw):
  dec = json.dumps(raw)
  return dec

def sign_data(secret, data):
    return base64.b64encode(str(HMAC(secret, data, sha512).digest()))


call = "1/info.php"
params = "testparams"
item = "BTC"
currency = "BTC"
id = "1234567890"

Code:
query1 = {'call':call, 'params':params, 'item':item, 'currency':currency, 'id':id, 'nonce':get_nonce()}
Quote
{'nonce': 133259594451109, 'item': 'BTC', 'currency': 'BTC', 'params': 'testparams', 'call': '1/info.php', 'id': '1234567890'}

Code:
query2 = encoder(query1)
Quote
{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}

Code:
sign = sign_data(secret, query2)
Quote
xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==

Code:
query3 = sign, query2
Quote
('xjAc1rbh0T4xFVJT/eZo/nxI4vXxiVmaZxbyM6a5RQtHqld2RW1y9HYdE9z/TlM5e9++P+MMWEd9YFdf6Nenqg==', '{"nonce": 133259594451109, "item": "BTC", "currency": "BTC", "params": "testparams", "call": "1/info.php", "id": "1234567890"}')

Code:
call = {'op':'call', 'call':base64.b64encode(str(query3)), 'id':id, 'context':'mtgox.com'}
Quote
{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU 5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMD ksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiL CAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}


Finally I take this last message and paste it in my socket.send:
Code:
socket.send("4:::{'call': 'KCd4akFjMXJiaDBUNHhGVkpUL2Vaby9ueEk0dlh4aVZtYVp4YnlNNmE1UlF0SHFsZDJSVzF5OUhZZEU5ei9UbE01ZTkrK1ArTU1XRWQ5WUZkZjZOZW5xZz09JywgJ3sibm9uY2UiOiAxMzMyNTk1OTQ0NTExMDksICJpdGVtIjogIkJUQyIsICJjdXJyZW5jeSI6ICJCVEMiLCAicGFyYW1zIjogInRlc3RwYXJhbXMiLCAiY2FsbCI6ICIxL2luZm8ucGhwIiwgImlkIjogIjEyMzQ1Njc4OTAifScp', 'id': '1234567890', 'context': 'mtgox.com', 'op': 'call'}")

Non-descriptive error-reply:

Quote
{u'debug': {u'data': False, u'uuid': u'249bcb53-ccb0-43f1-bac6-e5ae0748e2e1', u'op': u'client'}, u'message': u'Unknown command', u'success': False, u'op': u'remark'}


If someone has enough mercy to have a closer look I would be really happy!
Since this probably takes more than just a few moments, dont forget to quote your bitcoin-address too ;-)

Ente

/who now goes doing some woodwork to clear his head from this
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 24, 2012, 02:17:24 PM

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
March 24, 2012, 02:35:44 PM

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 24, 2012, 03:03:17 PM

I tried several hours to create signed commands to send over the socket instead of the old api.

I used this: https://bitcointalk.org/index.php?topic=49789.msg592388#msg592388, it works.


Yes, I used that too, for my old http api. I try to update to socket-only, which supposedly is almost the same.. Almost, but for me not close enough *sigh*

Thanks for replying!

Ente

"socket-only"? I know of the "new" and "old" (1, 2) http apis. And I know of socket.io interface for depth/trades (no auth necessary). Care to explain?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
March 24, 2012, 03:14:01 PM

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
Ente
Legendary
*
Offline Offline

Activity: 2048



View Profile
March 24, 2012, 03:28:05 PM

According to the wiki API reference, you should be able to do authentication stuff over the WebSocket interface.

Yep:

Quote
Authenticated commands
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called.
https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands

..with more details and a php example too

Ente
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 25, 2012, 08:38:58 PM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 26, 2012, 07:38:30 AM

The above code doesn't work. Don't bother trying it out. break'ing isn't enough, we need to raise an exception to exit the thread. I've written something else that I have yet to test. I will report back how it went when I have tested it.

Thanks so much for your analysis and help so far, runeks.

I think you are correct in saying we need to "decommission" those 2 threads on disconnect.

I'm currently trying

Code:
raise SystemExit

EDIT: looks good so far, seeing some "error: [Errno 32] Broken pipe" and it reconnects

EDIT2: great, more "broken pipes", but still running (exeptionally long). So I figure with the help of runeks, I got the threading issues fixed.

EDIT3: here's socketio.py snapshot of the current version: http://pastebin.com/b5CmTy8a, git is here (might change in the future): https://github.com/molecular/traidor/blob/master/socketio.py



Hi!

Totally forgot about this. Thanks for your token. I've modified your code a bit to see if I can get it completely robust. I'm basically doing the controlling from thread_func instead, because exceptions are almost always detected here first (because "msg = S.ws.recv()" is always hanging there just waiting for a message to come). This means we won't experience a 0-15 second delay before reconnecting (by having the keepalive thread do it). It adds some more complexity though. I still experience the occasional SSL error "EOF occurred in violation of protocol" (besides the ConnectionClosedException) nonetheless. But it seems to be running very robustly so far. No running amok like the previous script. The occasional error comes, but never multiple errors in a row. I will release the code when it has proved to be stable. I'm not experiencing any "Broken pipe" messages though. Only the two aforementioned errors.

I remember while coding it it didn't feel right to detect for reconnect (only) in the keepalive thread.

Thanks for fixing that, please pastebin your version if you consider it robust.

My one is running stably for days now. I might just not care about the 15 seconds.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
March 27, 2012, 03:18:58 PM

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
March 27, 2012, 10:12:20 PM

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote
def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
April 01, 2012, 03:55:33 PM

Hi,

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".
Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 01, 2012, 04:33:18 PM

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
April 01, 2012, 08:41:54 PM

Does anyone know if it is possible to receive market depth updates for other currencies (EUR) in the socket.io/websocket API?
I found this list of channels: https://en.bitcoin.it/wiki/Talk:MtGox/API but couldn't subscribe using the API.
The API only takes a "type" argument (Possible values: "trades, ticker, depth") e.g. {"op":"mtgox.subscribe","type":"trades"}
but there is no documented value for "depth.EUR".

You can subscribe with ?Channel=EUR added to the subscribe URL. I have a working EUR order book, chart, and time & sales on my website. Simply choose EUR from the drop down menu at the top: bitcoin.clarkmoody.com
Thanks but what subscribe URL do you mean ?
I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .
It sends a request to "https://socketio.mtgox.com/socket.io/1" and then opens a websocket to "wss://socketio.mtgox.com/socket.io/1/1/websocket/{id}"

Clark
Hero Member
*****
Offline Offline

Activity: 540


So much code.


View Profile WWW
April 01, 2012, 08:53:18 PM

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

PGP KEY | 1Bitcoin3Tg2KWyAq3wzivdqwYqGwKYaGd
sirk390
Full Member
***
Offline Offline

Activity: 153


View Profile
April 01, 2012, 11:01:43 PM

Thanks but what subscribe URL do you mean ?

Sorry, it should have been ?Currency=

Like this:

https://socketio.mtgox.com/mtgox?Currency=EUR

Thanks you! It works now although i'm not using exactly the same URL.
I'm using https://socketio.mtgox.com/socket.io/1/?Currency=EUR .

 
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
April 01, 2012, 11:09:38 PM

I'm connection with python using something like this https://github.com/osmosis79/gox_socketio_py .

a newer version of which (with improved thread handling, much more stable concerning reconnects) can be found here https://github.com/molecular/traidor/blob/master/socketio.py

osmosis79 seems to have copied it about a month ago (no objections)

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
runeks
Legendary
*
Offline Offline

Activity: 952



View Profile WWW
April 04, 2012, 12:55:16 AM

Here's my version. Just yesterday I found out that I need to handle exceptions in "response = urllib2.urlopen(req)" line in connect(). I couldn't understand why the script had stopped, but found out it was because this line had thrown a ConnectionRefused exception. So it's continually improving as I find out the various ways it can fail. Still haven't gotten rid of the SSL errors though.

https://github.com/runeksvendsen/traidor/blob/master/socketio.py

thanks, so cool.

it bails in line 94 for me, though, with:

Code:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 530, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/nick/bitcoin/traidor/socketio.py", line 94, in thread_func
    S.callback(S, msg[10:])
TypeError: onMessage() takes exactly 2 arguments (3 given)

this is because you added a parameter to the callback function and I still pass a callback function that takes one parameter less.

note that the "test_callback" function is in global scope, not in the class.

suggestion: If you need the instance of the socket.io in testing mode you can just use the global variable "sio" instead of passing S as follows (changes marked in bold):

Quote
def test_callback(msg):
   import ast
   print 'msg: ', msg
   #convert received message string into dictionary
   msg_dict = ast.literal_eval(msg)
   #print msg_dict
   if msg_dict['op'] == 'subscribe':
      if msg_dict['channel'] == 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f' or msg_dict['channel'] == '24e67e0d-1cad-4cc0-9e7a-f8523ef460fe':
         sio.unsubscribe(msg_dict['channel'])

that way everything remains compatible and I can use your code

Right you are. I think I added it to be able to unsubscribe when a message arrives. But as you say, a global variable works better for this. Fixed in the newest revision.
knite
Jr. Member
*
Offline Offline

Activity: 33


View Profile
April 07, 2012, 02:20:53 PM

runeks and molecular:

I've reviewed and begun using a combination of your codebases. I have a few questions and comments.

- Why the different approaches to your keepalive threads? Molecular's version is more concise. Is it an issue with the ws.recv() delay on disconnect?
- Is there a reason for using the urllib2 module? The requests module is much nicer.
- Why aren't you using the logging module instead of print statements?
- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.
- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.
knite
Jr. Member
*
Offline Offline

Activity: 33


View Profile
April 20, 2012, 07:07:36 PM

Threads are still melting my brain. Can I get the best of both worlds by doing something like this?

Code:
def start_thread(function):
t = Thread(target=function)
t.setDaemon(True)
t.start()
return t

def connect(S):
#setup
start_thread(thread_func)

def reconnect(S):
S.ws.close()
S.connect()

def thread_func(S):
S.ws = create_connection(S.ws_url)
S.run = True
start_thread(keepalive_func)

msg = S.ws.recv()
while msg is not None and S.run:
S.process(msg)
try:
msg = S.ws.recv()
except:
if S.run:
S.reconnect()
break
else:
if run:
S.reconnect()

def keepalive_func():
while S.run:
try:
S.ws.send('2::')
except:
S.ws.close()
break
sleep(S.heartbeat)

The keepalive thread does nothing except send heartbeats and close the connection in the event of an exception. Thread_func reconnects if it hits an exception or exits the while loop, if run is True. Thoughts?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2366



View Profile
May 14, 2012, 06:04:04 PM

Hi knite, nice to see the code being used / looked at and thanks for your comments. I'll comment on some of them...

- Is there a reason for using the urllib2 module? The requests module is much nicer.

No, not from my side. This (coding traidor) was the first time I did any requesting in python. Tried urllib (didn't work for some reason), the tried urllib2 (worked) so I just stuck with that.

- Why aren't you using the logging module instead of print statements?

Forgive my ignorance of the logging module, will take a look next time I need to dump some info.

On a more general note: there are never reasons for not doing something, there can only be reasons for doing something Wink. Why do you not believe in the flying spaghetti monster?

- Consider making your SocketIO classes more generic and creating an MTGox sub-class which inherits from it, passing in event handlers (on_connect, on_msg, etc). This will make things easier if and when other exchanges add streaming APIs.

Consider this considered. However, when thinking about integrating other exchanges, there is a lot more work on other parts of my code to be done. I've been wanting to do this for a while now but decided if I would do it, I'd reimplement in another language that better suits my experience. Let me point you to Goncalo Pinheira (goncalopp, don't know if on forum, he mailed me, will give you his email upon request in pm), who has been asking me about this and if he could use my code (been asking for lgpl) and who seems to want to do exactly that (exchange abstraction)

- I strongly recommend reducing your client heartbeat interval. I've set mine to 85% of the value given by the server, rounded down.

done


PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Seal
Donator
Hero Member
*
Offline Offline

Activity: 836


View Profile WWW