[BETA] MTGox websocket API, testers wanted

[MagicalTux]

Another question: how can I re-subscribe from a previously unsubscribed channel? I have tried sending

Code:

{'channel': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

and

Code:

{'key': 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f', 'op': 'mtgox.subscribe'}

but neither works at getting the ticker feed back after I unsubscribe from it.

Just added:

Code:

{"op":"mtgox.subscribe","type":"trades"}

Possible types: trades, ticker, depth

[slush]

Example usage of this API from Python: MtGox->Sierrachart feed

[0x6763]

Occasionally the websocket sends a depth message to remove an ask or bid (has a negative volume) with a "null" price.  I've only noticed this right before some (not all) trade messages.  I haven't yet seen it happen at volumes unrelated to a trade.

Could it maybe be a race condition in your code where most of the time it gets the depth information before the trade clears it, but sometimes it doesn't?

[0x6763]

Now that I've looked at more instances of it, the volumes are not always obviously related to the subsequent trades, though I think they're related at least most of the time, since these nulls only seem to occur right before a trade or throughout a series of trades.

Here are some examples of the "null" depth message with some surrounding depth/trade info (I removed some messages coming in around these same times that did not seem relevant):

Code:

REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
REMOVE BID -10 @ 1.1832
 BID 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-56},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -56 @ nil
TRADE 10 @ 1.1832
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-46},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -46 @ nil
TRADE 17 @ 1.1827
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-29},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -29 @ nil
TRADE 23.216 @ 1.18151

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-5.784},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -5.784 @ nil
TRADE 5.784 @ 1.1621

Code:

 ASK 0.026 @ 1.1875
 ASK 9.974 @ 1.1875
 ASK 0.03 @ 1.1875
REMOVE ASK -9.97 @ 1.1875
 BID 1 @ 1.18751
 ASK 0.026 @ 1.1875
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-9.974},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -9.974 @ nil
TRADE 1 @ 1.18751

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-4},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -4 @ nil
TRADE 4 @ 1.18751

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":null,"type":2,"volume":-19.966},"op":"private","origin":"broadcast","private":"depth"}
REMOVE BID -19.966 @ nil
TRADE 19.966 @ 1.18751

[MagicalTux]

Hi,

I have located the source of the null and fixed it


Mark

[0x6763]

@MagicalTux

When do you think you'll be able to stop broadcasting dark orders and orders with insufficient funds on the websocket?

[jed]

This depth channel is broadcasting the changes to the depth. It would be nice if you could send something in the beginning to get the current depth table in the same format. Then you can use this feed to change the original depth table as time goes on.
I know I can get it with the old API but it would be nice if the messages were in the same format.
Also is there a point to all these extra fields: "op":"private","origin":"broadcast","private":"depth" ?

[MagicalTux]

Hi,

Broadcasting of depth changes fixed, I think (mostly).

As for the extra fields, they are there mainly to avoid mistaking the messages for something else, however you can safely ignore them

[0x6763]

It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!

[MagicalTux]

It appears that you fixed the dark orders ok (at least the dark asks...I got an error when I tried to make a dark bid, but I have to play around with that to see what the error was since my GUI currently doesn't display error messages to me, yet.)

The orders with insufficient funds still seem to have some problems, though.

I tried an ask of 2000 @ $4, and I only saw the amount I had funds for come across the websocket, but when I canceled both orders (the one with the funds, and the one without), they both came through the websocket as two negative asks @ $4 totaling to 2000.

I then tried a bid of 2000 @ $1, and I only saw the amount I had funds for come across the websocket, but when I was viewing my orders through the old API, I only saw one order of 2000 @ $1 marked as not having sufficient funds (seems to be an unrelated problem to the websocket...I do have funds to cover part of that order), and when I canceled it, a bid of -2000 @ $1 came across the websocket.

So the depth channel is still causing some inconsistencies, but it's definitely improved.  I think there's also the occasional issue where a the price of a trade doesn't exactly match the price of the bid or ask being removed from the depth tables.

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

Thanks for what you've done so far, though!

Fixed notices for orders without enough funds.

As for the trade price, it'll be fixed in a few days when we switch to a new backend~

[jed]

I noticed that sometimes the price is sent in quotes:

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

[MagicalTux]

I noticed that sometimes the price is sent in quotes:

Code:

{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":"3","type":2,"volume":-6},"op":"private","origin":"broadcast","private":"depth"}
{"channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe","depth":{"price":1,"type":2,"volume":10},"op":"private","origin":"broadcast","private":"depth"}

Not a big deal but weird that it isn't consistent.

When the price comes from mysql, it is as a string. I could cast it to float to make sure it's a float, but I guess it doesn't change much for anyone (shouldn't, anyway)

[jed]

Shouldn't the order_add message tell you if it was a buy or sell order?

[jed]

Anybody else occasionally miss trade messages?

[molecular]

I grabbed a copy of the pywsc Websockets library for Python and am receiving trades, orders, and ticker events.

Hmm, I also tried this. Can't seem to succeed.

What port do I have to use? I tried like this:

Code:

from pywsc.websocket import WebSocket
ws = WebSocket('ws://websocket.mtgox.com:80/mtgox')

This bails with "error receiving" in receiver.py. I inserted debugging output to see what it's receiving. It seems to receive the <html/> page you get when you request http://websocket.mtgox.com/.

I'm confused and obviously doing something wrong. Can someone help?

[Clark]

Well, I had to change my WebSocket._send function to do this:

Code:

self._sendRaw('%s%s%s' % ('\x00', data, '\xff'))

because it was breaking the socket trying to send a non-string.

My receive code is identical except that I added a Threading.Event for that loop instead of just while True.

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

[molecular]

I connect to the exact same address, so I really can't tell what the problem. Maybe the handshake is not executing properly before it enters the main receive loop?

SOLVED! Thanks for your help.

Stupid me forgot about that transparent squid proxy.

Am receiving messages, now to the fun stuff!

[Clark]

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

[toffoo]

Awesome!

I'm working at the moment to integrate the websockets feed into a nice GUI, akin to what you'd expect from standard day trading software in the finance world.

Fantastic news.  Please let us know when you have something ready to test out.

[molecular]

ok, I'm receiving messages, nice

Now I want to update the market depth info, which I'm initially getting through http://mtgox.com/code/data/getDepth.php.

so I'm listening on channel "24e67e0d-1cad-4cc0-9e7a-f8523ef460fe" for messages that have op="private", and am getting depth updates like this:

Code:

{u'volume': -10, u'price': 5.4000000000000004, u'type': 2}

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Above message is an example of a message that caused this... I didn't have an entry for price "5.4000" (I'm using 4 decimal places for the index in my depth "dictionary", but that should do no harm, right), so the result is a new entry with negative volume.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

EDIT:

Also, I agree with jed that being able to get the current depth table (as well as trade table, and my own full order list) through the websocket instead of the old API would be an eventual nice-to-have.

If my problem actually stems from the two timepoints (initial depth request, start of depth updates) not being equal, I would consider this to be not a nice-to-have but a must-have.

You could maybe simply (from client point of view) send the complete depth table as a burst of updates after websocket connect?