[BETA] MTGox websocket API, testers wanted

[Shrapnel]

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Another point I'm unclear on: How can I possibly ensure that the timepoint of my request of the initial depth table and the starting timepoint of the websocket depth change messages match?

I'm also wondering the same thing. I think you can't unless the API send the complete data first.
I'm trying to do a realtime order book but it's currently very hard due to dark pool and sync issues.

[1714979865]

1714979865

[1714979865]

1714979865

[molecular]

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

[jed]

I think it doesn't send every update to the order book. It seems like some random ones get dropped.

[0x6763]

It seems, though, that I'm doing something wrong: I'm getting negative volumes in my depth data after a while.

Actually the negative values are cancelled orders.

Of course. So to keep my copy of the "order book" synced, I add that negative value at the given price and should get 0, or (if some other order at the same price still exitsts) some positive value, but never a negative value, which happens to me.

Mtgox has some bugs.  The websocket often entirely skips sending some depth messages, sends some at a slightly different price than what they should be, and the APIs (or site in general) has rounding inconsistencies.  I've only partially worked around these issues by rounding all depth message prices to the 5th decimal place and volumes to the 3rd decimal place, and creating a button in my program that I can click to manually refresh the entire depth tables when I start noticing odd stuff.

An example is where a bot will download it's own open orders, see that one of it's orders is something like 6.253, but it wants it to be 6.252999, so it removes the order for 6.253 and creates a new one at 6.252999, and both of those come over the websocket just like that.  And the bot does this over and over again because the API for getting your own open orders rounds to the 4th decimal place, but the websocket will do at least 6 decimal places for prices and more than that for volume, so your table ends up removing the order at 6.253 over and over again, resulting in a negative number, while it keeps adding orders to 6.252999 over and over again, making some huge number that's not accurate.  The bugs and inconsistencies are pretty annoying.

[Keefe]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

[molecular]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

MagicalTux, can you give us an estimate when you'll get around to implementing a solution to these problems or talk to us about a solution?

These problems are really not acceptable: we can't have the client side depth data out of sync or even worse stuff happening (missing data?)

I'm here to help with testing.

[jaybny]

my program is listening to the WebSocket and publishing to twitter once an hour, on new highs or lows.

im using c# and am not  using JSON schema, just parsing directly... so if anything changes - I will break..

http://twitter.com/newberg_feed

[Ycros]

Herp, disregard what I posted before, I found my client library was sending an extraneous \r\n\r\n on the end there.

[molecular]

I was going to make a nice interface for the feed as well, but I'm not going to bother until the problems are resolved:
Rounding issues
No perfect way to start out in-sync
Alleged missing data and dark pool inconsistencies

I'm seeing a slight problem for MagilTux to solve the "keep client depth tables in sync" problem, I hadn't thought about:

(Firstly, wether a client gets the initial depth data through the "classic API" or through an initial burst of depth-change messages doesn't make a difference for my thinking here.)

Problem is: the depth data a client receives initially is chopped off at the ends at a certain distance to the current last trade value.

Now whenever a trade happens, this window moves and some trades exit the window on the one end while others become "visible" on the other end.

There's 2 ways (at least) to solve this:

• Screw that window completely, give the complete depth data to clients
  
• On trade, determine entering/leaving orders and send them as new/cancelled over the channel
  

I'd go for the first option in combination with initially bursting change-messages instead of using the "classic API" to initially fill client depth table.

[toffoo]

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

[mindtheft]

A quick js chrome extension hack for WebSocket -

https://github.com/mindtheft/mtgox-Chrome-Extension

You can use the crx file to install the extension, or better yet check the source code and pack your own crx file to make sure there's no foul smells from mine.

[0x6763]

Hi guys,
Does anybody have some example python code that just connects to the Mt.Gox websocket feed and maybe just spits out trades or order changes?
I'd like to start playing around with this, but I hate to try to re-create the wheel from scratch when I'm such a rusty python programmer.
Many thanks for any help!

Here's a naive Mt. Gox websocket client I wrote in python: http://pastebin.com/FH6eWhFm
By default it just prints out trades as they happen, but you can modify it to do whatever you want.  If someone really wants an event-based version or something to make it a little easier to work with, you might be able to convince me (with a few bitcoins ) to put something together with pywsc or something.  I may eventually release an executable jar (clojure/java) file for those willing to buy a copy that will give you a GUI to view live trades and depth changes (though mtgox's websocket is still quite buggy in respect to depth changes...there would be a button to let you do manual refreshes as well).

[martin]

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

[0x6763]

I've been playing around a bit with this webcosket feed in C#, and I'm not sure if I'm doing things wrong or the feed is slightly broken.

Every time I receive a depth update I check the volume, if it's positive I add it to my local depth table, if it's negative I search the table for an exact matching depth entry, and remove it. After a while, this ends up with sell prices being less than buy prices - which is clearly wrong. Should this work, or am I missing something?

w.r.t all the people saying that sending the complete depth table on connection would be useful, I agree. It seems vital for the client to be able to request depth data, I would implement it as the client sending a request message for depth data, something like:

{ "op":"request-depth" }

And then the socket sends back the *complete* depth data in a *single message* in the same form that the web API currently returns it. In this way, once you receive the complete table you know you have a reliable dataset to perform updates to as they arrive.

The obvious caveat of this is that once you send the complete depth data, it must act as a "barrier", you can't send the complete state, and then an update to a stale state. (does this make sense?)

Mtgox currently has multiple bugs in the websocket's depth channel...your tables are behaving exactly the same as mine are.  Mtgox is moving to a new system in june which will hopefully have all of these problems taken care of.

[martin]

Ah excellent, any information on the form of this new system?

[zelyony]

I wrote some code in C# for MonoDevelop or Visual Studio
MtGox & IMtGoxHandler & TestMtGox - for making orders
  FetchCash
  FetchOrders
  Place/CancelBuyOrder
  Place/CancelSellOrder
  .Handler - interface for responses & errors
WebSocket & Program - for read trades events
  .OnData - delegate received string (in JSON format) for any channel

https://rapidshare.com/files/3646016978/MtGoxApp.zip

[zelyony]

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

technically:
JSON changes to this: add new attr to "depth": "abs"=X: where X = "1" for absolute volume; "0" or absent attr - relative volume as now
That data can be cashed on server for all new clients. Depth changes updates cache. Need to generate JSON (for sent) on request only - more expansive operation than just update some structs in cache.

PS
Sorry for my Russian English

[zelyony]

maybe
needs some authorization vs WebSockets' flood from bag guys (direction: from client to server)

something like this:
- some cookie received from Trade API for websocket connection to MtGox. this cookie valid only next hour (5-10-30min)
- on new connection to websocket client must send this cookie to server. if cookie is valid automatic subscribing will do..
cons: more difficult process for connection

or:
- using SSL/TLS encrypted WebSocekts.
pro: can refuse polling Trade API and do all orders and receive statuses over WS
cons: need implementations for different languages

[zelyony]

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?
for price enough.. 2 digits for cents.. and.. ok, let be 4digits after dot
for volume.. hmm.. similar numbers less than 0.001 are considered as zero.. = so 3 digits after dot for volumes

[martin]

There's a new MTGox websocket API. This API works by subscription to channels, and each channel is represented by an UUID.
...

imho
It will be good to send 10-30 best Sell and Buy depths one new connection after subsribe to depth channel : price, type, ABSOLUTE-volume
After that depth data sent in relative manner: +volume, -volume
It gives knowledge to robots(or men) of current volumes for asks & bids.

I prefer my suggestion of a request packet which the client can send at any time to request the complete depth state, and then relative +volume -volume stream continues from there.

volumes and prices sometimes received as :X.YYYYYYYYYYY
who needed so many digits after dot?

We need as many digits as people can specify, which as far as I know is an unlimited amount.