[BETA] MTGox websocket API, testers wanted

[Fireball]

...I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.
...
For those who are looking for yet another example: Here is my python client implementation: https://github.com/prof7bit/goxtool  it implements both protocols: plain old websocket and the bizarre socket.io-monstrosity that is always lagging behind the websocket by at least 5 seconds (you can configure which one to use)

I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

At ICBIT, I provide the API fully via socket.io not only for trading bots, but for the web trading too. As far as I can tell (and our users can confirm that), it works without this kind of glitches, very smooth, and indeed compatible with many different platforms.

I am not that skilled in Python, alas, however I could help you debug/test the problem by using ICBIT's socket.io connection and see if your problems appear there.

[molecular]

...I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.
...
For those who are looking for yet another example: Here is my python client implementation: https://github.com/prof7bit/goxtool  it implements both protocols: plain old websocket and the bizarre socket.io-monstrosity that is always lagging behind the websocket by at least 5 seconds (you can configure which one to use)

I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

At ICBIT, I provide the API fully via socket.io not only for trading bots, but for the web trading too. As far as I can tell (and our users can confirm that), it works without this kind of glitches, very smooth, and indeed compatible with many different platforms.

I am not that skilled in Python, alas, however I could help you debug/test the problem by using ICBIT's socket.io connection and see if your problems appear there.

I can confirm the connection drops (especiall when a lot of shit happens).

I can assure you the problems have already existed before the switch from websocket to socket.io.

[prof7bit]

I want to put my 2 cents here. First of all, Mt.Gox's socket.io connection indeed drops very often, I confirm this too. However, as for socket.io vs websocket, the thing is that socket.io may use websocket as the underlying transport layer, so that essentially it turns out to be a websocket connection with a few additional characters added into the websocket stream.

yes, its a websocket connection too, but before connecting you first do a http request to get your own session ID and then use that to get your very own websocket stream. This puts a lot of unnecessary load on the server during connection for *nothing*, especially when one needs to reconnect often because it drops the connection often. The first step of allocating this personalized websocket URL alone takes 30 seconds or longer sometimes, one can almost hear the server groaning on the other side while it is lifting heavy stuff around to allocate me a personalized socket.io session before I can even connect to it.

Plain websocket most of the time connects instantly.

So if data lags by 5 seconds, it might be that there is a bug in your trading client.

No, there is no bug on my side, I'm logging the messaes to the console as they arrive, its lagging behind severely. Just try it yourself, run a minimalistic websocket client that does nothing except printing the messages to the console and at the same time run a minimalistic socket.io client in another console window and print the same type of messages and then compare how the messages that you have seen scrolling by in the websocket client will appear 10 or more seconds delayed in the socket.io client.

[alpet]

For last days, automaticaly subscription for data (ticker,trades,depth) is not performed after connection to socketio.mtgox.com in most attempts. Some times is performed with late, after 10-30 minutes.

[prof7bit]

is plain websocket still considered deprecated? On https://en.bitcoin.it/wiki/MtGox/API/Streaming I cannot find a word mentioning this (anymore). Did they finally get back some common sense? Now the next step would be to fire the PHP-kiddies and hire some experienced Erlang programmers instead to build a brand new matching engine from scratch (or buy one) *before* it all collapses under the sheer weight of unbelievable 5 orders per second.

[molecular]

is plain websocket still considered deprecated? On https://en.bitcoin.it/wiki/MtGox/API/Streaming I cannot find a word mentioning this (anymore). Did they finally get back some common sense? Now the next step would be to fire the PHP-kiddies and hire some experienced Erlang programmers instead to build a brand new matching engine from scratch (or buy one) *before* it all collapses under the sheer weight of unbelievable 5 orders per second.

I don't know, but I heard MagicalTux is planning some updates. He's getting bashed pretty badly in the "wall thread" about the trade engine performance. Maybe he's getting rid of the socket.io (it could be a bottleneck for some reason).

I don't know what it is but there must be some severe problems when you look at the machine specs: https://mtgox.com/img/pdf/20120831/Transparency.008.jpg and compare that to estimates of the order throughput that has to be digested and resulting lag that happens.

[prof7bit]

@MagicalTux:

When will the streaming API leave the beta phase so that it can be used for financial applications?

[Scrat Acorns]

No, there is no bug on my side, I'm logging the messaes to the console as they arrive, its lagging behind severely. Just try it yourself, run a minimalistic websocket client that does nothing except printing the messages to the console and at the same time run a minimalistic socket.io client in another console window and print the same type of messages and then compare how the messages that you have seen scrolling by in the websocket client will appear 10 or more seconds delayed in the socket.io client.

The problem is (and has always been) the fallback transports. They are also the reason that socket.io exists. In a world where every browser has RFC 6455 websocket support socket.io has no reason for being. Channels? Meh. Broadcast? Meh. Socket.io with a websocket transport also has significant overhead versus a pure websocket implementation.

Add to that that 0.9 is currently unmaintained (there are over 300 open issues on github). The only way to scale socket.io right now is to throw more machines at it and use RedisStore.

The polling transports (XHR/JSONP) are performance killers but if you disable them you lose IE6-9. They will both leak connections over long periods of time resulting in GC having to manage gigs of useless heap space slowing it down even more.

[writhe]

* I have not found any way to request fulldepth through the websocket API, again there is documentation missing, is this possible at all?

Yes, you can do this via call operations with the BTC{currency}/fulldepth endpoint.

Note that this is rate-limited and the rate is pretty low. I've gone over it quite easily during testing. Unless you need the entire depth information it's probably worth sticking with partial depth information via BTC{currency}/depth.

* The connection is lost very often, sometimes its even impossible to reconnect for periods of many minutes, please fix this. As a first step to remove load from your servers I would suggest completely abandoning this totally unnecessary Socket.io monstrosity that serves no other purpose than demonstrating that some crazy programmer somewhere was able to build the Frankenstein-Monster equivalent of network-protocols. Please drop this nonsense and revert back to good old plain websockets.

I think the server will deliberately disconnect clients after an hour (at least, this is what it looks like from my logs).

I've only had problems with WebSocket connections in the past few days and I can't connect at all at the moment. It looks as though the TCP connection is established but the server won't sent any responses when the TLS session is being established (or even send any ACK replies…). Ditto for unencrypted connections.

[Fireball]

Socket.io with a websocket transport also has significant overhead versus a pure websocket implementation.

Add to that that 0.9 is currently unmaintained (there are over 300 open issues on github). The only way to scale socket.io right now is to throw more machines at it and use RedisStore.

Could you please elaborate about significant overhead?

https://github.com/LearnBoost/socket.io/blob/master/History.md - recent releases history. I don't see any signs of it being abandoned.

[prof7bit]

Could you please elaborate about significant overhead?

Every layer on top of another layer adds overhead. The overhead in this case (it might also be the way MtGox is using it or how they hooked it up to their other software) must be enormous, otherwise it would not lag behind the plain websocket server by at least 10 seconds during idle times (up to a few minutes at busy times), silently drop API-commands (or not ack them for 2 or more minutes, etc), while at the same time the old websocket server (when it is running) will instantly ACK (op:result) all my commands (for example give me the order-IDs for new pending orders within fractions of a second even when order-lag is high).  On the socketio stream I can sometimes wait 2 minutes for my order-ID while the order lag reading says "idle" all the time.

[Scrat Acorns]

Could you please elaborate about significant overhead?

Javascript, among other things. Don't get me wrong I love node but it just can't handle memory copying / allocating data structures as fast as a compiled language, which is understandable as it is using a JIT compiler with GC. In any case the difference with native C implementations isn't that big but it's there. I do realise that I'm comparing apples with oranges here but it's a valid comparison since the only stable socket.io server implementation is in javascript.

I forgot to add that the big advantage of socket.io is its (almost) seamless horizontal scalability as long as your redis box is beefy.

https://github.com/LearnBoost/socket.io/blob/master/History.md - recent releases history. I don't see any signs of it being abandoned.

https://github.com/LearnBoost/socket.io/issues?page=1&sort=comments&state=open

Its creator is busy working on 1.0 which is a complete rewrite (uses engine.io as the abstraction for the underlying transports).

it might also be the way MtGox is using it or how they hooked it up to their other software

They are doing it wrong.

[prof7bit]

"Method not found"

is the answer I get from the server for every signed API request (such as private/idkey or private/info or private/orders or order/add, order/cancel etc).

It started happening exactly at 2013-03-14 23:53:48 (central european time), it did not close the connection, it just suddenly stopped accepting any API commands.

Both servers (socketio and websocket) behave this way. I still get ticker, trade and depth but cannot send any trading commands or other commands anymore.

[writhe]

"Method not found"

is the answer I get from the server for every signed API request (such as private/idkey or private/info or private/orders or order/add, order/cancel etc).

It started happening exactly at 2013-03-14 23:53:48 (central european time), it did not close the connection, it just suddenly stopped accepting any API commands.

Both servers (socketio and websocket) behave this way. I still get ticker, trade and depth but cannot send any trading commands or other commands anymore.

Yeah, I'm seeing this to. Also seeing "invalid call".

By the way, is it possible to connect to socketio.mtgox.com via WebSocket? That is, a regular RFC 6455 WebSocket and not via the Socket.IO library (which I believe is meant to be able to fall back to WebSocket). I've never had any success with this, I've only ever been able to make WebSocket connections to websocket.mtgox.com on port 80 / 443.

[prof7bit]

fixed within 5 minutes after quick chat in #mtgox on freenode ;-)
Now I know where to go when there are problems

[prof7bit]

Yeah, I'm seeing this to. Also seeing "invalid call".

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make an unsigned 64 bit integer and not one invalid calls anymore since the day I made that change, before i used something like time.time()*1E6 and saw a lot of "invalid call". One day I'm going to investigate what exactly it was that was wrong with my microseconds and why it works now.

The socketio server only accepts the socketio protocol (http request, then get a session ID, then use that ID to have a new URL to connect to a websocket service but this is not the same websocket as the plain old websocket, all messages are wrapped into some socketio protocol ("1::/mtgox" headers and the like). This is a whole new complicated layer on top of websocket.

According to the person I just talked to the plain websocket server is NOT (I repeat: NOT) deprecated, we can use it! Its also 10 seconds faster (no annoying lag, no sluggishness), reacts instantly to all commands.

[writhe]

Yeah, I'm seeing this to. Also seeing "invalid call".

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make a 64 bit integer and no invalid calls anymore, before i used something like time.Time()*1E6 and saaw a lot of "invalid call"

Thanks, I'll try this out.

The documentation and PHP example code led me to believe that the nonce needs to increment for each message. Currently I'm setting the nonce of the first request to the Unix time in microseconds and then incrementing by 1 in each additional request.

It's weird that this would cause problems if a random value works fine. I'll see if I can get some clarification from MtGox.

The socketio server only accepts the socketio protocol (http request, then get a session ID, then use that ID to have a new URL to connect to a websocket service but this is not the same websocket as the plain old websocket, all messages are wrapped into some socketio protocol ("1::/mtgox" headers and the like). This is a whole new complicated layer on top of websocket.

Sounds heinous.

[prof7bit]

Yeah, I'm seeing this to. Also seeing "invalid call".

I have had invalid call before i changed the way my nonce was created. Now I'm just using 8 bytes from urandom to make a 64 bit integer and no invalid calls anymore, before i used something like time.Time()*1E6 and saaw a lot of "invalid call"

Thanks, I'll try this out.

The documentation and PHP example code led me to believe that the nonce needs to increment for each message.

STOP

forget everything I just wrote about the nonce, I had a bug in my program (I used struct.unpack() which creates a tuple and put that tuple with the random number into my json instead of only the nonce). And the mtgox-server also has a bug so that it *accepts* this without complaining!

It will *not* accept it if the nonce is a plain random number and not incrementing. And it will never forget, you need a new API key after the change.

I have just spent another hour with my code and now my code look like this, I also requested a brand new virgin API key and now it works:

Code:

    def send_signed_call(self, api_endpoint, params, reqid):
        """send a signed (authenticated) API call over the socket.io.
        This method will only succeed if the secret key is available,
        otherwise it will just log a warning and do nothing."""
        if (not self.secret) or (not self.secret.know_secret()):
            self.debug("### don't know secret, cannot call %s" % api_endpoint)
            return

        key = self.secret.key
        sec = self.secret.secret

        nonce = str(int(time.time() * 1E6))

        call = json.dumps({
            "id"       : reqid,
            "call"     : api_endpoint,
            "nonce"    : nonce,
            "params"   : params,
            "currency" : self.currency,
            "item"     : "BTC"
        })

        # pylint: disable=E1101
        sign = hmac.new(base64.b64decode(sec), call, hashlib.sha512).digest()
        signedcall = key.replace("-", "").decode("hex") + sign + call

        self.debug("### calling %s" % api_endpoint)
        self.send(json.dumps({
            "op"      : "call",
            "call"    : base64.b64encode(signedcall),
            "id"      : reqid,
            "context" : "mtgox.com"
        }))

you find it at github: prof7bit/goxtool

[Puppet]

One thing isnt clear to me. when a trade occurs. does the API also send a depth message that reflects this trade? Or do I need to subscribe to the trade channel as well and then  match the trade with the orderbook to keep the orderbook up to date?

[prof7bit]

One thing isnt clear to me. when a trade occurs. does the API also send a depth message that reflects this trade? Or do I need to subscribe to the trade channel as well and then  match the trade with the orderbook to keep the orderbook up to date?

On trade messages you need to update the orderbook yourself, there will be no depth message. [Edit: this is wrong, there is a depth message following every trade message but it can't do any damage if you always only use the total_volume and never use the volume difference]

If the trade is type:"bid" then it has filled an ask order, update your asks, if trade is type:"ask" then it has filled a bid order, update your bids.

On *own* trades there will be two trade messages: One public that everybody receives (like above), use that to update your orderbook like above and one private only to notify you about your trade that happened. Also on every trade that affects your *own* orders there will be user_order messages, at least one that sets the volume to the remaining volume (or zero and another one to finally remove it).

[Edit: the code below is flawed (maybe even outright wrong: You should not use the volume difference in the depth message, you should use the total volume]

WRONG WRONG WRONG

Code:

def on_trade(typ, price, volume, own):
  typ_filled = {"ask":"bid", "bid":"ask"}[typ]
  if own:
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe notify your bot about this event, or notify the user.
    pass
  else:
    update_orderbook(typ_filled, price, -volume)

def on_depth(typ, price, volume):
  update_orderbook(typ, price, volume)

def on_user_order(oid, price, new_volume, new_status):
  update_own_orders(oid, price, new_volume, new_status)

/WRONG /WRONG /WRONG


Edit: Here is a corrected version:
you would have two methods updating the orderbook, one that accepts a delta and one that accepts the new total volume.

Code:

def on_trade(typ, price, trade_volume, own):
  typ_filled = {"ask":"bid", "bid":"ask"}[typ]
  if own:
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe notify your bot about this event, or notify the user.
    pass
  else:
    update_orderbook_with_delta(typ_filled, price, -trade_volume)

def on_depth(typ, price, total_volumel):
  update_orderbook_with_absolute(typ, price, total_volume)

def on_user_order(oid, price, new_volume, new_status):
  update_own_orders(oid, price, new_volume, new_status)

def update_orderbook_with_delta(typ, price, delta_volume):
  # do whatever is needed to update a price level,
  # remove it if it is <= 0 after the update
  [...]

def update_orderbook_with_absolute(typ, price, total_volume);
  # do whatever is needed to update a price level to the new total_volume
  # remove the level if total_volume == 0
  [...]

And here is a simplified version that just waits for the inevitable depth message and completely ignores the trade message:

Code:

def on_trade(typ, price, trade_volume, own):
  if own:
    # an own order has been filled (this is only received in the private channel)
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe notify your bot about this event, or notify the user.
    # the same message will be fired again with vol=False in the public
    # channel like all other trade message too.
    pass
  else:
    # a trade has happened on mtgox (this is the public message)
    # do nothing as far as bookkeeping the order lists is concerned
    # maybe log the message or notify your bot if it needs it
    pass

def on_depth(typ, price, total_volumel):
  update_orderbook_with_absolute(typ, price, total_volume)

def on_user_order(oid, price, new_volume, new_status):
  update_own_orders(oid, price, new_volume, new_status)

def update_orderbook_with_absolute(typ, price, total_volume);
  # do whatever is needed to update a price level to the new total_volume
  # remove the level if total_volume == 0
  [...]