fixed within 5 minutes after quick chat in #mtgox on freenode ;-)
Now I know where to go when there are problems

"Method not found"

is the answer I get from the server for every signed API request (such as private/idkey or private/info or private/orders or order/add, order/cancel etc).

It started happening exactly at 2013-03-14 23:53:48 (central european time), it did not close the connection, it just suddenly stopped accepting any API commands.

Both servers (socketio and websocket) behave this way. I still get ticker, trade and depth but cannot send any trading commands or other commands anymore.

I have just run my code through 2to3 and did some additional small adjustments and got it almost working (there are no fundamental problems, its just a few module names have changed, the () around the print (there are only few) and the change to the str class (strings in py3 are always unicode and for byte-strings i have to use bytes() because str() will not have a decode() method anymore, so a few things regarding the encryption/decryption of the api secret and and base64-encoding/decoding and signing stuff must be slightly changed to use bytes() where I currently (ab)use str() for byte strings, this should all be possible without too many problems. Its only few places that need to be changed.

The main problem was the websocket code from Hiroki Ohtani, I first tried to port it myself, it contains a lot of the same sort of string manipulation all over the place and after an hour or so I gave up. Then I saw that there already exists an experimental py3 version of this module from the same author already on github, so not all hope is lost, it can eventually be ported to python 3 if necessary.

I will try to make my own code more python3 friendly (in small steps to not break anything) so that it will be prepared for easy transition eventually but I won't switch to python 3 yet, not at this time.

I have just googled and it seems you could break your debian stable or bring it into some totally inconsistent state when installing 2.7 on it.

If you really want to experiment with new software then I think debian stable is not the right distribution for you. Its main "advantage" is that it consists only of stuff that is so old (old abandoned branches of software versions) that won't ever change anymore (not moving == stable (== dead?)). You can use it to run old server software that has existed for 20+ years and run them for another 20+ years, software that won't break or change APIs with new updates because the only updates they will ever receive at all will be security fixes. That the Debian definition of "stable": old, petrified, fossilized. Thats what Debian *stable* is meant for.

If you have a Desktop computer and want to be flexible and experiment with cool new stuff now and not in 3 years when its old, you should choose a different distribution. The simplest solution for you would probably to upgrade to Debian testing if you intend to test new (everything that is not really old) software and then stay on their testing release schedule from then on.

Bleeding edge and Debian-Stable just does not go together. Not even "pretty well stable" or "stable" or even "rock solid stable for 2 years now" and "Debian-stable". They have a different and very extreme definition of "stable" over there at Debian.

This is unfortunate because I am not going to refrain from using these weakref classes. Maybe it would be easier to port the entire program to python-3 than to backport weakref.WeakSet() to a deprecated old python-2.6 version, but then again the problem would be that I would then probably be tempted to use features from python 3.2 (Ubuntu) and Debian users with deprecated 3.0 would complain.

I'm doing this for *fun* and not for profit! If I see that there is there is python 2.7 installed on stable Ubuntu (and even was installed on the previous Ubuntu version too and probably even on the one before that) then I am going to use it. I choose python-2.7 because it is the largest stable island and common denominator in the entire python world that is most likely to be available *everywhere*.

There *must* exist a python-2.7 installer for your distribution, 2.7 is three(!) years stable already. Maybe somewhere in the backports repository. If not then there is something seriously wrong in Debian-land, sorry.

Debian. Isn't the whole point of willingly choosing Debian instead of practically *any* other distribution to explicitly *avoid* any software that is less than 5 years old and work only with old and proven stuff that does not move a single millimeter anymore?

Don't misunderstand me. I mean I really try my best to avoid using any bleeding edge stuff when writing software (I once criticized bitcoin-wx for using a bleeding edge wx-widgets version that was *not* in widespread use), I consider something stable and available and usable for mainstream development if it is in the standard repositories of Ubuntu LTS. I had the same kind of problems when I started TorChat 5(!) years ago with python 2.6 (stable on Ubuntu Hardy at that time and widely used everywhere), soon after the Debian users started complaining that it would not work with their ancient python version.

There were even a few Windows-98 users (imagine! win98!), I even installed win98 in wmvare to see what strange problem they had with python-2.6 on win-98 and to see if I could fix it (and fixed it!) but eventually I decided that it is just not reasonably possible to care about all these exotic corner cases (and a Debian stable user (==conservative by definition) who wants to install the latest bleeding edge software is an exotic corner case). I don't have an army of programmers and testers working for me, At some point I must make a decision and then stick to it. In my case this is the software and libs and their versions that are available in Ubuntu.

Not sure, I don't have python 2.6 installed to test it, I relied on the fact that the 2.x branch ended at 2.7 many years ago (I consider it as old and mature and stable) and I assumed that over the time (until now?) from all the different 2.x versions there would eventually only 2.7 be left as the only relevant 2.x version. What Linux distribution are you using? Debian?

What exactly is missing/wrong/different? If it is something trivial it might be possible, but the main problem is I can't (and don't want to) test and maintain it on 2.6 but maybe someone else can temporarily maintain a 2.6 fork until there is no 2.6 anymore (hopefully soon).

Edit: now its 52 minutes already :-(

It would really be a great help. Imagine a bot that needs to place orders at certain prices if certain conditions are met but under no circumstances may place the same order twice. Ideally it would look up in its own order list whether its orders are there already (sent but not yet acked or acked and pending or open) or whether they are still missing.

Currently this is not easy: One possibility at the moment would be for my bot to generate an ID and send it in the id field of the signed call and then wait for the op:result that returns this id *and* the official mtgox oid and then update my own database to link these two ids together. But unfortunately the op:result is the only message that will ever refer to this user-generated ID, its the only link between my ID and mtgox ID, if its ever lost for some reasons (temporary disconnect between order/add and op:result during busy times, etc) I cannot reconstruct it. did the order/add goo through? is it pending? is it missing? If it suddenly appears 20 minutes later how do I know where it came from? If I query the private/orders it will not contain these user-generated IDs, I cannot know which of the existing orders belong to which of my bots or belong to manual trading.

Ideally I would want to be able implement a send_order() function in my trading client that is non-blocking and immediately returns an (internal) ID that I can rely on that will survive disconnects, restarts, all kinds of errors, etc. and my cancel_order() function and other order functions would always be able to use this internal ID to refer to the order.

Currently the only reliable way to do it is to encode such additional information in the least significant bits of the order volume: Instead of buying 1 BTC my bot would buy 1.00000042 or <whatever_amount> + <some_small_ID_number>, so each of my bots can recognize their own orders and restore this important part of their state from a simple private/orders query. But I consider this a dirty hack. Its only a workaround.

Such an ID field would not have to be large, people would not need to be able to store their entire autobiography in this field, 32 bit unsigned int would be plenty enough for this purpose.

Implemented manual trading functions, now its a full featured trading client :-)



In the above screenshot you see the order cancel dialog (press F6 to open it). It is inspired by midnight commander: arrow keys to move up/down, insert to select/unselect an order, F8 to cancel all selected orders, F10 to exit the dialog (sorry, cannot use ESC without ugly hacks). F10 is the key to exit all dialogs.

F4: new buy order
F5: new sell order
F6: cancel order(s)
q quit the client

l (lowercase L) reload the strategy module (your trading bot)

other keys (a..z, except the two above) will emit the signal gox.signal_keypress, the default empty strategy class already has a slot connected to this signal, there you can trigger stuff to happen in your bot when you press certain keys.

Also added since my last posting: display total fiat on the orderbook bid side, total BTC on the orderbook ask side, ratio of these two numbers BTC/USD, everything updating in realtime (see screenshot)



happy trading :-)

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

A few suggestions: If you ever decide to make an API V2 then please consider the following:

 * remove all the redundant different representations of the same number (value, display, display_short), only use value_int. There is a lot of totally redundant data in these structures, if you remove it all it would probably cut down bandwidth by 2/3 and your server would not need to waste time with string formatting.
 * gzip all data
 * allow orders to carry a user defined ID number, so I can generate my own order-id *before* submitting it to mtgox. Then when requesting "private/orders" or inside the private:user_order message each order would have two IDs, my own and the official mtgox oid. Then I could use my own ID for all my internal bookkeeping in my own database during the entire lifetime of the order and use the mtgx id only for commands that I send to mtgox.
 * please provide historical OHLCV data of various time frames, they don't all need to have the entire history, just 100 bars each would be enough already, this would remove the need for many people (including me) to request huge amounts trading history if one just quickly want to plot a graph where for example each 15 minutes or 4 hours or an entire week of trading is consolidated into only 5 numbers.

And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

PS: the websocket server is working again, did you fix it or does it go off and on on its own every Friday? Maybe its the cleaning lady who needs a wall socket for the vacuum cleaner every Friday, maybe you should just lock the door to that room to avoid these problems?

Could you (or anyone who has the data) please make a graph that shows:

  (total_bid / total_ask) vs. time

and overlay that with a price graph?

It would be interesting to see if it can be used a an indicator for future bitcoin price. I noticed that at the moment total_bid/total_ask = 56.9 BTC/USD (and slowly rising). This number is strikingly close to the bitcoin market price (and it is of the same dimension). It would be interesting to see a graph of historical values of that number, especially how it behaves during the times immediately before a major price drop.

Every layer on top of another layer adds overhead. The overhead in this case (it might also be the way MtGox is using it or how they hooked it up to their other software) must be enormous, otherwise it would not lag behind the plain websocket server by at least 10 seconds during idle times (up to a few minutes at busy times), silently drop API-commands (or not ack them for 2 or more minutes, etc), while at the same time the old websocket server (when it is running) will instantly ACK (op:result) all my commands (for example give me the order-IDs for new pending orders within fractions of a second even when order-lag is high).  On the socketio stream I can sometimes wait 2 minutes for my order-ID while the order lag reading says "idle" all the time.

see here: (click for larger image)


5.487 Million USD total on the bid side vs 102931 BTC on the ask side,
this is a ratio of 53.31 USD per BTC :-)

The websocket server does not respond anymore since yesterday. It connects instantly but then its totally silent, not even sending headers.

(Socketio-websocket still "works" but with a delay of 2+ Minutes on top of the usual order-lag, it won't even ack my calls to give me an oid for my pending orders in less than 2 minutes even when order-lag says "idle", this is almost unusable for trading)

Is websocket.mtgox.com affected by the change? Is it a known problem, Is somebody already working on the problem and will it be fixed?

@MagicalTux:

When will the streaming API leave the beta phase so that it can be used for financial applications?

please excuse my ignorance but why does this need a fork at all? "fork" sounds like all users will be forced to make a complicated decision with uncertain consequences, inconveniences, etc and actively migrating from A to B. At least thats what it sounds like. Couldn't it simply be done like that:


and then commit that change to the master branch of the reference implementation like every other upgrade or bug-fix or protocol extension too, document it in the official specifications and be done with it?

Without many scalpers who have their orders on both sides of the book (real orders from real scalpers with the intention to be filled, not fake and bait orders from quote stuffers), orders that absorb parts of the panic-volume (and convert the loss of the n00bs into profits for themselves) it would drop even faster and much further on every temporary panic event.

I have smoked some basic logic: Scalping is converting volatility into profit (for the scalper). Since there is no such thing as a perpetuum mobile it must therefore *reduce* volatility until no profit can be made from it anymore.

I love chart patterns. I don't use any other technical indicators, only patterns. This pattern (yesterday) has the potential to indicate a spike to 55$ (next few days, maybe even today!).