Data source url change (ticker, depth, history)

[MagicalTux]

Many people around here use our ticker to get the latest Bitcoin price, or other of our public data API (depth, etc).

Because of this, the strain on the system has been growing day after day, and has reached a level where we need absolutely to do something. We now have the ticker, depth and trade history available at a new url.

Because not everyone can afford to change their software to support our api v1/v2 ticker, we made this in a way that just replacing the hostname should work (mtgox.com => data.mtgox.com). Other API calls (ie. any api where you authenticate) should still be done as usual on mtgox.com. Please post on this thread or contact MtGox's support (info@mtgox.com) if you have any issue with this change.

Caching is slightly more aggressive for now, but we will reduce it once we confirm everything works the way we expect it to work. Please note that we added a "now" value to the ticker that will allow you to know how old the ticker is.

Ticker

Code:

http://data.mtgox.com/api/2/BTCUSD/money/ticker
http://data.mtgox.com/api/1/BTCUSD/ticker
http://data.mtgox.com/api/0/data/ticker.php?Currency=USD
http://data.mtgox.com/code/data/ticker.php?Currency=USD

Depth

Regular depth data

Code:

http://data.mtgox.com/api/2/BTCUSD/money/depth/fetch
http://data.mtgox.com/api/1/BTCUSD/depth/fetch
http://data.mtgox.com/api/0/data/getDepth.php
http://data.mtgox.com/code/data/getDepth.php?Currency=USD

Full depth data

Code:

http://data.mtgox.com/api/2/BTCUSD/money/depth/full
http://data.mtgox.com/api/1/BTCUSD/depth/full

"Large" depth data (deprecated)

Code:

http://data.mtgox.com/api/0/data/getDepth.php?mode=large
http://data.mtgox.com/code/data/getDepth.php?mode=large&Currency=USD

Trades

Code:

http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch
http://data.mtgox.com/api/1/BTCUSD/trades/fetch
http://data.mtgox.com/api/0/data/getTrades.php?Currency=USD

Code:

http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1
http://data.mtgox.com/api/1/BTCUSD/trades/fetch?since=1
http://data.mtgox.com/api/0/data/getTrades.php?Currency=USD&since=1

We will eventually replace the URLs on MtGox.com with Location redirects, however since this will most likely break many applications (follow location is not enabled by default on curl for example) and have a negative impact on performance of said applications, we start with a public announcement.

Please update your applications/software/etc by April 1st 2013.

[1714832183]

1714832183

[1714832183]

1714832183

[franky1]

thank you magicaltux. with the ramp and dump the other day i suspect your bandwidth got stretched to breaking point with EVERYONE looking at the charts/tickers.

so im glad to hear your moving the tickers to separate area so that it helps to keep live data flowing and most probably helps ease the ability to add more servers later with higher demand over the next few years, without affecting the other parts of mtgox.

it might be worth adding that it is best to use these tickers for actual 'live day traders' and other people EG just for 'curious/browsing audience' to use charting/data servers such as clark moody/bitcoinity.org. as they are only a few milliseconds different. but it help offload/reduce the strain on mtgox themselves.

[Donatus]

Can you make the change more compatible with the existing api please?  For example, the fulldepth url is documented in the wiki as being...

https://mtgox.com/api/1/BTCUSD/fulldepth

but I don't see that in the OP?

--D

[klaus]

@Donatus

with http://data.mtgox.com they can separate it (even physically) from http://mtgox.com. Thats the intention and its good.

[jl2012]

Can you make the change more compatible with the existing api please?  For example, the fulldepth url is documented in the wiki as being...

https://mtgox.com/api/1/BTCUSD/fulldepth

but I don't see that in the OP?

--D

You should try before asking for that

https://data.mtgox.com/api/1/BTCUSD/fulldepth

[prof7bit]

The websocket server does not respond anymore since yesterday. It connects instantly but then its totally silent, not even sending headers.

(Socketio-websocket still "works" but with a delay of 2+ Minutes on top of the usual order-lag, it won't even ack my calls to give me an oid for my pending orders in less than 2 minutes even when order-lag says "idle", this is almost unusable for trading)

Is websocket.mtgox.com affected by the change? Is it a known problem, Is somebody already working on the problem and will it be fixed?

[MagicalTux]

Can you make the change more compatible with the existing api please?  For example, the fulldepth url is documented in the wiki as being...

https://mtgox.com/api/1/BTCUSD/fulldepth

but I don't see that in the OP?

--D

https://mtgox.com/api/1/BTCUSD/fulldepth is actually an alias to https://mtgox.com/api/1/BTCUSD/depth/full

[playtin]

IIRC, the "old" mtgox.com (non-socket) API had some "query it 10 times and get blocked for 24h" policy.
Are those limits lifted for the new data.mtgox.com API?

[Bitcoinpro]

thanks for the info

[MagicalTux]

IIRC, the "old" mtgox.com (non-socket) API had some "query it 10 times and get blocked for 24h" policy.
Are those limits lifted for the new data.mtgox.com API?

The limits are lifted, however spamming will still end in your IP getting blocked (not by us, but by CloudFlare, using their own method to choose to block you).

Either way getting the same data again and again will not result in new data being returned, remember this.

[prof7bit]

Either way getting the same data again and again will not result in new data being returned, remember this.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

A few suggestions: If you ever decide to make an API V2 then please consider the following:

 * remove all the redundant different representations of the same number (value, display, display_short), only use value_int. There is a lot of totally redundant data in these structures, if you remove it all it would probably cut down bandwidth by 2/3 and your server would not need to waste time with string formatting.
 * gzip all data
 * allow orders to carry a user defined ID number, so I can generate my own order-id *before* submitting it to mtgox. Then when requesting "private/orders" or inside the private:user_order message each order would have two IDs, my own and the official mtgox oid. Then I could use my own ID for all my internal bookkeeping in my own database during the entire lifetime of the order and use the mtgx id only for commands that I send to mtgox.
 * please provide historical OHLCV data of various time frames, they don't all need to have the entire history, just 100 bars each would be enough already, this would remove the need for many people (including me) to request huge amounts trading history if one just quickly want to plot a graph where for example each 15 minutes or 4 hours or an entire week of trading is consolidated into only 5 numbers.

And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

PS: the websocket server is working again, did you fix it or does it go off and on on its own every Friday? Maybe its the cleaning lady who needs a wall socket for the vacuum cleaner every Friday, maybe you should just lock the door to that room to avoid these problems?

[MagicalTux]

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

If that's for uses such as development, then it's fine. Some websites actually hit our ticker backend once for every page load on their side.

And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

None, but your suggestions are welcome on info@mtgox.com. I've read what you posted here and will use it. I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side, and all data is already gzipped if supported by the user agent. We'll be checking your other suggestions.

[prof7bit]

I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side

It would really be a great help. Imagine a bot that needs to place orders at certain prices if certain conditions are met but under no circumstances may place the same order twice. Ideally it would look up in its own order list whether its orders are there already (sent but not yet acked or acked and pending or open) or whether they are still missing.

Currently this is not easy: One possibility at the moment would be for my bot to generate an ID and send it in the id field of the signed call and then wait for the op:result that returns this id *and* the official mtgox oid and then update my own database to link these two ids together. But unfortunately the op:result is the only message that will ever refer to this user-generated ID, its the only link between my ID and mtgox ID, if its ever lost for some reasons (temporary disconnect between order/add and op:result during busy times, etc) I cannot reconstruct it. did the order/add goo through? is it pending? is it missing? If it suddenly appears 20 minutes later how do I know where it came from? If I query the private/orders it will not contain these user-generated IDs, I cannot know which of the existing orders belong to which of my bots or belong to manual trading.

Ideally I would want to be able implement a send_order() function in my trading client that is non-blocking and immediately returns an (internal) ID that I can rely on that will survive disconnects, restarts, all kinds of errors, etc. and my cancel_order() function and other order functions would always be able to use this internal ID to refer to the order.

Currently the only reliable way to do it is to encode such additional information in the least significant bits of the order volume: Instead of buying 1 BTC my bot would buy 1.00000042 or <whatever_amount> + <some_small_ID_number>, so each of my bots can recognize their own orders and restore this important part of their state from a simple private/orders query. But I consider this a dirty hack. Its only a workaround.

Such an ID field would not have to be large, people would not need to be able to store their entire autobiography in this field, 32 bit unsigned int would be plenty enough for this purpose.

[Seal]

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.

Thanks MagicalTux. I've switched one of my feeds over. Is data.mtgox.com based on another server? What is the latency between data.mtgox.com and mtgox.com?

I also have a suggestion for cutting your bandwidth usage in half... A typical ticker request looks like this:

Code:

{"result":"success","data":{"high":{"value":"48.46900","value_int":"4846900","display":"$48.47","display_short":"$48.47","currency":"USD"},"low":{"value":"45.54000","value_int":"4554000","display":"$45.54","display_short":"$45.54","currency":"USD"},"avg":{"value":"47.57262","value_int":"4757262","display":"$47.57","display_short":"$47.57","currency":"USD"},"vwap":{"value":"47.63996","value_int":"4763996","display":"$47.64","display_short":"$47.64","currency":"USD"},"vol":{"value":"41608.55637760","value_int":"4160855637760","display":"41,608.56\u00a0BTC","display_short":"41,608.56\u00a0BTC","currency":"BTC"},"last_local":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"last":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"last_orig":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"last_all":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"buy":{"value":"48.03003","value_int":"4803003","display":"$48.03","display_short":"$48.03","currency":"USD"},"sell":{"value":"48.37000","value_int":"4837000","display":"$48.37","display_short":"$48.37","currency":"USD"},"now":"1363048242058242"}}

The most common request must be for an up to date ticker, however I'm willing to bet that the majority of apps and bots will only ever require Buy, Sell and last price.

high, low, avg, vwap, vol, last_orig and last_all will be redundant to a lot of sites.

Following on from prof7bit's suggestions about cutting the data responses down for redundant data, can you produce a second ticker feed with a more condensed dataset? This will speed up response times for sites requesting data too.

Alternatively as a suggestion is there a way to code it so url parameters decide what should be returned? Eg something like http://data.mtgox.com/api/2/BTCUSD/money/ticker/bid/ask/last

[Puppet]

Is this preferred over websockets? Since the ws server is down more often than its up, Im falling back to "hammering" your system with ticker requests over http. This seems contraproductive ?

Also Im confused about the depth api. How can I request depth changes since a given change or txid?
What depth messages does http://data.mtgox.com/api/2/BTCUSD/money/depth/fetch provide ?

Never mind, figured it out. I missed the  "filter_min_price" and  "filter_max_price" records, that explains. How often can we call this api?

[genuise]

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

If that's for uses such as development, then it's fine. Some websites actually hit our ticker backend once for every page load on their side.

And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

None, but your suggestions are welcome on info@mtgox.com. I've read what you posted here and will use it. I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side, and all data is already gzipped if supported by the user agent. We'll be checking your other suggestions.

Requesting the same API more than 5 times per hour can be in production mode too - in case when my engine supports more than 5 currencies orderbooks at the same time - when socket reconnects I will need to synchronize all supported orderbooks in the same moment. Then I request full orderbooks for every currency i support. And first test show the first 5 http requests work and the remaining requests get timout response. Thanks to the quite complext logic of managing timeouts for each currency orderbook my engine did not get blocked yet. But it would be good that you give your understanding of this situation and maybe suggest some possible solution in this case?

[myself]

http://data.mtgox.com/api/1/BTCUSD/ticker

Database access error, please retry later

[Kris]

Don't know how I missed this thread.

Everything updated.

Will the ticker always stay at HTTP to prevent any overhead of HTTPS (SSL) ?

Thanks.

[myself]

http://data.mtgox.com/api/1/BTCUSD/ticker

Database access error, please retry later

That's what we are trying to avoid by having people use data.mtgox.com. Guess there was a long enough ticker lag to get even data.mtgox.com to fail.

Dear god.

[genuise]

http://data.mtgox.com/api/1/BTCUSD/ticker

Database access error, please retry later

That's what we are trying to avoid by having people use data.mtgox.com. Guess there was a long enough ticker lag to get even data.mtgox.com to fail.

Hi, You did not mention socket.io api url changes. Will it change too?

And if yes do you know about current socket.io behaviour? Client connects, connect event invoked, and then no any subscribtion message appear. Just silence.

Can you comment on this effect? What can be done?