Bitcoin Forum
February 08, 2023, 03:40:01 AM *
News: Latest Bitcoin Core release: 24.0.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 3 »  All
  Print  
Author Topic: Data source url change (ticker, depth, history)  (Read 17526 times)
MagicalTux (OP)
VIP
Hero Member
*
Offline Offline

Activity: 608
Merit: 501


-


View Profile
March 09, 2013, 01:45:33 AM
Last edit: March 09, 2013, 02:34:13 AM by MagicalTux
 #1

Many people around here use our ticker to get the latest Bitcoin price, or other of our public data API (depth, etc).

Because of this, the strain on the system has been growing day after day, and has reached a level where we need absolutely to do something. We now have the ticker, depth and trade history available at a new url.

Because not everyone can afford to change their software to support our api v1/v2 ticker, we made this in a way that just replacing the hostname should work (mtgox.com => data.mtgox.com). Other API calls (ie. any api where you authenticate) should still be done as usual on mtgox.com. Please post on this thread or contact MtGox's support (info@mtgox.com) if you have any issue with this change.

Caching is slightly more aggressive for now, but we will reduce it once we confirm everything works the way we expect it to work. Please note that we added a "now" value to the ticker that will allow you to know how old the ticker is.

Ticker

Code:
http://data.mtgox.com/api/2/BTCUSD/money/ticker
http://data.mtgox.com/api/1/BTCUSD/ticker
http://data.mtgox.com/api/0/data/ticker.php?Currency=USD
http://data.mtgox.com/code/data/ticker.php?Currency=USD

Depth

Regular depth data

Code:
http://data.mtgox.com/api/2/BTCUSD/money/depth/fetch
http://data.mtgox.com/api/1/BTCUSD/depth/fetch
http://data.mtgox.com/api/0/data/getDepth.php
http://data.mtgox.com/code/data/getDepth.php?Currency=USD

Full depth data

Code:
http://data.mtgox.com/api/2/BTCUSD/money/depth/full
http://data.mtgox.com/api/1/BTCUSD/depth/full

"Large" depth data (deprecated)

Code:
http://data.mtgox.com/api/0/data/getDepth.php?mode=large
http://data.mtgox.com/code/data/getDepth.php?mode=large&Currency=USD

Trades

Code:
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch
http://data.mtgox.com/api/1/BTCUSD/trades/fetch
http://data.mtgox.com/api/0/data/getTrades.php?Currency=USD

Code:
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1
http://data.mtgox.com/api/1/BTCUSD/trades/fetch?since=1
http://data.mtgox.com/api/0/data/getTrades.php?Currency=USD&since=1

We will eventually replace the URLs on MtGox.com with Location redirects, however since this will most likely break many applications (follow location is not enabled by default on curl for example) and have a negative impact on performance of said applications, we start with a public announcement.

Please update your applications/software/etc by April 1st 2013.

1675827601
Hero Member
*
Offline Offline

Posts: 1675827601

View Profile Personal Message (Offline)

Ignore
1675827601
Reply with quote  #2

1675827601
Report to moderator
1675827601
Hero Member
*
Offline Offline

Posts: 1675827601

View Profile Personal Message (Offline)

Ignore
1675827601
Reply with quote  #2

1675827601
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1675827601
Hero Member
*
Offline Offline

Posts: 1675827601

View Profile Personal Message (Offline)

Ignore
1675827601
Reply with quote  #2

1675827601
Report to moderator
1675827601
Hero Member
*
Offline Offline

Posts: 1675827601

View Profile Personal Message (Offline)

Ignore
1675827601
Reply with quote  #2

1675827601
Report to moderator
1675827601
Hero Member
*
Offline Offline

Posts: 1675827601

View Profile Personal Message (Offline)

Ignore
1675827601
Reply with quote  #2

1675827601
Report to moderator
franky1
Legendary
*
Online Online

Activity: 3752
Merit: 3367



View Profile
March 09, 2013, 02:12:41 AM
 #2

thank you magicaltux. with the ramp and dump the other day i suspect your bandwidth got stretched to breaking point with EVERYONE looking at the charts/tickers.

so im glad to hear your moving the tickers to separate area so that it helps to keep live data flowing and most probably helps ease the ability to add more servers later with higher demand over the next few years, without affecting the other parts of mtgox.

it might be worth adding that it is best to use these tickers for actual 'live day traders' and other people EG just for 'curious/browsing audience' to use charting/data servers such as clark moody/bitcoinity.org. as they are only a few milliseconds different. but it help offload/reduce the strain on mtgox themselves.

I DO NOT TRADE OR ACT AS ESCROW ON THIS FORUM EVER.
Please do your own research & respect what is written here as both opinion & information gleaned from experience. many people replying with insults but no on-topic content substance, automatically are 'facepalmed' and yawned at
Donatus
Newbie
*
Offline Offline

Activity: 9
Merit: 0


View Profile
March 09, 2013, 01:41:03 PM
 #3

Can you make the change more compatible with the existing api please?  For example, the fulldepth url is documented in the wiki as being...

https://mtgox.com/api/1/BTCUSD/fulldepth

but I don't see that in the OP?

--D
klaus
Legendary
*
Offline Offline

Activity: 1904
Merit: 1000



View Profile
March 09, 2013, 01:52:01 PM
 #4

@Donatus

with http://data.mtgox.com they can separate it (even physically) from http://mtgox.com. Thats the intention and its good.

bitmessage:BM-2D9c1oAbkVo96zDhTZ2jV6RXzQ9VG3A6f1​
threema:HXUAMT96
jl2012
Legendary
*
Offline Offline

Activity: 1792
Merit: 1030


View Profile
March 09, 2013, 03:52:58 PM
 #5

Can you make the change more compatible with the existing api please?  For example, the fulldepth url is documented in the wiki as being...

https://mtgox.com/api/1/BTCUSD/fulldepth

but I don't see that in the OP?

--D


You should try before asking for that

https://data.mtgox.com/api/1/BTCUSD/fulldepth

Donation address: 374iXxS4BuqFHsEwwxUuH3nvJ69Y7Hqur3 (Bitcoin ONLY)
LRDGENPLYrcTRssGoZrsCT1hngaH3BVkM4 (LTC)
PGP: D3CC 1772 8600 5BB8 FF67 3294 C524 2A1A B393 6517
prof7bit
Hero Member
*****
Offline Offline

Activity: 938
Merit: 500


https://youengine.io/


View Profile WWW
March 09, 2013, 04:01:50 PM
 #6

The websocket server does not respond anymore since yesterday. It connects instantly but then its totally silent, not even sending headers.

(Socketio-websocket still "works" but with a delay of 2+ Minutes on top of the usual order-lag, it won't even ack my calls to give me an oid for my pending orders in less than 2 minutes even when order-lag says "idle", this is almost unusable for trading)

Is websocket.mtgox.com affected by the change? Is it a known problem, Is somebody already working on the problem and will it be fixed?

MagicalTux (OP)
VIP
Hero Member
*
Offline Offline

Activity: 608
Merit: 501


-


View Profile
March 09, 2013, 10:37:16 PM
 #7

Can you make the change more compatible with the existing api please?  For example, the fulldepth url is documented in the wiki as being...

https://mtgox.com/api/1/BTCUSD/fulldepth

but I don't see that in the OP?

--D

https://mtgox.com/api/1/BTCUSD/fulldepth is actually an alias to https://mtgox.com/api/1/BTCUSD/depth/full
playtin
Full Member
***
Offline Offline

Activity: 201
Merit: 100


https://playt.in


View Profile WWW
March 10, 2013, 04:07:21 AM
 #8

IIRC, the "old" mtgox.com (non-socket) API had some "query it 10 times and get blocked for 24h" policy.
Are those limits lifted for the new data.mtgox.com API?

Bitcoinpro
Legendary
*
Offline Offline

Activity: 1344
Merit: 1000



View Profile
March 10, 2013, 05:03:27 AM
 #9

thanks for the info

WWW.FACEBOOK.COM

CRYPTOCURRENCY CENTRAL BANK

LTC: LP7bcFENVL9vdmUVea1M6FMyjSmUfsMVYf
MagicalTux (OP)
VIP
Hero Member
*
Offline Offline

Activity: 608
Merit: 501


-


View Profile
March 10, 2013, 07:01:49 AM
 #10

IIRC, the "old" mtgox.com (non-socket) API had some "query it 10 times and get blocked for 24h" policy.
Are those limits lifted for the new data.mtgox.com API?

The limits are lifted, however spamming will still end in your IP getting blocked (not by us, but by CloudFlare, using their own method to choose to block you).

Either way getting the same data again and again will not result in new data being returned, remember this.
prof7bit
Hero Member
*****
Offline Offline

Activity: 938
Merit: 500


https://youengine.io/


View Profile WWW
March 10, 2013, 09:03:00 AM
 #11

Either way getting the same data again and again will not result in new data being returned, remember this.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

A few suggestions: If you ever decide to make an API V2 then please consider the following:

 * remove all the redundant different representations of the same number (value, display, display_short), only use value_int. There is a lot of totally redundant data in these structures, if you remove it all it would probably cut down bandwidth by 2/3 and your server would not need to waste time with string formatting.
 * gzip all data
 * allow orders to carry a user defined ID number, so I can generate my own order-id *before* submitting it to mtgox. Then when requesting "private/orders" or inside the private:user_order message each order would have two IDs, my own and the official mtgox oid. Then I could use my own ID for all my internal bookkeeping in my own database during the entire lifetime of the order and use the mtgx id only for commands that I send to mtgox.
 * please provide historical OHLCV data of various time frames, they don't all need to have the entire history, just 100 bars each would be enough already, this would remove the need for many people (including me) to request huge amounts trading history if one just quickly want to plot a graph where for example each 15 minutes or 4 hours or an entire week of trading is consolidated into only 5 numbers.

And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

PS: the websocket server is working again, did you fix it or does it go off and on on its own every Friday? Maybe its the cleaning lady who needs a wall socket for the vacuum cleaner every Friday, maybe you should just lock the door to that room to avoid these problems?

MagicalTux (OP)
VIP
Hero Member
*
Offline Offline

Activity: 608
Merit: 501


-


View Profile
March 11, 2013, 12:23:45 AM
 #12

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

If that's for uses such as development, then it's fine. Some websites actually hit our ticker backend once for every page load on their side.


And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

None, but your suggestions are welcome on info@mtgox.com. I've read what you posted here and will use it. I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side, and all data is already gzipped if supported by the user agent. We'll be checking your other suggestions.
prof7bit
Hero Member
*****
Offline Offline

Activity: 938
Merit: 500


https://youengine.io/


View Profile WWW
March 11, 2013, 10:50:39 AM
Last edit: March 11, 2013, 11:11:34 AM by prof7bit
 #13

I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side

It would really be a great help. Imagine a bot that needs to place orders at certain prices if certain conditions are met but under no circumstances may place the same order twice. Ideally it would look up in its own order list whether its orders are there already (sent but not yet acked or acked and pending or open) or whether they are still missing.

Currently this is not easy: One possibility at the moment would be for my bot to generate an ID and send it in the id field of the signed call and then wait for the op:result that returns this id *and* the official mtgox oid and then update my own database to link these two ids together. But unfortunately the op:result is the only message that will ever refer to this user-generated ID, its the only link between my ID and mtgox ID, if its ever lost for some reasons (temporary disconnect between order/add and op:result during busy times, etc) I cannot reconstruct it. did the order/add goo through? is it pending? is it missing? If it suddenly appears 20 minutes later how do I know where it came from? If I query the private/orders it will not contain these user-generated IDs, I cannot know which of the existing orders belong to which of my bots or belong to manual trading.

Ideally I would want to be able implement a send_order() function in my trading client that is non-blocking and immediately returns an (internal) ID that I can rely on that will survive disconnects, restarts, all kinds of errors, etc. and my cancel_order() function and other order functions would always be able to use this internal ID to refer to the order.

Currently the only reliable way to do it is to encode such additional information in the least significant bits of the order volume: Instead of buying 1 BTC my bot would buy 1.00000042 or <whatever_amount> + <some_small_ID_number>, so each of my bots can recognize their own orders and restore this important part of their state from a simple private/orders query. But I consider this a dirty hack. Its only a workaround.

Such an ID field would not have to be large, people would not need to be able to store their entire autobiography in this field, 32 bit unsigned int would be plenty enough for this purpose.

Seal
Donator
Hero Member
*
Offline Offline

Activity: 845
Merit: 1060


View Profile WWW
March 12, 2013, 12:39:46 AM
 #14

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.

Thanks MagicalTux. I've switched one of my feeds over. Is data.mtgox.com based on another server? What is the latency between data.mtgox.com and mtgox.com?

I also have a suggestion for cutting your bandwidth usage in half... A typical ticker request looks like this:

Code:
{"result":"success","data":{"high":{"value":"48.46900","value_int":"4846900","display":"$48.47","display_short":"$48.47","currency":"USD"},"low":{"value":"45.54000","value_int":"4554000","display":"$45.54","display_short":"$45.54","currency":"USD"},"avg":{"value":"47.57262","value_int":"4757262","display":"$47.57","display_short":"$47.57","currency":"USD"},"vwap":{"value":"47.63996","value_int":"4763996","display":"$47.64","display_short":"$47.64","currency":"USD"},"vol":{"value":"41608.55637760","value_int":"4160855637760","display":"41,608.56\u00a0BTC","display_short":"41,608.56\u00a0BTC","currency":"BTC"},"last_local":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"last":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"last_orig":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"last_all":{"value":"48.36900","value_int":"4836900","display":"$48.37","display_short":"$48.37","currency":"USD"},"buy":{"value":"48.03003","value_int":"4803003","display":"$48.03","display_short":"$48.03","currency":"USD"},"sell":{"value":"48.37000","value_int":"4837000","display":"$48.37","display_short":"$48.37","currency":"USD"},"now":"1363048242058242"}}

The most common request must be for an up to date ticker, however I'm willing to bet that the majority of apps and bots will only ever require Buy, Sell and last price.

high, low, avg, vwap, vol, last_orig and last_all will be redundant to a lot of sites.

Following on from prof7bit's suggestions about cutting the data responses down for redundant data, can you produce a second ticker feed with a more condensed dataset? This will speed up response times for sites requesting data too.

Alternatively as a suggestion is there a way to code it so url parameters decide what should be returned? Eg something like http://data.mtgox.com/api/2/BTCUSD/money/ticker/bid/ask/last

Puppet
Legendary
*
Offline Offline

Activity: 980
Merit: 1031


View Profile
March 20, 2013, 07:25:35 AM
Last edit: March 20, 2013, 12:07:29 PM by Puppet
 #15

Is this preferred over websockets? Since the ws server is down more often than its up, Im falling back to "hammering" your system with ticker requests over http. This seems contraproductive ?

Also Im confused about the depth api. How can I request depth changes since a given change or txid?
What depth messages does http://data.mtgox.com/api/2/BTCUSD/money/depth/fetch provide ?


Never mind, figured it out. I missed the  "filter_min_price" and  "filter_max_price" records, that explains. How often can we call this api?

genuise
Sr. Member
****
Offline Offline

Activity: 379
Merit: 250


View Profile WWW
March 20, 2013, 09:21:17 AM
Last edit: March 21, 2013, 07:45:52 AM by genuise
 #16

In the last 24 hours, data.mtgox.com has served 143513 requests for a total of 25.9 GB bandwidth.

That's just a fraction of what's being served on mtgox.com, but that helps already quite a lot.

There are other reasons for requesting the same API more often than only 5 times per hour, for example when developing client software and testing it. Your API should just be able to take such usage without a problem, after all its mostly only reading and not writing, so it should not put any noticeable stress on your servers.

If that's for uses such as development, then it's fine. Some websites actually hit our ticker backend once for every page load on their side.


And finally: Is there a thread or a section on this forum specifically for mtgox feature requests/problems/questions/etc?

None, but your suggestions are welcome on info@mtgox.com. I've read what you posted here and will use it. I'm not sure about user-defined order ids, as it's going to be using a lot of storage on our side, and all data is already gzipped if supported by the user agent. We'll be checking your other suggestions.


Requesting the same API more than 5 times per hour can be in production mode too - in case when my engine supports more than 5 currencies orderbooks at the same time - when socket reconnects I will need to synchronize all supported orderbooks in the same moment. Then I request full orderbooks for every currency i support. And first test show the first 5 http requests work and the remaining requests get timout response. Thanks to the quite complext logic of managing timeouts for each currency orderbook my engine did not get blocked yet. But it would be good that you give your understanding of this situation and maybe suggest some possible solution in this case?

myself
Legendary
*
Offline Offline

Activity: 938
Merit: 1000


chaos is fun...…damental :)


View Profile
March 21, 2013, 07:12:08 AM
 #17

http://data.mtgox.com/api/1/BTCUSD/ticker

Quote
Database access error, please retry later
Sad

Los desesperados publican que lo inventó el rey que rabió, porque todo son en el rabias y mas rabias, disgustos y mas disgustos, pezares y mas pezares; si el que compra algunas partidas vé que baxan, rabia de haver comprado; si suben, rabia de que no compró mas; si compra, suben, vende, gana y buelan aun á mas alto precio del que ha vendido; rabia de que vendió por menor precio: si no compra ni vende y ván subiendo, rabia de que haviendo tenido impulsos de comprar, no llegó á lograr los impulsos; si van baxando, rabia de que, haviendo tenido amagos de vender, no se resolvió á gozar los amagos; si le dan algun consejo y acierta, rabia de que no se lo dieron antes; si yerra, rabia de que se lo dieron; con que todo son inquietudes, todo arrepentimientos, tododelirios, luchando siempre lo insufrible con lo feliz, lo indomito con lo tranquilo y lo rabioso con lo deleytable.
Kris
Donator
Hero Member
*
Offline Offline

Activity: 640
Merit: 500


View Profile
March 21, 2013, 04:25:36 PM
Last edit: March 21, 2013, 04:39:03 PM by Kris
 #18

Don't know how I missed this thread.

Everything updated.

Will the ticker always stay at HTTP to prevent any overhead of HTTPS (SSL) ?

Thanks.
myself
Legendary
*
Offline Offline

Activity: 938
Merit: 1000


chaos is fun...…damental :)


View Profile
March 21, 2013, 10:14:49 PM
 #19

http://data.mtgox.com/api/1/BTCUSD/ticker

Quote
Database access error, please retry later
Sad

That's what we are trying to avoid by having people use data.mtgox.com. Guess there was a long enough ticker lag to get even data.mtgox.com to fail.
Dear god.

Los desesperados publican que lo inventó el rey que rabió, porque todo son en el rabias y mas rabias, disgustos y mas disgustos, pezares y mas pezares; si el que compra algunas partidas vé que baxan, rabia de haver comprado; si suben, rabia de que no compró mas; si compra, suben, vende, gana y buelan aun á mas alto precio del que ha vendido; rabia de que vendió por menor precio: si no compra ni vende y ván subiendo, rabia de que haviendo tenido impulsos de comprar, no llegó á lograr los impulsos; si van baxando, rabia de que, haviendo tenido amagos de vender, no se resolvió á gozar los amagos; si le dan algun consejo y acierta, rabia de que no se lo dieron antes; si yerra, rabia de que se lo dieron; con que todo son inquietudes, todo arrepentimientos, tododelirios, luchando siempre lo insufrible con lo feliz, lo indomito con lo tranquilo y lo rabioso con lo deleytable.
genuise
Sr. Member
****
Offline Offline

Activity: 379
Merit: 250


View Profile WWW
March 21, 2013, 10:22:16 PM
 #20

http://data.mtgox.com/api/1/BTCUSD/ticker

Quote
Database access error, please retry later
Sad

That's what we are trying to avoid by having people use data.mtgox.com. Guess there was a long enough ticker lag to get even data.mtgox.com to fail.

Hi, You did not mention socket.io api url changes. Will it change too?

And if yes do you know about current socket.io behaviour? Client connects, connect event invoked, and then no any subscribtion message appear. Just silence.

Can you comment on this effect? What can be done?


Pages: [1] 2 3 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!