MtGox API version 2: Unofficial Documentation

[dfrechet]

Does anyone know what the deal is with those streaming API depth updates with volume_int set to zero? The total volume still changes, breaking cumulative checks. It appears that about 5% of all depth updates are of this form, with some clustering with respect to the price (for certain price values, depth updates consist only of these messages).

[1715630702]

1715630702

[1715630702]

1715630702

[1715630702]

1715630702

[btc_lurker]

Does anyone know what the deal is with those streaming API depth updates with volume_int set to zero? The total volume still changes, breaking cumulative checks. It appears that about 5% of all depth updates are of this form, with some clustering with respect to the price (for certain price values, depth updates consist only of these messages).

Is this thread about the HTTP API or the Streaming API ?

Anyway, volume_int set to zero is simply telling you that there are no more bids/asks at that value -- it was fully removed. This of course changes the total volume, since it reduces the total volume. Also note that volume_int isn't about the last change in volume for a given price, it is the current total volume for that price.

[dfrechet]

Are you sure? The documentation under https://en.bitcoin.it/wiki/MtGox/API/Streaming#Depth seems to indicate otherwise:

total_volume_int: total volume at this price, after applying the depth update, can be used as a starting point before applying subsequent updates.
volume_int: volume change * 1E8

More than 90% of all depth updates as measured during the last day fit this pattern (old_total_volume_int + volume_int = total_volume_int). Suspiciously, all depth updates that break it have volume_int set to zero...

PS: If there is any moderator around, feel free to move this discussion to a new topic ("MtGox Streaming API: Zero Volume Depth Updates").

[btc_lurker]

Are you sure? The documentation under https://en.bitcoin.it/wiki/MtGox/API/Streaming#Depth seems to indicate otherwise:

total_volume_int: total volume at this price, after applying the depth update, can be used as a starting point before applying subsequent updates.
volume_int: volume change * 1E8

That is exactly what I said: "Also note that volume_int isn't about the last change in volume for a given price, it is the current total volume for that price"

So if the total volume is zero, it means there are no more bids/asks at that value. That is caused because the users that entered them decided to remove them all, and the volume went to zero.

Edit: Ok, I misread volume_int for total_volume_int. I currently don't bother with volume_int in my applications, is there a reason to do so ? Maybe for performance reasons ? Given the low volume at max hundreds of bids/asks in a second (maybe ?) at MtGox, I haven't hit performance problems yet.

[dfrechet]

I use it mainly for error checking and finding individual depth changes I might have missed during a socket reconnect.

Either these mysterious packets have some special undocumented meaning, or they are the result of a bug on part of the API. In doubt, I rather assume the former, hence this question...

[btc_lurker]

For me in particular, it is mystery on how to really keep the depth market info correct and updated. The HTTP API provides methods for loading them (either partially or fully), but they are not updated. So you have to, for instance, load partial data using the HTTP API and then update it as the Streaming API sends new data regarding depth. As you receive data from the Streaming API you can easily discard some of the data feched earlier as it becomes invalid (not talking about zero volume), but you cannot be sure about the volume of every ask/bid at every price.

Hopefully, after running for some time X you will have correct data at least regarding the prices with most changes (i.e., those were trades are effectively happening). Too bad you might get disconnected, because then you will need some more time Y in order to "recorrect" the data you had. The partial solution is to never stop your application, so you can keep mostly correct information.

[cilphex]

Nitrous, you state this in your documentation:

14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

[btc_lurker]

Nitrous, you state this in your documentation:

14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

[cilphex]

Sweet, thank you.

[nitrous]

Nitrous, you state this in your documentation:

14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Sorry, I haven't been very clear. Fetching those methods by GET works fine, but if you fetch them by POST, some methods which work fine unauthenticated by GET start to produce unexpected behaviour and strange errors unless provided with a valid api key. Therefore, you should either incorporate both get and post requests, or provide a valid api key for all methods by post. (I said all, but I'm sure there are some which aren't so strict. The ticker methods used to not require auth, even by post, for example. Seeing as that seemed to change though, you shouldn't rely on post requests being lenient.)

[nitrous]

Nitrous, you state this in your documentation:

14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

That text is very misleading. You most certainly can access authenticated api methods on data.mtgox.com, in fact all methods go through that subdomain, including trading, sending bitcoins, merchant stuff, etc.

[btc_lurker]

Nitrous, you state this in your documentation:

14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

That text is very misleading. You most certainly can access authenticated api methods on data.mtgox.com, in fact all methods go through that subdomain, including trading, sending bitcoins, merchant stuff, etc.

No, not really. Till recently I could add a param version=2 to a json payload and request data from any method provided by the HTTP API through the address websocket.mtgox.com (that is, I was using entirely the streaming API, but still able to request data from the HTTP API through it).

Now, some of those methods are no longer accessible through websocket.mtgox.com, and must go through data.mtgox.com.

[nitrous]

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

That text is very misleading. You most certainly can access authenticated api methods on data.mtgox.com, in fact all methods go through that subdomain, including trading, sending bitcoins, merchant stuff, etc.

No, not really. Till recently I could add a param version=2 to a json payload and request data from any method provided by the HTTP API through the address websocket.mtgox.com (that is, I was using entirely the streaming API, but still able to request data from the HTTP API through it).

Now, some of those methods are no longer accessible through websocket.mtgox.com, and must go through data.mtgox.com.

That's being a bit pedantic isn't it? My point was that the text at data.mtgox.com was wrong - it says any api that requires authentication will not be available there, yet I have been able to access authenticate methods over the subdomain for over a month now, and others have used this for years to build their btc bots. That is the official way to access authenticated methods, so clearly it's wrong. Yes you are right, you can also access some of these methods over websockets, but, ostensibly at least, accessing rest methods over the socket seems to function simply as a proxy to data.mtgox.com.

[glon]

Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?

[btc_lurker]

Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?

Could it be a bug in this software itself ? I'm fetching ticker data just fine (including from the endpoint you mentioned).

[nitrous]

Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?

Briefly looking at the source code, it seems that the gadget just requests a URL using XMLHTTP, parses json, and picks out the value specified. Perhaps the problem is that the url you used is redirected to https, try https://data.mtgox.com/api/1/BTCEUR/ticker instead. If that doesn't work, perhaps it has problems with HTTPS?

[glon]

Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?

Briefly looking at the source code, it seems that the gadget just requests a URL using XMLHTTP, parses json, and picks out the value specified. Perhaps the problem is that the url you used is redirected to https, try https://data.mtgox.com/api/1/BTCEUR/ticker instead. If that doesn't work, perhaps it has problems with HTTPS?

Tried https and it now works! Not sure if it's that though, as I'm pretty sure I tried https earlier today and it didn't work then. But the gadget is pretty flakey so who's to say what's really going on with it?

Thanks a lot anyway!

[btc_lurker]

It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868

[nitrous]

It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868

And you can't request it by POST anymore, these unannounced changes will surely break numerous api implementations:
{"result":"error","error":"Invalid request method for this API","token":"invalid_request_method"}

EDIT:
From manual investigation, it seems that the tids jump to 1309108565842636, so when you reach 218868, go to since=1309108565000000 in order to get the next dataset.

[nitrous]

It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868

Do you know the best way to contact the MtGox API team? I've been able to download all USD trade data by performing the jump manually, but this fails for some currencies (e.g. GBP), because the data seems to be very dispersed, and so there are numerous jumps at unpredictable places.

Also, for anyone who wants to download EUR data, it seems that you need to start downloading from 1315289573000000 (I'm not sure if there is any earlier data). I hope there's a way around this, as otherwise anyone who wants to download less popular currency trades will have to do small test jumps repeatedly until they reach the next data 'pocket'!