Bitcoin Forum

Economy => Trading Discussion => Topic started by: nitrous on April 01, 2013, 09:55:03 PM



Title: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 01, 2013, 09:55:03 PM
Hi there,

I've always been interested in making trading bots, however before I had heard of bitcoins and MtGox, I never ended up doing it because the fees needed to do it always seemed prohibitively high, even for Oanda's forex platform. Now that I have found MtGox however, I am now looking forward to automated trading.

As anyone using MtGox's API will know, documentation is only avaiable on the bitcoin wiki (https://en.bitcoin.it/wiki/MtGox/API/HTTP (https://en.bitcoin.it/wiki/MtGox/API/HTTP)), and there are three versions:

Version 0 - The first API, currently deprecated, and soon to be dropped
Version 1 - The most widely used and most well documented
Version 2 - The most recent, though information is sparse

Even for version 1, documentation is far from complete or consistent. I have focused my attention of version 2, however, as from what I have read, MtGox is working to overhaul their API, and so I want to start out on the most recent version, so that I won't have to worry about deprecation of version 1 in the near future. For this reason, I have started a project to document version 2. Although there is published documentation, it only lists most of the methods, and most of them don't even have a description, let alone usage information. Details on how to access the API, including the relevance of the nonce value and the rest key and sign needed, are hard to find, and is mostly achieved through example. I'm sure there is more information distributed throughout this forum and available in IRC chatrooms, however I don't think this is an ideal situation.

That is why I have spent the last few days compiling all the information I have found and worked out into an unofficial document. I intend to eventually cover all the API methods consistently, in as much detail as possible, however I also plan to make it brief, comprehensible, and unambiguous. Although I currently have only covered some of the methods, I have made a lot of headway.

https://bitbucket.org/nitrous/mtgox-api/overview (https://bitbucket.org/nitrous/mtgox-api/overview)

The above is the result of my efforts. As you can see, it explains how the API works, what you need to access it, details of the terms and data used, including currency specifics. I have also given a short example of how to access the API using a key and secret. You will also notice that I have documented the most useful methods for a trading bot. Each method includes a short description, any known arguments it accepts (if applicable), an example of its use if I have been able to obtain one, and bullet points explaining the nuances of each.

I plan to continue extending this as I come across more information, and as I gain experience. I don't currently have any bitcoins in my MtGox account to give examples for the actual trading methods yet (I'm waiting for a significant dip in price), but I'm sure in the near future I will be able to do this as well. Whilst I can't promise regular updates as I am a student and sometimes have a lot of other work, I hope to eventually complete this in its entirety (unless, of course, official documentation comes along with more information, and makes this project obsolete).

Thank you for taking the time to read this, hopefully this will be useful to some of you. Please give me your opinions, and any suggestions for improvements you may have :)



TLDR: I have started work on unofficial documentation for version 2 of MtGox's API here: https://bitbucket.org/nitrous/mtgox-api/overview (https://bitbucket.org/nitrous/mtgox-api/overview). It is not yet complete, but it is freely available for all to use. I welcome your opinions.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 09, 2013, 04:06:02 PM
I just received an anonymous donation :) And I've used it to help update the currently documented methods, notably the actual trading methods, such as adding orders and cancelling them, and getting information on currently open orders. All the currently documented methods should now be complete, however not all the json response key purposes are known. If I have accidentally left any information out, please let me know :)

I don't currently plan on documenting the other methods listed at the bottom in the near future, as I don't have too much free time currently, however when I do I will endeavour to finish the entire documentation.

A list of the main updates is available near the top of the page, here https://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-updates (https://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-updates)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 09, 2013, 04:18:06 PM
good work, I couldn't even find how to obtain a key due to outdated docs pointing me to the wrong location - got it working in perl, now I just have to make it do something useful.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 09, 2013, 04:23:03 PM
good work, I couldn't even find how to obtain a key due to outdated docs pointing me to the wrong location - got it working in perl, now I just have to make it do something useful.

Yeah, the API may be well-featured and easy to use, but it's hard to find reliable usage information. Good luck on your trader, have you got any ideas yet? I'm looking at neural networks currently.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 09, 2013, 04:53:10 PM
all I want right now is to monitor and issue a sell order in case of a large drop :)  kinda weird that isn't a standard feature yet through the web interface.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 09, 2013, 05:24:58 PM
all I want right now is to monitor and issue a sell order in case of a large drop :)  kinda weird that isn't a standard feature yet through the web interface.

Yeah, it's weird that the big exchanges don't support standard exchange features such as order limits, short selling and margin trading. I assume the market just isn't deep enough and volumes aren't big enough to warrant it at the moment. Strange though, as MtGox has a monthly volume in the millions of bitcoins, maybe the features just aren't requested enough by users.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 10, 2013, 08:40:11 AM
Thank you so much! What do you think are the advantages of starting a production-project using the v2 versus using the v1 ?

For instance: during periods of DDoS, do they perform the same way?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 10, 2013, 11:09:54 AM
Thank you so much! What do you think are the advantages of starting a production-project using the v2 versus using the v1 ?

For instance: during periods of DDoS, do they perform the same way?

I think that, when starting any project, it is important to make use of the latest versions:
  • It is based off of the experience of previous versions, so is likely to be more reliable and robust
  • When previous versions are deprecated, you won't need to update your code as often
  • Each new version should be more scalable, so you should see performance benefits

As for DDoS periods, I can't be sure, but I would think it would be better able to function. This is because I don't believe many people are using version 2 currently due to the previous lack of documentation, so there is less load and less incentive to perform a DDoS on the version 2 api. So, assuming that MtGox is running each API on a separate server, it would seem that version 2 will be more resilient.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 12, 2013, 02:35:29 AM
Thank you, I started right away today. When I'm done I'll be happy to contribute with some code snippets.

My application is in Java and I'm having trouble with SSL.  Could you try to help me?

When I run a simple query I got an exception :
Code:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

with google and stackoverflow I understand what the problem was (maybe) . I should "say" to the java virtual machine to trust mtgox's certificate. In order to do so I :
  • downloaded mtgox certificate (file.cer)
  • created a keyStore.ks file with keytool -import -alias mtg -file mtgox.com.cer -keystore goxkeystore.ks
  • set the parameter to the IDE to run the JVM with the option -Djavax.net.ssl.keyStore=goxkeystore.ks

unfortunately when running the application I still got the same exception. Setting the debug channel on SSL I captured the output. Might be useful?
http://pastebin.com/AwXVFZQ9 (http://pastebin.com/AwXVFZQ9)



OT: I didn't want to start from scratch and to build the client I re-used the code from an example of the v1 : https://gist.github.com/christopherobin/2396722 (https://gist.github.com/christopherobin/2396722) . My assumption is that the method to sign is the same as before. Am I right?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 12, 2013, 10:20:55 AM
Thank you, I started right away today. When I'm done I'll be happy to contribute with some code snippets.

My application is in Java and I'm having trouble with SSL.  Could you try to help me?

When I run a simple query I got an exception :
Code:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

with google and stackoverflow I understand what the problem was (maybe) . I should "say" to the java virtual machine to trust mtgox's certificate. In order to do so I :
  • downloaded mtgox certificate (file.cer)
  • created a keyStore.ks file with keytool -import -alias mtg -file mtgox.com.cer -keystore goxkeystore.ks
  • set the parameter to the IDE to run the JVM with the option -Djavax.net.ssl.keyStore=goxkeystore.ks

unfortunately when running the application I still got the same exception. Setting the debug channel on SSL I captured the output. Might be useful?
http://pastebin.com/AwXVFZQ9 (http://pastebin.com/AwXVFZQ9)



OT: I didn't want to start from scratch and to build the client I re-used the code from an example of the v1 : https://gist.github.com/christopherobin/2396722 (https://gist.github.com/christopherobin/2396722) . My assumption is that the method to sign is the same as before. Am I right?

MtGox has changed the signing method very slightly, you have to prepend the post-data with a specific part of the URL (the part that follows the version number, so in 'https://data.mtgox.com/api/2/BTCUSD/money/ticker', it's 'BTCUSD/money/ticker'), followed by the null character, you can fix the signing just by adding a single line and changing one other:
Code:
    public String query(String path, HashMap<String, String> args) {
        try {
            // add nonce and build arg list
            args.put("nonce", String.valueOf(System.currentTimeMillis()));
            String post_data = this.buildQueryString(args);
            
                // new line
                String post_data_mac = path.substring(2) + "\0" + post_data
            
            // args signature
            Mac mac = Mac.getInstance("HmacSHA512");
            SecretKeySpec secret_spec = new SecretKeySpec((new BASE64Decoder()).decodeBuffer(this.secret), "HmacSHA512");
            mac.init(secret_spec);
            
                // changed line
                String signature = (new BASE64Encoder()).encode(mac.doFinal(post_data_mac.getBytes()));

You still need the original post_data as this will be sent as normal in the request.

Try it, hopefully it will fix your SSL problem :) If not, I'll have another look.

EDIT:
Also, make sure that your base url is "https://data.mtgox.com/api/", not "https://mtgox.com/api/", it changed very recently. Additionally, in version 1, the "private/" bit in the URL has been deprecated, but if you're switching to version 2 that doesn't matter.

The change of the base URL should definitely fix your SSL problem, make sure you have the SSL certificate for data.mtgox.com as well though, just in case they differ.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 12, 2013, 11:46:51 AM
Thank you, I started right away today. When I'm done I'll be happy to contribute with some code snippets.

That would be great ;D I've just set up a directory for example implementations here: https://bitbucket.org/nitrous/mtgox-api/src/master/examples (https://bitbucket.org/nitrous/mtgox-api/src/master/examples), and a Java example would be very useful to include. Hopefully at some point there can be a variety of languages to point everyone in the right direction with their respective languages :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 12, 2013, 04:36:42 PM
Thank you a lot for helping ;) Can't wait to get it working asap!

I did the modifications and still got the same error ... Here is the complete code :

http://pastebin.com/4mBrc83n (http://pastebin.com/4mBrc83n)

To test it I'm trying to get my current balance and I call the function without any query_args :

Code:
 GoxJavaExample client = new GoxJavaExample(
                        apyKeys.getApiKey(),
                        apyKeys.getPrivateKey()
                );
                HashMap<String, String> query_args = new HashMap<>();
                client.query("BTCUSD/money/info", query_args);

It might be related with the Hash function not working properly in java?
The SSL error apparently is :

Code:
% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_RC4_128_SHA]
JavaFX Application Thread, SEND TLSv1 ALERT:  fatal, description = certificate_unknown
JavaFX Application Thread, WRITE: TLSv1 Alert, length = 2
JavaFX Application Thread, called closeSocket()
JavaFX Application Thread, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target


Plus, while building I'm receiving the following warning related to the Obsolescence of the BASE64Decoder package :

Code:
/Users/advanced/Dropbox/Botcoin.io/development/Botcoin/src/io/botcoin/api/GoxJavaExample.java:12: warning: BASE64Decoder is internal proprietary API and may be removed in a future release
import sun.misc.BASE64Decoder;
/Users/advanced/Dropbox/Botcoin.io/development/Botcoin/src/io/botcoin/api/GoxJavaExample.java:13: warning: BASE64Encoder is internal proprietary API and may be removed in a future release
import sun.misc.BASE64Encoder;
/Users/advanced/Dropbox/Botcoin.io/development/Botcoin/src/io/botcoin/api/GoxJavaExample.java:55: warning: BASE64Decoder is internal proprietary API and may be removed in a future release
            SecretKeySpec secret_spec = new SecretKeySpec((new BASE64Decoder()).decodeBuffer(this.secret), "HmacSHA512");
/Users/advanced/Dropbox/Botcoin.io/development/Botcoin/src/io/botcoin/api/GoxJavaExample.java:58: warning: BASE64Encoder is internal proprietary API and may be removed in a future release
            String signature = (new BASE64Encoder()).encode(mac.doFinal(post_data_mac.getBytes()));
4 warnings


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 12, 2013, 05:31:43 PM
Thank you a lot for helping ;) Can't wait to get it working asap!

I did the modifications and still got the same error ... Here is the complete code :

http://pastebin.com/4mBrc83n (http://pastebin.com/4mBrc83n)

To test it I'm trying to get my current balance and I call the function without any query_args :

First, now that you've moved the api version into the BASE_URL, path.substring(2) should now just be path (I used substring to trim the '1/' off in your original code).

It might be related with the Hash function not working properly in java?
The SSL error apparently is :

I'm not really sure why you're getting this SSL error, but I haven't used Java in a while. Try -Djavax.net.ssl.trustStore instead of .keyStore.

Plus, while building I'm receiving the following warning related to the Obsolescence of the BASE64Decoder package :

For the base 64 code, apparently sun.misc functions are indeed deprecated, try replacing your base64 calls with the equivalents from here: http://commons.apache.org/proper/commons-codec/ (http://commons.apache.org/proper/commons-codec/)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 13, 2013, 09:51:53 AM
Thanks again.

Ok so , i got rid of the .substring()

Code:
String post_data_mac = path + "\0" + post_data;

 I run the JVM with the trustStore instead of keyStore
Code:
-Djavax.net.ssl.trustStore=resources/goxkeystore.ks -Djavax.net.ssl.trustStorePassword=xxx -Djavax.net.debug=ssl 
,

 and I replaced the Base64 functions with apache's ones

Code:
     // add nonce and build arg list
            args.put("nonce", String.valueOf(System.currentTimeMillis()));
            String post_data = this.buildQueryString(args);
            String post_data_mac = path + "\0" + post_data;
       
            // args signature
            Mac mac = Mac.getInstance("HmacSHA512");
            SecretKeySpec secret_spec = new SecretKeySpec(Base64.decodeBase64(this.secret), "HmacSHA512");
            mac.init(secret_spec);
            String signature = Base64.encodeBase64String(mac.doFinal(post_data_mac.getBytes()));
 
            // build URL
            URL queryUrl = new URL(BASE_URL + path);


Still got the same SSL error... it has to be related to the certificate I downloaded from mtgox.com with chrome. Maybe they use a different one for the data.mtgox.com ?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 13, 2013, 12:27:05 PM
Thanks again.

Ok so , i got rid of the .substring()

Code:
String post_data_mac = path + "\0" + post_data;

 I run the JVM with the trustStore instead of keyStore
Code:
-Djavax.net.ssl.trustStore=resources/goxkeystore.ks -Djavax.net.ssl.trustStorePassword=xxx -Djavax.net.debug=ssl 
,

 and I replaced the Base64 functions with apache's ones

Code:
     // add nonce and build arg list
            args.put("nonce", String.valueOf(System.currentTimeMillis()));
            String post_data = this.buildQueryString(args);
            String post_data_mac = path + "\0" + post_data;
       
            // args signature
            Mac mac = Mac.getInstance("HmacSHA512");
            SecretKeySpec secret_spec = new SecretKeySpec(Base64.decodeBase64(this.secret), "HmacSHA512");
            mac.init(secret_spec);
            String signature = Base64.encodeBase64String(mac.doFinal(post_data_mac.getBytes()));
 
            // build URL
            URL queryUrl = new URL(BASE_URL + path);


Still got the same SSL error... it has to be related to the certificate I downloaded from mtgox.com with chrome. Maybe they use a different one for the data.mtgox.com ?

Ah yes, that must be it, I just checked and mtgox.com is signed by VeriSign, whereas data.mtgox.com is signed by StartCom - two different certificates unfortunately. Go to https://data.mtgox.com (https://data.mtgox.com) and you should be able to get the correct certificate, it might be useful adding it to the same keystore as well, just in case it ever needs both. Hopefully this will finally fix it :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 13, 2013, 01:46:21 PM
hey do you know whether these chart sites, listentobitcoins etc that use websockets relay data from mtgox or just redirect people to mtgox?

if the latter I'm thinking it must be pretty taxing to their bandwidth.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: Grover on April 13, 2013, 02:18:27 PM
all I want right now is to monitor and issue a sell order in case of a large drop :)  kinda weird that isn't a standard feature yet through the web interface.

Yeah, it's weird that the big exchanges don't support standard exchange features such as order limits, short selling and margin trading. I assume the market just isn't deep enough and volumes aren't big enough to warrant it at the moment. Strange though, as MtGox has a monthly volume in the millions of bitcoins, maybe the features just aren't requested enough by users.

This one has those features.

https://www.kraken.com/
https://bitcointalk.org/index.php?topic=159974.0;topicseen


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 13, 2013, 03:01:13 PM
all I want right now is to monitor and issue a sell order in case of a large drop :)  kinda weird that isn't a standard feature yet through the web interface.

Yeah, it's weird that the big exchanges don't support standard exchange features such as order limits, short selling and margin trading. I assume the market just isn't deep enough and volumes aren't big enough to warrant it at the moment. Strange though, as MtGox has a monthly volume in the millions of bitcoins, maybe the features just aren't requested enough by users.

This one has those features.

https://www.kraken.com/
https://bitcointalk.org/index.php?topic=159974.0;topicseen

Kraken looks quite promising, but are you sure it's legitimate? The information on the front page is filled with lorem ipsum text, and its FAQ is not complete yet either. Plus, their price and volume data look suspiciously similar to MtGox's. It might just be that they're still in beta, but I'll definitely follow them if they prove to be trustworthy. Do you know if they have/plan to have an API?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 13, 2013, 03:15:07 PM
hey do you know whether these chart sites, listentobitcoins etc that use websockets relay data from mtgox or just redirect people to mtgox?

if the latter I'm thinking it must be pretty taxing to their bandwidth.


Listentobitcoins is nice, never heard of it before :) I checked their source, and the mtgox connection is client-side, but you have to click the checkbox first, so it probably isn't too taxing at the moment. I also checked bitcoinity, and it appears to cache it server-side for the website to collect by JSON (though in the source, there is reference to websockets, but I don't think it actually connects to MtGox). It depends on the site I guess, but MtGox can cope with it I think, especially as they have anti-DDoS filters in place in case the load gets too taxing. Their main issue is with their trading engine, which they are trying to improve currently.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 13, 2013, 05:33:35 PM
yea I downloaded the source to listentobitcoins too, it only shows transactions from blockchain for me, can't get it to show trades for some reason, it connects but nothing comes through.

I was thinking of using it as a starting point for another little project but don't really want to convert it to socket.io just cos mtgox websocket maybe temporarily dodgy.


---
edit:  never mind just spotted this:

<shockdiode> websocket is officially deprecated now?
<MagicalTux> shockdiode: yep, replaced by socket.io server


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 13, 2013, 06:07:49 PM
edit:  never mind just spotted this:

<shockdiode> websocket is officially deprecated now?
<MagicalTux> shockdiode: yep, replaced by socket.io server


Hmm, ironic that 10 days ago listentobitcoin was switched from socket.io to websocket! I looked at their github repo, and I think this edit https://github.com/MaxLaumeister/Listen-To-Bitcoin/blob/778273ae2a91a89f230bd88302fa69fdee61a7da/socket.js (https://github.com/MaxLaumeister/Listen-To-Bitcoin/blob/778273ae2a91a89f230bd88302fa69fdee61a7da/socket.js) might have what you're looking for.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 13, 2013, 06:16:03 PM
Ah yes, that must be it, I just checked and mtgox.com is signed by VeriSign, whereas data.mtgox.com is signed by StartCom - two different certificates unfortunately. Go to https://data.mtgox.com (https://data.mtgox.com) and you should be able to get the correct certificate, it might be useful adding it to the same keystore as well, just in case it ever needs both. Hopefully this will finally fix it :)

Getting there !  SSL works!

I added the two certificate to the same file with

Code:
keytool -import -alias mtgox-data -file data.mtgox.com.cer -keystore mtgox.jks
keytool -import -alias mtgox-com -file mtgox.com.cer -keystore mtgox.jks

Then set the system property (trust store)

Code:
System.setProperty("javax.net.ssl.trustStore",settings.KEYSTORE_PATH);
System.setProperty("javax.net.ssl.trustStorePassword",settings.KEYSTORE_PWD);

Now it ... works! Well, kind of. I got a 403 instead... :(

Code:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/BTCUSD/money/info


     


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 13, 2013, 08:11:35 PM
Ah yes, that must be it, I just checked and mtgox.com is signed by VeriSign, whereas data.mtgox.com is signed by StartCom - two different certificates unfortunately. Go to https://data.mtgox.com (https://data.mtgox.com) and you should be able to get the correct certificate, it might be useful adding it to the same keystore as well, just in case it ever needs both. Hopefully this will finally fix it :)

Getting there !  SSL works!

I added the two certificate to the same file with

Code:
keytool -import -alias mtgox-data -file data.mtgox.com.cer -keystore mtgox.jks
keytool -import -alias mtgox-com -file mtgox.com.cer -keystore mtgox.jks

Then set the system property (trust store)

Code:
System.setProperty("javax.net.ssl.trustStore",settings.KEYSTORE_PATH);
System.setProperty("javax.net.ssl.trustStorePassword",settings.KEYSTORE_PWD);

Now it ... works! Well, kind of. I got a 403 instead... :(

Code:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/BTCUSD/money/info

Ah good, it's connecting properly now :) The reason it gives a 403 error is likely that the HMAC is wrong  or not being sent, try setting the path to 'BTCUSD/money/ticker' - this doesn't care about the authentication and should work, just to check that it can actually get a valid response, then:
  • Set your secret to "SGVsbG8gV29ybGQh", then send me your post_data_mac and signature results so that I can check your HMAC is working correctly, although it seems like your code is correct.
  • Are both your API key and secret valid? Consider generating a new pair, and making sure all the rights are selected.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 13, 2013, 09:57:51 PM
Hmm, ironic that 10 days ago listentobitcoin was switched from socket.io to websocket! I looked at their github repo, and I think this edit https://github.com/MaxLaumeister/Listen-To-Bitcoin/blob/778273ae2a91a89f230bd88302fa69fdee61a7da/socket.js (https://github.com/MaxLaumeister/Listen-To-Bitcoin/blob/778273ae2a91a89f230bd88302fa69fdee61a7da/socket.js) might have what you're looking for.

nice, that works thanks.. I wonder why he switched.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 14, 2013, 02:12:50 PM
Ah yes, that must be it, I just checked and mtgox.com is signed by VeriSign, whereas data.mtgox.com is signed by StartCom - two different certificates unfortunately. Go to https://data.mtgox.com (https://data.mtgox.com) and you should be able to get the correct certificate, it might be useful adding it to the same keystore as well, just in case it ever needs both. Hopefully this will finally fix it :)

Getting there !  SSL works!

I added the two certificate to the same file with

Code:
keytool -import -alias mtgox-data -file data.mtgox.com.cer -keystore mtgox.jks
keytool -import -alias mtgox-com -file mtgox.com.cer -keystore mtgox.jks

Then set the system property (trust store)

Code:
System.setProperty("javax.net.ssl.trustStore",settings.KEYSTORE_PATH);
System.setProperty("javax.net.ssl.trustStorePassword",settings.KEYSTORE_PWD);

Now it ... works! Well, kind of. I got a 403 instead... :(

Code:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/BTCUSD/money/info


      

MtGox has changed their API so that you are required to use a key and secret for every method. Also, if you are still having errors, your nonce could not be working properly, so do try generating a new api key/secret.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: bittencoin on April 15, 2013, 08:01:43 AM
Hi Nitrous,

I tried to read and follow your guide to try to pull up some data, but I am getting errors in the response when I just try to get the ticker info. 

array(3) { ["result"]=> string(5) "error" ["error"]=> string(20) "Chosen API not found" ["token"]=> string(13) "unknown_error" }

Basically, all I did was taking the example php template and put in my api key and secret (not shown here), and changed the base URL to
Code:
https://data.mtgox.com/api/2/
and changed the path to
Code:
BTCUSD/money/ticker
but the var dump says Chosen API not found.  I tested with many new api key/secret codes, but that appears not to be the source of the problem.  Any insight is appreciated.  Thank you.


Code:
<?php
 
function mtgox_query($path, array $req = array()) {
// API settings
$key 'my valid key';
$secret 'my secret code';
 
// generate a nonce as microtime, with as-string handling to avoid problems with 32bits systems
$mt explode(' 'microtime());
$req['nonce'] = $mt[1].substr($mt[0], 26);
 
// generate the POST data string
$post_data http_build_query($req'''&');
 
$prefix '';
if (substr($path02) == '2/') {
$prefix substr($path2)."\0";
}
 
// generate the extra headers
$headers = array(
'Rest-Key: '.$key,
'Rest-Sign: '.base64_encode(hash_hmac('sha512'$prefix.$post_database64_decode($secret), true)),
);
 
// our curl handle (initialize if required)
static $ch null;
if (is_null($ch)) {
$ch curl_init();
curl_setopt($chCURLOPT_RETURNTRANSFERtrue);
curl_setopt($chCURLOPT_USERAGENT'Mozilla/4.0 (compatible; MtGox PHP client; '.php_uname('s').'; PHP/'.phpversion().')');
}
curl_setopt($chCURLOPT_URL'https://data.mtgox.com/api/2/'.$path); //Base URL
curl_setopt($chCURLOPT_POSTFIELDS$post_data);
curl_setopt($chCURLOPT_HTTPHEADER$headers);
curl_setopt($chCURLOPT_SSL_VERIFYPEERFALSE);
 
// run the query
$res curl_exec($ch);
if ($res === false) throw new Exception('Could not get reply: '.curl_error($ch));
$dec json_decode($restrue);
if (!$dec) throw new Exception('Invalid data received, please make sure connection is working and requested API exists');
return $dec;
}
 
// example 1: get infos about the account, plus the list of rights we have access to
var_dump(mtgox_query('BTCUSD/money/ticker'));// PATH
 
// old api (get funds)
//var_dump(mtgox_query('0/getFunds.php'));
 
// trade example
// var_dump(mtgox_query('0/buyBTC.php', array('amount' => 1, 'price' => 15)));


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 10:11:24 AM
I generated a new pair of keys and it works. Weird, I'm using the other pair with the v1 api and another node.js bot.

Now I have another problem related with reading the http answer : It only gets printed the first line

Code:
{"result":"success","data": 
 
and then it stops .


This is the code I'm currently using, taken from the wiki example of the v1:

Code:
// write post
            connection.getOutputStream().write(post_data.getBytes());
            // read info
            byte buffer[] = new byte[16384];
            int len = connection.getInputStream().read(buffer, 0, 16384);
            System.out.print(new String(buffer, 0, len, "UTF-8"));

Quote
MtGox has changed their API so that you are required to use a key and secret for every method.
For every method? I'm not sure about the actual meaning of this. Should I generate a pair of keys for each method? one for orders, one for getinfo, one for withdraw .... ?  And how I do the association keypair <---> method ?


Sounds crazy ! ;|

Thank you a lot for the help, I feel like we are getting closer. I will write a Trading Class for the v2 and make it open asap!





Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 15, 2013, 10:43:19 AM
I went through this recently in perl, hopefully you can make sense of it.

$ticker_ref = __request('BTCUSD/money/ticker');

sub __request {
   my ($method, $args) = @_;

   $args = "" unless defined $args;

   my $nonce = 'nonce=' . sprintf "%d", gettimeofday * 1e6;

   my $hash_data = $method . chr(0) . $nonce . $args;

   my $req = HTTP::Request->new(POST => 'https://data.mtgox.com/api/2/' . $method );
   $req->content_type('application/x-www-form-urlencoded');
   $req->content($nonce . $args);
   $req->header('Rest-Key' => $key);
   $req->header('Rest-Sign' => encode_base64( hmac_sha512( $hash_data, decode_base64($secret) ) ));
 
    my $result = $lwp->request($req);

   unless ( $result->is_success ) {
       __msg "failed $method";
      exit 1;
   };

   return $json->decode( $result->content );

};


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 11:35:29 AM
I have two additional un-related questions :

is there a method to request withdrawal like in the v1 of the API (https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Withdraw_bitcoins (https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Withdraw_bitcoins))? I can't see it in your doku but I've found in the official pdf :

Quote
4.4 send simple
URL: https://mtgox.com/api/2/money/bitcoin/send_simple
Calling this method requires to be identified with an API key with the
withdraw right.
List of parameters address Target bitcoin address
7

amount int Amount of bitcoins to withdraw
fee int Fee amount to be added to transaction (optional), maximum 0.01
BTC
no instant Setting this parameter to 1 will prevent transaction from being processed internally, and force usage of the bitcoin blockchain even if receipient is also on the system
green Setting this parameter to 1 will cause the TX to use MtGox’s green address
Send bitcoins to a given address
This API allows sending a given number of bitcoins to any bitcoin address. Transfers to bitcoin address on the same system are processed instantly, unless the no instant flag is set.
On success, this method will return the transaction id (in offser trx ) which will contain either the bitcoin transaction id as hexadecimal or a UUID value in case of internal transfer.

In the  Wiki they wrote :
Quote
BTCUSD/money/ticker_fast
Solves the problem of ticker lag. (supposedly)
Did you try it? Is not in your doku ... How come it solves the lag problem?


PS: who is the software architect that returns the balance in EUR in a function that is called from "BTCUSD/money/info" ? :\


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 01:43:07 PM
Hi Nitrous,

I tried to read and follow your guide to try to pull up some data, but I am getting errors in the response when I just try to get the ticker info. 

array(3) { ["result"]=> string(5) "error" ["error"]=> string(20) "Chosen API not found" ["token"]=> string(13) "unknown_error" }

Basically, all I did was taking the example php template and put in my api key and secret (not shown here), and changed the base URL to
Code:
https://data.mtgox.com/api/2/
and changed the path to
Code:
BTCUSD/money/ticker
but the var dump says Chosen API not found.  I tested with many new api key/secret codes, but that appears not to be the source of the problem.  Any insight is appreciated.  Thank you.



Hi bittencoin,

You are correct that the API code and secret are not the problem, the problem is with this part of your code:

Code:
	$prefix = '';
if (substr($path, 0, 2) == '2/') {
$prefix = substr($path, 2)."\0";
}
 
// generate the extra headers
$headers = array(
'Rest-Key: '.$key,
'Rest-Sign: '.base64_encode(hash_hmac('sha512', $prefix.$post_data, base64_decode($secret), true)),
);
 

Your prefix is empty in a normal scenario, instead, you should change this code to the following:
Code:
	$prefix = $path;
if (substr($path, 0, 2) == '2/') {
$prefix = substr($path, 2);
}
 
// generate the extra headers
$headers = array(
'Rest-Key: '.$key,
'Rest-Sign: '.base64_encode(hash_hmac('sha512', $prefix."\0".$post_data, base64_decode($secret), true)),
);

I have made sure your prefix variable contains the path, and I've moved the null character into the rest-sign to make sure it's always included. This code worked for me after doing those fixes :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 02:05:44 PM
I generated a new pair of keys and it works. Weird, I'm using the other pair with the v1 api and another node.js bot.

Now I have another problem related with reading the http answer : It only gets printed the first line

Code:
{"result":"success","data": 
 
and then it stops .


This is the code I'm currently using, taken from the wiki example of the v1:

Code:
// write post
            connection.getOutputStream().write(post_data.getBytes());
            // read info
            byte buffer[] = new byte[16384];
            int len = connection.getInputStream().read(buffer, 0, 16384);
            System.out.print(new String(buffer, 0, len, "UTF-8"));

Quote
MtGox has changed their API so that you are required to use a key and secret for every method.
For every method? I'm not sure about the actual meaning of this. Should I generate a pair of keys for each method? one for orders, one for getinfo, one for withdraw .... ?  And how I do the association keypair <---> method ?


Sounds crazy ! ;|

Thank you a lot for the help, I feel like we are getting closer. I will write a Trading Class for the v2 and make it open asap!


Sorry about that misunderstanding, no you don't need individual API keys for each method! You can use the same API key, it just seems that some methods complain or display undefined behaviour when issued with an invalid/empty API key, whereas they used to be more lenient (e.g. money/ticker).

As for the short response, that's strange as the response is usually all on one line anyway, so there shouldn't be a problem. Perhaps the server simply hasn't had time to send the entire JSON response yet, as the read() method doesn't necessarily wait for EOF, try using something along these lines instead:
Code:
byte current;
byte buffer[] = new byte[16384];
int len = 0;
InputStream s = connection.getInputStream();
while (true) {
    current = s.read();
    if (current == -1) break;
    buffer[len] = current;
    len++;
}
System.out.print(new String(buffer, 0, len, "UTF-8"));

Note - I haven't tested the above code, so it might not work as written, but the essence is to keep trying to read from the connection (in this case, byte by byte, although I'm sure you could adapt it to read larger chunks at a time), until EOF is reached. Note also that some methods, e.g. money/depth/full, request more than 16384 bytes (full depth can be as large as 1mb, maybe even bigger), so you will need to watch out for that and maybe extend the buffer if necessary.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 02:18:45 PM
Thank you, I made some progress in the meantime (current code -> http://pastebin.com/RXQadJPA (http://pastebin.com/RXQadJPA)) :

I changed the method to print the HTTP request with something similar to your code (

Code:
   if (connection.getResponseCode() != 200) {
System.err.println("Failed : HTTP error code : "
+ connection.getResponseCode());
}           
           
            BufferedReader br = new BufferedReader(new InputStreamReader((connection.getInputStream())));
           
            String output;
            System.out.println("HTTP response \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
                        answer+=output;
}
            connection.disconnect();

Now the ticker response works like a charm and gets printed, while I get a 403 when trying to access my informations .

Code:
Failed : HTTP error code : 403
Apr 15, 2013 4:09:58 PM io.botcoin.api.MtGox query
java.io.IOException: Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/BTCUSD/money/info

This is weird: with the old method to read the answer, the beginning seems to be "response:success " ...

meh... Should I try to sign the request with the private key you indicated and provide it to you so we can verify if it is correctly generated?





Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 02:25:05 PM
here is what I get using this as secret key : SGVsbG8gV29ybGQh

Code:
post_data_mac : BTCUSD/money/infononce=1366035867142
signature: ZprNub9nISb5NdCig1LMXI9VYCdlbV/3/klZ78EJXHsTASg12xvWEZyrUTaVw5MUcJQtDMf1LrbguKka6BDlMA==


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 02:35:01 PM
I went through this recently in perl, hopefully you can make sense of it.

Code:
$ticker_ref = __request('BTCUSD/money/ticker'); 

sub __request {
my ($method, $args) = @_;

$args = "" unless defined $args;

my $nonce = 'nonce=' . sprintf "%d", gettimeofday * 1e6;

my $hash_data = $method . chr(0) . $nonce . $args;

my $req = HTTP::Request->new(POST => 'https://data.mtgox.com/api/2/' . $method );
$req->content_type('application/x-www-form-urlencoded');
$req->content($nonce . $args);
$req->header('Rest-Key' => $key);
$req->header('Rest-Sign' => encode_base64( hmac_sha512( $hash_data, decode_base64($secret) ) ));
 
  my $result = $lwp->request($req);

unless ( $result->is_success ) {
__msg "failed $method";
exit 1;
};

return $json->decode( $result->content );

};

Before you exit, `print $result->content` - when I did Perl complained to me about SSL, but the warning could be disabled by inserting the following line:
Code:
$ENV{'PERL_LWP_SSL_VERIFY_HOSTNAME'} = 0;
(This is only temporary however, if you get an SSL warning you should install the certificate for data.mtgox.com to be safe)

I also recommend making sure you join your arguments to your nonce properly using the following, and updating it wherever necessay:
Code:
my $post_data = $nonce . ($args == '' ? '' : '&' . $args);
my $hash_data = $method . chr(0) . $post_data;

It didn't matter in this case but it might solve some problems later on. The rest of your code seems fine - I was able to get a response. Make sure you're using all the modules you need, I needed the following to get your code to work:
Code:
use Time::HiRes qw(gettimeofday);
use Digest::SHA qw( hmac_sha512 );
use HTTP::Request::Common qw(POST GET);
use MIME::Base64;
use LWP::UserAgent;

And if it still doesn't work, send me the result of printing $result->content and your $post_data


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 02:46:50 PM
I have two additional un-related questions :

is there a method to request withdrawal like in the v1 of the API (https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Withdraw_bitcoins (https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Withdraw_bitcoins))? I can't see it in your doku but I've found in the official pdf :

Quote
4.4 send simple
URL: https://mtgox.com/api/2/money/bitcoin/send_simple
Calling this method requires to be identified with an API key with the
withdraw right.
List of parameters address Target bitcoin address
7

amount int Amount of bitcoins to withdraw
fee int Fee amount to be added to transaction (optional), maximum 0.01
BTC
no instant Setting this parameter to 1 will prevent transaction from being processed internally, and force usage of the bitcoin blockchain even if receipient is also on the system
green Setting this parameter to 1 will cause the TX to use MtGox’s green address
Send bitcoins to a given address
This API allows sending a given number of bitcoins to any bitcoin address. Transfers to bitcoin address on the same system are processed instantly, unless the no instant flag is set.
On success, this method will return the transaction id (in offser trx ) which will contain either the bitcoin transaction id as hexadecimal or a UUID value in case of internal transfer.

In the  Wiki they wrote :
Quote
BTCUSD/money/ticker_fast
Solves the problem of ticker lag. (supposedly)
Did you try it? Is not in your doku ... How come it solves the lag problem?


PS: who is the software architect that returns the balance in EUR in a function that is called from "BTCUSD/money/info" ? :\

Yes there are quite a few methods I haven't got round to documenting yet as they can take up quite a bit of time. As I'm not affiliated with MtGox I don't have access to their protocols, so I have to do detective work :) I should probably contact MagicalTux to get the official information. I don't have too much spare time at the moment though, exams and such, so I probably won't be able to make too much progress on the rest of the documentation (but if there's a specific method you need out of those I'm aware of - https://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-other-methods (https://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-other-methods) - then I'll have a look).

I was not aware of ticker_fast though, must be new in response to recent events, I'll try to add that to the documentation ASAP, the ticker methods are quite simple anyway, they don't seem to accept any arguments so it shouldn't take me too long. I imagine it supersedes the current ticker and connects the same way the websocket does, which is claimed to be realtime. I'll compare the two if I notice MtGox's lag increasing substantially again.

BTCUSD/money/info isn't technically the method, the actual method is money/info, but it can be accessed with a currency context as well. money/info gives your account details, including all the currency wallets you have, so if you have an EUR wallet, it will show up, as well as your BTC wallet, and any other currencies.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 02:50:49 PM
here is what I get using this as secret key : SGVsbG8gV29ybGQh

Code:
post_data_mac : BTCUSD/money/infononce=1366035867142
signature: ZprNub9nISb5NdCig1LMXI9VYCdlbV/3/klZ78EJXHsTASg12xvWEZyrUTaVw5MUcJQtDMf1LrbguKka6BDlMA==

Ok, your hmac code is definitely working properly :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 03:08:47 PM
Thank you, I made some progress in the meantime (current code -> http://pastebin.com/RXQadJPA (http://pastebin.com/RXQadJPA)) :

I changed the method to print the HTTP request with something similar to your code (

Code:
   if (connection.getResponseCode() != 200) {
System.err.println("Failed : HTTP error code : "
+ connection.getResponseCode());
}           
           
            BufferedReader br = new BufferedReader(new InputStreamReader((connection.getInputStream())));
           
            String output;
            System.out.println("HTTP response \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
                        answer+=output;
}
            connection.disconnect();

Now the ticker response works like a charm and gets printed, while I get a 403 when trying to access my informations .

Code:
Failed : HTTP error code : 403
Apr 15, 2013 4:09:58 PM io.botcoin.api.MtGox query
java.io.IOException: Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/BTCUSD/money/info

This is weird: with the old method to read the answer, the beginning seems to be "response:success " ...

meh... Should I try to sign the request with the private key you indicated and provide it to you so we can verify if it is correctly generated?

Do you know what the response is that comes with the 403 error? Your function should still generate and return the answer, right?

There are quite a few things that can cause it, but from what you've told me it seems pretty clear that your api key and secret are correct, and your nonce is always increasing. Your API key doesn't even need any rights to get a successful response (but it won't give you any information if you don't give it the rights).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 15, 2013, 03:35:10 PM
403 probably means failed authentication - I don't think the post data should contain the method should it? (as shown above)

mine works with just nonce + any other arguments.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 15, 2013, 03:41:14 PM
Before you exit, `print $result->content` - when I did Perl complained to me about SSL, but the warning could be disabled by inserting the following line:

the code I pasted was working - I was just showing others who were having problems :]  thanks though.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 15, 2013, 03:47:24 PM
I also recommend making sure you join your arguments to your nonce properly using the following, and updating it wherever necessay:
Code:
my $post_data = $nonce . ($args == '' ? '' : '&' . $args);
my $hash_data = $method . chr(0) . $post_data;

I think it's ok as it is - the ? doesn't seem to be required in post data and I'm supplying the args in full:

print Dumper __request('BTCUSD/money/order/add', "&type=$type&amount_int=$amount");

I've created and cancelled orders with it anyway so seems fine.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 03:52:03 PM
Thanks, as always!

I've found out about that method by reading through the pdfs @ https://github.com/MtGox/mtgox-doc/tree/master/api (https://github.com/MtGox/mtgox-doc/tree/master/api) , it seems quite useful.

Quote
BTCUSD/money/info isn't technically the method, the actual method is money/info, but it can be accessed with a currency context as well. money/info gives your account details, including all the currency wallets you have, so if you have an EUR wallet, it will show up, as well as your BTC wallet, and any other currencies

Are you sure? I tried removing the BTCUSD from the ticker ending with (money/ticker) and I get a http 500.


Quote
Do you know what the response is that comes with the 403 error? Your function should still generate and return the answer, right?

Actually I'm afraid that I can't get the answer as the instruction to read the http inputstream throws the IOException... I'll try to go back to the buffer example to be able to read the answer from there


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 03:57:46 PM
I also recommend making sure you join your arguments to your nonce properly using the following, and updating it wherever necessay:
Code:
my $post_data = $nonce . ($args == '' ? '' : '&' . $args);
my $hash_data = $method . chr(0) . $post_data;

I think it's ok as it is - the ? doesn't seem to be required in post data and I'm supplying the args in full:

print Dumper __request('BTCUSD/money/order/add', "&type=$type&amount_int=$amount");

I've created and cancelled orders with it anyway so seems fine.


Oh sorry, yeah if you pass it with a leading ampersand it's most definitely fine, yeah that works great then  :)

403 probably means failed authentication - I don't think the post data should contain the method should it? (as shown above)

mine works with just nonce + any other arguments.


The API can return 403 for quite a few different reasons (wrong API key, bad nonce, etc), but it usually hints in the JSON the specific reason for the error.
post_data_mac is the equivalent of your $hash_data


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 04:03:49 PM
Thanks, as always!

I've found out about that method by reading through the pdfs @ https://github.com/MtGox/mtgox-doc/tree/master/api (https://github.com/MtGox/mtgox-doc/tree/master/api) , it seems quite useful.

Quote
BTCUSD/money/info isn't technically the method, the actual method is money/info, but it can be accessed with a currency context as well. money/info gives your account details, including all the currency wallets you have, so if you have an EUR wallet, it will show up, as well as your BTC wallet, and any other currencies

Are you sure? I tried removing the BTCUSD from the ticker ending with (money/ticker) and I get a http 500.


Quote
Do you know what the response is that comes with the 403 error? Your function should still generate and return the answer, right?

Actually I'm afraid that I can't get the answer as the instruction to read the http inputstream throws the IOException... I'll try to go back to the buffer example to be able to read the answer from there

With money/ticker, a currency context is required as it returns information specific to a given currency, but money/info does not require a context as it is independent of the currency - it just returns an informative overview of your entire account (all wallets inclusive), so you can call BTCUSD/money/info if you want, but it's really just redirecting to money/info.

Could you catch the exception or is the response data completely blocked from you? The JSON response should contain an error key pointing to the problem, although it can be somewhat cryptic.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 04:24:06 PM
All right, I'll try to get that JSON right away. In the meantime another question: is it correct that the HTTP requests are GETs, right?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 04:27:51 PM
All right, I'll try to get that JSON right away. In the meantime another question: is it correct that the HTTP requests are GETs, right?

I should have spotted that! No, it's POST. I'm surprised you managed to get money/ticker working without POST but yes, you need to send the post_data by POST during the connection.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 15, 2013, 04:42:00 PM
ah, really? :\  ??? Writing this client is driving me nuts.


This is the behaviour at the moment :
Doing nothing or specifing the request method with this line :
Code:
connection.setRequestMethod("GET");
allows me to read the ticker and gives me the 403 on my info.

Using post, on the other hand, stars giving me a HTTP 500 on the ticker! (and still 403 on the info)
Code:
connection.setRequestMethod("POST");
/code]



This is everything I was able to get so far... unfortunately not any JSON.



Code:
Failed : HTTP error code : 403 -
Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/money/info
Forbidden("java.net.SocketPermission" "data.mtgox.com:80" "connect,resolve")
Apr 15, 2013 6:39:36 PM io.botcoin.api.MtGox query


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 15, 2013, 05:06:30 PM
if only the world would settle on 1 language.  :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: bittencoin on April 15, 2013, 05:30:52 PM
Hi Nitrous,

I tried to read and follow your guide to try to pull up some data, but I am getting errors in the response when I just try to get the ticker info.  

array(3) { ["result"]=> string(5) "error" ["error"]=> string(20) "Chosen API not found" ["token"]=> string(13) "unknown_error" }

Basically, all I did was taking the example php template and put in my api key and secret (not shown here), and changed the base URL to
Code:
https://data.mtgox.com/api/2/
and changed the path to
Code:
BTCUSD/money/ticker
but the var dump says Chosen API not found.  I tested with many new api key/secret codes, but that appears not to be the source of the problem.  Any insight is appreciated.  Thank you.



Hi bittencoin,

You are correct that the API code and secret are not the problem, the problem is with this part of your code:

Code:
	$prefix = '';
if (substr($path, 0, 2) == '2/') {
$prefix = substr($path, 2)."\0";
}
 
// generate the extra headers
$headers = array(
'Rest-Key: '.$key,
'Rest-Sign: '.base64_encode(hash_hmac('sha512', $prefix.$post_data, base64_decode($secret), true)),
);
 

Your prefix is empty in a normal scenario, instead, you should change this code to the following:
Code:
	$prefix = $path;
if (substr($path, 0, 2) == '2/') {
$prefix = substr($path, 2);
}
 
// generate the extra headers
$headers = array(
'Rest-Key: '.$key,
'Rest-Sign: '.base64_encode(hash_hmac('sha512', $prefix."\0".$post_data, base64_decode($secret), true)),
);

I have made sure your prefix variable contains the path, and I've moved the null character into the rest-sign to make sure it's always included. This code worked for me after doing those fixes :)

Thank you Nitrous.  I will test it out and report back.  But by looking at the original template code from mtgox, I can see why it does not work now.  

This block of code is suppose to check if the "path" is api version 2
Code:
$prefix = '';
if (substr($path, 0, 2) == '2/') {
$prefix = substr($path, 2)."\0";
}

the check is supposed to look for the "2/" at end of the string, but in this case the $path variable is "BTCUSD/money/ticker" and does not contain the "2/".  If I am correct, the original code tried to look for the "2/" in the wrong string, it should be looking for the version in the base url
Code:
https://data.mtgox.com/api/2/
using the code
Code:
substr($path, -2, 2) == '2/'
.  So if that is fixed, then you don't need to hard code the null character into the REST sign.  

UPDATE:

Here is my fix of that block of code

Code:
$prefix = '';
$baseurl = 'https://data.mtgox.com/api/2/';
if (substr($baseurl, -2, 2) == '2/') {
$prefix = $path."\0";

}


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 06:22:56 PM

Thank you Nitrous.  I will test it out and report back.  But by looking at the original template code from mtgox, I can see why it does not work now.  

This block of code is suppose to check if the "path" is api version 2
Code:
$prefix = '';
if (substr($path, 0, 2) == '2/') {
$prefix = substr($path, 2)."\0";
}

the check is supposed to look for the "2/" at end of the string, but in this case the $path variable is "BTCUSD/money/ticker" and does not contain the "2/".  If I am correct, the original code tried to look for the "2/" in the wrong string, it should be looking for the version in the base url
Code:
https://data.mtgox.com/api/2/
using the code
Code:
substr($path, -2, 2) == '2/'
.  So if that is fixed, then you don't need to hard code the null character into the REST sign.  

UPDATE:

Here is my fix of that block of code

Code:
$prefix = '';
$baseurl = 'https://data.mtgox.com/api/2/';
if (substr($baseurl, -2, 2) == '2/') {
$prefix = $path."\0";

}

Ok, let me know how it goes, your code is very close to working, the error you received was simply not signing the request properly, which the above should fix :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 06:31:52 PM
ah, really? :\  ??? Writing this client is driving me nuts.


This is the behaviour at the moment :
Doing nothing or specifing the request method with this line :
Code:
connection.setRequestMethod("GET");
allows me to read the ticker and gives me the 403 on my info.

Using post, on the other hand, stars giving me a HTTP 500 on the ticker! (and still 403 on the info)
Code:
connection.setRequestMethod("POST");
/code]



This is everything I was able to get so far... unfortunately not any JSON.



Code:
Failed : HTTP error code : 403 -
Server returned HTTP response code: 403 for URL: https://data.mtgox.com/api/2/money/info
Forbidden("java.net.SocketPermission" "data.mtgox.com:80" "connect,resolve")
Apr 15, 2013 6:39:36 PM io.botcoin.api.MtGox query

Your code is so close to working, which makes it that much more frustrating. It must be some small, easy to miss, error. Anyway, I quickly wrote this using my own basic API as a reference: http://pastebin.com/SmrxuJQj

I was able to get it to work, so if you compare each line hopefully you'll finally find whatever's causing the problem :) Also, I found a way to get the JSON even with the IOException: declare the query function as throwing java.io.Exception, then you can check the response code of the connection - if it is >= 400, use getErrorStream instead of getInputStream, hopefully this will help you debug.

Note: I added "000" to the nonce because my api uses miroseconds, and otherwise I would be rejected by the server, feel free to remove them as your nonces are milliseconds.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 15, 2013, 06:47:53 PM
not sure if it's an issue with v2 but others have had probs with their nonce not being long enough, something to keep in mind - but remember if you fiddle with it, you can't make it shorter again :)

don't know the requirement but mine's working at 16 chars.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 15, 2013, 07:13:13 PM
not sure if it's an issue with v2 but others have had probs with their nonce not being long enough, something to keep in mind - but remember if you fiddle with it, you can't make it shorter again :)

don't know the requirement but mine's working at 16 chars.


Yeah I noticed that recently, but testing today with a separate API key I found 13 also seems to work, so millisecond time should also be ok. I'll put it on my todo list to find the minimum length at some point :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 09:58:28 AM
Thanks! My today's resolution is : don't go to sleep unless it works.

So, lets try to capture exceptions and errors:

Code:
            if (connection.getResponseCode() != 200) {
                 BufferedReader errBr = new BufferedReader(new InputStreamReader((connection.getErrorStream())));
               
System.err.println("Failed : HTTP error code : "
+ connection.getResponseCode() + " \n Response message:"
                                        + connection.getResponseMessage() + " \n Permission :"
                                        + connection.getPermission());
                         String output;
                         System.out.println("Error Stream: : \n"); //TODO Remove
                         while ((output = errBr.readLine()) != null) {
                            System.err.println(output);

                    }                                         
}           
             

And here we go baby :

Code:
Failed : HTTP error code : 403 
Response message:Forbidden
Permission :("java.net.SocketPermission" "data.mtgox.com:80" "connect,resolve")
Error Stream: {"result":"error","error":"Identification required to access private API","token":"login_error_missing_nonce"}



missing nonce?



Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 10:09:34 AM
update : the call to the ticker works only via GET


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 16, 2013, 10:14:00 AM
works via POST for me - should paste the rest of the code in question.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 10:23:32 AM
Spotted new difference between your code

Code:
HttpsURLConnection c = (HttpsURLConnection)query.openConnection();

and my code

Code:
HttpURLConnection connection = (HttpURLConnection)query.openConnection();


still missing_nonce_error.


Plus, I don't understand howcome it works for you without specifying the keystore file .... :\ To make your code work I had to add the keystore to it! (And yes, it works with the same api keys I'm try to use)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 10:25:14 AM
works via POST for me - should paste the rest of the code in question.


I receive this error when "POSTing" it : {"result":"error","error":"Invalid request method for this API","token":"invalid_request_method"}


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 16, 2013, 10:50:53 AM
I've only posted perl code.. is that what you ran?

paste your java equivalent of the following:


my $nonce = 'nonce=' . sprintf "%d", gettimeofday * 1e6;
my $hash_data = $method . chr(0) . $nonce . $args;

my $req = HTTP::Request->new(POST => 'https://data.mtgox.com/api/2/' . $method );
$req->content_type('application/x-www-form-urlencoded');
$req->content($nonce . $args);
$req->header('Rest-Key' => $key);
$req->header('Rest-Sign' => encode_base64( hmac_sha512( $hash_data, decode_base64($secret) ) ));

my $result = $lwp->request($req);


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 10:52:30 AM
I've only posted perl code.. is that what you ran?

paste your java equivalent of the following:


my $nonce = 'nonce=' . sprintf "%d", gettimeofday * 1e6;
my $hash_data = $method . chr(0) . $nonce . $args;

my $req = HTTP::Request->new(POST => 'https://data.mtgox.com/api/2/' . $method );
$req->content_type('application/x-www-form-urlencoded');
$req->content($nonce . $args);
$req->header('Rest-Key' => $key);
$req->header('Rest-Sign' => encode_base64( hmac_sha512( $hash_data, decode_base64($secret) ) ));

my $result = $lwp->request($req);


Hi Mike, thanks for your support. However, if you go back in this conversation you can follow the track of the java discussion .


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 11:02:57 AM
I ran your code, inspected it line by line, tried to go back to deprecated Base64 functions, but nothing.

The only difference I was able to spot is the way you (don't) encode the post data arguments. I'm trying to use this function to add the arguments, which maybe does something weird to the request . I don't see how anyway :


Here is how I call the query and add the argument :


Code:
        String urlPath = "money/info";
        HashMap<String, String> query_args = new HashMap<>();  
        String queryResult = query(urlPath, query_args);

Code:
 public String query(String path, HashMap<String, String> args) {
.//here args should be empty
.
.
  args.put("nonce", nonce);        
//here the nonce is added to the args
  String hash_data = path + "\0" + this.buildQueryString(args);
//here the nonce should be added to hash_data: indeed if I print I have money/info◙nonce=1366110562837
.
.
.
}


And below is the function buildQueryString


Code:
   //Build the query string given a set of query parameters
     protected String buildQueryString(HashMap<String, String> args) {
        String result = new String();
        for (String hashkey : args.keySet()) {
            if (result.length() > 0) result += '&';
            try {
                result += URLEncoder.encode(hashkey, "UTF-8") + "="
                        + URLEncoder.encode(args.get(hashkey), "UTF-8");
            } catch (Exception ex) {
                Logger.getLogger(MtGox.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
        return result;
    }


I wonder if is something related to the ◙  symbol, but I see it in your hash_data as well!


EDIT: nope, I tried to build the string manually and I get the same damn 'missing nonce' error.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 16, 2013, 11:20:36 AM
I see in the pastebin java code that this is being assigned:

Code:
String post_data = this.buildQueryString(args);

but not used except as part of 'Rest-Sign'


Code:
$req->content($nonce . $args);

$hash_data = $method . chr(0) . $nonce . $args;
$req->header('Rest-Sign' => encode_base64( hmac_sha512( $hash_data, decode_base64($secret) ) ));

notice the nonce and args are being used in both 'Rest-Sign' and 'content' - this content is needed with POST but not GET where it forms part of the url.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 16, 2013, 11:29:08 AM
doesn't help that I've only done a bit of maintenance of java code but are you doing the following for the post?  seems this 'content' i'm referring is sent afterwards via DataOutputStream.

so ugly compared to perl.. may as well be using C :)

Code:
String urlParameters = "param1=a&param2=b&param3=c";
String request = "http://example.com/index.php";
URL url = new URL(request);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();          
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("Content-Length", "" + Integer.toString(urlParameters.getBytes().length));
connection.setUseCaches (false);

DataOutputStream wr = new DataOutputStream(connection.getOutputStream ());
wr.writeBytes(urlParameters);
wr.flush();
wr.close();
connection.disconnect();


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 11:36:55 AM
yes its ugly and it doesn't work I don't understand why.

if you want to see how it looks like at the moment : http://pastebin.com/4UZWYZix (http://pastebin.com/4UZWYZix) . Maybe another pair of eye can help me in finding what should be something really small and stupid compared to this which work http://pastebin.com/SmrxuJQj (http://pastebin.com/SmrxuJQj)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 11:42:03 AM
Spotted another difference in my code (which doesn't seem to make a difference :( ) :

Code:
connection.setRequestProperty("Rest-Sign", signature.replaceAll("\n", ""));


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 16, 2013, 11:51:53 AM
the + "000" might be needed:

Code:
String nonce = "nonce=" + String.valueOf(System.currentTimeMillis()) + "000";

plus you probably need this before reading response:

Code:
DataOutputStream output = new DataOutputStream(c.getOutputStream());
            output.writeBytes(post_data);
            output.close();


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 16, 2013, 12:40:24 PM
 8) Where is your "donate to" address ? :D

Thanks!  It was that! Now it works the ticker only with POST too ;) Yeeeeeeeeeeeeeeee


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 16, 2013, 01:09:33 PM
nice, unless you're a bitcoin miliionaire don't worry bout it :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 16, 2013, 01:41:36 PM
Glad you finally got it working, advanced. Clearly MikeH was paying closer attention  ;)
I was just giving the query as a string because I wanted to set it up quickly, using a HashMap is better.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 17, 2013, 08:14:32 AM
Thank you nitrous ;) I'm reporting new findings here :

I'm currently using ticker_fast and it works pretty good, despite I'm not sure how. "BTCUSD/money/ticker_fast"

Works pretty good with USD, even though the answer is slightly different in the ordering:

Code:
Query to :BTCUSD/money/ticker_fast - HTTP response : 

{"result":"success","data":{"last_local":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"last":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"last_orig":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"last_all":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"buy":{"value":"77.61110","value_int":"7761110","display":"$77.61110","display_short":"$77.61","currency":"USD"},"sell":{"value":"77.61641","value_int":"7761641","display":"$77.61641","display_short":"$77.62","currency":"USD"},"now":"1366188127158506"}}
Query to :BTCUSD/money/ticker - HTTP response :

{"result":"success","data":{"high":{"value":"82.39000","value_int":"8239000","display":"$82.39000","display_short":"$82.39","currency":"USD"},"low":{"value":"53.61111","value_int":"5361111","display":"$53.61111","display_short":"$53.61","currency":"USD"},"avg":{"value":"68.58637","value_int":"6858637","display":"$68.58637","display_short":"$68.59","currency":"USD"},"vwap":{"value":"68.56407","value_int":"6856407","display":"$68.56407","display_short":"$68.56","currency":"USD"},"vol":{"value":"337373.36971644","value_int":"33737336971644","display":"337,373.36971644\u00a0BTC","display_short":"337,373.37\u00a0BTC","currency":"BTC"},"last_local":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"last_orig":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"last_all":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"last":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"buy":{"value":"77.60002","value_int":"7760002","display":"$77.60002","display_short":"$77.60","currency":"USD"},"sell":{"value":"77.61641","value_int":"7761641","display":"$77.61641","display_short":"$77.62","currency":"USD"},"item":"BTC","now":"1366188104561566"}}


Be careful with EUROS because is not working as expected, giving different values. I will stick to the normal ticker for now with euros (...):

Code:
Query to :BTCEUR/money/ticker - HTTP response : 

{"result":"success","data":{"high":{"value":"64.51361","value_int":"6451361","display":"64.51361\u00a0\u20ac","display_short":"64.51\u00a0\u20ac","currency":"EUR"},"low":{"value":"41.78000","value_int":"4178000","display":"41.78000\u00a0\u20ac","display_short":"41.78\u00a0\u20ac","currency":"EUR"},"avg":{"value":"52.17449","value_int":"5217449","display":"52.17449\u00a0\u20ac","display_short":"52.17\u00a0\u20ac","currency":"EUR"},"vwap":{"value":"53.49364","value_int":"5349364","display":"53.49364\u00a0\u20ac","display_short":"53.49\u00a0\u20ac","currency":"EUR"},"vol":{"value":"30454.26156026","value_int":"3045426156026","display":"30,454.26156026\u00a0BTC","display_short":"30,454.26\u00a0BTC","currency":"BTC"},"last_local":{"value":"60.72397","value_int":"6072397","display":"60.72397\u00a0\u20ac","display_short":"60.72\u00a0\u20ac","currency":"EUR"},"last_orig":{"value":"77.77699","value_int":"7777699","display":"$77.77699","display_short":"$77.78","currency":"USD"},"last_all":{"value":"59.24060","value_int":"5924060","display":"59.24060\u00a0\u20ac","display_short":"59.24\u00a0\u20ac","currency":"EUR"},"last":{"value":"60.72397","value_int":"6072397","display":"60.72397\u00a0\u20ac","display_short":"60.72\u00a0\u20ac","currency":"EUR"},"buy":{"value":"60.10000","value_int":"6010000","display":"60.10000\u00a0\u20ac","display_short":"60.10\u00a0\u20ac","currency":"EUR"},"sell":{"value":"60.72397","value_int":"6072397","display":"60.72397\u00a0\u20ac","display_short":"60.72\u00a0\u20ac","currency":"EUR"},"item":"BTC","now":"1366188316676566"}}
Query to :BTCEUR/money/ticker_fast - HTTP response :

{"result":"success","data":{"last_local":{"value":"60.22337","value_int":"6022337","display":"60.22337\u00a0\u20ac","display_short":"60.22\u00a0\u20ac","currency":"EUR"},"last":{"value":"60.22337","value_int":"6022337","display":"60.22337\u00a0\u20ac","display_short":"60.22\u00a0\u20ac","currency":"EUR"},"last_orig":{"value":"77.79000","value_int":"7779000","display":"$77.79000","display_short":"$77.79","currency":"USD"},"last_all":{"value":"59.25051","value_int":"5925051","display":"59.25051\u00a0\u20ac","display_short":"59.25\u00a0\u20ac","currency":"EUR"},"buy":{"value":"60.22337","value_int":"6022337","display":"60.22337\u00a0\u20ac","display_short":"60.22\u00a0\u20ac","currency":"EUR"},"sell":{"value":"60.73178","value_int":"6073178","display":"60.73178\u00a0\u20ac","display_short":"60.73\u00a0\u20ac","currency":"EUR"},"now":"1366188333851800"}}

 






Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 17, 2013, 08:30:44 AM
I have another question: in the ticker, what is the difference between last, last_all, last_local?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 17, 2013, 01:55:50 PM
it's documented somewhere - last_all is last sale price in any currency converted to your specified currency, not sure difference between other 2 - not looking at documentation now :)


I have a question too - anyone know when buying through the api whether you're able to issue a buy at say 10 bitcoins at a specified price even if you only have funds for 5 ?

Just wondering if it will just buy the 5 - otherwise you have to do the calculations work out how many bitcoins you can buy.

--

edit: never mind, I tested it and you don't need to specify exact # of bitcoins to clean out your $ balance.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: ||bit on April 17, 2013, 02:34:52 PM
I'm interested in an automated trading platform myself. Glad you did some compiling for the API at least.

I had a fairly strict trailing stop strategy, and was using it in the recent climb. However, because there is no way to enter dynamic stops & sleep is a requirement - for normal humans - I found myself waking up to the recent crash. :P

Anyway, maybe some automatic trading software could have helped. I didn't think there was a DDOS according to the MtGox report. Evenso, I think the stop would have been placed & executed before the peak velocity of the crash.

Anyone know of any available automated trading applications out there yet?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 17, 2013, 02:52:48 PM
I have another question: in the ticker, what is the difference between last, last_all, last_local?

From what I can tell:
 - last and last_local are the same, and give the last trade of the selected currency
 - last_orig is the most recent trade of ALL currencies (so could be USD, GBP, EUR, anything...)
 - last_all is the same as last_orig, but it has been converted to the selected currency

As for ticker_fast, I imagine it was quickly put together so may have bugs in it. I just tested it now for BTCEUR and BTCUSD, however, and they both seemed to work and returned the same JSON properties, can you tell me what part seems not to work for EUR for you? Were you confused by last_orig being in USD?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 17, 2013, 02:54:57 PM
it's documented somewhere - last_all is last sale price in any currency converted to your specified currency, not sure difference between other 2 - not looking at documentation now :)


I have a question too - anyone know when buying through the api whether you're able to issue a buy at say 10 bitcoins at a specified price even if you only have funds for 5 ?

Just wondering if it will just buy the 5 - otherwise you have to do the calculations work out how many bitcoins you can buy.

--

edit: never mind, I tested it and you don't need to specify exact # of bitcoins to clean out your $ balance.


Don't rely on being able to buy however many btc you want even if unfunded, MtGox claims this is the source of a lot of lag and they are transitioning their engine to deny such requests.

Can you clarify what happened in your test? i.e: what you asked the api to do (how many and for how much), and what happened, and whether you still had an incomplete order pending afterwards or not?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 17, 2013, 03:06:12 PM
I'm interested in an automated trading platform myself. Glad you did some compiling for the API at least.

I had a fairly strict trailing stop strategy, and was using it in the recent climb. However, because there is no way to enter dynamic stops & sleep is a requirement - for normal humans - I found myself waking up to the recent crash. :P

Anyway, maybe some automatic trading software could have helped. I didn't think there was a DDOS according to the MtGox report. Evenso, I think the stop would have been placed & executed before the peak velocity of the crash.

Anyone know of any available automated trading applications out there yet?

I'm sorry to hear that, that's really unlucky especially with your strategy! Did you lose a lot?

I have heard sporadic reports of user-friendly automated trading platforms for MtGox, but I'm not sure how much you can trust these. Look out for a MetaTrader extension, as MT is the norm for trading fiat forex currencies, and I think is quite easy to use. It's tailored for monitoring and auto trading, but I don't think there's any MtGox extension yet (or any other exchanges, for that matter). I was thinking of releasing a graphical client whilst I work on my own high frequency trader, but I don't currently have the time to do any major development (not for a couple months at least).

Do you know any programming languages? If not, maybe you could take this opportunity to learn one - you would then have full control over how it works (although you'd also be responsible for any money lost due to program failures :P).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 17, 2013, 03:15:39 PM
I put in an order of 10 btc at $90 even though I only had funds for 5.. the order completed and I don't think I had any orders outstanding - don't now anyway.

I'm guessing it just checked my balance and adjusted # of btc before placing the order.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 17, 2013, 03:55:55 PM
I put in an order of 10 btc at $90 even though I only had funds for 5.. the order completed and I don't think I had any orders outstanding - don't now anyway.

I'm guessing it just checked my balance and adjusted # of btc before placing the order.


That sounds likely, I guess it would confuse a lot of people if they just denied all orders over the current balance.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 17, 2013, 05:29:07 PM
Nitrous, my fault. The fast ticker seem to work with EUR too. I got confused with last_all in the crowded debug console ;)

OT:

I'm actually building a simple trading bot myself if anybody is interested in participating to the beta test, PM me!  I already collected 15 volunteers + myself. Of course you should trust it with your life, but this bot here runs locally on your machine. you can see the code yourself and the API keys are stored locally on your machine ;)

Here you can see an example of the readme :> http://www.lize.it/up/prototype-botcoin/ (http://www.lize.it/up/prototype-botcoin/)

This was how the node version looked like  it saved me from big price drops), now I'm doing it better with java.

If any suggestion , please add ;D


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 17, 2013, 08:47:09 PM
Nitrous, my fault. The fast ticker seem to work with EUR too. I got confused with last_all in the crowded debug console ;)

OT:

I'm actually building a simple trading bot myself if anybody is interested in participating to the beta test, PM me!  I already collected 15 volunteers + myself. Of course you should trust it with your life, but this bot here runs locally on your machine. you can see the code yourself and the API keys are stored locally on your machine ;)

Here you can see an example of the readme :> http://www.lize.it/up/prototype-botcoin/ (http://www.lize.it/up/prototype-botcoin/)

This was how the node version looked like  it saved me from big price drops), now I'm doing it better with java.

If any suggestion , please add ;D

Have you got the JSON decoded yet? You could make debugging easier by pretty-printing it :) I've updated the documentation with my findings for the new method so hopefully that will be helpful - although I still haven't been able to test the differences during a high lag period.

That sounds interesting :) Does it generate the trading strategy automatically or is it user-programmable? I haven't been able to dedicate any large amounts of time to tradebot development but I was planning to incorporate some forex technical indicators, have a look at http://en.wikipedia.org/wiki/MACD for example. It seems fairly accurate (although there is a bit of lag), moreso than simple limit orders. I think bitcoin needs a trading platform to monitor forex technical indicators in order to make more informed trading decisions, especially with the high volatility, you could make a really powerful tool here :D


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 18, 2013, 06:55:03 AM
Nitrous, my fault. The fast ticker seem to work with EUR too. I got confused with last_all in the crowded debug console ;)

OT:

I'm actually building a simple trading bot myself if anybody is interested in participating to the beta test, PM me!  I already collected 15 volunteers + myself. Of course you should trust it with your life, but this bot here runs locally on your machine. you can see the code yourself and the API keys are stored locally on your machine ;)

Here you can see an example of the readme :> http://www.lize.it/up/prototype-botcoin/ (http://www.lize.it/up/prototype-botcoin/)

This was how the node version looked like  it saved me from big price drops), now I'm doing it better with java.

If any suggestion , please add ;D

Have you got the JSON decoded yet? You could make debugging easier by pretty-printing it :) I've updated the documentation with my findings for the new method so hopefully that will be helpful - although I still haven't been able to test the differences during a high lag period.

That sounds interesting :) Does it generate the trading strategy automatically or is it user-programmable? I haven't been able to dedicate any large amounts of time to tradebot development but I was planning to incorporate some forex technical indicators, have a look at http://en.wikipedia.org/wiki/MACD for example. It seems fairly accurate (although there is a bit of lag), moreso than simple limit orders. I think bitcoin needs a trading platform to monitor forex technical indicators in order to make more informed trading decisions, especially with the high volatility, you could make a really powerful tool here :D

Yes I decoded the JSON and prettified it ;)



The strategy is totally up to the user, I'm not a trading expert my self and I just want to provide a powerful tools to the people. The strategy(.csv) could then be shared among users and exchanged according to domain experties, forex indicators, and others.

I'm myself a follower of the mailing list on bitcoinbullbear.com  and I take decisions accordingly, but I'm not a speculator crazy trader ;)

thanks for the feedback!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: TimJBenham on April 18, 2013, 12:32:09 PM
all I want right now is to monitor and issue a sell order in case of a large drop :)  kinda weird that isn't a standard feature yet through the web interface.

Yeah, it's weird that the big exchanges don't support standard exchange features such as order limits, short selling and margin trading.

In most markets those are broker features, not exchange features.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: brox on April 18, 2013, 01:19:59 PM
Typo:

comprised <- compromised
completed useless, currently <- completely useless, currently


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on April 18, 2013, 03:37:09 PM
Typo:

comprised <- compromised
completed useless, currently <- completely useless, currently

got the right thread? :)



Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 18, 2013, 04:40:13 PM
Typo:

comprised <- compromised
completed useless, currently <- completely useless, currently

Thanks, fixed :)


got the right thread? :)

He was pointing out typos in the documentation :P


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: Viceroy on April 18, 2013, 05:03:47 PM
all I want right now is to monitor and issue a sell order in case of a large drop :)  kinda weird that isn't a standard feature yet through the web interface.

Yeah, it's weird that the big exchanges don't support standard exchange features such as order limits, short selling and margin trading.

In most markets those are broker features, not exchange features.

Please consider contributing such ideas to this post:  https://bitcointalk.org/index.php?topic=179147.0


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: baracuda on April 19, 2013, 09:06:10 AM
At the end of the document, money/wallet/history has an optional parameter of page (in addition to the required currency parameter already listed.)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 19, 2013, 09:55:20 PM
At the end of the document, money/wallet/history has an optional parameter of page (in addition to the required currency parameter already listed.)

Thanks, the ones at the bottom I haven't got around to investigating yet so the only information is what I found on the official documentation, so any information is very helpful :) I've published what I could find out for the method. Have you confirmed this parameter works? I don't have more than 50 transactions in any of my wallets so I can't test it myself.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 23, 2013, 09:35:50 PM
Just added some brief information about streaming here - https://bitbucket.org/nitrous/mtgox-api/src/master/socket (https://bitbucket.org/nitrous/mtgox-api/src/master/socket), including a basic websocket client for quick testing. Also documented the `stream/list_public method`. Note that I have not actually documented any of the actual streaming API, as the bitcoin wiki has a very good page on it themselves - https://en.bitcoin.it/wiki/MtGox/API/Streaming (https://en.bitcoin.it/wiki/MtGox/API/Streaming).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 24, 2013, 07:36:40 AM
thanks! ;)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 26, 2013, 11:11:33 AM
I finally got some spare time to put together a small repo for documenting the Java Client.
Hope it helps. Thank you guys.

https://github.com/adv0r/mtgox-api-v2-java (https://github.com/adv0r/mtgox-api-v2-java)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 26, 2013, 01:48:22 PM
I finally got some spare time to put together a small repo for documenting the Java Client.
Hope it helps. Thank you guys.

https://github.com/adv0r/mtgox-api-v2-java (https://github.com/adv0r/mtgox-api-v2-java)


Thanks, that's great work, I linked it in my API examples section :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 26, 2013, 02:45:06 PM
You are welcome, thanks for the good words you used ;)

PS: Today I observed another unexpected behaviour of the APIs. You cannot access the method MONEY/BITCOIN/SEND_SIMPLE if you are using an yubikey. Not even if you gave permissions to the API keys.  This is weird : i mean, i need the yubikey to login into my account, but this is not stopping me to be able to place orders via API. I opened a ticket on mtg.

 


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 26, 2013, 07:56:35 PM
You are welcome, thanks for the good words you used ;)

PS: Today I observed another unexpected behaviour of the APIs. You cannot access the method MONEY/BITCOIN/SEND_SIMPLE if you are using an yubikey. Not even if you gave permissions to the API keys.  This is weird : i mean, i need the yubikey to login into my account, but this is not stopping me to be able to place orders via API. I opened a ticket on mtg.


No problem, it's important to add a spectrum of examples, from quite simple to more sophisticated, to help people get started and then improve their projects, as well as having multiple languages :)

Thanks, I've updated the docs to note this. Update me if you get an official response, as it does make sense for it to be denied with yubikey enabled, but it is indeed inconsistent.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: IIzzII on April 27, 2013, 02:46:14 AM
Can someone help me out, I feel like I'm going crazy here. I've been reading your documentation (which is great, by the way) and I can't seem to make even the basic calls work predictably. I'm trying to get some historical data. For testing reasons, I've started at 'since=0'. This url:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=0

properly returns trades. The timestamp of the first trade is "1279408157" which is a date of  07/17/10 @ 6:09:17pm. The timestamp of the last trade is "1282128968" which is a date of 08/18/10 @ 5:56:08am EST. Now if I take that last timestamp and attach that to the url:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1282128968

the first timestamp of the first trade is "1309108565" which is a date of 06/26/11 @ 12:16:05pm EST. I've also tried using microseconds as suggested by the api but that returns the same thing. The url:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1282128968000000

returns the same thing as above. I've tried other timestamps from the first response and the all point to this 06/25/11 set of trades.

Any idea what I'm doing wrong?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: TimJBenham on April 27, 2013, 05:50:33 AM
Not exactly a V2 issue but https://data.mtgox.com/api/1/BTCUSD/ticker_fast just started failing for me with the error "This query needs to be done via data.mtgox.com". It gives an endless redirect loop. Is it still working in V2?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: TimJBenham on April 27, 2013, 06:26:26 AM
Looks like it's working again. Possible Cloudfare false positive?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 27, 2013, 10:28:12 AM
Can someone help me out, I feel like I'm going crazy here. I've been reading your documentation (which is great, by the way) and I can't seem to make even the basic calls work predictably. I'm trying to get some historical data. For testing reasons, I've started at 'since=0'. This url:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=0

properly returns trades. The timestamp of the first trade is "1279408157" which is a date of  07/17/10 @ 6:09:17pm. The timestamp of the last trade is "1282128968" which is a date of 08/18/10 @ 5:56:08am EST. Now if I take that last timestamp and attach that to the url:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1282128968

the first timestamp of the first trade is "1309108565" which is a date of 06/26/11 @ 12:16:05pm EST. I've also tried using microseconds as suggested by the api but that returns the same thing. The url:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1282128968000000

returns the same thing as above. I've tried other timestamps from the first response and the all point to this 06/25/11 set of trades.

Any idea what I'm doing wrong?

Hmm, that's very strange, it seems to work for dates after 1309108565, but not for the data gap between. I'll have to have another look later when I have some more time, but for now maybe just try from 1309108565 instead. Sorry about that, I may need to contact mtgox and see if they can help.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 27, 2013, 10:29:50 AM
Looks like it's working again. Possible Cloudfare false positive?

Could be. At times of high lag/DDoS, the API can stop working properly, and it can result in undefined behaviour.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 27, 2013, 10:55:30 AM
currently you can get the last price of 1 BTC from 4 sources. might be worth comparing them  and plot them for a testing period.this way we could check inconsistencies . Someone up for this?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: IIzzII on April 27, 2013, 01:37:16 PM
Sorry about that, I may need to contact mtgox and see if they can help.

You don't need to apologize! I'll see if I can get it to work from that date on. Thanks so much.

UPDATE: It seems to be working fine as long as you don't provide a timestamp before 1309108565.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 28, 2013, 11:22:10 AM
You are welcome, thanks for the good words you used ;)

PS: Today I observed another unexpected behaviour of the APIs. You cannot access the method MONEY/BITCOIN/SEND_SIMPLE if you are using an yubikey. Not even if you gave permissions to the API keys.  This is weird : i mean, i need the yubikey to login into my account, but this is not stopping me to be able to place orders via API. I opened a ticket on mtg.


No problem, it's important to add a spectrum of examples, from quite simple to more sophisticated, to help people get started and then improve their projects, as well as having multiple languages :)

Thanks, I've updated the docs to note this. Update me if you get an official response, as it does make sense for it to be denied with yubikey enabled, but it is indeed inconsistent.



Here we go :

Quote
Hello,

You will have to provide an OTP value with the API. The easiest is to add a TOTP and generate it when calling the API.

Thanks,

MtGox.com Team


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 28, 2013, 01:38:36 PM
Sorry about that, I may need to contact mtgox and see if they can help.

You don't need to apologize! I'll see if I can get it to work from that date on. Thanks so much.

UPDATE: It seems to be working fine as long as you don't provide a timestamp before 1309108565.

From what I can make out, it seems that there was a database change around 1309108565, as when you use since=0, the tid values are sequential integers, but at 1309108565, they become unix microstamps. Using this, I managed to work out that `since` refers to the tid instead of the date! I have now tested it successfully, and I can get all data from 0 to 1309108565 and beyond by passing in the last tid of the data as the new since value, i.e.:

since=0 gives tids from 1 to 1007
since=1007 gives tids from 1008 to 2007

It also appears that instead of a 24 hour period, the method provides 1000 trades at a time. Passing in the last tid works across the database change boundary as well, so if you pass in since=218500, it will give you tids from 218501 to 218868, and then from 1309108565842636 onwards.

Here we go :

Quote
Hello,

You will have to provide an OTP value with the API. The easiest is to add a TOTP and generate it when calling the API.

Thanks,

MtGox.com Team

Thanks advanced, I'll be adding that in to the docs along with the newly found trades/fetch info later today. Do you know what the argument name for the opt value is? Is it just "otp=123456"?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 28, 2013, 09:31:27 PM
Ok, I've overhauled the trades/fetch documentation :)



Here we go :

Quote
Hello,

You will have to provide an OTP value with the API. The easiest is to add a TOTP and generate it when calling the API.

Thanks,

MtGox.com Team

Have you been able to find a working argument yet? I tried 'otp', 'totp', 'OTP', 'TOTP' and none of them seemed to work, giving a 403 forbidden error. I don't currently have any funds in my MtGox account, so that might explain why. Perhaps it should be in the header or passed in to the HMAC encryption?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: advanced on April 29, 2013, 10:09:24 AM
I have no idea.
I'll put it in my backlog. However I just got an answer from the Yubikey team and apparently there is no way whatsoever I can read the OTP from the usb device unless the user physically press the button (i.e., not of much use for my bot ). :/


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: pescador on April 29, 2013, 11:29:46 AM
Is it possible to withdraw USD from MtGox to an OKPAY account using the API? I can't seem to find the commands. However I can assign withdraw rights to an API key, or is that only for withdrawing BTC?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 29, 2013, 01:33:04 PM
I have no idea.
I'll put it in my backlog. However I just got an answer from the Yubikey team and apparently there is no way whatsoever I can read the OTP from the usb device unless the user physically press the button (i.e., not of much use for my bot ). :/

You could change only the trade access to google auth, and keep the others on yubikey. There's little reason why someone would hack your account just to make some trades, as all they could do would be to throw away your money. That way, you can generate the codes with your bot, as the google auth algorithm is public I think.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on April 29, 2013, 01:46:51 PM
Is it possible to withdraw USD from MtGox to an OKPAY account using the API? I can't seem to find the commands. However I can assign withdraw rights to an API key, or is that only for withdrawing BTC?


I don't think so. If there is, then there's no information about it. Currently, the withdraw permission only seems to apply to BTC, but as MtGox claim to be overhauling their trade engine and api, it may be possible in the future to perform OKPAY withdraws, amongst others, via the API.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: IIzzII on May 02, 2013, 01:10:20 PM
I have two random questions:

1) Is there an advantage to using websockets as opposed to POST requests? Are websockets faster? More secure?

2) What does the price on the ticker actually mean? Is it the price of the last trade that was made before the request was received? Is it an average? If you're calculating EMAs and it is at the end of a given time period, is it best to subscribe to the ticker feed and use that value or subscribe to the trades feed and use the value of the last trade?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 02, 2013, 07:39:37 PM
I have two random questions:

1) Is there an advantage to using websockets as opposed to POST requests? Are websockets faster? More secure?

2) What does the price on the ticker actually mean? Is it the price of the last trade that was made before the request was received? Is it an average? If you're calculating EMAs and it is at the end of a given time period, is it best to subscribe to the ticker feed and use that value or subscribe to the trades feed and use the value of the last trade?

1) The websocket feed should be faster and less taxing on MtGox, so it's ideal for getting ticker data. As for security, I'm not sure. It may be more reliable, but for trades, info, etc, make sure you're using wss instead of ws, and even then, be careful - if in doubt, POST. Also note that there are two ways to access it, websockets and socket.io, websockets are the currently recommended method (socket.io apparently uses buggy code, while the ws code was rewritten from scratch).

2) The ticker method yields multiple values (see http://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-moneyticker (http://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-moneyticker)). Most of the values are from the most recent trade (see a more detailed breakdown at the link); however, there is an average available (VWAP (http://en.wikipedia.org/wiki/Volume-weighted_average_price)), which I believe is for the last 24 hours.

For calculating EMAs, I recommend keeping a local database of all trades as far back as necessary. Information for downloading these is available here (https://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-moneytradesfetch). Contact me if you need more detailed information to get this working. Then you can keep it up to date however you want, but I would strongly suggest using websockets. Of course, that depends on how up-to-date you need your database. If you don't need the most recent hour or so of data, then you can just keep accessing `money/trades/fetch` periodically, but if you need to calculate in realtime then definitely use websockets.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on May 03, 2013, 09:58:42 AM
my little app is using data.mtgox v2 with posts, it uses a 15 second time out and it reaches this limit a lot.. how's everyone else dealing with mtgox not responding?  I can just re-try of course but it seems dodgy that this occurs like 10 times / hour.

actually it's not just time outs - it'll run fine for a while then I get a bunch of 502 Bad Gateway's too.


Title: nitrous
Post by: nitrous on May 03, 2013, 05:36:13 PM
my little app is using data.mtgox v2 with posts, it uses a 15 second time out and it reaches this limit a lot.. how's everyone else dealing with mtgox not responding?  I can just re-try of course but it seems dodgy that this occurs like 10 times / hour.

actually it's not just time outs - it'll run fine for a while then I get a bunch of 502 Bad Gateway's too.


If you wait 15s between requests you really should be fine. That sounds like MtGox just being overloaded, which can happen fairly often. You can either put in some error handling to fail gracefully in the situation, and just try again (it sounds like you are doing this), or you could try using mtgox's websocket, which is probably more appropriate for a ticker app, although it will use a lot more network bandwidth (and battery, if it's a phone app). What I do in my api for post requests is try the same request repeatedly (up to 10 times) one after another, so you could try that (and just make sure that it doesn't do more than, say, 20 of these in a 15s period, and just sleep the thread until the next 15s window - you'll need to use multithreading to keep the app responsive). You should also change to ticker_fast, as it should be more reliable, and less prone to 502s (plus I think it has no request limit, so no danger of your app being blocked for DDoSing).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on May 03, 2013, 06:19:00 PM
so it's not correct that websocket is on the way out and socket.io's the replacement?  I got that from like a chat session snapshot a while back, I think it was one of the MtGox guys but I hadn't seen mention of it anywhere else.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 03, 2013, 10:26:44 PM
so it's not correct that websocket is on the way out and socket.io's the replacement?  I got that from like a chat session snapshot a while back, I think it was one of the MtGox guys but I hadn't seen mention of it anywhere else.


I've got mixed reports - on the one hand macarse got info on irc that websocket was rewritten and socket.io was buggy, but I've also heard that socket.io was replacing websocket, I'm not sure which is the more accurate/current status. If you hear any more recent official statement please tell me so I can update the docs :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 04, 2013, 05:21:38 AM
so it's not correct that websocket is on the way out and socket.io's the replacement?  I got that from like a chat session snapshot a while back, I think it was one of the MtGox guys but I hadn't seen mention of it anywhere else.


I've got mixed reports - on the one hand macarse got info on irc that websocket was rewritten and socket.io was buggy, but I've also heard that socket.io was replacing websocket, I'm not sure which is the more accurate/current status. If you hear any more recent official statement please tell me so I can update the docs :)

This latter thing you heard seems to come from someone completely unaware of what socket.io is. Socket.IO is mainly a compatibility layer for browsers that do not support Websocket. If you do not rely on a browser to use the Websocket API from MtGox, there is little reason to even consider Socket.IO.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on May 04, 2013, 05:54:08 PM
This latter thing you heard seems to come from someone completely unaware of what socket.io is. Socket.IO is mainly a compatibility layer for browsers that do not support Websocket. If you do not rely on a browser to use the Websocket API from MtGox, there is little reason to even consider Socket.IO.

maybe so but mtgox provided 2 separate streams for them (different IP's) and websocket wasn't working at all most the time for me while the specified server to use for socket.io worked fine.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 04, 2013, 09:51:31 PM
This latter thing you heard seems to come from someone completely unaware of what socket.io is. Socket.IO is mainly a compatibility layer for browsers that do not support Websocket. If you do not rely on a browser to use the Websocket API from MtGox, there is little reason to even consider Socket.IO.

maybe so but mtgox provided 2 separate streams for them (different IP's) and websocket wasn't working at all most the time for me while the specified server to use for socket.io worked fine.


I'm not sure on how to reply to that, but isn't it possible that you are using some custom websocket implementation that doesn't properly handle the protocol ? RFC 6455 describes the WebSocket protocol.

From my own experience, using MtGox through its socket.io stream with a "quickly-written-library" was causing a lot of disconnects for me. Then I moved on to use only websockets but together with a decent implementation of it. So far the I only get disconnected when requesting huge amounts of data in a short time (like downloading the trade history over several weeks), and even then I readily reconnect and everything is good again.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 04, 2013, 09:59:47 PM
This latter thing you heard seems to come from someone completely unaware of what socket.io is. Socket.IO is mainly a compatibility layer for browsers that do not support Websocket. If you do not rely on a browser to use the Websocket API from MtGox, there is little reason to even consider Socket.IO.

maybe so but mtgox provided 2 separate streams for them (different IP's) and websocket wasn't working at all most the time for me while the specified server to use for socket.io worked fine.


I'm not sure on how to reply to that, but isn't it possible that you are using some custom websocket implementation that doesn't properly handle the protocol ? RFC 6455 describes the WebSocket protocol.

From my own experience, using MtGox through its socket.io stream with a "quickly-written-library" was causing a lot of disconnects for me. Then I moved on to use only websockets but together with a decent implementation of it. So far the I only get disconnected when requesting huge amounts of data in a short time (like downloading the trade history over several weeks), and even then I readily reconnect and everything is good again.

I believe that recently they rewrote the websocket server to be more reliable (moreso than their socket.io server), so maybe that could have been responsible for the problems whilst they were switching over?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MikeH on May 05, 2013, 03:12:42 AM
Hmm, I don't recall whether this was the problem I was having before but I just noticed the old listentobitcoin code that uses socket.io works without a web server, the websocket version doesn't (in either Safari or Chrome), I need to kick off Apache before it will connect to MtGox.   It does connect to blockchain.info without the web server though, it's weird.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: jekatigr on May 06, 2013, 08:09:12 AM
Hello everyone,
I haven't found information about,

Through mtgox api I can buy or sell BTC by setting amount in BTC.
Can I buy BTC by setting amount in USD?
For example I'm sending request "price_int=***" and getting (***/market_price) BTC in my account.
Their api support this feature?


Title: nitrous
Post by: nitrous on May 06, 2013, 10:25:07 AM
Hello everyone,
I haven't found information about,

Through mtgox api I can buy or sell BTC by setting amount in BTC.
Can I buy BTC by setting amount in USD?
For example I'm sending request "price_int=***" and getting (***/market_price) BTC in my account.
Their api support this feature?

No there you cannot do that, you will have to calculate the (***/market_price) yourself and use that as the amount. For example, if market_price is $120 and you want to sell $600, then you will receive (600/120)=5 btc:

BTCUSD/money/order/add
type=bid
amount_int=500000000
price_int=12000000

The above will deduct $600 from your account and give you 5 btc, minus your fee of course.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: dfrechet on May 06, 2013, 03:16:06 PM
Does anyone know what the deal is with those streaming API depth updates with volume_int set to zero? The total volume still changes, breaking cumulative checks. It appears that about 5% of all depth updates are of this form, with some clustering with respect to the price (for certain price values, depth updates consist only of these messages).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 06, 2013, 08:32:21 PM
Does anyone know what the deal is with those streaming API depth updates with volume_int set to zero? The total volume still changes, breaking cumulative checks. It appears that about 5% of all depth updates are of this form, with some clustering with respect to the price (for certain price values, depth updates consist only of these messages).

Is this thread about the HTTP API or the Streaming API ? :)

Anyway, volume_int set to zero is simply telling you that there are no more bids/asks at that value -- it was fully removed. This of course changes the total volume, since it reduces the total volume. Also note that volume_int isn't about the last change in volume for a given price, it is the current total volume for that price.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: dfrechet on May 07, 2013, 01:18:56 AM
Are you sure? The documentation under https://en.bitcoin.it/wiki/MtGox/API/Streaming#Depth (https://en.bitcoin.it/wiki/MtGox/API/Streaming#Depth) seems to indicate otherwise:

total_volume_int: total volume at this price, after applying the depth update, can be used as a starting point before applying subsequent updates.
volume_int: volume change * 1E8

More than 90% of all depth updates as measured during the last day fit this pattern (old_total_volume_int + volume_int = total_volume_int). Suspiciously, all depth updates that break it have volume_int set to zero...

PS: If there is any moderator around, feel free to move this discussion to a new topic ("MtGox Streaming API: Zero Volume Depth Updates").


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 07, 2013, 03:43:26 PM
Are you sure? The documentation under https://en.bitcoin.it/wiki/MtGox/API/Streaming#Depth (https://en.bitcoin.it/wiki/MtGox/API/Streaming#Depth) seems to indicate otherwise:

total_volume_int: total volume at this price, after applying the depth update, can be used as a starting point before applying subsequent updates.
volume_int: volume change * 1E8

That is exactly what I said: "Also note that volume_int isn't about the last change in volume for a given price, it is the current total volume for that price"

So if the total volume is zero, it means there are no more bids/asks at that value. That is caused because the users that entered them decided to remove them all, and the volume went to zero.

Edit: Ok, I misread volume_int for total_volume_int. I currently don't bother with volume_int in my applications, is there a reason to do so ? Maybe for performance reasons ? Given the low volume at max hundreds of bids/asks in a second (maybe ?) at MtGox, I haven't hit performance problems yet.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: dfrechet on May 07, 2013, 07:06:43 PM
I use it mainly for error checking and finding individual depth changes I might have missed during a socket reconnect.

Either these mysterious packets have some special undocumented meaning, or they are the result of a bug on part of the API. In doubt, I rather assume the former, hence this question...


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 07, 2013, 07:31:41 PM
For me in particular, it is mystery on how to really keep the depth market info correct and updated. The HTTP API provides methods for loading them (either partially or fully), but they are not updated. So you have to, for instance, load partial data using the HTTP API and then update it as the Streaming API sends new data regarding depth. As you receive data from the Streaming API you can easily discard some of the data feched earlier as it becomes invalid (not talking about zero volume), but you cannot be sure about the volume of every ask/bid at every price.

Hopefully, after running for some time X you will have correct data at least regarding the prices with most changes (i.e., those were trades are effectively happening). Too bad you might get disconnected, because then you will need some more time Y in order to "recorrect" the data you had. The partial solution is to never stop your application, so you can keep mostly correct information.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: cilphex on May 07, 2013, 09:43:58 PM
Nitrous, you state this in your documentation:

Quote
14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 07, 2013, 09:54:24 PM
Nitrous, you state this in your documentation:

Quote
14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: cilphex on May 07, 2013, 10:02:15 PM
Sweet, thank you.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 07, 2013, 10:02:38 PM
Nitrous, you state this in your documentation:

Quote
14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Sorry, I haven't been very clear. Fetching those methods by GET works fine, but if you fetch them by POST, some methods which work fine unauthenticated by GET start to produce unexpected behaviour and strange errors unless provided with a valid api key. Therefore, you should either incorporate both get and post requests, or provide a valid api key for all methods by post. (I said all, but I'm sure there are some which aren't so strict. The ticker methods used to not require auth, even by post, for example. Seeing as that seemed to change though, you shouldn't rely on post requests being lenient.)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 07, 2013, 10:05:55 PM
Nitrous, you state this in your documentation:

Quote
14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

That text is very misleading. You most certainly can access authenticated api methods on data.mtgox.com, in fact all methods go through that subdomain, including trading, sending bitcoins, merchant stuff, etc.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 07, 2013, 10:34:15 PM
Nitrous, you state this in your documentation:

Quote
14TH APRIL, 2013:
The requirement of a valid API key and secret used to not be enforced for methods such as money/ticker, but now they are required for all methods.

But you can clearly access ticker and depth data, for example, without a key:

https://data.mtgox.com/api/2/BTCUSD/money/ticker
https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch

So I'm kind of confused.  What'll happen if I keep fetching this data without a key?  Nothing?  Or does this kind of API call purposely not require a key?

Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

That text is very misleading. You most certainly can access authenticated api methods on data.mtgox.com, in fact all methods go through that subdomain, including trading, sending bitcoins, merchant stuff, etc.

No, not really. Till recently I could add a param version=2 to a json payload and request data from any method provided by the HTTP API through the address websocket.mtgox.com (that is, I was using entirely the streaming API, but still able to request data from the HTTP API through it).

Now, some of those methods are no longer accessible through websocket.mtgox.com, and must go through data.mtgox.com.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 08, 2013, 07:01:59 AM
Directly from data.mtgox.com:

"""
This subdomain is for access to readonly APIs such as the ticker.

Any API that requires authentication will not be available here.
"""

So, nothing will happen and it purposely does not require a key.

That text is very misleading. You most certainly can access authenticated api methods on data.mtgox.com, in fact all methods go through that subdomain, including trading, sending bitcoins, merchant stuff, etc.

No, not really. Till recently I could add a param version=2 to a json payload and request data from any method provided by the HTTP API through the address websocket.mtgox.com (that is, I was using entirely the streaming API, but still able to request data from the HTTP API through it).

Now, some of those methods are no longer accessible through websocket.mtgox.com, and must go through data.mtgox.com.

That's being a bit pedantic isn't it? My point was that the text at data.mtgox.com was wrong - it says any api that requires authentication will not be available there, yet I have been able to access authenticate methods over the subdomain for over a month now, and others have used this for years to build their btc bots. That is the official way to access authenticated methods, so clearly it's wrong. Yes you are right, you can also access some of these methods over websockets, but, ostensibly at least, accessing rest methods over the socket seems to function simply as a proxy to data.mtgox.com.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: glon on May 09, 2013, 12:06:05 PM
Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?



Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 09, 2013, 07:20:41 PM
Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?



Could it be a bug in this software itself ? I'm fetching ticker data just fine (including from the endpoint you mentioned).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 09, 2013, 07:33:04 PM
Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?

Briefly looking at the source code, it seems that the gadget just requests a URL using XMLHTTP, parses json, and picks out the value specified. Perhaps the problem is that the url you used is redirected to https, try https://data.mtgox.com/api/1/BTCEUR/ticker instead. If that doesn't work, perhaps it has problems with HTTPS?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: glon on May 09, 2013, 07:40:16 PM
Hi guys,

seeing as the developer of this gadget is pretty much MIA, maybe you know what's going on with the Gox API?

http://domchi.cc/mybitcoins/
https://bitcointalk.org/index.php?topic=38450.20

I've been using http://data.mtgox.com/api/1/BTCEUR/ticker
and
json["return"].sell.value

in the gadget settings all along but today it does not fetch the Gox sell value any more?

Briefly looking at the source code, it seems that the gadget just requests a URL using XMLHTTP, parses json, and picks out the value specified. Perhaps the problem is that the url you used is redirected to https, try https://data.mtgox.com/api/1/BTCEUR/ticker instead. If that doesn't work, perhaps it has problems with HTTPS?

Tried https and it now works! Not sure if it's that though, as I'm pretty sure I tried https earlier today and it didn't work then. But the gadget is pretty flakey so who's to say what's really going on with it?

Thanks a lot anyway!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 11, 2013, 09:01:27 PM
It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 11, 2013, 09:38:18 PM
It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868

And you can't request it by POST anymore, these unannounced changes will surely break numerous api implementations:
{"result":"error","error":"Invalid request method for this API","token":"invalid_request_method"}

EDIT:
From manual investigation, it seems that the tids jump to 1309108565842636, so when you reach 218868, go to since=1309108565000000 in order to get the next dataset.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 21, 2013, 01:02:16 PM
It looks like MtGox changed its behaviour regarding fetch of old trades. We all know there is a jump at transaction id 218868, but I'm pretty sure it was possible to continue fetching data by specifying ?since=218868 and so on. But now MtGox is returning no trades at that point, and it is necessary to manually to do the jump.

This was reported to me at https://github.com/knowitnothing/btcx/issues/1 and can be seen by visiting https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=218868

Do you know the best way to contact the MtGox API team? I've been able to download all USD trade data by performing the jump manually, but this fails for some currencies (e.g. GBP), because the data seems to be very dispersed, and so there are numerous jumps at unpredictable places.

Also, for anyone who wants to download EUR data, it seems that you need to start downloading from 1315289573000000 (I'm not sure if there is any earlier data). I hope there's a way around this, as otherwise anyone who wants to download less popular currency trades will have to do small test jumps repeatedly until they reach the next data 'pocket'!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: btc_lurker on May 21, 2013, 01:43:54 PM
I suggest giving their IRC channel a try. Last time I went there, MagicalTux answered the questions I had. But this is a tad different, I'm not sure they are willing to change their API to the earlier behavior -- but hopefully it was just a mistake somewhere, and then the situation will be reverted.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 21, 2013, 03:15:13 PM
I suggest giving their IRC channel a try. Last time I went there, MagicalTux answered the questions I had. But this is a tad different, I'm not sure they are willing to change their API to the earlier behavior -- but hopefully it was just a mistake somewhere, and then the situation will be reverted.

Ok, thanks. I tried now but it seems he's offline, not surprising considering its 12am there :P I'll try to contact him tomorrow morning if I get the opportunity, I might be able to catch him around 6pm JST.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 23, 2013, 07:19:43 AM
I suggest giving their IRC channel a try. Last time I went there, MagicalTux answered the questions I had. But this is a tad different, I'm not sure they are willing to change their API to the earlier behavior -- but hopefully it was just a mistake somewhere, and then the situation will be reverted.

MagicalTux told me that the API change was intentional, and was because all of us are trying to download all their trade data! There are some database dumps though which he is going to prepare some links to, I'll update the docs and give some more info in about 8 hours time.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 23, 2013, 01:21:35 PM
Regarding the trade data API, MagicalTux told me the following over IRC:

Quote from: MagicalTux
this api change was made because of the very high load caused by all the people downloading the trade history that way. we've been considering different solutions for this, and there's already a very wide variety of available dumps for any currency available on mtgox.

I haven't seen any announcements or dumps anywhere around, however he told me he 'will prepare something' so that we can access these official dumps. In the meantime, please avoid downloading large quantities of data using money/trade/fetch. If enough people are interested before MtGox prepares their dumps, I have an (as of now) up to date dump for USD and EUR (and the very first 187 GBP trades :P). These include the microtime, the price (int), the type (0-bid, 1-ask), the quantity (int), and whether or not it's primary (0-no, 1-yes) - note that for USD, I didn't download whether or not each trade was primary, so I just assumed they all were, this may not be correct. Anyway, my internet connection is quite slow, but I'll endeavour to upload it somewhere given enough demand :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 23, 2013, 02:08:59 PM
Ok, it uploaded quicker than I thought :), here's the link:
https://drive.google.com/folderview?id=0B3hexlKVFpMpTHFjbU5WVUYxVHc&usp=sharing

The file is archive.db, and it is a sqlite3 archive, with tables:
  • meta
  • MtGox_USD
  • MtGox_EUR
  • MtGox_GBP
  • MtGox_BTC

meta gives the last tid downloaded for each table. MtGox_USD, _EUR, _GBP contain all the trade data I have been able to download (should be complete for USD and EUR, definitely incomplete for GBP though). Please ignore MtGox_BTC, that was a bug in my program and I forgot to delete it before uploading! Fields for the trade data tables are:

  • date (primary, int -- microstamp)
  • price (int -- amount of auxiliary currency per bitcoin, see docs for info in integer amounts)
  • quantity (int -- amount of btc traded, see docs again)
  • type (int -- 0:bid, 1:ask)
  • primary (int -- 0:N, 1:Y)

Keep in mind that the primary field may not be accurate for MtGox_USD as I didn't think to include it when I downloaded the USD data, and so I set them all to Y. The EUR data should be accurate though.

I won't be downloading any other currencies, and I doubt I'll keep this uploaded archive up to date, however it does have data up to 1369314732232756 / Thu, 23 May 2013 13:12:12 GMT. If this dump isn't months out of date by the time you download this, feel free to keep it up to date using money/trades/fetch though :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on May 30, 2013, 10:07:04 PM
Hi everyone, forgot to update here but if you haven't heard, there is now a resolution to the trade data problem!
See here for more info - https://bitcointalk.org/index.php?topic=218980.0 (https://bitcointalk.org/index.php?topic=218980.0)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on June 04, 2013, 12:37:05 PM
I've been asked about the correct procedure for sending authenticated requests to the socket API, and so I spent a little bit of time translating the PHP example, trying it out, and writing up some detailed instructions on correct usage here:

https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on June 04, 2013, 06:27:19 PM
Update to the API examples (https://bitbucket.org/nitrous/mtgox-api/src/master/examples?at=master (https://bitbucket.org/nitrous/mtgox-api/src/master/examples?at=master)):

There are now VB.Net, Python, Node.js (Ameen), Java (advanced) and Ruby (Renich) examples, thanks to all the contributors, and definitely check them out if you're writing your own tool or want a starting point :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 06, 2013, 11:52:50 AM
thank you for your great documentation. i am haveing problem with the "trades/fetch" API. maybe somebody knows something about it. i want to integrate the data into cryptocoincharts.info

when i am doing this:
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch
-> everything works fine

when i want to optimize the query and only get the trades since my last fetch
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=123456789
-> i get a timeout after some seconds

i think mtgox or cloudflare is blocking this requests from my php curl. the funny thing is when i open the URL with my browser also the "since" url is working!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 06, 2013, 12:02:58 PM
thank you for your great documentation. i am haveing problem with the "trades/fetch" API. maybe somebody knows something about it. i want to integrate the data into cryptocoincharts.info

when i am doing this:
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch
-> everything works fine

when i want to optimize the query and only get the trades since my last fetch
http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=123456789
-> i get a timeout after some seconds

i think mtgox or cloudflare is blocking this requests from my php curl. the funny thing is when i open the URL with my browser also the "since" url is working!

Hi xchrix,

First of all, please make sure you are not downloading ALL trade data using this method (see https://bitcointalk.org/index.php?topic=218980.0 (https://bitcointalk.org/index.php?topic=218980.0) and https://bitcointalk.org/index.php?topic=221055.0 (https://bitcointalk.org/index.php?topic=221055.0) for the method to download the majority of trade data. This is not yet completely ready for use, but it will let you download data up to May 2013 at the moment).

MtGox is very particular about what data is available through the trades/fetch api. To get trades since your last fetch, make sure you use the last tid as your since value, e.g. http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1370025354198298 (http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1370025354198298). 123456789 is not a valid tid, and so you will get problems using it. Tids range from 0 to 218868, and then they start again from 1309108565842636.

If you really can't get through to the server, perhaps you have made too many requests? You should limit your api requests to a maximum of 10 requests every 10 seconds, or else you risk being blocked.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 06, 2013, 06:07:03 PM
Hi xchrix,

First of all, please make sure you are not downloading ALL trade data using this method (see https://bitcointalk.org/index.php?topic=218980.0 and https://bitcointalk.org/index.php?topic=221055.0 for the method to download the majority of trade data. This is not yet completely ready for use, but it will let you download data up to May 2013 at the moment).

MtGox is very particular about what data is available through the trades/fetch api. To get trades since your last fetch, make sure you use the last tid as your since value, e.g. http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1370025354198298. 123456789 is not a valid tid, and so you will get problems using it. Tids range from 0 to 218868, and then they start again from 1309108565842636.

If you really can't get through to the server, perhaps you have made too many requests? You should limit your api requests to a maximum of 10 requests every 10 seconds, or else you risk being blocked.

thanks for the fast reply! i read your documentation carefully so i know that thing with the tid 218868 and then 1309108565842636. at first i tried to download very slow the whole data but it didnt worked :D so now i only want the latest trades. but mtgox isnt giving me even this data! if i use since=xxxx (12345 was only a placeholder. didnt used this number) i get NOTHING (all the time)! if i dont use "since" i get sometimes some trades. but most of the time a timeout and zero bytes. i am fetching every 15 minutes!!!! so this shouldnt be a problem.

php curl says
Code:
http://data.mtgox.com/api/2/BTCEUR/money/trades/fetch
Operation timed out after 10001 milliseconds with 0 bytes received

i resisted to add mtgox data to my charts for a long time. but more an more visitors are begging for mtgox charts. every other crap exchange managed to provide an easy and fast API which is working all the time. mtgox isnt able todo this?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 06, 2013, 06:41:26 PM
thanks for the fast reply! i read your documentation carefully so i know that thing with the tid 218868 and then 1309108565842636. at first i tried to download very slow the whole data but it didnt worked :D so now i only want the latest trades. but mtgox isnt giving me even this data! if i use since=xxxx (12345 was only a placeholder. didnt used this number) i get NOTHING (all the time)! if i dont use "since" i get sometimes some trades. but most of the time a timeout and zero bytes. i am fetching every 15 minutes!!!! so this shouldnt be a problem.

php curl says
Code:
http://data.mtgox.com/api/2/BTCEUR/money/trades/fetch
Operation timed out after 10001 milliseconds with 0 bytes received

i resisted to add mtgox data to my charts for a long time. but more an more visitors are begging for mtgox charts. every other crap exchange managed to provide an easy and fast API which is working all the time. mtgox isnt able todo this?

Hmm, that's strange that your requests are timing out... the same request worked fine for me. I can only think of two reasons why it might not work. For some API methods, only GET requests are allowed, so make sure you're using that. The other reason is that you might have been blocked, probably when you tried to download the entire database.

Try doing the same request on another IP, and another computer. I know it's not exactly feasible to change the IP of a server, so if it turns out that it has been banned, you might need to go over to #mtgox on freenode IRC and ask if they can unban you (if it was indeed MtGox who did this, it might be cloudflare).

Lastly, if you still want to download the whole data, follow this thread (https://bitcointalk.org/index.php?topic=218980.0 (https://bitcointalk.org/index.php?topic=218980.0)). MtGox has setup a Google bigquery database with all data in all currencies up to May 2013, and soon they're going to begin automatically updating this at least every hour. There's some basic instructions at that thread, but I've also written a GUI tool here (https://bitcointalk.org/index.php?topic=221055.msg2669142#new) to download the data. Obviously this isn't suitable for a website, but you might be able to look at the source code for some hints at how to access the bigquery data.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 06, 2013, 07:42:11 PM
got a hint from "Delirium" on mtgox IRC channel.  its working now! the "workaround" is to set some curl options which are not documented anywhre at the mtgox API
now i am using this options and everything is working fine:

Code:
		$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($curl, CURLOPT_POSTFIELDS, null);
curl_setopt($curl, CURLOPT_POST, FALSE);
curl_setopt($curl, CURLOPT_HTTPGET, TRUE);

curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MtGox PHP client; '.php_uname('s').'; PHP/'.phpversion().')');
curl_setopt($curl, CURLOPT_HEADER, 0);         
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_AUTOREFERER, 1);   
curl_setopt($curl, CURLOPT_MAXREDIRS, 120);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);

$data = curl_exec($curl);


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 06, 2013, 08:10:25 PM
got a hint from "Delirium" on mtgox IRC channel.  its working now! the "workaround" is to set some curl options which are not documented anywhre at the mtgox API
now i am using this options and everything is working fine:

Code:
		$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($curl, CURLOPT_POSTFIELDS, null);
curl_setopt($curl, CURLOPT_POST, FALSE);
curl_setopt($curl, CURLOPT_HTTPGET, TRUE);

curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MtGox PHP client; '.php_uname('s').'; PHP/'.phpversion().')');
curl_setopt($curl, CURLOPT_HEADER, 0);         
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_AUTOREFERER, 1);   
curl_setopt($curl, CURLOPT_MAXREDIRS, 120);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);

$data = curl_exec($curl);

Great, glad you got it working :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 06, 2013, 08:13:25 PM
Great, glad you got it working :)

mtgox looks like microsoft to me -> trying to make live of devs suck more than others


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: zhangweiwu on July 12, 2013, 08:07:13 AM
Hi. I have been trying to follow the unofficial documentation for 2 days without success. What frastrate me is even the simpliest case directly taken from example code doesn't work.

The first example code is, forgive me putting up all:
Code:
import hmac, base64, hashlib, urllib2
base = 'https://data.mtgox.com/api/2/'

def makereq(key, secret, path, data):
    hash_data = path + chr(0) + data
    secret = base64.b64decode(secret)
    sha512 = hashlib.sha512
    hmac = str(hmac.new(secret, hash_data, sha512))

    header = {
        'User-Agent': 'My-First-Trade-Bot',
        'Rest-Key': key,
        'Rest-Sign': base64.b64encode(hmac),
        'Accept-encoding': 'GZIP',
        'Content-Type': 'application/x-www-form-urlencoded'
    }

    return urllib2.Request(base + path, data, header)

post_data = 'nonce=123'
request = makreq('abc123..', 'aBc7/+..', 'BTCUSD/money/ticker', post_data)
response = urllib2.urlopen(request, post_data)
# if gzip encoding, decode
# try to decode json into dictionary
# raise exception if response contains error key

And it fail out of the box:
Code:
$ python2.7 test.py 
Traceback (most recent call last):
  File "test.py", line 23, in <module>
    request = makereq(key, secret, 'BTCUSD/money/ticker', post_data)
  File "test.py", line 8, in makereq
    hmac = str(hmac.new(secret, hash_data, sha512))
UnboundLocalError: local variable 'hmac' referenced before assignment

I "fix" it by renaming the variable hmac to hmac2, and I get:

Code:
$ python2.7 test.py 
Traceback (most recent call last):
  File "test.py", line 24, in <module>
    response = urllib2.urlopen(request, post_data)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error

This is really clueless, if the server just give an 5xxx without any detail. The key/secret used in my case works fine on API_1, and I tried again by applying a new key/secret, which failed with exactly the same 500.

Do you have a code snippet of minimal working code to access an authenticated-user-only feature? E.g. to access wallet information would be perfect. I usually use python3 and am happy to morph a piece of working python2 code to a piece of working python3 code.

The documentation is a rather big topic, worth making up a mailing list or separate board. I feel guilty to make this disordered thread even longer.

The next example I can look up is this:

http://pastebin.com/aXQfULyq

Which also stopped working as soon as I need to do something that requires authentication. From their source it seems they don't even encrypt post data as part of their Sign, thus must fail authentication -> only work for public accessible URIs.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: zhangweiwu on July 12, 2013, 08:16:29 AM
On another topic, there is little information about how to authenticate against MtGox STREAMING API. Do I must overcome the API2 authentication problem in order to attempt Stream API?

Quote from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands
Quote
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called. Responses are op:result

So, if I use Streaming API I cannot use API2, right? I really need the feature like MONEY/ORDER/RESULT which doesn't seem to work in API_1†

--

† with API_1, MONEY/ORDER/RESULT always complain:
{'error': 'No data found', 'result': 'error', 'token': 'unknown_order_id'}

When I know the order ID is prefect existing and open, by double checking result from MONEY/ORDERS, and finding the order ID I request is byte-identical to what MONEY/ORDERS returns. I also know the POST parameters are correct, because if I change post parameter from 'order_id' to 'oid' (as the way it was returned from MONEY/ORDERS) I got missing order_id parameter error.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: zhangweiwu on July 12, 2013, 08:30:24 AM
with API_1, MONEY/ORDER/RESULT always complain:
{'error': 'No data found', 'result': 'error', 'token': 'unknown_order_id'}

When I know the order ID is prefect existing and open

Turns out this is exactly the problem: if the order is open, there won't be any result of it. Only closed orders can be inquired with this API.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 12, 2013, 01:46:59 PM
Hi zhangweiwu,

With regards to the example, sorry I thought I had tested it. I've pushed a new version that should work and uses the new tonce instead of nonce. Please note also that some API methods must be performed by GET now, and others by POST.

On another topic, there is little information about how to authenticate against MtGox STREAMING API. Do I must overcome the API2 authentication problem in order to attempt Stream API?

Quote from https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands
Quote
These commands require an API key and secret pair to sign requests. Any of the HTTP API version 1 methods can be called. Responses are op:result

So, if I use Streaming API I cannot use API2, right? I really need the feature like MONEY/ORDER/RESULT which doesn't seem to work in API_1†

--

† with API_1, MONEY/ORDER/RESULT always complain:
{'error': 'No data found', 'result': 'error', 'token': 'unknown_order_id'}

When I know the order ID is prefect existing and open, by double checking result from MONEY/ORDERS, and finding the order ID I request is byte-identical to what MONEY/ORDERS returns. I also know the POST parameters are correct, because if I change post parameter from 'order_id' to 'oid' (as the way it was returned from MONEY/ORDERS) I got missing order_id parameter error.

I've compiled some information about the socket api here https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master (https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master) with example python code.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: zhangweiwu on July 12, 2013, 02:54:26 PM
Hi zhangweiwu,

With regards to the example, sorry I thought I had tested it. I've pushed a new version that should work and uses the new tonce instead of nonce. Please note also that some API methods must be performed by GET now, and others by POST.


Thanks a lot! Decyphering the MtGox API really need some team work, because it is plain trial and error. The error message was never clear and some thing works something not. It is a waste of intelligence to try individually. For example I cannot figure out why I used to be able to repeatedly check order result this mid day and now I am not able to do it even once (HTTP Error 403: Forbidden) with exactly the same code -- guessing tounce can help exclude some causes. I am going to try it as soon as I am sober, now I am drinking because I got sick of it.

Sadly, It again showed that one doesn't have to be able to design well in order to occupy the market...


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: zhangweiwu on July 13, 2013, 04:35:42 AM
I've compiled some information about the socket api here https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master with example python code.

Thanks! tested and it works for me. Here is a minimalist sample that works on python3. There are a lot of changes needed for python3, including frequently converting str to bytes, incompatible urlencode() and lack of urllib2. This example uses python-requests to shorten the code in attempt to manifest the security measurement:
Code:
import hmac, base64, hashlib, urllib, time, requests

base = 'https://data.mtgox.com/api/2/'
key = ""
sec = ""

def request(get_or_post, path, inp={}):

    def sign(path, data): # not used by any other functions
        return hmac.new(base64.b64decode(bytes(sec, 'UTF-8')),
            bytes(path+chr(0)+data, 'UTF-8'), hashlib.sha512)

    if get_or_post == 'get':
        return requests.get(base+path)

    inp[u'tonce'] = str(int(time.time()*1e6))
    post_data = urllib.parse.urlencode(inp)
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded',
        'Rest-Key': key,
        'Rest-Sign': base64.b64encode(sign(path, post_data).digest())
    }
    return requests.post(base+path, data = post_data, headers = headers)

# example: get
#r = request('get', 'BTCUSD/money/ticker_fast')
#print(r.text)

# example: post
#r = request('post', 'BTCUSD/money/info')
#print(r.text)

# example: post with params (use your own order ID here)
#data = {'order': 'd5111d57-c495-48a0-923d-59b729dba331', 'type':'ask'}
#r = request('post', 'BTCUSD/money/order/result', data)
#print(r.text)

Output (activated the last most-complicated example)
Code:
!python3 /tmp/test.py
[No se ha escrito nada al disco desde el último cambio]
{"result":"success","data":{"order_id":"d5111d57-c495-48a0-923d-59b729dba331","trades":[{"trade_id":"1373688054789976","primary":"Y","currency":"USD","type":"ask
","properties":"limit","item":"BTC","amount":{"value":"0.01000000","value_int":"1000000","display":"0.01000000\u00a0BTC","display_short":"0.01\u00a0BTC","currenc
y":"BTC"},"price":{"value":"91.89874","value_int":"9189874","display":"$91.89874","display_short":"$91.90","currency":"USD"},"spent":{"value":"0.91899","value_in
t":"91899","display":"$0.91899","display_short":"$0.92","currency":"USD"},"date":"2013-07-13 04:00:54"}],"total_amount":{"value":"0.01000000","value_int":"100000
0","display":"0.01000000\u00a0BTC","display_short":"0.01\u00a0BTC","currency":"BTC"},"total_spent":{"value":"0.91899","value_int":"91899","display":"$0.91899","d
isplay_short":"$0.92","currency":"USD"},"avg_cost":{"value":"91.89900","value_int":"9189900","display":"$91.89900","display_short":"$91.90","currency":"USD"}}}


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 13, 2013, 04:43:18 PM
I've compiled some information about the socket api here https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master with example python code.

Thanks! tested and it works for me. Here is a minimalist sample that works on python3. There are a lot of changes needed for python3, including frequently converting str to bytes, incompatible urlencode() and lack of urllib2. This example uses python-requests to shorten the code in attempt to manifest the security measurement:

That's great, thanks :) If you don't mind, I've uploaded it to the example code section.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 19, 2013, 09:11:17 AM
okay i think i got banned from the mtgox API :(

Quote
Website is currently unreachable (1)
The website that you are trying to access is in Offline Mode, which means the server is not currently responding.
cloudflare.com

have fetched the trades once in 5 minutes. what shall i do?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 19, 2013, 12:08:15 PM
okay i think i got banned from the mtgox API :(

Quote
Website is currently unreachable (1)
The website that you are trying to access is in Offline Mode, which means the server is not currently responding.
cloudflare.com

have fetched the trades once in 5 minutes. what shall i do?

That doesn't sound like you've been blocked by cloudflare, it should be more explicit than that. More likely, MtGox was actually offline at that moment. Are you still getting that message?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 19, 2013, 02:08:02 PM
if i run the script from another IP everything is working fine! yes i get this message the since 24hours i think...

okay i think i got banned from the mtgox API :(

Quote
Website is currently unreachable (1)
The website that you are trying to access is in Offline Mode, which means the server is not currently responding.
cloudflare.com

have fetched the trades once in 5 minutes. what shall i do?

That doesn't sound like you've been blocked by cloudflare, it should be more explicit than that. More likely, MtGox was actually offline at that moment. Are you still getting that message?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 19, 2013, 02:42:54 PM
if i run the script from another IP everything is working fine! yes i get this message the since 24hours i think...

okay i think i got banned from the mtgox API :(

Quote
Website is currently unreachable (1)
The website that you are trying to access is in Offline Mode, which means the server is not currently responding.
cloudflare.com

have fetched the trades once in 5 minutes. what shall i do?

That doesn't sound like you've been blocked by cloudflare, it should be more explicit than that. More likely, MtGox was actually offline at that moment. Are you still getting that message?

That's weird, maybe you have been then. Try going to #mtgox on freenode IRC and asking for some help. Your best bet is probably 9am-5pm tokyo time so you can ask someone actually at MtGox.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on July 19, 2013, 03:53:36 PM
fixed my problem. i have switched the url from http to https...
cloudflare is really doing a bad job!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: marcomagno on July 30, 2013, 12:27:19 PM
Hi Guys, I have been beating my head on this for more than 10hours...

I have written the following javascript code, with the incredibly generous help of the un-official guide:

Code:
function auth() {
var RestKey= 'my API KEY';
var tonce = microtime();
var secret = 'my secret';

var message = 'BTCUSD/money/info' + '\0';
var RestSign = hmac_512(message,secret);
var path = 'https://data.mtgox.com/api/2/BTCUSD/money/info/';
var result = $.ajax({
                        url: path,
                        headers:{'tonce': tonce, 'Rest-Key': RestKey, 'Rest-Sign': RestSign},
                        type: 'POST',
                        success: function (data) {console.log(data);}
                        });
}

where hmac_512 is calculated with jsSHA:

Code:
function hmac_512(message, secret) {
    var shaObj = new jsSHA(message, "TEXT");
    var hmac = shaObj.getHMAC(secret, "B64", "SHA-512", "B64");
    return hmac;
}

the issue is that i keep getting the following error. I think I have tried all possible combinations, but no success. It seems like my headers are not accepted. It does not seem to be an origin-related kind of issue, because that is a different message normally.

Quote
OPTIONS https://data.mtgox.com/api/2/BTCUSD/money/info/ Request header field Rest-Sign is not allowed by Access-Control-Allow-Headers. jquery.min.js:4

XMLHttpRequest cannot load https://data.mtgox.com/api/2/BTCUSD/money/info/. Request header field Rest-Sign is not allowed by Access-Control-Allow-Headers.


Thanks in advance to any helpful soul that could take me out of my misery...


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on July 30, 2013, 11:19:15 PM
Hi Guys, I have been beating my head on this for more than 10hours...

I have written the following javascript code, with the incredibly generous help of the un-official guide:

Code:
function auth() {
var RestKey= 'my API KEY';
var tonce = microtime();
var secret = 'my secret';

var message = 'BTCUSD/money/info' + '\0';
var RestSign = hmac_512(message,secret);
var path = 'https://data.mtgox.com/api/2/BTCUSD/money/info/';
var result = $.ajax({
                        url: path,
                        headers:{'tonce': tonce, 'Rest-Key': RestKey, 'Rest-Sign': RestSign},
                        type: 'POST',
                        success: function (data) {console.log(data);}
                        });
}

where hmac_512 is calculated with jsSHA:

Code:
function hmac_512(message, secret) {
    var shaObj = new jsSHA(message, "TEXT");
    var hmac = shaObj.getHMAC(secret, "B64", "SHA-512", "B64");
    return hmac;
}

the issue is that i keep getting the following error. I think I have tried all possible combinations, but no success. It seems like my headers are not accepted. It does not seem to be an origin-related kind of issue, because that is a different message normally.

Quote
OPTIONS https://data.mtgox.com/api/2/BTCUSD/money/info/ Request header field Rest-Sign is not allowed by Access-Control-Allow-Headers. jquery.min.js:4

XMLHttpRequest cannot load https://data.mtgox.com/api/2/BTCUSD/money/info/. Request header field Rest-Sign is not allowed by Access-Control-Allow-Headers.


Thanks in advance to any helpful soul that could take me out of my misery...

Most (all?) web browsers don't allow you to send ajax requests to other domains, as it presents a security vulnerability. In some browsers you can lower the security settings I think, and you may be able to get it working from localhost, but otherwise you'll need to use an intermediate server-side script to perform the request, and use ajax to talk to this script instead of MtGox.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: marcomagno on July 31, 2013, 02:43:58 PM
Thanks. I was scouting for another set of few hours to understand how a trading bot extension for Chrome (entirely written in JS) could do it. Then i found that there is a setting that applies only to extensions to disable access control. One can disable access control for Chrome but then the browser needs to be launched with arguments.

I will go for your idea and create a php page server side to handle the requests.

Unwanted setback... :-\

but, thanks a lot!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: divergenta on August 02, 2013, 02:10:45 PM
I've compiled some information about the socket api here https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master with example python code.

Thanks! tested and it works for me. Here is a minimalist sample that works on python3. There are a lot of changes needed for python3, including frequently converting str to bytes, incompatible urlencode() and lack of urllib2. This example uses python-requests to shorten the code in attempt to manifest the security measurement:
Code:
import hmac, base64, hashlib, urllib, time, requests

base = 'https://data.mtgox.com/api/2/'
key = ""
sec = ""

def request(get_or_post, path, inp={}):

    def sign(path, data): # not used by any other functions
        return hmac.new(base64.b64decode(bytes(sec, 'UTF-8')),
            bytes(path+chr(0)+data, 'UTF-8'), hashlib.sha512)

    if get_or_post == 'get':
        return requests.get(base+path)

    inp[u'tonce'] = str(int(time.time()*1e6))
    post_data = urllib.parse.urlencode(inp)
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded',
        'Rest-Key': key,
        'Rest-Sign': base64.b64encode(sign(path, post_data).digest())
    }
    return requests.post(base+path, data = post_data, headers = headers)

# example: get
#r = request('get', 'BTCUSD/money/ticker_fast')
#print(r.text)

# example: post
#r = request('post', 'BTCUSD/money/info')
#print(r.text)

# example: post with params (use your own order ID here)
#data = {'order': 'd5111d57-c495-48a0-923d-59b729dba331', 'type':'ask'}
#r = request('post', 'BTCUSD/money/order/result', data)
#print(r.text)

Output (activated the last most-complicated example)
Code:
!python3 /tmp/test.py
[No se ha escrito nada al disco desde el último cambio]
{"result":"success","data":{"order_id":"d5111d57-c495-48a0-923d-59b729dba331","trades":[{"trade_id":"1373688054789976","primary":"Y","currency":"USD","type":"ask
","properties":"limit","item":"BTC","amount":{"value":"0.01000000","value_int":"1000000","display":"0.01000000\u00a0BTC","display_short":"0.01\u00a0BTC","currenc
y":"BTC"},"price":{"value":"91.89874","value_int":"9189874","display":"$91.89874","display_short":"$91.90","currency":"USD"},"spent":{"value":"0.91899","value_in
t":"91899","display":"$0.91899","display_short":"$0.92","currency":"USD"},"date":"2013-07-13 04:00:54"}],"total_amount":{"value":"0.01000000","value_int":"100000
0","display":"0.01000000\u00a0BTC","display_short":"0.01\u00a0BTC","currency":"BTC"},"total_spent":{"value":"0.91899","value_int":"91899","display":"$0.91899","d
isplay_short":"$0.92","currency":"USD"},"avg_cost":{"value":"91.89900","value_int":"9189900","display":"$91.89900","display_short":"$91.90","currency":"USD"}}}

Hi, Currently trying this code. Works well with get but when I try send I get syntax error at this line:
    inp[u'tonce'] = str(int(time.time()*1e6))

why is that? I'm using python3.2 @ linux mint


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on August 05, 2013, 09:13:56 PM
I've compiled some information about the socket api here https://bitbucket.org/nitrous/mtgox-api/src/master/socket?at=master with example python code.

Thanks! tested and it works for me. Here is a minimalist sample that works on python3. There are a lot of changes needed for python3, including frequently converting str to bytes, incompatible urlencode() and lack of urllib2. This example uses python-requests to shorten the code in attempt to manifest the security measurement:
Code:
import hmac, base64, hashlib, urllib, time, requests

base = 'https://data.mtgox.com/api/2/'
key = ""
sec = ""

def request(get_or_post, path, inp={}):

    def sign(path, data): # not used by any other functions
        return hmac.new(base64.b64decode(bytes(sec, 'UTF-8')),
            bytes(path+chr(0)+data, 'UTF-8'), hashlib.sha512)

    if get_or_post == 'get':
        return requests.get(base+path)

    inp[u'tonce'] = str(int(time.time()*1e6))
    post_data = urllib.parse.urlencode(inp)
    headers = {
        'Content-Type': 'application/x-www-form-urlencoded',
        'Rest-Key': key,
        'Rest-Sign': base64.b64encode(sign(path, post_data).digest())
    }
    return requests.post(base+path, data = post_data, headers = headers)

# example: get
#r = request('get', 'BTCUSD/money/ticker_fast')
#print(r.text)

# example: post
#r = request('post', 'BTCUSD/money/info')
#print(r.text)

# example: post with params (use your own order ID here)
#data = {'order': 'd5111d57-c495-48a0-923d-59b729dba331', 'type':'ask'}
#r = request('post', 'BTCUSD/money/order/result', data)
#print(r.text)

Output (activated the last most-complicated example)
Code:
!python3 /tmp/test.py
[No se ha escrito nada al disco desde el último cambio]
{"result":"success","data":{"order_id":"d5111d57-c495-48a0-923d-59b729dba331","trades":[{"trade_id":"1373688054789976","primary":"Y","currency":"USD","type":"ask
","properties":"limit","item":"BTC","amount":{"value":"0.01000000","value_int":"1000000","display":"0.01000000\u00a0BTC","display_short":"0.01\u00a0BTC","currenc
y":"BTC"},"price":{"value":"91.89874","value_int":"9189874","display":"$91.89874","display_short":"$91.90","currency":"USD"},"spent":{"value":"0.91899","value_in
t":"91899","display":"$0.91899","display_short":"$0.92","currency":"USD"},"date":"2013-07-13 04:00:54"}],"total_amount":{"value":"0.01000000","value_int":"100000
0","display":"0.01000000\u00a0BTC","display_short":"0.01\u00a0BTC","currency":"BTC"},"total_spent":{"value":"0.91899","value_int":"91899","display":"$0.91899","d
isplay_short":"$0.92","currency":"USD"},"avg_cost":{"value":"91.89900","value_int":"9189900","display":"$91.89900","display_short":"$91.90","currency":"USD"}}}

Hi, Currently trying this code. Works well with get but when I try send I get syntax error at this line:
    inp[u'tonce'] = str(int(time.time()*1e6))

why is that? I'm using python3.2 @ linux mint

Hi, sorry for the late reply, I've been away from the internet for a few days. That's strange, I tried on my homebrew python 3.3.2 install and that line worked fine. Can you give any more diagnostics from the syntax error?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: Lotus on August 06, 2013, 07:24:55 AM
Is there a reliable way to get completed/past order IDs to pass them to the "/money/order/result" query? It seems that whatever they return in the wallet history is not what this API expects.

BTW, great job nitrous on the documentation in general.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on August 06, 2013, 12:52:38 PM
Is there a reliable way to get completed/past order IDs to pass them to the "/money/order/result" query? It seems that whatever they return in the wallet history is not what this API expects.

BTW, great job nitrous on the documentation in general.

Unfortunately, not really.

Ideally you would need to capture the order ID from a 'money/order/add' request. If not, I believe you may be able to get the order ID from the websocket api if and when a change occurs while the trade is being processed. After it has completed, however, I don't believe there is a way to get its ID.

Your only other option is to collate the information provided by the wallet history to try to collect the information you need.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: Lotus on August 07, 2013, 05:47:54 AM
Hmm... I figured as much. I hope they would fix their API soon. Thanks.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MusX on September 02, 2013, 09:32:20 PM
Hi,

Anybody could recommend how to handle cancel order launched on inactive order?
You can simply have an open order, fetch it's oid, but before you will launch cancel particular oid somebody can fill it.
Currently the mtgox api returns: Error: Internal Server Error, I'm almost sure it is with result = 'error'.
Would be nice direct "cancel all open orders on particular currency" method :)

Another question is time threshold described in the most logic way to code "antispam guard", there are some methods which has different api call limits than others in the different time frames, would be good to setup the antispam guardian in the most fitted way.

I remember there was an issue with full trades history download, I can code a script which can download the recent trades history and the csv full history from bitcoincharts and match them. It can be better way than some cloud solution. If anyone interested please write an output format: csv, sql script, sqlite db.

Regards


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on September 02, 2013, 10:01:28 PM
Hi,

Anybody could recommend how to handle cancel order launched on inactive order?
You can simply have an open order, fetch it's oid, but before you will launch cancel particular oid somebody can fill it.
Currently the mtgox api returns: Error: Internal Server Error, I'm almost sure it is with result = 'error'.
Would be nice direct "cancel all open orders on particular currency" method :)

Another question is time threshold described in the most logic way to code "antispam guard", there are some methods which has different api call limits than others in the different time frames, would be good to setup the antispam guardian in the most fitted way.

I remember there was an issue with full trades history download, I can code a script which can download the recent trades history and the csv full history from bitcoincharts and match them. It can be better way than some cloud solution. If anyone interested please write an output format: csv, sql script, sqlite db.

Regards

You usually shouldn't get an error. Errors should only occur if the server is having problems or if the order has already been filled. There is no way to cancel all open orders at once. Instead you can do this by calling BTCUSD/money/orders (https://bitbucket.org/nitrous/mtgox-api/overview#markdown-header-moneyorders), and then cancelling each of the orders individually.

Usually, a maximum of 10 API requests per 10 seconds will be a good antispam-guard. If you want to do more frequent API calls, try using MtGox's websocket API (https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands)

There are already some people working on this:
  • https://bitcointalk.org/index.php?topic=282154.0 -- user bitranox (https://bitcointalk.org/index.php?action=profile;u=146909)
  • http://wizb.it -- user whydifficult (https://bitcointalk.org/index.php?action=profile;u=99949)
But if you think you have a different approach and want to do it, feel free to create your own script :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MusX on September 02, 2013, 10:20:23 PM
or if the order has already been filled.
this the the case which I would like to handle, is there any better way to catch that error other than suppress market api error? there should be, open orders can disappear in any time.

MtGox's websocket API (https://en.bitcoin.it/wiki/MtGox/API/Streaming#Authenticated_commands)
this would be the whole new project which currently I don't have time to code.

But if you think you have a different approach and want to do it, feel free to create your own script :)
I think I will try


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on September 02, 2013, 10:23:15 PM
or if the order has already been filled.
this the the case which I would handle, is there any better way to catch that error other than suppress market api error? there should be, open orders can disappear in any time.

Sorry, no, you'll just have to check for errors yourself. Anyway, the HTTP API can be unreliable at times so you will need to handle unexpected errors for all methods.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MusX on September 03, 2013, 01:16:53 AM
But if you think you have a different approach and want to do it, feel free to create your own script :)
I think I will try
MtGox trades history download R script - https://bitcointalk.org/index.php?topic=286755
no bigquery, no middle hosts, simple logic, opensource, output to csv/sqlite


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: imrer on September 04, 2013, 10:32:28 PM
I can't believe they don't offer useful documentation about it.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MusX on September 04, 2013, 11:29:46 PM
there are more issues with their data/doc info. There is a threshold in the TID field, 4 of the past trades are with 0 amount. It needs to be handled in the volume weighting process. I recommend my scripts described above to fetch it's full and up to date archive.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: xchrix on September 05, 2013, 08:46:01 AM
I can't believe they don't offer useful documentation about it.

they gox you everytime you have something to do with them ;) for my part i stay far away from mtgox


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: alikim on September 21, 2013, 12:46:45 PM
Hi,

I have a couple of question about connecting to mtgox servers.

I tried to use your ws.client.html file, first of all https://socketio.mtgox.com/socket.io/socket.io.js doesn't seem to be there anymore, so I had to download one from repository. Now I'm trying to connect to ws://echo.websocket.org just for testing and all I got is "Connection timeout message".
I tried ws://echo.websocket.org, http://echo.websocket.org, ws://echo.websocket.org:80/, http://echo.websocket.org:80/ - how exactly do you input the server and the port to connect?

Also I tried to connect to socketio.mtgox.com/mtgox using a C++ program, can't connect to it either.
Can easily connect to ws://echo.websocket.org but not to mtgox one.
Do you know maybe there are some peculiarities in sending the handshake to mtgox, so it would switch to websockets?

Thank you!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on September 21, 2013, 01:03:24 PM
Hi,

I have a couple of question about connecting to mtgox servers.

I tried to use your ws.client.html file, first of all https://socketio.mtgox.com/socket.io/socket.io.js doesn't seem to be there anymore, so I had to download one from repository. Now I'm trying to connect to ws://echo.websocket.org just for testing and all I got is "Connection timeout message".
I tried ws://echo.websocket.org, http://echo.websocket.org, ws://echo.websocket.org:80/, http://echo.websocket.org:80/ - how exactly do you input the server and the port to connect?

Also I tried to connect to socketio.mtgox.com/mtgox using a C++ program, can't connect to it either.
Can easily connect to ws://echo.websocket.org but not to mtgox one.
Do you know maybe there are some peculiarities in sending the handshake to mtgox, so it would switch to websockets?

Thank you!

Hi alikim,

Sorry, my ws.client.html file is specific to socket.io servers. MtGox's socket.io server is unreliable, deprecated and is being phased out in favour of their websocket server (wss://websocket.mtgox.com/mtgox). I believe that it is not difficult to set up websocket access in JS (even using socket.io as your library), however ws.client.html was just a quick little project to play around with the API and I don't really have much free time at the moment to keep it up to date. The reason you could not connect to those servers was because they are purely websocket and my client cannot interface with them.

Perhaps you might have some luck adapting my client or rewriting it using this resource http://www.tutorialspoint.com/html5/html5_websocket.htm (http://www.tutorialspoint.com/html5/html5_websocket.htm)? If anyone updates my client to support the websocket protocol I'd be happy to update the documentation with it :)

Good luck!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinLeader on September 29, 2013, 08:52:26 PM
Anyone knows a PHP Class for the MtGox API v2 ?

Or should I make one?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on September 29, 2013, 09:06:23 PM
Anyone knows a PHP Class for the MtGox API v2 ?

Or should I make one?

All I've seen is this -- https://en.bitcoin.it/wiki/MtGox/API/HTTP#PHP
Not really a fully functional class though, so if you want to then I'd say go for it! I'll link it in my documentation if you want as well :)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinLeader on September 29, 2013, 09:27:48 PM
Anyone knows a PHP Class for the MtGox API v2 ?

Or should I make one?

All I've seen is this -- https://en.bitcoin.it/wiki/MtGox/API/HTTP#PHP
Not really a fully functional class though, so if you want to then I'd say go for it! I'll link it in my documentation if you want as well :)
Yeah well that is for v1.

I'm working on a v2 Class now.

Update: I got it up and running, I will post the link when I have implemented all API functions :)
Update 30 september 2013: I have released v0.1, you can download it here MtGox-PHP-API-V2 (https://github.com/pathaksa/MtGox-PHP-API-V2)


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on November 21, 2013, 12:42:05 PM
where i can find MTGOX API Key Secret?

Go to https://www.mtgox.com/security and click on 'Advanced API Key Creation'. Enter a name for your key, select the rights you want:

  • Get Info - find out your balance, fees, orders, etc
  • Trade - buy and sell
  • Deposit
  • Withdraw
  • Merchant - manage merchant services

Then click create key, and you will be given your key and secret. Note down your secret because you will not be able to get it again. If you forget your secret or think someone else is using your key, you can delete the key and make another one. Click on 'Current API Keys'. From here, you can change the rights, and delete the key if necessary using the red cross button. Be careful to only allow the rights you need, you don't want someone else draining your account because you allowed withdrawal rights.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: mfarshada on December 11, 2013, 11:51:08 PM
Dear nitrous,

First and foremost, thank you for kindly writing the API v2 documentation!

I was just going to add a small update. In my recent tests, the method "money/trades/fetch" with no "since" argument returned all transactions for the past 24 hours. However, when "since" is passed, it behaves as described in your current documentation.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: mfarshada on December 12, 2013, 12:05:34 AM
Anyone knows a PHP Class for the MtGox API v2 ?

Or should I make one?

All I've seen is this -- https://en.bitcoin.it/wiki/MtGox/API/HTTP#PHP
Not really a fully functional class though, so if you want to then I'd say go for it! I'll link it in my documentation if you want as well :)

Here is a more complete version of it:
https://github.com/Someguy123/MtGOX-PHP-API/blob/master/Gox.class.php

It is functional and uses v2 authentication method. Most of its functions use still-functioning v0 methods, but very little modification is required to make them v2 compatible. It contains only a POST query function, so if you would like to use GET-only methods such as "fetch", the following seems to work:

Code:
	public function mtgox_query_get($path, array $req = array())
{
// generate the GET data string
$get_data = http_build_query($req, '', '&');

// our curl handle (initialize if required)
static $ch = null;
if (is_null($ch))
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT,
'Mozilla/4.0 (compatible; MtGox PHP client; ' . php_uname('s') . '; PHP/' .
phpversion() . ')');
}
curl_setopt($ch, CURLOPT_URL, 'https://data.mtgox.com/api/' . $path.'?'.$get_data);

// run the query
$res = curl_exec($ch);
if ($res === false)
throw new Exception('Could not get reply: ' . curl_error($ch));
$dec = json_decode($res, true);
if (!$dec)
throw new Exception('Invalid data received, please make sure connection is working and requested API exists');
return $dec;
}


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on December 12, 2013, 11:41:15 PM
Dear nitrous,

First and foremost, thank you for kindly writing the API v2 documentation!

I was just going to add a small update. In my recent tests, the method "money/trades/fetch" with no "since" argument returned all transactions for the past 24 hours. However, when "since" is passed, it behaves as described in your current documentation.


Dear mfarshada,

I'm glad you've found it useful :)

Thanks for finding this! I've pushed it to the docs now.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 02, 2014, 04:34:38 PM
hy,
i cant get this to work, whats wrong? (couldnt find anything on the forum or the net; i just want to get the most recent trades including tid)

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1388679721
{"result":"success","data":[]}

i know that <218867 its the tid and after that it should be the unix timestamp with 10 digits, but for everything > 218867 it just returns nothing


thx!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 02, 2014, 05:39:12 PM
hy,
i cant get this to work, whats wrong? (couldnt find anything on the forum or the net; i just want to get the most recent trades including tid)

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1388679721
{"result":"success","data":[]}

i know that <218867 its the tid and after that it should be the unix timestamp with 10 digits, but for everything > 218867 it just returns nothing


thx!

Hi,

Actually, it needs to be the unix timestamp with 16 digits! After 218867 they switched to micro timestamps, so you need to include the microseconds as well. Try:

https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1388679721000000

In fact, you're actually supplying the tid after which you want the trades, not the date (though after 218867, they're equivalent). However, if you give too small a tid, the API won't return any results (to try to conserve bandwidth I think). Therefore you need to be within something like 86400000000 of your desired tid in order to get results.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 02, 2014, 06:25:12 PM
thanks a lot; it´s especially tricky as the api outputs 10 digits (date:..)..
:)



Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 02, 2014, 09:29:00 PM
thanks a lot; it´s especially tricky as the api outputs 10 digits (date:..)..
:)



No problem :) To understand it better, have a look at the 'tid' values -- these are what you're passing to the 'since' argument.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: albus on January 07, 2014, 04:54:17 AM
Hi,
is there a reason why the API uses only POST for private function and never GET ?
The security difference seems small, or is it because GET is cacheable ? In that case they could set the cache reset at a very low value...
Or am I missing something else ?
thanks


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 07, 2014, 05:28:23 AM
Hi,
is there a reason why the API uses only POST for private function and never GET ?
The security difference seems small, or is it because GET is cacheable ? In that case they could set the cache reset at a very low value...
Or am I missing something else ?
thanks


All respectable secure site use POST with HTTPS because its essential to effectively hide potentially secret information:
  • The url and query string of a request are easily accessible and often recorded, so it would undermine the entire point of HTTPS to use GET.
  • Another reason is that GET query strings have a maximum length which is quite short, like 1 or 2 kb, so POST makes sense for this reason as well.
  • And yet another, as you identified, is the problem of caching; and no, they would never consider setting the cache refresh at a very low value because that would eliminate the whole point of a cache, to reduce server load (which often has been exploited to DDOS MtGox, so I'm pretty sure they don't want to take any chances with this).

Of course, you could argue that a computer program accessing the API probably won't record the URLs and the query strings are likely far less than 1kb and you could control caching for individual resources, but frankly there's little reason to use GET and every to use POST.

Is there some tool or library you're trying to use which doesn't support POST requests?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: albus on January 07, 2014, 10:33:38 AM
Quote
Is there some tool or library you're trying to use which doesn't support POST requests?

Thanks for the answer, it was more a curiosity question.

I am trying write code to access the MtGox API and I was wondering why all exchanges (in the BTC world) use POST everywhere there is authentication, whereas it is not the case in many other APIs like Amazon, Twitter, Twilio etc...

About your first point, isn't it possible to record all the data for a hacker, and not just the URL ? In that case the security improvement is really about using the API in a browser and not showing the request to user or some server recording the requests, but in that case anyway they all use some kind of HMAC. It seems this would protect only against very trivial attacks, that are anyway put off by using SSL+HMAC.
For example a GET on your open orders would show your request (to your browser, but not elsewhere as we are in SSL) but sign it so it cannot be forged to do something else with it.

I am a bit new to HTTP so might be this is not the right way to see it.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: albus on January 07, 2014, 03:37:24 PM
after searching around a bit (Wikipedia... where I should have started...) I understood your explanation a bit better, what I did not get is that URL does not get encrypted as well when using SSL...

thanks for your explanations !


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 07, 2014, 03:58:26 PM
after searching around a bit (Wikipedia... where I should have started...) I understood your explanation a bit better, what I did not get is that URL does not get encrypted as well when using SSL...

thanks for your explanations !

No problem :)

The URL should also get encrypted, but it is also stored in server logs. It is certainly possible to work around all these things, but with all the things you'd have to work around you'd probably miss something or introduce a bug. Simply put, it's unwise to use GET when passing secure information, and far easier to use POST, whose attributes make perfect sense for use with SSL.

Hmm, I haven't used those APIs but if you're right, that's really surprising. I wouldn't expect such large companies to use GET for secure requests, especially Amazon where money is involved.

EDIT: Another important point, though obviously not a factor with a simple text API, is that on a webpage any third-party resources that are accessed will be sent a referrer url, which is the url of the webpage (query string and all!).


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 08, 2014, 07:47:17 PM
hy nitrous!

what do you consider the best method to always get the last trade (every latest trade)?

i tried (python):

get the latest time

letztertid=str(int(time.time()*1000000));

fetch trades since that

x=req('BTCUSD/money/trades/fetch?since='+jetzt, {}, True)

take the latest trade from that result and repeat.

but its lagging 3-4 trades. if i try something like that

letztertid=str(int(time.time()*1000000-864000));

it shows one lates trade and then lags some trades and then shows another latest trade.

i also tried to start with time*1000000, take the tid of the latest trade and fetch all trades since that, but that doesnt return no data (i think on your website you describe this as beeing some kind of ddos protection).

what is the minimum x i can subtract from time?
thx!



Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 09, 2014, 12:17:18 PM
hy nitrous!

what do you consider the best method to always get the last trade (every latest trade)?

i tried (python):

get the latest time

letztertid=str(int(time.time()*1000000));

fetch trades since that

x=req('BTCUSD/money/trades/fetch?since='+jetzt, {}, True)

take the latest trade from that result and repeat.

but its lagging 3-4 trades. if i try something like that

letztertid=str(int(time.time()*1000000-864000));

it shows one lates trade and then lags some trades and then shows another latest trade.

i also tried to start with time*1000000, take the tid of the latest trade and fetch all trades since that, but that doesnt return no data (i think on your website you describe this as beeing some kind of ddos protection).

what is the minimum x i can subtract from time?
thx!



I'm not quite sure what behaviour you're seeing...
Visiting BTCUSD/money/trades/fetch, the last few trades had the following tids:

Code:
1. 1389268906937622
2. 1389268907156855
3. 1389268907286795
4. 1389268919402914
5. 1389268919674069

Then going to fetch?since=1389268906937622, ie. the first one, gives:

Code:
2. 1389268907156855
3. 1389268907286795
4. 1389268919402914
5. 1389268919674069
6. 1389269081502565
7. 1389269099578993
8. 1389269100231335
9. 1389269128381212

It returned all trades after the tid I specified, as expected. If I subtract 1 (this should be the minimum you need to use for x) from tid#1, fetch?since=1389268906937621, then the results also include the first trade as well:

Code:
1. 1389268906937622
2. 1389268907156855
3. 1389268907286795
4. 1389268919402914
5. 1389268919674069
6. 1389269081502565
7. 1389269099578993
8. 1389269100231335
9. 1389269128381212

Also, you probably shouldn't use time.time() as it's very possible there won't be any trades that recent. On your first request, just call BTCUSD/money/trades/fetch instead, then start using the last trades from the last results set.

Anyway, I don't seem to be seeing your problem, all the results I got were exactly as I expected, what do you mean by lagging 3-4 trades?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 09, 2014, 01:26:16 PM
hy! thx for your help, i will test that out with -1. i was comparing bitcoinwisdom with the output of my script and as my script only displayed the last tid, bitcoinwisdom showed some trades in between that. but with -1 i didnt yet experiment.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 12, 2014, 07:13:04 PM
once=str(int(time.time()*1000000-1))
works like a charm :)

next thing i need is the orderbook. all i found is

/money/depth/fetch

but i think i might run into ddos protection, cause what i want to do, is to closely watch the orderbook max 6 btc away from the last price, so i would have to update it at least after each new trade. websocket isnt for me.. do you have a tip for me?
thx!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: MusX on January 12, 2014, 07:41:32 PM
once=str(int(time.time()*1000000-1))
works like a charm :)

next thing i need is the orderbook. all i found is

/money/depth/fetch

but i think i might run into ddos protection, cause what i want to do, is to closely watch the orderbook max 6 btc away from the last price, so i would have to update it at least after each new trade. websocket isnt for me.. do you have a tip for me?
thx!
there is a 'quote' method on mtgox - check it - as I remember it works only one way, not sure which
thread about mtgox market depth (much more than 6btc): https://bitcointalk.org/index.php?topic=194868


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 15, 2014, 01:12:00 PM
once=str(int(time.time()*1000000-1))
works like a charm :)

next thing i need is the orderbook. all i found is

/money/depth/fetch

but i think i might run into ddos protection, cause what i want to do, is to closely watch the orderbook max 6 btc away from the last price, so i would have to update it at least after each new trade. websocket isnt for me.. do you have a tip for me?
thx!

Glad it works :)

Sorry, unfortunately the only way to maintain a realtime copy of the orderbook is with the websocket - there is a maximum of 5 requests per hour for /money/depth/fetch, so it is simply impossible to call it any more often than once per 12 minutes, which I assume is not good enough for your purposes. If you can't use the websocket, you may want to reconsider your timescale or strategy. (What is wrong with using the websocket though? I've successfully used the websocket in python using the websocket-client (https://pypi.python.org/pypi/websocket-client/) package).

The thread posted by MusX says the same thing.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 18, 2014, 05:51:41 PM
so websockets ..
https://pypi.python.org/pypi/websocket-client/
no classes for me as example, cause i dont get 2 new things combined togehter (classes + websockets)
so i´m trying this first ("Another example"):
http://stackoverflow.com/questions/3142705/is-there-a-websocket-client-implemented-for-python
hard time to make sense of the websocket module calls..



Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 18, 2014, 07:30:57 PM
ok, here it is for all python-noobs as me - for starters:

Code:
#!/usr/bin/python
import websocket
import thread
import time

def on_message(ws, message):
        print 'ola, messages incoming';
        print message;
        print 'endofmessagedude';


def on_error(ws, error):
        print 'ola, error incoming:';
        print error;

def on_close(ws):
        print "### closed ###";

def fuckitin(ws):
    def run(*args):
        #for i in range(5):
         #   time.sleep(1)
          #  ws.send("1")
            #ws.send("Hello %d" % i)
        #time.sleep(1)
        #ws.send('');
        ws.close()
        print "thread terminating..."
    thread.start_new_thread(run, ())


if __name__ == "__main__":

    websocket.enableTrace(True)
    ws = websocket.WebSocketApp("wss://websocket.mtgox.com:443/mtgox?Currency=USD",
                                on_message = on_message,
                                on_error = on_error,
                                on_close = on_close)
    #ws.on_open = fuckitin

    ws.run_forever()


how do i tell  him which channel i want?


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 19, 2014, 12:27:08 AM
ok, here it is for all python-noobs as me - for starters:

Code:
#!/usr/bin/python
import websocket
import thread
import time

def on_message(ws, message):
        print 'ola, messages incoming';
        print message;
        print 'endofmessagedude';


def on_error(ws, error):
        print 'ola, error incoming:';
        print error;

def on_close(ws):
        print "### closed ###";

def fuckitin(ws):
    def run(*args):
        #for i in range(5):
         #   time.sleep(1)
          #  ws.send("1")
            #ws.send("Hello %d" % i)
        #time.sleep(1)
        #ws.send('');
        ws.close()
        print "thread terminating..."
    thread.start_new_thread(run, ())


if __name__ == "__main__":

    websocket.enableTrace(True)
    ws = websocket.WebSocketApp("wss://websocket.mtgox.com:443/mtgox?Currency=USD",
                                on_message = on_message,
                                on_error = on_error,
                                on_close = on_close)
    #ws.on_open = fuckitin

    ws.run_forever()


how do i tell  him which channel i want?

There are different ways depending on the type of channel, try sending the following:

Code:
{
    "op": "mtgox.subscribe",
    "channel": "channel-id"
}

where channel-id is one from http://data.mtgox.com/api/2/stream/list_public. You can do this as a python object, and then use json.dumps (import the json module). Have a look here for more information on websocket methods :- https://en.bitcoin.it/wiki/MtGox/API/Streaming. There's some conflicting information, but it should be helpful.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on January 19, 2014, 12:29:34 PM
thx again, your tips are really helpful :)

in principle i can do what i want now.

i can subscribe to any channel listed on
https://mtgox.com/api/2/stream/list_public

with this:

Code:
message={
"op":"mtgox.subscribe",
"channel":"trade.lag"
}
ws.send(json.dumps(message))

or i can have a single channel opened by just modifying the uri and not doing any send:


Code:
 ws = websocket.WebSocketApp("wss://websocket.mtgox.com:443/?Channel=trade.BTC",
                                on_message = on_message,
                                on_error = on_error,
                                on_close = on_close)

(btw that was in the comments of prof7bits goxapi.py, line 1137, https://github.com/prof7bit/goxtool/blob/master/goxapi.py)


so, the ws: uri is important, if calling
wss://websocket.mtgox.com:443/mtgox?Currency=USD"
i can subscribe to additional channels and always get btc.depth and btc.trades but cant unsubscribe to btc.depth and btc.trades.

if using
wss://websocket.mtgox.com:443
i get
"{\"message\":\"Now online (no channels)\",\"op\":\"remark\"}"
and can subscribe to any channel.

so i think unsubscribe may be only usable to unsubscribe channels one subscribed but not for avoiding messages like btc.trades that result from calling
wss://websocket.mtgox.com:443/mtgox?Currency=USD"






Title: Re: MtGox API version 2: Unofficial Documentation
Post by: nitrous on January 19, 2014, 02:35:52 PM

so, the ws: uri is important, if calling
wss://websocket.mtgox.com:443/mtgox?Currency=USD"
i can subscribe to additional channels and always get btc.depth and btc.trades but cant unsubscribe to btc.depth and btc.trades.

if using
wss://websocket.mtgox.com:443
i get
"{\"message\":\"Now online (no channels)\",\"op\":\"remark\"}"
and can subscribe to any channel.

so i think unsubscribe may be only usable to unsubscribe channels one subscribed but not for avoiding messages like btc.trades that result from calling
wss://websocket.mtgox.com:443/mtgox?Currency=USD"

To unsubscribe, try using

Code:
{
    "op": "unsubscribe",
    "channel": "id"
}

If I remember correctly, I think the id has to be the long guid code instead of the channel name when you unsubscribe.


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on February 01, 2014, 10:48:54 AM
hy,
when receiving ticker data by websocket, is there another possibility to get the volume of the last_local trade other than taking the difference of the actual volume to the subsequent volume?
thx!


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinPappi on February 13, 2014, 11:07:42 AM
Hi Guys

Guidance on Calculating Depth Or Order Book.

From what I understand so far.

1. A initial HTTP request has to be made to get the full order book (https://data.mtgox.com/api/2/BTCUSD/money/depth/fetch)
2.After receiving the full order book. You can then calculate the changes using websocket updates a soon below.


My objective is to obtain and keep track of the order book, what is the best way to do this. Are the above steps the correct way ?

Are there some factor or calculations  that need to be made to get the actual volume. "total_volume_int" is too large and "volume"  is deprecated.

Code:
{

    "channel":"24e67e0d-1cad-4cc0-9e7a-f8523ef460fe",
    "channel_name":"depth.BTCUSD",
    "op":"private",
    "origin":"broadcast",
    "private":"depth",
    "depth":{
        "price":"511.55897",
        "type":1,
        "type_str":"ask",
        "volume":"10.5",
        "price_int":"51155897",
        "volume_int":"1050000000",
        "item":"BTC",
        "currency":"USD",
        "now":"1392264454601642",
        "total_volume_int":"1050000000"
    },
    "stamp":1392264454602040

}


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: wilfried on February 13, 2014, 11:41:15 AM
i use vol_int: if its 0 i delete the order from my db, if its not 0 i replace every order where price matches; but you need to delete trades from the orderbook too


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinPappi on February 13, 2014, 02:26:42 PM
i use vol_int: if its 0 i delete the order from my db, if its not 0 i replace every order where price matches; but you need to delete trades from the orderbook too

Thanks. Can you tell me what  volume_int represents. Looking on other charts and third party order book data like bitcoincharts. The value that comes back is way too large to be the volume at that single price. I am missing something in understanding what is being returned


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinPappi on February 14, 2014, 01:03:49 AM
i use vol_int: if its 0 i delete the order from my db, if its not 0 i replace every order where price matches; but you need to delete trades from the orderbook too

Thanks. Can you tell me what  volume_int represents. Looking on other charts and third party order book data like bitcoincharts. The value that comes back is way too large to be the volume at that single price. I am missing something in understanding what is being returned


So  volume_int and total_volume_int seem to be bigger by a factor of 1E8 and price_int bigger by factor of 1E6 . Now that the numbers make sense, I can work on implementation


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinPappi on February 14, 2014, 03:39:55 AM
For some reason I get the following when i try to subscribe to channels. I get unknown command "op" :s

Code:
{"debug":{"data":{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"subscribe"},"data_raw":"{\"channel\": \"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\", \"op\": \"subscribe\"}","op":"client","uuid":"4a1f761a-4f50-4a21-91ff-e9037a2fa91f"},"message":"Unknown command","op":"remark","success":false}


Title: Re: MtGox API version 2: Unofficial Documentation
Post by: BitcoinPappi on February 14, 2014, 03:53:30 AM
For some reason I get the following when i try to subscribe to channels. I get unknown command "op" :s

Code:
{"debug":{"data":{"channel":"d5f06780-30a8-4a48-a2f8-7ed181b4a13f","op":"subscribe"},"data_raw":"{\"channel\": \"d5f06780-30a8-4a48-a2f8-7ed181b4a13f\", \"op\": \"subscribe\"}","op":"client","uuid":"4a1f761a-4f50-4a21-91ff-e9037a2fa91f"},"message":"Unknown command","op":"remark","success":false}

So it appears that you can only unsubscribe from channels when using the wss://websocket.mtgox.com:443/mtgox endpoint. Just unsubscribe from the channels your not interested in vs choosing which one to subscribe to since it will give the exception above.