API insanity: HmacSHA512 variations on MtGox vs Bitfloor vs CryptoXChange

[eldentyrell]

TL;DR: although all three exchanges technically "use HmacSHA512 authentication", there are important differences and you can't use exactly the same code for all three of them.

The APIs of MtGox, Bitfloor, and CryptoXChange all authenticate requests by having the caller include the HmacSHA512-signature of the HTTP POST message in an extra header.  However, there are some differences I want to make people aware of:

MtGox and Bitfloor use keys that are one block (64 bytes) long when Base64-decoded, whereas CryptoXChange uses funny 43-byte keys.  But see below regarding Bitfloor.

All three exchanges supply your key to you in Base64-encoded form.  However -- and this is the really weird part:

- Bitfloor wants you to use the literal ASCII bytes of the Base64-encoded key as your signing key, giving an 88-byte key.

- CryptoXChange wants you to Base64-decode the key into 43 bytes and use those bytes.

- MtGox will let you do either of the above if you include your password in the POST as &pass=XXX.  In fact, you can include a totally bogus Rest-Sign signature.  Go ahead, try it!

NONE OF THIS IS MENTIONED IN ANY OF THEIR API DOCUMENTS.  I have confirmed all of the above (both what works and what doesn't on each exchange) by actual tests against the three APIs.

In my opinion, what Bitfloor is doing is broken, what CryptoXChange is doing is correct, and MtGox is being nice by supporting calls made with broken code.

Here is some Java code that illustrates the difference in precise terms:

    public static String computeSignature(String algorithm, String baseString, byte[] key)
        throws GeneralSecurityException, UnsupportedEncodingException {
        SecretKey secretKey = null;
        secretKey = new SecretKeySpec(key, algorithm);
        Mac mac = Mac.getInstance(algorithm);
        mac.init(secretKey);
        mac.update(baseString.getBytes());
        return new String(Base64.encodeBase64(mac.doFinal())).trim();
   }

So for CryptoXChange you do

    computeSignature("HmacSHA512", postBody, Base64.decode(secretkey))

for Bitfloor you do

    computeSignature("HmacSHA512", postBody, secretKey.getBytes())

And for MtGox you can do either as long as your password is in the POST body.

[cryptoxchange]

Hi Elden,

Much appreciated, this will be very helpful for people and Arbitrage as we will all have BitInstant as of the end of this week.

Thank you very much for letting people know as most traders do want to trade between these exchanges.

Regards

Crypto X Change

[eldentyrell]

this will be very helpful for people and Arbitrage as we will all have BitInstant as of the end of this week.

BitInstant might get rid of the arbitrage opportunities that sit open for hours (or days) on end, but I don't think it's going to change the ones that last for a half hour.  BitInstant is fast, but not that fast.

If you want the exchanges to become more integrated (which is to the advantage of the smaller exchanges) the most important thing you can do is make sure that it is possible to withdraw BTC via the API.  This will make arbitrageurs more willing to leave large balances at your exchange (since they can be moved elsewhere by the tradebot in the middle of the night if necessary) which will increase liquidity.

MtGox and CampBX are the only two exchanges that allow withdrawal-via-API.  Tradehill claims to offer it, but it doesn't actually work (exchanged a few emails with their tech support and we came to that conclusion).

It's not such a big deal for me personally since I'm mainly a miner and my presence on the exchanges is mostly just a way of making sure I get the best prices for my mined coins.  But now that I've implemented all the APIs I might as well run a bit of arbitrage.

[cryptoxchange]

Hi,

Yes we haven't added withdraw VIA API just yet as safety for our clients, we can definately turn it on, we just want to make sure first that there are no disasters, we have the withdraw PinCode etc, which will need to be added to each persons bot etc for this feature which may help the security additionally.

Thanks!

Regards

Crypto X Change

[notme]

Mtgox's behavior here is unacceptable.  It completely nullifies the Yubikey.  Anyone can withdraw my funds with just my un/pw.  Fix this now, force people to update their code instead of making pointless all the effort you have put into making a secure API.  I've already updated the ruby bindings so I know it's not that hard to switch to the new authentication mechanism.  People will whine, but you can't afford another security fiasco.

[shtylman]

Why do you think what bitfloor is doing is broken? I would be happy to offer alternative ways to send the bytes however to me it seems very clear that sending the ASCII string we provide to you from the site is very straight forward. I will try to make the documentation more clear on this matter in either case.

[eldentyrell]

Why do you think what bitfloor is doing is broken? I would be happy to offer alternative ways to send the bytes however to me it seems very clear that sending the ASCII string we provide to you from the site is very straight forward.

Shtylman, I think you have mixed up what you call the "api-key" and the "sec-key" (it seems that each exchange makes up their own words for "username" and "password" -- argh).

What you call the "api-key" is like a username.  You're right, it should be sent as ASCII bytes in the "bitfloor-key" header.

However, your handling of "sec-key" is broken.  You provide it to users as a Base64 string; what they SHOULD do is Base64-decode it into 64 bytes and use those to sign the request.  Instead you are forcing your users to use the 88 ASCII bytes of the sec-key to do the signing.  This is silly!  Why bother with Base64-encoding if the user isn't going to ever Base64-decode it?  You might as well use some random string of characters.

[shtylman]

You are correct in that the user should base64-decode the string and use those bytes to sign (ending up with 64 bytes they can use to sign). This was an oversight on my part when doing the initial signing handling. I will make those changes.

The reason we base64 encode the bytes in the first place is because we need to provide the user with 64 bytes of random data. In order to do this, we generate those random bytes and then need to encode them for display on the web. Otherwise the characters would be gibberish. If we used just a random string of length 64 that would not be 64 random bytes (and the search space would be much smaller). Likewise, just a random string of 88 characters would still not be 88 random bytes (if done in ascii for easily display).

I am not sure why CryptoXChange is only using 44 bytes as that is not enough for a sha512. And the MtGox behavior with bypassing the sign field altogether seems completely wrong and misses the point of not using your password in the requests (I have not confirmed your statement on this). To say that MtGox will let you do either if you specify the password seems incorrect. It seems like that just bypasses the signing altogether.

[shtylman]

I have pushed a new version of the REST gateway to both the testnet as well as the production environment. This new version supports both variants (using the ascii string directly, and using the 64 decoded bytes).

Users of the api should switch over to the decoded variant in the coming weeks. I will be deprecating the previous usage of just the raw ascii characters within a month (after sending out an email to all users and giving them time to migrate).

Please let me know if you experience any problems or issues with the new scheme and thank you for the issue report.