Print Page - Easy way to get detailed MTGOX trading history

Title: Easy way to get detailed MTGOX trading history
Post by: pasky on January 10, 2014, 03:03:25 PM

Hi! I wanted to get MtGox trading history for backtesting of few simple trading strategies, but found that there is no easy way to do it - just some very old methods that don't work with the current API, a crazy way using Google's BigQuery and storing the data to SQLite, etc. There is a BitcoinCharts export, but it does not include information on whether the trade was bid/ask, making the data useless for backtesting especially in periods of big ask/bid spreads.

It turns out that if I just wrote a script right away instead of looking for an existing solution, my life would be easier. :-)

Code:

#!/usr/bin/perl

use warnings;
use strict;
use JSON;

my $since = 1354767190000000; # XXX: Fill your own; use e.g. `date -d '2013-01-01' +%s`

while (1) {
        my $json = decode_json `curl http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=$since`;
        for my $i (@{$json->{data}}) {
                print join(',', $i->{date}, $i->{price}, $i->{amount}, $i->{trade_type})."\n";
                $since = $i->{tid} + 1;
                print STDERR "$since\n";
        }
}

This Perl script produces a simple CSV file with timestamp, price, amount and type of trade. Downloading a year's worth of data takes a few hours. As it is downloading the data, it will print an updated "since" timestamp value for resuming the download later.

Hope that's useful for someone.

Title: Re: Easy way to get detailed MTGOX trading history
Post by: Bitsky on January 10, 2014, 08:43:06 PM

Apart from the chance that mtgox will probably ban your IP for abusing the API, there are other methods too.

Use BigQuery: https://bitcointalk.org/index.php?topic=218980.0
Or download a mirror: http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/

Title: Re: Easy way to get detailed MTGOX trading history
Post by: pasky on February 07, 2014, 02:04:10 PM

Thanks for your reply! I don't see why would this be considered abusing the API. What'd be the ?since= functionality good for otherwise?

Re BigQuery, I didn't like that way since it seems to require you to get a Google account and overally seems real complex.

Re the mirror, it suffers the major problem I was trying to solve, specifically not containing information about direction of the trade, which makes it mostly useless for trading backtesting.

Title: Re: Easy way to get detailed MTGOX trading history
Post by: Bitsky on February 07, 2014, 08:04:44 PM

I vaguely remember that I read about a request limit somewhere when using the api and your script is looping without any delays.
Btw, if you use LWP::Simple you'll be independant of curl.

Title: Re: Easy way to get detailed MTGOX trading history
Post by: pasky on February 08, 2014, 12:55:58 AM

Quote from: Bitsky on February 07, 2014, 08:04:44 PM

I vaguely remember that I read about a request limit somewhere when using the api and your script is looping without any delays.

I considered adding a delay, but the HTTP reply is actually delayed by a second or two, so I assume there is a delay that's built-in. But I agree having another sleep there may be friendlier to MtGox.

Quote from: Bitsky on February 07, 2014, 08:04:44 PM

Btw, if you use LWP::Simple you'll be independant of curl.

I agree, that was just a quick hack. :)

Title: Re: Easy way to get detailed MTGOX trading history
Post by: InCoinsITrust on February 09, 2014, 10:33:52 AM

Thank you all for answers, interesting and usefull reading.

Title: Re: Easy way to get detailed MTGOX trading history
Post by: coinglomerate on February 09, 2014, 11:25:58 AM

really

Title: Re: Easy way to get detailed MTGOX trading history
Post by: Sonny on February 09, 2014, 08:03:36 PM

Quote from: coinglomerate on February 09, 2014, 11:25:58 AM

really

huh ???
What do you mean?

Title: Re: Easy way to get detailed MTGOX trading history
Post by: pasky on January 25, 2015, 10:03:22 AM

N.B. this data seems to have gone completely off the internet. It's still sitting on my disk, so I decided to check if it might be useful for anyone at https://cryptothrift.com/auctions/download-misc/full-mtgox-trading-history-w-saletype-4backtesting/

Title: Re: Easy way to get detailed MTGOX trading history
Post by: okthen on January 25, 2015, 10:09:13 AM

Yeah this doesn't work anymore.
I find it kind of upsetting that they keep withdrawing info, it all should be accessible and crystal clear.
Let's see how they handle it this year.

Title: Re: Easy way to get detailed MTGOX trading history
Post by: Yuki1988 on January 25, 2015, 11:53:51 AM

Quote from: okthen on January 25, 2015, 10:09:13 AM

Yeah this doesn't work anymore.
I find it kind of upsetting that they keep withdrawing info, it all should be accessible and crystal clear.
Let's see how they handle it this year.

What are you talking about? The thread was made a year ago, and a month later mtgox went belly up in Feb 2014.
Of course their API is not working now lol.

Title: Re: Easy way to get detailed MTGOX trading history
Post by: findftp on January 25, 2015, 12:15:02 PM

If you want to do some technical analysis on old MtGox data you could go to bitcoinwisdom. They still provide the data.
I still do tech analysis on it and it brings me profit.

Bitcoin Forum

Other => Beginners & Help => Topic started by: pasky on January 10, 2014, 03:03:25 PM