Bitcoin Forum
May 17, 2024, 11:41:46 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Easy way to get detailed MTGOX trading history  (Read 1102 times)
pasky (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 0


View Profile
January 10, 2014, 03:03:25 PM
 #1

Hi! I wanted to get MtGox trading history for backtesting of few simple trading strategies, but found that there is no easy way to do it - just some very old methods that don't work with the current API, a crazy way using Google's BigQuery and storing the data to SQLite, etc. There is a BitcoinCharts export, but it does not include information on whether the trade was bid/ask, making the data useless for backtesting especially in periods of big ask/bid spreads.

It turns out that if I just wrote a script right away instead of looking for an existing solution, my life would be easier. :-)

Code:
#!/usr/bin/perl

use warnings;
use strict;
use JSON;

my $since = 1354767190000000; # XXX: Fill your own; use e.g. `date -d '2013-01-01' +%s`

while (1) {
        my $json = decode_json `curl http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=$since`;
        for my $i (@{$json->{data}}) {
                print join(',', $i->{date}, $i->{price}, $i->{amount}, $i->{trade_type})."\n";
                $since = $i->{tid} + 1;
                print STDERR "$since\n";
        }
}

This Perl script produces a simple CSV file with timestamp, price, amount and type of trade. Downloading a year's worth of data takes a few hours. As it is downloading the data, it will print an updated "since" timestamp value for resuming the download later.

Hope that's useful for someone.
Bitsky
Hero Member
*****
Offline Offline

Activity: 576
Merit: 514


View Profile
January 10, 2014, 08:43:06 PM
 #2

Apart from the chance that mtgox will probably ban your IP for abusing the API, there are other methods too.

Use BigQuery: https://bitcointalk.org/index.php?topic=218980.0
Or download a mirror: http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/


Bounty: Earn up to 68.7 BTC
Like my post? Feel free to drop a tip to 1BitskyZbfR4irjyXDaGAM2wYKQknwX36Y
pasky (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 0


View Profile
February 07, 2014, 02:04:10 PM
 #3

Thanks for your reply! I don't see why would this be considered abusing the API. What'd be the ?since= functionality good for otherwise?

Re BigQuery, I didn't like that way since it seems to require you to get a Google account and overally seems real complex.

Re the mirror, it suffers the major problem I was trying to solve, specifically not containing information about direction of the trade, which makes it mostly useless for trading backtesting.
Bitsky
Hero Member
*****
Offline Offline

Activity: 576
Merit: 514


View Profile
February 07, 2014, 08:04:44 PM
 #4

I vaguely remember that I read about a request limit somewhere when using the api and your script is looping without any delays.
Btw, if you use LWP::Simple you'll be independant of curl.

Bounty: Earn up to 68.7 BTC
Like my post? Feel free to drop a tip to 1BitskyZbfR4irjyXDaGAM2wYKQknwX36Y
pasky (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 0


View Profile
February 08, 2014, 12:55:58 AM
 #5

I vaguely remember that I read about a request limit somewhere when using the api and your script is looping without any delays.

I considered adding a delay, but the HTTP reply is actually delayed by a second or two, so I assume there is a delay that's built-in. But I agree having another sleep there may be friendlier to MtGox.

Btw, if you use LWP::Simple you'll be independant of curl.

I agree, that was just a quick hack. Smiley
InCoinsITrust
Member
**
Offline Offline

Activity: 112
Merit: 10


View Profile
February 09, 2014, 10:33:52 AM
 #6

Thank you all for answers, interesting and usefull reading.
coinglomerate
Newbie
*
Offline Offline

Activity: 28
Merit: 0


View Profile
February 09, 2014, 11:25:58 AM
 #7

really
Sonny
Hero Member
*****
Offline Offline

Activity: 868
Merit: 1000


View Profile
February 09, 2014, 08:03:36 PM
 #8

really

huh  Huh
What do you mean?
pasky (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 0


View Profile
January 25, 2015, 10:03:22 AM
 #9

N.B. this data seems to have gone completely off the internet. It's still sitting on my disk, so I decided to check if it might be useful for anyone at https://cryptothrift.com/auctions/download-misc/full-mtgox-trading-history-w-saletype-4backtesting/
okthen
Hero Member
*****
Offline Offline

Activity: 490
Merit: 500


View Profile
January 25, 2015, 10:09:13 AM
 #10

Yeah this doesn't work anymore.
I find it kind of upsetting that they keep withdrawing info, it all should be accessible and crystal clear.
Let's see how they handle it this year.
Yuki1988
Hero Member
*****
Offline Offline

Activity: 614
Merit: 500



View Profile
January 25, 2015, 11:53:51 AM
 #11

Yeah this doesn't work anymore.
I find it kind of upsetting that they keep withdrawing info, it all should be accessible and crystal clear.
Let's see how they handle it this year.

What are you talking about? The thread was made a year ago, and a month later mtgox went belly up in Feb 2014.
Of course their API is not working now lol.

findftp
Legendary
*
Offline Offline

Activity: 1022
Merit: 1006

Delusional crypto obsessionist


View Profile
January 25, 2015, 12:15:02 PM
 #12

If you want to do some technical analysis on old MtGox data you could go to bitcoinwisdom. They still provide the data.
I still do tech analysis on it and it brings me profit.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!