MtGox trades history download R script

[MusX]

There is some issues with fetching all the history from MtGox api.
I've prepared a script which joins trades history from bitcoincharts csv and mtgox recent trades to full set.
Output format supported are csv and sqlite db, configurable.
CSV separator and decimal mark configurable.
Support other currencies.
SSL support.
Open source.

environment: http://www.r-project.org/ (http://www.rstudio.com/)
script (v1.3): http://pastebin.com/DWhZfGTy

how to use:
1. open R
2. open script
3. if using first time after installation run commented "install.packages" function call, else skip this function call

Actually it is a bit more complicated on some platforms
Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.

4. adjust params if needed: mtgox currency, output types (csv,sqlite), csv config
5. run the full script
6. wait until it will download and dump to files
7. look for .csv and/or .db files in R working directory (default on windows /Documents)

timing:
 - downloading mtgoxUSD history from bitcoincharts: ~10min
 - downloading mtgoxUSD recent trades from mtgox: ~5sec
 - transform and merge: ~1min
 - save to csv: ~1.5min (~296mb)
 - save to sqlite: ~1min (~225mb, not compressed)

please report any bugs you will find

[1714210488]

1714210488

[1714210488]

1714210488

[1714210488]

1714210488

[MusX]

anybody used it? any experience, feedback?

[Adriano]

anybody used it? any experience, feedback?

I saved your topic in order to test it when I have the time, but it wont happened before next monday at least... If no one else comment then I'll post my thoughts when I use it.


Thanks


Adriano

[arvin8]

http://pastebin.com/DWhZfGTy
this link seems to be offline?

[MusX]

http://pastebin.com/DWhZfGTy
this link seems to be offline?

now it's working fine, was the whole pastebin offline or only this one link?

[chenchunyu88]

This is super cool.

[razorfishsl]

3. if using first time after installation run commented "install.packages" function call, else skip this function call

Actually it is a bit more complicated on some platforms

Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.

data.table
RCurl
RJSONIO
RSQLite

From CRAN(binaries)

Ensure that you also check the box for "lnstall dependancies"

Then load the script…. and off it goes.

[MusX]

good to see any feedback
I put your comment in the instruction.

[chenchunyu88]

I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.

[MusX]

I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.

The dates field in data from script should be in GMT already. To have the same TZ during the processing I'm using

Code:

Sys.setenv(TZ = 'GMT')

adjusted globally for whole project in RStudio.
It would be great to have a R package related to bitcoin, I have already a lot of code for that (which include mtgox/bitstamp private api calls with authorization (also v2 for mtgox) and it is possible to add new markets).
The blocker currently for me is a lack of documentation for that code. If anybody with experience of publishing R package would be interested to create one I would be happy to contribute. I would need to simplify some codes, drop some dependencies (example. RpostgreSQL), but I wanted to create package from my codes so it is on my todo list someday.

[trepper]

Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23".

[MusX]

Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23".

simple way: pass as.numeric(substr(date,1,10)) to as.POSIXct. You will lose 0.1s precision if available
most correct way: (as.numeric(date)/x) - x should be kind of 1000, 1000000, to have a result with 10 numbers on left left from decimal sign. Than it will keep the microseconds after conversion to posix.

check this one: https://bitcointalk.org/index.php?topic=343504.0