Bitcoin Forum

Economy => Trading Discussion => Topic started by: MusX on September 03, 2013, 12:42:28 AM



Title: MtGox trades history download R script
Post by: MusX on September 03, 2013, 12:42:28 AM
There is some issues with fetching all the history from MtGox api.
I've prepared a script which joins trades history from bitcoincharts csv and mtgox recent trades to full set.
Output format supported are csv and sqlite db, configurable.
CSV separator and decimal mark configurable.
Support other currencies.
SSL support.
Open source.

environment: http://www.r-project.org/ (http://www.rstudio.com/)
script (v1.3): http://pastebin.com/DWhZfGTy

how to use:
1. open R
2. open script
3. if using first time after installation run commented "install.packages" function call, else skip this function call
Actually it is a bit more complicated on some platforms
Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.
4. adjust params if needed: mtgox currency, output types (csv,sqlite), csv config
5. run the full script
6. wait until it will download and dump to files
7. look for .csv and/or .db files in R working directory (default on windows /Documents)

timing:
 - downloading mtgoxUSD history from bitcoincharts: ~10min
 - downloading mtgoxUSD recent trades from mtgox: ~5sec
 - transform and merge: ~1min
 - save to csv: ~1.5min (~296mb)
 - save to sqlite: ~1min (~225mb, not compressed)

please report any bugs you will find


Title: Re: MtGox trades history download R script
Post by: MusX on September 06, 2013, 06:54:03 PM
anybody used it? any experience, feedback? :)


Title: Re: MtGox trades history download R script
Post by: Adriano on September 07, 2013, 01:07:53 AM
anybody used it? any experience, feedback? :)

I saved your topic in order to test it when I have the time, but it wont happened before next monday at least... If no one else comment then I'll post my thoughts when I use it.


Thanks


Adriano


Title: Re: MtGox trades history download R script
Post by: arvin8 on September 12, 2013, 06:39:24 PM
http://pastebin.com/DWhZfGTy
this link seems to be offline?


Title: Re: MtGox trades history download R script
Post by: MusX on September 25, 2013, 06:24:16 PM
http://pastebin.com/DWhZfGTy
this link seems to be offline?
now it's working fine, was the whole pastebin offline or only this one link?


Title: Re: MtGox trades history download R script
Post by: chenchunyu88 on October 25, 2013, 08:36:56 PM
This is super cool.


Title: Re: MtGox trades history download R script
Post by: razorfishsl on October 28, 2013, 04:18:15 AM
Quote
3. if using first time after installation run commented "install.packages" function call, else skip this function call

Actually it is a bit more complicated on some platforms

Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.

data.table
RCurl
RJSONIO
RSQLite

From CRAN(binaries)

Ensure that you also check the box for "lnstall dependancies"

Then load the script…. and off it goes.


Title: Re: MtGox trades history download R script
Post by: MusX on October 29, 2013, 09:31:50 AM
good to see any feedback :)
I put your comment in the instruction.


Title: Re: MtGox trades history download R script
Post by: chenchunyu88 on October 29, 2013, 05:48:19 PM
I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.


Title: Re: MtGox trades history download R script
Post by: MusX on October 30, 2013, 09:34:44 AM
I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.
The dates field in data from script should be in GMT already. To have the same TZ during the processing I'm using
Code:
Sys.setenv(TZ = 'GMT')
adjusted globally for whole project in RStudio.
It would be great to have a R package related to bitcoin, I have already a lot of code for that (which include mtgox/bitstamp private api calls with authorization (also v2 for mtgox) and it is possible to add new markets).
The blocker currently for me is a lack of documentation for that code. If anybody with experience of publishing R package would be interested to create one I would be happy to contribute. I would need to simplify some codes, drop some dependencies (example. RpostgreSQL), but I wanted to create package from my codes so it is on my todo list someday.


Title: Re: MtGox trades history download R script
Post by: trepper on November 22, 2013, 04:50:35 PM
Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". :-\



Title: Re: MtGox trades history download R script
Post by: MusX on November 23, 2013, 08:36:52 AM
Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". :-\


simple way: pass as.numeric(substr(date,1,10)) to as.POSIXct. You will lose 0.1s precision if available
most correct way: (as.numeric(date)/x) - x should be kind of 1000, 1000000, to have a result with 10 numbers on left left from decimal sign. Than it will keep the microseconds after conversion to posix.

check this one: https://bitcointalk.org/index.php?topic=343504.0