Title: MtGox trades history download R script Post by: MusX on September 03, 2013, 12:42:28 AM There is some issues with fetching all the history from MtGox api.
I've prepared a script which joins trades history from bitcoincharts csv and mtgox recent trades to full set. Output format supported are csv and sqlite db, configurable. CSV separator and decimal mark configurable. Support other currencies. SSL support. Open source. environment: http://www.r-project.org/ (http://www.rstudio.com/) script (v1.3): http://pastebin.com/DWhZfGTy how to use: 1. open R 2. open script 3. if using first time after installation run commented "install.packages" function call, else skip this function call Actually it is a bit more complicated on some platforms 4. adjust params if needed: mtgox currency, output types (csv,sqlite), csv configBetter you should go to "package installer" in 'R' and ensure the following are loaded and updated. 5. run the full script 6. wait until it will download and dump to files 7. look for .csv and/or .db files in R working directory (default on windows /Documents) timing: - downloading mtgoxUSD history from bitcoincharts: ~10min - downloading mtgoxUSD recent trades from mtgox: ~5sec - transform and merge: ~1min - save to csv: ~1.5min (~296mb) - save to sqlite: ~1min (~225mb, not compressed) please report any bugs you will find Title: Re: MtGox trades history download R script Post by: MusX on September 06, 2013, 06:54:03 PM anybody used it? any experience, feedback? :)
Title: Re: MtGox trades history download R script Post by: Adriano on September 07, 2013, 01:07:53 AM anybody used it? any experience, feedback? :) I saved your topic in order to test it when I have the time, but it wont happened before next monday at least... If no one else comment then I'll post my thoughts when I use it. Thanks Adriano Title: Re: MtGox trades history download R script Post by: arvin8 on September 12, 2013, 06:39:24 PM http://pastebin.com/DWhZfGTy
this link seems to be offline? Title: Re: MtGox trades history download R script Post by: MusX on September 25, 2013, 06:24:16 PM http://pastebin.com/DWhZfGTy now it's working fine, was the whole pastebin offline or only this one link?this link seems to be offline? Title: Re: MtGox trades history download R script Post by: chenchunyu88 on October 25, 2013, 08:36:56 PM This is super cool.
Title: Re: MtGox trades history download R script Post by: razorfishsl on October 28, 2013, 04:18:15 AM Quote 3. if using first time after installation run commented "install.packages" function call, else skip this function call Actually it is a bit more complicated on some platforms Better you should go to "package installer" in 'R' and ensure the following are loaded and updated. data.table RCurl RJSONIO RSQLite From CRAN(binaries) Ensure that you also check the box for "lnstall dependancies" Then load the script…. and off it goes. Title: Re: MtGox trades history download R script Post by: MusX on October 29, 2013, 09:31:50 AM good to see any feedback :)
I put your comment in the instruction. Title: Re: MtGox trades history download R script Post by: chenchunyu88 on October 29, 2013, 05:48:19 PM I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.
Title: Re: MtGox trades history download R script Post by: MusX on October 30, 2013, 09:34:44 AM I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis. The dates field in data from script should be in GMT already. To have the same TZ during the processing I'm using Code: Sys.setenv(TZ = 'GMT') It would be great to have a R package related to bitcoin, I have already a lot of code for that (which include mtgox/bitstamp private api calls with authorization (also v2 for mtgox) and it is possible to add new markets). The blocker currently for me is a lack of documentation for that code. If anybody with experience of publishing R package would be interested to create one I would be happy to contribute. I would need to simplify some codes, drop some dependencies (example. RpostgreSQL), but I wanted to create package from my codes so it is on my todo list someday. Title: Re: MtGox trades history download R script Post by: trepper on November 22, 2013, 04:50:35 PM Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.
Thank you very much for it!!!! I have just a Rnewbie question: what is the origin of the $date timeline? In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". :-\ Title: Re: MtGox trades history download R script Post by: MusX on November 23, 2013, 08:36:52 AM Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful. simple way: pass as.numeric(substr(date,1,10)) to as.POSIXct. You will lose 0.1s precision if availableThank you very much for it!!!! I have just a Rnewbie question: what is the origin of the $date timeline? In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". :-\ most correct way: (as.numeric(date)/x) - x should be kind of 1000, 1000000, to have a result with 10 numbers on left left from decimal sign. Than it will keep the microseconds after conversion to posix. check this one: https://bitcointalk.org/index.php?topic=343504.0 |