Bitcoin Forum
October 23, 2017, 05:23:19 PM *
News: Latest stable version of Bitcoin Core: 0.15.0.1  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: MtGox trades history download R script  (Read 5004 times)
MusX
Full Member
***
Offline Offline

Activity: 175


View Profile
September 03, 2013, 12:42:28 AM
 #1

There is some issues with fetching all the history from MtGox api.
I've prepared a script which joins trades history from bitcoincharts csv and mtgox recent trades to full set.
Output format supported are csv and sqlite db, configurable.
CSV separator and decimal mark configurable.
Support other currencies.
SSL support.
Open source.

environment: http://www.r-project.org/ (http://www.rstudio.com/)
script (v1.3): http://pastebin.com/DWhZfGTy

how to use:
1. open R
2. open script
3. if using first time after installation run commented "install.packages" function call, else skip this function call
Actually it is a bit more complicated on some platforms
Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.
4. adjust params if needed: mtgox currency, output types (csv,sqlite), csv config
5. run the full script
6. wait until it will download and dump to files
7. look for .csv and/or .db files in R working directory (default on windows /Documents)

timing:
 - downloading mtgoxUSD history from bitcoincharts: ~10min
 - downloading mtgoxUSD recent trades from mtgox: ~5sec
 - transform and merge: ~1min
 - save to csv: ~1.5min (~296mb)
 - save to sqlite: ~1min (~225mb, not compressed)

please report any bugs you will find

1508779399
Hero Member
*
Offline Offline

Posts: 1508779399

View Profile Personal Message (Offline)

Ignore
1508779399
Reply with quote  #2

1508779399
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1508779399
Hero Member
*
Offline Offline

Posts: 1508779399

View Profile Personal Message (Offline)

Ignore
1508779399
Reply with quote  #2

1508779399
Report to moderator
1508779399
Hero Member
*
Offline Offline

Posts: 1508779399

View Profile Personal Message (Offline)

Ignore
1508779399
Reply with quote  #2

1508779399
Report to moderator
1508779399
Hero Member
*
Offline Offline

Posts: 1508779399

View Profile Personal Message (Offline)

Ignore
1508779399
Reply with quote  #2

1508779399
Report to moderator
MusX
Full Member
***
Offline Offline

Activity: 175


View Profile
September 06, 2013, 06:54:03 PM
 #2

anybody used it? any experience, feedback? Smiley

Adriano
Staff
Legendary
*
Offline Offline

Activity: 1058



View Profile
September 07, 2013, 01:07:53 AM
 #3

anybody used it? any experience, feedback? Smiley

I saved your topic in order to test it when I have the time, but it wont happened before next monday at least... If no one else comment then I'll post my thoughts when I use it.


Thanks


Adriano

arvin8
Newbie
*
Offline Offline

Activity: 9


View Profile
September 12, 2013, 06:39:24 PM
 #4

http://pastebin.com/DWhZfGTy
this link seems to be offline?
MusX
Full Member
***
Offline Offline

Activity: 175


View Profile
September 25, 2013, 06:24:16 PM
 #5

http://pastebin.com/DWhZfGTy
this link seems to be offline?
now it's working fine, was the whole pastebin offline or only this one link?

chenchunyu88
Full Member
***
Offline Offline

Activity: 127



View Profile WWW
October 25, 2013, 08:36:56 PM
 #6

This is super cool.

razorfishsl
Sr. Member
****
Offline Offline

Activity: 406


View Profile WWW
October 28, 2013, 04:18:15 AM
 #7

Quote
3. if using first time after installation run commented "install.packages" function call, else skip this function call

Actually it is a bit more complicated on some platforms

Better you should go to "package installer" in 'R' and ensure the following are loaded and updated.

data.table
RCurl
RJSONIO
RSQLite

From CRAN(binaries)

Ensure that you also check the box for "lnstall dependancies"

Then load the script…. and off it goes.

High Quality USB Hubs for Bitcoin miners
https://bitcointalk.org/index.php?topic=560003
MusX
Full Member
***
Offline Offline

Activity: 175


View Profile
October 29, 2013, 09:31:50 AM
 #8

good to see any feedback Smiley
I put your comment in the instruction.

chenchunyu88
Full Member
***
Offline Offline

Activity: 127



View Profile WWW
October 29, 2013, 05:48:19 PM
 #9

I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.

MusX
Full Member
***
Offline Offline

Activity: 175


View Profile
October 30, 2013, 09:34:44 AM
 #10

I am wondering if you can add code to convert the date and time to GMT time. I am trying to do some analysis on BTC price and its correlation and some more analysis with stock market index and some other index. Once we can figure out those stuffs, maybe we can publish a R package for BTC analysis.
The dates field in data from script should be in GMT already. To have the same TZ during the processing I'm using
Code:
Sys.setenv(TZ = 'GMT')
adjusted globally for whole project in RStudio.
It would be great to have a R package related to bitcoin, I have already a lot of code for that (which include mtgox/bitstamp private api calls with authorization (also v2 for mtgox) and it is possible to add new markets).
The blocker currently for me is a lack of documentation for that code. If anybody with experience of publishing R package would be interested to create one I would be happy to contribute. I would need to simplify some codes, drop some dependencies (example. RpostgreSQL), but I wanted to create package from my codes so it is on my todo list someday.

trepper
Member
**
Offline Offline

Activity: 86


View Profile
November 22, 2013, 04:50:35 PM
 #11

Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". Undecided


Donate and something fine could happen to you: 15jBKffDHUfARkgHWnDyaebzVBE5G4kcPg
MusX
Full Member
***
Offline Offline

Activity: 175


View Profile
November 23, 2013, 08:36:52 AM
 #12

Your cool script works great on my desktop (win7x64+R3.0.2) and I find it very useful.

Thank you very much for it!!!!

I have just a Rnewbie question: what is the origin of the $date timeline?
In fact when I set the parameter "origin" to "1970-01-01" and apply as.date(mtgoxtrade$date) in order to have a more readable time column , I get dates like "4870-07-23". Undecided


simple way: pass as.numeric(substr(date,1,10)) to as.POSIXct. You will lose 0.1s precision if available
most correct way: (as.numeric(date)/x) - x should be kind of 1000, 1000000, to have a result with 10 numbers on left left from decimal sign. Than it will keep the microseconds after conversion to posix.

check this one: https://bitcointalk.org/index.php?topic=343504.0

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!