New method to pull MtGox trade data

[btc_lurker]

Just pay attention when using bigquery.cloud.com. It is very easy to get charged.

[1714149487]

1714149487

[1714149487]

1714149487

[1714149487]

1714149487

[1714149487]

1714149487

[1714149487]

1714149487

[1714149487]

1714149487

[Loozik]

Hello dexX7,

This is a fantastic guide

I needed the data sorted in ascending order (by time: from earliest to most recent date) so I changed DESC to ASC and <2013-05-19 to > 2010-07-16 and it works.

Is the number of 16k rows imposed by Google or could it be somehow increased by the user to e.g. 160k? I am not complaining, just asking

[nitrous]

Hello dexX7,

This is a fantastic guide

I needed the data sorted in ascending order (by time: from earliest to most recent date) so I changed DESC to ASC and <2013-05-19 to > 2010-07-16 and it works.

Is the number of 16k rows imposed by Google or could it be somehow increased by the user to e.g. 160k? I am not complaining, just asking

The issue is that Google won't let you download a CSV copy of the results if there are more than 16k rows. It is possible to download more, but you have to do it programatically. My tool can currently download a complete copy of the database (https://bitcointalk.org/index.php?topic=221055.0), but I don't think it's ready for your needs yet. I would be interested in your opinion on the poll, though. I need to know whether you would be fine for it to download the entire database to a sqlite3 dump (about 460mb) before then processing it to your specifications. This would also enable you to then quickly create a csv copy of any query without having to download any more data.

[Loozik]

The issue is that Google won't let you download a CSV copy of the results if there are more than 16k rows.

What a misfortune  

It is possible to download more, but you have to do it programatically. My tool can currently download a complete copy of the database (https://bitcointalk.org/index.php?topic=221055.0), but I don't think it's ready for your needs yet. I would be interested in your opinion on the poll, though. I need to know whether you would be fine for it to download the entire database to a sqlite3 dump (about 460mb) before then processing it to your specifications. This would also enable you to then quickly create a csv copy of any query without having to download any more data.

I am just a ''paste and copy'' guy. I write movie dialogues and song lyrics for a living (not in English; my English is terrible). My hobby is trading forex and futures. I will of course leave my opinion in your poll, but not until a few days from now when I get an understanding of what ''sqlite3 dump'' is. Yes, I am that dumb when it comes to computers  

What I normally do if I want to get data for my charting application is I go here http://iqfeed.net/ and pay on a monthly basis for data feeds I want, from exchanges I want. And get the historical and real-time data in format customized to my platform and streamed to my platform, without the need to worry about technicalities.

Maybe in future MtGox could speak to some data vendors and avail its feed to these data vendors. These data vendors will then redistribute (using their data centres) the data to many users for a fee. Here is http://www.iqfeed.net/index.cfm?displayaction=data&section=software#products a list of charting packages that MtGox can reach in a minute without the need to invest in infrastructure too much.

[nitrous]

I am just a ''paste and copy'' guy. I write movie dialogues and song lyrics for a living (not in English; my English is terrible). My hobby is trading forex and futures. I will of course leave my opinion in your poll, but not until a few days from now when I get an understanding of what ''sqlite3 dump'' is. Yes, I am that dumb when it comes to computers  

Don't worry about that, sqlite3 is just a type of database, which I can hopefully embed into the app. I still can't guarantee that it'll be trivial to create a self-contained application, but if I can do, you should be able to just download and double click Basically, the dump would just be a local copy of the bigquery database behind the scenes. The app would first download it, and then process it to create any format you want and much more quickly (without downloading any more data, nor risking any processing charges from google - completely free!).

The only issue is that it will require a half gigabyte of overhead, if you don't mind that, you won't need to worry about the dump, it'll just sit there hidden [this is option 1 -- download a complete dump then process]. If, however, I don't use the sqlite3 dump in the app, then each time you want to download a copy of the data, it will have to download it all again (and depending on how many queries are needed, it could end up costing you money) [this is option 2 -- process during download].

[Loozik]

Okay. I voted for option 1. Disk space is cheap now-a-days.

[Loozik]

This is how beautifully BTCUSD can look with high quality data in a charting software I downloaded and merged only a tiny portion of the tick data from Google.

You can easily add indicators, e.g. geometrical ones, you can draw trendlines, you can apply algorithmic strategies and whatnot. On the right there is a panel for manual entering orders, without the need to go to the broker's / exchange's website.

Yeah, I would love to have this possibility with MtGox.

At the moment I need go to bitcoincharts. com, copy the BTCUSD image and then process the image in an image processing software to draw a single trendline  

[nitrous]

This is how beautifully BTCUSD can look with high quality data in a charting software I downloaded and merged only a tiny portion of the tick data from Google.

You can easily add indicators, e.g. geometrical ones, you can draw trendlines, you can apply algorithmic strategies and whatnot. On the right there is a panel for manual entering orders, without the need to go to the broker's / exchange's website.

Yeah, I would love to have this possibility with MtGox.

At the moment I need go to bitcoincharts. com, copy the BTCUSD image and then process the image in an image processing software to draw a single trendline  

Well I have my tool generating data according to your format now There's no integrity checking of the csv currently though. I may not be able to work on it very much for the next few days/weeks though as I've got some important exams coming up.

[Loozik]

I may not be able to work on it very much for the next few days/weeks though as I've got some important exams coming up.

It's okay. I can wait. Thank you very much for the work that you do 

[Loozik]

Guys, I have a couple of questions:

1. Is the minimum price tick size of BTCUSD at MtGox 0.00001?

2. Is Bid and Ask data from the dark pool https://bitcointalk.org/index.php?topic=5564.0;all reflected in the data?

3. Is traded volume from the dark pool reflected in the data?

4. What is MtGox time (Greenwich, Tokyo, other)?

[nitrous]

Guys, I have a couple of questions:

I don't know about the dark pool, but for 1 and 4:

1. Is the minimum price tick size of BTCUSD at MtGox 0.00001?

Yes, so a price of 12345678 for a USD order would be $123.46 (rounded). I've compiled a list of minimum tick sizes for each currency here: https://bitbucket.org/nitrous/mtgox-api#markdown-header-currencies - USD is listed as having 100,000 divisions, so the tick size is 1/100000 = 0.00001

4. What is MtGox time (Greenwich, Tokyo, other)?

The timezone used for the dates in the bigquery data is GMT

[Loozik]

Thank you.

[8bitPunk]

Loozik - I discovered that Big Query has an Excel connector which is handy for auto-updating spreadsheets: https://bigquery-connector.appspot.com/

You still need to understand the setup of the Big Query project to load the Mt-Gox dataset and the query example provided by dexX7.

Best of all it isn't limited to 16K rows like the csv option, rather my Excel spreadsheet stopped at 110K rows, so I added another worksheet and pulled the next 110K results into there.

[Loozik]

Loozik - I discovered that Big Query has an Excel connector which is handy for auto-updating spreadsheets: https://bigquery-connector.appspot.com/

You still need to understand the setup of the Big Query project to load the Mt-Gox dataset and the query example provided by dexX7.

Best of all it isn't limited to 16K rows like the csv option, rather my Excel spreadsheet stopped at 110K rows, so I added another worksheet and pulled the next 110K results into there.

Thanks 8bitPunk for the info. This Big Query thing is too difficult for me. I am a silly non-techie I will wait till a more elegant solution is availed.

I found a temporary remedy for my charting pains as far BTCUSD from MtGox goes https://www.tradingview.com/e/?symbol=MTGOX:BTCUSD# It isn't a perfect solution (I can't make use of custom indicators and strategies) but at least I can draw trendlines.

[buyer]

so this appears to mirror large datasets.  is anyone mirroring mt gox last?  I need to built an effective ticker but I find mt gox's feed is wholly unreliable.  Any advice?

[bitranox]

I am thinking to provide the tradedata by another method to the users.

I fetched all the data for myself, so if a considerable amount of users wants to use that, I will provide the tradedata in a different way (not rely on mtgox updating the data, not rely on big query).

please check out following post : https://bitcointalk.org/index.php?topic=282154.msg3017272#msg3017272 and let me know.

yours sincerely

bitranox

[frito]

Does the data end in May 2013?

[nitrous]

Does the data end in May 2013?

Unfortunately yes. MtGox has not updated the BigQuery database in quite some time, however the data is complete up to May 2013.

[frito]

Does the data end in May 2013?

Unfortunately yes. MtGox has not updated the BigQuery database in quite some time, however the data is complete up to May 2013.

It's like MagicalTux has been on strike. Everybody's mean to him I guess