Discussion for MtGox trade data downloader

[nitrous]

It seems that some windows users have been experiencing trouble with the initial authentication, so I've overridden part of the google code to create a GUI method that should work cross-platform. Please let me know if there are still problems authenticating with Google on any platform, or if the instructions aren't clear enough.

In addition, it seems that you do not need to supply your own client_secrets.json. Apparently, it is up to me as the app developer to do this, so there is no longer any need to dig through Google's API Console as I have incorporated the codes into the program, you just need to complete the authorisation in a browser.

[nitrous]

Ok everyone, I've completed the tool!

You can download a binary for mac or windows here:
https://bitbucket.org/nitrous/bq/downloads

The source is still available here:
https://bitbucket.org/nitrous/bq
And you can build your own binary if you want.


The tool is ready to download from the bigquery database, and to export to a number of formats.

• CSV (price/volume) [ISO]
• CSV Candle [ISO]
• CSV Candle [Unix]
• Sqlite3 PhantomCircuit

The first option is the format Loozik needed, a csv file with headers date,time,price,volume. Date and time are in ISO 8601 format.

The next two generate candlestick data using a port of https://bitcointalk.org/index.php?topic=239815.0. The first one uses ISO date formats and the second, Unix timestamps.

The last should be compatible with PhantomCircuit's script http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/.


Feel free to download, try it out, and let me know if you find any bugs. Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).

I haven't gotten around to documenting the usage of this tool yet, but hopefully it's pretty self-explanatory.


Windows DLL requirements:

There are a number of DLL files which py2exe seems to require. I think those that are come preloaded with Windows, except for the MSVC runtime. Try running the app first without downloading any new stuff. If it doesn't work, first you should try installing the following:

http://www.microsoft.com/en-us/download/details.aspx?id=29

If you still can't run it, tell me and I'll try to see what the problem is.

[Loozik]

Ok everyone, I've completed the tool!

Fantastic.

Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).

Do you perhaps know when MtGox will start updating the database?

[nitrous]

Ok everyone, I've completed the tool!

Fantastic.

Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).

Do you perhaps know when MtGox will start updating the database?

Sorry, not yet. I've asked a couple times but I've yet to receive an official response.

[Loozik]

Ok everyone, I've completed the tool!

Fantastic.

Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).

Do you perhaps know when MtGox will start updating the database?

Sorry, not yet. I've asked a couple times but I've yet to receive an official response.

Many thanks. Please let us know when you know something 

[2weiX]

trying thiso out ASAP.
might have some feature requests, happy to donate.

[nitrous]

trying thiso out ASAP.
might have some feature requests, happy to donate.

Great Let me know and I'll see what I can do.

[Diabolicus]

It crashes when trying to export, no matter what settings I choose.
Tested on Windows 7 64bit and 32bit.
Log says:

Traceback (most recent call last):
  File "app.py", line 80, in __call__
  File "export.pyc", line 210, in begin_export
  File "formatters\Candles.pyc", line 181, in latest
ValueError: invalid literal for float(): 2010-07-22

or

Exception in thread Thread-1:
Traceback (most recent call last):
  File "threading.pyc", line 808, in __bootstrap_inner
  File "app.py", line 98, in thread_bootstrap
  File "app.py", line 52, in fatal
NameError: global name 'time' is not defined

Traceback (most recent call last):
  File "app.py", line 96, in thread_bootstrap
  File "mtgox.pyc", line 185, in run
  File "formatters\LoozikCandles.pyc", line 117, in open
IOError: Invalid CSV file to append - wrong number of fields

[nitrous]

It crashes when trying to export, no matter what settings I choose.
Tested on Windows 7 64bit and 32bit.
Log says:

Traceback (most recent call last):
  File "app.py", line 80, in __call__
  File "export.pyc", line 210, in begin_export
  File "formatters\Candles.pyc", line 181, in latest
ValueError: invalid literal for float(): 2010-07-22

or

Exception in thread Thread-1:
Traceback (most recent call last):
  File "threading.pyc", line 808, in __bootstrap_inner
  File "app.py", line 98, in thread_bootstrap
  File "app.py", line 52, in fatal
NameError: global name 'time' is not defined

Traceback (most recent call last):
  File "app.py", line 96, in thread_bootstrap
  File "mtgox.pyc", line 185, in run
  File "formatters\LoozikCandles.pyc", line 117, in open
IOError: Invalid CSV file to append - wrong number of fields

Hi Diabolicus,

Thanks, the 'time' error was a bug (though it only occurs after the program crashes). Your other errors appear to be you trying to update a CSV file which was created with a different format. There are five types of CSV file that can currently be produced:

CSV (price/volume) [ISO]
CSV Candle [ISO] - no volume
CSV Candle [ISO] - with volume
CSV Candle [Unix] - no volume
CSV Candle [Unix] - with volume

None of these are compatible with each other. It should work as long as you remember which options you used (I recommend saving your export settings with the Save button in the export window and then reloading them with the Load button so that the settings are exactly as they were before).

I've now fixed the time bug, and I've made the app produce some user-friendly error messages instead of crashing on a CSV file error. I haven't updated the binaries yet though in case there are any changes to make.

Hopefully there aren't any bugs specific to Windows 7, but please let me know as I've only personally tested it on OS X Mountain Lion and Windows XP.

[Diabolicus]

Ok, thanks, I will test again as soon as you release the update.

[Epoch]

I'm also getting a crash under Win7/64 when trying to export a CSV (candles, UNIX time, do not include volume, 1800s duration ... haven't tried the others). Log says:

Code:

Traceback (most recent call last):
  File "app.py", line 80, in __call__
  File "gui.pyc", line 95, in save_dump
  File "gui.pyc", line 109, in load_dump
IndexError: list index out of range
Traceback (most recent call last):
  File "app.py", line 80, in __call__
  File "gui.pyc", line 95, in save_dump
  File "gui.pyc", line 109, in load_dump
IndexError: list index out of range

Once a new release is out with improved error logging, I'll run it again to see what it shows.

A question: (assuming the crash is fixed in the next release) does anyone know what settings should be chosen for compatibility with WhyDifficult's trading bot?

[nitrous]

I'm also getting a crash under Win7/64 when trying to export a CSV (candles, UNIX time, do not include volume, 1800s duration ... haven't tried the others). Log says:

Code:

Traceback (most recent call last):
  File "app.py", line 80, in __call__
  File "gui.pyc", line 95, in save_dump
  File "gui.pyc", line 109, in load_dump
IndexError: list index out of range
Traceback (most recent call last):
  File "app.py", line 80, in __call__
  File "gui.pyc", line 95, in save_dump
  File "gui.pyc", line 109, in load_dump
IndexError: list index out of range

Once a new release is out with improved error logging, I'll run it again to see what it shows.

A question: (assuming the crash is fixed in the next release) does anyone know what settings should be chosen for compatibility with WhyDifficult's trading bot?

Ok, that bug should be fixed. Expect a new release later today (I'll post again when it's ready). Improved error logging will only apply to the bits of code found to have the potential to cause an error, unexpected errors will still cause a crash unfortunately. When I have enough time I may read through all the code to try to find any more that could happen.

Gekko should work with the format set to 'CSV Candle [Unix]', and volume unchecked. I think in the future he plans to include volume in his candles. Technically, Gekko should still work with volume left checked as his backtest loader script currently ignores any extra fields, but that could change. Make sure to select only one currency, and the other settings are up to you. Also, WhyDifficult's published CSVs included non-primary trades. I usually don't include them, but again that's up to you.

[nitrous]

Ok, I've released a new version It should fix the bugs that have been reported and give more helpful error messages in the cases identified.

[Epoch]

I've just downloaded and tried the new version; still crashes on CSV export. However, this time it did not generate any .log file (the previous version created a logfile in the same directory as the executable). Any special steps needed to get the debugging info?

[Diabolicus]

Same here, still crashes and no log.

[nitrous]

Weird, it worked fine on my XP but I just tested it on windows 7 and I see what you mean. For some reason, Tkinter (the library responsible for the graphics of the app) is causing a crash, and I'm not sure why at the moment. I'll keep looking into it. Also, I have access to OS X 10.8, Windows XP and Windows 7, does this problem occur in Vista as well?

[nitrous]

Ok, my current conclusion is that Tkinter is broken in windows 7 I thought I found the (sort-of) problem, I fixed it, and then another bug came up, entirely due to tkinter. (Tkinter is the graphical library I use, and I use it for around a third of my code!) I believe I have now isolated the bug though and I think I've got around it satisfactorily. Sorry about that.

There are some other changes I also need to make (you still get an error when trying to update a CSV (price/volume) [ISO], for example), so expect a new release tomorrow, and I'll try to do all I can to make Tkinter work. In future, you'll know if it crashes because of Tkinter because you won't get a traceback, just something like 'python.exe has stopped working'. If you get that, please tell me the exact circumstances that led you to get that error so I can try to reproduce it, as without a traceback I have no idea where the bug is coming from.

[nitrous]

Ok, I'm now cautiously optimistic that the new version I just pushed should now work on windows 7 (at least 32 bit). I quickly tested it right now though (OS X 10.8, XP 32bit, 7 32bit) and it seems to be fully functional, touch wood.

Also, I've only experienced this a couple times, but when the Google library fails to get a connection to the Google servers quickly enough it can sometimes crash, I'll see if I can stop it but in the meantime just open the program again and it should be fine. You'll get a logfile if this happens.

[Epoch]

Ok, I'm now cautiously optimistic that the new version I just pushed should now work on windows 7 (at least 32 bit). I quickly tested it right now though (OS X 10.8, XP 32bit, 7 32bit) and it seems to be fully functional, touch wood.

Nope, same problem as the previous build. It crashes on export and does not generate any .log file. I'm running Win7/64.

[Diabolicus]

Windows 7, 64 Bit:
When trying to build 1h candles from the whole 640MB dump.sql, after about 10-15% done it crashes with the following error:

Traceback (most recent call last):
  File "app.py", line 119, in <module>
  File "gui.pyc", line 280, in run
  File "bq.pyc", line 127, in __init__
  File "bq.pyc", line 192, in complete
  File "oauth2client\util.pyc", line 128, in positional_wrapper
  File "apiclient\discovery.pyc", line 192, in build
  File "oauth2client\util.pyc", line 128, in positional_wrapper
  File "oauth2client\client.pyc", line 490, in new_request
  File "httplib2\__init__.pyc", line 1570, in request
  File "httplib2\__init__.pyc", line 1317, in _request
  File "httplib2\__init__.pyc", line 1286, in _conn_request
  File "httplib.pyc", line 1033, in getresponse
httplib.ResponseNotReady
Traceback (most recent call last):
  File "app.py", line 81, in __call__
  File "export.pyc", line 223, in begin_export
  File "formatters\PhantomCircuit.pyc", line 69, in latest
DatabaseError: file is encrypted or is not a database

Windows 7, 32 Bit:
Runtime error after ~20% done:

Traceback (most recent call last):
  File "app.py", line 119, in <module>
  File "gui.pyc", line 280, in run
  File "bq.pyc", line 127, in __init__
  File "bq.pyc", line 192, in complete
  File "oauth2client\util.pyc", line 128, in positional_wrapper
  File "apiclient\discovery.pyc", line 192, in build
  File "oauth2client\util.pyc", line 128, in positional_wrapper
  File "oauth2client\client.pyc", line 490, in new_request
  File "httplib2\__init__.pyc", line 1570, in request
  File "httplib2\__init__.pyc", line 1317, in _request
  File "httplib2\__init__.pyc", line 1286, in _conn_request
  File "httplib.pyc", line 1033, in getresponse
httplib.ResponseNotReady
Traceback (most recent call last):
  File "app.py", line 81, in __call__
  File "export.pyc", line 223, in begin_export
  File "formatters\PhantomCircuit.pyc", line 69, in latest
DatabaseError: file is encrypted or is not a database