nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 03, 2013, 02:35:04 PM Last edit: September 13, 2013, 11:54:35 AM by nitrous |
|
![](https://ip.bitcointalk.org/?u=https%3A%2F%2Fi.imgur.com%2FUMbAQ7n.png&t=665&c=KCzHCVd7mKiREw) ![](https://ip.bitcointalk.org/?u=https%3A%2F%2Fi.imgur.com%2FGOquc4R.png&t=665&c=5voCS7NdD3RSyw) It seems that some windows users have been experiencing trouble with the initial authentication, so I've overridden part of the google code to create a GUI method that should work cross-platform. Please let me know if there are still problems authenticating with Google on any platform, or if the instructions aren't clear enough. In addition, it seems that you do not need to supply your own client_secrets.json. Apparently, it is up to me as the app developer to do this, so there is no longer any need to dig through Google's API Console as I have incorporated the codes into the program, you just need to complete the authorisation in a browser.
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 06, 2013, 06:23:37 PM Last edit: July 06, 2013, 07:06:56 PM by nitrous |
|
Ok everyone, I've completed the tool! You can download a binary for mac or windows here: https://bitbucket.org/nitrous/bq/downloadsThe source is still available here: https://bitbucket.org/nitrous/bqAnd you can build your own binary if you want. The tool is ready to download from the bigquery database, and to export to a number of formats. - CSV (price/volume) [ISO]
- CSV Candle [ISO]
- CSV Candle [Unix]
- Sqlite3 PhantomCircuit
The first option is the format Loozik needed, a csv file with headers date,time,price,volume. Date and time are in ISO 8601 format. The next two generate candlestick data using a port of https://bitcointalk.org/index.php?topic=239815.0. The first one uses ISO date formats and the second, Unix timestamps. The last should be compatible with PhantomCircuit's script http://cahier2.ww7.be/bitcoinmirror/phantomcircuit/. Feel free to download, try it out, and let me know if you find any bugs. Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...). I haven't gotten around to documenting the usage of this tool yet, but hopefully it's pretty self-explanatory.
Windows DLL requirements: There are a number of DLL files which py2exe seems to require. I think those that are come preloaded with Windows, except for the MSVC runtime. Try running the app first without downloading any new stuff. If it doesn't work, first you should try installing the following: http://www.microsoft.com/en-us/download/details.aspx?id=29If you still can't run it, tell me and I'll try to see what the problem is.
|
|
|
|
Loozik
Sr. Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 378
Merit: 250
Born to chew bubble gum and kick ass
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2013, 11:15:39 AM |
|
Ok everyone, I've completed the tool! Fantastic. Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).
Do you perhaps know when MtGox will start updating the database?
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2013, 11:31:01 AM |
|
Ok everyone, I've completed the tool! Fantastic. Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).
Do you perhaps know when MtGox will start updating the database? Sorry, not yet. I've asked a couple times but I've yet to receive an official response.
|
|
|
|
Loozik
Sr. Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 378
Merit: 250
Born to chew bubble gum and kick ass
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2013, 12:28:41 PM |
|
Ok everyone, I've completed the tool! Fantastic. Please note, however, that the bigquery database is not yet being updated, nor is it sorted. When it is, any current dumps will probably be incompatible, so I suggest you don't download a full dump yet unless you need/want the data. When automatic bigquery updates do start, you will need to generate a new dump (File->New Dump...).
Do you perhaps know when MtGox will start updating the database? Sorry, not yet. I've asked a couple times but I've yet to receive an official response. Many thanks. Please let us know when you know something ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif)
|
|
|
|
2weiX
Legendary
Offline
Activity: 2058
Merit: 1005
this space intentionally left blank
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2013, 12:31:28 PM |
|
trying thiso out ASAP. might have some feature requests, happy to donate.
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2013, 03:09:37 PM |
|
trying thiso out ASAP. might have some feature requests, happy to donate.
Great ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) Let me know and I'll see what I can do.
|
|
|
|
Diabolicus
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 90
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 08, 2013, 01:55:02 PM Last edit: July 08, 2013, 02:24:54 PM by Diabolicus |
|
It crashes when trying to export, no matter what settings I choose. Tested on Windows 7 64bit and 32bit. Log says: Traceback (most recent call last): File "app.py", line 80, in __call__ File "export.pyc", line 210, in begin_export File "formatters\Candles.pyc", line 181, in latest ValueError: invalid literal for float(): 2010-07-22
or Exception in thread Thread-1: Traceback (most recent call last): File "threading.pyc", line 808, in __bootstrap_inner File "app.py", line 98, in thread_bootstrap File "app.py", line 52, in fatal NameError: global name 'time' is not defined
Traceback (most recent call last): File "app.py", line 96, in thread_bootstrap File "mtgox.pyc", line 185, in run File "formatters\LoozikCandles.pyc", line 117, in open IOError: Invalid CSV file to append - wrong number of fields
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 08, 2013, 05:10:03 PM |
|
It crashes when trying to export, no matter what settings I choose. Tested on Windows 7 64bit and 32bit. Log says: Traceback (most recent call last): File "app.py", line 80, in __call__ File "export.pyc", line 210, in begin_export File "formatters\Candles.pyc", line 181, in latest ValueError: invalid literal for float(): 2010-07-22
or Exception in thread Thread-1: Traceback (most recent call last): File "threading.pyc", line 808, in __bootstrap_inner File "app.py", line 98, in thread_bootstrap File "app.py", line 52, in fatal NameError: global name 'time' is not defined
Traceback (most recent call last): File "app.py", line 96, in thread_bootstrap File "mtgox.pyc", line 185, in run File "formatters\LoozikCandles.pyc", line 117, in open IOError: Invalid CSV file to append - wrong number of fields Hi Diabolicus, Thanks, the 'time' error was a bug (though it only occurs after the program crashes). Your other errors appear to be you trying to update a CSV file which was created with a different format. There are five types of CSV file that can currently be produced: CSV (price/volume) [ISO] CSV Candle [ISO] - no volume CSV Candle [ISO] - with volume CSV Candle [Unix] - no volume CSV Candle [Unix] - with volume None of these are compatible with each other. It should work as long as you remember which options you used (I recommend saving your export settings with the Save button in the export window and then reloading them with the Load button so that the settings are exactly as they were before). I've now fixed the time bug, and I've made the app produce some user-friendly error messages instead of crashing on a CSV file error. I haven't updated the binaries yet though in case there are any changes to make. Hopefully there aren't any bugs specific to Windows 7, but please let me know as I've only personally tested it on OS X Mountain Lion and Windows XP.
|
|
|
|
Diabolicus
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 90
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 09, 2013, 10:08:12 AM |
|
Ok, thanks, I will test again as soon as you release the update.
|
|
|
|
Epoch
Legendary
Offline
Activity: 922
Merit: 1003
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 04:01:50 AM |
|
I'm also getting a crash under Win7/64 when trying to export a CSV (candles, UNIX time, do not include volume, 1800s duration ... haven't tried the others). Log says: Traceback (most recent call last): File "app.py", line 80, in __call__ File "gui.pyc", line 95, in save_dump File "gui.pyc", line 109, in load_dump IndexError: list index out of range Traceback (most recent call last): File "app.py", line 80, in __call__ File "gui.pyc", line 95, in save_dump File "gui.pyc", line 109, in load_dump IndexError: list index out of range
Once a new release is out with improved error logging, I'll run it again to see what it shows. A question: (assuming the crash is fixed in the next release) does anyone know what settings should be chosen for compatibility with WhyDifficult's trading bot?
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 09:02:01 AM |
|
I'm also getting a crash under Win7/64 when trying to export a CSV (candles, UNIX time, do not include volume, 1800s duration ... haven't tried the others). Log says: Traceback (most recent call last): File "app.py", line 80, in __call__ File "gui.pyc", line 95, in save_dump File "gui.pyc", line 109, in load_dump IndexError: list index out of range Traceback (most recent call last): File "app.py", line 80, in __call__ File "gui.pyc", line 95, in save_dump File "gui.pyc", line 109, in load_dump IndexError: list index out of range
Once a new release is out with improved error logging, I'll run it again to see what it shows. A question: (assuming the crash is fixed in the next release) does anyone know what settings should be chosen for compatibility with WhyDifficult's trading bot? Ok, that bug should be fixed. Expect a new release later today (I'll post again when it's ready). Improved error logging will only apply to the bits of code found to have the potential to cause an error, unexpected errors will still cause a crash unfortunately. When I have enough time I may read through all the code to try to find any more that could happen. Gekko should work with the format set to 'CSV Candle [Unix]', and volume unchecked. I think in the future he plans to include volume in his candles. Technically, Gekko should still work with volume left checked as his backtest loader script currently ignores any extra fields, but that could change. Make sure to select only one currency, and the other settings are up to you. Also, WhyDifficult's published CSVs included non-primary trades. I usually don't include them, but again that's up to you.
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 06:03:56 PM |
|
Ok, I've released a new version ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif) It should fix the bugs that have been reported and give more helpful error messages in the cases identified.
|
|
|
|
Epoch
Legendary
Offline
Activity: 922
Merit: 1003
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 08:18:22 PM |
|
I've just downloaded and tried the new version; still crashes on CSV export. However, this time it did not generate any .log file (the previous version created a logfile in the same directory as the executable). Any special steps needed to get the debugging info?
|
|
|
|
Diabolicus
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 90
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 08:26:15 PM |
|
Same here, still crashes and no log.
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 09:35:27 PM |
|
Weird, it worked fine on my XP but I just tested it on windows 7 and I see what you mean. For some reason, Tkinter (the library responsible for the graphics of the app) is causing a crash, and I'm not sure why at the moment. I'll keep looking into it. Also, I have access to OS X 10.8, Windows XP and Windows 7, does this problem occur in Vista as well?
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 10, 2013, 10:37:16 PM |
|
Ok, my current conclusion is that Tkinter is broken in windows 7 ![Angry](https://bitcointalk.org/Smileys/default/angry.gif) I thought I found the (sort-of) problem, I fixed it, and then another bug came up, entirely due to tkinter. (Tkinter is the graphical library I use, and I use it for around a third of my code!) I believe I have now isolated the bug though and I think I've got around it satisfactorily. Sorry about that. There are some other changes I also need to make (you still get an error when trying to update a CSV (price/volume) [ISO], for example), so expect a new release tomorrow, and I'll try to do all I can to make Tkinter work. In future, you'll know if it crashes because of Tkinter because you won't get a traceback, just something like 'python.exe has stopped working'. If you get that, please tell me the exact circumstances that led you to get that error so I can try to reproduce it, as without a traceback I have no idea where the bug is coming from.
|
|
|
|
nitrous (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 11, 2013, 02:38:38 PM |
|
Ok, I'm now cautiously optimistic that the new version I just pushed should now work on windows 7 (at least 32 bit). I quickly tested it right now though (OS X 10.8, XP 32bit, 7 32bit) and it seems to be fully functional, touch wood.
Also, I've only experienced this a couple times, but when the Google library fails to get a connection to the Google servers quickly enough it can sometimes crash, I'll see if I can stop it but in the meantime just open the program again and it should be fine. You'll get a logfile if this happens.
|
|
|
|
Epoch
Legendary
Offline
Activity: 922
Merit: 1003
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 11, 2013, 03:01:48 PM |
|
Ok, I'm now cautiously optimistic that the new version I just pushed should now work on windows 7 (at least 32 bit). I quickly tested it right now though (OS X 10.8, XP 32bit, 7 32bit) and it seems to be fully functional, touch wood.
Nope, same problem as the previous build. It crashes on export and does not generate any .log file. I'm running Win7/64.
|
|
|
|
Diabolicus
Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 90
Merit: 10
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 11, 2013, 03:26:58 PM Last edit: July 11, 2013, 03:38:54 PM by Diabolicus |
|
Windows 7, 64 Bit: When trying to build 1h candles from the whole 640MB dump.sql, after about 10-15% done it crashes with the following error: Traceback (most recent call last): File "app.py", line 119, in <module> File "gui.pyc", line 280, in run File "bq.pyc", line 127, in __init__ File "bq.pyc", line 192, in complete File "oauth2client\util.pyc", line 128, in positional_wrapper File "apiclient\discovery.pyc", line 192, in build File "oauth2client\util.pyc", line 128, in positional_wrapper File "oauth2client\client.pyc", line 490, in new_request File "httplib2\__init__.pyc", line 1570, in request File "httplib2\__init__.pyc", line 1317, in _request File "httplib2\__init__.pyc", line 1286, in _conn_request File "httplib.pyc", line 1033, in getresponse httplib.ResponseNotReady Traceback (most recent call last): File "app.py", line 81, in __call__ File "export.pyc", line 223, in begin_export File "formatters\PhantomCircuit.pyc", line 69, in latest DatabaseError: file is encrypted or is not a database
Windows 7, 32 Bit: Runtime error after ~20% done: Traceback (most recent call last): File "app.py", line 119, in <module> File "gui.pyc", line 280, in run File "bq.pyc", line 127, in __init__ File "bq.pyc", line 192, in complete File "oauth2client\util.pyc", line 128, in positional_wrapper File "apiclient\discovery.pyc", line 192, in build File "oauth2client\util.pyc", line 128, in positional_wrapper File "oauth2client\client.pyc", line 490, in new_request File "httplib2\__init__.pyc", line 1570, in request File "httplib2\__init__.pyc", line 1317, in _request File "httplib2\__init__.pyc", line 1286, in _conn_request File "httplib.pyc", line 1033, in getresponse httplib.ResponseNotReady Traceback (most recent call last): File "app.py", line 81, in __call__ File "export.pyc", line 223, in begin_export File "formatters\PhantomCircuit.pyc", line 69, in latest DatabaseError: file is encrypted or is not a database
|
|
|
|
|