ncurses based MtGox live monitor and trading-bot-framework

[prof7bit]

...written in python. Curses based UI, live orderbook, ascii-art chart of latest 24h, trading functions can be defined in separate module "strategy.py" which receives all events from the gox client and all keystrokes from the terminal. Strategy module can be edited and reloaded at runtime which is great for experimenting with bots. Works also without MtGox API-key to view only without trading.


(still experimantal, license: GPL3)
http://prof7bit.github.com/goxtool/
https://github.com/prof7bit/goxtool

screenshot


Usage:

• it needs python 2.7 (and maybe a few additional packages but nothing that would not be found in the standard ubuntu repositories). It will NOT work on Windows because there exists no ncurses (maybe it works with cygwin, not tested)
• Download and unzip (or better git clone the repo from github).
• ./goxtool.py

This should give you the orderbook on the left side and an M15 candlestick chart of the lat 24 hours (if your terminal is wide enough) on the right side and a log window at the bottom.

To use the trading features do the following:

./goxtool.py --add-secret

This will ask you for your MtGox API key and secret, you get them at the mtgox website, copy and paste key and secret, supply a passphrase which will be used to encrypt the secret. Then it will write the encrypted secret to goxtool.ini and exit. The next time you start ./goxtool.py it will ask for the passphrase to access the API secret and then all trading functions will be enabled, you will see your own orders in the orderbook and in the chart and your account balance at the top of the terminal.

To use it with other currencies, just edit the ini file. If it is one of these days when the socket.io just wont want to connect, edit the ini file and change use_plain_old_websockets to True. If you don't need history or full orderbook you can turn it off in the ini too.

press q to quit the program.
press F4 to buy
press F5 to sell
press F6 to view orders / cancel orders
press l (lowercase L) to reload the strategy module at runtime.
any other key will call the slot_keypress() in the Strategy class in strategy.py, see the examples. Here you can call methods like gox.buy() or .sell() or .cancel()  or peek around in the order book to decide what fancy stuff to do.

(this forum posting and the instructions might eventually come out of sync with the current version, see here for the official manual: http://prof7bit.github.com/goxtool/)

There are also methods that will be fired on signals from the gox object, you can use them to build fully automated trading bots (there exist even more signals to connect to, for example orderbook.signal_changed or gox.signal_wallet, etc. Look into the source of goxtool to find them all). You can also instantiate more Gox() objects from within your strategy for different currencies (you will also need separate GoxConfig() objects for each of them that use separate config files) to trade multiple currencies at once (useful for arbitrage strategies).

And finally you can rip out parts of this for your own programs, you can just import goxapi and use the Gox() class on its own in your own program, connect your slots to its signals and make your own UI for it (or no UI at all).

[1714846178]

1714846178

[1714846178]

1714846178

[molecular]

cool stuff man, just took a look at the code.

I couldn't find it so you probably haven't implemented keeping an up-to-date synced orderbook? Maybe I didn't look hard enough.

[prof7bit]

I couldn't find it so you probably haven't implemented keeping an up-to-date synced orderbook? Maybe I didn't look hard enough.

Please pull a recent version from github, I just made some changes an hour ago and fixed some bugs and also refactored some stuff (I'm still moving a lot of stuff around, thats why I called it "experimental" ;-).

The orderbook is updated automatically. The Gox() instance contains an OrderBook() instance, gox is generating signals for trade, depth, ticker, user_order, etc and orderbook will connect itself to all these signals. Look for signal_xxxx.connect() calls in the constructors to see what is connected to what and the slot_xxxx() functions to see how they all react to these signals. OrderBook will process the signals fulldepth, ticker, depth, trade and user_order and orderbook itself will then emit signal_changed for the UI to update.

[OpenYourEyes]

Fantastic work. I love curses based programs.

I'll chuck a donation you way soon. Any chance you could display the high/low/last prices at the top near the account info.
It will just make it easier to see the current BTC at a quick glance.

[prof7bit]

Today is again one of these days where MtGox's SocketIO server is almost unusable.

Open the .ini file and switch to plain old websocket protocol:

use_plain_old_websocket = True

Both protocols are implemented, you can compare how they behave and which one is better for you, default is socketio because they say the other one is deprecated but it turns out that plain old websocket is *much* more reliable on days like today while socketio won't connect at all or the messages lag behind websocket by 20 seconds or more!

(If you are going to experiment with a lot of restarts of the application then please temporarily turn off load_fulldepth and load_history because this is eating a lot of bandwidth)

[prof7bit]

Added:
* y-axis labels for chart
* better looking candle bodies in chart
* display current order-lag
* display last trade price, bid and ask in the xterm window title bar
* fixed some bugs and flaws (and probably also introduced new ones)

[Herodes]

Very cool project - I will look into this!

[prof7bit]

Implemented manual trading functions, now its a full featured trading client :-)



In the above screenshot you see the order cancel dialog (press F6 to open it). It is inspired by midnight commander: arrow keys to move up/down, insert to select/unselect an order, F8 to cancel all selected orders, F10 to exit the dialog (sorry, cannot use ESC without ugly hacks). F10 is the key to exit all dialogs.

F4: new buy order
F5: new sell order
F6: cancel order(s)
q quit the client

l (lowercase L) reload the strategy module (your trading bot)

other keys (a..z, except the two above) will emit the signal gox.signal_keypress, the default empty strategy class already has a slot connected to this signal, there you can trigger stuff to happen in your bot when you press certain keys.

Also added since my last posting: display total fiat on the orderbook bid side, total BTC on the orderbook ask side, ratio of these two numbers BTC/USD, everything updating in realtime (see screenshot)

Code:

$ ./goxtool.py -h
usage: goxtool.py [-h] [--add-secret] [--strategy STRATEGY]
                  [--protocol PROTOCOL] [--no-fulldepth] [--no-history]

MtGox live market data monitor and trading bot experimentation framework

optional arguments:
  -h, --help           show this help message and exit
  --add-secret         prompt for API secret, encrypt it and then exit
  --strategy STRATEGY  name of strategy module file, default=strategy.py
  --protocol PROTOCOL  force protocol (socketio or websocket), ignore setting
                       in .ini
  --no-fulldepth       do not download full depth (useful for debugging)
  --no-history         do not download full history (useful for debugging)

happy trading :-)

[K1773R]

is it possible to make it backward compatible to 2.6?

[Deafboy]

+1 for 2.6
Debian squeeze has 2.5, 2.6 and 3.0 in repo.

[prof7bit]

is it possible to make it backward compatible to 2.6?

Not sure, I don't have python 2.6 installed to test it, I relied on the fact that the 2.x branch ended at 2.7 many years ago (I consider it as old and mature and stable) and I assumed that over the time (until now?) from all the different 2.x versions there would eventually only 2.7 be left as the only relevant 2.x version. What Linux distribution are you using? Debian?

What exactly is missing/wrong/different? If it is something trivial it might be possible, but the main problem is I can't (and don't want to) test and maintain it on 2.6 but maybe someone else can temporarily maintain a 2.6 fork until there is no 2.6 anymore (hopefully soon).

[prof7bit]

+1 for 2.6
Debian squeeze has 2.5, 2.6 and 3.0 in repo.

Debian. Isn't the whole point of willingly choosing Debian instead of practically *any* other distribution to explicitly *avoid* any software that is less than 5 years old and work only with old and proven stuff that does not move a single millimeter anymore?

Don't misunderstand me. I mean I really try my best to avoid using any bleeding edge stuff when writing software (I once criticized bitcoin-wx for using a bleeding edge wx-widgets version that was *not* in widespread use), I consider something stable and available and usable for mainstream development if it is in the standard repositories of Ubuntu LTS. I had the same kind of problems when I started TorChat 5(!) years ago with python 2.6 (stable on Ubuntu Hardy at that time and widely used everywhere), soon after the Debian users started complaining that it would not work with their ancient python version.

There were even a few Windows-98 users (imagine! win98!), I even installed win98 in wmvare to see what strange problem they had with python-2.6 on win-98 and to see if I could fix it (and fixed it!) but eventually I decided that it is just not reasonably possible to care about all these exotic corner cases (and a Debian stable user (==conservative by definition) who wants to install the latest bleeding edge software is an exotic corner case). I don't have an army of programmers and testers working for me, At some point I must make a decision and then stick to it. In my case this is the software and libs and their versions that are available in Ubuntu.

[K1773R]

is it possible to make it backward compatible to 2.6?

Not sure, I don't have python 2.6 installed to test it, I relied on the fact that the 2.x branch ended at 2.7 many years ago (I consider it as old and mature and stable) and I assumed that over the time (until now?) from all the different 2.x versions there would eventually only 2.7 be left as the only relevant 2.x version. What Linux distribution are you using? Debian?

What exactly is missing/wrong/different? If it is something trivial it might be possible, but the main problem is I can't (and don't want to) test and maintain it on 2.6 but maybe someone else can temporarily maintain a 2.6 fork until there is no 2.6 anymore (hopefully soon).

python 2.6/2.7 are considered stable, therefore used by most ppls.
well it dosnt work:

Code:

$ python goxtool.py 
Traceback (most recent call last):
  File "goxtool.py", line 1031, in <module>
    main()
  File "goxtool.py", line 1023, in main
    curses.wrapper(curses_loop)
  File "/usr/lib/python2.6/curses/wrapper.py", line 44, in wrapper
    return func(stdscr, *args, **kwds)
  File "goxtool.py", line 947, in curses_loop
    gox = goxapi.Gox(secret, config)
  File "/home/k1773r/git/goxtool/goxapi.py", line 779, in __init__
    BaseObject.__init__(self)
  File "/home/k1773r/git/goxtool/goxapi.py", line 243, in __init__
    self.signal_debug = Signal()
  File "/home/k1773r/git/goxtool/goxapi.py", line 172, in __init__
    self._functions = weakref.WeakSet()
AttributeError: 'module' object has no attribute 'WeakSet'

[prof7bit]

Code:

AttributeError: 'module' object has no attribute 'WeakSet'

This is unfortunate because I am not going to refrain from using these weakref classes. Maybe it would be easier to port the entire program to python-3 than to backport weakref.WeakSet() to a deprecated old python-2.6 version, but then again the problem would be that I would then probably be tempted to use features from python 3.2 (Ubuntu) and Debian users with deprecated 3.0 would complain.

I'm doing this for *fun* and not for profit! If I see that there is there is python 2.7 installed on stable Ubuntu (and even was installed on the previous Ubuntu version too and probably even on the one before that) then I am going to use it. I choose python-2.7 because it is the largest stable island and common denominator in the entire python world that is most likely to be available *everywhere*.

There *must* exist a python-2.7 installer for your distribution, 2.7 is three(!) years stable already. Maybe somewhere in the backports repository. If not then there is something seriously wrong in Debian-land, sorry.

[K1773R]

yes, if backporting isnt possible i will run python2.7 coexistent, that should be fine

[prof7bit]

yes, if backporting isnt possible i will run python2.7 coexistent, that should be fine

I have just googled and it seems you could break your debian stable or bring it into some totally inconsistent state when installing 2.7 on it.

If you really want to experiment with new software then I think debian stable is not the right distribution for you. Its main "advantage" is that it consists only of stuff that is so old (old abandoned branches of software versions) that won't ever change anymore (not moving == stable (== dead?)). You can use it to run old server software that has existed for 20+ years and run them for another 20+ years, software that won't break or change APIs with new updates because the only updates they will ever receive at all will be security fixes. That the Debian definition of "stable": old, petrified, fossilized. Thats what Debian *stable* is meant for.

If you have a Desktop computer and want to be flexible and experiment with cool new stuff now and not in 3 years when its old, you should choose a different distribution. The simplest solution for you would probably to upgrade to Debian testing if you intend to test new (everything that is not really old) software and then stay on their testing release schedule from then on.

Bleeding edge and Debian-Stable just does not go together. Not even "pretty well stable" or "stable" or even "rock solid stable for 2 years now" and "Debian-stable". They have a different and very extreme definition of "stable" over there at Debian.

[K1773R]

yes, if backporting isnt possible i will run python2.7 coexistent, that should be fine

I have just googled and it seems you could break your debian stable or bring it into some totally inconsistent state when installing 2.7 on it.

If you really want to experiment with new software then I think debian stable is not the right distribution for you. Its main "advantage" is that it consists only of stuff that is so old (old abandoned branches of software versions) that won't ever change anymore (not moving == stable (== dead?)). You can use it to run old server software that has existed for 20+ years and run them for another 20+ years, software that won't break or change APIs with new updates because the only updates they will ever receive at all will be security fixes. That the Debian definition of "stable": old, petrified, fossilized. Thats what Debian *stable* is meant for.

If you have a Desktop computer and want to be flexible and experiment with cool new stuff now and not in 3 years when its old, you should choose a different distribution. The simplest solution for you would probably to upgrade to Debian testing if you intend to test new (everything that is not really old) software and then stay on their testing release schedule from then on.

Bleeding edge and Debian-Stable just does not go together. Not even "pretty well stable" or "stable" or even "rock solid stable for 2 years now" and "Debian-stable". They have a different and very extreme definition of "stable" over there at Debian.

you can install another python into a specific prefix, so u dont overwrite the systems versions. or you can do it with virtualenv too

[prof7bit]

I have just run my code through 2to3 and did some additional small adjustments and got it almost working (there are no fundamental problems, its just a few module names have changed, the () around the print (there are only few) and the change to the str class (strings in py3 are always unicode and for byte-strings i have to use bytes() because str() will not have a decode() method anymore, so a few things regarding the encryption/decryption of the api secret and and base64-encoding/decoding and signing stuff must be slightly changed to use bytes() where I currently (ab)use str() for byte strings, this should all be possible without too many problems. Its only few places that need to be changed.

The main problem was the websocket code from Hiroki Ohtani, I first tried to port it myself, it contains a lot of the same sort of string manipulation all over the place and after an hour or so I gave up. Then I saw that there already exists an experimental py3 version of this module from the same author already on github, so not all hope is lost, it can eventually be ported to python 3 if necessary.

I will try to make my own code more python3 friendly (in small steps to not break anything) so that it will be prepared for easy transition eventually but I won't switch to python 3 yet, not at this time.

[algorista]

nice work.

thank you.

[dutt]

Great stuff mate.

[toffoo]

Congratulations and thank you to prof7bit (I've tried many CLI-type tools posted here and this is probably the best) and also to MagicalTux because the new beta streaming socket.io feed is FINALLY the first real improvement in reliability since pretty much ever.  You can get it with something like this:

Code:

goxtool.py --protocol=socketio-beta

I am on Mac and can confirm it is working really good.

I believe new Mac OS X versions ship with python v2.7 installed.

Along with prof7bit's code you need to install this pycrypto package from here:

https://pypi.python.org/pypi/pycrypto

which of course will not install unless you have XCode installed, because it needs to compile something.  If you don't want to install or don't have XCode you can also download a Command Line Only Tools package which can also compile/install this if you register as an Apple Developer.

If you can post us some boilerplate examples on how to use strategy.py I will certainly tip your donation address if I manage to make any money autotrading with this thing.

[wumpus]

Ha, bravo. Looks great.

[01BTC10]

What is missing?

Code:

~/goxtool$ ./goxtool.py 
Traceback (most recent call last):
  File "./goxtool.py", line 30, in <module>
    import goxapi
  File "/home/r2d2/goxtool/goxapi.py", line 34, in <module>
    from Crypto.Cipher import AES
ImportError: No module named Crypto.Cipher

~/goxtool$ ./goxtool.py --add-secret
Traceback (most recent call last):
  File "./goxtool.py", line 30, in <module>
    import goxapi
  File "/home/r2d2/goxtool/goxapi.py", line 34, in <module>
    from Crypto.Cipher import AES
ImportError: No module named Crypto.Cipher

~/goxtool$ python --version
Python 2.7.3

[toffoo]

Along with prof7bit's code you need to install this pycrypto package from here:

https://pypi.python.org/pypi/pycrypto

[01BTC10]

Along with prof7bit's code you need to install this pycrypto package from here:

https://pypi.python.org/pypi/pycrypto

Working thanks very much!

[refer_2_me]

Hi, this looks great. I am trying to get started writing some custom trading strategies, but I am having some trouble figuring out the API. Could you post some sample strategy files for me to work off of? I'm not looking for your secret strategy, but rather a more in depth framework, i.e. where does the code for deciding to buy/sell go etc.

Thanks again for this great looking tool!

[ScarecrowMagick]

So far this looks awesome. How long does it take for Account: No Info (yet)  to update, or am I doing something wrong?

Nevermind, I had a dificulty with my key entry. All is awesome now.

Scarecrowmagick

[JWU42]

I keep getting

Code:

Account: No info (yet)

EDIT - Seems using

Code:

--protocol socketio

didn't work to ever get account info.

[ErebusBat]

I keep getting

Code:

Account: No info (yet)

EDIT - Seems using

Code:

--protocol socketio

didn't work to ever get account info.

You need to create a GOX API Key if you want to see your account info.  If you just want to display market conditions you don't need to do this:

To use the trading features do the following:

./goxtool.py --add-secret

This will ask you for your MtGox API key and secret, you get them at the mtgox website, copy and paste key and secret, supply a passphrase which will be used to encrypt the secret. Then it will write the encrypted secret to goxtool.ini and exit. The next time you start ./goxtool.py it will ask for the passphrase to access the API secret and then all trading functions will be enabled, you will see your own orders in the orderbook and in the chart and your account balance at the top of the terminal.

[JWU42]

Seriously think I didn't do that first 

The issues all related to the API being down for the first bit after the re-start last night.

Using

Code:

./goxtool.py --protocol socketio --use-http

got it working as it fetched account data via HTTP instead of the API.

[fnordfnordfnord]

This is rad thanks prof7bit. Now if Mtgox will just handle their business.

[fnordfnordfnord]

New ticker

http://data.mtgox.com/api/2/BTCUSD/money/ticker_fast

http://www.reddit.com/r/Bitcoin/comments/1c9npl/mtgox_fast_ticker_last_only_no_rate_limit_1second/

[fujiwara]

Great! I even managed to set it up. But what about strategy.py - any examples?

[dutt]

prof7bit, the people demand bitfloor support

[prof7bit]

Great! I even managed to set it up. But what about strategy.py - any examples?

Here is an example: https://gist.github.com/prof7bit/5395900

Read everything below carefully and don't run it if you not 100% understand what it does and what its implications are, including temporary loss when BTC price goes down! I'm not giving any support or guarantee for this, its only example code, I'm using it myself but I'm not going to give support for any trading bot code, you can only run a trading bot if you 100% understand what it does!


* Save the above file in _balancer.py (I'm using underscores because these filenames are in .gitignore)

* Update goxtool to the very latest version (important because there is a new slot in strategy.py!)

* Then start goxtool:

./goxtool.py --strategy=_balancer.py --protocol=socketio --use-http

* bring your account into 50/50 balance manually (exactly half of the money in USD, other half in BTC at current price, use a pocket calculator to have really exact numbers, then cancel all open orders)

* Then press p (this will place 2 orders above and below price. If they seem wrong press c to cancel them again and find the error. if anything is wrong press c to cancel orders and effectively stop the bot)

The bot will trade if one of the orders has been filled, it will then cancel the other one and place two fresh orders above and below last trade price. The bot will maintain a 50/50 ratio of your USD and BTC account balances. Please read about portfolio rebalancing to understand the implications and why you would want (or not want) to do this. This should only be regarded as exemplary code, study it to learn how to use the trading API, don't just run it blindly. I repeat: I will not be responsible for any damage you do to your money!

If you look closely you will see that it is working around some limitations of the gox streaming API. Trade messages and user_order messages will not arrive in the order we would ideally like to have, I cannot trigger any action from within slot_trade() because at this time the number of open orders will not yet have been updated (fortunately the wallet *will* be updated already), so I need to jump through one additional hoop. If you design a bot yourself then print debug output until you understand the exact order in which the messages arrive and which information is available at which time.

[ErebusBat]

Here is an example: https://gist.github.com/prof7bit/5395900

Thanks for this prof7bit.  Taking a look now.

[infebious]

I just sent a donation your way!  Thanks for this it's exactly what I needed.   

[ErebusBat]

I just sent a donation your way!  Thanks for this it's exactly what I needed.   

As did I.... 

I was able to make some money, nothing huge, but sent it your way for all your hard work.

[JWU42]

Just another friendly mention that a Bitfloor "version" would be of interest.  If there needs to be a minimum pledged let us know as I am fairly sure it can be accomplished.

[prof7bit]

Just another friendly mention that a Bitfloor "version" would be of interest.  If there needs to be a minimum pledged let us know as I am fairly sure it can be accomplished.

I have thought about this too already. Its just a matter of time. But since its GPL maybe someone else who is faster than me can fork it and make a bitfloor version, maybe with some refactoring it might even be possible later to combine both into one tool again. Its not imposible its just some work that needs to be done and time to put into it and I also have so many other unfinished things that occasionally need my attention, I wish I could fork myself into many parallel processes to get more stuff done ;-)

[teek]

wow this looks awesome,  checking it out!

[JWU42]

Just another friendly mention that a Bitfloor "version" would be of interest.  If there needs to be a minimum pledged let us know as I am fairly sure it can be accomplished.

I have thought about this too already. Its just a matter of time. But since its GPL maybe someone else who is faster than me can fork it and make a bitfloor version, maybe with some refactoring it might even be possible later to combine both into one tool again. Its not imposible its just some work that needs to be done and time to put into it and I also have so many other unfinished things that occasionally need my attention, I wish I could fork myself into many parallel processes to get more stuff done ;-)

Appreciate the reply.  If I had the skills I would have tried to write it myself...

[ErebusBat]

I was wondering what someone would charge to calculate arbitrary EMAs in the strategy file.

I am no python expert, nor a GOX api specialist.

[toffoo]

Just another friendly mention that a Bitfloor "version" would be of interest.  If there needs to be a minimum pledged let us know as I am fairly sure it can be accomplished.

I have thought about this too already. Its just a matter of time. But since its GPL maybe someone else who is faster than me can fork it and make a bitfloor version, maybe with some refactoring it might even be possible later to combine both into one tool again. Its not imposible its just some work that needs to be done and time to put into it and I also have so many other unfinished things that occasionally need my attention, I wish I could fork myself into many parallel processes to get more stuff done ;-)

Appreciate the reply.  If I had the skills I would have tried to write it myself...

No point now!

Think of it like you just saved yourself the trouble of doing lots of extra work for nothing!

[JWU42]

Toffoo,

Had no idea what you were speaking of until I visited the site just now 

[fnordfnordfnord]

prof7bit, the people demand bitfloor support

Belay that order, disregard.

[smooth]

Is this actually usable?  socketio seems to drop constantly and lag horribly, especially on orders.  websocket just sits there and doesn't update.

[dutt]

Yea. The problem lies with mtgox though, not with the code.

[fnordfnordfnord]

Is this actually usable?  socketio seems to drop constantly and lag horribly, especially on orders.  websocket just sits there and doesn't update.

Been running it for a week, it is surprisingly robust considering the problems with mtgox, and me suspending my laptop a couple of times per day.

Please read about portfolio rebalancing to understand the implications and why you would want (or not want) to do this.

So, about that, got any good links or books to recommend? Googling for "portfolio rebalancing" gets me a bunch of noise about how I should own a diversity of stocks, bonds, and other garbage. Googling for "Forex rebalancing" is only a bit better.

[ErebusBat]

A very very basic overview on rebalancing would be that it protects you from volitilty of one asset.

[smooth]

Been running it for a week, it is surprisingly robust considering the problems with mtgox, and me suspending my laptop a couple of times per day.

What API configuration are you using?  Do you trade or just use it to watch the market?

[fnordfnordfnord]

Been running it for a week, it is surprisingly robust considering the problems with mtgox, and me suspending my laptop a couple of times per day.

What API configuration are you using?  Do you trade or just use it to watch the market?

I installed it last weekend with git.

Code:

git clone https://github.com/prof7bit/goxtool.git

I run it just like this.

Code:

./goxtool.py  --protocol=socketio --use-http 

I tried other configurations, but I can't say anything good or bad considering the conditions. I wouldn't call it a controlled experiment in other words.

I traded manually with it this week a bit, and it worked fine. I changed the colors up a bit since I have 256 color xterm. Until today I was just entering trades manually but then I checked the thread and saw the _balancer so I did a git stash and a git pull and I have been playing with it this evening. Fun stuff, thanks to OP for posting the example. It's better (more complete) than what I was throwing together. A good starting point.

PS If anyone is interested, I found a book on Amazon http://www.amazon.com/Trading-Systems-Methods-Website-Wiley/dp/1118043561

[smooth]

I run it just like this.

Code:

./goxtool.py  --protocol=socketio --use-http 

Thanks, that does seem to work a lot better.

[prof7bit]

A very very basic overview on rebalancing would be that it protects you from volitilty of one asset.

and it is producing a small profit in the long run and outperforms a buy and hold strategy. This is because you are always selling higher than you have bought. Its only a small amount because only a tiny fraction of your BTC is bought or sold on every step but if BTC price is very volatile and does two digit percent changes every single day and huge ups and downs this can become significant quite fast.

Its a very conservative strategy and probably one of the few strategies that can cannot accidentally burn all your money in one night if the bot is crashing/hanging while you are sleeping.

Maybe we could make a separate forum thread where people post their bots and discuss them.

[ErebusBat]

@prof7bit

I noticed that sometimes with the balancer + gox connection issues sometimes the balancer gets out of whack.  Last night with the spike at $130 was a prime example.

Would it be possible to add a slot to the strategy so that on a gox reconnection the strategy can then re-asses if anything needs to be done (a cancel and re-place orders in the case of the balancer).

[K1773R]

Bug Report: please check if the terminal is a terminal emulation and not a real tty otherwise the CUI looks ugly because of this code

Code:

# update the xterm title (this is not handled by curses)
        if self.gox.config.get_bool("goxtool", "set_xterm_title"):
            last_candle = self.gox.history.last_candle()
            if last_candle:
                title = goxapi.int2str(last_candle.cls, self.gox.currency).strip()
                title += " - goxtool -"
                title += " bid:" + goxapi.int2str(book.bid, self.gox.currency).strip()
                title += " ask:" + goxapi.int2str(book.ask, self.gox.currency).strip()
                curses.putp("\033]0;%s\007" % title)

if you want i can provide a screenshot

EDIT: im aware of the following code, still its not nice. altough its good that you can disable it with ease.

Code:

if self.gox.config.get_bool("goxtool", "set_xterm_title"):

[prof7bit]

Would it be possible to add a slot to the strategy so that on a gox reconnection the strategy can then re-asses if anything needs to be done (a cancel and re-place orders in the case of the balancer).

Yes, something like that is on my todo list because i have exactly the same problems with the balancer.bot. As a workaround I have slightly modified the _balancer.py code to also trade when it suddenly detects only one order while the trade signal has been missed:

Code:

"""
The portfolio rebalancing bot will buy and sell to maintain a
constant asset allocation ratio of exactly 50/50 = fiat/BTC
"""

import strategy
import goxapi

DISTANCE    = 7     # percent price distance of next rebalancing orders
MARKER      = 7     # lowest digit of price to identify bot's own orders
COIN        = 1E8   # number of satoshi per coin, this is a constant.

def add_marker(price, marker):
    """encode a marker in the price value to find bot's own orders"""
    return price / 10 * 10 + marker

def has_marker(price, marker):
    """return true if the price value has the marker"""
    return (price % 10) == marker

def mark_own(price):
    """return the price with our own marker embedded"""
    return add_marker(price, MARKER)

def is_own(price):
    """return true if this price has our own marker"""
    return has_marker(price, MARKER)



class Strategy(strategy.Strategy):
    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.last_trade = 0
        self.temp_halt = False

    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
        """a key has been pressed"""

        if key == ord("c"):
            # remove existing rebalancing orders
            self.debug("canceling all rebalancing orders")
            self.temp_halt = True
            self.cancel_orders()

        if key == ord("p"):
            # create the initial two rebalancing orders. Before you
            # do this the portfolio should already be balanced.
            self.debug("adding new initial rebalancing orders")
            book = self.gox.orderbook
            self.temp_halt = False
            self.place_orders((book.bid + book.ask) / 2)

        if key == ord("u"):
            gox.client.channel_subscribe()

        if key == ord("i"):
            price = (gox.orderbook.bid + gox.orderbook.ask) / 2
            vol_buy = self.get_buy_at_price(price)
            self.debug("BTC difference:",
                goxapi.int2float(vol_buy, "BTC"))

        if key == ord("b"):
            price = (gox.orderbook.bid + gox.orderbook.ask) / 2
            vol_buy = self.get_buy_at_price(price)
            if abs(vol_buy) > 0.01 * COIN:
                self.temp_halt = True
                self.cancel_orders()
                if vol_buy > 0:
                    self.debug("buy %f at market" %
                        goxapi.int2float(vol_buy, "BTC"))
                    gox.buy(0, vol_buy)
                else:
                    self.debug("sell %f at market" %
                        goxapi.int2float(-vol_buy, "BTC"))
                    gox.sell(0, -vol_buy)



    def cancel_orders(self):
        """cancel all rebalancing orders, we identify
        them through the marker in the price value"""
        must_cancel = []
        for order in self.gox.orderbook.owns:
            if is_own(order.price):
                must_cancel.append(order)

        for order in must_cancel:
            self.gox.cancel(order.oid)

    def get_buy_at_price(self, price_int):
        """calculate amount of BTC needed to buy at price to achieve
        rebalancing. price and return value are in mtgox integer format"""
        currency = self.gox.currency
        fiat_have = goxapi.int2float(self.gox.wallet[currency], currency)
        btc_have  = goxapi.int2float(self.gox.wallet["BTC"], "BTC")
        price_then = goxapi.int2float(price_int, currency)

        btc_value_then = btc_have * price_then
        diff = fiat_have - btc_value_then
        diff_btc = diff / price_then
        must_buy = diff_btc / 2
        return goxapi.float2int(must_buy, "BTC")

    def place_orders(self, center):
        """place two new rebalancing orders above and below center price"""
        currency = self.gox.currency
        step = int(center * DISTANCE / 100.0)
        next_sell = mark_own(center + step)
        next_buy  = mark_own(center - step)

        sell_amount = -self.get_buy_at_price(next_sell)
        buy_amount = self.get_buy_at_price(next_buy)

        if sell_amount < 0.01 * COIN:
            sell_amount = int(0.01 * COIN)
            self.debug("WARNING! minimal sell amount adjusted to 0.01")

        if buy_amount < 0.01 * COIN:
            buy_amount = int(0.01 * COIN)
            self.debug("WARNING! minimal buy amount adjusted to 0.01")

        self.debug("new buy order %f at %f" % (
            goxapi.int2float(buy_amount, "BTC"),
            goxapi.int2float(next_buy, currency)
        ))
        self.gox.buy(next_buy, buy_amount)

        self.debug("new sell order %f at %f" % (
            goxapi.int2float(sell_amount, "BTC"),
            goxapi.int2float(next_sell, currency)
        ))
        self.gox.sell(next_sell, sell_amount)


    def slot_trade(self, gox, (date, price, volume, typ, own)):
        """a trade message has been receivd"""
        # not interested in other people's trades
        if not own:
            return

        # not interested in manually entered (not bot) trades
        if not is_own(price):
            return

        text = {"bid": "sold", "ask": "bought"}[typ]
        self.debug("*** %s %f at %f" % (
            text,
            goxapi.int2float(volume, "BTC"),
            goxapi.int2float(price, gox.currency)
        ))
        self.last_trade = price
        self.check_trades()

    def slot_owns_changed(self, orderbook, _dummy):
        """status or amount of own open orders has changed"""
        self.check_trades()

    def check_trades(self):
        """find out if we need to place new orders and do it if neccesary"""

        # bot temporarily disabled
        if self.temp_halt:
            return

        # still waiting for submitted orders,
        # can wait for next signal
        if self.gox.count_submitted:
            return

        # we count the open and pending orders
        count = 0
        count_pending = 0
        book = self.gox.orderbook
        for order in book.owns:
            if is_own(order.price):
                if order.status == "open":
                    count += 1
                else:
                    count_pending += 1

        # as long as there are ANY pending orders around we
        # just do nothing and wait for the next signal
        if count_pending:
            return

        # if count is exacty 1 then one of the orders must have been filled,
        # now we cancel the other one and place two fresh orders in the
        # distance of DISTANCE around current price.
        if count == 1:
            price = self.last_trade
            self.last_trade = 0
            if not price:
                price = (book.ask + book.bid) / 2
                self.debug(
                    "*** missed trade message, using current price %f" %
                    goxapi.int2float(price, self.gox.currency)
                )
            self.cancel_orders()
            self.place_orders(price)

there are also some new keyboard commands:

"u": update own order list, depth, history, wallet and everything
"i": display how much it is out of balance at current price (negative: must sell, positive: must buy)
"b":  balance immediately (cancel orders and then buy/sell at market to rebalance)

[ErebusBat]

Would it be possible to add a slot to the strategy so that on a gox reconnection the strategy can then re-asses if anything needs to be done (a cancel and re-place orders in the case of the balancer).

Yes, something like that is on my todo list because i have exactly the same problems with the balancer.bot. As a workaround I have slightly modified the _balancer.py code to also trade when it suddenly detects only one order while the trade signal has been missed:

Thanks for this, esp the 'b' command I just deposited this morning and would have been very helpful.   

Do you have the bot in a separate git repo that can be tracked?

For those playing along at home, here is a command summary & launch instructions for prof7bit's balancer bot:

Code:

# "p": place 2 orders above and below price. 
# "c": cancel orders and effectively stop the bot
# "u": update own order list, depth, history, wallet and everything
# "i": display how much it is out of balance at current price (negative: must sell, positive: must buy)
# "b":balance immediately (cancel orders and then buy/sell at market to rebalance)
./goxtool.py --protocol=socketio --use-http --strategy=_balancer.py

[prof7bit]

Do you have the bot in a separate git repo that can be tracked?

No, not yet.

[havardr]

I just tested the balancer.py now, and i must say it had a huge spread.

Had about 7BTC, divided in two, and got 433 USD in cash. Current price is 120.
Started the bot with p, and it put in a sell at 129.03137, and a buy at 112.14877   

[prof7bit]

I just tested the balancer.py now, and i must say it had a huge spread.

Had about 7BTC, divided in two, and got 433 USD in cash. Current price is 120.
Started the bot with p, and it put in a sell at 129.03137, and a buy at 112.14877   

Yes, its a compromise between how often should it trade and how big should be the effect of the the rebalancing (the profit after a consecutve buy and sell). I cannot mathematically prove it but my gut feeling tells me in a normally trading market the overall profit should be the same (half the distance leads to 1/4 the profit but 4 times more trades [and more mtgox fees, so it would actually be less profitable]).

Faster trades also means more problems when mtgox becomes slow (order lag) or disconnects often during fast price moves. You can experiment with smaller numbers by changing the value of DISTANCE, don't make it smaller than 1.2, at 1.2 you will pay 100% of your profits as mtgox fees, maybe 2 would be a good minimum DISTANCE to experiment with when the market is slow but I guarantee that as soon as there are sudden large price moves mtgox will start lagging at exactly the wrong times and it won't be able to cancel and place new orders fast enough.

This bot is not intended to make you rich quickly, you need to run it for months to see an effect. Write down the BTC price and your account balance once a day and plot the curves on top of each other after a few months. You could also program a simulation if you find historical price data to experiment with different distances, I haven't done this yet.

[molecular]

prof7bit, I didn't try out the balance bot, but from the description it seems doesn't work automtically but you have to tell it to place orders manually?

If it does work automatically, how does it decide at which point to "balance out" (i.e. how much divergence from 50/50 does it tolerate)? (I've written a similar bot in my traidor "framework". Never got it to fully work, even after 2nd try, I kept scrwing it up, then came the api change at gox (this was quite a while ago) but the goal was the same. I thought about putting the asks/bids just below/above walls. Also it didn't place just 2 bids, but a whole array all the way up/down so it wouldn't run into problems with lag)

[prof7bit]

If it does work automatically, how does it decide at which point to "balance out"

It will place orders in the distance of 7% price change. So for example if price is at $100/BTC and you have 1000USD and 10BTC in your account is exactly balanced, it would place a sell order 0.327 at 107 and a buy order 0.376 at 93. Then it will wait until price moves and one of these orders is filled and automatically cancel the other and place 2 fresh orders in +- 7% distance around the new price. Thats all. It should happen automatically. BUT it will only work if you don't confuse the logic by having other open orders in your account! It will try to tell bot orders from manually entered orders by looking at the last digit of the order price but thats only an ugly hack! Make sure you dont have any other open orders whose price ends with "7" (or whatever you configured the value of MARKER)!

Standard disclamer for algorithmic trading: If you don't understand the logic of the bot by reading the source code and maybe inserting some more debug output then please don't use it. Please don't ask me, you cannot run a bot if you are not able to debug it yourself! That is simply not possible!

If the distance (default = 7%) is chosen too narrow then it might need to place orders faster than mtgox lag allows and it might come out of sync.

If it is messed up completely (happens maybe once a week) press c to cancel all orders and if it won't cancel them all or is behaving strange then cancel them manually, maybe restart goxtool or press u to get a manually updated list of open orders, use F6 to check your open orders to make sure they are all canceled. then press i to see how far you are out of balance at current price and if needed press b to rebalance manually, press p to place 2 fresh orders.

And as i have mentioned before already: make sure you ALWAYS have an up to date version of goxtool. git pull (and restart if there were updates) at least once a day, I'm still fixing things in goxtool itself occasionally that make it more reliable.

[fnordfnordfnord]

Maybe we could make a separate forum thread where people post their bots and discuss them.

That's a good idea.

(half the distance leads to 1/4 the profit but 4 times more trades [and more mtgox fees, so it would actually be less profitable]).

Correct me if I am wrong but Mtgox charges a 0.6% fee per trade so it makes little difference whether you make large* or small ones. But, as you say, you do need to ensure that you make approximately >(0.6*2)% profit on trades or you will simply be slowly giving away your account to Mtgox in the form of fees.

* Large being relative here.

[daybyter]

The channel #cryptocoin-bots is quiet at the moment, but was used for such discussion a while ago....

[prof7bit]

Correct me if I am wrong but Mtgox charges a 0.6% fee per trade so it makes little difference whether you make large* or small ones. But, as you say, you do need to ensure that you make approximately >(0.6*2)% profit on trades or you will simply be slowly giving away your account to Mtgox in the form of fees.

* Large being relative here.

Yes, its the price difference between buy and sell, not the trade volume that matters. 2 * 0.6% fee translates into a minimum price difference of 1.2% between buy and sell price to reach the profit zone, if you buy at 100 you need to sell above 101.2 to become profitable, so the samller the distance, the larger (relative) the part that goes to mtgox on every trade.

[fnordfnordfnord]

Okay, here are a couple questions,
Q1. Are you interested in bug reports at your github?
Q2. Do you mind contributions from others?
Q2. a. Even from individuals who are obviously less capable programmers than yourself? It might require some patience on your part, especially in the beginning.
Q3. I might want to add some features that some may think is really goofy (like play sounds when trades execute),

let's say I make some changes to stragegy.py, and I want to test them. We can consider several testing strategies

• Test on historical data from mtgox.
  
  • interesting since goxtool grabs this data when it connects, and because this data is real, and relevant to current mkt conditions
  • bad because it's not a controlled test, and therefore difficult to compare strategies
  • dangerous since we're connected to a live account. Maybe we should make for ourselves different API access for testing vs trading

• Test on fake data we make ourselves. This could be:
  
  • Old data we've saved
  • Data collected from other exchanges
  • Fake sample data sets.

There is also various levels of diligence to consider,

• simply iterate over the data?
• iterate, and try to account for realistic mkt timings, liquidity, etc.

Do you think it would be easier to write a new program for testing that imports some strategy.py, or add testing functionality to goxtool?
Is that a feature that you might be personally interested in, or would consider adding to the codebase?

[prof7bit]

Okay, here are a couple questions,
Q1. Are you interested in bug reports at your github?

yes. I just cannot promise that I will include every new feature but even then its easy on github to maintain a modified fork (there are a few already) and still exchange patches with each other that are important.

Do you think it would be easier to write a new program for testing that imports some strategy.py, or add testing functionality to goxtool?

Not 100% sure. A full blown simulator could load the strategy and implement the same interface and emit the same signals as Gox() does, it could run as multiple processes (maybe even on multiple machines) to test different parameters of the same strategy in parallel, maybe its really a good idea to make such a thiing a completely separate application and not complicate the goxapi too much, since only few people will use it.

[prof7bit]

Maybe we could make a separate forum thread where people post their bots and discuss them.

That's a good idea.

I have made a new thread for the _balancer.py now, maybe a separate thread for every separate bot is even better: https://bitcointalk.org/index.php?topic=181584.0

[rm187]

Love this. Plan to do my own custom changes.

Also setup a server for public to check out the script/watch gox. Basically just another ticker; but useful for some people depending on internet restrictions.

gox.ruststorm.com:22
user:goxtool
pass:password

[JWU42]

Interesting idea...

Just tested it out and looks cool.  Wish I knew how to script to have it load goxtool and exit the ssh session as you did...

[K1773R]

Love this. Plan to do my own custom changes.

Also setup a server for public to check out the script/watch gox. Basically just another ticker; but useful for some people depending on internet restrictions.

gox.ruststorm.com:22
user:goxtool
pass:password

thanks, now i can tell all the win kids they can use goxtool too with putty

[rm187]

Interesting idea...

Just tested it out and looks cool.  Wish I knew how to script to have it load goxtool and exit the ssh session as you did...

I just setup the script as the user shell.
Add the script location to the shells(/etc/shells), change the user's shell with chsh username, then when prompted, enter the location of goxtool.py(same as what u added in shells). (Debian env here, RH may be little different)

Love this. Plan to do my own custom changes.

Also setup a server for public to check out the script/watch gox. Basically just another ticker; but useful for some people depending on internet restrictions.

gox.ruststorm.com:22
user:goxtool
pass:password

thanks, now i can tell all the win kids they can use goxtool too with putty

np. I threw in shellinabox too for HTTP display. Though it doesn't respond as well as a real terminal.
http://gox.ruststorm.com/

[prof7bit]

np. I threw in shellinabox too for HTTP display. Though it doesn't respond as well as a real terminal.
http://gox.ruststorm.com/

wow, this is a funny idea! I think I should add a configuration option for (or automatic detection of) black&white terminals too to make this look even better :-)

[c0inbuster]

Thanks prof7bit for your hard work.
I ever see you on a FX forum (sorry I don't remember which one)

It will be very nice if you could add features such as:
- stop loss (sell stop)
- trailing stop
- stepping stop

You should also add support of Python Pandas library
http://pandas.pydata.org/
because some technical analysis indicators have been ported to this lib
https://code.google.com/p/ultra-finance/wiki/pyTaLib

of course making this software available to other markets (Bitcoin-central for
example) will be great !

Kind regards

[ErebusBat]

np. I threw in shellinabox too for HTTP display. Though it doesn't respond as well as a real terminal.
http://gox.ruststorm.com/

wow, this is a funny idea! I think I should add a configuration option for (or automatic detection of) black&white terminals too to make this look even better :-)

On that note... When connecting to my screen instance from my ipad it isn't immediately clear what the last price was because f no title bar.  Would you be willing to accept a pull request to add this by account balance?

[K1773R]

np. I threw in shellinabox too for HTTP display. Though it doesn't respond as well as a real terminal.
http://gox.ruststorm.com/

even more awesome
i will use this too!

[prof7bit]

np. I threw in shellinabox too for HTTP display. Though it doesn't respond as well as a real terminal.
http://gox.ruststorm.com/

even more awesome
i will use this too!

This is the true power of the unix philosopy "one job one tool", you can just plug a few small existing tools together without needing to change or adapt any them and instantly create something totally new :-)

[K1773R]

np. I threw in shellinabox too for HTTP display. Though it doesn't respond as well as a real terminal.
http://gox.ruststorm.com/

even more awesome
i will use this too!

This is the true power of the unix philosopy "one job one tool", you can just plug a few small existing tools together without needing to change or adapt any them and instantly create something totally new :-)

thats why i love linux (not unix, unix has been a mess too often ) myself so much
its like LEGO, you build your tools/program and can wrap em with whatever u want!

[rm187]

Well, shellinabox does not close sessions on browser closes. Had 9.99 load and few hundred goxtools trying to run. I have setup a script to kill processes that continue after a session is lost. Hopefully fixes the problem.

I am adding more GUI elements, options (currency/graph type, etc), personal modifications. prof7bit, are you currently working on any additions?

[c0inbuster]

It will be nice to also add email notifications (and/or push notifications) when trading

If you are looking for email notifier
I did a Python class for that purpose
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/email_notifier.py

If you are looking for push notifier I found a very useful web service called PushOver
I also did a Python class for that purpose
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/pushover_notifier.py

[prof7bit]

It will be nice to also add email notifications (and/or push notifications) when trading

Functionality like this could be wrapped into separate strategy modules. Also another strategy module could be made whose only purpose is to load a list of other strategies so that you could for example run the email notifier strategy in parallel with for example a trailing stop strategy.

[c0inbuster]

I like this idea...
but a strategy like trailing stop should also have access to other strategies (such as notifiers)

object programming using python is permissive so it shouldn't be a problem

but you will have to use a kind of "dependency graph" between strategies

(if you want that trailing stop strategy will be able to send notifications you must ensure that
email_notifier strategy or pushover_notifier was load)

[prof7bit]

(if you want that trailing stop strategy will be able to send notifications you must ensure that
email_notifier strategy or pushover_notifier was load)

A simple pragmatic solution: Any notifyer strategy could connect a slot to gox.signal_debug, it could look like this:

Code:

import strategy
import whatever_message_api

RECEIVER = "somebody@example.com"

class Strategy(strategy.Strategy):
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        gox.signal_debug.connect(self.slot_debug)

    def slot_debug(self, sender, (msg)):
        if "NOTIFY" in msg:
            whatever_message_api.sendmsg(RECEIVER, msg)

       

and every other strategy that wants to send an important message would then simply do something like this:

Code:

        self.debug("NOTIFY", "foo bar %f happened at %f" % (bla, blub))

The string "NOTIFY" would then be established as some kind of convention, it wouldn't do any harm if no notifier is present (it would just go to the log file like all other messages too) and if an older strategy does not send messages containing "NOTIFY" or notifications for more events are desired then the notification strategy could optionally do some more advanced filtering of debug messages.

[c0inbuster]

Sometimes I'm not pragmatic

Do you know how to load list of strategies ?

Do you plan to add notifier (email or push) in your code base ?

Thanks a lot

[prof7bit]

Do you know how to load list of strategies ?

_loader.py (untested code, just wrote it down here without testing but should work this way unless I have forgotten something, for example not tested what happens on reload during runtime)
(use it like this: ./goxtool --strategy=_loader.py)

(my [personal] convention is to prefix all strategy module filenames (and other eperimental stuff in this folder) with _ to not confuse git too much with unversioned files in the same folder, I have put _*.* into my .gitignore file)

Code:

import strategy

import _notifierstrat
import _foostrat
import _barstrat
import _whateverstrat

class Strategy(strategy.Strategy):
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)

        self.notifierstrat  =  _notifierstrat.Strategy(gox)
        self.foostrat       =       _foostrat.Strategy(gox)
        self.barstrat       =       _barstrat.Strategy(gox)
        self.whateverstrat  =  _whateverstrat.Strategy(gox)

[c0inbuster]

Nice!

I prefer not to get away from your code...
so I will git pull when it will be ready

I think it's both a good idea :
- to avoid breaking a large part of code (and that's what you are doing elegantly)
- to have notifications when auto trading (I prefer to know very quickly [bad] surprises)

You should also have a look at my suggestion
https://bitcointalk.org/index.php?topic=181584.msg1911355#msg1911355
it will allow us to see several symbols (BTCUSD, BTCEUR...)

[prof7bit]

Do you know how to load list of strategies ?

_loader.py
[ ...code...]

This is obsolete now. After pulling the latest revision you can now just pass a comma separated list of modules on the command line, no need to load them from a separate loader strategy, this is much more flexible now:

Code:

./goxtool --strategy=foo.py,bar.py,baz.py

[c0inbuster]

Thanks for this !

Have you ever try to implement MA crossover strategy ?
http://www.onlinetradingconcepts.com/TechnicalAnalysis/MASimple2.html

How can I use indicator MACD, RSI... ?

[prof7bit]

Have you ever try to implement MA crossover strategy ?

I experimented with this back in my old forex days and never found something that would work really reliable when I simulated it with new (not seen before) test data. All the myriads of MA-crossover strategies (or whatever-indicator-cross-this-or-that-strategies) that looked promising on the data they were developed on failed miserably when simulating them on new test data. All of them! Every time! Even the most sophisticated stuff with neural networks and other machine learning technologies trying to predict the trend that seemed to work well on the training set failed miserably and immediately every time when confronted with new data, so I won't even try to do that anymore.

[BTC-engineer]

Have you ever try to implement MA crossover strategy ?

I experimented with this back in my old forex days and never found something that would work really reliable when I simulated it with new (not seen before) test data. All the myriads of MA-crossover strategies (or whatever-indicator-cross-this-or-that-strategies) that looked promising on the data they were developed on failed miserably when simulating them on new test data. All of them! Every time! Even the most sophisticated stuff with neural networks and other machine learning technologies trying to predict the trend that seemed to work well on the training set failed miserably and immediately every time when confronted with new data, so I won't even try to do that anymore.

It looks like you have large experience with automatic trading technologies. I have only some basic knowledge. One of the things I know about it, is that it's important to test/fine-tune your trading strategy with historical or synthesizer new data before going live. I think this is also what you are talking about your experience with various MA-crossover strategies. Is there a plan to implement such a test-mode into goxtool, so you are able to test your trading strategy?

[hugolp]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

[ErebusBat]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

STUPID QUESTION:  Nothing in goxtool.log?

[hugolp]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

STUPID QUESTION:  Nothing in goxtool.log?

I almost did not want to answer for the embarasment. Yes, it was there. Thank you.

[ErebusBat]

I almost did not want to answer for the embarasment. Yes, it was there. Thank you.

NP... we all have our moments so it is good to ask   Plus your question will help others.

[prof7bit]

Is there a plan to implement such a test-mode into goxtool, so you are able to test your trading strategy?

That would certainly be a nice thing to have. I'm not yet sure if it should be part of goxtool or a separate application that just exposes the same interface and can load the same strategy files. I cannot promise anything yet (don't have unlimited time), maybe also someone else might want to write something that could be used for this purpose.

[MtQuid]

+1 for 2.6
Debian squeeze has 2.5, 2.6 and 3.0 in repo.

I just installed goxtool on a squeeze box and I made this script to help anyone else in the same situation.

Code:

# Installing python2.7 on debian stable and running goxtool
# Tested on virgin debian squeeze install with sudo installed

####################################################
# Part 1: Install python2.7

# First get some tools
sudo apt-get install build-essential
# and some libs
sudo apt-get install libsqlite3-dev zlib1g-dev libncurses5-dev libgdbm-dev libbz2-dev libreadline5-dev libssl-dev libdb-dev
 
# Optional bits and bobs
#sudo apt-get build-dep python

# starting at home
cd ~

# Get and compile Python-2.7.4
wget wget http://python.org/ftp/python/2.7.4/Python-2.7.4.tar.bz2
tar xf Python-2.7.4.tar.bz2 
cd Python-2.7.4/
./configure --prefix=/usr/local
make

# This make takes about 5 mins on an old virtual machine

# Now you have 2 options:
# 1. install the new python
#  sudo make altinstall  
# 2. create a new debian package and install python 2.7 into debian package system
#  sudo checkinstall make altinstall

# I prefer checkinstall, though it will take longer 
sudo apt-get install checkinstall 

# There is a bug and checkinstall will fail because some directories could not be created.
# So just create them manually right now.
sudo mkdir -p /usr/local/lib/pkgconfig
sudo mkdir -p /usr/local/lib/python2.7/config

# Install and create the debian package
sudo checkinstall --pkgname=python2.7 --pkgversion=2.7.4 make altinstall

# checkinstall took about 45 minutes on my slow VM

# have a look at the new python
python2.7 --version
# and still we have the original 2.6 as the system wide default 
python --version

# get back to home
cd ~

# install a python package manager
# this will install the script 'easy_install-2.7'
wget https://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
tar xvf distribute-0.6.36.tar.gz 
cd distribute-0.6.36/
sudo python2.7 setup.py install

####################################################
# Part 2: Install goxtool

# get back to home
cd ~

# install a means to segregate python applications
sudo easy_install-2.7 virtualenv

# get latest goxtool
git clone https://github.com/prof7bit/goxtool.git

# create a new python2.7 environment for goxtool
virtualenv-2.7 --distribute goxtool

# switch to the new environment
source goxtool/bin/activate

# check it worked - you should see 2.7.4 as the version number
python --version

# get the packages required for goxtool
easy_install-2.7 pycrypto

# fire her up and get trading
python goxtool/goxtool.py --protocol=socketio --use-http

I've only just discovered this project with it's balance bot and I love it and have been running it for the last day and a half on the 7% default.
I'm going to try some different margins on other accounts with like half a coin balance and see how they each perform over time.
My first trading bot.

One issue.  During setup I balanced the account and then hit 'p' to place the buy/sell orders but I think I went too fast because the price was around 123 and the bot miscalculated my center and put in buys and sells at 80/90 something.  MtGox ignored the low sell but I did panic.

Also the graph is using a lot of screen space so could do with a toggle for switching between other information sets.  I'm a C++ programmer by trade but I'm keen to help out with this projects so I'm currently looking at the code.
Graph area could switch between log, graph, more detailed order book stats, strategy provided info.
Strategy provided would be the obvious solution to all as then you could have empty strategies that only provide information screens.  Then just have a mechanism to cycle through them.  So even the current graph would end up in a strategy.

Thanks for your work prof7bit.

[ErebusBat]

I thought I would share several of my helper scripts I have written to help with goxtool

Bot Start Script
This script is used to rotate the log file and start the bot with the appropriate connection params, as well as a log of where and when it was said to use those.

Code:

#!/bin/zsh
# https://bitcointalk.org/index.php?topic=148462.msg1854636#msg1854636
# https://gist.github.com/prof7bit/5395900
# https://bitcointalk.org/index.php?topic=148462.msg1886914#msg1886914

# "p": place 2 orders above and below price. 
# "c": cancel orders and effectively stop the bot
# "u": update own order list, depth, history, wallet and everything
# "i": display how much it is out of balance at current price (negative: must sell, positive: must buy)
# "b":balance immediately (cancel orders and then buy/sell at market to rebalance)

# Backup log file
if [[ -f goxtool.log ]]; then 
  mv goxtool.log goxtool_$(date '+%Y%m%d-%H%M%S').log
fi
# Per https://bitcointalk.org/index.php?topic=181584.msg1923260#msg1923260
./goxtool.py --protocol=websocket --strategy=_balancer.py

botlog.sh - View/watch bot's log
This filters out the 'noise' and lets me see just the output I want from the log.
As a side effect of the above log rotation you will need to restart this every time you restart the tool with the script above

Code:

#!/bin/zsh
tail -n500000 -f ~/goxtool/goxtool.log | egrep "(Strategy|got ack|got|have)"

startbot.sh - Enforce Screen Session
Forces that you are in a screen session so you don't start the bot and then logout and think it is running when it is not 

Code:

#!/bin/zsh
if [[ -z $STY ]] then
	echo "Not in screen! Run ~/botwindow.sh"
else
	cd ~/goxtool
	./bot_balancer.sh
fi

botwindow.sh
Starts (or re-connects to) screen with the same params every time

Code:

#!/bin/sh
screen -D -R -A -S bot

[supert]

Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

[ErebusBat]

Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

You can add (or change) the following in your ini file and then the title will not update

Code:

set_xterm_title = False

[supert]

You can add (or change) the following in your ini file and then the title will not update

Code:

set_xterm_title = False

thanks.

Is there a way to moderate what is logged / verbosity (I guess I can comment out the code)
-- would also be a nice .ini option

[hugolp]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

Im hitting this problem again and now I just get a message saying:

Code:

2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

The program keeps running but without running the strategy obviously. And yes, the file mybot.py exists and is the right one. No other indication as to what might be happening. Any idea of what I could do to know where I got the code wrong?

[ErebusBat]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

Im hitting this problem again and now I just get a message saying:

Code:

2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

The program keeps running but without running the strategy obviously. And yes, the file mybot.py exists and is the right one. No other indication as to what might be happening. Any idea of what I could do to know where I got the code wrong?

The code that handles that error is ~953:

Code:

    def reload(self):
        """reload and re-initialize the strategy module"""
        self.unload()
        for name in self.strategy_name_list:
            name = name.replace(".py", "").strip()

            try:
                strategy_module = __import__(name)
                try:
                    reload(strategy_module)
                    strategy_object = strategy_module.Strategy(self.gox)
                    self.strategy_object_list.append(strategy_object)

                # pylint: disable=W0703
                except Exception:
                    self.gox.debug(traceback.format_exc())

            except ImportError:
                self.gox.debug("### could not import %s.py" % name)

I would change that log message to output more information if I were you... by the looks of it I would guess (not being a python person at all) that your class name doesn't match your file name.

[prof7bit]

Code:

2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

Make sure the current working directory is the goxtool directory and the mybot.py is in the same directory too.

[smooth]

Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

I'm very surprised by a load of ~1 for this on a rpi.  A (possible) memory leak surprises me less.  I'll test this myself soon, but I think something is likely going wrong in your configuration.

[hugolp]

The code that handles that error is ~953:

Code:

    def reload(self):
        """reload and re-initialize the strategy module"""
        self.unload()
        for name in self.strategy_name_list:
            name = name.replace(".py", "").strip()

            try:
                strategy_module = __import__(name)
                try:
                    reload(strategy_module)
                    strategy_object = strategy_module.Strategy(self.gox)
                    self.strategy_object_list.append(strategy_object)

                # pylint: disable=W0703
                except Exception:
                    self.gox.debug(traceback.format_exc())

            except ImportError:
                self.gox.debug("### could not import %s.py" % name)

I would change that log message to output more information if I were you... by the looks of it I would guess (not being a python person at all) that your class name doesn't match your file name.

Thanks, that helped. I had an typo in one of my import statements so the module refused to load. prof7bit, it would be better if the log signaled the particular error otherwise you are in the dark making it hard to debug. Basically this change:

Code:

            except ImportError as e:
                self.gox.debug("### could not import %s.py, reason: %s" % (name,e))

[ghostshirt]

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

And as i have mentioned before already: make sure you ALWAYS have an up to date version of goxtool. git pull (and restart if there were updates) at least once a day, I'm still fixing things in goxtool itself occasionally that make it more reliable.

[prof7bit]

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

The balancer does not need to be re-initialized after you update/restart goxtool, it does not have any state, it can trade from the information about currently open orders and account balance alone, it will continue where it left automatically.

Are you using the newest version of the balancer that I posted on the separate thread that I started for the balancer? Questions about it should go to this thread.

Also please note (if you haven't seen it already) that since a few days (since a server upgrade at mtgox took place) the most reliable connection to mtgox is achieved with these command line options:
--protool=websocet --use-http
I'm mentioning this again because it is very important.

[ghostshirt]

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

The balancer does not need to be re-initialized after you update/restart goxtool, it does not have any state, it can trade from the information about currently open orders and account balance alone, it will continue where it left automatically.

Are you using the newest version of the balancer that I posted on the separate thread that I started for the balancer? Questions about it should go to this thread.

Also please note (if you haven't seen it already) that since a few days (since a server upgrade at mtgox took place) the most reliable connection to mtgox is achieved with these command line options:
--protool=websocet --use-http
I'm mentioning this again because it is very important.

Sorry about the wrong thread.
I used the other protocol unfortunately. But the _balancer.py is the right one. I left my desktop running at office without a remote access.
Strangely, I seem to be on profit side in terms of BTC and a very small loss of $ after the crash we saw a few hours ago. newbie luck probably

[hugolp]

Excuse if this is offtopic, but I want my goxtool bot to notify my Android phone, and even sound an alarm (in case Im sleeping) when certain conditions are met. Whats the best way of achieving this? I know how to program for Android (but would be happy if I can avoid having to create my own program just for this) and own Tasker. One option seems to use Google Cloud Messaging, anyone has any other suggestion?

[c0inbuster]

@hugolp
you should have a look at PushOver API https://pushover.net/, it works with both Android and iOS.
You just have to install PushOver app on your Android device and use their HTTP API
https://play.google.com/store/apps/details?id=net.superblock.pushover&hl=fr
It's a paid apps but it worth it
you can find some inspiration here
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/pushover_notifier.py
Have a look at priority
https://pushover.net/api#priority

@prof7bit
I'm looking for a way to run goxtool at startup (connected to my MtGox account) in a GNU Screen
I tried

Code:

$ python goxtool --add-secret

I set API and SECRET key
but I'm asked for a password... I can't set a blank password
What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret: 

I would like to bypass this step

[prof7bit]

What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret: 

I would like to bypass this step

You are not the first one to ask me and I am going to implement one of these two options (not sure which one, probably the second one):

(1) A command line option (with a huge warning) to pass the pasword on the command line. It is insecure because the password will end up in the ~/.bash_history if you do this.

(2) The other alternative would be to make a separate .ini setting for the unencrypted secet that is automatically used (and the password question bypassed) when this setting is found in the ini.

[K1773R]

What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret: 

I would like to bypass this step

You are not the first one to ask me and I am going to implement one of these two options (not sure which one, probably the second one):

(1) A command line option (with a huge warning) to pass the pasword on the command line. It is insecure because the password will end up in the ~/.bash_history if you do this.

(2) The other alternative would be to make a separate .ini setting for the unencrypted secet that is automatically used (and the password question bypassed) when this setting is found in the ini.

how about reading from STDIN?

Code:

python goxtool.py <<< YOURPASSWORDHERE

[prof7bit]

how about reading from STDIN?

Code:

python goxtool.py <<< YOURPASSWORDHERE

Doesn't work for me. I'm not a bash expert, so I'm not sure why it won't work (maybe the reason is how readline internally operates or the python getpass module). Also this would still leave the password in the .bash_history even if it worked.

[prof7bit]

I have added a command line option --password that can be used when starting it from within a script (don't use it on the command line unless you understand the implications)

Code:

./goxtool.py --password=YOURPASSWORD

I have also added two new ini settings (they will be added to the .ini with default values after you start it the first time after update):

Code:

[goxtool]
orderbook_group = 0
orderbook_sum_total = False

The dafault values will display the orderbook exactly like it was in earlier versions of goxtool all the time already, so if you leave them at defaults then nothing will change. orderbook_group will have the same effect as it is used on the clarkmoody website, if you set it for example to 0.1 it will bin the price levels into $0.10 intervals (it won't display empty bins). 0 means no grouping at all, display every price level separately. orderbook_sum_total = True would sum up the total amount of BTC that needs to be bought or sold to reach and break that level instead of printing only the amount at that particular level like it was in earlier versions.

Please git pull to update.

[supert]

I'm very surprised by a load of ~1 for this on a rpi.  A (possible) memory leak surprises me less.  I'll test this myself soon, but I think something is likely going wrong in your configuration.

having run it a bit longer it seems about 0.5 load most of the time and 55%-75% memory (256MB rpi).
The memory usage seems stable so it doesn't seem there is a (serious) leak.

[ErebusBat]

I have added a command line option --password that can be used when starting it from within a script (don't use it on the command line unless you understand the implications)

Code:

./goxtool.py --password=YOURPASSWORD

I would HIGHLY recommend that you store your password on disk (with appropriate file permissions) then you can launch the bot with this command line, keeping the password out of your history file:

Code:

./goxtool.py --password=`cat ~/.mygoxtoolpassword`

Note that those are backticks (same key as the tilde, left of the 1 on most keyboards)  NOT single quotes!

[prof7bit]

Here is something not really serious but someone might still find it useful:

_speaker.py

Code:

"""
_speaker.py - A simple fun bot (no real usage) for goxtool.

This bot will read big market buys and sells loudly, a female voice will read
all buys and a male voice will read all sells that are above VOL_MIN. A 3rd
voice will read own orders that have been filled. It will attempt to sum up
partial trades by waiting until either a new tick message arrives or a trade
in opposite direction.

This will only work if espeak is installed.
On Ubuntu this is apt-get install espeak
"""
import os
import goxapi
import strategy

# minimum trade volume, market orders below that
# volume will be ignored. value is in BTC.
VOL_MIN = 10

# see the manual of espeak to understand how to tweak the voices.
VOICE_BUY   = "-ven+f5"
VOICE_SELL  = "-ven"
VOICE_OWN   = "-ven+f5 -p10 -s150"

# if mbrola us voices are installed these also work
# VOICE_BUY   = "-vmb-us1 -s150 -p70"
# VOICE_SELL  = "-vmb-us2 -s150"
# VOICE_OWN   = "-ven-us1 -s130 -p0"

def say(txt, voice="-ven"):
    """say the text (needs espeak installed)"""
    os.system("espeak %s '%s' 2>/dev/null 1>/dev/null &" % (voice, txt))

class Strategy(strategy.Strategy):
    """text to speech bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.vol = 0
        self.action = 0

    def slot_trade(self, gox, (date, price, volume, typ, own)):
        prc = goxapi.int2float(price, gox.currency)
        vol = goxapi.int2float(volume, "BTC")

        if own:
            say("Order filled: %0.2f at %0.2f." % (vol, prc), VOICE_OWN)
        else:
            action = {"bid": "buy", "ask": "sell"}[typ]
            if action != self.action:
                self.say_trade()
            self.action = action
            self.vol += vol

    def slot_tick(self, gox, (bid, ask)):
        self.say_trade()

    def say_trade(self):
        """read the last trade (if any). a female voice will read
        the buy orders, a male voice the sell orders"""
        if self.vol >= VOL_MIN:
            if self.action == "buy":
                voice = VOICE_BUY
            else:
                voice = VOICE_SELL
            say("%s %0.0f." % (self.action, self.vol), voice)
        self.vol = 0

[c0inbuster]

I'm not very interested by you "speaker" strategy ... for now... but maybe one day it could be useful...

I tried to play with a Python wrapper for TA-Lib
http://ta-lib.org/
http://mrjbq7.github.io/ta-lib/
this wrapper is total...
you just need to install cython, TA-Lib, and this wrapper

I just did a test strategy for now

Code:

"""
trading robot breadboard
"""

import goxapi
import datetime
import numpy as np
import talib
#import pandas as pd

class Strategy(goxapi.BaseObject):
    # pylint: disable=C0111,W0613,R0201

    def __init__(self, gox):
        goxapi.BaseObject.__init__(self)
        self.signal_debug.connect(gox.signal_debug)
        gox.signal_keypress.connect(self.slot_keypress)
        gox.signal_strategy_unload.connect(self.slot_before_unload)
        gox.signal_ticker.connect(self.slot_tick)
        gox.signal_depth.connect(self.slot_depth)
        gox.signal_trade.connect(self.slot_trade)
        gox.signal_userorder.connect(self.slot_userorder)
        gox.orderbook.signal_owns_changed.connect(self.slot_owns_changed)
        gox.signal_wallet.connect(self.slot_wallet_changed)
        
        gox.signal_orderlag.connect(self.slot_orderlag)
        
        self.gox = gox
        self.name = "%s.%s" % \
            (self.__class__.__module__, self.__class__.__name__)
        self.debug("%s loaded" % self.name)
        
        self.is_lagging = True

    def __del__(self):
        self.debug("%s unloaded" % self.name)

    def slot_before_unload(self, _sender, _data):
        self.debug("%s before unload" % self.name)

    def slot_keypress(self, gox, (key)):
        if key == ord("t"):
            #self.debug("Test")
            len_hist = gox.history.length()
            self.debug("Test history with {N} candles".format(N=len_hist))
            #z = np.zeros(len)
            #self.df = pd.DataFrame(z)
            
            #opn = np.array(gox.history.candles)
            #iterable = (gox.history.candles[i].tim for i in range(len_hist))
            #a_tim = np.fromiter(iterable, ???)

            rng = range(len_hist)
            iterable = (gox.history.candles[i].opn for i in rng)
            a_opn = np.fromiter(iterable, np.float)
            
            iterable = (gox.history.candles[i].hig for i in rng)
            a_hig = np.fromiter(iterable, np.float)
            
            iterable = (gox.history.candles[i].low for i in rng)
            a_low = np.fromiter(iterable, np.float)
            
            iterable = (gox.history.candles[i].cls for i in rng)
            a_cls = np.fromiter(iterable, np.float)

            iterable = (gox.history.candles[i].vol for i in rng)
            a_vol = np.fromiter(iterable, np.float)
            
            a_sma = talib.SMA(a_opn)

            """
            self.tim = tim
            self.opn = opn
            self.hig = hig
            self.low = low
            self.cls = cls
            self.vol = vol
            """
            #self.debug(gox.history.candles[len-1].cls)
            #self.debug(self.df)
            
            self.debug(a_opn)
            #self.debug(a_sma)
            
        else:
            self.debug("someone pressed the %s key" % chr(key))

    def slot_tick(self, gox, (bid, ask)):
        pass

    def slot_depth(self, gox, (typ, price, volume, total_volume)):
        pass

    def slot_trade(self, gox, (date, price, volume, typ, own)):
        """a trade message has been received. Note that this might come
        before the orderbook.owns list has been updated, don't rely on the
        own orders and wallet already having been updated when this fires."""
        pass

    def slot_userorder(self, gox, (price, volume, typ, oid, status)):
        """this comes directly from the API and owns list might not yet be
        updated, if you need the new owns list then use slot_owns_changed"""
        pass

    def slot_owns_changed(self, orderbook, _dummy):
        """this comes *after* userorder and orderbook.owns is updated already"""
        pass

    def slot_wallet_changed(self, gox, _dummy):
        """this comes after the wallet has been updated"""
        pass
        
    def slot_orderlag(self, dummy_sender, (usec, text)):
        """slot for order_lag messages"""
        lag = datetime.timedelta(microseconds=usec)
        lag_max = datetime.timedelta(seconds=60)
        #self.debug(usec)
        if lag>lag_max:
            self.debug("MtGox is lagging {lag}".format(lag=lag))
            self.is_lagging = True
        else:
            self.is_lagging = False

I was very surprised to see that candles are stored in a list...
why not using a Numpy Array ?

It will much efficient to implement indicators on top of a Numpy array than on top
of a Python list.
 
You should add a length for array as Numpy arrays have a fixed size (contrary to python list).

I was also surprised to see output for price such as

Code:

2013-04-26 18:02:23,152:DEBUG:test_strategy.Strategy:[ 13399024.  13590000.  13680000.  13610000.  13649999.  13680000.

In my mind (but I ever said something similar) in strategy" I should have "real" values.

I think that a "layer" is actually missing but I don't know how to implement it.

[prof7bit]

I'm not very interested by you "speaker" strategy

maybe others are - for various other reasons.

I was very surprised to see that candles are stored in a list...
why not using a Numpy Array ?

The entire history was initially only a quick hack to be able to plot the recent 24 hours prices into the chart view. Ideally i would eventually implement somethiing more sophisticated that would provide more history, more different timeframes and then maybe also with numpy arrays. I'm waiting until magicaltux implements the API to query trade history of several different (most commonly used) timeframes in OHLCV format. Until then it remains only a quick and dirty hack to be able to plot the price chart.

Until then your bot needs to construct its Numpy arrays itself (maybe even from a local database or some other web service) and keep it updated with the trade signals. You could write a class that does all this, connects to needed gox signals and has a well architected and intuitive interface for usage by bots and also integrates TA-lib and then instantiate this entire thing from your bot during __init__. Once it is working I could look at it for some inspiration of how to design my own new history class once mtgox has the history api ready.

I was also surprised to see output for price such as

Code:

2013-04-26 18:02:23,152:DEBUG:test_strategy.Strategy:[ 13399024.  13590000.  13680000.  13610000.  13649999.  13680000.

In my mind (but I ever said something similar) in strategy" I should have "real" values.

The format is the one used by mtgox api, all currency values are integers. BTC is an integer of satoshi (1e8), JPY is 1e3, USD and all others are 1e5.

[c0inbuster]

Maybe price should be stored in Decimal http://docs.python.org/2/library/decimal.html

[Ban Curtain]

Id like to create hierarchy of strategies. The first strategy observes for a complex event and then emits signal which should be handled by slot of the second strategy. How is it better to implement?

As far as I understand to connect to a signal of a dummy emitter I must have access to instance of emitter, but while initializing strategies I have only instance of gox. Is it possible to get access to another strategies' instances?

[prof7bit]

Is it possible to get access to another strategies' instances?

Not in the current implementation, not when passing a list of strategies to the command line. But what you could do is to explicitly load a strategy from within another strategy. This "master strategy" could then load "slave strategies" in an exactly defined order and pass them references to itself or to each other in their constructors so they can refer to each other and then you could have your strategies communicate with custom signals or directly calling each other.

Edit: Another (maybe even better) solution would be that you pass the strategies to the command line in the right order so that the first strategy would be the one that wants to emit signals would simply create a new signal in the gox instance and all the other strategies that load after it would check if a member of that name exists in the gox instance and if it exists connect a slot to it.

Emitting Strategy:
In the constructor __init__ of the emitting strategy (that one should be loaded first) do the following to create the new the "signal_foo":

Code:

        gox.signal_foo = goxapi.Signal()

and to emit the signal you do something like this:

Code:

        self.gox.signal_foo(self, (42, 23))

Receiving Strategy:
in the constructor __init__ of the receiving strategies (these must appear *after* the above one on the command line) you add the following to detect the existence of the signal and connect a slot to it to it:

Code:

        if hasattr(gox, 'signal_foo'):
            gox.signal_foo.connect(self.slot_foo)

and a slot:

Code:

    def slot_foo(self, sender, data):
        """slot for custom foo signal"""
        (a, b) = data
        self.debug("foo signal detected: %g %g" %  (a, b))

That should do the trick, I just tested it and it works very well. If the emitter was not loaded then the slot will just never be called but it won't crash (you can add an else to the if to print a debug warning if that happens)

[Ban Curtain]

Is it possible to get access to another strategies' instances?

Not in the current implementation, not when passing a list of strategies to the command line. But what you could do is to explicitly load a strategy from within another strategy. This "master strategy" could then load "slave strategies" in an exactly defined order and pass them references to itself or to each other in their constructors so they can refer to each other and then you could have your strategies communicate with custom signals or directly calling each other.

Edit: Another (maybe even better) solution would be that you pass the strategies to the command line in the right order so that the first strategy would be the one that wants to emit signals would simply create a new signal in the gox instance and all the other strategies that load after it would check if a member of that name exists in the gox instance and if it exists connect a slot to it.

Emitting Strategy:
In the constructor __init__ of the emitting strategy (that one should be loaded first) do the following to create the new the "signal_foo":

Code:

        gox.signal_foo = goxapi.Signal()

and to emit the signal you do something like this:

Code:

        self.gox.signal_foo(self, (42, 23))

Receiving Strategy:
in the constructor __init__ of the receiving strategies (these must appear *after* the above one on the command line) you add the following to detect the existence of the signal and connect a slot to it to it:

Code:

        if hasattr(gox, 'signal_foo'):
            gox.signal_foo.connect(self.slot_foo)

and a slot:

Code:

    def slot_foo(self, sender, data):
        """slot for custom foo signal"""
        (a, b) = data
        self.debug("foo signal detected: %g %g" %  (a, b))

That should do the trick, I just tested it and it works very well. If the emitter was not loaded then the slot will just never be called but it won't crash (you can add an else to the if to print a debug warning if that happens)

Thank you, I gonna try the solution you recommend.

[prof7bit]

goxtool.py can now display a depth chart. Also it has grouping in the orderbook and optionally adding up the totals:

git pull

there are a lot of new keyboad shortcuts to control all this,
I have added them to the manual: http://prof7bit.github.io/goxtool/

Screenshot:

[ErebusBat]

Thanks prof7bit!

Question:  what do the yellow Os indicate?

[prof7bit]

Thanks prof7bit!

Question:  what do the yellow Os indicate?

They are your own orders. If you zoom out the orderbook (- key) you will see them there too

[ErebusBat]

They are your own orders. If you zoom out the orderbook (- key) you will see them there too

Oh, awesome!  Thanks!

[MtQuid]

I'm not very interested by you "speaker" strategy ... for now... but maybe one day it could be useful...

I tried to play with a Python wrapper for TA-Lib
http://ta-lib.org/
http://mrjbq7.github.io/ta-lib/
this wrapper is total...
you just need to install cython, TA-Lib, and this wrapper

I just tried this out and wanted to plot the results so I dumped the data to some files and used Highstock.js to render the view.

Code:

import goxapi
from goxapi import OHLCV
import datetime
import numpy as np
import talib
import json

class OHLCV_Encoder(json.JSONEncoder):
    """JSONEncoder for class OHLCV()"""
    def default(self, obj):
        if isinstance(obj, OHLCV):
            return [obj.tim, obj.opn, obj.hig, obj.low, obj.cls, obj.vol]
        else:
            return json.JSONEncoder.default(self, obj)

class Strategy(goxapi.BaseObject):
    ........
    ........

    def slot_keypress(self, gox, (key)):
        if key == ord("t"):
            """testing talib"""        
            candles = []

            for c in reversed(self.gox.history.candles):
                candles.append(
                    OHLCV(
                        # fix time for HighStock.js JSON import
                        c.tim * 1000,
                        # adjust values to be human readable
                        goxapi.int2float(c.opn, self.gox.currency),
                        goxapi.int2float(c.hig, self.gox.currency),
                        goxapi.int2float(c.low, self.gox.currency),
                        goxapi.int2float(c.cls, self.gox.currency),
                        goxapi.int2float(c.vol, "BTC")
                    )
                )  

            self.debug("Test history with %d candles" % len(candles))                

            rng = range(len(candles))
            iterable = (candles[i].cls for i in rng)
            a_cls = np.fromiter(iterable, np.float)
                                  
            iterable = (candles[i].tim for i in rng)
            a_tim = np.fromiter(iterable, np.int64)


            a_sma = talib.SMA(a_cls, 10)
            a_wma = talib.WMA(a_cls, 25)

            # create json compatible with HighStock.js
            with open("talib_ohlcv.json", 'w') as outfile:                    
                json.dump(candles, outfile, cls = OHLCV_Encoder)
            with open("talib_sma.json", 'w') as outfile:
                # first 10 elements contain Nan
                json.dump(np.dstack((a_tim[10:], a_sma[10:])).tolist()[0], outfile)
            with open("talib_wma.json", 'w') as outfile:
                # first 25 elements contain Nan
                json.dump(np.dstack((a_tim[25:], a_wma[25:])).tolist()[0], outfile)

with

Code:

<!DOCTYPE HTML>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>BTC Highstock - Goxtool/TA-Lib</title>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script>
<script type="text/javascript" src="http://code.highcharts.com/stock/highstock.js"></script>
<script type="text/javascript" src="http://code.highcharts.com/modules/exporting.js"></script>
<script type="text/javascript">
$(function () {
    var seriesOptions = [],
        ohlc = [],
        volume = []
        yAxisOptions = [],
        seriesCounter = 0,
        names = ['OHLCV', 'SMA', 'WMA'],
        colors = Highcharts.getOptions().colors,
        groupingUnits = [
            ['week', [1]],
            ['month', [1, 2, 3, 4, 6]]
        ];

    $.each(names, function (i, name) {
        $.getJSON('/talib_' + name.toLowerCase() + '.json', function (data) {
            console.log("Loading " + name);
            if (name == 'OHLCV') {
                // split the data set into ohlc and volume		        			
                var dataLength = data.length;
                for (i = 0; i < dataLength; i++) {
                    ohlc.push([
                        data[i][0], // the date
                        data[i][1], // open
                        data[i][2], // high
                        data[i][3], // low
                        data[i][4] // close
                    ]);

                    volume.push([
                        data[i][0], // the date
                        data[i][5] // the volume
                    ])
                }
            } else {
                seriesOptions[i] = {
                    name: name,
                    data: data
                };
            }

            // As we're loading the data asynchronously, we don't know what order it will arrive. So
            // we keep a counter and create the chart when all the data is loaded.
            seriesCounter++;

            if (seriesCounter == names.length) {
                createChart();
            }
        });
    });

    // create the chart when all data is loaded

    function createChart() {
        console.log("Creating chart");

        $('#container').highcharts('StockChart', {
            credits: {
                enabled: false
            },
            chart: {
                renderTo: container,
                backgroundColor: {
                    linearGradient: [0, 0, 0, 500],
                    stops: [
                        [0, 'rgb(255, 255, 255)'],
                        [1, 'rgb(215, 229, 240)']
                    ]
                }
            },
            rangeSelector: {
                buttons: [{
                        type: 'day',
                        count: 1,
                        text: '1d'
                    }, {
                        type: 'week',
                        count: 1,
                        text: '1w'
                    }, {
                        type: 'month',
                        count: 1,
                        text: '1m'
                    }
                ],
                selected: 0,
                inputEnabled: false
            },

            title: {
                text: 'BTC Historical - Goxtool/TA-Lib'
            },

            yAxis: [{
                    title: {
                        text: 'OHLC'
                    },
                    type: 'logarithmic',
                    height: 200,
                    lineWidth: 2
                }, {
                    title: {
                        text: 'Volume'
                    },
                    type: 'linear',
                    top: 300,
                    height: 100,
                    offset: 0,
                    lineWidth: 2
                }
            ],

            plotOptions: {
                candlestick: {
                    color: '#DD2F29',
                    upColor: '#87DA3B',
                    lineColor: '#666'
                }
            },

            tooltip: {
                pointFormat: '<span style="color:{series.color}">{series.name}</span>: <b>{point.y}</b><br/>',
                valueDecimals: 2
            },

            series: [{
                    type: 'candlestick',
                    name: 'BTC',
                    data: ohlc,
                    dataGrouping: {
                        units: groupingUnits
                    }
                }, {
                    type: 'column',
                    name: 'Volume',
                    data: volume,
                    yAxis: 1,
                    dataGrouping: {
                        units: groupingUnits
                    }
                },
                seriesOptions[1], seriesOptions[2]
            ]
        });

    }

});
</script>
</head>
<body>
<div id="container" style="height: 500px; min-width: 600px"></div>
</body>
</html>

Serve index.html up with

Code:

python -m SimpleHTTPServer 8080

EDIT:  Using close instead of open for EMAs and added python server example

[ErebusBat]

@MtQuid

How far back does that 'historical' day pull from?  Since GoxTool started?

[MtQuid]

@MtQuid

How far back does that 'historical' day pull from?  Since GoxTool started?

It's using the same history candles that prof7bit uses for the trade graph.  So it goes back a day in 15 minute blocks.
Though you can change the history_timeframe value in goxtool.ini to change the block size.

[c0inbuster]

I had some problems to see HTML file
so I contacted MtQuid using PM
I post here for other members

in fact you have to run a webserver (and not directly open HTML file with a browser)

python -m SimpleHTTPServer 8080

Open index.htm in your web browser using this url http://127.0.0.1:8080/index.hml


You will get a beautiful chart



It will be nice to also make orderbook depth chart with a JS lib

Maybe adding this feature to goxgui (goxtool GUI) will be fine
https://bitcointalk.org/index.php?topic=176489.80

Thnaks MtQuid for your code

[MtQuid]

Yep they do look good.  I robbed some of the colour styling straight from MtGox.com

I only wanted the graph output to make sure the ta-lib functions were returning the correct values so I could use them to control my bot.  I don't think this functionality needs to be part of the main goxtool branch but a good strategy plugin using ta-lib would be handy.  Like providing values and firing signals that can be caught.

And because the data sets are small (1 days worth of trade) you could write the whole index.html file to include the data and then there would be no need for the .json files or a web server.  Just pops out a stand alone html page every now and again if that's what you require.

I've also moved my function so it automatically gets called when new history arrives.

Code:

    def slot_history_changed(self, history, _dummy):
        """History has changed so recalculate EMAs"""

Now to try and find the best usage for the ta-lib functions and make some coins.. 

[hugolp]

I only wanted the graph output to make sure the ta-lib functions were returning the correct values so I could use them to control my bot.

And how are the results?.

I manually created a ema function, basically following the formula you see everywhere in the Internet (f.e. http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:moving_averages) but Im getting results quite different from bitcoincharts.com, specially for the hourly ema's but almost identical for the 1minute ema's. Because of this my guess was: the formula calculates the sma with the first periods and then starts adding the data as a exponential. I assumed this means that over time with enough data it would converge and that is why the 1minute ema's were almost identical while the hourly ema's were not, since Im getting data of two days and there is only 48 hours while there is 48 * 60 minutes. Does ta-lib get better results with data from one day only?

I don't think this functionality needs to be part of the main goxtool branch but a good strategy plugin using ta-lib would be handy.  Like providing values and firing signals that can be caught.

I actually would love to have the the web part in goxtool, maybe just as a plugin or something.

EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

[ErebusBat]

MtQuid...
Would you be willing to share a basic demo of the Ta-lib integration?  I do not live in python so it would be very helpful for me, and I am sure others. 

[MtQuid]

MtQuid...
Would you be willing to share a basic demo of the Ta-lib integration?  I do not live in python so it would be very helpful for me, and I am sure others. 

Ok.

We will use TA-Lib for bython from these lovely chaps
http://mrjbq7.github.io/ta-lib/index.html

Assuming you are working on a debian system you will need to do the following to install ta-lib for python
If on windows or mac then check http://mrjbq7.github.io/ta-lib/install.html

Code:

sudo apt-get install python-dev 
sudo easy_install cython 

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
tar -xvf ta-lib-0.4.0-src.tar.gz
cd ta-lib
./configure --prefix=/usr
make

#can use make install here if you want
sudo checkinstall

sudo easy_install TA-Lib

Once you have TA-Lib installed this is your strategy

Code:

"""
_talib.py

This example strategy will produce some HighStock.js compatible JSON files
"""

import numpy as np
import talib
import datetime
import json
import strategy
import goxapi
from goxapi import OHLCV

class OHLCV_Encoder(json.JSONEncoder):
    """JSONEncoder for class OHLCV()"""
    def default(self, obj):
        if isinstance(obj, OHLCV):
            return [obj.tim, obj.opn, obj.hig, obj.low, obj.cls, obj.vol]
        else:
            return json.JSONEncoder.default(self, obj)

class Strategy(strategy.Strategy):
    """a protfolio rebalancing bot"""
    def __init__(self, gox):        
        strategy.Strategy.__init__(self, gox)

    def slot_history_changed(self, history, _dummy):
        """History has changed so recalculate EMAs"""
        candles = []

        # read them all - don't wory about the history parameter
        for c in reversed(self.gox.history.candles):
            candles.append(
                OHLCV(
                    # fix time for HighStock.js JSON import
                    c.tim * 1000,
                    # adjust values to be human readable
                    goxapi.int2float(c.opn, self.gox.currency),
                    goxapi.int2float(c.hig, self.gox.currency),
                    goxapi.int2float(c.low, self.gox.currency),
                    goxapi.int2float(c.cls, self.gox.currency),
                    goxapi.int2float(c.vol, "BTC")
                )
            )  

        self.debug("New EMAs from history with %d candles" % len(candles))                

        rng = range(len(candles))
        iterable = (candles[i].opn for i in rng)
        a_opn = np.fromiter(iterable, np.float)

        iterable = (candles[i].hig for i in rng)
        a_hig = np.fromiter(iterable, np.float)
        
        iterable = (candles[i].low for i in rng)
        a_low = np.fromiter(iterable, np.float)
        
        iterable = (candles[i].cls for i in rng)
        a_cls = np.fromiter(iterable, np.float)

        iterable = (candles[i].vol for i in rng)
        a_vol = np.fromiter(iterable, np.float)

        iterable = (candles[i].tim for i in rng)
        a_tim = np.fromiter(iterable, np.int64)


        a_sma = talib.SMA(a_cls, 10)
        a_wma = talib.WMA(a_cls, 25)
        a_chaikin = talib.AD(a_hig, a_low, a_cls, a_vol)
        a_cdlCounterAttack = talib.CDLCOUNTERATTACK(a_opn, a_hig, a_low, a_cls)

        # create json compatible with HighStock.js
        with open("talib_ohlcv.json", 'w') as outfile:                    
            json.dump(candles, outfile, cls = OHLCV_Encoder)
        with open("talib_sma.json", 'w') as outfile:
            # first 10 elements contain Nan
            json.dump(np.dstack((a_tim[10:], a_sma[10:])).tolist()[0], outfile)
        with open("talib_wma.json", 'w') as outfile:
            # first 25 elements contain Nan
            json.dump(np.dstack((a_tim[25:], a_wma[25:])).tolist()[0], outfile)
        with open("talib_chaikin.json", 'w') as outfile:
            json.dump(np.dstack((a_tim, a_chaikin)).tolist()[0], outfile)
        with open("talib_cdlCounterAttack.json", 'w') as outfile:
            json.dump(np.dstack((a_tim, a_cdlCounterAttack)).tolist()[0], outfile)

        # Current price in relation to EMA
        self.debug("SMA = %f" % a_sma[-1])
        self.debug("WMA = %f" % a_wma[-1])
        self.debug("CLS = %f" % a_cls[-1])

Run goxtool like so

Code:

python ./goxtool.py --protocol=websocket --strategy=_talib.py

Create index.html like this

Code:

<!DOCTYPE HTML>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>BTC Highstock - Goxtool/TA-Lib</title>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script>
<script type="text/javascript" src="http://code.highcharts.com/stock/highstock.js"></script>
<script type="text/javascript" src="http://code.highcharts.com/modules/exporting.js"></script>
<script type="text/javascript">
$(function () {
    var seriesOptions = [],
        ohlc = [],
        volume = []
        yAxisOptions = [],
        seriesCounter = 0,
        names = ['OHLCV', 'SMA', 'WMA', 'chaikin'],
        colors = Highcharts.getOptions().colors,
        groupingUnits = [
            ['week', [1]],
            ['month', [1, 2, 3, 4, 6]]
        ];

    $.each(names, function (i, name) {
        $.getJSON('/talib_' + name.toLowerCase() + '.json', function (data) {
            console.log("Loading " + name);
            if (name == 'OHLCV') {
                // split the data set into ohlc and volume		        			
                var dataLength = data.length;
                for (i = 0; i < dataLength; i++) {
                    ohlc.push([
                        data[i][0], // the date
                        data[i][1], // open
                        data[i][2], // high
                        data[i][3], // low
                        data[i][4] // close
                    ]);

                    volume.push([
                        data[i][0], // the date
                        data[i][5] // the volume
                    ])
                }
            } else if (name == 'chaikin') {
                seriesOptions[i] = {

                    yAxis: 2,
                    name: name,
                    data: data
                };

            } else {
                seriesOptions[i] = {
                    name: name,
                    data: data
                };
            }

            // As we're loading the data asynchronously, we don't know what order it will arrive. So
            // we keep a counter and create the chart when all the data is loaded.
            seriesCounter++;

            if (seriesCounter == names.length) {
                createChart();
            }
        });
    });

    // create the chart when all data is loaded

    function createChart() {
        console.log("Creating chart");

        $('#container').highcharts('StockChart', {
            credits: {
                enabled: false
            },
            chart: {
                renderTo: container,
                backgroundColor: {
                    linearGradient: [0, 0, 0, 500],
                    stops: [
                        [0, 'rgb(255, 255, 255)'],
                        [1, 'rgb(215, 229, 240)']
                    ]
                }
            },
            rangeSelector: {
                buttons: [{
                        type: 'day',
                        count: 1,
                        text: '1d'
                    }, {
                        type: 'week',
                        count: 1,
                        text: '1w'
                    }, {
                        type: 'month',
                        count: 1,
                        text: '1m'
                    }
                ],
                selected: 0,
                inputEnabled: false
            },

            title: {
                text: 'BTC Historical - Goxtool/TA-Lib'
            },

            yAxis: [{
                    title: {
                        text: 'OHLC'
                    },
                    type: 'logarithmic',
                    height: 200,
                    lineWidth: 2
                }, {
                    title: {
                        text: 'Volume'
                    },
                    type: 'linear',
                    top: 300,
                    height: 100,
                    offset: 0,
                    lineWidth: 2
                }, {
                    title: {
                        text: 'Chakin'
                    },
                    type: 'linear',
                    top: 450,
                    height: 100,
                    offset: 0,
                    lineWidth: 2
                }
            ],

            plotOptions: {
                candlestick: {
                    color: '#DD2F29',
                    upColor: '#87DA3B',
                    lineColor: '#666'
                }
            },

            tooltip: {
                pointFormat: '<span style="color:{series.color}">{series.name}</span>: <b>{point.y}</b><br/>',
                valueDecimals: 2
            },

            series: [{
                    type: 'candlestick',
                    name: 'BTC',
                    data: ohlc,
                    dataGrouping: {
                        units: groupingUnits
                    }
                }, {
                    type: 'column',
                    name: 'Volume',
                    data: volume,
                    yAxis: 1,
                    dataGrouping: {
                        units: groupingUnits
                    }
                },
                seriesOptions[1], seriesOptions[2], seriesOptions[3]
            ]
        });

    }

});
</script>
</head>
<body>
<div id="container" style="height: 1000px; min-width: 600px"></div>
</body>
</html>

And run a simple web server like this

Code:

python -m SimpleHTTPServer 8080

Then go to http://localhost:8080/index.html and see sunshine

You could have the index.html auto reload with simple script or refresh with your mouse

EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

To reduce debug clutter for writing strategies I use a crafty tail.  I did have a cut on the end but it lagged

Code:

tail -f goxtool.log| grep -E 'Strat|Trace|^[^0-9]'

And how are the results?.

I just loaded the chart along with current data from bitcoin charts and they are pretty similar. 
http://bitcoincharts.com/charts/chart.json?m=mtgoxUSD&r=1&i=15-min
I don't know what weighted moving average function they are using but the data is close enough for my use though I probably should check it out on a few more days before I invest the yacht
If you use this data you will need to alter the timestamp for HighStock.js  add on 000 (three zeros)

EDIT: I haven't checked for the 1 hour EMAs but will do some more testing and experimenting later.  Using the TA-Lib you can specify the timeperiod and then ignore those first timeperiod output values as the results settle down.  I've used the default values from bitcoincharts.  Simple at 10 and Weighted at 25.

[hugolp]

EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

To reduce debug clutter for writing strategies I use a crafty tail.  I did have a cut on the end but it lagged

Code:

tail -f goxtool.log| grep -E 'Strat|Trace|^[^0-9]'

Yeah, thats an option, but I still would preffer to have a different file so I can check it from time to time.

And how are the results?.

I just loaded the chart along with current data from bitcoin charts and they are pretty similar. 
http://bitcoincharts.com/charts/chart.json?m=mtgoxUSD&r=1&i=15-min
I don't know what weighted moving average function they are using but the data is close enough for my use though I probably should check it out on a few more days before I invest the yacht
If you use this data you will need to alter the timestamp for HighStock.js  add on 000 (three zeros)

EDIT: I haven't checked for the 1 hour EMAs but will do some more testing and experimenting later.  Using the TA-Lib you can specify the timeperiod and then ignore those first timeperiod output values as the results settle down.  I've used the default values from bitcoincharts.  Simple at 10 and Weighted at 25.

Thanks for the direction, I will try using ta-lib between today and tomorrow, hopefully it gives better results that my manually created one (which its not difficult...).

[MtQuid]

EDIT: Also, can anyone give me a hint on how to create a separate debug file for my bot? The regular debug goxtool file gets too big and its very cumbersome to check.

To reduce debug clutter for writing strategies I use a crafty tail.  I did have a cut on the end but it lagged

Code:

tail -f goxtool.log| grep -E 'Strat|Trace|^[^0-9]'

Yeah, thats an option, but I still would preffer to have a different file so I can check it from time to time.

And how are the results?.

I just loaded the chart along with current data from bitcoin charts and they are pretty similar. 
http://bitcoincharts.com/charts/chart.json?m=mtgoxUSD&r=1&i=15-min
I don't know what weighted moving average function they are using but the data is close enough for my use though I probably should check it out on a few more days before I invest the yacht
If you use this data you will need to alter the timestamp for HighStock.js  add on 000 (three zeros)

EDIT: I haven't checked for the 1 hour EMAs but will do some more testing and experimenting later.  Using the TA-Lib you can specify the timeperiod and then ignore those first timeperiod output values as the results settle down.  I've used the default values from bitcoincharts.  Simple at 10 and Weighted at 25.

Thanks for the direction, I will try using ta-lib between today and tomorrow, hopefully it gives better results that my manually created one (which its not difficult...).

I just put this together.
This will create a log file for each loaded strategy as well as sending log messages to the main log.
Maybe prof7bit could check it over and put it into the main source.
It's a quick hack but it works.
And you call self.debug() just like before and there is a new function self.info() for different log level.

Change Strategy.py to look like this.

Code:

"""
trading robot breadboard
"""

import goxapi
import logging


class Strategy(goxapi.BaseObject):
    # pylint: disable=C0111,W0613,R0201

    def __init__(self, gox):
        goxapi.BaseObject.__init__(self)
        self.signal_debug.connect(gox.signal_debug)
        gox.signal_keypress.connect(self.slot_keypress)
        gox.signal_strategy_unload.connect(self.slot_before_unload)
        gox.signal_ticker.connect(self.slot_tick)
        gox.signal_depth.connect(self.slot_depth)
        gox.signal_trade.connect(self.slot_trade)
        gox.signal_userorder.connect(self.slot_userorder)
        gox.orderbook.signal_owns_changed.connect(self.slot_owns_changed)
        gox.history.signal_changed.connect(self.slot_history_changed)
        gox.signal_wallet.connect(self.slot_wallet_changed)
        self.gox = gox
        self.name = "%s.%s" % \
            (self.__class__.__module__, self.__class__.__name__)

        self.logger = logging.getLogger(self.name)
        self.fh = logging.FileHandler(self.name + '.log')
        formatter = logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')
        self.fh.setFormatter(formatter)
        self.fh.setLevel(logging.DEBUG)
        self.logger.addHandler(self.fh)

        self.info("%s loaded" % self.name)

    def info(self, *args):
        """write info event to log.  use our logger then send to the base logger"""
        msg = " ".join([str(x) for x in args])
        self.logger.info(msg)
        goxapi.BaseObject.debug(self, *args)


    def debug(self, *args):
        """write debug event to log.  use our logger then send to the base logger"""
        msg = " ".join([str(x) for x in args])
        self.logger.debug(msg)
        goxapi.BaseObject.debug(self, *args)

    def __del__(self):
        """the strategy object will be garbage collected now, this mainly
        only exists to produce the log message, so you can make sure it
        really garbage collects and won't stay in memory on reload. If you
        don't see this log mesage on reload then you have circular references"""
        self.debug("%s unloaded" % self.name)
        self.logger.removeHandler(self.fh)

    def slot_before_unload(self, _sender, _data):
        """the strategy is about to be unloaded. Use this signal to persist
        any state and also use it to forcefully destroy any circular references
        to allow it to be properly garbage collected (you might need to do
        this if you instantiated linked lists or similar structures, the
        symptom would be that you don't see the 'unloaded' message above."""
        pass

    def slot_keypress(self, gox, (key)):
        """a key in has been pressed (only a..z without "q" and "l")"""
        self.debug("someone pressed the %s key" % chr(key))

    def slot_tick(self, gox, (bid, ask)):
        """a tick message has been received from the streaming API"""
        pass

    def slot_depth(self, gox, (typ, price, volume, total_volume)):
        """a depth message has been received. Use this only if you want to
        keep track of the depth and orderbook updates yourself or if you
        for example want to log all depth messages to a database. This
        signal comes directly from the streaming API and the gox.orderbook
        might not yet be updated at this time."""
        pass

    def slot_trade(self, gox, (date, price, volume, typ, own)):
        """a trade message has been received. Note that this signal comes
        directly from the streaming API, it might come before orderbook.owns
        list has been updated, don't rely on the own orders and wallet already
        having been updated when this is fired."""
        pass

    def slot_userorder(self, gox, (price, volume, typ, oid, status)):
        """this comes directly from the API and owns list might not yet be
        updated, if you need the new owns list then use slot_owns_changed"""
        pass

    def slot_owns_changed(self, orderbook, _dummy):
        """this comes *after* userorder and orderbook.owns is updated already.
        Also note that this signal is sent by the orderbook object, not by gox,
        so the sender argument is orderbook and not gox. This signal might be
        useful if you want to detect whether an order has been filled, you
        count open orders, count pending orders and compare with last count"""
        pass

    def slot_wallet_changed(self, gox, _dummy):
        """this comes after the wallet has been updated. You can access the
        new balance like so: gox.wallet["BTC"] or gox.wallet[gox.currency]"""
        pass

    def slot_history_changed(self, history, _dummy):
        """this is fired whenever a new trade is inserted into the history,
        you can also use this to query the close price of the most recent
        candle which is effectvely the price of the last trade message.
        Contrary to the slot_trade this also fires when streaming API
        reconnects and re-downloads the trade history, you can use this
        to implement a stoploss or you could also use it for example to detect
        when a new candle is opened"""
        pass

EDIT1: Changed formatter to

Code:

formatter = logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')

EDIT2: Changed call to base debug to use *arg instead of arg so is now

Code:

goxapi.BaseObject.debug(self, *args)

No more edits.  Code is flawless
EDIT3: Need to remove file handler on strategy unload so that it is not reloaded twice or more during a strategy reload
I think that should do it now.  Code is never flawless
Price is also going down.... Time for tea

[hugolp]

MtQuid that looks terrific. Hopefully prof7bit can include that or something similar in the source.

Another solution might be being able to select what goes into the debug at runtime, so one could select to only have in the debug file the messages from the modules s/he wants. Im fine with any of both solutions.

EDIT: MtQuid tried your code and its working perfectly. Im sending you a small tip.

[MtQuid]

EDIT: MtQuid tried your code and its working perfectly. Im sending you a small tip.

Tip recieved. 
Thanks hugolp

[wilfried]

hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:

import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
	if key == ord("b"):
	     self.debug("canceling all rebalancing orders")
			
			
			

If key "b" is pressed goxtool displays: someone pressed key b,
But it doesnt display "canceling all rebalancing orders".
so, how to write messages to the log output?


btw: is there an easy way to get rid of the Winconsole output in the main window? I am running tail on the log in a second window, so its just superfluid for me?

[MtQuid]

hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:

import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
	if key == ord("b"):
	     self.debug("canceling all rebalancing orders")
			
			
			

If key "b" is pressed goxtool displays: someone pressed key b,
But it doesnt display "canceling all rebalancing orders".
so, how to write messages to the log output?


btw: is there an easy way to get rid of the Winconsole output in the main window? I am running tail on the log in a second window, so its just superfluid for me?

Your strategy file works perfectly though it won't pay the bills.
Make sure you are loading it.  e.g.

Code:

python ./goxtool.py --protocol=websocket --strategy=_test.py

And you should see the load and unload in goxtool.log

Code:

2013-05-01 14:47:32,433:DEBUG:_test.Strategy:_test.Strategy loaded
...
2013-05-01 14:47:38,099:DEBUG:_test.Strategy:canceling all rebalancing orders
...
2013-05-01 14:47:40,974:DEBUG:_test.Strategy:_test.Strategy unloaded

[wilfried]

thx, so your log shows the message, mine doesnt; there is something wrong with my setup, but i cant figure out what.
i.e. this script prints out the messages:
http://www.rugatu.com/questions/14260/python-trading-bot-configuration-file

my bot  does load - debug message appears:

2013-04-30 15:30:24,177:DEBUG:testbot.Strategy:testbot.Strategy unloaded

and keystroke is registered, but the self.debug message doesnt print.

If I use it i.e. with an non existent bot file (goxtool.py --strategy=anything.py

it does not recognize the key pressed.
I am stuck with this

[prof7bit]

hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:

import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
	if key == ord("b"):
	     self.debug("canceling all rebalancing orders")
			

This can't work. Please fix the formatting and indentations. And set your editor to 4 spaces, no tabs (recommended style, used by most others too) and dont ever mix tabs and spaces. You should first make yourself comfortable with some essential concepts of the Python programming language with simpler examples and simple isolated standalone programs, follow some tutorials. You should not attempt to ride a motorbike until you can safely ride a bicycle. This will avoid many frustrations.

[wilfried]

Hy prof7bit,
maybe i am blind, but i left the parts as they were in the rebalancing bot.

what i saw now from looking into
http://www.rugatu.com/questions/14260/python-trading-bot-configuration-file

is that the rebalancing bot does
import strategy
and in strategy.py there is a function slot_keypress
in the bot of tarzan he doesnt import strategy and defines the class Strategy(goxapi.BaseObject):
- but if thats the problem, how does the balancing bot work?
thx

[prof7bit]

what i saw now from looking into
http://www.rugatu.com/questions/14260/python-trading-bot-configuration-file

Tarzan's bots are not autoritative. Also he wrote some comments that indicate he did not really understand goxtool yet at the time he attempted to write these bots.

One can either make a copy of the existing strategy.py and use it directy or one can (more elegant and recommended) just derive a new class from it and override only the virtual slot methods that are needed for the bot and leave away the ones that are not needed (in Python all methods are always virtual and defining a method that exists in the inherited class will automatically override it)

Also please paste your code as it is (it was clearly not formatted properly in your previous posting, and you know (because you followed my advice to learn Python) that in Python indentation is a critical part of the syntax. Also post *all* relevant log output)

[wilfried]

thx again, i re-pasted the rebalancing bot and now it works, still i cant see what the error in the above code was, anyway it prints out now.
I got this code now:

Code:

"""
The portfolio rebalancing bot will buy and sell to maintain a
constant asset allocation ratio of exactly 50/50 = fiat/BTC
"""

import strategy
import goxapi

DISTANCE    = 7     # percent price distance of next rebalancing orders
MARKER      = 7     # lowest digit of price to identify bot's own orders
COIN        = 1E8   # number of satoshi per coin, this is a constant.

def add_marker(price, marker):
    """encode a marker in the price value to find bot's own orders"""
    return price / 10 * 10 + marker

def has_marker(price, marker):
    """return true if the price value has the marker"""
    return (price % 10) == marker

def mark_own(price):
    """return the price with our own marker embedded"""
    return add_marker(price, MARKER)

def is_own(price):
    """return true if this price has our own marker"""
    return has_marker(price, MARKER)



class Strategy(strategy.Strategy):
    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False

    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
        """a key has been pressed"""

        if key == ord("c"):
            # cancel existing rebalancing orders and suspend trading
            self.debug("canceling all rebalancing orders")
            self.temp_halt = True
            #self.cancel_orders()

could you please point me in the right direction, so i can do some reading, about the "virtual slots"- i suppose the functions in the Strategy Class are those slots, i.e.
"slot_history_changed" - one is supposed to use this function to put some code in, when it is "fired"  - what i dont get is, the def is only fount in the strategy.py file - how does the goxtool and goxapi fire something into it?? thx!

[prof7bit]

thx again, i re-pasted the rebalancing bot and now it works, still i cant see what the error in the above code was

Because you messed up the indentations. This totally changes the program.

In Python whitespace has a syntactical meaning to get rid of the ugly and redundant and hard to reach (on a german keyboard) curly braces. The absence of curly braces and the clean and readable syntax are one of the main reasons I prefer python over all other scripting languages and Pascal over C++.

[wilfried]

i know about indentation (just a beginner yet), what i ment was, that i didnt see which indentation was messed up, but maybe due to the fact that since 6 hours i am studying your code and its the first time i actually try to cope with classes..

[prof7bit]

i know about indentation (just a beginner yet), what i ment was, that i didnt see which indentation was messed up

The last two lines in your other posting, the if statement was outside the slot_keypress method, it was even outside the entire class. MtQuid intuitively corrected it and fixed the indentation without asking further questions when he tested it (because he understood what was probably your intention) but the Python interpreter cannot guess what you might have wanted to do, it will interpret it such that the block belonging to the slot_keypress method ends as soon as the indentation level changes.

[wilfried]

AAAAAAHHHHH, its the variables in the braces of the class-functions, right?

i.e.

def slot_history_changed(self, history, _dummy):

history is found in goxapi - class History(BaseObject)

so goxapi "fills" its classes and strategy.py is defining other classes filled with goxapi results?

I think i start to understand

[wilfried]

i am blind as a fish, cant see it..

i posted:

hy
i started to look into goxbot and i must admit, i dont get anything, but anyway i will reach the goal of my own little bot. plz help!

in my strategy module i got (copied most of it of the 50/50-balance boot)

Code:

import strategy
import goxapi


class Strategy(strategy.Strategy):

    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False


    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
	if key == ord("b"):
	     self.debug("canceling all rebalancing orders")
			
			
			

If key "b" is pressed goxtool displays: someone pressed key b,
But it doesnt display "canceling all rebalancing orders".
so, how to write messages to the log output?


btw: is there an easy way to get rid of the Winconsole output in the main window? I am running tail on the log in a second window, so its just superfluid for me?

what i run now is (snipplet):

Code:

class Strategy(strategy.Strategy):
    """a protfolio rebalancing bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.temp_halt = False

    def slot_before_unload(self, _sender, _data):
        pass

    def slot_keypress(self, gox, (key)):
        """a key has been pressed"""

        if key == ord("c"):
            # cancel existing rebalancing orders and suspend trading
            self.debug("canceling all rebalancing orders")
            self.temp_halt = True
            #self.cancel_orders()

do you mean that self.debug.. is under the "e" of key and above it was under the "y" of key?

[prof7bit]

AAAAAAHHHHH, its the variables in the braces of the class-functions, right?
i.e.
def slot_history_changed(self, history, _dummy):
history is found in goxapi - class History(BaseObject)
so goxapi "fills" its classes and strategy.py is defining other classes filled with goxapi results?
I think i start to understand

lol, yes, almost. Although your usage of the above terms indicates that you are only at the beginning of your journey, learning Python. You should really try to work through some tutorials, write your own simple programs from scratch, first start with simple functon calls, understand what function arguments are, global and local variables, then much later maybe learn what classes are and methods and how these are related, inheritance, virtual methods and all that stuff. And then beneath all that there also exist an entire universe of hidden Python-specific black magic that can also be used. This is not learned and understood in one single day, it needs some time.

[ErebusBat]

You REALLY want to listen to prof7bit.  Especially because we don't have backtesting so you are effectively PAYING to test your bot on the live system.

[prof7bit]

Code:

    def slot_keypress(self, gox, (key)):
	if key == ord("b"):
	     self.debug("canceling all rebalancing orders")

do you mean that self.debug.. is under the "e" of key and above it was under the "y" of key?

the entire two last lines don't seem to belong into the slot_keypress method because the if line needs to be intended further than the def line to be considered inside the method and the line after the if needs to be intended even further than the if to be considered inside the if block.

Also be careful with mixing tabs and spaces, you dont see them, everything looks right but isnt. In python it has been established that everybody use 4 spaces and no tabs to have a consistent coding style everywhere and I am trying to follow the styleguides as close as possible myself. So set your editor to

(depending on what editor you use, every editor has similar options somewhere in its config)
indentation 4 spaces
no tabs
convert tabs to spaces

Do this once and from then on everything becomes much easier.

You should also make it a habit to run your code through pylint, I'm doing this everytime I save the code in the editor, before I even attempt to run it, Its a great help, it forces you to adhere to the established standards and coding style and it also finds a lot of problems and errors (my favorite are misspelled variable names, missing function arguments, missing self argument in methods) that could otherwise only be detected at runtime and maybe even not immediately after program start but only after a few hours when its attempting to do its first trade.

[wilfried]

thx, i use your code as leraning object; isnt that a good motivation - as soon as it works it brings btc

[c0inbuster]

I just want to inform you, guys, that an interesting python open source software for backtesting strategy is available here
http://gbeced.github.io/pyalgotrade/

mixing both goxtool and pyalgotrade could make a good "recipe".

I'm looking for a 1 hour timeframe data history for MtGox BTCUSD

I tried http://bitcoincharts.com/t/trades.csv?symbol=mtgoxUSD&start=1335830400&end=1367366400

as 1335830400 is UNIX Timestamp for Tue, 01 May 2012 00:00:00 GMT
and 1367366400 for Wed, 01 May 2013 00:00:00 GMT

but Bitcoincharts only reply with data from 1367249267 (Mon, 29 Apr 2013 15:27:47 GMT)

[DonBit]

Excellent work prof7bit!!! Keep it up the good job! 

Just made some changes to the stop loss bot and posted it in the stop loss bot thread (I added the start gain feature and full wallet amount autofill).

Donating 0.1 for this amazing tool. See transaction dd9dc2b56c328d7a1bb57d07381c789d2312249ee84dd79b578d75e49b4b854b

-DonBit

[MtQuid]

I just want to inform you, guys, that an interesting python open source software for backtesting strategy is available here
http://gbeced.github.io/pyalgotrade/

mixing both goxtool and pyalgotrade could make a good "recipe".

I'm looking for a 1 hour timeframe data history for MtGox BTCUSD

I tried http://bitcoincharts.com/t/trades.csv?symbol=mtgoxUSD&start=1335830400&end=1367366400

as 1335830400 is UNIX Timestamp for Tue, 01 May 2012 00:00:00 GMT
and 1367366400 for Wed, 01 May 2013 00:00:00 GMT

but Bitcoincharts only reply with data from 1367249267 (Mon, 29 Apr 2013 15:27:47 GMT)

Cool.
Have a look here http://www.reddit.com/r/Bitcoin/comments/11iz5b/the_history_of_gox_mountain/
I just downloaded the torrent and this is the head

Code:

http://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=0
insert into trades values ("USD", "1279408157", "Y", "BTC", "limit", "1", "2000000000", "20", "4951", "", "0.04951")
insert into trades values ("USD", "1279424586", "Y", "BTC", "limit", "2", "5001000000", "50.01", "5941", "", "0.05941")
insert into trades values ("USD", "1279475336", "Y", "BTC", "limit", "3", "500000000", "5", "8080", "", "0.0808")
insert into trades values ("USD", "1279489451", "Y", "BTC", "limit", "4", "1000000000", "10", "8585", "", "0.08585")
insert into trades values ("USD", "1279490426", "Y", "BTC", "limit", "5", "500000000", "5", "8584", "", "0.08584")
insert into trades values ("USD", "1279490436", "Y", "BTC", "limit", "6", "500000000", "5", "8584", "", "0.08584")
insert into trades values ("USD", "1279511584", "Y", "BTC", "limit", "7", "500000000", "5", "9090", "", "0.0909")
insert into trades values ("USD", "1279556653", "Y", "BTC", "limit", "9", "8000000000", "80", "9307", "", "0.09307")
insert into trades values ("USD", "1279559013", "Y", "BTC", "limit", "10", "10000000000", "100", "8911", "", "0.08911")

and the tail

Code:

insert into trades values ("USD", "1367047706", "Y", "BTC", "limit", "1367047706332857", "9980000", "0.0998", "13683040", "ask", "136.8304")
insert into trades values ("USD", "1367047791", "Y", "BTC", "limit", "1367047791768232", "25712723", "0.25712723", "13688009", "bid", "136.88009")
insert into trades values ("USD", "1367047804", "Y", "BTC", "market", "1367047804704062", "80186290", "0.8018629", "13682010", "ask", "136.8201")
insert into trades values ("USD", "1367047805", "Y", "BTC", "limit", "1367047805062520", "79726342", "0.79726342", "13688009", "bid", "136.88009")
insert into trades values ("USD", "1367047825", "Y", "BTC", "limit", "1367047825872682", "497020000", "4.9702", "13682020", "ask", "136.8202")
insert into trades values ("USD", "1367047826", "Y", "BTC", "limit", "1367047826181247", "500000000", "5", "13682000", "ask", "136.82")
insert into trades values ("USD", "1367047853", "Y", "BTC", "limit", "1367047853175671", "502980000", "5.0298", "13682000", "bid", "136.82")
insert into trades values ("USD", "1367047879", "Y", "BTC", "market", "1367047879486765", "800000000", "8", "13686000", "bid", "136.86")
insert into trades values ("USD", "1367047879", "Y", "BTC", "market", "1367047879775307", "600000000", "6", "13688000", "bid", "136.88")
insert into trades values ("USD", "1367047879", "Y", "BTC", "market", "1367047879824265", "122345236", "1.22345236", "13688009", "bid", "136.88009")

Would be cool to link up with pyalgotrade.

[evilscoop]

can anyone help me with coding a strategy for this tool...

I need to calc the EMAs and want to use the differance in my stratagy...
I have no clue atm how to do this....

Im looking to create a macd stratagy on the dema from mtgox

Pm me pls, as I dont want to derail the thread, thx

[hugolp]

can anyone help me with coding a strategy for this tool...

I need to calc the EMAs and want to use the differance in my stratagy...
I have no clue atm how to do this....

Im looking to create a macd stratagy on the dema from mtgox

Pm me pls, as I dont want to derail the thread, thx

MtQuid already showed how to include ta-lib in a strategy. Look early pages in this thread.

[evilscoop]

ahh thx ill look into it

[hugolp]

I hit a problem with my bot and its that the last days the number of trades has increased a lot at mtgox so downloading two days of data everytime I load the bot takes for ever. So Ive been programming a way to save the trades in a database. It obviously saves the trades in the database, but also on load my strategy checks the first received trade and the oldest trade in the database and downloads the trades missing between them so the history is always complete. The idea is to use it for backtesting as well. If anyone is interested I can share the code.

Also, with more datapoints the hourly ema formula gives almost exacts results to bitcoincharts (as suspected) so thats good.

[FoxMURDER]

I never liked how depth/tick messages keep flooding the console window. At first I was disabling those. But today I tried to hack it into a separate console window (I am using a large screen and half the console window was unused anyway).

So I got something like this: http://www.imagehosting.cz/thumbs/goxtool1.png

At the moment it is rather an ugly hack. And the question is: anyone else interested? Should I clean it up?

[prof7bit]

So I got something like this:

Did you change the character that is used for the depth bars or is that the way putty displays the ACS_CKBOARD character?

(Also from your screenshot it seems you are still using the socketio server, you should switch to websocket, much better experience, faster connect and no disconnects anymore)

[FoxMURDER]

That is just a mess from misconfigured (not using UTF8) putty. I use it just to check if I broke something before pushing to my server. The "real thing" is running in different window and bars are just fine. Just trying to figure out what protocol I'm using there.

[prof7bit]

Ok, I'm quickly inserting this post here before I forget it, this might be of interest for others trying to use goxtool with putty (there seem to be a lot), After seeing the screenshot I did some quick experiments with the Linux version of putty myself and line-drawing and other special characters looked totally screwed up here too. There seem to be the following few important settings that must be changed from the default, this will also solve the problem with F4 key not working:

Terminal > Keyboard > "Function keys and keypad" : Xterm R6
Wndow > Tranlation > "use font encoding"
Wndow > Tranlation > "[_] Override with UTF-8"  (remove the checkmark)

the equivalent options in the session settings file are:

LinuxFunctionKeys=2
LineCodePage=Use font encoding
UTF8Override=0

PS: It also seems that the UTF-8 problem will not occur when running goxtool through screen, screen seems to handle the character translation a different way but if not running through screen then these settings should help.

[supert]

Some beer money on the way to prof7bit from me.

[supert]

Having been mightily impressed by MtQuid's .js tomfoolery, perhaps someone is interested in using something like this to display the orderbook history:
http://code.google.com/p/javascript-surface-plot/

[MtQuid]

Having been mightily impressed by MtQuid's .js tomfoolery, perhaps someone is interested in using something like this to display the orderbook history:
http://code.google.com/p/javascript-surface-plot/

Ok I tried this out and got it working even though I think there must be better ways to represent the data.
See the files
https://github.com/MtQuid/goxtool/blob/master/strategies/dump_order_book.py
https://github.com/MtQuid/goxtool/blob/master/www-root/orderbook-3d.html

It can take some time to calculate the points within the browser window so I'm sure the solution can be optimised, though it is a lot faster when var fillPly = false;
Default settings are added to goxtool.ini on first load.  I have used this config solution in other strategies and I like it because all the configs are in one file.
If you want to keep the same folder layout then have a look at some of my other commits.

http://www.imagehosting.cz/thumbs/3dorderboo.png

See what you think and tell me if there are any bugs.

[c0inbuster]

Hello,

I don't see why using a 3D plot for a problem with only 2 variables (price, volume)...
unless third variable is time (but I'm not sure that it's what you are drawing)

In my mind it will be a much better idea to work on a Qt widget to display
candlesticks (like Highcharts Highstock) and also a Qt widget to display orderbook depth
Qwt could be help http://qwt.sourceforge.net/
or this project https://code.google.com/p/candelabro/
so it could be add to goxgui https://bitcointalk.org/index.php?topic=176489.0

On my side, I have now historical data M1, M5, M15, M30, H1, H2, H6, H12, D1, W1 data
for mtgoxBTCUSD.

I've done all the work to test if data are ok (no missing data, no overlapping data, constant timeframe)

I'm now working on a script to generate pseudo tick data (generate control points from candlestick).

I will release this soon...

Kind regards

[MtQuid]

Hello,

I don't see why using a 3D plot for a problem with only 2 variables (price, volume)...
unless third variable is time (but I'm not sure that it's what you are drawing)

Yep it's time.
Here is a graphic of an hours worth of change on the orderbook using rickshaw.
I just think these things look cool.

http://www.imagehosting.cz/thumbs/3dordebsb.png

I'll have a look at the Qt but I'm having fun messing around with python and js at the moment.
I hadn't looked at that project before - now exploring.

[c0inbuster]

Ok very well !!! in fact I prefer 2D version with colormap to display much recent orderbook depth darker
you should use opacity (if you can)

I hadn't looked at that project before - now exploring.

Which project are you talking of ?

• Qt
• Qwt
• Candelabro
• Goxgui

[supert]

MtQuid your work is awesome and I look forward to playing with it this weekend.

[MtQuid]

Which project are you talking of ?

goxgui
I had seen it before but when looking for a trading framework I skipped over it as I wasn't looking for a GUI.
In fact I still don't really need anything other than goxtool but I'll keep goxgui in mind when playing around.
What I really like about goxtool is that it's not a lot of source code and it's not hard to digest the whole lot.
If you plan to build a bot that plays dice with your own money then you really must look at all the code and have a feel for it.
Also goxtool is not compiled, or anything like that, and once you have examined the source you can safely use it and only
have to review each new commit.
There are many things I really like about goxtool.

I just tried goxgui but it was too big for my laptop screen and I couldn't shrink the dialog any more.
Not a great start....

Anyway I've been working on a buy/sell helper bot that could be useful to some people and I'm just finishing of the testing.
My plan is to have this advanced buying/selling being controlled by a strategy bot that takes inputs from other providers like
lumptrade/lag/ta-lib/...  Have it all modular.  I'll release it all when I'm happy it works as planned.

Would be nice to have a fake MtGox to test out the buying and selling on and simulate market events.  Less cost and less time.

And at the moment in that bot I'm using key-press with feedback in the log file but it would be nice to be able to pop up some ncursors
dialogs from inside a strategy.

Also we should think about what to do with multiple accounts or accounts trading in different quotes and bases
Should each account need another instance of goxgui running? If so at least a colour change would be useful for recognising an account.
Colours in goxtool.ini anyone??
Maybe an account selector via command line argument or start-up prompt.  With each account able to colourise the gui, along with inline switching.
I know I'm saying goxtool is lean but this does not add much weight.

Lots to play about with

Edit: Scrap the online account switching idea as this would be a nightmare for the bots...I have been drunk.
I think all that's needed is select config from command line argument and have some limited colour options in the config.

[c0inbuster]

I don't think having differents instances of goxtool will be a very good idea...
because you can design a strategy which need to know price of both mtgox|BTC/USD and mtgox|BTC/EUR.
for now, we can't (in my mind) have such a strategy implemented

[jbsnyder]

Been lurking for a bit, mainly because it took a while to get out of newbie jail.

I've been playing around with some strategies using the pandas python module and EMAs.  As I think prof7bit noted somewhere, things like this seem to work well in some cases but then fall apart in others resulting in somewhat of a wash.  The latest thing I've given a try adds thresholding based on volume on top of using EMA crossovers.  We'll see :-)

One thing that might be of interest to those on here developing such algorithms might be quantopian.com which is a Python-based stackexchange-ish online forum for backtesting and discussing algorithms (they'll run the algorithms online and generate some nice plots/stats).  Their main data sources do not include bitcoin, but they do provide an ability to pull in CSV data from anywhere and there's at least one database that will provide daily data for that that one can test against it:

https://www.quantopian.com/posts/anyway-to-import-bitcoin-data

It might be interesting to build an adapter within a template to match the API here, or perhaps one of the other backtesting frameworks out there to be able to test algorithms.

Edit: Also, regarding GUIs why not use something based on matplotlib or maybe Enthought's Chaco which is designed for interactive plotting:
http://code.enthought.com/chaco/

Edit2: quantopian's backend is built around the zipline module, so perhaps that could figure into offline backtesting:
https://github.com/quantopian/zipline

[limpbrains]

Been lurking for a bit, mainly because it took a while to get out of newbie jail.

I've been playing around with some strategies using the pandas python module and EMAs.  As I think prof7bit noted somewhere, things like this seem to work well in some cases but then fall apart in others resulting in somewhat of a wash.  The latest thing I've given a try adds thresholding based on volume on top of using EMA crossovers.  We'll see :-)

One thing that might be of interest to those on here developing such algorithms might be quantopian.com which is a Python-based stackexchange-ish online forum for backtesting and discussing algorithms (they'll run the algorithms online and generate some nice plots/stats).  Their main data sources do not include bitcoin, but they do provide an ability to pull in CSV data from anywhere and there's at least one database that will provide daily data for that that one can test against it:

https://www.quantopian.com/posts/anyway-to-import-bitcoin-data

It might be interesting to build an adapter within a template to match the API here, or perhaps one of the other backtesting frameworks out there to be able to test algorithms.

Edit: Also, regarding GUIs why not use something based on matplotlib or maybe Enthought's Chaco which is designed for interactive plotting:
http://code.enthought.com/chaco/

Edit2: quantopian's backend is built around the zipline module, so perhaps that could figure into offline backtesting:
https://github.com/quantopian/zipline

Thanks for great links

Good video to understand how to work with zipline and ipython
http://vimeo.com/53064082
If you want to try it, you should run ipython with pylab inline

Code:

ipython notebook --pylab inline

Have anybody succeed to load mtgox data into it?

[c0inbuster]

zipline seems to be exactly what I was looking for !!!
a python algo trading environment for backtesting (but event-driven !)

[MtQuid]

Great links with the zipline stuff

Here is my notebook with dual moving average on bitcoin data with 20/10 day windows.
http://nbviewer.ipython.org/5561936
I just don't know why I have to add 2 items to the moving average lists.

Though this stuff does not really belong in this goxtool thread..ooOOO  what to do?
In fact I only post in this thread so I don't care, the rest of the forum is the wild west to me.
And this stuff will belong in here when we make zipline work with a goxtool strategy.

Someone make a new thread..I'm going to bed...

[c0inbuster]

New post for zipline / Quantopian here
https://bitcointalk.org/index.php?topic=202808.new#new

[K1773R]

just saw this, whats that?

Code:

### (https) calling https://data.mtgox.com/api/2/money/info                                                                                                                                                                                  
{u'success': False, u'message': u'CONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.', u'
id': u'info', u'token': u'unknown_error', u'op': u'remark'}

this happens for everything i wanna do.
err, seems mtgox is stuck. dont mind this post then

[whydifficult]

Code:

### (https) calling https://data.mtgox.com/api/2/money/info                                                                                                                                                                                  
{u'success': False, u'message': u'CONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.', u'
id': u'info', u'token': u'unknown_error', u'op': u'remark'}

That is an error straight from MtGox.

[K1773R]

yea, as i said "dont mind this post"
was about to paste here in case it happens only for me, but well everyone had the json error obviously.

[supert]

prof7bit, would you consider supporting any other exchanges, esp. bitstamp? Even in a partial way?

I am getting a bit nervous about mtgox's long term chances of survival.

[nii236]

Is there a way to do market orders or "all in" orders? I can't do it easily with the current goxtool as I have to type in USD price and quantity of BTC.

Is there a way for goxtool to:

• Calculate how many BTC given a certain amount of USD using the market price?
  
• Calculate how much USD given a certain amount of BTC using the market price?

Once I have this number I can offset accordingly.

[Aido]

Here's my 'all in' version of prof7bit's original _stoploss.py:

https://gist.github.com/aido/5623833

I have a few other ones at https://gist.github.com/aido/public

[nii236]

Thanks! AidoATP rocks too by the way.  

Nice, I think the _panic.py is what I need.

[c0inbuster]

Hi,

it is unfortunate that this project does not advance much. Maybe a decoupling of features is required. Have you considered using an Advanced Message Queuing Protocol http://fr.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol (AMQP like RabbitMQ http://www.rabbitmq.com/) ?

Kind regards

[nanobtc]

prof7bit, thank you for your work. goxtool + balancer.py has been good to me. I am not a good manual day-trader, but balancer.py has been. Donation sent.

[JWU42]

Sad this has died down but still working...

Can anyone advise how to increase the number of rows visible at the bottom.  It looks like 8 is the default - would like to increase to 12 or so.

I have peaked into the files and cannot find it.

TIA!

[prof7bit]

in goxtool.py around line 48 there are the sizes defined

Note that after changing your local copies of the files you might need a local git branch or git stash to be able to continue pulling and merging future updates of goxtool (sounds more complicated than it actually is)

[JWU42]

Not sure how I missed that 

Thanks!!

[frankenmint]

in goxtool.py around line 48 there are the sizes defined

Note that after changing your local copies of the files you might need a local git branch or git stash to be able to continue pulling and merging future updates of goxtool (sounds more complicated than it actually is)

Wish I did this 6 months ago...so I'm trying to run it using cygwin and I have everything setup, but when I try to use it, its is like the bash window doesnt move forward after entering in:

  python goxtool.py --protocol=socketio --use-http


It just goes to a new line.  Dont know what else I could be missing  

[nanobtc]

No trading fees on Gox till Monday (Tokyo time). If you're running balancer.py this could be a good opportunity to set your distance as small as possible. You don't want it placing orders smaller than .01 or Gox will just round them off.

frankenmint, I'm not familiar with running Goxtool in Cygwin. It may not be finding the path, even though it looks like you are in the goxtool directory.
Try specifying ./goxtool.py as part of your command (preface goxtool.py with dot slash)?

[prof7bit]

please

git pull

and then run with

--protocol=pubnub

because websocket is currently down. Implemented this in a hurry, might still be buggy but seems to work for me. Will fix bugs later as they become known.

[Aido]

Seems to be working fine.

I'm impressed prof7bit. Very quick turnaround, thanks.

[piramida]

To add to prof7bit's awesome opensource, posted the bot I'm using: https://bitcointalk.org/index.php?topic=376352.new#new

Based on the balancer bot but different logic - trading inside a predefined price range buying below current price selling higher; since it sets orders on many levels across the range, is pretty resilient to gox lag spikes and API delays. Does not need balancing, uses all fiat in equal increments and the same with BTC, amounts automatically balance out after several spikes.

It can go full-BTC or full-FIAT, so this bot definitely requires supervision - you set it's trading ranges, you can select how aggressive it will be. Setting a range of 0-10000 would be really conservative, and using something like 450-900 as I do now would yield the most.

Testing it with significant deposit for several weeks, does not lose orders, is stable (mainly due to goxtool's architecture and stability, of course), worked really well during that crash.

[opinologo]

@prof7bit I noticed that Goxtool goes to great lengths identifying self orders but according to: https://bitbucket.org/nitrous/mtgox-api#markdown-header-moneyinfo MONEY/ORDER/ADD should return "success" (or error) along with the the OID (order identifier). I still didn't try sending orders, but I am wondering if it is possible to identify the Bot's orders when you are sending them.

[prof7bit]

@prof7bit I noticed that Goxtool goes to great lengths identifying self orders but according to: https://bitbucket.org/nitrous/mtgox-api#markdown-header-moneyinfo MONEY/ORDER/ADD should return "success" (or error) along with the the OID (order identifier). I still didn't try sending orders, but I am wondering if it is possible to identify the Bot's orders when you are sending them.

Initially (early versions of goxtool) only used the streaming API to send orders, so it all happened asynchronously, the function call (buy, sell, cancel) did immediately return without any feedback. When I later implemented the http api I made it behave the same way, so all existing bots did not experience any different behavior when goxtool was switched to http api, the call still immediately returns while a separate thread is trying to do the http requests. As soon as the http response is received it will put the order into the owns list and fire a bunch of signals while the order is graduating from acked to pending to open.

Its all centered around the idea that nothing you do within your bot should ever block (so you should never wait for a http request to return) because that would block the entire client because only exactly one thread is allowed to be inside a signal call at any given time, any other thread in goxtool that attempts to emit a signal too will have to wait until all slots of the previous signal have returned. Signals can be emitted by the stream receive thread, the http thread and the main thread (keypress signals) and none of them can ever be processed simultaneously, they will all run into the global signal lock and wait until your strategy (or any other component of goxtool/goxapi) returns from the slot it is currently processing.

Sou you should not attempt to program your strategy in an imperative sequential manner, you should instead move the stuff that needs to happen after the order has been placed into the owns_changed slot or other slots that are appropriate. I admit that this might make some things more complicated but I don't see any easy way around this. If you absolutely cannot avoid it to do things in a sequential manner then you might try to start a new separate thread that does it and which calls the protected (prefixed wth "_") blocking http request methods directly.

[yuppie]

Can I get a collection of strategies? Something that just uses some form of EMA and only needs BTC in the wallet to start?

[dddbtc]

BTCounty for the first person to port goxtool to btc-e with support for all currency pairs.

Just because Gox is dead, doesn't mean goxtool has to die.  Goxtool was always my favorite thing in the world, even compared to some of the complex bots or trading platforms

[supert]

Bitstamp has introduced a websocket API.

https://www.bitstamp.net/websocket/

Aside: sorry for any people who lost any money on gox.

[prof7bit]

There will be no goxtool for bitstamp (and no single satoshi from me into their accounts) until they hire some competent programmers and get rid of that crap of a dysfunctional matching engine:

[supert]

They claim it was a one off glitch
https://bitcointalk.org/index.php?topic=38711.msg5523544#msg5523544
but point taken.

Nonetheless it would be a shame to let goxtool sit unused.