ncurses based MtGox live monitor and trading-bot-framework

[c0inbuster]

It will be nice to also add email notifications (and/or push notifications) when trading

If you are looking for email notifier
I did a Python class for that purpose
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/email_notifier.py

If you are looking for push notifier I found a very useful web service called PushOver
I also did a Python class for that purpose
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/pushover_notifier.py

[prof7bit]

It will be nice to also add email notifications (and/or push notifications) when trading

Functionality like this could be wrapped into separate strategy modules. Also another strategy module could be made whose only purpose is to load a list of other strategies so that you could for example run the email notifier strategy in parallel with for example a trailing stop strategy.

[c0inbuster]

I like this idea...
but a strategy like trailing stop should also have access to other strategies (such as notifiers)

object programming using python is permissive so it shouldn't be a problem

but you will have to use a kind of "dependency graph" between strategies

(if you want that trailing stop strategy will be able to send notifications you must ensure that
email_notifier strategy or pushover_notifier was load)

[prof7bit]

(if you want that trailing stop strategy will be able to send notifications you must ensure that
email_notifier strategy or pushover_notifier was load)

A simple pragmatic solution: Any notifyer strategy could connect a slot to gox.signal_debug, it could look like this:

Code:

import strategy
import whatever_message_api

RECEIVER = "somebody@example.com"

class Strategy(strategy.Strategy):
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        gox.signal_debug.connect(self.slot_debug)

    def slot_debug(self, sender, (msg)):
        if "NOTIFY" in msg:
            whatever_message_api.sendmsg(RECEIVER, msg)

       

and every other strategy that wants to send an important message would then simply do something like this:

Code:

        self.debug("NOTIFY", "foo bar %f happened at %f" % (bla, blub))

The string "NOTIFY" would then be established as some kind of convention, it wouldn't do any harm if no notifier is present (it would just go to the log file like all other messages too) and if an older strategy does not send messages containing "NOTIFY" or notifications for more events are desired then the notification strategy could optionally do some more advanced filtering of debug messages.

[c0inbuster]

Sometimes I'm not pragmatic

Do you know how to load list of strategies ?

Do you plan to add notifier (email or push) in your code base ?

Thanks a lot

[prof7bit]

Do you know how to load list of strategies ?

_loader.py (untested code, just wrote it down here without testing but should work this way unless I have forgotten something, for example not tested what happens on reload during runtime)
(use it like this: ./goxtool --strategy=_loader.py)

(my [personal] convention is to prefix all strategy module filenames (and other eperimental stuff in this folder) with _ to not confuse git too much with unversioned files in the same folder, I have put _*.* into my .gitignore file)

Code:

import strategy

import _notifierstrat
import _foostrat
import _barstrat
import _whateverstrat

class Strategy(strategy.Strategy):
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)

        self.notifierstrat  =  _notifierstrat.Strategy(gox)
        self.foostrat       =       _foostrat.Strategy(gox)
        self.barstrat       =       _barstrat.Strategy(gox)
        self.whateverstrat  =  _whateverstrat.Strategy(gox)

[c0inbuster]

Nice!

I prefer not to get away from your code...
so I will git pull when it will be ready

I think it's both a good idea :
- to avoid breaking a large part of code (and that's what you are doing elegantly)
- to have notifications when auto trading (I prefer to know very quickly [bad] surprises)

You should also have a look at my suggestion
https://bitcointalk.org/index.php?topic=181584.msg1911355#msg1911355
it will allow us to see several symbols (BTCUSD, BTCEUR...)

[prof7bit]

Do you know how to load list of strategies ?

_loader.py
[ ...code...]

This is obsolete now. After pulling the latest revision you can now just pass a comma separated list of modules on the command line, no need to load them from a separate loader strategy, this is much more flexible now:

Code:

./goxtool --strategy=foo.py,bar.py,baz.py

[c0inbuster]

Thanks for this !

Have you ever try to implement MA crossover strategy ?
http://www.onlinetradingconcepts.com/TechnicalAnalysis/MASimple2.html

How can I use indicator MACD, RSI... ?

[prof7bit]

Have you ever try to implement MA crossover strategy ?

I experimented with this back in my old forex days and never found something that would work really reliable when I simulated it with new (not seen before) test data. All the myriads of MA-crossover strategies (or whatever-indicator-cross-this-or-that-strategies) that looked promising on the data they were developed on failed miserably when simulating them on new test data. All of them! Every time! Even the most sophisticated stuff with neural networks and other machine learning technologies trying to predict the trend that seemed to work well on the training set failed miserably and immediately every time when confronted with new data, so I won't even try to do that anymore.

[BTC-engineer]

Have you ever try to implement MA crossover strategy ?

I experimented with this back in my old forex days and never found something that would work really reliable when I simulated it with new (not seen before) test data. All the myriads of MA-crossover strategies (or whatever-indicator-cross-this-or-that-strategies) that looked promising on the data they were developed on failed miserably when simulating them on new test data. All of them! Every time! Even the most sophisticated stuff with neural networks and other machine learning technologies trying to predict the trend that seemed to work well on the training set failed miserably and immediately every time when confronted with new data, so I won't even try to do that anymore.

It looks like you have large experience with automatic trading technologies. I have only some basic knowledge. One of the things I know about it, is that it's important to test/fine-tune your trading strategy with historical or synthesizer new data before going live. I think this is also what you are talking about your experience with various MA-crossover strategies. Is there a plan to implement such a test-mode into goxtool, so you are able to test your trading strategy?

[hugolp]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

[ErebusBat]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

STUPID QUESTION:  Nothing in goxtool.log?

[hugolp]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

STUPID QUESTION:  Nothing in goxtool.log?

I almost did not want to answer for the embarasment. Yes, it was there. Thank you.

[ErebusBat]

I almost did not want to answer for the embarasment. Yes, it was there. Thank you.

NP... we all have our moments so it is good to ask   Plus your question will help others.

[prof7bit]

Is there a plan to implement such a test-mode into goxtool, so you are able to test your trading strategy?

That would certainly be a nice thing to have. I'm not yet sure if it should be part of goxtool or a separate application that just exposes the same interface and can load the same strategy files. I cannot promise anything yet (don't have unlimited time), maybe also someone else might want to write something that could be used for this purpose.

[MtQuid]

+1 for 2.6
Debian squeeze has 2.5, 2.6 and 3.0 in repo.

I just installed goxtool on a squeeze box and I made this script to help anyone else in the same situation.

Code:

# Installing python2.7 on debian stable and running goxtool
# Tested on virgin debian squeeze install with sudo installed

####################################################
# Part 1: Install python2.7

# First get some tools
sudo apt-get install build-essential
# and some libs
sudo apt-get install libsqlite3-dev zlib1g-dev libncurses5-dev libgdbm-dev libbz2-dev libreadline5-dev libssl-dev libdb-dev
 
# Optional bits and bobs
#sudo apt-get build-dep python

# starting at home
cd ~

# Get and compile Python-2.7.4
wget wget http://python.org/ftp/python/2.7.4/Python-2.7.4.tar.bz2
tar xf Python-2.7.4.tar.bz2 
cd Python-2.7.4/
./configure --prefix=/usr/local
make

# This make takes about 5 mins on an old virtual machine

# Now you have 2 options:
# 1. install the new python
#  sudo make altinstall  
# 2. create a new debian package and install python 2.7 into debian package system
#  sudo checkinstall make altinstall

# I prefer checkinstall, though it will take longer 
sudo apt-get install checkinstall 

# There is a bug and checkinstall will fail because some directories could not be created.
# So just create them manually right now.
sudo mkdir -p /usr/local/lib/pkgconfig
sudo mkdir -p /usr/local/lib/python2.7/config

# Install and create the debian package
sudo checkinstall --pkgname=python2.7 --pkgversion=2.7.4 make altinstall

# checkinstall took about 45 minutes on my slow VM

# have a look at the new python
python2.7 --version
# and still we have the original 2.6 as the system wide default 
python --version

# get back to home
cd ~

# install a python package manager
# this will install the script 'easy_install-2.7'
wget https://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
tar xvf distribute-0.6.36.tar.gz 
cd distribute-0.6.36/
sudo python2.7 setup.py install

####################################################
# Part 2: Install goxtool

# get back to home
cd ~

# install a means to segregate python applications
sudo easy_install-2.7 virtualenv

# get latest goxtool
git clone https://github.com/prof7bit/goxtool.git

# create a new python2.7 environment for goxtool
virtualenv-2.7 --distribute goxtool

# switch to the new environment
source goxtool/bin/activate

# check it worked - you should see 2.7.4 as the version number
python --version

# get the packages required for goxtool
easy_install-2.7 pycrypto

# fire her up and get trading
python goxtool/goxtool.py --protocol=socketio --use-http

I've only just discovered this project with it's balance bot and I love it and have been running it for the last day and a half on the 7% default.
I'm going to try some different margins on other accounts with like half a coin balance and see how they each perform over time.
My first trading bot.

One issue.  During setup I balanced the account and then hit 'p' to place the buy/sell orders but I think I went too fast because the price was around 123 and the bot miscalculated my center and put in buys and sells at 80/90 something.  MtGox ignored the low sell but I did panic.

Also the graph is using a lot of screen space so could do with a toggle for switching between other information sets.  I'm a C++ programmer by trade but I'm keen to help out with this projects so I'm currently looking at the code.
Graph area could switch between log, graph, more detailed order book stats, strategy provided info.
Strategy provided would be the obvious solution to all as then you could have empty strategies that only provide information screens.  Then just have a mechanism to cycle through them.  So even the current graph would end up in a strategy.

Thanks for your work prof7bit.

[ErebusBat]

I thought I would share several of my helper scripts I have written to help with goxtool

Bot Start Script
This script is used to rotate the log file and start the bot with the appropriate connection params, as well as a log of where and when it was said to use those.

Code:

#!/bin/zsh
# https://bitcointalk.org/index.php?topic=148462.msg1854636#msg1854636
# https://gist.github.com/prof7bit/5395900
# https://bitcointalk.org/index.php?topic=148462.msg1886914#msg1886914

# "p": place 2 orders above and below price. 
# "c": cancel orders and effectively stop the bot
# "u": update own order list, depth, history, wallet and everything
# "i": display how much it is out of balance at current price (negative: must sell, positive: must buy)
# "b":balance immediately (cancel orders and then buy/sell at market to rebalance)

# Backup log file
if [[ -f goxtool.log ]]; then 
  mv goxtool.log goxtool_$(date '+%Y%m%d-%H%M%S').log
fi
# Per https://bitcointalk.org/index.php?topic=181584.msg1923260#msg1923260
./goxtool.py --protocol=websocket --strategy=_balancer.py

botlog.sh - View/watch bot's log
This filters out the 'noise' and lets me see just the output I want from the log.
As a side effect of the above log rotation you will need to restart this every time you restart the tool with the script above

Code:

#!/bin/zsh
tail -n500000 -f ~/goxtool/goxtool.log | egrep "(Strategy|got ack|got|have)"

startbot.sh - Enforce Screen Session
Forces that you are in a screen session so you don't start the bot and then logout and think it is running when it is not 

Code:

#!/bin/zsh
if [[ -z $STY ]] then
	echo "Not in screen! Run ~/botwindow.sh"
else
	cd ~/goxtool
	./bot_balancer.sh
fi

botwindow.sh
Starts (or re-connects to) screen with the same params every time

Code:

#!/bin/sh
screen -D -R -A -S bot

[supert]

Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

[ErebusBat]

Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

You can add (or change) the following in your ini file and then the title will not update

Code:

set_xterm_title = False