ncurses based MtGox live monitor and trading-bot-framework

[supert]

You can add (or change) the following in your ini file and then the title will not update

Code:

set_xterm_title = False

thanks.

Is there a way to moderate what is logged / verbosity (I guess I can comment out the code)
-- would also be a nice .ini option

[hugolp]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

Im hitting this problem again and now I just get a message saying:

Code:

2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

The program keeps running but without running the strategy obviously. And yes, the file mybot.py exists and is the right one. No other indication as to what might be happening. Any idea of what I could do to know where I got the code wrong?

[ErebusBat]

Im writting a bot (Strategy class) and Im having the problem that goxtool sometimes will just quit leaving a grey screen with no error message, making it hard to debug where I have fucked up in the code. Sometimes the error will show but others it just completely kills the program with no explanation. Is there a way to find out what happen or a way to solve it so one can get a debug message?

Im hitting this problem again and now I just get a message saying:

Code:

2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

The program keeps running but without running the strategy obviously. And yes, the file mybot.py exists and is the right one. No other indication as to what might be happening. Any idea of what I could do to know where I got the code wrong?

The code that handles that error is ~953:

Code:

    def reload(self):
        """reload and re-initialize the strategy module"""
        self.unload()
        for name in self.strategy_name_list:
            name = name.replace(".py", "").strip()

            try:
                strategy_module = __import__(name)
                try:
                    reload(strategy_module)
                    strategy_object = strategy_module.Strategy(self.gox)
                    self.strategy_object_list.append(strategy_object)

                # pylint: disable=W0703
                except Exception:
                    self.gox.debug(traceback.format_exc())

            except ImportError:
                self.gox.debug("### could not import %s.py" % name)

I would change that log message to output more information if I were you... by the looks of it I would guess (not being a python person at all) that your class name doesn't match your file name.

[prof7bit]

Code:

2013-04-24 20:21:58,510:DEBUG:goxapi.Gox:### could not import mybot.py

Make sure the current working directory is the goxtool directory and the mybot.py is in the same directory too.

[smooth]

Fantastic work prof7bit.

It runs OK on a raspberry pi raspbian / wheezy (wanted an always-on low power bot server) at a load average of ~0.99 and most of the memory. Hopefully there is no memory leak.

I'm not running X and in the console the xterm title (price/bid/ask) seems to cause some trouble appearing at wherever the cursor was last.

With screen over ssh in an xterm it looks fine though.

I'm very surprised by a load of ~1 for this on a rpi.  A (possible) memory leak surprises me less.  I'll test this myself soon, but I think something is likely going wrong in your configuration.

[hugolp]

The code that handles that error is ~953:

Code:

    def reload(self):
        """reload and re-initialize the strategy module"""
        self.unload()
        for name in self.strategy_name_list:
            name = name.replace(".py", "").strip()

            try:
                strategy_module = __import__(name)
                try:
                    reload(strategy_module)
                    strategy_object = strategy_module.Strategy(self.gox)
                    self.strategy_object_list.append(strategy_object)

                # pylint: disable=W0703
                except Exception:
                    self.gox.debug(traceback.format_exc())

            except ImportError:
                self.gox.debug("### could not import %s.py" % name)

I would change that log message to output more information if I were you... by the looks of it I would guess (not being a python person at all) that your class name doesn't match your file name.

Thanks, that helped. I had an typo in one of my import statements so the module refused to load. prof7bit, it would be better if the log signaled the particular error otherwise you are in the dark making it hard to debug. Basically this change:

Code:

            except ImportError as e:
                self.gox.debug("### could not import %s.py, reason: %s" % (name,e))

[ghostshirt]

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

And as i have mentioned before already: make sure you ALWAYS have an up to date version of goxtool. git pull (and restart if there were updates) at least once a day, I'm still fixing things in goxtool itself occasionally that make it more reliable.

[prof7bit]

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

The balancer does not need to be re-initialized after you update/restart goxtool, it does not have any state, it can trade from the information about currently open orders and account balance alone, it will continue where it left automatically.

Are you using the newest version of the balancer that I posted on the separate thread that I started for the balancer? Questions about it should go to this thread.

Also please note (if you haven't seen it already) that since a few days (since a server upgrade at mtgox took place) the most reliable connection to mtgox is achieved with these command line options:
--protool=websocet --use-http
I'm mentioning this again because it is very important.

[ghostshirt]

prof7bit, thanks for this great tool. I've just set it up and started to experiment with a bit of amount. I am intending to run it for a week. If I git pull during this time, and restart goxtool, should I press 'b' before 'p', or let it rebalance on its own with the price movement?

The balancer does not need to be re-initialized after you update/restart goxtool, it does not have any state, it can trade from the information about currently open orders and account balance alone, it will continue where it left automatically.

Are you using the newest version of the balancer that I posted on the separate thread that I started for the balancer? Questions about it should go to this thread.

Also please note (if you haven't seen it already) that since a few days (since a server upgrade at mtgox took place) the most reliable connection to mtgox is achieved with these command line options:
--protool=websocet --use-http
I'm mentioning this again because it is very important.

Sorry about the wrong thread.
I used the other protocol unfortunately. But the _balancer.py is the right one. I left my desktop running at office without a remote access.
Strangely, I seem to be on profit side in terms of BTC and a very small loss of $ after the crash we saw a few hours ago. newbie luck probably

[hugolp]

Excuse if this is offtopic, but I want my goxtool bot to notify my Android phone, and even sound an alarm (in case Im sleeping) when certain conditions are met. Whats the best way of achieving this? I know how to program for Android (but would be happy if I can avoid having to create my own program just for this) and own Tasker. One option seems to use Google Cloud Messaging, anyone has any other suggestion?

[c0inbuster]

@hugolp
you should have a look at PushOver API https://pushover.net/, it works with both Android and iOS.
You just have to install PushOver app on your Android device and use their HTTP API
https://play.google.com/store/apps/details?id=net.superblock.pushover&hl=fr
It's a paid apps but it worth it
you can find some inspiration here
https://code.google.com/p/working4arbitrage/source/browse/trunk/src/pushover_notifier.py
Have a look at priority
https://pushover.net/api#priority

@prof7bit
I'm looking for a way to run goxtool at startup (connected to my MtGox account) in a GNU Screen
I tried

Code:

$ python goxtool --add-secret

I set API and SECRET key
but I'm asked for a password... I can't set a blank password
What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret: 

I would like to bypass this step

[prof7bit]

What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret: 

I would like to bypass this step

You are not the first one to ask me and I am going to implement one of these two options (not sure which one, probably the second one):

(1) A command line option (with a huge warning) to pass the pasword on the command line. It is insecure because the password will end up in the ~/.bash_history if you do this.

(2) The other alternative would be to make a separate .ini setting for the unencrypted secet that is automatically used (and the password question bypassed) when this setting is found in the ini.

[K1773R]

What is the "best" way to do this (even if that's a bit insecure) ?
because when I run goxtool at startup I always get this message

Code:

enter passphrase for secret: 

I would like to bypass this step

You are not the first one to ask me and I am going to implement one of these two options (not sure which one, probably the second one):

(1) A command line option (with a huge warning) to pass the pasword on the command line. It is insecure because the password will end up in the ~/.bash_history if you do this.

(2) The other alternative would be to make a separate .ini setting for the unencrypted secet that is automatically used (and the password question bypassed) when this setting is found in the ini.

how about reading from STDIN?

Code:

python goxtool.py <<< YOURPASSWORDHERE

[prof7bit]

how about reading from STDIN?

Code:

python goxtool.py <<< YOURPASSWORDHERE

Doesn't work for me. I'm not a bash expert, so I'm not sure why it won't work (maybe the reason is how readline internally operates or the python getpass module). Also this would still leave the password in the .bash_history even if it worked.

[prof7bit]

I have added a command line option --password that can be used when starting it from within a script (don't use it on the command line unless you understand the implications)

Code:

./goxtool.py --password=YOURPASSWORD

I have also added two new ini settings (they will be added to the .ini with default values after you start it the first time after update):

Code:

[goxtool]
orderbook_group = 0
orderbook_sum_total = False

The dafault values will display the orderbook exactly like it was in earlier versions of goxtool all the time already, so if you leave them at defaults then nothing will change. orderbook_group will have the same effect as it is used on the clarkmoody website, if you set it for example to 0.1 it will bin the price levels into $0.10 intervals (it won't display empty bins). 0 means no grouping at all, display every price level separately. orderbook_sum_total = True would sum up the total amount of BTC that needs to be bought or sold to reach and break that level instead of printing only the amount at that particular level like it was in earlier versions.

Please git pull to update.

[supert]

I'm very surprised by a load of ~1 for this on a rpi.  A (possible) memory leak surprises me less.  I'll test this myself soon, but I think something is likely going wrong in your configuration.

having run it a bit longer it seems about 0.5 load most of the time and 55%-75% memory (256MB rpi).
The memory usage seems stable so it doesn't seem there is a (serious) leak.

[ErebusBat]

I have added a command line option --password that can be used when starting it from within a script (don't use it on the command line unless you understand the implications)

Code:

./goxtool.py --password=YOURPASSWORD

I would HIGHLY recommend that you store your password on disk (with appropriate file permissions) then you can launch the bot with this command line, keeping the password out of your history file:

Code:

./goxtool.py --password=`cat ~/.mygoxtoolpassword`

Note that those are backticks (same key as the tilde, left of the 1 on most keyboards)  NOT single quotes!

[prof7bit]

Here is something not really serious but someone might still find it useful:

_speaker.py

Code:

"""
_speaker.py - A simple fun bot (no real usage) for goxtool.

This bot will read big market buys and sells loudly, a female voice will read
all buys and a male voice will read all sells that are above VOL_MIN. A 3rd
voice will read own orders that have been filled. It will attempt to sum up
partial trades by waiting until either a new tick message arrives or a trade
in opposite direction.

This will only work if espeak is installed.
On Ubuntu this is apt-get install espeak
"""
import os
import goxapi
import strategy

# minimum trade volume, market orders below that
# volume will be ignored. value is in BTC.
VOL_MIN = 10

# see the manual of espeak to understand how to tweak the voices.
VOICE_BUY   = "-ven+f5"
VOICE_SELL  = "-ven"
VOICE_OWN   = "-ven+f5 -p10 -s150"

# if mbrola us voices are installed these also work
# VOICE_BUY   = "-vmb-us1 -s150 -p70"
# VOICE_SELL  = "-vmb-us2 -s150"
# VOICE_OWN   = "-ven-us1 -s130 -p0"

def say(txt, voice="-ven"):
    """say the text (needs espeak installed)"""
    os.system("espeak %s '%s' 2>/dev/null 1>/dev/null &" % (voice, txt))

class Strategy(strategy.Strategy):
    """text to speech bot"""
    def __init__(self, gox):
        strategy.Strategy.__init__(self, gox)
        self.vol = 0
        self.action = 0

    def slot_trade(self, gox, (date, price, volume, typ, own)):
        prc = goxapi.int2float(price, gox.currency)
        vol = goxapi.int2float(volume, "BTC")

        if own:
            say("Order filled: %0.2f at %0.2f." % (vol, prc), VOICE_OWN)
        else:
            action = {"bid": "buy", "ask": "sell"}[typ]
            if action != self.action:
                self.say_trade()
            self.action = action
            self.vol += vol

    def slot_tick(self, gox, (bid, ask)):
        self.say_trade()

    def say_trade(self):
        """read the last trade (if any). a female voice will read
        the buy orders, a male voice the sell orders"""
        if self.vol >= VOL_MIN:
            if self.action == "buy":
                voice = VOICE_BUY
            else:
                voice = VOICE_SELL
            say("%s %0.0f." % (self.action, self.vol), voice)
        self.vol = 0

[c0inbuster]

I'm not very interested by you "speaker" strategy ... for now... but maybe one day it could be useful...

I tried to play with a Python wrapper for TA-Lib
http://ta-lib.org/
http://mrjbq7.github.io/ta-lib/
this wrapper is total...
you just need to install cython, TA-Lib, and this wrapper

I just did a test strategy for now

Code:

"""
trading robot breadboard
"""

import goxapi
import datetime
import numpy as np
import talib
#import pandas as pd

class Strategy(goxapi.BaseObject):
    # pylint: disable=C0111,W0613,R0201

    def __init__(self, gox):
        goxapi.BaseObject.__init__(self)
        self.signal_debug.connect(gox.signal_debug)
        gox.signal_keypress.connect(self.slot_keypress)
        gox.signal_strategy_unload.connect(self.slot_before_unload)
        gox.signal_ticker.connect(self.slot_tick)
        gox.signal_depth.connect(self.slot_depth)
        gox.signal_trade.connect(self.slot_trade)
        gox.signal_userorder.connect(self.slot_userorder)
        gox.orderbook.signal_owns_changed.connect(self.slot_owns_changed)
        gox.signal_wallet.connect(self.slot_wallet_changed)
        
        gox.signal_orderlag.connect(self.slot_orderlag)
        
        self.gox = gox
        self.name = "%s.%s" % \
            (self.__class__.__module__, self.__class__.__name__)
        self.debug("%s loaded" % self.name)
        
        self.is_lagging = True

    def __del__(self):
        self.debug("%s unloaded" % self.name)

    def slot_before_unload(self, _sender, _data):
        self.debug("%s before unload" % self.name)

    def slot_keypress(self, gox, (key)):
        if key == ord("t"):
            #self.debug("Test")
            len_hist = gox.history.length()
            self.debug("Test history with {N} candles".format(N=len_hist))
            #z = np.zeros(len)
            #self.df = pd.DataFrame(z)
            
            #opn = np.array(gox.history.candles)
            #iterable = (gox.history.candles[i].tim for i in range(len_hist))
            #a_tim = np.fromiter(iterable, ???)

            rng = range(len_hist)
            iterable = (gox.history.candles[i].opn for i in rng)
            a_opn = np.fromiter(iterable, np.float)
            
            iterable = (gox.history.candles[i].hig for i in rng)
            a_hig = np.fromiter(iterable, np.float)
            
            iterable = (gox.history.candles[i].low for i in rng)
            a_low = np.fromiter(iterable, np.float)
            
            iterable = (gox.history.candles[i].cls for i in rng)
            a_cls = np.fromiter(iterable, np.float)

            iterable = (gox.history.candles[i].vol for i in rng)
            a_vol = np.fromiter(iterable, np.float)
            
            a_sma = talib.SMA(a_opn)

            """
            self.tim = tim
            self.opn = opn
            self.hig = hig
            self.low = low
            self.cls = cls
            self.vol = vol
            """
            #self.debug(gox.history.candles[len-1].cls)
            #self.debug(self.df)
            
            self.debug(a_opn)
            #self.debug(a_sma)
            
        else:
            self.debug("someone pressed the %s key" % chr(key))

    def slot_tick(self, gox, (bid, ask)):
        pass

    def slot_depth(self, gox, (typ, price, volume, total_volume)):
        pass

    def slot_trade(self, gox, (date, price, volume, typ, own)):
        """a trade message has been received. Note that this might come
        before the orderbook.owns list has been updated, don't rely on the
        own orders and wallet already having been updated when this fires."""
        pass

    def slot_userorder(self, gox, (price, volume, typ, oid, status)):
        """this comes directly from the API and owns list might not yet be
        updated, if you need the new owns list then use slot_owns_changed"""
        pass

    def slot_owns_changed(self, orderbook, _dummy):
        """this comes *after* userorder and orderbook.owns is updated already"""
        pass

    def slot_wallet_changed(self, gox, _dummy):
        """this comes after the wallet has been updated"""
        pass
        
    def slot_orderlag(self, dummy_sender, (usec, text)):
        """slot for order_lag messages"""
        lag = datetime.timedelta(microseconds=usec)
        lag_max = datetime.timedelta(seconds=60)
        #self.debug(usec)
        if lag>lag_max:
            self.debug("MtGox is lagging {lag}".format(lag=lag))
            self.is_lagging = True
        else:
            self.is_lagging = False

I was very surprised to see that candles are stored in a list...
why not using a Numpy Array ?

It will much efficient to implement indicators on top of a Numpy array than on top
of a Python list.
 
You should add a length for array as Numpy arrays have a fixed size (contrary to python list).

I was also surprised to see output for price such as

Code:

2013-04-26 18:02:23,152:DEBUG:test_strategy.Strategy:[ 13399024.  13590000.  13680000.  13610000.  13649999.  13680000.

In my mind (but I ever said something similar) in strategy" I should have "real" values.

I think that a "layer" is actually missing but I don't know how to implement it.

[prof7bit]

I'm not very interested by you "speaker" strategy

maybe others are - for various other reasons.

I was very surprised to see that candles are stored in a list...
why not using a Numpy Array ?

The entire history was initially only a quick hack to be able to plot the recent 24 hours prices into the chart view. Ideally i would eventually implement somethiing more sophisticated that would provide more history, more different timeframes and then maybe also with numpy arrays. I'm waiting until magicaltux implements the API to query trade history of several different (most commonly used) timeframes in OHLCV format. Until then it remains only a quick and dirty hack to be able to plot the price chart.

Until then your bot needs to construct its Numpy arrays itself (maybe even from a local database or some other web service) and keep it updated with the trade signals. You could write a class that does all this, connects to needed gox signals and has a well architected and intuitive interface for usage by bots and also integrates TA-lib and then instantiate this entire thing from your bot during __init__. Once it is working I could look at it for some inspiration of how to design my own new history class once mtgox has the history api ready.

I was also surprised to see output for price such as

Code:

2013-04-26 18:02:23,152:DEBUG:test_strategy.Strategy:[ 13399024.  13590000.  13680000.  13610000.  13649999.  13680000.

In my mind (but I ever said something similar) in strategy" I should have "real" values.

The format is the one used by mtgox api, all currency values are integers. BTC is an integer of satoshi (1e8), JPY is 1e3, USD and all others are 1e5.