goxtool bot: portfolio rebalancing

[prof7bit]

I'm making separate threads for individual goxtool bots to keep the goxtool thread for goxtool related stuff only. I choose this forum because the goxtool thread was moved into this forum too. (Similar software sometimes gets moved into different forums, I believe there is a forum missing for all kinds of software that is bitcoin related but *not* a bitcoin client/wallet itself)

And here is the balancer bot:
_balancer.py: https://gist.github.com/prof7bit/5395900

This bot will maintain a constant asset allocation ratio Bitcoin/Fiat = 50/50
http://en.wikipedia.org/wiki/Rebalancing_investments#Rebalancing_bonus
It will do so by placing limit orders above and below current price that will restore the 50/50 ratio once price moves there and fills the order. As soon as one order is filled it will cancel the other and then calculate and place 2 new orders above and below that new price. The price distance of the orders is 7%, that can be changed in the variable DISTANCE you should not make it smaller than 2%).

This strategy will produce a small profit in the long run (a few percent per week if there are huge price swings or nothing at all when its going sideways for many days), it seems a promising strategy for BTC/USD because of its insanely high volatility but it will also make your account balance go up and down directly in tandem with BTC price, so prepared to see huge swings in your account. You should write down BTC-price, USD and BTC balance in your account once a week and plot these curves after a few months or years to see the effect. This is not a get rich quick bot, this needs patience!

Installation and usage:

• prerequisites: you are familiar with the usage of goxtool.py
• (most important) update goxtool to the latest version (git pull)
• save the file from the above link as _balancer.py in the same directory as goxtool
• ./goxtool.py --strategy=_balancer.py --protocol=websocket --use-http
• (optional) open a separate terminal and tail -f goxtool.log | grep Strategy to follow the debug output of the strategy module
• cancel all orders and bring your account into 50/50 balance, do so by pressing the b key
• start the bot by pressing the p key

Disclaimer: If the above sounds chinese to you or you cannot read the code or don't understand what it does or you are not a Linux user (needed for goxtool) then algo trading is not for you. You should have the skills to implement this yourself from scratch given the description of the strategy or be able to read, understand and debug every aspect of the existing code or it will only end with a huge disappointment or maybe even a desaster.

As you can see in the code the the bot has some some hotkeys bound to certain actions to control its operations:

i show info about current status (how much out of balance)
c cancel all open rebalancing orders and suspend trading
b rebalance immediately with market order at current price
p place a pair of fresh rebalancing orders and start/resume trading
u manually force update of own order list and other stuff (should only be needed when socketio connection behaves strange or is down or other related problems, not used for normal operations)

like all other goxtool strategies it can be reloaded at runtime without restarting goxtool (for example after editing DISTANCE) by pressing the l key (lowercase L).

As you also can see from the code it is triggering its trades when it has detected the number of orders being exactly 1, this does not happen in the trade message, the user_order message is the last API message that is sent after a series of messages updating the wallet and the trade message (and this signal is also triggered after reconnect), so this is the best place to trigger new orders, knowing that the wallet is updated already (we need the new account balance for the calculations).

When the bot ever comes out of sync, I have seen this happening when price is jumping around too fast and you choose DISTANCE too small and mtgox is under ddos at the same time and the connection becomes really unreliable then you need to cancel all orders (press c to cancel and halt trading, also check with F6 that really all orders have been canceled, maybe use u to manually update if socketio is down and if its screwed up totally then restart goxtool. This should not happen very often) and then check if you need to rebalance (press i and see the log) and if needed manually rebalance (press b) and then resume automatic rebalancing (press p).

[jubalix]

is there a debug mode/test mode where it can just hook into gox data, and i put in a test fiat /btc amount and see how it runs

with mocked up fees and every thing....

I guess my worry is in spikey fast trading with bad connections sometimes computer programs behave in unexpected manners.

I guess the other tool would be a face feed to simulate a jumpy/flaky gox feed....

then we would have a good test environs....

or is this already present?

I'm making separate threads for individual goxtool bots to keep the goxtool thread for goxtool related stuff only. I choose this forum because the goxtool thread was moved into this forum too. (Similar software sometimes gets moved into different forums, I believe there is a forum missing for all kinds of software that is bitcoin related but *not* a bitcoin client/wallet itself)

And here is the balancer bot:
_balancer.py: https://gist.github.com/prof7bit/5395900

This bot will maintain a constant asset allocation ratio Bitcoin/Fiat = 50/50
http://en.wikipedia.org/wiki/Rebalancing_investments#Rebalancing_bonus
It will do so by placing limit orders above and below current price that will restore the 50/50 ratio once price moves there and fills the order. As soon as one order is filled it will cancel the other and then calculate and place 2 new orders above and below that new price. The price distance of the orders is 7%, that can be changed in the variable DISTANCE you should not make it smaller than 2%).

This strategy will produce a small profit in the long run (a few percent per week if there are huge price swings or nothing at all when its going sideways for many days), it seems a promising strategy for BTC/USD because of its insanely high volatility but it will also make your account balance go up and down directly in tandem with BTC price, so prepared to see huge swings in your account. You should write down BTC-price, USD and BTC balance in your account once a week and plot these curves after a few months or years to see the effect. This is not a get rich quick bot, this needs patience!

Installation and usage:

• prerequisites: you are familiar with the usage of goxtool.py
• (most important) update goxtool to the latest version (git pull)
• save the file from the above link as _balancer.py in the same directory as goxtool
• ./goxtool.py --strategy=_balancer.py --protocol=socketio --use-http
• (optional) open a separate terminal and tail -f goxtool.log | grep Strategy to follow the debug output of the strategy module
• cancel all orders and bring your account into 50/50 balance, do so by pressing the b key
• start the bot by pressing the p key

Disclaimer: If the above sounds chinese to you or you cannot read the code or don't understand what it does or you are not a Linux user (needed for goxtool) then algo trading is not for you. You should have the skills to implement this yourself from scratch given the description of the strategy or be able to read, understand and debug every aspect of the existing code or it will only end with a huge disappointment or maybe even a desaster.

As you can see in the code the the bot has some some hotkeys bound to certain actions to control its operations:

i show info about current status (how much out of balance)
c cancel all open rebalancing orders and suspend trading
b rebalance immediately with market order at current price
p place a pair of fresh rebalancing orders and start/resume trading
u manually force update of own order list and other stuff (should only be needed when socketio connection behaves strange or is down or other related problems, not used for normal operations)

like all other goxtool strategies it can be reloaded at runtime without restarting goxtool (for example after editing DISTANCE) by pressing the l key (lowercase L).

As you also can see from the code it is triggering its trades when it has detected the number of orders being exactly 1, this does not happen in the trade message, the user_order message is the last API message that is sent after a series of messages updating the wallet and the trade message (and this signal is also triggered after reconnect), so this is the best place to trigger new orders, knowing that the wallet is updated already (we need the new account balance for the calculations).

When the bot ever comes out of sync, I have seen this happening when price is jumping around too fast and you choose DISTANCE too small and mtgox is under ddos at the same time and the connection becomes really unreliable then you need to cancel all orders (press c to cancel and halt trading, also check with F6 that really all orders have been canceled, maybe use u to manually update if socketio is down and if its screwed up totally then restart goxtool. This should not happen very often) and then check if you need to rebalance (press i and see the log) and if needed manually rebalance (press b) and then resume automatic rebalancing (press p).

[smooth]

I guess my worrie is in spikey fast tradeing with bad conections sometimes computer programs behace in unexpected manners.

There are definitely issues with lag, dropped gox connections, etc.  Some of those are discussed on the main thread, as they aren't really specific to this strategy.

Another rebalancing strategy discussed on the same wiki page might also be interesting to play with:

http://en.wikipedia.org/wiki/Constant_proportion_portfolio_insurance

[1Pakis]

F*ck! I was doing that by hand when I was awake with emotions envolved.
I wanted an emotionless robot to do that for me all day long.
Thanks for that bot.
Donation is coming your way.
https://blockchain.info/tx/b7945ec1febb42338e96d9a05019593751dbf8f3775e56ca74789e2de661b1a0
Thanks again

[1Pakis]

I set DISTANCE at 5.
I was balanced at 126.30
First trade went smoothly at 119.97
New orders opened

[ErebusBat]

So... Did anyone else have the bot freak out and place 30+ orders because of GOX lag today?

[jubalix]

So... Did anyone else have the bot freak out and place 30+ orders because of GOX lag today?

hmm i may have mentioned this.....need to make something like

if(watch.goxLag())
{
trade
}


goxLag()
{

if
{last.ping() > choose interval
return false}

...condition
false

....condition
true

}

etc

[ErebusBat]

So... Did anyone else have the bot freak out and place 30+ orders because of GOX lag today?

hmm i may have mentioned this.....need to make something like

if(watch.goxLag())
{
trade
}


goxLag()
{

if
{last.ping() > choose interval
return false}

...condition
false

....condition
true

}

etc

It wasn't just lag tho. When I looked there was only like 60s.  However I kept getting 503 errors all over the place. 

I think a nice feature (although far from full proof) would be to cancel all orders if there are >2 bot orders, then place them again.  However it is still possible that some of those rouge orders get filled in the meantime.

[whizter]

I like the idea, is it also able to trade BTC/EUR ?

[prof7bit]

I think a nice feature (although far from full proof) would be to cancel all orders if there are >2 bot orders, then place them again.  However it is still possible that some of those rouge orders get filled in the meantime.

I'm trying to understand wat might have happened to cause these symptoms, i don't have any idea yet. It should not trade if there are != 1 open orders, it should not trade if there are != 0 submitted but not yet acked orders.

How did you start goxtool, did you use --use-http or did you send the orders via the streming API? I cannot imagine any scenario where sending an order would *not* increment gox.count_submitted *or* the number of own orders in orderbook.owns. Either the sent order doesn't get acked then count_submitted will be != 0 or there comes the ack with order id for each order then count_submitted = 0 but orderbook.owns has a new element added. In both cases it shoudl stop sending more orders. The only way I could vaguely imagine was if you did not --use-http and submitted via socketio, never received the ack and order ID, a reconnect would reset the count_submitted counter [Edit: actually only a *successful* download of own orders will reset count_submitted, not a simple reconnect] and because of ddos and 502 error you would never get the full own order list download but for some reason all the submitted orders (submitted through socketio and never acked) would later go through all at once [because they were hanging around somewhere on the server in some kind of separate (undocumented?) queue all the time]. Thats why I recommend --use-http because that way you *WILL* get an ack for every order under all circumstances because the http call will either succeed (ack) or fail.

Are you running the latest version of goxtool.py and the latet version of the bot (3rd revision) from the github gist in post #1 of this thread or are you still using one of the older versions from the other thread?

If this happens again please save the log file (*before* you restart goxtool because restarting will truncate the old log)

[prof7bit]

I like the idea, is it also able to trade BTC/EUR ?

Yes, just change the currency in goxtool.ini from USD to EUR.

[c0inbuster]

I don't like the idea of this .ini file

because you can't have 2 goxtool instance running in 2 differents GNU screen
 (one for BTC/USD and an other one for BTC/EUR)

(if you want to do this you will have to copy goxtool)

maybe you should add flag to CLI

Code:

--cur1 BTC --cur2 EUR

or

Code:

--symbol BTCEUR

[c0inbuster]

an other idea could be to allow to pass config file as parameter like this

Code:

python goxtool.py --config /path/to/goxtool.ini

so it will be possible to have a screen with BTCUSD and an other one with BTCEUR

**but** if your strategies are writing files it could be a problem because 2 differents goxtools
are writing on same file...
Let's say that you are implementing a trailing stop strategy...
you will store stop loss price in a file (because you want not to lose value if goxtool is
stopped and started)
let's call sl.txt this file (ini, json or yaml file)

maybe you should add a "name" parameter in goxtool.ini

Code:

[gox]
name = MyGox1
currency = USD
use_ssl = True
use_plain_old_websocket = False
use_http_api = False
load_fulldepth = True
load_history = True
history_timeframe = 15
secret_key = 
secret_secret = 

[goxtool]
set_xterm_title = True

and strategy that writes files (to store data such as last stop price) will store data as
MyGox1_sl.txt

so differents goxtool could work together

(english is not my mother tongue... so please excuse me if it's not very clear)


Adding config parameter is quite easy to do

in goxtool.py file

Code:

    argp.add_argument('--config', action="store", default="goxtool.ini",
        help="name of config file, default=goxtool.ini")

    args = argp.parse_args()

    config = goxapi.GoxConfig(args.config)

So you can now have a goxtoolBTCEUR.ini with

Code:

[gox]
currency = EUR
use_ssl = True
use_plain_old_websocket = False
use_http_api = False
load_fulldepth = True
load_history = True
history_timeframe = 15
secret_key = 
secret_secret = 

[goxtool]
set_xterm_title = True

and call goxtool with this config file as parameter using

Code:

python goxtool.py --config goxtoolBTCUSD.ini

or with my Raspberry Pi running my Bitcoin/Litecoin softwares in GNU Screen

Code:

screen -t goxtoolBTCUSD 50 python /home/pi/src/goxtool/goxtool.py --config /home/pi/src/goxtool/goxtoolBTCUSD.ini
screen -t goxtoolBTCEUR 51 python /home/pi/src/goxtool/goxtool.py --config /home/pi/src/goxtool/goxtoolBTCEUR.ini

[prof7bit]

I don't like the idea of this .ini file

because you can't have 2 goxtool instance running in 2 differents GNU screen
 (one for BTC/USD and an other one for BTC/EUR)

I'm aware of the problem. And there is also the logfile that will be written by each running gox instance. I'm already thinking about possible solutions. Currently the simplest workaround is to either copy the entire folder or start it from within different cwd. This will improve eventually but at the moment I'm still finding new bugs in the client code (handling some erroneous conditions, missed messages, previously unknown error replies from mtgox API that result from various kinds of mtgox outage and other strange mtgox behavior that could happen) these kinds of bugfixes have highest priority.

[c0inbuster]

Thanks...
maybe the "name" parameter in config file is a KISS solution (Keep it Simple, Stupid)
(even for log file)

good luck hunting bugs 

[K1773R]

I don't like the idea of this .ini file

because you can't have 2 goxtool instance running in 2 differents GNU screen
 (one for BTC/USD and an other one for BTC/EUR)

I'm aware of the problem. And there is also the logfile that will be written by each running gox instance. I'm already thinking about possible solutions. Currently the simplest workaround is to either copy the entire folder or start it from within different cwd. This will improve eventually but at the moment I'm still finding new bugs in the client code (handling some erroneous conditions, missed messages, previously unknown error replies from mtgox API that result from various kinds of mtgox outage and other strange mtgox behavior that could happen) these kinds of bugfixes have highest priority.

then how about applying the patch i did send u?

[whizter]

got this running on my raspberry pi now for testing purposes with small amount of funds, and it's working as intended so far.
But I got a little problem: how can I run the bot in background, I can't fork the process to background using "&" because I won't be prompted for the decryption password, and if I close the SSH session the process gets killed.

[arsenische]

got this running on my raspberry pi now for testing purposes with small amount of funds, and it's working as intended so far.
But I got a little problem: how can I run the bot in background, I can't fork the process to background using "&" because I won't be prompted for the decryption password, and if I close the SSH session the process gets killed.

probably you can run it from screen

[c0inbuster]

You can use GNU screen

Code:

$ apt-get install screen

run it using

Code:

$ screen -t goxtool 51 python /home/pi/src/goxtool/goxtool.py

CTRL + a + " to change screen
CTRL + a + c to create new screen

If you want to run it at startup, I can say you what I did
(there is probably a better solution)

put in /home/pi/.screenrc

Code:

screen -t goxtool 51 python /home/pi/src/goxtool/goxtool.py

and add

Code:

su - pi -c "/usr/bin/screen -dmS rclocal"

in your /etc/rc.local

If you consider that what I'm doing for you is valuable you can send me some crypto-coins.
https://sites.google.com/site/working4coins/donate

[ErebusBat]

I think a nice feature (although far from full proof) would be to cancel all orders if there are >2 bot orders, then place them again.  However it is still possible that some of those rouge orders get filled in the meantime.

I'm trying to understand wat might have happened to cause these symptoms, i don't have any idea yet. It should not trade if there are != 1 open orders, it should not trade if there are != 0 submitted but not yet acked orders.

How did you start goxtool, did you use --use-http or did you send the orders via the streming API? I cannot imagine any scenario where sending an order would *not* increment gox.count_submitted *or* the number of own orders in orderbook.owns. Either the sent order doesn't get acked then count_submitted will be != 0 or there comes the ack with order id for each order then count_submitted = 0 but orderbook.owns has a new element added. In both cases it shoudl stop sending more orders. The only way I could vaguely imagine was if you did not --use-http and submitted via socketio, never received the ack and order ID, a reconnect would reset the count_submitted counter [Edit: actually only a *successful* download of own orders will reset count_submitted, not a simple reconnect] and because of ddos and 502 error you would never get the full own order list download but for some reason all the submitted orders (submitted through socketio and never acked) would later go through all at once [because they were hanging around somewhere on the server in some kind of separate (undocumented?) queue all the time]. Thats why I recommend --use-http because that way you *WILL* get an ack for every order under all circumstances because the http call will either succeed (ack) or fail.

Are you running the latest version of goxtool.py and the latet version of the bot (3rd revision) from the github gist in post #1 of this thread or are you still using one of the older versions from the other thread?

If this happens again please save the log file (*before* you restart goxtool because restarting will truncate the old log)

I am using the version from this post: https://bitcointalk.org/index.php?topic=148462.msg1886914#msg1886914

My startup script for the bot:

Code:

./goxtool.py --protocol=socketio --use-http --strategy=_balancer.py

OK.. perhaps it was user error then.  I did notice that the bot did not get the updated orders OR the trade so I canceled the remaining single bot order and DID manually try to "P"lace orders multiple times.  I didn't think it was that high, and I do believe the log indicated that it was also placing the orders itself.   Go and put my son down for a nap... fell asleep with him and my gox history page is full of sells.

It isn't a big deal... like you said things can and will go wrong (esp when dealing with the gox api under load), and in fact I was able to buy it back without loss so it wasn't a huge deal anyway.

So it was very possible it was just PEBKAK.