goxtool bot: portfolio rebalancing

[prof7bit]

I am on commit 2e615b9b6272ac99de88693b148f6aa8f2e41708

This is from friday, you should git pull, I'm updating goxtool almost every day, fixing little bugs and problems.

Starting with ./goxtool.py --protocol=websocket --strategy=_balancer.py.

This is ok. [Edit: I just noticed that the huge delay problem is still not solved, maybe you should continue combining the above with the --use-http option (contrary to what I have said earlier), the --use-http will not hurt anyways and trading commands become slightly more reliable then, seems we still need to keep using http for order/add/cancel for a while until all problems are fixed at mtgox]

the latest _balancer.py is the one linked in post #1 in the github gist https://gist.github.com/prof7bit/5395900.
you can also just clone the gist (its a git repository too, you can fork it or clone it):
git clone https://gist.github.com/5395900.git
and then symlink the _balancer.py from this repository into your goxtool directory (you might need to make local branches if you intend to edit the _balancer.py locally yourself)

I'm going to keep this gist updated when  I find a bug. What makes me wonder is why the two orders have so widely different volume in your log file, this makes me assume you are running an older version of the balancer that did not yet calculate the new center price from the account balances. The new version (linked in post#1) is more rubust (although still not yet 100% perfect).

[ErebusBat]

Thanks for the reply... I thought I had updated, obviously not :/

All updated now... (tool+bot).  Re-balanced and orders placed

I will checkout the gist rather than my butchered solution as it is now, thanks!

[smooth]

It looks to me like the bot doesn't consider the trading fee when calculating the new two distance orders points and the two new order volumes.

This is true. The fees are low enough to be not significant for the calculation of the new acount balance, so I just ignore them and if you choose the distance >> 1.2% there is also enough room for profit for every consecutive buy-sell-sequence.

I think it does adjust to your current balances, so the effect of fees  (as well as any slippage due to lag, downtime, etc.) will be factored into a future trade.

Regarding the issue of trends (some other posts on the thread), if you expect a sufficiently strong uptrend, the better strategy is to go long.  Likewise if you expect a strong downtrend, you should just go long USD (hold 0 BTC).  Both of these are ignoring the possibility of shorting or leverage.  There is a narrow range of expectations where balancing is the best approach. 

[prof7bit]

damn... I introduced a bug. Every goxtool revision between 12 hours ago and just now would break the balancer bot (it would place immense amounts of orders and not detect them as own orders).

Please git pull the latest goxtool, fixed it a minute ago.

[fnordfnordfnord]

Just a comment for those newly experimenting with _balancer. This is not a bug or anything but be careful not to set DISTANCE too small. What will happen on a rally is that you will sell several times before the bot gets a chance to buy, then after the inevitable peak the bot will buy the cumulative of the next buy and the two or three missed buys, but at a price nearer to the peak of the rally than you'd like.

[limpbrains]

Hello, prof7bit, thanks for great tool.

I'm running this bot from yesterday and currently can't see any open orders on my mtgox accout.
Here is trading history: https://dl.dropboxusercontent.com/u/1788271/history_BTC.csv
and Strategy log: https://dl.dropboxusercontent.com/u/1788271/srat.log

Looks like bot have tried to open new orders at 2013-04-25 23:11:32
but I don't have them in my gox page.

Is it correct behavior ?
What should I do to fix this issue?

Thanks

[prof7bit]

and Strategy log: https://dl.dropboxusercontent.com/u/1788271/srat.log

You are running it with much too small account. It needs at least a few hundred USD in the acount, otherwise it would adjust the orders to the minimum allowed order size and if this adjustment is too severe (it wouldn't matter if it adjusted them from say 0.009 to 0.01 but larger adjustments will matter) then the next calculation of the correct center price will be totally off. It will be off by more than DISTANCE and instead of placing limit orders above and below it would place *both* orders below price (or both above price) and that would result in a market order and crazy behavior and BURNING OF MONEY!

As I have already initially stated in my disclaimer: If you don't understand what this bot is doing (if you need to ask what is correct behavior) then you have not read the code! And if you did not read the code you cannot run the bot!

[dwolfman]

[Edit: I just noticed that the huge delay problem is still not solved, maybe you should continue combining the above with the --use-http option (contrary to what I have said earlier), the --use-http will not hurt anyways and trading commands become slightly more reliable then, seems we still need to keep using http for order/add/cancel for a while until all problems are fixed at mtgox]

Just to clarify, does the ini entry "use_http_api = False" line do the same as --use-http would from the command line?

Kinda prefer to not have to type more command line than I need to. 

[prof7bit]

Just to clarify, does the ini entry "use_http_api = False" line do the same as --use-http would from the command line?

True, not False!

use_http_api = True

is the same as --use-http. This will be the default in newer versions (unless you have an old ini file where it is still set to False). The following is what I am currently using:

use_http_api = True
use_ssl = False
use_plain_old_websocket = True

use_plain_old_websocket is the same as protocol=websocket, this is currently the most reliable streamng api server.

I'm also using use_ssl=False (this means http instead of https and ws instead of wss), I only do this because connections and http requests will be slightly faster. The disadvantage of this is my ISP, the BKA (Germany), the CIA, and the NSA and everybody else who is able to sniff my internet connection would be able to know how much money is in my MtGox account but I guess they would be able to find out anyways if they would care about my activities, they would probably just ask MtGox to send them all information and MtGox would probably comply instantly because they live under the permanent fear of losing their license and bank accounts.

[dwolfman]

Thanks, and you did understand what I meant.  Guess I should have just asked about "use-http" 

[1Pakis]

Is the goxtool keeps getting stuck today? Or it 's just me and my system?

[ErebusBat]

Is the goxtool keeps getting stuck today? Or it 's just me and my system?

Seems fine for me... although lots of trades today

[bitmango]

I just installed this bot and set the distance to 3.
I rebalanced and then when I hit p it places orders correctly.

I'm seeing the sell orders go through, but the bot doesn't then place a new order at that point. I'm I right in thinking that this is incorrect behavior? There is a single bid order left in my mtgox account..

Is it due to this?

Code:

Code:
   # as long as there are ANY pending orders around we
   # just do nothing and wait for the next signal
   if count_pending:
            return

Also sometimes, I see 2 ask orders the same, and also 2 bid orders the same as well.

Currently i'm seeing an order go through and then I press c, followed by p to get new orders in. But this kind of renders the bot useless for me...

[ewibit]

Is the goxtool keeps getting stuck today? Or it 's just me and my system?

I have running it now a few days and it made no trades...

[ghostshirt]

Is the goxtool keeps getting stuck today? Or it 's just me and my system?

It's been working OK for me for 2 days. I'm using websocket.

[prof7bit]

I just installed this bot and set the distance to 3.
I rebalanced and then when I hit p it places orders correctly.

I'm seeing the sell orders go through, but the bot doesn't then place a new order at that point. I'm I right in thinking that this is incorrect behavior? There is a single bid order left in my mtgox account..

Is it due to this?

Code:

Code:
   # as long as there are ANY pending orders around we
   # just do nothing and wait for the next signal
   if count_pending:
            return

The following would be correct behavior: If there is only one order left it will cancel that order and then place 2 new orders above and below the new center. (The center is calculated from the account balance, not the price of the last trade but usually it differs only a few cents, the center is the BTCUSD price at which perfect balance would be established).

If you see anything other than exactly 1 bid and 1 ask order  then there is something wrong. If there is only one existing order left and it does not attempt to trade then here is something wrong.

The code snipped you cited makes it wait until there is no more pending order left, this is important, otherwise it would not work at all and probably burn a lot of money! Orders can be pending for several seconds up to many minutes, depending how high the order lag is, the bot will not try to place more orders or do anything as long as there are stlll pending orders in the pipeline, just wait (the o_lag display at the top gives you a hint how much order lag there is currently)

DISTANCE = 3 is very low. Its extremely low, given how the order lag can spike on mtgox when there is much activity and price is moving fast. You will inevitably run into problems with such an extremely low distance. I have set mine to 11 now (because 7 was still too small) and it will trade only a few times per week (and on crazy days that happen every once in a while it might trade a few times a day). The larger the distance the fewer trades it will do, the less problems you will have and the less fees you pay to MtGox (and therefore more profit for you). With DISTANCE=11 I can leave it running unattended for days and can sleep well without worrying.

[prof7bit]

I have running it now a few days and it made no trades...

It does not suddenly begin to "trade". It places 2 limit orders above and below immediately after you initialized it according to the instructions (no need to wait here, you can see that it worked immediately) and then it waits for market to move there and fill one of them. Please read the description and the source code and also insert self.debug() statements at all the places where you are unsure what exactly is happening to learn what the values of the variables are and why it won't work as expected.

If you followed the instrucions then it has placed 2 orders (you see them with F6 or if you zoom out the order book or by looking at the mtgox.com trading web interface). You just have to wait until price moves to fill one of these orders. Then when this happens it will automatically cancel the other remaining order and place 2 new orders above and below. You may not have any other open orders in your account (whose price ends with the MARKER digit) or it will confuse the bot and it wont operate correctly. Also if account balance suddenly changes due to other trading it will confuse the bot and its calculations.

Look at your open orders and cancel or change all orders that have a price ending with 7. If you really need a permanent order to buy a million BTC at $0.00007 in case MtGox gets hacked again then either change this order to $0.00008 or alternatively change the MARKER value to some other number between 1 and 9 that is not used by you, then manually cancel all wrong orders (F6) and reload and reinitialize according to instrutions in post #1.

[ghostshirt]

DISTANCE = 3 is very low. Its extremely low, given how the order lag can spike on mtgox when there is much activity and price is moving fast. You will inevitably run into problems with such an extremely low distance. I have set mine to 11 now (because 7 was still too small) and it will trade only a few times per week (and on crazy days that happen every once in a while it might trade a few times a day). The larger the distance the fewer trades it will do, the less problems you will have and the less fees you pay to MtGox (and therefore more profit for you). With DISTANCE=11 I can leave it running unattended for days and can sleep well without worrying.

prof, I'm experimenting with DISTANCE=2. I know it's s very small distance but it seems working fine even with a lag up to 60 seconds. My thinking is as follows (in a quite generalized way), I get more profits doing 6 trades with 2$ profit each time than only 1 trade with $10 profit. Can we come up with a optimum distance level given a portfolio?

[prof7bit]

My thinking is as follows (in a quite generalized way), I get more profits doing 6 trades with 2$ profit each time than only 1 trade with $10 profit. Can we come up with a optimum distance level given a portfolio?

Its not a function of the portfolio size. The profit per trade is a function of distance d and fees f (f = 2*0.6 = 1.2 at MtGox). Lets first only look at the ideal situation with zero fees for smplicity, then the profit for one trade would be proportional to d².

The number of trades n that happen during a fixed time period is also a function of the distance, lets call this n(d). So the profit would be

profit(d) ~ d² * n(d)

and with fees it would be:

profit(d) ~ (d-f) * d * n(d)

And now we need to know how exactly does n(d) look like? Does it have an 1/d² somewhere? I don't know it. We need to simulate it with real historical bitcoin market data (and maybe also with data from other markets). Maybe someone wants to volunteer and make a quick python (or other) script to simulate it on old data and then run it with a lot of distances (and different MtGox fees) (and over a time period where start price and end price are not too far away) and then plot a few curves:

overall profit vs. distance
number of trades vs. distance
etc.

I was hoping BTC-Engineer would come back and share his results and do more simulations but if he wont come back I will probably do it myself but I'm lazy and probably someone else will do it earlier (and hopefully post his results here)

[supert]

My thinking is as follows (in a quite generalized way), I get more profits doing 6 trades with 2$ profit each time than only 1 trade with $10 profit. Can we come up with a optimum distance level given a portfolio?

Its not a function of the portfolio size. The profit per trade is a function of distance d and fees f (f = 2*0.6 = 1.2 at MtGox). Lets first only look at the ideal situation with zero fees for smplicity, then the profit for one trade would be proportional to d².

The number of trades n that happen during a fixed time period is also a function of the distance, lets call this n(d). So the profit would be

p(d) ~ d² * n(d)

and with fees it would be:

p(d) ~ (d-f) * d * n(d)

And now we need to know how exactly does n(d) look like? Does it have an 1/d² somewhere? I don't know it. We need to simulate it with real historical bitcoin market data (and maybe also with data from other markets). Maybe someone wants to volunteer and make a quick python (or other) script to simulate it on old data and then run it with a lot of distances (and different MtGox fees) (and over a time period where start price and end price are not too far away) and then plot a few curves:

overall profit vs. distance
number of trades vs. distance
etc.

I was hoping BTC-Engineer would come back and share his results and do more simulations but if he wont come back I will probably do it myself but I'm lazy and probably someone else will do it earlier (and hopefully post his results here)

I looked at this before, oddly enough. The size of order follows, roughly, a Pareto distribution. That is, a 1 coin bid is roughly equally likely as a 10 coin as a 100 coin, in a given time period.

There are caveats: I looked at a lot of historical data, so it may be a function of bitcoins being cheaper in the past (so then larger trades in bitcoin terms).

Also, integer trades are much more likely; 1 btc much more than 1.01 and 10 more than 9.7 for example.

Ignoring the integer size preference, I fitted a curve to the trade size data. Where f(x) is the probability that the trade size is greater than x bitcoins,

Code:

f=1-(0.1/x)^2.

So the probability of execution at a price is

Code:

p(price)=min( 1, f(s + bid_size) )

where s is the cumulative bids (i.e. depth) at that price.

You can validate f(x) with the following matlab code:

Code:

% empiral data on probability of trade at size
trades=csvread('../data/recent_trades.csv');
[F,X]=ecdf(trades(:,3));

   
In the end, I don't really use this information in my strategy. I don't think your bots performance under a balancing strategy is that sensitive to where you place the orders, if they are in a reasonable range. I would strongly echo prof7bit's warning about placing them too close together, though. This is because of fees.