Best practices around backtesting strategies

[whydifficult]

Greetings,

I wasn't sure where to post this question, but I figured finding profitable trade strategies using backtesting would fall under speculating about what the price is going to do. I am working on a trading bot and I'm currently implementing backtesting functionality. My background is in IT and not in trading, so hence I got a couple of questions on how this is normally done.

My current backtesting feature is pretty simple: define a TA strategy with its parameters, define the dates from where to where you want to backtest it and my bot will run the simulations. Currently it works really simple by just receiving a new candle every tick and the strategy can decide to act based on this new candle and all previous ones, this loops until all candles in the timerange are dealt with. When it's done it will spit out the results like so:

Code:

2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) start time:			 2013-04-24 07:00:00
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) end time:			 2013-05-23 16:00:00
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) timespan:			 29 days

2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) start price:			 121.6
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) end price:			 125.44
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) Buy and Hold profit:		 3.158%

2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) amount of trades:		 15
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) original simulated balance:	 245.404 USD
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) current simulated balance:	 281.819 USD
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) simulated profit:		 36.415 USD (14.839%)
2013-06-30 13:25:30 (INFO):	(PROFIT REPORT) simulated yearly profit:	 447.030 USD (182.161%)

So I got a couple of questions for those more familiar with trading

• Is this simple way of implementing backtest solid enough? I would say there is a big gap between fine tuning parameters until the profit % becomes the highest and finding a strategy that would work on the real market. I know backtesting doesn't tell you anything about the future philosophically, but statistically it can be pretty smart, right?
• Isn't it smarter run a bunch of simulations (with different parameters) on the first part of the timerange, and apply the most profitable ones on the second part to get a more accurate number towards the current time and thus current environment of the market
• How should I deal with previous price bubbles, I know another bubble can happen again but I doubt they will look the same every time. Therefor I do not think it is wise to tweak a strategy towards a bubble that happened in the past, right?
• Is the output data something you'd expect on backtesting functionality? Are there things missing that also need to be taken into account

Any feedback is appreciated!

[telemaco]

I think that is a good idea and backtesting is essential. I am doing the same.

This are my model results for the last two months also running with a bot.

Red line is BTCUSD MtGox price.
Blue line is the equity value starting with 10K USD.
Green line is the equity value over a Buy and Hold strategy.

Vertical dotted red and green lines are buys and sells.



These are the results for this model over the past 2 months / 1 month / 15 days and 7 days

I have designed it to maximize results over BH and for a very plain/dead market like we have had for the last two months




I suggest you don't use for backtesting all the bubble data as it will probably confuse the model. We are now at a very plain market it seems. Add maybe some movement from the very last part of the bubble so the model also reacts to some harsh movements. If you use all the bubble it might do some great decisions during the bubble and not a single signal during the plain period and you probably don't want that.

Also make sure the model makes enough buy/sell signals. Only two or three signals might mean that your model has overlearned the data (like if it has been designed specifically for the underlying data and it for example buys at the lowest and sells at the highest giving a very high but not "real" model.) and that probably won't work with new info. If it consistently executes buy/sell signals  duing all the time it probably is good. So add a limitation so it performs for example a minimum of one signal at least every X days.

You might also want to test your model against somewhat random created series of data (or from other stocks) to check how well it performs with those so you cover aswell the unpredicted and it does not perform horribly in a unexpected market situation.


Also take in account the commission and the slippage depending on the amount traded every time.

[zamazama]

back testing is never anywhere near true results.

I have unsuccessfully messed around with metatrader for fx trading and designed some system that looked great when back tested, but fall flat (or eventually fall) over time.

If possible forward test for a month .. don't place trades, rather just record what the trade would have been and work out the PnL from there.

A month lost in trading will mean nothing in the long run if the system is profitable, but can save you big bucks.

[telemaco]

back testing is never anywhere near true results.

I have unsuccessfully messed around with metatrader for fx trading and designed some system that looked great when back tested, but fall flat (or eventually fall) over time.

If possible forward test for a month .. don't place trades, rather just record what the trade would have been and work out the PnL from there.

A month lost in trading will mean nothing in the long run if the system is profitable, but can save you big bucks.

That is true. Nevertheless no system is as horrible trading as myself when i do it manually. I absolutely always buy and sell at the worst possible moment so i use bots.

[whydifficult]

I think that is a good idea and backtesting is essential. I am doing the same.

Thanks for your example, nice to see others doing the same thing. Right now my tool is purely command line and not graphical, but I think when backtesting it is important to communicate what is happening inside the simulation. So I think I'll also add graphs to the backtester so users can study the strategies behaviour across the timerange, instead of just the output.

I suggest you don't use for backtesting all the bubble data as it will probably confuse the model. We are now at a very plain market it seems. Add maybe some movement from the very last part of the bubble so the model also reacts to some harsh movements. If you use all the bubble it might do some great decisions during the bubble and not a single signal during the plain period and you probably don't want that.

It appears that we currently are not in a bubble, but if one were to come (or a crash) suddenly I do think it is important to know that how the strategy will handle harsh situations (I think in a broader sense than one specific part just before or behind a bubble). Also the definition of a crash and bubble is pretty subjective I think, right? What if the there is a more gradual crash (like from now till december the BTC price dropping 10$ every month)?

Also make sure the model makes enough buy/sell signals. Only two or three signals might mean that your model has overlearned the data (like if it has been designed specifically for the underlying data and it for example buys at the lowest and sells at the highest giving a very high but not "real" model.) and that probably won't work with new info. If it consistently executes buy/sell signals  duing all the time it probably is good. So add a limitation so it performs for example a minimum of one signal at least every X days.

Good point, I am currently logging the timespan in days and the amount of trades that happened but not the average time between trades (simple division of those two). Will add this.

You might also want to test your model against somewhat random created series of data (or from other stocks) to check how well it performs with those so you cover aswell the unpredicted and it does not perform horribly in a unexpected market situation.

Good point, especially testing against assets that are in the real stock market at this point as we all hope that one day trading BTC would be just as normal as any other currency (like FX). Will add some datasets for non crypto related assets.

Also take in account the commission and the slippage depending on the amount traded every time.

The bot does take fees into account (same as commission?) which you can configure yourself at the moment, it's not really advanced in that it can take a fee structure and calculate the fees dynamically (based on trade amount across the passed month like how Mt. Gox does it). So I will look into that.

The slippage is a bit harder one, as right now I only need to keep track of candle data. I think that the difference when trading small amounts is negligible but becomes more important when the investment increases. Not sure how to handle this yet, does your simulation take slippage into account?

--

back testing is never anywhere near true results.

I have unsuccessfully messed around with metatrader for fx trading and designed some system that looked great when back tested, but fall flat (or eventually fall) over time.

If possible forward test for a month .. don't place trades, rather just record what the trade would have been and work out the PnL from there.

A month lost in trading will mean nothing in the long run if the system is profitable, but can save you big bucks.

I also think it's pretty hard to predict the future based on the history. However it is the only thing we have (apart from things like fundamental trading). But it at least provides some grip. My bot already supports paper trading (simulating trades in the live market).

But when you are evaluating the results a month later you are doing the same thing as running a backtest for the previous month, right? Because after that month you still don't know if it will work the next month.

[ElectricMucus]

In one way I'm glad this thread exists, in another way sad that I am too exhausted from all the meddling to participate and disappointed that this kind of discussion is only possible here in very rare cases.

Currently I just can throw around some terms, Bayesian non-linear regression & feed forward neural networks.
Oh an visiting classes... 

[oda.krell]

Nice thread. One remark before I add my thoughts on the topic itself:

It appears that we currently are not in a bubble, [...]

...would be a conviction that will make you feel pretty lonely here quickly, I'm afraid. I think nearly everyone here agrees that we were in a bubble until April 11, and  are now either in a correction trend or a full bubble deflation. Practically, I would suggest to exclude (or treat separately, for learning purposes) the time around June 2011 (the bubble + deflation everyone here recognizes), and the run-up and sharp decline around April 11 2013. Maybe other points as well.

I'm in the specification phase of my own trading/analysis system, but not yet programming anything. Here's a number of thoughts I had in the beginning that might apply to your situation as well:

* think about what you mean by "parameter". are you starting your search/optimization from the premise that you only use, as an example, some EMAs and, say, their crossovers, and parameters are which time intervals and number of intervals you range over, or is the method (EMA? SMA? regression on particular time intervals?) itself under optimization?

* if the latter, the search becomes (exponentially, d'uh) more difficult. so you might need to perform separate searches for separate method/parameter combinations. which then leads to the question if and how you want to combine the results from these separate searches.

* finally, the biggest conceptual challenge I'm facing right now is not so much how to train the system to yield the best times for sell/buy signals, but the optimal *volume* of trading. Probably it would help if I knew more about TA or theory of trading, but right now, that's where I have to admit defeat.