Let's Backtest Some Indicators

[let_me_backtest_that]

I will be posting a series of backtests for various indicators on BTC historical data.   I posted on the Aroon last week, but thought I would begin a larger thread to see if anyone is interested in this research project.   This time I am backtesting the ADX indicator

The ADX is one of the more complicated indicators in a trader's arsenal.   The reason why the ADX is complex is because it is a compilation of many indicators created by J. Welles Wilder in his book New Concepts in Technical Trading.  The book is worth the read because his ideas have been so influential on previous and contemporary traders.  (I also heard from a little bird that it is easy to find a pdf of the book from a basic google search.)

I am going to give a basic overview of the indicator then backtest it on historical BTC data.   The ADX is designed to indicate changes in momentum and the direction and strength of a trend.   The indicator does this by first calculating the Directional Movement Index (DMI).

The DMI generally consists of the difference between two different values the DM+(blue or green) and the DM-(red).  The DM+ and DM- consist of the net difference between the most recent highs and lows.  Changes in momentum are generally detected when the DM+ crosses the DM- and vice versa.    Additionally, the direction of the trend is determined by whether or not the DM+ or DM- has a higher value.

The creation of the ADX is based on the DMI and various smoothing techniques.  This involves taking the difference between the absolute value of the DM+ and DM- and dividing it by the sum of the DM+ and DM-.  The calculation is somewhat complex and can be found in more detail at stockcharts.com

http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:average_directional_index_adx

The traditional interpretation of the indicator involves using the ADX line (green or blue) to indicate a trend.  When the ADX is below 20, then the markets are moving sideways.  If it is above 20, then the markets are trending.  Trading signals are determined when there is a crossover of the DM+ and DM-.   Here is an example:

https://i.imgur.com/QC1vLGn.png

Now that we have the basics of the ADX down let's do some historical research. I am going to do two backtests using the default parameters for the ADX and a basic non-optimized position sizing strategy that trades approx $1,000 (or less) on each trade.  The position sizing strategy helps to focus on the performance of the indicator itself.  Since this is only research on an indicator and not a fully developed system, then our focus consists on how it works on various individual trades which I give below.

I will create two separate strategies.  The first will be to trade when the DM+ crosses above the DM- and sell with the reverse.  Also, we will use the ADX line above 20 as a filter.  Call this ADX Research.   

The second will merely use the DMI without the ADX filter (DMI Research).  As usual, I will use NinjaTrader for the backtest and analysis.  I will post images of the results and provide links for the code below.

Here is a summary of  the results of the ADX results:

https://i.imgur.com/0kLTgHV.png

Here is a summary of each trade

https://i.imgur.com/LwmtVkg.png

Here is short simulation of the strategy on Youtube to give you a picture of each trade.

https://www.youtube.com/watch?v=OCWIXg-xxGg&feature=youtu.be

Here is a summary of the DMI results

https://i.imgur.com/Zkif51J.png

Here is a summary of each trade

https://i.imgur.com/m3gH4sR.png

Here is a short simulation on Youtube to give you a picture of each trade.

https://www.youtube.com/watch?v=9dR97Uq6Gww&feature=youtu.be

Here are some things to notice:

1. The ADX system does succeed at filtering out many whipsaw trades when the market is flat.  However, this increase in % profitable also comes with the cost of less total profit and a lower profit fact.

2. The total drawdown of each system is the same.  Importantly, the ADX filter does not decrease drawdown risk.  The ADX filter does not perform better on a return to risk basis.

3. One reason that the DMI indicator does well is because BTC prices trend and have low volatility when moving sideways.  If these features were to change, then the DMI indicator would do less well.   I am merely pointing out one of the many limitations that occur through backtesting.

You can download the code for on my blog
http://www.signalstrengthfinance.com/better-know-an-indicator-adx/

You can also find a free daily updating link to upload historical BTC data into NinjaTrader here

http://www.signalstrengthfinance.com/bitconnector-bitcoin-trading-on-ninjatrader/

Trade Well and Prosper

[wobber]

Absolutely brilliant! Waiting for more! Thanks!

[HeliKopterBen]

Nice!  You should charge for your services.

[let_me_backtest_that]

DOES MACD BEAT BUY AND HOLD?

The MACD is a fairly straightforward momentum indicator.  It consists of the difference between a 12 and 26 bar EMA.  The difference between these two moving averages is the MACD.  It is smoothed by a 9 Day EMA that is called the "signal line."

The MACD  is usually plotted with a histogram that represents the difference between the MACD and the signal line.  The histogram can give either positive or negative values.  On the traditional interpretation, a buy signal occurs when the signal line crosses the MACD and a sell signal occurs when the signal line crosses below the MACD.   See the example below:

https://i.imgur.com/Llda1qf.png

The MACD is a popular indicator on BTC forums.  How does it hold up historically? Does it beat  buy and hold?  How does a long and short strategy perform?  We will perform two backtests.  The first will be a MACD long and short strategy.  The second will be a long only strategy.  Finally, we will compare this to buy and hold.

We will make the following assumptions: (1) The backtest will cover 1/'1/2012-6/1/2012.  (2) We will reinvest profits beginning with a $1,000 account.  (3) We will keep the default parameters for the MACD.

EDIT: The backtest is performed over 1/1/2012-6/1/2014

How does the long/short strategy perform?

The strategy entered 41 total trades - 59% were profitable- net profit was $9,820- there was a 98% a drawdown and a profit factor of 6.42.  See below

https://i.imgur.com/UcJD6aH.png

I am also attaching a youtube replay of each trade in the system to get a better feel for how the system would've worked.

https://www.youtube.com/watch?v=3mwCZOr-6fc&feature=youtu.be

You will notice that the system trades a different number of BTC depending on the account size.  Once it hit a large drawdown at the end of April 2013, then the system did not have the equity to trade larger quantities of BTC.

https://i.imgur.com/vwQTIqI.png

Now let's backtest the long only system.

The strategy entered a total of 21 trades - 76.19% were profitable - net profit was $135,000,  there was a 19% drawdown and a profit factor of 60.41.

https://i.imgur.com/ZFZLGI5.png

Here is the youtube replay.

https://www.youtube.com/watch?v=jE_eDDAe10w&feature=youtu.be

Ultimately, MACD outperforms a buy/hold strategy as seen below.  The Buy/Hold had a net profit of $126,600 and a drawdown of 73%.  (For Buy/Hold I entered a buy at the open of everyday with the same amount of BTC 202 on each trade to make the performance equivalent. )

https://i.imgur.com/kz70wD9.png

Here are a few takeaways:

1. Ultimately, MACD long only does beat buy and hold on both a net profit and drawdown.   It caught every major uptrend even though it lagged the market.

2. However, MACD long only underperformed buy and hold during the large price increases.  At one point, someone would've seen their BTC holdings drop from 202 to 58 due to price increases, lag and some losing trades.

3.  The drawdown  in USD on the MACD strategy is superior to buy/hold.  Returns should always be related to risk.  Someone trading the MACD would've had a $1,982 drawdown in USD, while the buy/hold drawdown equaled $159,000.

You can download updated BTC historical data formatted for NinjaTrader here:

http://www.signalstrengthfinance.com/bitconnector-bitcoin-trading-on-ninjatrader/

You can download the code for your own analysis on my blog;

http://www.signalstrengthfinance.com/bitcoinchartsmacd/


Trade Well and Prosper

[oda.krell]

Very interesting, as before. Thanks!

"long/short" MACD strategy means on a buy signal, the system buys in with the entire account value, and on a sell signal, it sells and actively shorts? While "long" only means the system only buys and sells, without entering a margin position?

[let_me_backtest_that]

Very interesting, as before. Thanks!

"long/short" MACD strategy means on a buy signal, the system buys in with the entire account value, and on a sell signal, it sells and actively shorts? While "long" only means the system only buys and sells, without entering a margin position?

Yes, thanks for clarifying that.  I think its important to show how a going long and actively selling short with one's full account value can have some pretty horrendous consequences.

EDIT: I had the order backwards in the post.  It now makes more sense.

[oda.krell]

I keep being surprised by how well daily MACD still seems to work for this market, even after it being mentioned and discussed over and over again. In fact, I suspect during the 2013/2014 bear market it would have performed even better than in your 2012 data backtest.

[fallinglantern]

Interesting to read that trading MACD beats buy hold. Does that factor in the tax savings (long term capital gains vs. short term capital gains) inherent in the buy+hold strategy that MACD trading doesn't have?

[BitchicksHusband]

Outperformed "buy and hold DAILY".  That's not a valid measure.  Nobody buys and holds daily.

Also: "The backtest will cover 1/'1/2012-6/1/2012", the cherry-picked deadest period possible for buy and hold.

[oda.krell]

Outperformed "buy and hold DAILY".  That's not a valid measure.  Nobody buys and holds daily.

AFAIK, he doesn't include slippage and fees in his calculations, so assuming no gaps between daily candles, I believe selling at candle close and buying at candle open is just his way to simulate a buy&hold strategy. It is (for the reasons mentioned above) equivalent to "real" b&h.

[let_me_backtest_that]

Outperformed "buy and hold DAILY".  That's not a valid measure.  Nobody buys and holds daily.

AFAIK, he doesn't include slippage and fees in his calculations, so assuming no gaps between daily candles, I believe selling at candle close and buying at candle open is just his way to simulate a buy&hold strategy. It is (for the reasons mentioned above) equivalent to "real" b&h.

Yes, that's right.  I am buy on the open and selling on the close without slippage and commissions to simulate both the gains and losses of buy/hold.  Also, I don't see how the data is cherry picked, not intentionally anyway, I am using the same data set as the other indicator backtests.  The bitstamp data is pretty shoddy before 1/1/2012.   In fact, starting at the beginning of the most recent move to 1,200USD and decline would've given the best results because the MACD has had a very good record of shorting during the recent bear market.

[let_me_backtest_that]

Interesting to read that trading MACD beats buy hold. Does that factor in the tax savings (long term capital gains vs. short term capital gains) inherent in the buy+hold strategy that MACD trading doesn't have?

Good point. I am not factoring in slippage, commissions or taxes which should be thoroughly taken into account before using this as a stand alone strategy.  I want to emphasize that this thread is just for testing indicators not fully developed strategies.

[BitchicksHusband]

Outperformed "buy and hold DAILY".  That's not a valid measure.  Nobody buys and holds daily.

AFAIK, he doesn't include slippage and fees in his calculations, so assuming no gaps between daily candles, I believe selling at candle close and buying at candle open is just his way to simulate a buy&hold strategy. It is (for the reasons mentioned above) equivalent to "real" b&h.

Yes, that's right.  I am buy on the open and selling on the close without slippage and commissions to simulate both the gains and losses of buy/hold.  Also, I don't see how the data is cherry picked, not intentionally anyway, I am using the same data set as the other indicator backtests.  The bitstamp data is pretty shoddy before 1/1/2012.   In fact, starting at the beginning of the most recent move to 1,200USD and decline would've given the best results because the MACD has had a very good record of shorting during the recent bear market.

Change it to first six months of 2013 and see if it still beats it.

[oda.krell]

@BitchicksHusband Let it go, man. Just by eyeballing MACD, I can already tell that during the first 6 months of 2013, b&h would perform better. Then skip forward to 2013/2014, and MACD signal is better again.

What matters are overall profits, and this entire thread is basically an invitation to backtest your own ideas. Instead, you come in here, with your "only b&h makes sense" attitude, and take cheap shots. Why not _download_ the motherf'ing program and test it for yourself?

EDIT: that said, any reason why you don't run the test over the entire trading history, let_me_backtest_that?

EDIT 2: sounded more aggressive than intended. apologies.

[let_me_backtest_that]

@BitchicksHusband Let it go, man. Just by eyeballing MACD, I can already tell that during the first 6 months of 2013, b&h would perform better. Then skip forward to 2013/2014, and MACD signal is better again.

What matters are overall profits, and this entire thread is basically an invitation to backtest your own ideas. Instead, you come in here, with your "only b&h makes sense" attitude, and take cheap shots. Why not _download_ the motherf'ing program and test it for yourself?

EDIT: that said, any reason why you don't run the test over the entire trading history, let_me_backtest_that?

EDIT 2: sounded more aggressive than intended. apologies.

Thanks oda.krell-  You are right, this post is a an invitation to test people's own ideas with a some data and code to get started with.  I am responsible for some of the misunderstanding because the backtest length was a typo.  I am including the MACD backtest info up till 6-29-14.  People can see that it is similar to the other results.

https://i.imgur.com/bFcicsE.png

[Wary]

Thanks for the tests!

I've looked at the MACD video. The indicator seems to be good for quiet periods, but not for the rally time.

At April's rally it
    1 Completely missed the rally's top
    2 Sold exactly at the following bottom
    3 Bought exactly at the following top

At October's rally it was planning to replicated April's disaster   , but was saved by an accident: it was going to sell at the bottom again, but since the rally had two overlapping peaks, it sold around the second peak.

Ergo: 1d MACD is good at quiet periods, but at rally time, because of the lag, it's signals becoming worse than useless (namely harmful).

(Sorry for harsh words. My anger is directed not to participants of this discussion, but to the indicator itself. I guess it wouldn't mind )

EDIT: grammar.

[let_me_backtest_that]

Thanks for the tests!

I've looked at the MACD video. The indicator seems to be good for quiet periods, but not for the rally time.

At April's rally it
    1 Completely missed the rally's top
    2 Sold exactly at the following bottom
    3 Bought exactly at the following top

At October's rally it was planning to replicated April's disaster   , but was saved by an accident: it was going to sell at the bottom again, but since the rally had two overlapping peaks, it sold around the second peak.

Ergo: 1d MACD is good at quiet periods, but at rally time, because of the lag, it's signals becoming worse than useless (namely harmful).

(Sorry for harsh words. My anger is directed not to participants of this discussion, but to the indicator itself. I guess it wouldn't mind )

EDIT: grammar.

My purpose here is to give the backtested information so people can determine the usefulness of the indicator for themselves.  One reason I provide the videos is so that people can see the exact issues that you point out.  I will say this about the MACD, as with any lagging momentum indicator, it will not catch the exact top.  Also, any price that decreases precipitously in one or two bars will give back a lot of gains. 

What I am less comfortable with is the following inference: If the MACD lags (and thereby doesn't sell at the top of a rally), then it is only useful in non-trending periods.  Also, I want to point out that my purpose is not to give  full blown strategy.  I think a person could find various exit parameters alongside the MACD indicator that would be useful.  Just something to think about. 

[Wary]

Thanks for the tests!

I've looked at the MACD video. The indicator seems to be good for quiet periods, but not for the rally time.

At April's rally it
    1 Completely missed the rally's top
    2 Sold exactly at the following bottom
    3 Bought exactly at the following top

At October's rally it was planning to replicated April's disaster   , but was saved by an accident: it was going to sell at the bottom again, but since the rally had two overlapping peaks, it sold around the second peak.

Ergo: 1d MACD is good at quiet periods, but at rally time, because of the lag, it's signals becoming worse than useless (namely harmful).

(Sorry for harsh words. My anger is directed not to participants of this discussion, but to the indicator itself. I guess it wouldn't mind )

EDIT: grammar.

My purpose here is to give the back-tested information so people can determine the usefulness of the indicator for themselves.  One reason I provide the videos is so that people can see the exact issues that you point out.  I will say this about the MACD, as with any lagging momentum indicator, it will not catch the exact top.  Also, any price that decreases precipitously in one or two bars will give back a lot of gains.  

What I am less comfortable with is the following inference: If the MACD lags (and thereby doesn't sell at the top of a rally), then it is only useful in non-trending periods.  Also, I want to point out that my purpose is not to give  full blown strategy.  I think a person could find various exit parameters alongside the MACD indicator that would be useful.  Just something to think about.  

Your back-tests gave us very useful information and it is much appreciated. Especially appreciated is the videos, which gives much more clear picture for table-challenged persons like myself  
I realize that no single indicator can give perfect signals. I also realize that catching the exact tops&bottoms is Holy Grail that can be searched for but hardly can be found.  What I'm trying to figure out about 1d MACD is when it should be used and when avoided. It seems to be it is more useful for quiet time, i.e. not during the parabolic rise or immediately after. It is too lagging for it. The quiet time can be trending, just things should be moving slow enough for the indicator to catch up. Especially good it is for "buy" signals. For violent time and "sell" signals it should be used more cautiously, with more confirmations or not at all. Does it make any sense?

[BitchicksHusband]

First off, Oda, he said it beats buy and hold and I intend to make him prove it more than his simple statements.  He brought it up, he should have expected criticism.

My point: Buy and Hold is easier to perform perfectly than ADX.  So I have the following questions:

How do you perform all these trades?

How do you perform all these trades perfectly? 

Do you ever sleep?

Do you program a bot to do them for you? 

Do you leave your money on an exchange such as Mt. Gox?  If so, your carefully traded stash just went to zero.  If you say Bitstamp, how do you know Bitstamp isn't the next Mt. Gox?

All of these questions must be answered adequately in order to beat buy and hold in a cold storage wallet, which takes zero effort and gives 93% of the same result (by the numbers given).

Mistakes can easily lose you that 7%.

[oda.krell]

@BitchicksHusband

Okay, I'll jump to let_me_backtest_that's defense one more time... I'm starting to feel I'm whiteknighting here :D

Anyway. Your points are all absolutely valid questions.

However, let_me_backtest_that said clearly, that he doesn't provide a fully developed strategy. Call it "a hypothetical strategy based on a single signal", if you want.

This is not just a semantic difference. Understanding the difference between a single indicators's signal and a strategy is the difference between making profitable trades and incurring losses in the long run. I learned that lesson the last year.

The backtest is just the evaluation of a single indicator signal, and it potentially invites the interested trader to continue investigating it. Since the signal all by itself slightly outperforms b&h (in this very theoretical, non-practical setting), it is a somewhat interesting invitation.