Checking Under the Hood

Now it is time to check under the hood and see if this new system holds up to scrutiny.  My system evaluation is reasonably extensive and may take several posts to cover in it's entirety.

Scrutinizing the Industry Breakout Rule

I started this development using industry bullishness (or in this case industry breakout) as a foundation for my system.  Now I find that I have gone in a different direction by implementing market timing rules and a powerful ranking system to achieve high annual profits and low max drawdown.  And in the end it appears, at least on the surface, that the industry breakout rule Fmedian("RSI(7)",#Industry) > 60 is not doing much.  So I am going to take a closer look at whether it is worthwhile to keep the buy rule.  Let's start with a side-by-side comparison of system performance.

The equity curve above starts with $100 trading capital and for both systems grows to $1000+ over approximately 1900 "periods" , each period being one week.  As can be seen the system without the Fmedian industry breakout rule has a slight advantage in total equity.

From the annualized profit per calendar year I cannot draw any conclusions.  Each system had better or worse perfomance on any given year.

Looking at the risk stats, it appears that the system with the industry breakout rule removed has marginally better Sortino Ratio and Alpha. So why would I want to keep the rule when the performance appears to be worse?

System Degradation Testing

To answer this question I decided that it might be a good idea to perform some degradation testing to see how each system responds.  I would first like to address the possibility that the super ranking system isn't so super in the future.  Then I would like to look at the possibility that the finely tuned market timing is slightly off in the future.

A lot about a system and it's relationship with the ranking system can be learned by adjusting the weights of each factor while re-running the back sim.  In some camps this is called sensitivity testing.  This can be a very time consuming activity so I have decided to employ a simpler approach.  Instead I decided to substitute in different ranking systems and compare the outcomes for the two systems.  Here are the results:

Ranking system	Profit Per Trade	% Profitable	Annual Profit	Drawdown
With industry breakout buy rule  FMedian("RSI(7),#Industry) > 60
Olikea's Brainchild	14.09%	62.70%	36.42%	12.56%
Balanced4	6.69%	55.96%	20.22%	14.44%
Technamental	4.76%	51.53%	12.85%	17.24%
Without industry breakout rule
Olikea' Brainchild	12.76%	60.56%	38.62%	15.62%
Balanced4	6.65%	55.57%	21.71%	19.77%
Technamental	3.25%	48.84%	8.85%	23.39%

It can be concluded that the industry breakout rule does have some benefit for combatting drawdown.  In all cases tested, the rule reduced drawdown.  In some by more than 6%.

Now what happens if the market timing rules are "off" i.e. slightly out of tune with the markets.  To test this I changed the benchmark index from S&P500 to Russell2000.  Below is a graph of the two system equity curves with the new benchmark.

Again both systems start with $100 and the vertical axis represents total equity.  The horizontal axis represents periods in weeks.  The concern is quite visibly illustrated at the end of the equity curve.  The market timing didn't do it's job but when the industry breakout rule was in effect there was a 21% drawdown while in the case of the system with no industry breakout rule there was a 27+% drawdown.

The conclusion that I have arrived at is that the industry breakout rule is beneficial but the effect is subtle.  It appears to improve maximum drawdown.  It also seems to slightly improve profit per trade, an important factor for system degradation.  So I will leave the rule in.

Negatives So Far

This post is getting kind of long so I am going to wrap it up soon.  I will be doing some robustness testing in future post(s).  To wrap up for tonight I would like to highlight some negatives I see for this system.

For several years we experienced record low levels of market volatility. For the last year+ market volatility has been rising and we are back at more normal levels (except for December which was extreme).  Now this system relies on market timing rules that are more likely to come into play with higher levels of volatility.  See the graph below for reference.

You can see that the system was in and out of the market during times of high market volatility (2002-2003 and 2007-2008).  The quiet low volatility periods seen for the middle part of this simulation will likely not be experienced in the future. 

What this means is I will have to dump stocks on Monday mornings on a regular basis.  (Not something I look forward to).  The profit per trade will also drop.  This in itself is not a problem given that I may be saving my account, but combined with other factors such as degradation of ranking system performance, the system might ultimately become marginally profitable.  For example, the Technamental ranking system produced a profit per trade of under 5% before slippage and commissions.  Combine this with the whipsaw of market timing in a volatile environment and this system could be a problem.


I hope you enjoyed this post.  I'll be doing some robustness testing next time out.
Steve

Post a Comment Share Article Email Article Permalink