System Robustness

In the previous post I performed some system degradation testing.  This is a taste of what I like to call "System Robustness Testing".  Now it is time to get a little more serious with this aspect of trading system development.

System Robustness

System robustness testing is one of the most important aspects of trading system development. In the last post I swapped one benchmark (SP500) for another (Russell2000) to see what effect it would have. I also tried a couple of different ranking systems. Today I am going to try running the simulation backtest with different subsets of stock universes.

Odd and Even Stock Universes

The first method I tried was to use the rule EvenID = TRUE {or FALSE} to test half the stock universe at a time. See below.

Now I would like to point out here that the EvenID rule is used by some as an "in sample" / "out of sample" development tool. The idea is to set EvenID = TRUE for system development. Once the system is finished and polished up then set EvenID = FALSE. This allows one to develop a system using one set of stocks and verify the system using an out of sample completely different second set of stocks. This is a powerful concept provided you understand that although the stocks are out of sample the time period is not.

I am using the EvenID rule here for robustness testing only as a method for creating a different stock universe than before. Different stocks will be bought and different results will be obtained.

Test	Annualized Profit	Max D.D.	%profit /trade	Realized  Winners	3Yr Sortino  Ratio
All	36.42%	12.56%	14.09%	62.7%	1.52
EvenID	28.48%	13.38%	11.96%	62.78%	1.33
OddID	27.58%	16.61%	11.81%	62.53%	0.9

The results I got using the odd and even ID stock universes were quite good and really reflects on the extremely good ranking system being used. So this is one method of testing on a sub-set of the stock universe.

Excluding Stocks from Previous Backtests

Another method is to perform multiple backtests, each time excluding the stocks bought during the previous backtest. The stock universe is reduced each time and different stocks are bought/sold.   The stock exclusion is done on the review page prior to initiating the simulation backtest as shown below.

I was able to complete two runs with reduced stock universes before the system ran out of steam.

Although the subset universe backtests don't produce the same level of profits overall, I don't see any major drawdowns. For the results I see here I believe that the trading capital is not being fully utilized. (I have a Rank > 98 as a buy rule.) In this case I like to look at the trade statistics to get a better feel for performance.

Test	Annualized Profit	Max D.D.	%profit/trade	Realized Winners	3Yr Sortino  Ratio
All	36.42	12.56	14.09	62.7	1.52
All-1	18.2	15.65	8.83	59.06	0.16
All-2	17.45	9.29	13.91	66.96	1.19
All-3	5.18	5.64	15.16	70.24	0.24

In my opioion the $profit/trade and %winners are quite reasonable.  I'm not sure why the 3 year Sortino Ratio is so bad for the first exclusion run.  But other than that things look good so far.  I attempted a third stock exclusion run and included the results here just to show that the %profit/trade and realized winners were holding up even though the system was incapable of using the trading capital effectively.

Sector Exclusion

The final test I did today was excluding individual sectors from the stock universe one at a time.  I did this by first copying my custom universe and editing it:

I did twelve separate backtests, each time excluding a different sector.  The sectors are listed below.

The Portfolio123 user's interface can be quite cumbersome.  I would like to show you a little trick I use to speed up my testing.  I open up two sessions of Portfolio123 in separate windows.  Yes - I log on twice.  Within one window I go to my custom universe and stay there.  Within the second window I go to the simulation summary.  I also open up a copy of the sector list.  You can find it by selecting the rule "Sector" and selecting the full description.  So I have three windows open on my computer as shown below:

I use a four step process.  First copy the neumonic for the sector of interest.  Then copy it into the custom universe as shown.  Save the custom universe.  Then move to the other window and run the simulation.  Repeat for all the sectors.  Below are the results.

The primary reason for doing this exercise is to make sure the system isn't biased towards individual sectors- energy for example.  I have to say that the results are pretty consistent across the board.

All for tonight.

Steve

Comments

[Dean Llewellyn - The Stock Market Report]

what system are you using for backtesting? I use tradesim, but cant see any reference to the software you are using.

[StockMarketStudent]

Portfolio123.com

[Alisa]

I feel very happy to visit this type of post.I really enjoy to visit your post.Thank you for that.

subcultures