Backtesting pitfalls, Thoughts.... Options

[SideShowBob]

I've been backtesting a few systems mostly on QQQ and SMH (mostly for their high liquidity and high volatility -- the same reasons everyone trades them) but I'm not sure how to interpret the results.

Over the historical period for both the buy and hold return is very poor (obviously) since most of it was during a pretty brutal bear market. However I'm not sure how to interpret system backtesting results for this period.

On one hand a system that can make money at all during that large a decline (especially a long only system!) must be decent. However, it partially benefits from lower exposure, only being in the market 25% of the time. This low exposure then becomes a slight underperformance during a rally like the one in 2003.

What are people's thoughts on a system like this and on backtesting during the bear market in general? I'm thinking maybe if you filter for bull/bear (using 200 day SMA?) then maybe you can decide when to buy/hold vs. trading a system that doesn't do as well in bull markets.

Thoughts?

SSB

[mini-trader]

If you can tolerate a big initial draw down, a system that stays in long or short 100% of the time and trades off long term charts will do wonderful.

[ecpinto]

I've been backtesting a few systems mostly on QQQ and SMH (mostly for their high liquidity and high volatility -- the same reasons everyone trades them) but I'm not sure how to interpret the results. 

Over the historical period for both the buy and hold return is very poor (obviously) since most of it was during a pretty brutal bear market.  However I'm not sure how to interpret system backtesting results for this period. 

On one hand a system that can make money at all during that large a decline (especially a long only system!) must be decent.  However, it partially benefits from lower exposure, only being in the market 25% of the time.  This low exposure then becomes a slight underperformance during a rally like the one in 2003. 

What are people's thoughts on a system like this and on backtesting during the bear market in general?  I'm thinking maybe if you filter for bull/bear (using 200 day SMA?) then maybe you can decide when to buy/hold vs. trading a system that doesn't do as well in bull markets. 

Thoughts?

SSB

A trend following system will need a filter to keep you in trends. You can use a 14 period ADX to determine trendiness of market. Other methods include Vertical Horizontal filter, R2 when using linear regression, Aroon etc.
No amount of backtesting will get you prepared to trade the markets. It will give you a historical perspective and not much else. It will give you the confidence when you put money on the line.
Systems can be complex. My view is take a simple system. One that determines trend and then keeps you in it. you can then apply the system to a broader spectrum of stocks.

[mini-trader]

Here are three [edited: they are weekly, not monthly] charts of S&P 500, Naz comp, and Dow. All have an ADX trading system plotted. 99% percent of the time they are in a buy (green) or sell (red). There are two or three bars neutral (dark blue). The big thing would be to stay commited to the system.

S&P



COMP



Dow

[ecpinto]

Monthly charts are great...but I think most people are looking for swing trading. Using monthy charts to trade IMO is for folks with deeeeeeeeeeeep pockets. Something I dont have..
You could use weekly charts to determine direction and then take daily trades in the direction of the trend.

[mini-trader]

They are weekly charts. That was a typo and I can no longer edit. I rarely even look at monthly charts and have no idea why I wrote that.

[Affirmed]

Alot depends upon the time frame you are using for your back test period. For instance the QQQ began trading in 1990, does your back test go back that far or is it from another time. Buy and hold since then shows a 6 times your money return, 10,000 invested at 5.2 (in 1990) is worth roughly 64,000 today (8/30/04). Naturally you sat through alot in 2000-2002 if you could stand it.

Below is a very simple long only back test using Welles Wilder's Directional Movement with one caveat. If the +DM crosses the -DM up then you use a buy stop 1/8 over the highest value of the previous 12 days. This simple caveat reduced whipsaws quite a bit. You exit the trade when the +DM crosses the -DM to the downside on the opening the next day. And the results over the 14 year period shows a sharpe ratio of 1.38 which is pretty good.

I guess what I am really trying to say is the time period used means alot. Try adding filters to your buy and sell signals and see what turns up. You should look for a sharpe ratio greater 1.



Good Luck
Bill

[david_neeran]

Here are three [edited: they are weekly, not monthly] charts of S&P 500, Naz comp, and Dow. All have an ADX trading system plotted. 99% percent of the time they are in a buy (green) or sell (red). There are two or three bars neutral (dark blue). The big thing would be to stay commited to the system.

an ADX system, is this the conventional ADX system, out there in the
public domain? or, is this a "proprietary" system?

[mini-trader]

It is convention except that I took away all the filters. All it does is plot a buy when +DI (14) is higher than -DI (14) and a sell when -DI is above +DI.

I guess ADX could just be used for your "level of commitment" or amount of funds dedicated to the trade.

This post has been edited by mini-trader: Aug 31 2004, 09:34 AM

[Lurk]

Did some testing of the performance of a simple SMA or EMA system on daily data. Best MA combination for S&P 500 i found was SMA(225) SMA(210) which did yield a non-leveraged annual return of 16%. I tested total 1 milion uniqe SMA/EMA combinations, ie (500x500)x4.

CODE

Signal         Date         Price     MA1     MA2    (Points : Percent : Percent)
BUY SIGNAL  :  19830817  165.29  0.00  152.73     (-0.00 : -0.00 : -0%)
SELL SIGNAL :  19840323  156.86  164.26  164.25  (-8.43 : -0.05 : -5%)
BUY SIGNAL  :  19840328  159.89  164.16  164.16  (-3.03 : -0.02 : -2%)
SELL SIGNAL :  19840411  155.00  163.85  163.85  (-4.89 : -0.03 : -3%)
BUY SIGNAL  :  19841214  162.69  159.70  159.74  (-7.69 : -0.05 : -5%)
SELL SIGNAL :  19880115  252.07  288.00  287.96  (89.38 : 0.55 : +55%)
BUY SIGNAL  :  19880831  261.52  258.34  258.39  (-9.45 : -0.04 : -4%)
SELL SIGNAL :  19900613  364.90  343.73  343.70  (103.38 : 0.40 : +40%)
BUY SIGNAL  :  19900830  318.71  343.80  343.81  (46.19 : 0.14 : +14%)
SELL SIGNAL :  19901001  314.94  341.36  341.36  (-3.77 : -0.01 : -1%)
BUY SIGNAL  :  19901128  317.95  334.73  334.74  (-3.01 : -0.01 : -1%)
SELL SIGNAL :  19910111  315.23  333.70  333.69  (-2.72 : -0.01 : -1%)
BUY SIGNAL  :  19910305  376.72  335.74  335.76  (-61.49 : -0.16 : -16%)
SELL SIGNAL :  19910312  370.03  336.48  336.47  (-6.69 : -0.02 : -2%)
BUY SIGNAL  :  19910617  380.13  343.47  343.49  (-10.10 : -0.03 : -3%)
SELL SIGNAL :  19921110  418.62  413.03  413.02  (38.49 : 0.10 : +10%)
BUY SIGNAL  :  19921204  432.06  413.96  413.97  (-13.44 : -0.03 : -3%)
SELL SIGNAL :  19940815  461.23  460.58  460.56  (29.17 : 0.07 : +7%)
BUY SIGNAL  :  19950206  481.14  458.73  458.82  (-19.91 : -0.04 : -4%)
SELL SIGNAL :  20000516  1466.04  1387.77  1387.68  (984.90 : 2.05 : +205%)
BUY SIGNAL  :  20000523  1373.86  1388.83  1388.84  (92.18 : 0.07 : +7%)
SELL SIGNAL :  20001108  1409.28  1441.58  1441.51  (35.42 : 0.03 : +3%)
BUY SIGNAL  :  20001120  1342.62  1438.61  1438.78  (66.66 : 0.05 : +5%)
SELL SIGNAL :  20010124  1364.30  1424.14  1423.61  (21.68 : 0.02 : +2%)
BUY SIGNAL  :  20020205  1090.02  1164.13  1164.28  (274.28 : 0.25 : +25%)
SELL SIGNAL :  20020226  1109.38  1160.16  1159.88  (19.36 : 0.02 : +2%)
BUY SIGNAL  :  20020726  852.84  1084.84  1085.36  (256.54 : 0.30 : +30%)
SELL SIGNAL :  20020822  962.70  1071.38  1071.25  (109.86 : 0.13 : +13%)
BUY SIGNAL  :  20030530  963.59  885.41  885.66  (-0.89 : -0.00 : -0%)
SELL SIGNAL :  20030619  994.70  892.28  892.14  (31.11 : 0.03 : +3%)
BUY SIGNAL  :  20030721  978.80  899.77  899.78  ( 15.90 : 0.02 : +2%)

Time: 5517 trading days
Points: +2058.99
Trades: 31
Return: +372%