I think I'm still not totally with you. Let me try an example.
Let's say you have a series of 4 indicators that were developed strictly from backtesting, or curve fitting. You backtested them over the last 8 years, and over that period of time they were accurate in picking short-term bottoms 115 times, and on 25 occassions they were wrong. That would be an 82% win ratio spread out over 8 years with a "lot" of signals. Assume also that they pretty much consistently achieved that same 82% win ratio in each of the 8 years even though those 8 years included both bull and bear markets. This is still curve fitting, right, which if I am understanding you correctly you don't look favorably upon? But is it not also technical analysis, i.e., the recognition of a pattern that repeats and can be expected to continue repeating? Does the size of the sample have a bearing on whether you would consider it a legitimate noteworthy pattern versus just curve fitting?
Yes, what you are describing is curve fitting. Now unless one knows the distribution of the signals over the bull and bear market periods, it's hard to know what your odds of success are. To give you a simple example:
Lets say, i have a simple bottom picking system that uses RSI and has been backtested over 10 years. Let's say the signal is when the RSI goes below 30 and comes back above it, it's a buy signal. Here are 6 signals out of it. 10 days after the signal was generated, the gains from the signals were
Sig #1 : +4.0%
Sig #2 : -1.0%
Sig #3 : +2.0%
Sig #4 : +3.0%
Sig #5 : +1.0%
Sig #6 : -2.0%
Now, let's assume that signal Sig #1, Sig #2, Sig #3, Sig#4 were generated in a bull context and Sig #5 and Sig #6 was generated in a bear context.
Now looking at all the 6 signals (bull and bear context), it would give a 66% win/loss ratio.
Whereas considering Sig#1 - Sig#4 only (bull context only), would give a 75% win/loss ratio.
Whereas considering Sig#5 and Sig#6 only (bear context only), would give a 50% win/loss ratio i.e a toss of coin.
Now let's say, if we were to enter a protracted bear market and i get two more signals.
Sig #7 : -3.0%
Sig #8 : -6.0%
Now the bear context win/loss ratio becomes 25%, making it a worthless system.
So if the character of the market changes to completely different than from the backtested years, the results could be very different from what you have observed from your backtested years. That's why every mechanical system which works like a charm during a certain period, fails miserably during another period.
It's not to say that backtested systems are useless, but one needs to have statistics for the signals from both bull/bear market contexts and keep those statistics seperate. One should also adapt/modify their systems, when the context changes. That's the reason i am not in favor of pure mechanical systems.
If anyone has a mechanical system with 80% win/loss ratio in both bull and bear contexts, i would really love to see one, cuz i haven't seen one to date. I would strongly encourage them to post their signals in real-time so that we all can verify that.
I met a sales guy, a few years back, who showed me a mechanical system with 85% accuracy. When i tested the system with data from 73-75, it gave me a win/loss ratio of 35%. When i presented my results to the sales guy, his response was "Hey, we are not living in the 70s dude ".