If you backtest a sufficient amount of strategies you're statistically bound to find a "successful" strategy eventually though. Put differently, you can find "patterns" in any snippet of random data if you look long enough, that doesn't make the pattern applicable to larger amounts of random data from the same source though.
How to ensure that the backtesting results are actually sound and not just some random fluke? Serious question, not a rhetorical one.
That's a big problem in social sciences and medicine. It is called "p-hacking". Researchers tweak their models until they come up a result with 95% confidence, and they think they have found something.
The reality is that 95% confidence means that 1 in 20 is just fluke, and they have found the fluke. The same issues applies to tweaking a trading strategy until the back-testing results look good.how can you make such a drastic assumption? what evidence do you have that p-hacking is occurring when a trader backtests strategies? p-hacking involves the
deliberate misuse of data to
knowingly present false findings. it's used to fool
other people.
why would a trader do that? so they can purposefully lose their own money? a self-employed trader has strong incentive
not to do that because improper backtesting (such as deliberately omitting data, as with p-hacking) will only lose his shirt. what you're saying only applies to snake oil salesmen trying to sell indicators to noobs because they have incentive to overstate effectiveness. traders, on the other hand, have no incentive to deliberately misuse backtesting data.
overfitting is a legitimate (and separate) concern though. in the case of either p-hacking or overfitting, one simply needs to test the model against data outside the sample used to develop it, to ensure statistical significance. rigorous testing is not easy, but your assertion that any statistically significant backtesting data must be a fluke is really not grounded in reality.
