In this talk Adrian Reid explains that backtesting is fundamentally limited because it only uses historical data, so past profitability does not guarantee future success. He distinguishes good from bad backtesting, illustrating the danger of over‑optimizing parameters—e.g., testing hundreds of thousands of moving‑average combinations and selecting the single best historical fit. That approach leads to overfitting and likely failure when deployed live. Reid stresses that proper backtesting should focus on validating a strategy’s stability, robustness across parameter ranges, the durability of its edge, and the behavior of its volatility, rather than merely maximizing the equity curve. The goal of backtesting, he argues, is to reduce the chance of a strategy failing in real time, not just to showcase past profits.
Chapters
What are the limitations of backtesting? Well, probably the biggest limitation is that it's on past data. So just because it backtests profitably doesn't mean that it's gonna be profitable in the future. It gives, it increases the chances, it tilts the odds in your favor, but there's good backtesting and there's bad backtesting. So for, let me give you an example. If you have a trading strategy, let's say that uses two moving averages, maybe three moving averages even. And you code that strategy into your backtesting software and you test every possible combination of those 3 moving averages and you test, let's say you test 300,000 different combinations of parameter values for that trading rule. And you come up with the one combination that over the last 20 years happened to work the best. Then you say, okay, cool, I've got this great trading strategy. The equity curve looks amazing. I've got really profitable trading results. Let's start trading that in real time. Chances are, as soon as you start trading that in real time, you're going to lose money. Why? Because you didn't backtest correctly. What you did was look for the historically best parameter values and you applied those to the, to the future. What you didn't do is validate, is that system stable over time? Is that system robust across a wide range of parameter values? Is that system Is the edge of that system stable? As in, is it eroding or is it growing over time? Is the volatility of the system increasing and therefore getting more and more dangerous? So you don't want to backtest just to optimize profit or just to improve the equity curve. You want to backtest to eliminate the possi— or reduce the possibility that the strategy is bad failing or about to fail. So the limitations of backtesting really come about when you don't backtest correctly.
