Backtesting Failures 5 Mistakes That Sabotage Even Proven Trading Systems

What are the pitfalls of backtesting that I should avoid? There's obviously a lot of challenges because if it was easy, everyone would do it and everyone would be a profitable trader. We need to understand what are the pitfalls of backtesting. And there's five that I wrote down leading into this session, which I wanted to really, I guess, emphasize focus on. The first one is blatantly cheating what I would call future leaks. So writing a system that you can't reproduce in real life. So, for example, if you have a system that enters at the Open today, but cheats by looking at what the closing price today is, you know, when you buy at the Open, you don't know what the close is, right, because it hasn't happened yet. But if you look in the backtest it in the historical data, you know what the open, high, low, close is in advance because the data is all there. So when you develop a system and you back test it, the first thing you've got to be careful of is to make sure it's realistic. You've got to be very, really careful to make sure that whenever the backtest is making a decision to buy or sell, that you would know in that moment all the information required for you to place that trade and generate that order. There's a couple of examples that people frequently get wrong when they're developing systems. One of them is that real blatant one is I'm entering at the at the Open today, but the system refers to the closed today as part of its calculations. That's a real blunt, kind of dumb one. A more subtle one is you enter at the Open today and you have a stop loss that's 5 ATRs wide, 5 times the average true range wide. But you're using today's average true range in your backtest. You can't do that because today at the Open, you don't know today's average true range, right? You don't. You won't know today's average true range until the close today, but you're entering at the Open placing a stop loss. That means you have to use yesterday's average true range. So you've got to be really careful and think through the process of what would I know at each point in the trading the trade cycle? And make sure that your system rules only refer to things that you would actually realistically know. And you'll be surprised. I get a lot of systems sent to me by people saying, hey, look at this great system. Can you check it out for me and see if there's anything wrong. And inevitably when it looks amazing, there's always a future leak. It's always cheating by looking forward into the future somehow. And there's so many ways to do it, but just by being aware that you have to check very carefully all the entry conditions, all the exit conditions, and make sure that you're only acting on information you would know at the time so that when you backtest it, your results are realistic. So the first rule of backtesting really is when you run a backtest and you generate the results and it shows that nice, beautiful equity curve. If it's laser sharp and it looks like you're going to be rich by next week, you've done something wrong. That's the first rule. So it took me a long time, but now I have this automatic alarm bell that goes off when I produce a back test. And it's this beautiful laser sharp line. And it makes anything more than 30 or 40% return per year, immediate alarm bells goes off because there's a mistake in the code, guaranteed. So if you just adopt that, if you take nothing else away from this, if you take that, you have alarm bells when it looks really good and you start to get excited, oh, I'm excited, there's something wrong, that's going to help because that will prompt you to look into the code, the backtest and the rules and make sure they're actually realistic. That makes sense. How many of you have done a backtest and gone, wow, that's amazing. I'm so excited about this system, anyone. And then found that it was actually there was a mistake in the code. Yeah. So this is really important. So number one rule of backtesting is have those alarm bells. So we're talking about the pitfalls of back testing, what you should avoid making sure the code is realistic, avoiding future leaks is the first one. The second one is ignoring commissions and slippage. So commissions are actually pretty low now, certainly compared to when I started trading. When I started trading, I was paying $40 to buy and $40 to sell as a minimum commission for stocks in Australia. And at the time, if I wanted to trade US stocks, it was $150 to buy and $150 to sell. It's just crazy town, right? You can't ignore those sorts of commissions. But now with interactive brokers, you might have a $6 minimum on Australian stocks, a $1 minimum of US stocks. You think, yeah, it's nothing. I don't really need to take that into account. Right. Wrong. You do because if you ignore slippage and commissions, what happens is these trading systems that have a really fine edge, like a really small average profit per trade start to look really good and you optimize it and the equity curve gets sharper and it looks like you're going to make more and more money and the average profit per trade is like point 3%. And you think, okay, this system is beautiful. Low drawdown, high returns, I'm going to be rich. Again, alarm bells. If you haven't taken into account slippage and commission when you do, you'll find that the equity curve deteriorates massively, even if it's only a very low commission rate. So my guidance would be, when we're doing this sort of, this sort of work, overestimate the costs in your analysis so that you're pleasantly surprised by what happens in real life. Rather than under estimating the cost and getting cut to pieces by commissions and slippage and losing money. How does that feel? Does that feel right? Yeah. So we definitely want to overestimate the commissions. So that means on stocks, if you're trading short to medium term, as in days to weeks, you might be allowing for half a percent commission for entry and exit. That's going to allow slippage and commission and probably a little bit left over if you're paying. If you're trading high price stocks that are quite liquid, you know it'll be more than enough. But if you're only allowing the minimum, like Interactive brokers, What is it? 0.08% commission with a minimum of a dollar on US stocks, something like that. If you're only allowing that and you haven't allowed slippage, a little bit extra for slippage, you're probably going to underestimate how much you're going to pay. And that means your equity curve, when you do, your back test is going to look too good. And if it looks too good, then you're going to position size too high and you're going to use too much leverage. And then when you do those things and something goes wrong and the market turns down, you're overly optimistic, overly aggressive, and you get, you get a much bigger drawdown than what you'll be comfortable with otherwise. Does that all make sense? Okay, so we've covered pitfalls of trading, future leaks, basically systems cheating, ignoring slippage and commissions. Next big one is overfitting past data with too many rules. Yeah. How many rules have you got in your system? 1, 2, 3, 4, 5, 10, 15. If you've got a system right now, how many Rules has got. Now I have systems that vary from as low as two rules to as high as about 15 rules. Fifteen sounds like a lot, but there's reasons for that. And there's also a lot of history on that system, a lot of trades generated and so it's very carefully analyzed to make sure that everyone makes a difference. But most of my systems are very simple thinking, two to five rules. So we don't want to have lots of rules because the more rules we add, the more we're just overfitting to exactly what happened in the past and the less likely it is to work on future unseen data, which is always going to be a little bit different than what happened in the past. Right? So the fewer rules you have, the simpler it is, the more likely you are to capture the essence of the edge in the market. But the more rules you add, the more likely you are to be capturing the exact nuance of just those stocks and those exact moves in the past, which won't work in real time trading. And the dead giveaway is when you run a back test and it looks amazing and then you start trading in real life and it loses money. That's a sign of the strategy was overfit too complicated. One exception is if you're adding a liquidity filter. So let's say I only want to trade stocks that have a million dollars per day turnover on average. That's required. So you filter out the junk so you don't get a lot of slippage. I'm not so worried about that as a rule. Yeah, it's a rule, but I'm not stressed about that in terms of overfitting past data because it's there to make the system tradable. I'm more thinking about, okay, what's my trend filter is above a blow moving average, what's my entry trigger, the breakout or the dip or whatever it is, how wide is the stop loss, how wides the profit target, how many days in the time stop? All of those sorts of things, position sizing, even. Those are all kind of rules that are important to count. So we want to minimize that, keep it as simple as possible. All systems should be as simple as possible. The next problem is over over optimization by cherry picking parameter values. Okay, so we've got our rules, we've got four to five rules, right? When we optimize them, optimizing means we take the parameters, the input values think like the 200 in the 200 day moving average, the 5 in the 5 ATR stop loss. These are parameters Right. And they're just numbers that we input and we don't know what the best numbers are. The best numbers. So we're going to vary them. Like we might take our Trend filter from 100 day moving average to 150 day moving average to a 200 day moving average to a 250 day moving average. And we might test all of those. We test all of those values by optimizing and seeing which is the best. And choosing the best value of a parameter in your optimization is actually one of the biggest problems in back testing because the best value is a cherry picked value which performed extremely well in the past. And if we choose the very best value from the past, is the future likely to be that good or worse? Probably worse. Because the very best parameter value in the past is probably going to be different to the very best parameter value in the future. So instead of choosing the best, what we want to do instead is, is choose the most stable. Again, let's go back to the moving average example. If we go from 100 and we optimize from 100 to 300 in steps of 20, so we go 120, 140, 160, all the way up to 300 and we look at the compound return, the returns of this, of our system, we're going to get this curve and it's going to kind of look pretty jagged and there might be a spike where the very best parameter value is, but either side of it it falls away. And so we're going to look at this whole profile and we say okay, I see that's the best one, the spike. But what I really want to know is where is the most stable area of performance. So we don't want to optimize for the best. We're not optimized to put out choose our parameter in the center of the most stable area of good performance. Because the past and the future aren't exactly the same. So if we choose a parameter value which is stable, so if at 150 it works, and if at 125 it works, and if 175 it works and they're all similar, it's nice and stable. It doesn't matter that the future is going to be different to the past because our parameter is pretty stable, we can push it up or down a bit and it doesn't matter. That's what we want. We want parameter values, inputs into our systems where the number doesn't matter that much no matter where we put it, plus or minus a bit. It still works. That's a stable system, a robust system that's more likely to work in the future. If you just optimize every single rule in your system and you choose the very best value for each one, the backtest is going to look amazing. But in the future it won't work because all you've done is cherry pick the very best combinations from the past. You're overfitting to the past data and then the system will fall apart. That's what causes those beautiful sharp back tested equity curves to turn into volatile disasters in real time trading. But okay. The fifth major pitfall of backtesting is failing to validate your testing. You've done your backtest, you've done your optimization, you've improved the system as much as you can. Now what you need to do is test it on some other data, some other markets, some data it hasn't seen to make sure that it still works. It still holds up. Because if you test, optimize, fine tune your strategy over here and then you apply to a different set of data, maybe a different market, or maybe a different time period in the same market, and it falls apart and it loses money. What have you done over here? You've overfit it to the past data. But if you develop and optimize and fine tune your strategy and you choose real stable parameter values, simple robust rules at work over a broad range of market conditions, and you've done that well, when you apply that same system to unseen data, it should still be profitable. This is just like developing your system and then trading in future unseen data. If the system doesn't work on old unseen data, it's not going to work on future unseen data. Right? So again, we have to validate our backtest because there's always this risk that we're overfitting what we're doing, overfitting our rules to pass behavior past data, and it might fall apart in the future.