Common Backtesting Mistakes

Welcome back to our series on backtesting a trading strategy. My name is Adrian Reed. This is enlightened stock trading. And in this video I want to talk to you about six common mistakes that traders make when backtesting their trading strategy. But before we go any further, make sure you hit the subscribe button and give this video a thumbs up so that you get notified of all of my future videos because I've got new trading content coming out all the time and I don't want you to miss a thing. So what are these mistakes and what is the impact on you if you make them? Well, all, all of these mistakes have the potential to lose you huge amounts of money. So you don't get this right, you're going to have problems in your live account. So listen up, let's go through the mistakes one by 1. Mistake 1 Ignoring slippage and commissions. I can't tell you how many times I've looked at backtest reports from signal services, from system vendors, from website selling courses that don't include slippage and commissions in their backtest results. Now this is just criminal because it's basically outright deceptive. Why is it deceptive? Because slippage and commissions are a real cost of trading and you cannot trade without them. I mean, yes, there are some brokers in some markets lay your trade commission free, but they don't allow you to trade slippage free. There's always a spread and you need to take into account the fact that you may not always get the price that you really want. Slippage and commission are really, really important. Now what is the impact if you don't include slippage and commission in your backtest results? Well, for a short term high frequency trading system that has a very fine edge, the impact can be catastrophic. If you have a mean reversion system, for example, and the average profit per trade is half a percent and you get 0.25% of slippage on the entry and the exit on average, then your system is not profitable. It's breakeven at best. And that's only with slippage before commission. So the impact of slippage and commission actually is most significant for short term small edge or low expectancy strategies. Strategies like mean reversion, where you're trading many, many signals and they're very short term moves so you get a small positive gain when you win and hopefully a small loss when you lose. Slippage and commission can really destroy the profitability of those systems. So if you buy a system or you're looking at a system to get signals from and they don't adequately account for slippage and commission, then you should run the other way or at the very least backtest it yourself with a generous allowance for slippage and commission before committing real money to the trading. Slippage and commission is a real and significant cost of trading, even though commission nowadays is very small. When I started trading, I was paying $40 to buy and $40 to sell. It was unbelievable. So much harder to make money. But now you can get a couple of bucks or 0.08% commission and there's even commission free. But, but what happens if you ignore slippage and commission?

It makes short term low edge strategies look relatively more attractive. And so you're drawn as a result of ignoring the slippage and commission, you're drawn to strategies which are shorter term, high frequency, small profit per trade. And if you trade those with real money, then you will incur some slippage in commission and the profitability will very quickly fall apart. With longer term systems like a long term trend following system, it doesn't make as much of a difference. It does make a difference, but it's less likely to destroy the system. So the first mistake is ignoring slippage and commission. You must have an allowance. How much? Well, it depends on what markets you're trading, how liquid they are and what commission you have to pay. So first of all, check with your broker and see what commission you're actually going to have to pay on the buys and the sells. Second of all, look at the markets you're trading and note how much slippage on average you're actually getting. If you enter a market order to buy the stock market before the market opens. When the market opens, look at the price you get. Compare it to the official opening price, that is the slippage that you incurred. Monitor the slippage on a few trades, maybe 10 or 20 trades, and see what allowance you should have. And then put that into your backtest. At the very least 0.25% or maybe even 0.5%. Slippage in commission is a good rule of thumb to start with. So that's mistake number one. Mistake number two is allowing survivorship bias in into your backtest results. There's two types of survivorship bias. The first one is index constituency. The second one is stock listing or delisting survivorship bias. Let's talk about them separately. Index constituency survivorship bias is where you back test your strategy on.

The constituents of a certain index. So let's say you want to trade stocks in the S&P 500 so you get the 500 stocks that are currently in the S&P 500 list. And you back test your strategy on those stocks over the last 30 years to see how well it would have performed. Sounds like a great idea, right? Trouble is, the strategy is cheating because 30 years ago you didn't know that in 2023 those 500 stocks were going to be in the s and P500 index. They could have been small caps way back then, but they're in the index now, which means they probably went through a meteoric rise in share price. And you wouldn't have known that in advance if you were trading 30 years ago. So the way you avoid this is to have historically accurate index constituents. So every single day in the backtest, every bar in the backtest, the backtesting software looks at that stock and says, is that stock in the S&P 500 index today? If it is, I can buy that stock. If it's not, I can't buy that stock. Now, the only way that I know that you can really get that data is through nor gate. If you want a link, there's a link in the comments below or on the blog post if you're watching this on my website. So, index survivorship bias. Index constituency survivorship bias is probably the biggest one. The second one is stock listing or delisting survivorship bias. That's where if you trade only the stocks that are listed on the market today, you're ignoring a whole bunch of stocks that existed in the past that have since disappeared. Now, there's several reasons why a stock might have disappeared. Maybe it went bankrupt and the price went to zero. That's probably the worst case scenario. But very commonly what happens is a stock actually gets acquired by another company. Now, what happens if a stock gets acquired? Typically it's at a premium to the share price at the time of acquisition. So often if there's an acquisition, you'll get a little boost in share price and then that stock delists. So it's fashionable to think that you need all the delisted stock data to properly evaluate your trading strategy and so that you don't delude yourself. And while it's way better to have delisted data than not, because you're getting a complete view, what I found is that actually including the delisted stocks in my backtest generally makes my systems better. If you design a system that will only hold a stop in the long side if it's going up and has some sort of exit, like a trailing stop or a stop loss or Something like that that stops you holding a stock going down, down, down, down, down. Then delisting due to bankruptcy is not really an issue because you're going to get an exit signal and you're going to get out well before the stock goes bankrupt. Very rarely will a stock be going up and then disappear in a puff of smoke. Bankrupt in 25 years of trading, it's happened to me twice, both of them. The Shanghai Stock Exchange never happened to me on Australian stocks, never happened to me on US Stocks, never happened to me on Canadian stocks. I'm not saying it can't, but very unlikely. And I've taken tens of thousands of trades so it can happen. And having the delisted data in your data set would show you those situations. But it's not really that big of a deal. The much bigger deal in terms of survivorship buyers is index constituency. So index allowing sellership bias is the second common trading mistake that gets systematic traders into trouble. The third common mistake is allowing look ahead bias. Now, look ahead bias is where your trading rules look into the future. They peek at information that you wouldn't know at the time of the signal. What's an example? Let's imagine you are. Okay, here's a real subtle example. You have a trading strategy that enters at the close of the market. It enters market on close. But in order to calculate the signal to buy or sell that stock at the close, you need the closing price. So if the closing price is above a certain level, then buy the stock on close.

Can you always execute that faithfully? Well, maybe not, because it cheats a little bit. It needs information that's available right now to place a trade right now. And you may not be able to get that in time. The market can close before you get that information. Now, you could approximate that. You could say, okay, in the last five minutes of trade. If this price is above that level, then buy market on close. It's an approximation, but it's not perfect. A worse example or a more serious example is having an indicator that looks forward into the future. Like maybe you reference tomorrow's opening price accidentally. Or maybe you look at tomorrow's moving average. Or a good one is you use.

Today'S average true range to set your stop loss level today. Now this is really subtle but significant because let's say you open your your trade at the, you enter your trade at the open today and you want to place your stop loss in the backtest. You could code your system to set that stop loss as five times the average true range. Right now, five times today's average true range will be a nice stop loss. And that system could perform really, really well. The trouble is, at the Open, when you're setting that stop loss, you don't yet know what today's average true range is. You only know yesterday's average true range. So by using today's average true range, you're cheating. Your system is looking into the future, thereby rendering your backtest incorrect and faulty. And if you use future information to make a decision in your trading now, you inevitably look, it inevitably looks fantastic in the backtest. As soon as you get a backtest result, there's a beautiful smooth line from bottom left to top right. The equity curve is really smooth. Low volatility. Chances are you've got look ahead bias. You've got a future leak in your, in your system code. So you have to avoid this at all costs. How do you do it? The way to avoid future leaks in your data is to look at every rule and say, okay, at the time of the trade, would I know that information? Do I, can I see that on the chart at the time I'm supposed to be placing the trade? If you can't yet see that information on the chart at the time you're supposed to be placing the trade, then it's not yet available and it's a future leak. And that causes your back test to be overly optimistic, which will cause you to be too aggressive and you actually won't be able to make money in real time trading. So look ahead bias or future leak is the biggest.

One of the biggest downfalls of backtesting. If you get it wrong, the next one is, the next mistake is information availability, timing. Now this is like a future leak, but a little more subtle. Let's say, for instance, you're using a trading system and you incorporate some external data. Maybe you incorporate the rate of inflation or the current interest rates or some other fundamental data series you incorporate. You suck that in as a foreign function into your backtest. You use that to generate your.

Trading signals. Now, if you have the information up to date at the time you're supposed to place the trade, great. But some of that fundamental information is released late. So you need to know what the release schedule is and line up the information you have available at the time of the technical signal from your system with the information that's available from the fundamental data source you're using. And if the data source, if the fundamental data takes several weeks or months to come out and be updated, then you got to make sure that you're only using what is currently available. This is really important because if you get this wrong again, it's just like a future leak. You're going to get artificially positive results and you won't be able to replicate them in real life trading. So that's information availability timing. The next mistake I want to talk about is execution timing mistakes. Now this is where your backtest assumes that the trade gets placed at a certain time and gets filled. But in reality you couldn't do that. So you generate a signal at the close, but you place the entry. The backtest is entering at the open on the same day. You know, again, it's like a, it's a future leak of sorts. You're using today's bar to generate a signal, but then you're assuming in the backtest that you enter at today's open. Can't do that in real life trading, I'm afraid to say. And if you do have a mistake like that in your backtest code, you're going to find you get hugely optimistic results. And so that's your clue. As soon as you see hugely optimistic results, very smooth equity curves, really large rates of return with low drawdown, you've probably got a future leak in your data, in your, in your backtest rules. So execution timing mistakes are critical. You must check the logic and make sure you get the data to generate the signal and then place the trade and you can do those steps in, in sequence, in time and you can replicate that in real life trading. If unsure, then try and paper trade your system and see if it works. If you've got all the information you need, great. If you don't yet have the information you need at the time you've got to place the trade, that's your clue to tell you that you've got something wrong. So that's execution timing mistakes. The next mistake and the final mistake I want to talk about today is in sample and out of sample data. Now it's tempting to just throw all of your data into the back tester and optimize your strategy over the entire set of data and generate the best system you possibly can out of that. The problem is if you do that, you don't have any data left to validate your strategy on. So if you test your Data on all 30 years of stock history, you've got over every stock on the market and you optimize it and you fine tune it, you do everything and you've got this great system, how do you then validate that you didn't curve fit it. How do you validate that it works in different sets of conditions? How do you validate that you can have confidence to trade that in the future, which is not going to be the same as the past? You can't, or at least it's very difficult. The only way to do it if you burn all your data like that Is to wait 6 months or 12 months and get some new data coming in and incubate your strategy on that new data and then start trading live once you've got six or four months of history on it. Trouble is, that wastes a ton of time. So a better way is to take your original data set and segregate it into some sort of in sample and out of sample groups. Maybe you use half of the stocks for the design optimization and half the stocks for validation. Maybe you use half the history for optimization and the rest of the history for validation. And there's lots of different mechanisms for doing that. But the key point I want to get across rather than going into all the detail right now, is you must have some out of sample data to validate your strategy on. If you don't, then you need to incubate it for long enough to be sure that you didn't curve fit it and you don't have any faults in your backtest. Having an in sample group and an out of sample group of data and an in sample and out of sample time series time frame is very, very important to make sure that your backtest works effectively. So there's several trading mistakes that trip up most new systematic traders. Go back and watch this video again. Make sure that you're not making any of those mistakes. Because if you do, your backtest is worthless. If you avoid them, chances are your backtest is giving you a really good indication of how profitable your system might be in the future. In the next video in this series, I'm going to talk about data requirements for backtesting. So click and watch that video or scroll down if you're watching on the blog post and check out that next video. My name is Adrian Reed. This is enlightened stock trading. Make sure you avoid the most common backtesting mistakes. That's all for this video. See you in the next one. Bye for now. It.