Adrian Reid shows how to spot and handle outliers in backtests: find the extreme periods, set them aside, then optimize on normal data. If results change when you add the outliers back in, they're skewing your system. Trend-following strategies especially suffer when outliers push exits too loose.
I want the system fine tuned to normal market behavior, not astronomically good market behavior. So outliers in terms of stocks and in terms of time can make a huge difference. And how you treat them is important. One thing that I like to do is run the backtest with everything in it from the start of time to the end of your in sample. Still don't want to burn my outer sample, but to the the end of my in sample and see if there are any periods of market history where it is just bonkers. Like it's just so good. Because there are times when markets go through just massive bull runs or where they're just consistently so volatile that mean reversion just absolutely kills it and it always works. And those can be really destructive for future trading. So if there's a period of time where the market is just really great, in my first initial test, I just take that and move it to the side and do my optimization on the rest of the in sample period. That's not crazy good, because I want the system fine tuned to normal market behavior, not astronomically good market behavior that just happened to happen for one year. That way you've got a system that's more likely to perform in the future without those precisely brilliant market conditions where it worked well in 1999 or something. So take that set aside with the trade outliers, the really good trade outliers. One of my favorite things to do is to run the optimization with them and then run it again without them and see if I get different answers. If I get different answers, then we know that outlier is skewing the results. If I get the same or very similar answers, then we know that they don't worry too much, that doesn't matter too much. And for trend following, the impact of the massive outlier is often that you have your exit set too loose.
And then if your exits are set too loose, what happens is in real time trading you end up with some really poor quality trades where you have a big move, but then it goes a long way against you before you get out and you end up giving up a lot of profit. I found that optimizing with outliers and trend following just causes me to have inefficient exits and bigger drawdowns in the future than I had in the past.
Sam.
