Adrian Reid shows how to randomly exclude trades during backtesting to fight curve fitting. He pairs a trailing-stop optimization (20, 40, 60) with a dummy variable (1-5) so each parameter set runs multiple times on different trade subsets, then averages the results in Excel for more reliable optimization.
Questions this video answers
November 18, 2021
Random number < 0.7 (i.e., 70% chance to allow a trade)
20, 40, 60 (optimized from 20 to 60 in steps of 20)
Integer values 1 to 5 (demo); recommends running ~100 in practice
Aggregate results (average, maximum, minimum, standard deviation) across runs; e.g., average compound annual return or CAR/MDD
Chapters
One of the biggest problems when we're developing and optimizing trading systems is curve fitting. And curve fitting often comes because in our compounding backtest we have a limited number of trades that make it into the equity curve. And so when there's only a limited number of trades in the equity curve.
Our optimization choices are almost always skewed by the best trades in the equity curve. So one way to get around this is to run the backtest many times or the optimization many times, but.
Randomly exclude trades from the, from the backtest, so that on each optimization run different trades are included or excluded. So this is really powerful, particularly for systems that either don't have many trades simultaneously or are very long term. So there's not that many trades in the database. And it really helps you run an optimization, get more insight into whether you're curve fitting and trying to avoid curve fitting. So let me show you how you do it and then we'll talk about it some more.
Okay.
So what I've got here is.
I'm just using a really simple system example. I've got a liquidity filter, an index filter and an entry trigger, which is just a breakout. I'm just saying the 20 day moving average is the highest, highest that the 20 day moving average has been for the last two or days. So the 20 day moving average is breaking out. And I've got this random function here. So randomly, every single trade will produce a random number. And if that random number is less than 0.7, then the trade will be allowed. If it's not less than 0.7, the trade will be rejected. Okay, so on its own this is not very helpful. It doesn't do anything, this random function, but when we combine it with optimization and a dummy variable, it becomes really powerful.
Here's what I'm doing. I have a trailing stop here. This is the exit. There's no, there's no indicator exit, just a trailing stop. And I'm optimizing for the trailing stop width. So trailing stop width is an optimization statement and I'm going to optimize from 20 to 60 in steps of 20. Now if you've done a bit of optimization on long term trend following systems before, you'll know that sometimes if you make the exit really, really, really wide, you get some monster trades into the portfolio and you end up with a very small number of trades driving the vast majority of your profit. And so your system runs the risk of getting curve fit to those monster trades. So what I'm doing is instead of just running an optimization on the trailing stop width. I'm also, I also created this dummy variable and this dummy variable doesn't do anything. It's not linked to anything. Okay. It's just dummy variable. And it's also an optimization.
Function. I've just called it. This is a dummy variable and it's going to optimize from one to five in steps of one. So what this means I've got two optimizations that are running simultaneously. The first one is the trailing stop width. The second one is dummy variable. So when the trailing stop width is going to take a value of 20, and then it's going to optimize.
For a dummy variable value of 1, 2, 3, 4 and 5, and then the trailing stop is going to take a value of 40, and then dummy variable is going to take a value of 1, 2, 3, 4, 5, and then the trailing stops can take a value of 60 and the double variable is going to take a value of 1, 2, 3, 4, 5. Why do this? Because each time we run the optimization, the random trade selection is rejecting different trades because it's random. So by optimizing the trailing stop width with a dummy variable, we're getting five times the data here because I'm running it five times now. You can run this a hundred times if you like, and you'll get a lot more data. It's just a question of computing power. So let me show you what this does. I'm going to send this to the analysis window and let's just hit optimize and see what happens. So first thing, I'm going to scroll all the way to the right hand side and I know it's really small, I apologize for that. But you can see the trailing stop value of 20, 40, 60 and the type, the.
The dummy variable is a value of 1, 1, 1. And then. Hang on, sorry, it's just reordered them incorrectly.
And then the trailing stop value goes back to 20 for the dummy variable value of 2, then 40 for the dummy variable value of 2, then 60 and then 20, 40, 60 again. 20, 40, 60 again, 20, 40, 60 again. So the dummy variable has basically forced the optimization of the trailing stop width to run five times. But each time it runs, we get slightly different values. So let me just highlight the 20% trailing stops so that we can see how the values are different. So we've got net profit of 3.09, 3.43, 1.61, 2.11, 1.96. And the reason net profits are different is because every time the optimization is run for, for a trailing stop width of 20. It's, it's randomly rejected different trades.
So if we run the same optimization many times with this dummy variable, we get different trades into the portfolio. Now what we can do is we can take a, an average of the trade statistics across many different optimization runs with many different trade selections and decide on a more significant basis which is the best value for the optimization variable.
So if you just ran this optimization once, you might, you know, you look at this and you say, okay, well, the Net profit is 3 million and the others were 1.5 and 1.8. Clearly, 3 is, is the best. So I'm going to go for a.
Trailer stock value of 20. But that might have been driven by just one monster trade or two monster trades, and you don't know. So by running this many times, rejecting a whole bunch of trades, you get far more confidence because you've got more different portfolio compositions run by the same rules with the different parameter values. You take this into Excel and you can aggregate it up and look at the averages. So here's how, here's how you would do that. Let's just select all of this, copy it, make a new Excel sheet here, paste it in, and what I want to do is insert a pivot table.
And I'll just pop the pivot table down here.
Make this a bit bigger so you can see it.
And in this pivot table.
Let'S, let's look at the trailing stop width there. And let's average the compound annual return.
So this compound annual return, this value here of 3.89, is the average of the five different optimization runs that we did. That includes different trades in the portfolio. And this simile is the average of the five different optimization runs that had a trailing stop with the 40 and so on. So you can now go look at the drawdown and average that.
And you can look at the CAR over mdd, for example.
And average that.
You can say, okay, well, now I've got a lot more data. I know that I'm less likely to be curve fitting because I've run the optimization on many different portfolio compositions and this can show you the trend. Now, clearly, the tighter moving average gives. Sorry, the tighter trailing stop gives you higher car, lower drawdown, and better CAR over mdd. Okay, this is just a nonsensical. That's just a simple example to illustrate how to do this. Often it's less subtle. So, you know, if you, if you've got not many trades fitting into your portfolio and your data is very noisy. Running the optimization with a random trade selection trigger like this and a dummy variable gives you a lot more data to smooth out the noise so you can see through the noise to the signal and actually make a sensible optimization decision.
I ran five values of the dummy variable in this example, but if I was doing this in practice, I'd probably run 100. So you get a lot more data, and by running 100 then you can do something like looking at the average, the maximum and the minimum and the standard deviation of those values so that you can get a sense for okay, well, how wide is the spread and or the skew of results? And can I can I be confident in choosing this value from my optimization?
So that's using the random trade selection switch and a dummy variable to improve your optimization quality and get more data.
