Uses and Abuses of Optimization - The Most Powerful Tool That Most Traders Use Incorrectly

When I sat down to write the presentation about what people do right and wrong with optimization, to be honest, it took me by surprise how much there was, how many different concepts and subtleties that people miss. And so I want to share some of that with you today and I really hope you enjoy. So let's get into it. And a question for you. Have you ever worried about whether or not your backtest results were realistic? So have you ever worried about whether or not your backtest results were realistic? Or maybe you've tested trading rules and they seem to work in the past, but then when you start using them on real time data on the right hand edge of the chart, you suddenly start losing money even though your backtest result was great. Or maybe you've thought that backtesting doesn't really work because it's based on past data and the past is not the same as the future. This is a common concern that a lot of traders have. And another one is have you ever designed or optimized a system only to find that when you test it out of sample on unseen data, the system falls apart? Now, the interesting thing is all of these problems, all of these questions have roots in the same issue, the same root cause, and that is incorrect optimization or misunderstanding of optimization. We're going to cover some of those concepts here today. Some of this you may have seen a little bit before. There may be a new twist. Some of this might be some subtlety that maybe you haven't thought of. But either way, I do hope and expect that you'll come up with something valuable and interesting out of this. So here's what we're going to cover today. Three things. How the industry drives you to fail in system development.

And also in optimization, typically by encouraging to ask the wrong questions. The second thing we're going to cover is what your evil nemesis is as a system trader and the biggest pitfalls you have to avoid, particularly if you want to have a profitable trading system like I assume we all do. I trade purely systematically, I have zero discretion in my trading. And optimizing correctly is therefore a really critical issue for me and no doubt for you as well. And the third point is how to quickly build unshakable confidence. Because it's all fine having a system and optimizing it correctly. But if you don't have 100% confidence in that system, how are you going to feel in weeks like this week when things are going wrong, when your positions are moving against you, when you're getting stopped out of multiple trades? So I Want you to be able to build quickly, unshakable confidence in your system, no matter what the market is doing. And that's the key to consistent, profitable trading in the long run. Okay, just by way of background, I'm not going to spend too much of time on myself because I know you want to get into the content. In 2012, I walked away from the corporate world. And the reason I did was because in that year I worked between 12 and 14 hours a day, every single day, six days a week. And I earned about $230,000 in my job that year. But that year I also earned more than that in my trading. And my trading was taking me 20 to 30 minutes a day and causing zero stress. And so on one hand I was stressed out of my mind, working long hours, feeling like a slave, and on the other hand I was making, I'm not going to call it easy money, but certainly stress free money in my trading. And so it became a very easy decision for me at that point to walk away from the corporate world and trade full time for a living. Through my trading, I've managed to pay off all our debts. We paid off our mortgage a long time ago from trading profits and built a seven figure account. And I've coached and mentored hundreds of traders now all over the world, lots of different countries. My trading still takes 20 to 30 minutes a day. Now I say I'm a full time trader, but really full time is not what it takes. It only takes a few systems and the discipline to follow your rules each day for 20 to 30 minutes. And that's all that it certainly all that I need and all of my students need. And this trading life allowed me in 2019 to spend basically the whole year traveling with my family. So we spent three months in the us, three months in Europe and three months in Asia traveling and trading and seeing the world. We took the kids out of school and homeschooled them for the year, which was very cool. And certainly one of the benefits of trading systematically and building the sort of confidence that we're going to talk about today. So that's all I'm going to say about me. The first principle for optimizing correctly and building that confidence is to ask the right questions. And what I'm talking about here is how the industry drives you to fail in system development and optimization by asking the wrong questions. Make sense? So when you open your trading software, what questions do you naturally ask when you look at the screen and you look at your code and you look at your Systems and your rules typically ask questions like this, right? What's the best indicator? Have you ever asked that? Which indicator should I use? What are the best input parameters for my system? What length of moving average should I use? What rsi? What number of bars should I put in my rsi? And what's my overbought oversold threshold? What should it be? What's the best value? What rules could I add to make my trading system better? What filters could I use to get rid of some of the bad trades to avoid sidestep some of the market conditions? Maybe what we're experiencing this week and what tweaks could I make to make my system more profitable? These are the sorts of questions that you're naturally driven and drawn to ask when you open up the trading software, when you read about optimization and when you look at the code of your trading system. If you've asked if I asked yourself or anyone else any of these questions, give me a yes in the comments, let me know. I just want to make sure I'm on the right track here because when I started systematic trading these were the questions I was asking and this is a big problem. Why is that? The problem is that all of those questions are backwards looking.

Now what does that mean? They're looking at the past. They're looking at the backtest result, which sounds sensible, right? Because we've only got the past. But you need to focus on. So when you're looking backwards you're focusing on proving your backtest results. But the trouble is when you're only looking backwards, you fail to make your trading system any better. And what you need to do instead. Let me share a story here about a student of mine. I'm going to call him Harry, not his real name, because I wasn't sure if Harry was going to be on the call. But Harry came to me quite a few years ago and his situation was interesting. He'd been trading forex using automated systems and just wasn't making any money. And he had a relatively simple system, made complete logical sense and I liked the system. But he used his trading software to optimize the system to find the best parameter values. And this is a very common sort of situation. Now Harry spent or tested for 360,000 odd combinations of indicator, different indicator values, different parameter values being fed into his system and he found was was finding the best parameter values to put into a system and the system would always look amazing in back testing but would lose money as soon as he started trading it. Now why is that? It's because that focus, that optimization, it's all backwards looking, only focused on finding the best historical parameter values. So what I did working with this student was to change the question from what parameter combination gives the best backtest to instead, what rules and parameters give the greatest chance of future profitability? See, we don't really care about the past. The past is an indication of whether the system is any good. But what we want to know is that we have the greatest chance of profit potential in the future with real money, in real time trading make sense? So as soon as we flip that question, the type of analysis that he was doing changed. And so his system started making money in real time because of this renewed focus, this new focus. So let's move on. So you need to focus on the right question when we're thinking about optimization. What is the right question? What rules and parameters will give me the greatest chance of future profitability? And I'll give you some, some more precise techniques about how to do that. But for now I just want that question to sink in. Because if you ask the wrong question, you get the wrong answers. And the quality of everything in our life is determined by the quality of the questions that we ask. And trading is exactly the same. You see online, you go to a Facebook group about stock trading and people asking questions about what stock should I buy, about which indicator should I use, which moving average should I use, should I place a stop loss, should I exit this trade? The market's down 10%, should I get out, should I go to cash? All these questions, unfortunately, are low quality questions and so they get low quality results. Now a lot of you are systematic traders, so you're already getting better results, most likely. But if we ask the right question, even higher level, higher quality of question, we're going to get even better results.

Principle two is to avoid the optimization traps, your evil nemesis in trading. And the biggest pitfalls you must avoid if you want a profitable system. That's what we're going to cover here. This is some of the meat, the meat and potatoes of good optimization in this section. But first I want to just make sure we're all on the same page with some quick definitions. You probably know this, but just so that we're speaking the same language, and if I say something, I just want to make sure you know exactly what I mean. So what is backtesting me? Back testing is applying your trading rules to historical data to see how well they performed in the past. And when I do this, it's always a portfolio back test. So I'll back test over many stocks in a portfolio to see how my portfolio would have performed, basically to simulate as closely as possible how my brokerage account would have performed had I been trading that system real time. What is optimization? Optimization is varying the rules and the parameter values to improve the performance of a trading system. And what is curve fitting? Curve fitting is fitting the rules and parameters of your trading system to past conditions, events, price behavior so closely that the model loses its ability to profit on future unseen data. And this is probably the most important point. Curve fitting is the evil nemesis of system traders. Okay. This is the thing that kills more systematic traders than anything else. And it can be so subtle and it can be painfully obvious at the same time if you know what you're looking for. And so what I'm going to do now is share with you some of the obvious and subtle causes of curve fitting so that you know how to avoid them. Does that sound good? So look at a high level curve fitting can be a result of several things. It's not always just about pressing the optimization button and getting the absolute best historical results and using them in real time trading. There's some real subtleties here. So the first cause can be the optimization technique that you use, the process that you go through. The second cause can be the structural effects, things about the system that cause optimized curve fitting. And I'll go through some examples of that trade inclusion and exclusion, which is a really eye opening concept which I talk about with a lot of my students, which is almost the worst kind of optimization problem. And then also the market dynamics can cause you co fitting optimization problems. So I'm going to go through these one at a time. We'll start with optimization technique. Optimization technique. This is curve fitting that comes from when the parameter values are too finely tuned to past conditions. This is the most obvious form of curve fitting. And I've got a couple of concepts for each type of curve fitting here or optimization problem that I want to share. And the first one is unstable profit peaks. Okay, so the situation is this. With unstable profit peaks, whenever you optimize a parameter, you've got a risk that when you press optimize and you find the best value, that value may be a profit peak which is unsustainable or not really a real illustration or estimation of the system's performance. Look at this chart here on the screen. If you vary the optimization of the parameter value on the horizontal axis and look at a profit measure, whatever that profit measure is on the vertical axis, this Parameter value doesn't really make a lot of difference to the performance of the system at the very far left. If I can draw on this screen, let me see if I can draw here on the very far left of the screen here. Performance decays at the low end of the parameter value range, and there's some evidence that it's decaying at the very far right. So at the extremes of the parameter space, the system performance suffers in the middle. Here you've got this profit peak. And if you just press optimize on your trading software, this is the value. This profit peak here is the value that your trading software is going to tell you to use. But the reality is, in this optimization example, all of these values in this entire region are basically the same. Everything, all the ups and downs are probably just noise. When you just press optimize and you don't really take the time to evaluate where the profit is in the parameter space.

The chances are you're going to end up just selecting an unstable profit peak, which is good performing or performs really well in the past, but it's not broadly representative of what you can expect from the system. What's interesting, let me just get rid of this pen here.

Okay. What's interesting is you've got the profit peak, but the more realistic, the more representative performance level is actually a little down from that profit peak because the surrounding values show you where what you can really expect to achieve when you're trading real time. So when you're selecting your parameter values, what I encourage you to do is intentionally avoid those spiky profit peaks from your backtest and choose a value. Choose a parameter value which is representative of all surrounding.

Parameter values performance in your backtest. And this is why it's so important to visualize the results when you run an optimization. Here's the drawing that I usually start with for new students. And you may have seen something like this in the past. If you took a parameter value like this one and you varied it from a value of 10 to a value of 30 odd, and you look at the historical performance on the left, you've got an unstable profit peak of high performance. Now, it looks great in the backtest. And if you press optimize and let your software choose the best value for you, this is the one it's going to choose. It's going to choose this high profit peak on the left. But it's clearly not sustainable because that requires the precise conditions that were that you got that you had in past trading in order to achieve this profit level. And if the conditions are just a little different either side of it, what happens to the performance of the system? It drops dramatically, as you can see here. So rather, what you want to do is find stable parameter values like you've got out here on the right, so where the performance of the parameter value you choose is very similar to the performance of the surrounding parameter values. And this is the key point. Now, one of the challenges with optimization is visualizing the data, visualizing the results. And if you press optimize on five or six parameters of rules simultaneously, you can't visualize the stability of the parameter space. And I like to optimize one at a time so that I can see this going on and intentionally choose or select parameter values that have stable performance areas. Does that make sense?

So the extreme version of this, which is what my student Harry was doing before he came to me, is brute force optimization. Now, brute force optimization is brute force optimization is what my student Harry was doing when he came to me, which is basically four or five rules in this system, 20 or 30 values or more for each rule, and optimizing all of them simultaneously to try and find the best performance. Now, here's what happens. You remember the profit peak slide that we had at the beginning? Now imagine you had that profit peak and you selected a profit peak for one parameter in your system, and then compound that with another unstable profit peak from another rule and another one. Pretty soon a bunch of random spikes in performance that your software has selected compound on each other to give you a grossly unrealistic backtest result. Just because there's a bit of noise in the data and you've curve fit or you've chosen the value that performed absolutely the best in the past.

That slight advantage through curve fitting compounds with all of those other rules. And so what happens is when you optimize together, you magnify the risk that you're overstating the performance of the system. In your backtest, imagine you get a 10% uplift from each of five different rules that 10% compounds on each other, and the system all of a sudden looks amazing. Now, if you do a brute force optimization in the past, pretty much any trading system will come up looking amazing. It's got to be a pretty bad trading system to not to not optimize well or to optimize to look profitable in a back test when you do a brute force optimization. But think about this analogy. Walk into the most expensive exclusive shopping mall in town, then go to the best jewelry store in that town in that shopping mall. Then go to the best salesperson in that jewelry store and then ask for the best diamond. Now what do you think is going to happen? Do you think you're going to overpay for that diamond? Absolutely. Because you've stacked disadvantages on top of each other over and over again and they compound on each other. That salesman is going to convince you that it's good value even though it's not. The jewelry store already has high rent and high paid salespeople and big markups, and the shopping center already charges high rent and they've got the best, most wealthiest clientele coming in. This is what happens. The disadvantages are stacked against you. And with brute force optimization, this little bit of curve fitting from each different rule in your system really stacks on top of each other and ends up causing grief because your backtest looks amazing, but it's just fitting it closer and closer to past data.

So with brute force optimization, it sounds really clever because you can basically set up your code, set up the parameter space for each rule in your system, press optimize, and you get all of these results in just a couple of hours or a couple of minutes, depending on how fast your computer and your software is. And you can use the technique, but you cannot afford to blindly take the results from your trading software. I use, I use amibroker and teach most of my students on amibroker as well. When you just press optimize on amibroker, it'll give you the best result and all the other results ranked according to whatever profit measure you choose. But the profit measure is only a small part of the battle. And so this is what I really want to share with you in this. You need to do more analysis to find stability in the results, stability in the parameter space. Because if the parameter values you choose in your optimization are not broadly representative of the surrounding parameter values, then your backtest is going to mislead you. It's going to make you think the system is way better than it is. The trouble with that, actually, there's several problems with that. Anyone guess what they are? That's a big guess. If you think that your system is better than it is, what's it going to lead you to do?

Any ideas? Put them in the comments. Overconfidence. Great caller, thank you. And what does overconfidence cause? What are you going to get wrong? Over trading. Good. Over trading in terms of what? Size, frequency, risk per trade, exposure, leverage, all of these things. And when we're overconfident about those things, what Happens.

You have a week like today, and if you're overconfident on a long side system, what happens? You blow up. So getting optimization wrong and not understanding the true performance of the system. And you do that by looking at the surrounding parameter values, parameter sets, not the very best parameter set from your backtest and your optimization look at the surrounding values that surround the best one. That's going to give you a better indication of how well the system performs. And when you've got that, then you're going to do your position sizing more sensibly. You're going to select your, your, your exposure level, your weighting to that system, your overall leverage level. You're going to set all of those things much more conservatively because you're not deluding yourself about how amazing your system is. Does that make sense? As soon as you're overconfident, you're guaranteed that you're going to have a drawdown bigger than you're comfortable with or blow up your account.

Okay.

So the next type of co fitting comes from the structural effects of the system, the way the system fits together. Now, there's quite a few rules that can interact in your system and cause you problems. I don't have time today to go through all of them. I did spend two hours on this with my mastermind, explaining all the different components of the system and how they interact and where they can cause problems. But I do have a few good examples for you. And this will at least be enough to get your mind thinking about how your system is coming together. So the structural effects are curve cutting that results from the system structure or interaction between system components during the design process because the entry interacts with the setup and the entry trigger interacts with the stop loss and the stop loss interacts with the profit target and so on. Make sense? So the first one is the number of trades that you've got. Now one of the things to really keep in mind with trading is that statistics are almost worthless. As much as it would be nice to be able to analyze our trading data with traditional statistics, the variance in the trade outcomes that we can get or that our system can generate over time is just so huge and so variable over time that optimization becomes very sensitive to what is included in the basket which your system actually touches or sees. Now as you probably be aware, you can have a set of trading rules and they could generate thousands and thousands of trades over a 20 year backtest. But when you run a portfolio back test, you're not going to get most of those trades into the portfolio. So I'm talking about trading stocks here. If you're trading the whole universe of stocks, your trading system over 20 years might generate 5,000 trades if it was unconstrained. But you've got a constrained amount of capital, you've got position sizing. So the whole backtest may only have five or six hundred trades in it. And this is the problem, because depending on which trades make it into your backtest, you become very sensitive to the results that or the trades that are in there. So to make a credible optimization decision, you need an awful lot of data. You can't. The fewer trades you have, the bigger the problem you're going to get in choosing the right optimization values. And each additional rule you add requires exponentially more data, exponentially more trades. If we add an extra filter to try and make our system better, we better test that system with the extra rule on even more trades to make sure that we're not just curve fitting to a small number of trades. And then the third part of the problem here is when you're optimized with a compounding equity curve, an equity curve which is like what you trade with, let's say you trade with, let's say, 10% of your account going into each trade. Right. How many trades can you have if you're not taking leverage, you can only have about 10 trades. Now, if you've got 10 trades and one of those is a massive winner, your system result is going to be very dependent on that one massive winner. But if you've got 10 trades and you don't have the massive winner, but instead you've got one massive loser, then your system results and your optimization is going to be very heavily influenced by that massive loser. So you've got to find a way to have your optimization touch more trades rather than less. So one of the things that I do is, is optimize on a compounding equity curve. So I can see how the system would perform traded like a portfolio. But I also optimize on an unconstrained equity curve, which means, let's say, give the system a million dollars worth of capital and make every trade like $1,000 or $10,000, so very small compared to the capital. And when you optimize the system rules like that in an unconstrained way, you're going to get many more trades. The optimization is going to touch many more trades, which means you get better informed about how well the system will perform. Yeah. What the average profit per trade is more likely to be Accurate because you're not being fooled by a small selection of the trades. If you're optimizing and your system is only touching 500 trades, compare that to if you're optimizing and the system touches 2,000 trades in an unconstrained optimization, you're much less likely to curve fit in the second example in the unconstrained. Now, the challenge is someone's going to say which way is better? And my answer is do both. Because you need to understand how a system compounds when you position size properly, but you also need to understand how the parameter space looks, how stable it is, how variable it is when you expose it to all possible trades that the system can generate. Does that make sense? The compounding optimization is like where as your account grows, your position size grows. So 5% of equity or 10% of equity or 1% risk per trade. As your capital grows, the trades get bigger. And you've usually got a small number of trades in the portfolio when you do it that way.

And so therefore there's a high risk of curve fitting. And a linear or unconstrained sort of optimization is where you have a huge amount of capital and every trade is small for the life of the backtest. And that means that all of the trades can make it into the back test in the optimization. And so when you vary the optimization parameters, it affects all of those trades, not just the small number that actually make it into the equity curve. And what that that means is you're not going to. You've got less chance about selecting an unstable profit peak because of one particular trade. But do both. Don't skimp on the time here, because understanding how the compounding works and also how stable the parameter space is using an unconstrained model really help. So the analogy is just watching a batter in baseball hit two home runs and assuming that they're a star batter. Okay, but two home runs can be a fluke. It could just be two good trades that happened to make it into the portfolio that made it look amazing. And then when you optimize that, you could fit to those two amazing trades, but you have to look over many seasons for the batter to see how good a batter they are. And just like in your backtesting optimization, you need to look at how the parameter values affect. Affect the system performance over many trades far more than you could actually fit into the portfolio because you want as much information about your system performance as possible.

Good. That making sense. I know there's a lot in that. So the more complex the system is the more trades you need to give the optimization access to. Now, sometimes what happens is when you create your system, you add your filters and your rules and you constrain the system to only trade in a package particular set of circumstances. And so you may have very few trades in the system, in the backtest. Now, if you want to optimize that system and you've got only a couple hundred trades, you're going to curve fit it. It's almost guaranteed. And so what you need to do is loosen it up, allow more trades into the portfolio or into the backtest so that you can really understand how the system performance changes when you vary each of the parameters in the system. So how can you do that? You can. First thing you can do is you can reduce your position size. I would never optimize with say 20% of equity position sizing, putting five positions, 20% into each position. Even if you were going to trade that way, you shouldn't optimize that way because you're guaranteeing you've got very few trades in the portfolio and so you're going to curve fit to the trades that are in there. So if you want to do a compounding optimization, do it on a much smaller position size, maybe 2 and a half percent or 5%, because then you're going to get a lot more positions into the backtest and into the optimization will touch more trades and so you get a better result, right? A less delusional result, let's put it that way. It may not be better in absolute performance because. But it's going to be less delusional because you're not curve it as much. Right? And sometimes what you can do is you can use a Monte Carlo optimization technique where you run the same optimization many times over and you have a Monte Carlo switch which randomly selects which trades make it in and which trades get rejected from the system. And the cool thing about that is you can optimize then with compounding, but do it many times over with different trades going into the portfolio. And then when you plot that performance or the optimize the parameter space performance, you can get a lot more information about whether the performance is stable or whether it was an unstable profit peak and you curve fit it. And then the last thing you can do is optimize on unconstrained capital or fixed and small position size. And I think I probably covered that already. Okay, so here's the key point. The more trades your optimization touches, the less chance you have of curve fitting. If in doubt, find A way to allow your optimization to touch more trades, and that will help help improve performance. Even simple systems need at least 500 trades. If you've got three or four, three or four rules. I've seen very few systems with only three rules, but if you've got three or four or five rules, 500 trades in your back test is really risky because it's very easy to co fit in that case and only tolerate less trades. If the system is exceptionally stable across a very broad parameter space, you can trade, you can optimize. A system that has less than 500 trades can be done, but what you're optimizing for is not performance. You're checking for stability. And if you've only got a few hundred trades, that system needs to be absolutely stable across a very broad range of parameters. And then it's okay to trade, right? Yeah, you don't have a lot of evidence, but you can trade it probably with a smaller size with a small exposure because you can have confidence in the system if it performs over a wide range of parameter values. So that's, so that's the number of trades with structural effects. The next point is the number of trading rules that you've got. Now, you've probably seen something like this chart on the top, right? And if you remember back to year 10 or year 11 math I think they talked about, if you've got four data points, if you've got a fifth order polynomial or something like that, I can't quite remember the math, then you can fit those data points, any data points with an equation of that complex, complex complexity. So it explains those data points precisely. But when you go outside those data points, and let me just circle this here.

On outside the data points, if we say at the extremes, the equation bears zero resemblance to what's actually happening in the real world, and also in between data points, it bears zero resemblance to what's happening in the real world. And generally, if you want a practical rule that will actually give you predictive value, then a linear line of best fit is probably going to do better than a fancy polynomial equation, right? And so in trading systems, trading system development, it's exactly the same issue. The more rules you add, the more you're explaining and capturing the past performance. The exact trades that were in your backtest. The more simple your system, the more you're catching the general behavior of the market.

That occurred in the past. And if you're capturing the general behavior, like stocks tend to trend and stocks that break out tend to continue to trend and they'll continue to trend and then if they turn down, then you want to get out. A simple trend following system, basically a breakout and a moving average exit, has a pretty strong edge because it's simple. If you start adding filters and filters and rules to try and improve the performance so much, all you're doing is describing the precise stock movements that happened in the past. You're not just, you're not capturing the general behavior that gives you the market edge. And so the more rules you add, the more you ruin your ability to profit. Make sense. The problem here is each additional rule requires exponentially more trades to optimize it with confidence. And I would go so far as to say if you've got five entry rules and one exit, then your system is probably curve fit on the entries, unless you've been exceptionally careful. Right? If you've got two entry rules and four exit rules, then you've probably curved fit the exits to the past data because four exit rules, highly unlikely to be significant.

Make sense. But if you've got two entry rules and two exit rules and a couple of filters, your system is probably going to be okay again as long as you've optimized correctly. And we've already covered the analogy here of the complex math equation. So the idea is what you want to do is capture or describe the general behavior of the market, not the specific behavior of particular trades. Because the more your rules zero in and focus on the particular behavior of the trades, the less predictive value they'll have in the future. Because all you're doing is curve fitting and fine tuning your system to those particular trades and they're gone, they're in the past. You can't trade them. It would be lovely, but you can't. So you need a system which is capturing general behavior, not specific historical moves.

So what I, what I want to encourage you to do here is invest more time simplifying the rules and making sure that they catch the general or describe the general behavior you're trying to profit from in the market rather than testing additional filters and rules and optimizing. And you want to seek the simplest way to capture your ideal move. And this ideal move is a really great concept to help you avoid curve fitting. I haven't gone into a lot of detail here, but let me just pick it up before you start developing a trading system. If you have in your mind what is the ideal moves that I'm trying to catch with this system, and if you've described that to yourself, written it down in plain English, Then your rules should reflect that ideal move. If it's a mean reversion, if it's a long term trend or whatever it is, make sure that your rules reflect that ideal move. Use additional rules and filters sparingly because the more you add, the more you move away from that ideal move that you're trying to catch and the more you tailor it and custom and fine tune it to past price moves which are not going to be repeated in the future. So your system becomes worthless. So aim for four to six rules wherever possible. Keep them fairly general. Make sure that they reflect the ideal move and any rules that you add to improve the performance of your system, you should use with extreme caution and skepticism. And one of the concepts that I describe when I'm teaching new traders to develop systems and working with my mastermind guys is, is everything you add you need to be extremely skeptical about. Why is this wrong? Why is this not working? Why could this be curve fitting? Is this really real? How can I disprove this idea? If you go into it with that mindset, that skeptical mindset, it's going to help you uncover whether or not you've actually curve fit the system or not. And I know it's hard to be cautious and skeptical in trading and then optimistic and positive and happy in the real world, but that's what we've got to do in order to be good traders. We need to be cautious and skeptical of the performance that we're seeing, the back test results that we're getting.

The next thing which is really interesting is trade inclusion and exclusion. Now what I mean by this is let's define these two separately. Trade inclusion is curve fitting that comes from when you have parameters or rules that catch or tune the system to specific trades which are not generally representative of the strategies distribution. For example, you catch a lucky big win. Now sometimes this can happen in a trend following system. Let's imagine you develop a trend following system and you have a really tight stop loss in that trend following system and you run the trend following System over a 25 year backtest and with that really tight stop loss and there's a couple of amazing trends over the last 25 years. And there has been in the Australian stock market and in other markets and if you follow trends, you will catch some of those or you have a good chance of catching some of those big amazing trends in the future. But what happens in your backtest if you've got a really tight stop loss and one of those huge trends just happened to not hit the stop Loss just happened to by random chance you got in and it just went straight up from the entry price and it just went. And it was a big move and it was volatile, went up and down, but it went to the moon up a thousand percent. It was a lucky big win that just happened to not get stopped out. Including that trade in your back test when you're optimizing almost guarantees that you're curve fitting to that trade. Because that one monster move has such a big impact on the total profit of your system that it can't help but skew the results. Make sense? Another example might be if you developed a system and I don't trade cryptocurrencies, but if you developed a trend following system for cryptocurrencies, it would be pretty hard to not curve fit the system to the monster move that they had in that market a little while back. Right? And that may or may not be representative of what's going to happen in the future. I'm not here to talk about cryptos one way or the other, but imagine that it wasn't and that was just a once off. That system that you've optimized for the past that included that one monster trade.

Will be worthless in the future because it was curve fit to that one trade. Now a more subtle one and probably almost more problematic one is trade exclusion optimization or curve fitting. Trade exclusion is curve fitting for when you have parameters or rules and that exclude specific trades which actually are representative of a strategies trade distribution like when you sidestep luckily a loss. Now what's a good example of this? Let's imagine that you trade penny stocks, like really tiny stuff. I'm talking about micro caps. And every now and then one of those companies goes bust, right? It goes bust and goes to zero or has a 95 or 99% drop in share price. Okay, now let's say you've, you've developed your trading system and you add a filter which happens to avoid one or two of these catastrophic losses. The stock, there are a couple of stocks in the backtest which had a 95% drop in price overnight, basically bad announcement, corporate fraud, whatever it was. And you add a filter which removes those two, two or three monster losses from your system. And then what happens? You optimize your system and you get the performance of your system and you look at the backtest and it looks amazing. You think, yes, someone a winner. I'm going to make millions of dollars. I'm going to be rich. I can retire next week. Honey, I'm Resigning from my job, this is what happens, right? And we get all excited and then you start trading. What happens if one of those losses comes along in the future? You've developed your system assuming that won't happen, because your extra filter that you add conveniently and luckily sidestep those in the past. But it doesn't mean it will always sidestep them in the future, because some of those big losses don't conform to a particular pattern. They don't come after a particular indicator, crossover or whatever. Right? So if monster losses could be part of the true distribution of trade outcomes from your system, like if you're trading microcat, then you want to make sure that you're not diluting yourself by adding a filter which removes those from the database from your backtest. Because if you do, what's going to happen, you're going to be too aggressive on your position sizing, you're going to be too optimistic about how much leverage you can use. You're going to assume your drawdown is going to be small, when in fact it could be huge. And everything is going to be too aggressive and chances are you can blow up your account. So trade exclusion is actually one of the most insidious forms of curve fitting and over optimization, because you're deluding yourself about how bad it can be. And we know if you've been trading for more than a couple of years, you know that the most important thing is to be prepared for the worst that can happen. Because if you're not prepared for the worst that can happen, you're not going to be able to be, to stay in the game and to be profitable in the long run, you've got to survive. You're going to be in the game to win it. If you get knocked out of the game, can't win, it's game over. Right. So excluding trades by curve fitting your rules, to ignore a couple of trades, to remove them is a killer for sustainable trading. Right. So you have to avoid that. It's probably the most important one in my view. So trade inclusion exclusion, positive outliers. So we've covered both types of trading. Sorry, we've covered both types of included trades that you could either include or exclude as a curve fitting. So what are you going to do about this for positive outliers? If too much profits, too much of the profit of the system comes from too few of the trades, you've got a really high risk of curve fitting. And if you're curve fitting, you're getting a false expectation about how good the system is and all of those position sizing risk management problems that come from it. So my rule of thumb is if more than 10% of the total system profits come from one or two trades, then you've got a very high risk that you've curve fit the system. And so the solution is to take those trades, remove them from the backtest and then re optimize. And when you re optimize, if you get the same or similar parameter values as your ideal, then you know that you're okay, right? You didn't cope it to those, those big trades, the outliers that were in there. But if you take the outliers, the positive outliers out of your system and you re optimize and you get drastically different parameter values, you know you've got a big problem, right, because you were curved to those couple of outliers. And if things don't play out exactly the same way in the then your system won't perform make sense. Okay, so when we're optimizing, first remove huge positive outliers so you don't curve it to a few large trades. That way you're more likely to get parameter values which give you sustainable performance in the future. Okay, now the trade include the avoided negative outliers is again more troublesome, right? So occasionally a stock might collapse by 50% or more or a stock might, you know, get delisted all of a sudden. I've been trading for 18 years, actually almost 20 now. And on several occasions I've been holding a stock which has been in a strong uptrend and it's been suspended and then delisted because of a corporate fraud or something crazy happened, just basically never traded again. And if I didn't have those sorts of things in my backtest and if I didn't position size and manage my portfolio thinking that those types of events could happen, they can wipe you out. We want to have in mind that those things are possible. They may not be likely, but possible because then even if we have curve fit, we're not going to be too aggressive in our portfolio management. Occasionally what happens, we find a parameter or rule that improves the system by removing some of those big negative traits. And that looks great because it reduces our drawdown, makes the equity curve looks better. But the trouble is those shocks can actually happen in your portfolio. So what I like to do is again, be suspicious, right? We've got to be detectives when we're developing systems. Be suspicious of any rule that improves the system only by removing a small number of large losing trades, because chances are it's a fluke. It's just luck that those rules happen to take those three or four large negative trades out. Right? So what you're going to do is run your optimization and look at the trade distribution over the whole parameter space. And if your chosen parameters have much smaller losses than the broad distribution of losses would be across all the different parameter values, then you've got to be suspicious about curve fitting because you've avoided some large negative outliers. Okay? So again, you can't just press optimize and take the results. You've got to optimize and look at what's going on as you vary the values and make sure whether or not your chosen rules or the filters that you've added have avoided some of these large negative outliers. Okay, I hope that makes sense as well. So just because our backlist avoids them doesn't mean we'll avoid them in the future. Shocks are unexpected, right? You don't know when they're coming. Who would think that from an absolute all time equity high or absolute all time high in the stock indices this week or in the US stock indices anyway, we'd have a drawdown or a dip in the market this strong over just a couple of days. It's unexpected. Okay. Some people might say, yeah, I called it good. Can you do that consistently over 20 years? Okay, maybe not. But most drops like that haven't occurred in the past from a high. From an all time high. It was unexpected. And if we had rules that avoided sudden large drops in our back test in the past, we would have thought our system was amazing. And then we would have got to this week and what would have happened, we would have experienced those sudden large drops. And so that's actually part of system performance. So you've got to check whether or not you're diluting yourself by looking at the full range of values of the parameter values when you're optimizing. Okay? So any rules that remove a small number of shock adverse trades are suspicious, highly suspicious. And filters are only valid. So when you want to filter a market, filter like a rule that says the stock index has to be above the X day moving average or the stock should have volatility above or below a certain level. Any overlays that you put in your system to make the performance better, they're only valid if they impact a large number of trades spread over a long period of time. If you add a rule or a filter to your system and it impacts a Handful of trades in a precise period of time. That filter is not real. It's no good. It's probably not going to perform in real time. A filter you add to your system has to affect many trades over a wide time frame. A timeline in your backtest. That's how you know it's significant, right? Your rules have to be significant.

Now the last, the last cause of curve fitting is this market dynamics. Now market dynamics is curve fitting from designing a system to a particular set of market conditions that may not be readily repeatable. Now there's a couple of examples here. So let's talk about unusual market moves. So imagine you developed a system that profited from the 2008 bear market in stocks or the technology stocks rally and bull market and then also maybe the tech bust in 2000 leading up to 2000 and then the decline. If you developed a system around those market dynamics that may or may not happen again in your trading lifetime, right? It's not a consistent and repeatable behavior. So you're going to be really careful. Developing a short side system is something that you that requires caution. Because if you develop a short side system that only trades in a bear market when stocks are really getting beaten up, you don't actually have a lot of data to vet, test, validate and optimize that system. So you can do it, but you just require an extra level of caution. And I think a bear market system, trading short equities is probably one of the conditions that requires the most caution. So when you're developing a system, it can look great like this because of the one huge move. If you look at what happened if you have a short selling system on US equities that shorts in the bear market, basically the system goes sideways or loses a bit of money until 2008 where it rockets up and then it basically goes sideways, makes a little money go sideways for the rest of the period of time. You're basically curve fitting your rules to this profit spike in when the bear market happened. Now I'm not saying that it's impossible to have a system that does legit, right? But you've just got to be cautious. You need to make sure that the system is exceptionally stable. Don't fool yourself that by optimizing the system and making it better and better and better, your optimization that's going to be repeatable because you don't have enough data to know that you need to test your system over a wide range of parameter values and make sure that all of those parameter values perform well because you've got such a narrow time period and such a small number of trades that you need to be make sure the system is exceptionally stable. Otherwise it's probably just curve fit. I'm just going to say if the system.

The fewer trades the system has and the fewer number of trades that generate all the profit and the smaller the time window that the system generates most of the profit in, the more suspicious you should be. It's not necessarily about accuracy or not, because you can curve it on the entry to get it more accurate. You can curve fit on the exit to capture monster trades, right? You can curve in any aspect of the system. So your system, the curve could manifest in an artificially high profile percentage of winning trades, or your curve fitting could manifest in an unusually high average profit per trade. Both are a problem, right? In fact, equally so I'm going to say just be suspicious when you add an extra filter. You need to be cautious to make sure it's actually adding value. And the overall solution is what? Keep the system as simple as possible. And I know you've heard this before, but go one step further. Even simpler, right? And be a little more suspicious and check a broader range of parameter values and make sure that if you're developing a system over a particular market dynamic, like the 2008 bear market in stocks, or like a very strong bull market, go further afield and try and find some similar market dynamics elsewhere to test your rules on and see if they hold up right? Make sure that they stay profitable and then trade the system cautiously. If I had a system that made most of its money in a one or two year time window from a particular set of market conditions, I could be exceptionally confident in that system because I've optimized it carefully. It's very stable. But because it was such a precise set of market conditions that caused it to profit, I'm going to give it a small capital allocation because I just can't be as profitable as confident rather in that system as I can in the system that makes money year in year out over a 25 year back test and has a nice beautiful linear exponential growth in the equity curve.

So systems are not all equal. I like to. I weight my systems according to not only performance and drawdown and diversification benefits, but also in terms of what could go wrong with the system. Could I have been mistaken in some way or deluded myself about the problem performance? And I've done this wrong in the past and traded a system that I shouldn't have had as much confidence in and ended up with a big drawdown. Okay, so what we do about all of these problems in trading system development, the third principle, and this is how you solve it, right? It's the robustness method. And what I'm talking about here is how to quickly build unshakable confidence in your trading approach and keep you safe no matter what the market is doing. But so the first thing you've got to do is build a trading system or what you've got to do is build a trading system that actually works. And the robustness method is how I do it. So first of all you want stable parameters where the system performs well even if the parameters are varied either side of your chosen values. If you vary the parameters a bit, the system should be stable. Like this chart we saw earlier, right? I don't want the unstable profit peak on the left, I want the broad area of stability on the right. I don't want my system parameters to matter much. Does that make sense? If it varies a little bit, the performance should still be good. The second thing you got to do is make sure you've got a consistent market edge over a 20 year backtest through all market conditions. Now, a system not going to make money every year or in every type of market, but it's got to be stable. Look at this. This equity curve has a decaying edge. This is a linear equity curve without compounding. Okay? So it's a constant position size over the life of the backtest and you can see that it's curving over, the edge is dying. This system is no good. That's eventually going to lose you money. If a system has a broken edge, basically it's made money in the past, but now it started losing money over the last couple of years. It's no good, it's broken. If the system has increasing volatility over time, the drawdowns are getting bigger. Then again, it's no good, it's broken. If the system is highly dependent on a particular type of market, then it's broken. Or use extreme caution, right? And then if the system has high dependence on a very few number of trades, like you basically go along sideways, up, down, losing money and then all of a sudden you get one wants to trade in. The equity curve rockets up and then you have lots of losing trades and rockets up. This is highly suspicious. You probably could fit to a small number of trades and you need to be careful. What you want to see is an ideal stable edge, right? Where the system just grows nicely over time. The equity goes up and up. That's what I want to see when I'm developing a system. Ideally, of course, some of these others can be okay, but the less ideal it is, the more caution you need, the less capital you should weight to it, and the more skeptical you should be of the results you're going to get. Make sense? Good. The third thing is the system should perform well over the entire surrounding parameter space. Now, here's what I do, and this is one of my favorite parts of the back testing process. Let's say you've got a system and I've just chosen two arbitrary rules. But let's say you optimized your short term moving average to 50 days, your long term moving average of 200 days, and a breakout to 50 days.

If that, that backtest looks great. But if you vary each of those parameters, plus or minus 20% or even plus or minus 30%, you've got the default parameter value, the minus 30% value, and the plus 30% value. Actually, these are 20%, aren't they? So the minus 20% value and the plus 20% value. So three values for each parameter. How many combinations you've got? In this case, we've got three parameters, three combinations each. Three times three is 27. Every single one of those 27 parameter combinations should be profitable. Ideally should be tradable because that shows you the system is stable. You haven't deluded yourself about the performance because you've chosen some random, some profit spikes in the parameter space that were unrepresentative of what you can really expect from the system. When I run this test, this can take quite a long time because imagine if you've got five or six rules and you're doing three combinations of each rules, and you're doing this simultaneously, you're gonna have many results, and you want most of those results to be profitable and tradable. And if you got, if that happens, then you've got a high chance that those rules are actually real and stable. Now here, this is the magic. The stability of your chosen parameter values relative to the surrounding values is more important than the absolute performance.

And say that again, I just want to make sure this makes sense. The stability of your chosen parameter values relative to their surrounding values. So you've chosen value. The performance level relative to the surrounding parameter values is more important than the absolute value, absolute performance level that you're generating. Because if you want your system to perform in the future, the parameter values you choose need to be both profitable and stable.

Even if you vary them a little bit. And if you do that you're forward looking, you're thinking, okay, this system is probably going to profit in the future in unseen market conditions. So today what we've covered is principle one, Ask the right questions. Ask about how.

What you're doing increases your probability of future profit. Don't just look at the backtest and try and increase the profitability of the back test because that won't make you money in real time. Principle two is avoid the optimization traps. And there's many of them, I gave you a flavor for some of them. But over optimization, curve fitting is your evil nemesis. And if you don't avoid all the different traps, then you're going to end up with a back test that is not representative of what you can expect in real time. And then principle three is the robustness method. Build unshakable confidence by developing a system for stability and robustness over time so that then it can profit in the future. It's more likely to profit in the future. So let me ask you this, I'm just going to wrap up now. If you want to build unshakable confidence in your strategy, do you want a powerful step by step process to show you how? Do you want some free training on building a robust system and a profitable system? Right. If you want all of that, you want it for free, then what I'd like you to do is go to this link Bit ly Trading Confidence and you're going to get my trading system confidence cheat sheet and a series of training videos on building robust trading systems, totally free, so you can download that, go to Bit ly Trading Confidence and you'll get a step by step process and free training on how to do that. And that's the last slide of my presentation. So go grab that cheat sheet and watch the training videos and let me know what you think.