Adrian Reid explains how to choose in-sample and out-of-sample data when validating trading strategies. He emphasizes including multiple market cycles (bull, sideways, bear) and sufficient trade counts in the in-sample window to avoid overfitting, and recommends out-of-sample data that also contains each cycle or using other markets if necessary.
All right, the next question, closely related about validating your trading strategy. It is what is a good method for choosing your in sample and your out of sample data to validate, to build and to validate your system on how do you choose what is in sample and what's out of sample? Now let me just clarify. In sample data is the data that you develop, test, optimize your strategy on out of sample data is is unseen data that you use at the end of the entire process to check whether the rules you've developed. Over here work on unseen data. Okay, in sample, out of sample. There's several different ways to do this. The biggest driver of success is having an in sample data window that includes all different types of market behavior. Bull markets, sideways markets, bear markets, and preferably a couple of cycles so that you're not just overfitting to one particular cycle, you want to go through several cycles and that way you're developing and optimizing over the whole market cycle so you know how bad it gets in the bad times, how good it gets in the good times and you make sensible trade offs. If you only develop and optimize your system on bull market data, it's going to look amazing for bull market data, but when it gets to bear market, it's probably going to fall apart. So for in sample you want several bull sideways bear market cycles and then for outer sample, preferably at least one of each. One of each cycle, at least. And if you can't get that in one market, then make sure you go to another market to validate. That's how I think about it. You also want to make sure you get enough trades. So if you have a in sample window and you develop and optimize your system, and it's a quite complex System, let's say six or eight rules and you've only got 50 trades, you don't have enough data, so you need to use more data or make your rules looser so more trades come in so you're not overfitting to a small number of trades and invalidate it on other data.
It.
