Adrian Reid explains how to split data when building trading systems: use in-sample data to develop and optimize, then test on out-of-sample data you haven't seen yet. Your test window needs bull, sideways, and bear markets. Fifty trades from a complex rule set means you've overfit; you need either more data or simpler rules.
Alright, the next question, closely related about validating your trading strategy, it is, what is a good method for choosing your in-sample and your out-of-sample data to validate, to build and to validate your system on? How do you choose what is in-sample and what's out-of-sample? Now let me just clarify, in-sample data is the data that you develop, test, optimize your strategy on. Out-of-sample data is unseen data that you use at the end of the entire process to check whether the rules you've developed over here work on unseen data. And that way you're developing and optimizing over the whole market cycle. So, you know, how bad it gets in the bad times, how good it gets in the good times, and you make sensible trade-offs. If you only develop and optimize your system on bull market data, it's going to look amazing for bull market data. But when it gets to bear market, it's probably going to fall apart. So for in-sample, you want several bull, sideways, bear market cycles. And then for out-of-sample, preferably at least one of each, one of each cycle at least. And if you can't get that in one market, then make sure you go to another market to validate. That's how I think about it. You also want to make sure you get enough trades. So if you have an in-sample window and you develop and optimize your system and it's a quite complex system, let's say 6 or 8 rules, and you've only got 50 trades, You don't have enough data. So you need to use more data or make your rules looser so more trades come in. So you're not overfitting to a small number of trades and then validate it on other data.
