Short transcript excerpt of a conversation (Adrian Reid and another speaker) about the persistence of curve-fitting risks in trading-system backtests, even with large sample sizes (e.g., 10,000 trades). They discuss how outliers and concentrated sector market conditions can lead to overfitting and recommend splitting the universe/sample to test robustness.
Still things that you can curve fit.
To, you can, you can still curve fit to outliers. Even in 10,000 trades, there can still.
Be outliers which impact your decision.
There's also market conditions that you could be curve fitting to.
And so.
Maybe there's a particular sector.
That is super strong and you do.
Your optimization and let's say you're in.
Sample happened to be heavy in one sector.
I mean it should be random, but let's just say it was. And you did your optimization and it curve it to that sector and then you expand it to the whole universe, maybe.
It might give you a different result.
And when I look at a system.
That gives me 10,000 trades, I get excited because I know the in sample out of sample test is going to be easy to do. I'm not going to have to agonize over whether it's valid because I did my optimization on 150 trades. I did on 5,000 trades and now I've got a full universe of 10,000 trades. That's easy.
So I wouldn't, I mean it's only.
One extra step, right? It's just chopping the universe in half. You do the same optimization and then you've just got to compare.
So you didn't, you don't actually save.
Yourself much time, but you lose information which could be helpful. Yeah. So I wouldn't skip the step if.
That, if that was me, Sam.
