Adrian Reid explains the most harmful form of survivorship bias for trading-system backtests: index-constituent survivorship bias. He shows that backtesting on the current members of an index (e.g., S&P 500 or All Ordinaries) artificially selects firms that have grown into the index and therefore inflates performance. Reid recommends using historically accurate index constituent lists (validated each day of the backtest) or else backtesting across the entire market. He points listeners to Norgate (platinum subscription) for historically accurate index constituents for Australia, US, and Canada; other markets may lack good sources.
Hey, Adrian, here with your backtesting and trading system development tip of the week. And today I want to talk to you about survivorship bias and the worst type of survivorship bias problem that you can have in your trading system development.
One of the most common things that traders think about when you think about survivorship bias is backtesting on the current set of stocks that are actively listed and worrying about stocks that have been trading in the past but that have delisted and are no longer in your sample set. That's actually the least problematic type of survivorship bias. I want to talk to you about the most problematic survivorship bias, and that is index constituent survivorship bias. So index constituents are the stocks that make up the index. So let's say you want to trade stocks in the S&P 500. Right now there's about 500 stocks, surprisingly interesting, the S&P 500 index. And if you back test your trading system on the current set of 500 stocks, typically you get a really good result. Now why do you get a good result? Because the stocks that are currently as of today in The S&P 500 index have grown to become big companies today and so therefore made it into the index. But the problem is 30 years ago when you started your backtest and you tested all the way out to today, those stocks may not have been big companies, but we know because they're currently in The S&P 500 index today, that as of today, they are big companies. So in effect, if you backtest on the list of stocks that are currently in the index, then you're pre selecting at the start of your backtest small companies that are going to grow massively to become a member of the S&P 500. Think about Amazon, think about Tesla, think about Nvidia, think about all these big companies, companies that are massive today, that have gone through huge growth phases and are now part of the s and P500. If you run your backtest, assuming that they were always part of the S&P 500, you get amazing results. But the problem is they weren't always part of the S&P 500. And in fact, you're cheating hugely by by including them in your backtest. So what you've got to do instead is have historically accurate index constituents for your backtesting. So if you want to test on the stocks that are in the S&P 500 index, then every day of your backtest for the last 30 years or however long you're back testing for, you need to validate which list of stocks is currently on that day in the index and only take buy signals or shorting signals from those stocks and then the next day the same thing, revalidate what companies are in the list, which ones are not, and only take signals from those stocks and so on every single day. When you do it that way, you get a very, very different answer or very different backtest result because what you're doing is you're avoiding that cheating. You're looking with the knowledge that you would have had at the time which stocks were in the s and P 550 and making that that your universe on that day. And then the universe changes each day when to keep track of what stocks are in the S&P 500 over time. Now with the S&P 500, you know, a lot of people are sort of aware of this, but there's some other ways this can creep in. One thing I noticed this week is someone sent me some system results which looked really good and I got a bit excited about it, to be honest. And then I dug in a little bit further and what I noticed in their backtest is they had used stocks in the All Ordinaries watch list in Norgate data and the All Ordinaries sounds like it's all ordinary stocks, but it's not. It's actually an index somewhat similar to the S&P 500 index. It's actually 500 of the biggest stocks in Australia. And if you back just on the current list of all ordinaries, you are cheating just like you would be if you backtested on the current list of the S&P 500 stocks. So you need to be very cautious if you're using anything other than the entire market to backtest your results, you need to be very careful and make sure you're using historically accurate index constituents. Now where do you get this data from? You get it from Nor Gate and below this video I'll give you a link. You can go and get a trial or get a subscription to Nor Gate data. You need the platinum subscription to get the historically accurate index constituents and that will allow you to back test on any of the major the constituents of any of the major indices for Australia, US or Canada. Now if you're testing on markets outside Australia, US and Canada, you're out of luck because there are no really good sources other than Nor Gate for historically accurate index constituencies. So if you're back testing on any other markets, Hong Kong, Japan, any of the European markets, and so on, unless you've found yourself a source of the historically accurate index constituents. What you should do is backtest over the entire market all stocks that are currently listed because that eliminates this survivorship bias problem. Of course, you're left with the survivorship bias problem of your only backtesting stocks that are currently listed. But I've tested and compared.
Backtest with currently listed stocks and currently and delisted stocks and actually that survivorship bias is not very much of a problem. The index constituent survivorship bias is a far bigger problem, so you want to avoid that one at all costs. So that's today's trading system development and backtesting tip. Be cautious what list you're back testing on, and if you're doing anything other than the entire market, make sure you're not falling victim to that survivorship bias and you should instead be using the historically accurate index constituents. Hope that helps. My name is Adrian Reed. This is Enlightened Stock Trading and if you want to know more, click one of the links below and I'll see if I can help you out. See you next time. Bye for now.
