Sample size calculations for clinical prediction model research (aka "goodbye rules of thumb")

Hi everybody, my name is Professor Richard Riley from Keele University, and in this talk I'm going to discuss sample size calculations for clinical prediction model research. I'd like to thank various collaborators that you can see listed at the bottom of the slide. So what do we mean by a prediction model? Well, it's essentially a tool, usually a statistical equation, but it may be a machine learning algorithm, that utilizes multiple factors, also known as predictors or prognostic factors, in combination to produce estimates of risk or outcome values for an individual person. So in healthcare, what we're talking about usually is people that have come to see their doctor or have been admitted to a hospital. Here it's people who've had a traumatic brain injury, and the predictors in the equation behind the scenes are country, age of the patient, their Glasgow Coma Score, pupillary reactivity to light, and whether they've suffered an extracranial injury. Based on the values of the predictors that are entered into the model, the equation behind the scenes will give us an estimate of their risk. So here we see a risk of 14 days of 14% for mortality, and at 6 months unfavorable outcome, the risk is about 50%. So this is a prediction model, and these are utilized in practice to guide decision-making, patient care, maybe even treatment decisions. Prediction model research has 3 main phases. There's the model development phase, where the actual prediction model is produced, i.e., the equation is derived. External validation— this is where the performance of the model is checked in new data, sometimes from a different population. And the last one is where the model's impact on health outcomes is evaluated. Now unfortunately, lots of evidence shows that prediction model research is often substandard and improvements are needed throughout the whole process. And a major area where improvements are needed are in the sample size. So many prediction model studies are too small, they're not fit for purpose, and this leads to problems. So the aim of this talk is to calculate or describe how to calculate sample size required for developing and externally validating prediction models. So why is sample size important? Well, for model development, what we're trying to do is produce a robust equation that when applied to individuals from the target population will give accurate predictions. So when the sample size is too small, your model equation, your algorithm you're producing is more likely to be a chance finding, and therefore the, the predictions from that model are not likely to be reliable when applied in practice. Indeed, many models suffer from this problem that they don't perform well in new data, and that is usually due to low sample sizes, which leads to instability, chance models which are not going to be seen again if you apply them, overfitting— this is where your estimates of predictor effects are too extreme, and a lack of researchers using methods to adjust for optimism, also known as shrinkage methods. Now you may have heard of shrinkage and penalization methods such as the LASSO, elastic net, ridge regression, and you may think, well, don't they resolve overfitting? Don't they, don't they deal with small sample sizes? Well, this isn't true. Here are some quotes from 3 key papers in this area: shrinkage works on the average but may fail in the particular unique problem in which the statistician is working. The results imply that shrinkage methods do not solve problems associated with small sample size. Penalization methods are not a carte blanche. They do not guarantee a reliable prediction model is developed, and they are more unreliable when needed most, i.e., when overfitting may be large. So these are important methods, but only in the context of when the sample size is right. So how can we get our sample size right? Well, our proposal is that we need to move away from the rule of thumb, also known as 10 events per variable, and rather calculate for your particular setting and model of interest for development the sample size needed to minimize the potential overfitting. and to estimate parameters in that model precisely, at the very least to estimate the intercept of that model precisely, the overall risk. And we've published 3 or 4 papers in Statistics in Medicine and the British Medical Journal summarizing proposals for continuous, binary, and time-to-event outcomes. And the key thing is that the calculation requires the consideration of multiple criteria. So let's just focus on binary and time-to-event outcomes. We propose 3 criteria. First of all, to minimize overfitting is to target a sample size that aims to achieve a shrinkage factor of 0.9 or closer to 1. This corresponds to wanting overfitting of less than 10%. So this suggests that if penalization and shrinkage is needed, it's going to be quite small. So this is coming back to stability. We want to make sure subsample size is large enough so that we have low problems of instability caused by overfitting. Another way of measuring overfitting is to look at the difference between the overall model fit observed and the adjusted— the optimism-adjusted overall model fit as defined by R-squared or Nagelkerke's R-squared, which goes from 0 to 100%. And so we might target a small difference on this scale too. Let's, let's say less than 5%, or an absolute difference of R squared of 0.05. Thirdly, like I said, we want to make sure the intercept is estimated precisely. So let's make sure that the overall risk, i.e., the average risk in the population, is estimated precisely. So the confidence interval for the absolute risk has a width of of no bigger than 0.1, so an absolute error of 0.05. So these are our minimum criteria that we suggest, and the sample size needed to achieve all these 3 is the minimum required for the sample size calculation. Now we could go beyond this and also look at a sample size needed to estimate each predictor effect precisely, but this is the bare minimum. Like any sample size calculation, we need to pre-specify a number, a number of things. So for this, we need to specify the number of predictor parameters that you're considering putting in your model, the desired shrinkage factor S— remember I said that should be about 0.9 or above— the overall risk in the population, so the anticipated outcome prevalence or cumulative incidence in the population by a particular time point, and the anticipated value for the Cox-Snell R-squared. In the papers, we show that you can either specify R-squared or you can specify actually a C-statistic, an area under the curve, if that's more easier to gauge what that might be. And we get information about these things from previous studies and previous models in the field and utilize that information towards our input parameters for the sample size calculation. Now behind the scenes, there's lots of closed-form algebra But Joey Ensor has put this all into a lovely package called PMSumSize, which is available in R or Stata for free. Let's have a look at a couple of examples. So a binary outcome example. Let's say we're developing a model with a binary outcome. Maybe it's predicting a particular adverse outcome that's quite common, and so the outcome prevalence is about 50%. And we think that the model based on previous models in the field will have a Coxnell R-squared of about 0.2, which is a Nagel-Kirke R-squared of about 25 to 30%, and that our model— we're thinking of putting in about 30 candidate predictors. So in Stata, we would specify PM Sampsize, type B, which is binary. We'd say that our Coxnell R-squared is 0.2. 30 predictor parameters been put in, and the prevalence, the outcome proportion, is 0.5. So these are the 3 criteria that we talked about before. This is the sample size needed for each of these 3. We see that this first criteria, criteria 1, which is looking to minimize the overfitting defined by shrinkage of 0.9, it gives us the largest sample size of about 1,200. This is about 600 events. and an EPP of about 20. So EPP here is the events per predictor parameter, not per variable. It's a number of parameters in your model. We need about 20. So this is actually about twice, isn't it, the, uh, the quite poor 10 EPV rule of thumb. This is a tailored sample size calculation. So actually, in a different setting, if actually you thought because predictors are better known in that area that the R-squared is going to be a lot higher, say 0.5, which is about an R-squared Nagelkerke of about 66%, then if we change it to the R-squared of 0.5 in the equation, then actually now the second criteria is the one that comes out largest to 556, and we only need now 278 events. And actually now that the PP required is about 9, so the sample size depends on how much noise, how much variation you think you're going to explain. It depends on the outcome prevalence, it depends on the predictor parameters. So it's a tailored sample size calculation which is much more scientific and reliable than just saying 10 events per variable. Now what about external validation? So external validation, a different type of sample size calculation, because what we're trying to do is calculate the performance of the existing model precisely. So we want to precisely estimate discrimination, so the C statistics, area under the curve, and also calibration. So for example, the calibration slope or calibration at large, the O over E. We might, if we're interested in clinical decision-making, we may also want to calculate clinical utility, for example, as defined by net benefit and decision curves. So these are the measures of interest and we want to estimate them precisely. Rules of thumb for doing this suggest we need about 100 events, 100 non-events. For calibration, we may even need more than that, maybe even 200 events. But again, can we do something that's a bit more scientific and a bit more tailored to the situation at hand? So this is what we've proposed, that actually you can derive closed-form analytical solutions for the standard errors, i.e., the confidence intervals, of the measures of interest such as calibration slope and the C-statistic. And because you can do that, you can rearrange that equation in terms of the sample size required, and then this allows you to calculate the sample size required conditional on the user specifying the other components of the equation, which are the target confidence interval width or the target standard errors. Obviously these need to be quite small to be able to estimate something precisely. Again, as in for development, we need to specify the anticipated outcome proportion, overall risk. We need to specify whether we believe that the model is going to be well calibrated in the, in the validation population, and also crucially the distribution of the model's linear predictor. The linear predictor here, that just means the predicted values, and that's often denoted by LP. And also, if you're interested in clinical utility, you need to specify the risk thresholds at which clinical decisions are made. So if you're willing to specify these things, we can use closed-form solutions for continuous or binary outcomes. A more generalized approach is a simulation-based approach where you simulate data thousands and thousands of times in particular sample sizes to see the average standard errors that this produces. And that's a simulation-based approach. Let's look at an example. So let's say we were wanting to externally validate a prediction model for diagnosing deep vein thrombosis. And that we know that the existing prediction model in the, in the development dataset had a linear predictor that was approximately normally distributed with a mean of -1.75 and a standard deviation of 1.47. So we could assume that, okay, let's assume it's going to be well calibrated, and if it was well calibrated, that we'd want to estimate it precisely. And let's assume that the outcome proportion is 22%, which is the same as the development population. So based on these things and entering them into the closed-form solutions and targeting a precise C-statistic with a confidence interval width of 0.1 and a precise calibration slope with a width of 0.2, then the sample size calculation suggests that the calibration slope is the one that's driving it, and we need about 530 events with about 2,400 participants. So estimating calibration is the tricky thing for validating prediction models, and this is going to drive the calculation. Notice that again, this is much bigger in this particular example than needing 100 or 200 events. What about if we'd use a simulation-based approach? We see that the simulation-based approach gets 2,430 participants. Again, basically the same answers as the closed-form approach, which the latter is much quicker, but the simulation-based approach is, is more flexible. So they're both good tools to have in your armoury. Okay, let's finish with a second example looking at the sample size required to validate a model for predicting the risk of a mechanical heart valve failure. The development paper reported a histogram for the linear predictor which was skewed, and we were able to approximate it with a skewed normal distribution, and that also corresponded to an outcome proportion of about 1.8%. So using these assumed distribution values and assuming that the model is well calibrated, we again found that the calibration slope is the one that's driving the sample size calculation. We need about 9,800 participants, actually 177 events in this particular example. So much less in terms of number of events than the previous example, again illustrating that it's a tailored sample size calculation depending on the input assumptions and distributions. Instead of assuming it's well calibrated so that the slope is 1, we might assume that there was overfitting in the original development and therefore that the slope might be less than 1 in new data, let's say 0.8. Well, to estimate that precisely, we'd actually need about 2,000 less participants, so, and 137 events. Therefore, it's more conservative to assume that the model is well calibrated. In terms of the sample size calculation. So in summary, the sample size proposals that we've done are to try and do something a bit more tailored and scientific than using rules of thumb. So for model development, we suggest to use PM_SampSize in Stata or R, and for model validation, we suggest to estimate key performance measures precisely. The proposals are tailored to the setting at hand by pre-specifying things like the distribution of the linear predictor, the outcome proportion, the number of predictors. We've made them applicable to continuous, binary, and time-to-event outcomes and provided Stata and R code in our papers if it's not already contained within PMSumSize. So we hope you find those useful, and I'd just like to finish by giving you the references for the various papers and by giving reference to a new website, prognosisresearch.com, which houses good practice for prognosis research and gives details of online courses that you may find useful. Thanks very much for listening.