Sample size calculations for external validation of a clinical prediction model

Hi there. In this talk, I'm focusing on how to calculate the sample size you need for externally validating a clinical prediction model. Now, this is work I've done with many collaborators listed at the bottom of the screen there, so I really want to thank them. Let's get going. Okay, so just so that we're all on the same page, what do I mean by a clinical prediction model? Well, I mean a statistical equation like logistic regression, or a machine learning algorithm like a random forest, which based on the information in that algorithm, which is the predictors or features such as country, age, Glasgow Coma Score in this example, would give you a predicted value for an outcome or an estimated risk of the outcome in a particular individual. So this individual from when we apply the CRASH tool in those that have had a traumatic brain injury. This individual's from Australia, aged 47, and so forth. The model gives an estimated risk of 14-day mortality of 14%. So these estimated risks or predicted values are to be used in clinical decision-making to guide patient counselling, treatment decisions, monitoring strategies, and so forth. And they're hot topic because they're used in clinical practice, and indeed here are some examples of where they're recommended within NICE guidelines in the UK. So we've got the Framingham score, which you may have heard of, QRisk, the Nottingham Prognostic Index, the Apgar score, one of the oldest prediction models, the Leicester Diabetes Risk Score, and so forth. So lots of people are interested in developing prediction models. Prediction model research is the process of firstly developing the model and using methods to internally validate it. There's then usually an external validation phase where the model that you've developed is evaluated in new data, potentially from the same population but often from a different population. And then thirdly there may be an investigation of clinical impact. So when you use the model in practice, does it actually improve patient outcomes? Many improvements are needed throughout the whole of these different phases of prediction model research. One area that's really important and has been lacking is the thought about sample sizes. So when you develop your model, how large does the dataset need to be to develop a reliable model? And when you externally validate a model, how large does the sample size need to be to reliably examine the performance of that model? For model development, we've already got other talks about that that you may want to check out, but very, very briefly, for development it's about having a sample size that reliably estimates that model equation or the machine learning algorithm so that the predictions are likely to be reliable in new data. And what we mean by that is having a sample size that minimizes the potential for overfitting and estimates the parameters in the model precisely. So for example, at a basic level, we want the model intercept or the average risk in the population to be precisely estimated. Check out our PMsampsize module in Stata and R, which is just one line of code which allows you to enter the number of predictor parameters you're considering putting in your model, the outcome proportion in the population of interest, and a conservative estimate for the overall model performance, and it will give you the sample size. And check out our paper on this for the mathematical details. But today I want to focus on external validation. So this is where we are examining in new data the performance of an existing prediction model. So there's no model development. It's about an existing model which is being used to make predictions, and then new data— we're going to see whether those predictions are reliable. This means that we need a different sample size calculation compared to model development, and we need to be thinking now not about the model equation but about actually the performance measures of the model. So can we precisely estimate the performance in terms of usually measures such as discrimination, calibration, and clinical utility. So we want to make sure that when we estimate performance, for example, in terms of the area under the curve, the C-statistic, or the calibration plots and the calibration slope, that we estimate these things precisely so that we can reliably say whether this model appears to be robust or not. When I present the sample size calculations shortly, a main measure that often drives the sample size calculation is calibration. So I just want to pause this to talk about, you know, what we mean by calibration. Calibration is the agreement between the expected risk based on the model, predicted risk, the estimated risk, against the observed risk in the validation dataset. So an excellent calibration would have a 45-degree line here where basically the slope of the calibration line is equal to 1, and the overall risk and the— that's predicted from the model and the observed risk in the validation dataset agree such that the O/E, the observed of the expected risks, equal 1 overall. What we often get is miscalibration, particularly when a model has not been developed using very good methods, and we may, for example, have a calibration slope that's less than 1 if there's some overfitting in the model development dataset. For example, in new data, we might expect there to be a calibration slope less than 1. Notice here that actually what I've got is a calibration curve. So this is a flexible way of examining calibration rather than forcing a straight line through points. We can model the curve, so a nonlinear smooth relationship, which I'll come back onto later. That's important to consider. So what I want to suggest is that This calibration plot, which is telling us basically core information about a model's performance, should be the focus of the sample size required for external validation datasets. We want to make sure this calibration curve is estimated precisely so that in individuals whose predicted risk is around 0.8 If it's a good model, we'd know that based on the results of the validation exercise that this curve is precisely estimated around the 0.8 predicted value. Similarly, if you've got predicted values down at 0.2, we want a precise estimate for what the observed risk is around 0.2. So there's been a number of papers suggesting rules of thumb for the sample size needed for external validation, mainly focusing on binary and time-to-event outcomes. These generally suggest that we need about 100 events and 100 non-events at a minimum. And for calibration and examining calibration curves, we might even want greater than 200 events. So these are rules of thumb, but can we do something a bit more scientific in order to, in a particular application for a particular model in a particular population, can we identify the sample size that we're going to need more accurately than just saying, okay, we need 100 events and non-events. So let's have a look at that. So I'm going to focus on calibration. I'm going to look at what factors influence the precision of calibration measures that, that contribute towards that calibration plot, like the calibration slope. In step 1 in a validation study, you're going to take the existing prediction model, be it a logistic regression or a machine learning algorithm, and apply it to every individual in that validation dataset and obtain their estimated risk. I'm going to define that by P, and i is just an individual. So every individual in my validation dataset has a predicted risk from the model, and I'm going to model that actually on the logit scale, so the log odds scale, the log of p over 1 minus p. And I'm going to define that as the linear predictor, the LP. This is the natural scale for a logistic regression model. Um, and so every individual in my dataset has now got an estimated linear predictor value from the existing model. In step 2, I'm now going to examine the calibration of my predicted and observed risks, but on the logit scale, because generally we would use a logistic regression model, which is on the logit scale, to examine the calibration of our predictions. So if we did that, we would have this logistic regression model where the logit tree probabilities in my validation dataset are modeled by an intercept term and a slope term for the linear predictor. So remember, the linear predictor is the predicted linear predictor from my existing model, and I'm seeing basically if this equation when it's applied in my validation dataset suggests that the existing model is well calibrated. If it was well calibrated, my intercept would be 0 and my calibration slope beta would be 1. If we focus on beta, the question really is what influences the precision of the estimate of beta. So when I estimate my calibration slope What factors are going to contribute to whether it's precisely estimated or not? Well, clearly the first one is the sample size. The larger the sample size, the more precise my estimate of beta will be. Also, the linear predictor's distribution will have an impact. So a wider distribution will tend to have more information than a narrower distribution. The shape of the distribution may also be important. The number of events, or a broader thing is the outcome proportion, the overall risk in the population, will also contribute. And then lastly, because this is a logistic regression and not a linear regression, actually the values of alpha and beta themselves will have an influence on the precision of the estimates, basically because the error is a function of the parameters themselves. So we need to think about, for our sample size calculation, what the distribution of the linear predictor is likely to be, what the population is that we're applying the model to, and what's the overall risk in that population. And lastly, whether we think the model will perform well in our population. So do we think the intercept will be 0 and the slope will be 1? We suggest that that's a good place to start. Let's say, okay, if the model is likely to be very well calibrated, then let's make sure that we are estimating that calibration equation here precisely. So under the assumption that alpha is 0 and beta is 1, and under an assumption about the distribution of the linear predictor, and an assumption about the outcome proportion, then what is the sample size we need to estimate alpha and beta precisely? Now I want to avoid detailed mathematical formula in this broad talk because you can always go to the papers to look at this. But what we've done in our papers in Statistics in Medicine— and we've got papers for binary outcomes, for time-to-event outcomes, and also continuous outcomes— we derive analytical solutions for the standard errors, maybe like the confidence intervals, for a calibration slope estimate, for an O/E estimate, for a C-statistic estimate, and for a net benefit estimate, which we'll come back on to. And then we rearrange these equations so that we can work out the sample size required to precisely estimate each of these measures. And then basically the largest sample size needed is the one that we would suggest to be the minimum required. So the user needs to specify a target standard error or target confidence interval width. As I said, the anticipated outcome proportion in the population the model's anticipated calibration performance, for example, an intercept of 0 and a slope of 1, and the distribution, in particular the variance of LP values, in the validation population. If we're interested in clinical utility in terms of net benefit, we may also think about the potential risk thresholds for making clinical decisions. Now, as I said, I want to avoid mathematical formula, but it's actually essential that I just illustrate it briefly for just one measure, which is the calibration slope. So remember, the premise is we are going to calculate the sample size n conditional on specifying a target standard error We can rewrite that in terms of the confidence interval width if we wanted to, and the true values of alpha and beta in my equation, and also the distribution of the linear predictor. Now we show in the paper that the sample size depends on the values of the information matrix when we fit that logistic regression equation divided by the square of the standard error. And the information matrix here, the unit, unit information matrix, is equal to the expected value of the transformation of the logistic regression equation. So the exponential of alpha plus beta times the linear predictor over 1 plus the exponential of alpha plus beta times the linear predictor, all squared. And similarly, we have different components, but they're all based on a function of alpha and beta and the linear predictor. So how do we obtain these expected values for the information matrix? Well, what we're going to do is— and we've got R and Stata code that does this for you— you basically simulate a very large data set that generates the values of LP and the outcome values for each individual based on the assumptions. So based on the distribution of LP Based on the proportion of outcomes in the population and the assumed values of alpha and beta. And then we're going to compute the value of each of these elements of this matrix for every individual, and then work out the average. And that's going to give us the expected values for each of these elements. And that becomes then the i alpha and i alpha beta and i I beta. So this is all done behind the scenes in the computer programs, but it's all based on the assumptions. And then these are the values of the unit information matrix, and the sample size is a function of these, like I said, and the target standard error. So now we're in business to do our sample size calculation. So the distribution of the linear predictor is an important aspect to this calculation, but how do we gauge that distribution? Well, the place to start is by going to the model development study and trying to extract information from that study that would tell us approximately what the distribution might be. So fundamentally, you might have done that development study, so you might have the raw data, the individual patient data, so you could calculate it directly. But if not, you could ask the model developers to provide you with a histogram of the linear predictor distribution for the model development dataset. You may have a histogram provided by the authors in the original papers. Sometimes that's given, or sometimes it's at the bottom of a calibration plot. Sometimes there may just be a mean and a variance of the linear predictor provided. We might be able to assume it's normal distribute— normally distributed perhaps. Here's a real example where I've got the histogram for an existing prediction model, and I've tried to approximate the underlying distribution by a, um, either a normal distribution— here's the dotted curve— or a skewed normal distribution, which is the solid curve. And actually, when we apply the sample size calculation, it doesn't make much difference either, either way we do this. But the key thing is we're using these distributions to try and mimic an approximation for the underlying true distribution as based on the observed distribution from the model development paper, which was this histogram. If you don't have a histogram like that, and you don't have means and variances or the IPD, you may want to then go to a stronger assumption, which is to assume that the distributions of the linear predictor in the— in those with the outcome and those without the outcome are normally distributed with a common variance. Because in that situation, what we can use is the C-statistic, the area under the curve, which is nearly always provided in moral development studies, and that would then inform the variance of the linear predictor. And we can, as we show in the paper, we can use that information to get the distribution in terms of the mean and the variance, assuming normality. So this is sort of a last resort, but often that may be the only thing we can do if we can't get any other information. Right, let's look at some examples. So the first example is a diagnostic prediction model looking at diagnosing a deep vein thrombosis. So the question is, an existing model has been proposed and the model development study reported that the linear predictor distribution was approximately normal with a mean of -1.75 and a variance of 1.47 squared. So what is the sample size we're going to need to precisely estimate Let's say the C statistic and the calibration slope in a new population where we assume that this distribution holds, that the model is well calibrated, so the intercept and the slope of the logistic regression equation is 0 and 1, and that the outcome proportion is about 22%. So when we apply our closed-form solutions such as the one for the calibration slope that I showed you, we see that to get a targeted 95% confidence interval width for the calibration slope of 0.2— so basically that means that the confidence interval goes from 0.9 to 1.1, assuming the calibration slope is 1 in truth— that we need 2,407 participants, which actually, based on this outcome proportion, is about 530 events. Now clearly that's a lot more than the rule of thumb of 100 events or 200 events. The z-statistic, again, it depends on the confidence interval width that we choose, but let's assume that we want a confidence interval width of 0.1 so that the, you know, if the, the true value of the z-statistic was 0.82, that the, the range would be 0.77 to 0.87. The sample size required for the z-statistic is much lower, 488. 8 events, 400 participants. So like I said, the calibration slope is often the one that's driving things. There is also a paper by Kim Snell and colleagues which uses a simulation-based approach, and that was the paper that I used to guide what is informing the precision of the calibration slope in terms of the distribution and things like that. If we use that simulation-based approach, it takes a lot longer than this closed-form approach, which it was just seconds. The Stell approach is minutes. Still not too long, but takes a lot longer. You see that the results are very, very similar: 400, 385, 2,400, 2,400. So either approach is getting very similar answers. The closed-form approach is much quicker. And is the one that we would recommend. Let's look at the second example. This is looking at a prediction model for the risk of a mechanical heart valve failure. So we want to know what's the sample size needed to externally validate this model. And the histogram for the linear predictor was reported in the development paper, and we approximated it using a skewed normal distribution and ensured that the outcome proportion was about 0.018, which is the outcome proportion in the development population. Let's see then what sample size we need. Again, let's jump into the calibration slope, and if we assume it's 1 and that again we want a targeted 95% confidence interval width of 0.2, then we need 9,800 participants and about 180 events to estimate, um, to have to target this confidence interval width, to estimate it this precisely. If we lower the calibration slope— remember earlier I said that often the calibration slope is less than 1 in new data— then actually we need smaller sample size to estimate that precisely. Therefore, the original calibration slope of 1 is a more conservative sample size calculation. C-statistics, again, these, you know, assuming different values, these need a lower sample size of 4,000 to 5,000. Interestingly, in this example, of course, the calibration slope of about 177 events is not far off the 100 or 200 events that the rules of thumb proposed, but it wasn't in the previous example. I'm coming towards the end now. Let's just think about extension to survival outcomes because the closed-form solutions that I mentioned before were really for binary outcomes. Survival outcomes have this issue of censoring that not everyone's followed for the same length of time, and therefore it does create problems for closed-form solutions. Therefore, the simulation-based approach of SNELL is probably more important in this situation. But it's complex. You need to specify the survival distribution, the censoring distribution. It's important to focus on a particular time point or time points. And to be able to examine calibration plots at particular time points and avoiding categorization and making sure that we can plot calibration curves, we need to use something called pseudo-values, which I'm not going to go into here, but we do in the paper. But the underlying premise is very similar to before, where we're going to simulate a large data set under assumptions of good calibration. We're going to simulate the linear predictor distribution based on maybe a histogram or something available in the model development paper, and then we're going to use assumed survival distributions for the survival and centering, probably just exponential distributions because that will keep things simple. And then work out, based on a particular sample size, a precision of our estimates, and then repeat that to see how the precision changes according to different sample sizes, and then eventually decide on the sample size that we'll use. So the third example, this is in our paper for survival models looking at the prediction for recurrent venous thromboembolism, and the original paper had this histogram And like I said earlier, we could use a skewed normal or a normal distribution. It doesn't make much difference in terms of the actual results in the end. So simulating different sample sizes and expecting the precision of estimates such as the z-statistic and calibration slopes, but also— and this holds true for all of the calculations— let's have a look at the precision of the calibration curves and the variability of calibration curves we might expect to see. Because defining precision is very tricky, isn't it? I have avoided this so far, and I've said, well, let's aim for a confidence interval width of 0.2 or, um, something similar. But the reality is that we're sort of guessing, and particularly when it's on the logit scale, we don't really know what is precise. So what I suggest that we do to help us in that regard is, after we've done our sample size calculation and identified a sample size that we think is useful, is then to generate datasets of that size and plot the calibration curves and repeat that maybe 200 times and just examine the variability in the calibration curves. And you might find that the variability is very narrow, you might be able to lower the sample size then, or you might find that the variability is too big and you may need to increase it. So I've got 2 plots here for this example. On the left, this is a sample size based on targeting a confidence width of 0.2 for the calibration slope, and on the right, it's targeting a confidence width of 0.4 for the calibration slope. Now clearly, as you increase the targeted width, you your sample size is coming down, and therefore the variability around the calibration curves that you might estimate goes up. On the left, it's much more precise and narrow, isn't it, the variability? So this is the ideal situation, but you may then say, I don't have the money or the resources to recruit 14,000 individuals. Therefore, do we really need this level of precision? Maybe if my clinical decision range is down here, I'm less worried about variability up here, and therefore I could perhaps relax the sample size. And if I did relax it, I would get more variability down here, but again, still reasonably narrow perhaps. And up here, yes, there's a lot of variability. However, I'm not so worried perhaps if somebody has a predicted risk of an observed risk of 0.5 when I'm predicting it to be 0.9, because maybe 0.5 and 0.9 are both still very high, whereas down here it's much more important to get precise estimates of the observed risk compared to the predicted risk. So this is just to raise this issue that this sample size approach, it's not quickly pressing a button and walking away with the sample size. You need to be very considered in terms of what is it you're trying to estimate, what precision is enough, and to think of those things in regards to where you want to use the model and if you're thinking about using it for clinical decision-making, for example. So this is not just a press the button and walk away. It's, it's actually needs a lot of considered time with clinical collaborators and statisticians. Lastly, I want to follow on from that to mention clinical utility. So clinical utility is another important measure alongside calibration and discrimination, and this is where we were thinking about using the model to make decisions. For example, somebody is higher than 5%, we may decide on using a treatment, and but less than 5%, we would not do anything differently. So clearly we'd like to look at precisely estimating the clinical utility defined by particular risk thresholds, and we may do that using the net benefit measure proposed by Andrew Vickers and colleagues. This measures the— or weighs the benefits against the harms at those particular thresholds that you've chosen based on consultation with clinicians and patients. So in the VTE example, actually the clinical team suggested that a range of thresholds between about 0.05 to 0.20 would often be used to guide clinical decision-making. So again, once the sample size has been identified, you might want to check the potential uncertainty or variability in the net benefit for that sample size across, say, 100, 200, or 500 samples. So with 3,600 participants— and let's focus just on a risk threshold of 0.05— we see that this is the distribution of the net benefit estimates that might be observed across 200 samples generated with 3,600 participants. So the average net benefit is around 0.125, and the range goes down to 0.11 and up to 0.14. This is all above 0, and therefore, you know, this is very precise. If you decided that a risk threshold of 0.5 was relevant, which I don't think the clinical team did in this situation, but just for example, we see that the net benefit now is on average around 0.005 but the range is much wider because it encompasses 0, so there would be uncertainty here potentially from some samples about whether the net benefit is positive, and therefore maybe a bigger sample size would be needed if you're trying to precisely estimate clinical utility based on this particular threshold, if that was clinically relevant. So again, just raising the point that, you know, once the sample size has been identified based on your target precision measures, it's important to plot that and just think about that a bit more before making a final decision. Okay, so in summary, I have presented today sample size calculations for prediction model research focusing on model validation. Remember, for model development, other talks have proposed the PM_SampleSize module in Stata or R. For validation though, this is about precisely estimating key performance measures like the C-statistic, the calibration slope, calibration curves, and potentially clinical utility. And I've encouraged you to move away from rules of thumb of like 100 or 200 events, although they're a useful starting point. But if we can do something more scientific, then we should. We need to think about therefore whether we can get information about the assumed linear predictor distribution, the overall outcome proportion, and be willing to have a look at sample size for potentially different assumed calibration fit. I focused on binary and time-to-event outcomes in the talk, but there's also a paper that we published in Statistics in Medicine with first author Lucinda Archer on continuous outcomes. And for all our papers, data and R code is provided so that you can have a look at those codes for the examples in the papers and then easily adapt it for your own purpose. So I hope this talk has been really useful to you. I'll leave you with our references for model development and also for model validation. Do check out our website where we provide links to the PDFs for these articles and information about our courses and training, which maybe we'll see some of you at soon. Do check out my other videos, and thank you for listening.