Importance of sample size for AI-based prediction models in healthcare

Hi everybody, it's Richard here. In this video, I'm going to look at the importance of sample size in research that aims to develop or evaluate prediction models for healthcare. Whether you're from an AI background, machine learning, or statistical profession, this lecture is for you. So the talk's going to be split into 3 parts. Part 1, I'm just going to introduce a bit of background to the field. Also, throughout the talk, I'm going to trying to entertain you a little bit by adding some quotes from various footballers. Here's the first one, Paul Gascoigne: I never make predictions and I never will. Obviously they're saying he'll never make prediction whilst making a prediction. But in healthcare at the minute, so many people are interested in making predictions. With the emergence of AI, machine learning, and advanced statistical methods, more and more research papers are being published thousands each year, to be honest, that are suggesting they can predict all sorts of outcomes and diseases. But should we really be trusting these models? Here's an example of a clinical prediction model. This is a web interface that's got a tool called the CRASH tool behind the scenes. And actually, behind the scenes, I think here it's a logistic regression-based model. What you have here is a tool that is predicting the risk or the probability of mortality in an individual who's been, who's had a traumatic brain injury. And this model has these predictors: country, age, Glasgow Coma Score, pupillary reactivity to light, and major extracranial injury. Now this individual, when you enter their values into the CRASH tool, you end up that this individual is estimated to have a risk of dying by 14 days of 14%. So this is what we mean by a clinical prediction model. You may also know it as a predictive algorithm or a variance on, on that term. The key thing is we're individualizing our estimates. So these are not population-level estimates, these are estimates for individuals that are then used to make decisions for those individuals. I refer to these characteristics as predictors. You may also know them as features, variables, or or covariates. Now why are these a hot topic? Well, because they are being used in practice. So here's a list of prediction models that have been referenced in NICE guidelines. You may have heard of some of them in the UK, for example QRisk, the Framingham risk score, Nottingham prognostic index, the Apgar score in newborns, and the Leicester diabetes risk score. All of them are being used to guide decisions on individuals based on expected outcomes or risk of those expected outcomes. So formally then, we might define a clinical prediction model as an equation, for example based on a regression, or an object, for example based on a machine learning method, a black box, that outputs or estimates or predicts an individual's outcome, either the value of that outcome or the probability of that outcome occurring, conditional on values of multiple predictors or features. There are 3 key phases of prediction model research. Model development— this is the process of using a dataset to estimate or produce a model equation or black box. Model evaluation— this is where we use a dataset to validate the performance of the developed model, usually in new data, sometimes from the same population, sometimes from a different population, to look at generalizability. And the third key phase is model impact. This is where we've developed a model and we think it performs well, but we want to see that if we actually start to use this model in practice to help make clinical decisions for patients, do we ultimately see a benefit on patient outcomes or related things like reduced costs or improved population health in some sense? How do we evaluate the performance of models? Well, throughout this talk I'm going to be focusing on 3 key areas: discrimination, calibration, and clinical utility. So briefly, discrimination tells us whether a model separates out those individuals that do and do not have an outcome event. This is typically measured by the C-statistic, also known as the error of under the receiver operating characteristic curve. A perfect C-statistic would be 1, and a C-statistic that's no better than chance would be 0.5. Calibration— this is a very important but hugely neglected performance measure in clinical prediction model research. Calibration says, do the observed and predicted values agree? So for example, if we had a group of individuals whose predicted risk was 10% we would want to see that in those individuals, 10% of them do experience the outcome. Calibration can be examined through calibration plots, and we'll see some of those shortly. And the fit of that calibration plot can be summarized by a measure called the calibration slope, which has an ideal value of 1, or the O/E, the observed over the expected, again with an ideal value, value of 1. Another measure which again is also neglected a lot of the time is clinical utility. This is where we start to use a model to inform decisions on patients if they are at or above a particular risk threshold. So for example, in the UK, often a QRisk would be applied to estimate an individual's risk of CVD by 10 years, and if that risk was 10% or more, we may start to give statins to that patient. So there's a decision threshold, and based on that decision threshold, we can work out measures such as the net benefit, which tells us the benefits compared to the harms of using a model to make decisions. So all these 3 measures and types of performance criteria will be considered throughout this lecture. Okay, but we need some realism. So lots of hype and lots of potential prediction models, and they are being used, but let's have some realism. Steve McLaren, when he was England football manager, was good at producing realism, and he said, we have to reduce our expectations of England, and we have the players to do it. And he certainly, he certainly Reduce our expectations. Now this slide was kindly provided by Martin van Smeden and basically shows that in each type of disease field hundreds of models are being produced and have been produced. And the issue is that those models that I mentioned before that are used in practice, they are in many ways the exception to the norm. And the norm is that most models used in practice— sorry, most models that are being developed are just not good enough or fit for purpose. For For example, there's over 400 models for COPD, 360 models for CVD prediction in the general population, and so forth. Everyone's trying to produce their own model. The reality is that sadly the quality of these models is often very, very poor at the moment. So everyone is developing a new model, and really we shouldn't be trusting them. Most of them are not fit for purpose. Why? Well, very few are validated or evaluated in new data, and the models that are developed, therefore, when we see their performance in the development data, is often very, very optimistic. They suffer from many common methodological weaknesses. For example, the data set that's being used to develop the model may not be representative of the population, the predictor is not measured at the right point, the outcome definition was poor. Overfitting— this is where models are overfitted to noise in their development data, and therefore they suggest relationships that are not observed in practice, and that leads to extreme predictions. Poor calibration— so miscalibrated estimated risks. Predictors that are dichotomized rather than models on the continuous scale, or poorly reported models— we don't know what the model equation is, so forth. We did a review during the COVID pandemic of 606 models that we identified. 90% basically were high risk of bias, and only a handful seemed to be— seemed to have potential for value. Okay, so that's a bit of a depressing picture. What can we, what can we do about it? Well, to improve prediction model research, what we need is more rigorous study design standards. And there are many different aspects to that, but today I'm going to focus on sample size requirements. So strangely, in the prediction model field, there doesn't seem to be much emphasis on people thinking about sample size for their research studies. What happens a lot of the time is that people just have an existing dataset. Sometimes I call that a what's-in-the-freezer approach. Let's see what's in the drawer. Let's let's just hand it to a PhD student and ask them to develop a prediction model. And it's not a very scientific way of doing research. We did various reviews looking at hundreds of models that have been developed in the literature, and most had not even justified or mentioned any statement about why their sample size was sufficient. And indeed, in many of them, we could think about that their sample size was probably far too small. So, okay, why is this relevant? Well, guidelines, in particular the new 10 Principles of Good Machine Learning Practice guidelines produced by the FDA, MHRA, and so forth, emphasize that sample size is relevant. So if you look at their guidance, there's at least 3 items that mention sample size. So they talk about Research should have samples of adequate size in the clinical study for training and testing. Samples should be of adequate size to manage any bias, promote generalizability of performance, and also so that we can identify circumstances where the models may underperform. In a risk of bias tool called Probust AI, there is a signalling question that says, was there evidence that the sample size was reasonable? In the Tripod AI reporting guideline by Collins et al., they have an item that's about sample size. Explain how the study size was arrived at separately for development and evaluation and justify that the study size was sufficient to answer the research question. Include any details of any sample size calculation. So there is a move to try and encourage people to think about sample size in their field through guideline makers and through these new tools, but But, um, so many people still struggle to understand this concept and are just, regardless of their dataset size, are developing models often with thousands of predictors. And I want to push back in the remainder of my talk on this aspect. This concern led me to lead this paper that was published in 2025 in Lancet Digital Health, The Importance of Sample Size on the Quality and Utility of AI-Based Prediction Models for Healthcare. And in this paper, we talk about why small sample sizes have a negative impact on the performance of prediction models in terms of when we develop them or in terms of when we evaluate them. So the aims today in this video are that I'm going to showcase the impacts of small sample size in various examples and And I'm going to talk about some potential solutions and emerging work. Okay, so let's move on to part 2, the impact of small sample sizes on clinical prediction models. Michael Owen, football commentator, used to be a player, but when he was commentating, he said, that is a fantastic penalty, but he'll be gutted it went wide. We've got all these fantastic AI machine learning statistical tools, but But if we're not using them appropriately, we will be gutted that we missed the target. So let's have a look at why. First of all, if you've got a small sample size, then your dataset is potentially not going to be representative of the target populations and settings. This means that models are unlikely to be generalisable, that for some subgroups or regions the model is unlikely to perform well in, And indeed, when we don't have big enough sample sizes, because we can't really see enough people in those subgroups or regions, we can't really include predictors in our model for those things. So we can't tailor models to be better in those places. Well, both these aspects are going to inflate and exaggerate fairness concerns. So if you have small datasets, then those individuals in minority groups, particular ethnic groups for example, may not be well reflected in your dataset at all, leading to problems that the model may not perform well in those individuals. A lovely paper by Thomason showed that even sometimes with 23,000 participants, the effective sample size can be, you know, less than 30— quite small for some individuals defined by particular predictor values. So if you think about combinations of predictor values, so for example, um, individuals with a particular ethnic group, a particular, um, family history, particular blood pressure or age. When you start to combine predictors like that, you start to narrow down how many people are in your data set that have those characteristics. So even with big data sets, you can have small effective sample sizes for some people. So a concern of generalizability is my first point. My second point is that there can be large uncertainty in the actual model that you develop. So when we develop a model, we obviously have to estimate associations between predictors and outcomes. And when we have small sample sizes, the magnitude of those associations, the predictor effects, could be hugely uncertain. Similarly, if we're trying to understand the importance of particular predictors, their weight towards the prediction or the estimated risk, then the uncertainty of their importance is going to be very, very big when the dataset is too small. So this concept I refer to in this talk as instability— that when you're trying to develop a model, that in your particular dataset you may select a certain set of predictors and estimate particular predictor effects and weights, but if you happen to have a different dataset of exactly the same size from the same population, then you may have selected very different predictors and have estimated very different predictor effects. I refer to this as a multiverse of very different models, that across all these potential models that you could have generated based on a random sample of the same size, the predictors, predictor effects can change dramatically. And therefore, any attempt to meaningfully explain such models And you know that there's, there's a drive to, to explain AI, explainable AI. This is just futile when the sample size is small because of the instability and because of the uncertainty. For more on that, have a look at our paper Clinical Prediction Models and the Multiverse of Madness. The knock-on effect of uncertainty and instability in a developed model is is uncertainty and instability in individual predictions. So what I mean by this is that one model, let's say if it's applied to me, may estimate a risk of, let's say, 0.2. So I've got a 20% chance of the outcome. But if there's a lot of uncertainty, although that's sort of the best estimate of my risk, 20%, if there's a lot of uncertainty, then maybe that actually across other plausible models that we could have generated, they may have said that your risk was 1%, and actually some of them may have said your risk was 90%. Well, that changes my understanding of whether a model is useful or not, because if that's the uncertainty around my individual risk from that model, then that model is pretty much worthless. So this relates to model-based error, also known as epistemic uncertainty, and we want to be reducing that. There's always other things that we don't know, but based on the predictors that we are modeling, we want to try and reduce uncertainty to have some stability based on the information that we do have available. We talk about this stability more in a paper, paper with myself and Gary Collins in the Biometrical Journal, Stability of Clinical Prediction Models Developed Using Statistical Machine Learning Methods. It's always nice to have some examples, so I'm going to have more figures and examples in the next few slides. In this slide, what I'm demonstrating is this multiverse of models concept and the instability in prediction around individuals. Now if we just focus on this bottom right situation here, I've got a development dataset of 5,000 people, and I'm going to use this dataset to fit a LASSO logistic regression model with 9 predictors, and the prevalence in the population is about 50%. And I'm going to repeat that process 1,000 times, so I'm going to get 1,000 models in my multiverse, and And for every individual, therefore, I'm going to have 1,000 models applied to them, so I'm going to have 1,000 predictions. On the x-axis, I've got 9 individuals. Now, these are their true risks because I know— because I've simulated data. So each of these individuals, their true risk. And then I've scattered the range of predictions from these 1,000 models that I've generated. So for an individual here whose true risk is 0.2. On average, we see that their estimated risk is about 0.2, and the scatter of predictions around all these different models that I could have generated is quite narrow, isn't it? It can go up to about 0.25, down to about 0.1. Similarly, everyone else, the scatter around the points is quite narrow, and we'd expect that because the dataset is huge relative to the number of predictors that I'm fitting. But what happens when I move down and I have lower sample sizes? Let's go down to the extreme. I've got 50 patients in my development dataset, and again, I've repeated this 1,000 times to see what happens. Now we see that on average, the individuals again not too far off their true risk, but that's on average. In a prediction modeling sense, on average doesn't really help us because we are just going to be developing one model. So when we develop one model, actually what we see is that predictions for everybody here from that one model could be anywhere between 0 and 1. Let's look at this individual whose true risk is 0.2 again. Although on average across all the models that could have developed, their true— their average risk is, is lower than 0.2, isn't it? It's not too far off 0.2, but in one particular model, their risk estimated risk could have been around 0, or it could have been up to 1. So this is what I mean by instability of predictions. Now, quite worryingly, you never get presented graphs like this in practice. I call these instability plots, and we could use something like bootstrapping to replicate this instability process and provide us with essentially predictive distributions of uncertainty reflecting our uncertainty around individual risk. At the minute, most prediction models just give you a point estimate of risk for each individual. I think that's highly misleading, and I think we need to change that. So small sample sizes lead to big problems of instability, which clearly are not going to be helpful to individual patients if their risks are going from 0 to 1. The knock-on effect of instability like that is also that we will end up with models that have worse performance. And worse performance means, for example, lower discrimination performance, so we can't separate out individuals that will and won't have the outcome, and therefore we have reduced clinical utility. So this is not a situation where we're making recommendations in Netflix about what films to watch. This is a situation where we're telling or informing patients about their risk to guide treatment decisions. This has big implications on people's lives. Therefore, to think of reducing clinical utility, i.e., reducing the positive benefit on patient outcomes from small sample sizes, is many ways unethical. Okay, so if you've got a small data set, think about the consequences for healthcare. We can define the reduction in a performance from smaller sample sizes as model degradation. Um, we also know this as a loss function, but I like to refer to it as model degradation. This is how your developed model will perform compared to a hypothetical true model, how that would perform. We can never do as well as if we knew the truth, um, unless we had infinite data and we modeled everything perfectly. So we're always going to do less than we could do if we knew the truth, and that's okay. But we don't want to drop too far down from the truth because the further we come down, the less the C-statistic will be, the less the clinical utility will be. And that's because with smaller sample sizes, we can't distinguish noise from signal. We're going to increase our epistemic uncertainty, our model-based error, and we're going to have issues such as more overfitting. Remember I said to you that overfitting is an issue because what happens is you end up with extreme predictions. Now there are methods such as LASSO, elastic nets, other machine learning methods like random forests that try to deal with this overfitting by, um, something called penalization or shrinkage, which pulls predictor effects down towards the average— sorry, yeah, essentially down towards the null. Um, however, they don't solve all the problems because we have to estimate the penalty terms and the shrinkage terms when we use our development data, essentially. So it doesn't solve all the problems. And yeah, this, this, this issue of shrinkage, of course, by definition will shrink predictor effects, which will shrink and reduce the C-statistic and the net benefit, the clinical utility. So don't be fooled in thinking machine learning or AI methods resolve this, because they don't. They lead to model degradation too. Let's have a look at an example again. Here I'm going to focus on the degradation in terms of R-squared, an overall measure of fit. So again, I'm going to fit models with different sample sizes and repeat the process 1,000 times. I'm actually going to fit 4 different models here: a standard logistic regression, a ridge regression, an elastic net, and a lasso model. If we look when the sample size is quite big, 1,000, we see that the scatter around the R-squared is quite narrow. People— these models are getting an R-squared of about 0.15. As we reduce our sample size down, we see that the R-squared on average is reduced and the scatter increases. So we get lower R-squareds on average compared to when we had larger sample sizes. So lower R-squareds mean lower C-statistics, lower clinical utility. And even, you know, in some models we're getting R-squareds of around 0. So the assurance you have when developing models with smaller sample sizes is low, and you're likely to get worse performance and even some ridiculously low performance because of that. Number 5 is miscalibrated predictions. And basically this is the same as the previous point, that your, your degradation leads to problems in performance. Here let's focus on miscalibration. So this means that your estimated risks, your predictions, won't agree with observed risks, observed values. Like I said, penalization approaches such as LASSO are supposed to deal with this problem, but they don't because their data is used to estimate the shrinkage. So there's uncertainty around the shrinkage, um, and therefore there's no guarantee that, that penalization methods and other methods like random forests will, will address overfitting and issues with miscalibration. Let's have a look at an example. So here's an example where I've applied a lasso logistic regression model to develop a prediction model. And this was a huge dataset of 40,000 individuals, 3,000 events, huge data. Again, about 10 predictors, so massive data. And I've repeated this 1,000 times with bootstrapping, and each time I've plotted the calibration curve. Um, so the solid black line there is ideal calibration, and these curves here, these lighter gray lines, are very narrowly scattered around each other, and they're very close to that diagonal line. So what we're seeing here is that there doesn't seem to be much instability in calibration, that it's very close to the 45-degree line. So we'd expect this to do— this model to do well in new data. However, if I reduce the data set down to 500 participants 35 events, and I repeat this process, we see huge scatter in calibration curves. So there's no guarantee that our calibration performance in new data will be close to the ideal line. In some situations, my model may, um, underpredict, and in other situations it may overpredict. So small sample sizes lead to miscalibration, likely miscalibration. In new data. Here's an example of miscalibration. Franz Beckenbauer, a famous German footballer: one year I played 15 months. So clear example of miscalibration. Let's just pause. Does this calibration plot suggest the model has good calibration? So I'm showing you here predicted risk against actual risk. And the red line is ideal calibration. And I've got a calibration curve fitted for my evaluation data. And clearly, I think you would agree, the calibration is horrific. We have people with estimated risks— sorry, predicted risk down towards 0 whose observed risk is around 0.5. And similarly down here, people with predicted risk of 0.3, their actual risk is about 0. So there's huge miscalibration. The trouble is there's a lot of spin in the prediction model healthcare field. Lots of spin that people essentially lie or try and interpret findings to suit their narrative. Here, this is what the authors said: the calibration curve showed a good agreement between the predictive risk and the actual probability. It's just, just not true. But the issue of small sample sizes perpetuates this narrative because people can try and spin things more with small sample sizes if there's a lot of uncertainty around it. So when you've got large uncertainty in evaluation, that means you're going to get wide confidence intervals around your performance measures, like a wide confidence interval around the C-statistic or the calibration slope. And this leads to people saying, oh, it validated well, or my model is validated, when actually Well, you haven't validated it at all. You've done a validation study, but the model is not valid because the uncertainty around it is so big. Let's have a look at an example. So this is one of our COVID models, and they used— they developed a model using XGBoost, very popular machine learning method, and concluded that the model can predict death risk accurately. That's a strong statement. Remember, this, this model could be used for making decisions on who to keep in hospital, for example. Um, but let's look more closely. The C-statistic is 1. So is this why they say the model can predict death risk accurately? So confidence interval goes from 1 to 1. So this looks like it's exciting, maybe. But the reality is, when you dig a bit deeper, there was just 7 events in this study. Um, and when you have 7 events and you're trying to work out the z-statistic, and there's basically— it's separating people out completely, statistically you can't really estimate the uncertainty. And this is why you get these odd confidence intervals that are way too precise. You can see the uncertainty simply by these huge steps in the plot. Remember I said that the C-statistic is the area under the ROC curve, and the ROC plot here is just hugely uncertain because of these huge steps. And there were no calibration checks in this plot either. So small sample sizes lead to some weird results, they lead to spin, and we need to push back against papers like this. They shouldn't be published. Okay, so that's, I hope, really relevant to a lot of you working in the field in terms of why sample size is important for healthcare research. In the last bit of this talk, I'm going to focus on some solutions for how we may think about what sample sizes we actually do need. Going back to him, commentating, when they don't score, they hardly ever win. Well, what I want is some of these sample size calculations to be used to give you the best chance of developing a prediction model that will be a winner. First thing to note is that sample size calculations for prediction model research are not about statistical power or type 1 or type 2 errors, which are synonymous with traditional medical research such as randomized trials. There are no hypothesis tests here, okay? So the focus is actually on the model predictions. There's no test of whether a treatment effect is, um, bigger than 1 or something. It's— there's nothing like that. Rather, actually what you're trying to do is you're trying to precisely estimate a model's predictive performance. Or in a development situation, you're trying to optimize the accuracy of your predictions, and that's relative to the development method that you're using, the number of predictors and the parameters you're thinking of fitting, the number of participants and events. So we want to identify a minimum sample size to have some confidence about prediction model's performance and in some way to make sure we maximize the accuracy of predictions that we can get out. Let's now focus firstly on model evaluation situations, so where we're validating an existing model. In this area, there's been rules of thumb proposed which are, you know, a reasonable starting point. Um, they have suggested things like 100 events are needed, at least 100 events, 100 events for calibration. Sometimes that suggests at least 200 events Um, but rules of thumb are rules of thumb, and they're not particularly scientific. Hence, we've tried to address this with a more tailored scientific approach that will allow you to actually account for aspects about the model that's being validated, aspects about the target population you're testing the model in, in terms of maybe the outcome prevalence, for example, and other aspects. I'm not going to go into the technical details, but if you want more, then you can read our BMJ paper that summarizes some statistical work on that topic that's listed below. But ultimately, you can apply the method in pmvalsumpsize, which is available in R, Stata, and Python. This is just one line of code, code, and it calculates the minimum sample size required to target precise estimates, and the user defines what precise means. In terms of calibration, discrimination, and clinical utility. So all the measures that I've mentioned so far, we can target having a narrow confidence interval around. We don't have to use all of these measures. Indeed, some people would not suggest to think about confidence intervals around net benefit, but you can decide which of these measures are the most relevant to your sample size calculation. The user defines the outcome prevalence, the calibration slope, and the O of E that are expected. So like any sample size calculation, we have to make assumptions. What is the true prevalence slope and O of E? We suggest defaults of 1 for these so that we assume the model is well calibrated. We could take a C-statistic from the development data that was suggested in that. We also have to specify the distribution of predicted risk. So what's the range of risks that we're expecting in the target population? If we don't know this, we can assume normality and approximate it from a z-statistic. And again, the package will do this for you. And like I said, we have to target what precision we want, so we have to specify what the target 95% confidence interval widths are. But the user needs to specify things like you would with any sample size calculation, but then the calculation is, is very, very quick. After that, here's an example. We looked at doing a sample size calculation for a model that had been developed to estimate, estimate the failure risk of mechanical heart valve within 12 months. We wanted to target a confidence interval width of 0.1 around the z-statistic and 0.3 for the calibration slope. We, based on the model development data, we knew that the distribution of logit predicted risk was approximately skewed normal. They presented the histogram of risks in the paper and we approximated with that essentially. The outcome proportion was very small, 0.018, and the C-statistic of 0.08— 0.8 is from the development data paper. And again, like I said before, we assume the model calibrates well. That's a good place to start. So the one line of code is pervalsumsize and then the options here. This is the Stata code, but these R and Python codes very similar. Type B, this is just denoting binary. There's continuous outcome options and survival options as well. We input the prevalence value, the C-stat, the distribution, um, of the linear predictor skew distribution that I mentioned, the target confidence interval widths for the various measures, and we apply that. And within a split second, it will give you the answer. And focusing on 2 criteria here See statistical calibration slope. They both suggest that we need about 4,300 participants, and ultimately this— the calibration slope is the one that's driving the final answer, which is a recommended minimum of 4,336 participants. Now remember that the outcome prevalence proportion is very low here, so that relates to 79 events. So even though those rules of thumb suggested at least 200 participants sorry, 200 events for calibration. What we're showing here is that that rule of thumb is not appropriate in this situation, that actually 79 events would, would be fine to estimate very precisely the calibration slope, and hence why we recommend this method rather than just blindly assuming 100 events or 200 events. Some emerging and extended ideas— we just published a paper Rebecca Whistle, myself, and colleagues on extensions of PMVal sample size to look at classification measures that are often used in machine learning evaluation models. So for example, the F1 score, accuracy, sensitivity, and specificity. Have a look at that, it's embedded in PMVal sample size now. And a paper that I, I did with, um, Mosin-Satoshi in Statistics in Medicine, it's literally just just come out, is where we, rather than assuming a single true value for each of the performance measures— so for example, rather than assume that the calibration slope is 1— we take a Bayesian approach and allow for uncertainty around those true values. And this allows us then to propagate that uncertainty through the sample size calculation, so we get essentially distributions of expected performance And we're— this allows us to calculate something called assurance probabilities, which I really, really like. And this will tell us the sample size needed to obtain, for example, a probability that the confidence interval width for the calibration slope will be less than 0.3 of 90%. So rather than targeting an expected confidence interval width of a particular size, we can say, well, what we want to do is, is make sure that in at least 95% of validation studies of this sample size that the width, the calibration slope, will be less than a certain magnitude of uncertainty. So check out that paper. I think you'll, you'll really like that paper. And to finish, sample size calculations for model development. So whereas model evaluation focuses on the precision of performance estimates, Here I'm going to be focusing more on sort of making sure that we're not overfitting our model too much. And when we think about overfitting, Pelé gave a really great example. He said that, I think France, Germany, Spain, Holland, and England will join Brazil in the semi-finals. Well, only 4 people can be in the semi-finals, so clearly that is an overfitted extreme prediction. There's been lots of rules of thumb and lots of confusion in this area too. So you— I'm sure many people have heard of the 10 events per variable rule. Um, some people suggest 5 events per variable. Lots of people, including myself, would say there's no rationale for any events per variable rule anymore. And the reason for that is that the work that we've done has really clearly shown that it's not just the number of events that's important. It's also the total number of participants. It's the anticipated performance of the model as well, like the R-squared value. So this led us to produce some more scientific equations for calculating the sample size needed to develop a prediction model conditional on the number of predictor parameters that you're thinking of putting in the model. This has been implemented in a package called PMsampsize. So the Validation one was pm_val_sumsize. Here this is pm_sumsize. So I'm not going to go into the maths behind the scenes, but basically the— there are 3 calculations that the package does. It looks to calculate the minimum sum size needed to have a small amount of overfitting in predictor effects, a small amount of overfitting in the overall fit, and to ensure precise estimate of the overall risk. I always say that If the overall risk cannot be precisely estimated, then what chance have you got to estimate an individual's risk precisely? So estimate the overall risk precisely, ensure that the R-squared is estimated well and not too optimistic, too much overfitting in that, and that in terms of the predictor effects that we have, let's say, targeting overfitting of 10% or less, which relates to a uniform shrinkage of 0.9. So you'd multiply the predictor effects by 0.9 in a standard regression. Now, this is therefore based on regression theory, but this is still relevant for people using machine learning or other approaches that are attributed to AI. Those methods tend to allow by default lots of interactions and nonlinearity, and therefore have more parameters. So I think the minimum for a regression model is usually still highly relevant for machine learning, and we'll see later on that you tend to need a lot bigger sample sizes for machine learning. Um, okay, so the focus now is on population-level stability, not individual level, hence minimum required. Like I said, this is available in pmsumsize for R, Stata, and Python. The user now inputs similar things to the validation sample size calculation, like the outcome proportion, the anticipated C-statistic or R-squared. But also now they also have to specify the number of candidate predictor parameters, the target shrinkage factor. Like I said, our default is 0.9, so 10% overfitting. Target optimism of R-squared, again we suggest that's quite small difference, absolute difference of 0.05. And we'd have to specify a target confidence interval width for the overall risk. Again, we specified about 0.1. So let's look at an example. Here we've got PMsump size type B for binary again. Again, there's continuous and time-to-event options. What's the C-statistic in this example based on previous models in the field? And remember I showed you earlier, there's always going to be previous models in the field. Previous models suggest they could do about a point— C-statistic about 0.86. We're going to use 11 predictor parameters, and the prevalence of the outcome in the population here is 0.5. We see the 3 criteria, and actually the largest sample size needed is for criteria 3, ensuring that the overall risk is precisely estimated, and that relates to about 200 events because there's 50% prevalence, and an events per predictor parameter of about 17. Okay, so remember the 10 events per variable rule? Well, clearly you can see here that that's quite different than 10 events per variable. Now what if we change the C-statistic to be much lower, 0.66? So this is a situation where there's less discrimination, so there's more noise, and How do we deal with that? Well, we need a larger sample size to ensure the same level of overfitting, which makes sense because there's more noise, so you need a bigger sample size to be able to depict it. So now it's criteria 1 that gives us the sample size required, which is about 1,200. So we need about 600 events now, an EPP of about 55. So the EPP, the events per predict parameter, does change depending on what assumptions you're using. And again, so this is why I say say it's a bit more scientific method than just blanket rules of thumb. So have a think about PMsumsize for your own research. Note some extensions and emerging ideas. Pavlo have shown that PMsumsize, which is based on closed-form mathematical approximations, can sometimes be inaccurate. And if they apply a simulation-based approach instead in their samplesize_dev package, then sometimes you get larger sample size, uh, minimum sample sizes that have been suggested. So check that out, it's a lovely, lovely paper. Also, like I said before, our criteria focus on population-level stability, but in recent work we've suggested extensions to individual-level stability. So this is where you're trying to ensure precise estimates of risk at the individual level, or for most individuals in the population. And essentially this needs an extension to PMsumsize, and this is called PMstabilityss, and this is available in Stata and R. So if you're interested in ensuring stability around individual level risks, which often we should be in prediction model research in healthcare, then check that one out. Now what about machine learning and AI approaches? So we've been talking about criteria based on regression, but some people say to me, oh, for machine learning and AI approaches it doesn't matter about sample size, do they address that? Well, that's not true. They are not magic. They also have uncertainty in their predictions. It's just not exposed or very clear, and I really want to emphasize that, um, you typically need a much bigger dataset to fit the machine learning and AI-based methods, particularly using default settings where, for example, in a random, random forest where you have lots of trees and branches by default, essentially to try and pick up lots of interactions and so forth and lots of nonlinear relationships, you need much bigger sample sizes to be able to look for those things and to get the advantages of those approaches. Than regression-based approaches. So don't be fooled into thinking machine learning approaches address all the problems. If you don't believe me, then have a look at these papers. So van der Ploeg in 2014: modern techniques may need over 10 times as many events per variable to achieve a stable AUC and a small optimism than logistic regression. This implies that such modern techniques should only be used in medical prediction problems if very large datasets are available. We also looked at penalization methods such as LASSO and elastic net and concluded they are not a carte blanche. Despite what people say, they do not guarantee a reliable prediction model is developed, and they are more unreliable when needed most, i.e., in situations where you have thousands of predictors and only, let's say, 20 or 100 events. These methods will allow you to fit a prediction model, but they do not guarantee it will be reliable. And what do I mean by that? Well, think about what I said earlier in terms of degradation of performance, instability in calibration, reduction in clinical utility. All those things are still applicable for machine learning and AI-based methods, and don't let anybody tell you otherwise. Lastly, I want to just mention a generalized approach which I think will be a solution for any statistical or AI development method. So this is available as a preprint at the moment. We're hoping it'll be published soon. There's the QR code, and I will put a link into the video underneath in the comments section. So basically what we're proposing is that researchers and the method anticipates what the posterior distribution of performance will be. So what the predictive distribution of individual predictions will be. Remember I talked about this multiverse of models and uncertainty of predictions ranging potentially from 0 to 1 sometimes. Well, if you can anticipate what that predictive distribution will be, then we can use that to work out assurance measures such as the probability that the calibration slope will be between 0.9 and 1.1, or the probability that the net benefit of your model will be greater than 90% of the net benefit that's maximum from a true model. So the, the process requires you to specify what you think your true model would be, a reference model, and based on, again, entering some very simple things a lot of the time, we can work out anticipated predictive distributions. Anyway, this is emerging work, and I'm sure you might be interested in that, but I'm not going to speak about this anymore here. That'll be on another video coming in the next few months. So lastly, a final quote: And Wigan Athletic are certain to be promoted barring a mathematical tragedy. Well, hopefully you don't think any of what I've talked about so far has been a mathematical tragedy. And hopefully it's the opposite of that. You've actually learned about the importance of sample size. So in summary, if you've got small sample sizes, don't ignore it, otherwise it could blow up in your face. And remember, small sample sizes lead to a lack of generalizability, they perpetuate fairness concerns, they reduce model performance. They lead to degradation. They lead to instability. They lead to imprecision. And they reduce clinical utility. So ensure your sample sizes are as large as you can possibly get them to have better models with better clinical utility, better impact on patient outcomes. Don't always think that your dataset that you've got in the freezer is fine. it may not be suitable. If it's too small, don't use it. And consider our packages PM_SumpSize and PM_Val_SumpSize and the emerging work that I've talked about for doing the calculations and for looking at this yourself. Okay, I'll leave you with some references for our work, and I really do hope that's been informative and that you do think about sample size when developing and validating prediction models for healthcare. Thanks.