A general sample size framework for clinical prediction models | Prof. Richard Riley

Hi everyone, so welcome to today's TAP seminar. Thank you all for joining us today. So I'm Rebecca Whittle, I'm one of the organisers of these TAP seminars here at the University of Birmingham. Before we begin, just a quick reminder for those who are joining us online, could you please keep your microphones muted during the presentation? We will have time for questions at the end, so please either add your question to the chat and we can read it out for you, Or you can raise your hand at the end to ask your question directly. And just to note that we are recording the seminar, so if you speak, you do speak during the session, you may be on the recording. Many of you have already signed up to join our mailing list to hear about future TAP seminars, but if anyone would like to join the mailing list, then please either add your email in the chat or send me a message and I can add you and you will hear about future seminars. So today we are delighted to welcome Professor Richard Riley from here at the University of Birmingham. Sir Richard is a professor in biostatistics and is also statistics editor at the BMJ. And today he's going to talk about the general sample size framework for developing or updating a predictor algorithm. So I will hand over to you, Richard.

Thanks so much, Becky. Thank you very much. And thank you to everybody who's come in person and online. Yeah, so I'm talking about sample size calculations for developing prediction models. The focus is on clinical prediction models, so prediction models applied to health, but really this could be applied to any setting where prediction models are used. Acknowledge my funders who supported us, and in particular, acknowledge the collaborators who have contributed heavily to the work that I'm going to present today. Right, so introduction. Just because it's very hot and I don't want people falling asleep, I've littered this talk with World Cup commentary quotes because it's obviously very topical. So these are funny quotes, supposedly. So John Watson in 1982 said, the World Cup is a truly international event. So every 6 or 7 slides or so, I'll introduce another quote just to wake you all up. Right. So what is a clinical prediction model? So a prediction model is any object. This could be an equation defined by a regression equation or a black box from machine learning methods, neural network, or random forest or something. that's used for estimating, predicting outcomes in individuals. So predicting outcome values, continuous values, or particularly I'm going to focus on today predicting, estimating an individual's risk of a particular outcome occurring. So in this particular model here, we have the CRASH tool, and behind the scenes we've got a logistic regression equation. And we've got multiple predictors, country, age, Glasgow Coma Score, and so forth. And to apply the model to an individual, we input values of those predictors for that individual, and the model will give us a prediction. So for this individual from Australia, their risk of 14-day mortality after a traumatic brain injury is 14%, as estimated by this clinical prediction model. Clearly, we want models that are useful and that we need to know about their predictive performance. So the estimated risks that we get out of the model, are they in some sense accurate and reliable? And we measure this in healthcare settings by 3 main strands. First is discrimination. This is the separation of risk between those that do and do not have the outcome events. Measured by C-statistics, area under the curve. Calibration, a measure of agreement between predicted risks and observed risks in the target population, measured by aspects such as the calibration slope and the observed or the expected. And clinical utility. So, for example, measures such as net benefit, which tell us if we make— we use the model's predictions to help make decisions, about whether people are above or below a particular threshold of risk, then do we have some benefits from using that model to make those decisions? Now, thousands of models are developed each year. I'm sure most of you know that most of them are absolutely awful. So methodology standards are really, really poor. Like I said, my focus today is on sample size, and sample size traditionally has been quite neglected in this area. So we've done various reviews of machine learning models and models in COVID, and they've all found that large percentages of the models that have been developed do not justify their sample sizes. And indeed, most of them are trained on very small datasets that we would suggest are probably too small to develop a model. Why is this an issue? I hear a lot of comments sometimes from data science field that sample size-related issues are not a concern to modern methods like machine learning methods. But the reality is that's a load of nonsense. And with small datasets, regardless of what model development approach you take, you end up with reduced predictive performance. So lower performance of the predictions than you would otherwise have obtained if your sample size was larger. I'm going to refer to this today as model degradation, so worsening in performance. Aspects related to that are you're going to get miscalibrated predictions, you're going to get lower discrimination performance, and you're going to get lower clinical utility. Now, lower clinical utility means less benefit for patients. So that's important to remember. It's not an arbitrary thing here. The consequences of having a dataset that are too small is that you'll have lower patient benefit than if your dataset was much bigger. You'll get large uncertainty around predictions at the individual level, and there's potential for unfair predictions. Small datasets exacerbate issues to do with certain groups not being well represented in the dataset. And therefore predictions are unlikely to be reliable for that. So I think hopefully you'll just go with me. The small datasets for model development are not a good idea, and we want to understand what size datasets are a good idea. So we've worked hard over the last decade really on sample size calculations for model development to try and improve standards. And we came up with a package called PMsumsize, which is available in R, Stata, and Python. that implements some very basic criteria. The first is that the dataset should be large enough to at least estimate the average risk in the population precisely. If you can't estimate the average risk in the population precisely, how can you estimate individualized risk? We would also suggest that we try and minimize overfitting. This is where small datasets lead to predictions that are too extreme, too high towards 1, too low towards 0. And we want to understand what sample sizes keep overfitting to a minimum, defined by a metric such as the shrinkage factor. So shrinkage of 1 means there's no overfitting. A shrinkage of 0.9, we could maybe target that, and that'd be like a 10% overfitting. So we can think about sample sizes to target small amounts of overfitting. and a precise estimate of the overall risk. We've extended this work also to look at stability of risk estimates at other levels too. The premise of PM sample size, though, that I want to clarify there is that we're sort of talking about a population-level perspective. Let's make sure we have some minimum that's at least ensuring some overall level of stability. In overall risk, in the overall overfitting of predictions. There's been some really nice work by Pavlo and colleagues, and very recently by Alexander Pace and colleagues, who have shown that PM subsized, when evaluated in simulation studies, does well in the majority of situations, but sometimes it gives the minimum end that's too small. if that even makes sense. It's the minimum N, but the minimum N is too small in some situations. What that means is that actually the sample size needed to target an expected shrinkage of 0.9 is slightly more than PM_SampSize recommends. And so they propose a simulation-based approach to address this. So rather than the closed-form mathematical solutions that we use in our original PM_SampSize, they suggest using simulation to evaluate across repeated samples what sample size is needed to obtain an average S of 0.9. So that's a really nice addition to the literature. We've also extended PM_SampleSize to another package called PM_StabilityAssess. This was looking at sample size calculations to try and identify the sample size needed to ensure individual level stability. So remember I talked to you before about PM sample size being about population stability. But if you really push that to saying we want to make sure every individual in the population has a precise prediction, then PM stability SS allows you to do that. So making sure that the confidence interval width around the predictions of everybody is of a certain narrow width. And we've adapted that for continuous, binary, and time-to-event outcomes. So you can see there's a lot of work going on in this area. Now, what we realized is that we needed to try and do something to bring it all together, to try and put it under one umbrella framework, because all these different approaches are not particularly helpful for the practicing statistician who wants to know what the right thing is to do. So we've just published in BMC Medical Research Methods a paper called A General Sample Size Framework, which does, I think, bring all this together in a coherent framework. So that's the aim of the talk today, is to describe this framework. I'm going to talk about how the framework allows us to think about producing sampling distributions and posterior distributions of expected performance, and how it allows us to examine model degradation. And any measure of interest, not just overfitting and average risk. It allows you to look at any measure of interest. So the premise is that embeds all previous sample size calculations, and it's flexible enough to allow any model development approach to be considered and any performance metric. As I begin, I need to introduce some key terminology because this is going to keep coming back. So the first thing is reference model. I'm going to assume in my calculations that there's a reference model. This is an assumed true model for a given set of predictors. So we need to, like in any sample size calculation, assume a truth. You know, if you're doing a sample size calculation for a trial, you assume a true treatment effect. And that's the same premise here. Now we have to assume a true model. All those previous calculations that have gone before They also assume a true model. They just don't quite say it explicitly. So PMSamSize is also assuming a true model under the conditions which we don't really explicitly state. But this is an approach to allow us to assume what the true model is. So the user is going to need to specify that true model. That means if you think about a regression equation, you're going to specify the intercept, the predictor. Weights of that function. And that whole equation would correspond to a particular C-statistic and average risk in the population. And I've mentioned this one already, but the other key thing is model degradation. This is the developed model's performance minus the reference model's performance. So the premise is that you can never do as well as the true model. Okay, so the true model is the best you could ever do if you had an infinite sample size. You cannot do as well as that if the sample size is not infinite. So you're always going to have worsening of performance. And the premise is that the smaller sample size, the further away you're going to get from the true performance is how far you're willing to go. So related concepts here are the loss function, the value of perfect information, epistemic uncertainty, and model-based error. These are all related terms to model degradation. And not doing as well as treatment. Right, methods. So Ron Atkinson quotes, they take their heads up off the ground and they now have a lot to carry on their shoulders. So sticking with this reference model idea, like I said, any sample size calculation assumes a truth. So here the reference model is our truth. And our premise is that we can't do better than the reference model. So our question for the sample size calculation is, what N is needed to target a particular level of degradation? So we're willing to accept we're not going to do as well as the perfect model. But how much lower are we willing to go? So this is a relevant question if you're designing a new cohort study to recruit a certain number of participants to then develop a model. It's a relevant question if you've got an existing dataset in your drawer or in the freezer or on some server somewhere, and you know it's of a particular size, and you want to know, is this going to be suitable to develop a model? This is a really common situation, isn't it? Because we often use healthcare datasets, CPRD, for example, and we'd say, okay, we can get this dataset. Is this going to be sufficient to develop a reliable model? It's also a relevant question in the context of understanding what model complexity we're allowing to have. So when we develop a model, we don't just think about how we're going to develop it. We need to think about what predictors we're going to put in the model. So what's the complexity of the model? And clearly, again, with a limited sample size, we may not be able to think about a very complicated model. And I'll come back onto the implications of that to the sample size calculation later. The reference model idea also aligns with Sir David Cox's response to, to Breiman. So they had this exchange in Statistical Science journal, and Breiman was arguing that machine learning is all you need. This is the way forward. Everyone should be doing machine learning. Statistical approaches are outdated. We don't need to formally specify models and so forth. But Sir David Cox replied, this is really useful in this setting as well. He said, formal models like regression equations are useful and often almost, if not quite, are essential for incisive thinking. They indicate the uncertainty that would apply under somewhat idealized, maybe very idealized conditions, and as such are often a lower bound to real uncertainty. So what we're doing in this sample size calculation is we're expressing our beliefs about what some true model would be based on information that we may have from previous models from the literature. And we're saying in the, in that context of that model, and we're gonna have a particular sample size, what's the uncertainty for developing that model when we've got a particular truth? So if we can understand our sample size in that context, then that gives us maybe some minimum level of uncertainty to move forward. So the reference model I'm going to use throughout this premise is going to be a regression model, because you can express that as an equation. Even if you develop a model using a machine learning method that's more black box, I suggest you have a reference model that's a very transparent regression-based equation for the reasons there. Okay. The next premise now is we've got our reference model, and we're going to choose a particular sample size to examine and call that N. The general premise is that what we're going to do is we want to anticipate what the sampling distribution— from a proper Bayesian perspective, you think of this as a posterior distribution. of model predictions. In other papers, we talked about a multiverse of possible models. So you happen to have a dataset of a particular size to develop your model. What if you happen to have a different dataset the same size from the same population and develop your model on that? And another one of a different size— of the same size, but, you know, different random sample from the same population. How do the models vary? And how do predictions from those multiverse of models vary? So that's the sampling distribution. So if you can try and anticipate the sampling distribution in advance, it tells you something about the uncertainty about how your model is going to be performing after you develop it for one single model. So it tells you distributions of model predictions, and it also tells you distributions of model performance and degradation. So again, this is always conditional on the target population you're wanting to develop the model in. The set of predictors that you're including, the reference model, like I said, and the modeling approach. So, given all those things, can we anticipate what the posterior distribution of predictions and model performance is going to be? How can we do this? There are 2 options. I'm going to focus on the first one today, which is a fully simulation-based approach. So, this is really exact. This allows you to model to have any model development approach to be used. So when you do the simulation and you fit models multiple times, you can fit the model using whatever approach you like that week. But this could be slow and often is slow to do. So therefore, in our paper, we do present some approximations based on Fisher's information matrix, which extends from our previous work, which allows you to, to do this approach in a much quicker but more approximate Wait, I'm not gonna go into that today because you don't want to be here for too long. That's a fantastic penalty, but he'll be gutted it went wide. These are all true. Right, so let's go through the fully simulation-based approach. So the setup phase, choose your sample size N you're interested in. Okay, that's arbitrary. You're just gonna focus on one. Choose a set of candidate predictors. Specify your reference model, and that reference model in that context of that particular target population must have a particular average risk in the population that you— is consistent with previous information, predictor effects, and C-statistics that you think are realistic for that target population. Again, based on previous evidence. You specify a case mix distribution. So you have to specify for the candidate predictors what their joint distribution is. Okay, so it's not necessarily an easy thing to, to do. We'll come back onto that. The next step is to generate a sample of size n. Participants contain their predictor and outcome values from the population. So remember, we know the reference model, so we can generate outcome states as conditional on that reference model, and we can generate predictor values from the case mix distribution. So we can develop— produce a sample of N participants, a random sample of N participants. We then develop our model using any strategy of choice. Doesn't have to be a regression model, could be any method. You then estimate the performance and degradation of that developed model back in your large datasets, maybe of 100,000 or 1 million individuals from that target population up here. We then repeat those steps 999 times, or 9,999 times to get 1,000 models, or 10,000 models. So this is now reflecting the multiverse of models And I'm going to apply each of those models to make predictions in my huge datasets from the population. And so every individual in that large dataset has 1,000 predictions now, one for each of the 1,000 models that I've developed. And for each of those 1,000 models, I'm going to be able to test the performance. And I'm going to test the degradation. Okay, so I know the reference model. And I've developed 1,000 models. I can then test each of those 1,000 models relative to the reference model and see how worse it's doing than the reference model. And so I'm getting a distribution, sampling distribution of predictions at the individual level, and also performance of the model in terms of calibration, net benefits, discrimination. So, so we then just need to summarize those sampling distributions. So how do we summarize? So we can summarize in terms of expected values. So we could say, okay, let's make sure that our sample size is sufficient to target a particular mean value for the C system, or a mean calibration slope of 1, or a mean net benefit of something. We could focus on the variability of the distribution. But this is the one that we really like. This is assurance. So based on the whole distribution of models that you could potentially develop, what's the probability that any single model is going to have a calibration slope between 0.9 and 1.1? So it's gonna be very well calibrated. What's the probability that the net benefit of the model that you develop is going to be greater than the net benefit of competing strategies? So those competing strategies could be just treat everybody or treat nobody. So we want to have an assurance that when we develop the model with a particular sample size, that we're going to do better than other options that are out there in terms of clinical performance. That to me makes a lot of sense in terms of why we're doing this. But degradation is also key here. So again, we could focus on expectation and assurance in terms of degradation. So what's the probability that the net benefit of my model that I'm going to produce is at least going to have 90% of the reference model's net benefit. I can't do as well as the reference model, but I want to do nearly as well, or up to 90% or more. It could be that the sample size that we looked at in the calculations didn't, didn't meet this criteria, didn't work as well. So then we have to change the sample size and repeat until we find the sample size. That does work. George Hamilton, 1994, that was only a yard away from being an inch-perfect pass. A lot of these quotes now are going to be more statsy. Right, so I've given you the premise, but you've probably also already gone, this is not an easy thing to do. So let's try and make it a little bit easier. So first of all, what end do you start with? Well, we recommend You start with the N recommended by PM sample size. So remember, PM sample size is the one that's focusing on trying to achieve some stability at the population level. So why not start with that as your initial N? Of course, if you've got an existing dataset, just put in the N from your existing dataset. Yeah, so you know you've got a dataset size 13,021, your N is 13,021. And you want to see anticipate performance in that concept. What about the case mix distribution? So essentially, for all the predictors you're thinking of looking at, you need to specify what the joint distribution of predictor variables is. So we could use pilot data. So there may be some initial data from that same target population where we can say, okay, based on this pilot data, we've got some indication of what the case mix distribution is. We may already have that existing dataset. So we're not modeling that existing dataset in terms of modeling the outcome. We're just looking at existing datasets to say, let's look at the predicted distribution. We're going to use that observed distribution in that dataset. Sometimes we don't have the data that we want, but there's this synthetic dataset that can be given to us by the people who do have the data. So, for example, CPRD, in many situations, they will give you a synthetic dataset to say this is A not real, but it's mimicking what our dataset would look like, and you can use that to inform the predictor distribution. What reference models to use? I've sort of told you this all the way along, but there are thousands of models out there. There's highly likely going to be at least one, maybe 100 models in your field. So go to the literature, look at reviews of prediction models. and see what typical C statistics are for those models. You might be optimistic and say, well, let's assume the C statistic in the model that I'm going to develop is slightly above that. You would be able to see if they're regression-based equations, predictor weights, and you can look at the relative weights in those equations. Usually, like I said, there's regression-based models, so you can do this. You could specify what the C statistic of the model is and just the relative predictor weights. So you could say that, oh, these predictors are likely to be twice as important as the other ones. So you just specify the relative weights and the C statistic, and then Joey's PMStabilityAssess package will work out the corresponding regression equation. That makes sure that the overall risk and the model equation has those relative predictor weights and the C-statistic. Yeah, and we're going to update PM sample size soon to embed all this as well. Come back on. You can also account for the uncertainty in the reference model. Okay, so in the paper, we actually show how you don't need to specify one reference model. You can actually specify multiple reference models. ASB fully Bayesian, put distributions on all the predictor parameters, and you could sample from the reference model if you want to go down that road. I think that's probably where this may end up, but not today. What if this is still completely disorienting? Well, remember, this is a pragmatic thing that you're doing before you do any modeling. So you do the best you can. So the pragmatic case is that You just specify the C statistic or the R squared value for the model that you anticipate based on previous models. The number of predictor parameters. We can assume all those predictors are independent. They're probably not going to be, but again, just practice. It's just a practical assumption. We can assume that they're all standardized, they're all normal 0, 1, and they all have the same relative weight on the standardized scale. So we're just stripping everything back. And actually, that is what PM sum size and a lot of those previous calculations already do. It's just we don't— we didn't say it at the time, probably because we didn't realize. So you can strip it right back. So the options that we'll produce in the future in our software packages will have that as the default. And then you can add in more stronger information than you do have. Right, let's have a look at a real example that we use in the paper, adverse outcomes in preeclampsia. So the aim here is to develop a prediction model for the risk of an adverse outcome in women who've got preeclampsia. So the question is, if we're going to develop a new model, what size cohort study do we need to have? We're going to assume that we're going to recruit new patients, so we need a new cohort study. So our setup phase is that we're going to think about 10 parameters related to 9 predictors. We're going to take the case mix distribution based on synthetic data that we got from a real previous cohort study. And we have our reference model based on a previous model published in the field, which was a logistic regression equation. And the performance of that model, when it was examined in the validation study, had a C-statistic of 0.76 and a net benefit of 0.41 at a threshold for making decisions of 50% risk. The equation is given on the bottom there, but we don't need to go into that. I'm just showing you that I'm using an existing equation, and that equation corresponds to a particular C-statistic. And so I'm going to now apply the fully simulation-based approach. I'm going to use 2,000 simulations. So I'm going to sample huge datasets from that population with true outcome values based on my reference model. I'm going to have a look at particular sample sizes and generate models. With those sample sizes and compare the performance to the true reference model. And I'm going to focus on different modeling approaches. So I don't need to focus on just one. I'm going to look at standard logistic regression. I'm going to look at penalized logistic regression with the LASSO and ridge penalties. I'm going to look at Bayesian approaches to LASSO and ridge regression with Bayesian priors. And I'm going to look at random forests. In the paper, we also look at some other more sexy machine learning methods which don't really change things. So let's not bother with those today. I'm going to look at various sample sizes as well. I'm going to start with 4, 5, 6, 310 events, which is the minimum sample size recommended by PM-SampStat. Gary Lineker has now scored 37 goals. That's precisely twice as many as last year. Right, some key results. So we've applied all these different sample sizes, and I'm just showing you some of the key messages, right? So first of all, your smaller sample sizes lead to lower expected performance, higher model degradation, and lower assurance probabilities. This just confirms what I already said to you. The lower your sample size, the less assurance you're going to get that any one model that you produce is going to meet a good standard in terms of calibration, discrimination, clinical utility. And the other key message is that the sample size required depends on you and what you choose to be your estimate of interest, modeling choices, and targets. So the sample size required will change depending on the modeling approach. If you have a method that allows for interactions and lots of terms, you're going to need a larger sample size than a method that keeps things much more simple. It also depends on your approaches to deal with overfitting. So how you deal with penalization of predictor effects, what— how you do tuning and shrinkage. It would also depend on how much data you hold out if you do that approach to tune your model. Like I said, it depends on the metrics you're interested in. It depends on whether your focus is on expectation, variance, or assurance probabilities, and it depends on clinical context. So it depends on your threshold choices. Right, so remember, 4, 5, 6 is the PM start times minimum, but I'm finally going to give you some results of these. So first of all, the main message, results 1. This approach allows you and provides for you posterior distributions for any metric. So what you output from this is a sampling posterior distribution. So on the left here, I've— so in context, I've used LASSO logistic regression to develop a model using PM sample sizes 4, 5, 6. This is the distribution calibration slopes that I would anticipate in the multiverse of all models that I could produce. in a dataset of that size. We see that it's centered on 1, and it goes from what, about 0 to up to 2. Net benefit, this is the posterior distribution. It goes from about low, just lower than 90% to 100% of the maximum net benefit we could have got from the reference model. So you can see a whole distribution that is beautiful. Result 2, lower sample sizes equals wider distributions and lower clinical utility. So this is what I just had on before. But if I was to lower the sample size to 75, clearly you see those distributions get a lot wider. 4, 5, 6, 75. So now we've got more uncertainty about performance, and indeed, in terms of net benefits relative to the true model. There's a shift down to the 90% mark, and there's not much above it. We can present our distributions in another way, which I call prediction instability plots. So again, this is a 4, 5, 6 with LASSO models. I'm presenting here for the individuals in the target population what the variability in their predictions is. compared to the true risk. So, for every individual in the population, I know what their true risk is because I've designed it. And look at the variability of their individual values. So, this is based on 4, 5, 6. This is quite wide still. You know, somebody who predicts a risk should be 0.4. Any one model might be up to 1 or down to 0. But that dotted line gives you a sort of a 95% range. And this is the calibration stability plot. So there's a little bit of variability in the calibration performance, but maybe not ridiculously so. Again, if I was to lower the sample size down to 75, so much lower than PM sample size would recommend, we get this ridiculous plot on the left where, you know, any individual could have a prediction from 0 to 1 depending on what model happens to be produced. So there's low assurance there that that model is going to work well for individuals. This is the calibration performance that we might anticipate in the population. Clearly, there's a lot of potential from this calibration. Rudy Hullett said we must have had 99% of the game. It was the other 3% that cost us the match. Fourthly, we can summarize any metric we're interested in, and we can compare the impacts on those metrics across different sample sizes. So let's look at using a standard logistic regression equation. And I'm going to use the PM sample size recommended, 456, and then this lower ridiculous one of 75. What's the expected C statistic? So what's the average C statistic of all the models that I could have produced? It's 0.75 at 456 and 0.68 at 75. There's the degradation. Remember, the true model C statistic was 0.76. So the degradation in the C statistic is 0.01 and 0.08. The expected calibration slope is 0.89. PM sample size is 456 and 0.45 at 75. This is based on logistic regression without any penalization. We can also produce probability measures. So what's the probability the calibration slope will be between 0.9 and 1.1, just 36% with PM subsize and 1% with a 75 subsize. We can look at the probability that our net benefit from our model is going to be at least 90% of the maximum net benefit that we could have got from a reference model. There's a probability of 100% of getting that from 4, 5, 6, and just 40% with 75. I've just highlighted in red there something very important. That's the fact that PM sample size is targeting— and one of the criteria is targeting an expected calibration slope of 0.9. Remember that S of 0.9, that's 10% shrinkage. So it's not doing badly. So that on average is pretty close to 0.9. So that's why PM sample size works well, but it's only focusing on one metric. So it's fine if you're just focused on making sure on average across all those models in the multiverse that you would get a calibration slope of 0.9. Then that's fine. PM sample size is fine. But what this method does is say, well, hang on. We actually don't have a lot of assurance of getting a calibration slope between 0.9 and 1.1. Surely that's what we wanted. And with Net benefit, there's a large probability of getting at least 90% of the information. C-statistic would only drop by 0.01. So we've got all these different things that we can look at, and it depends on the user in terms of what it is that's important to them. Of course, we'd never use logistic regression. We'd use LASSO logistic regression or some form of shrinkage after logistic regression. So now we use LASSO logistic regression. We see that our expected calibration slope is 0.99, which we want. That should be close to 1. And at 75, again, it's not too far off 1. And we now have better probabilities here, 0.53, that the calibration slope will be very close to 1. Remember, before it was about 0.3. And at 75, it's gone up from 1% to 13%. So what we're seeing here is that the choice of the modeling method has an impact on what the metric performance is. We can also compare different modeling strategies. So against the 4, 5, 6, we compare what a Bayesian LASSO approach will give you compared to fitting a logistic regression and then using a uniform post hoc shrinkage estimate. We get very, very similar results. Both approaches, 100% probability of getting at least 9% of the net benefit of the reference model. We can look at different tuning approaches. So let's say, oh, I am obsessed with random forests. I'm always going to use a random forest. My only decision is whether I'm going to use a depth of 3 or a depth of 5. Clearly, as you increase the depth, therefore you increase the complexity of the modeling. Your expected z-statistic goes down because you're essentially overfitting, and there's more noise. So you can compare and see, for example, the bottom probability of getting at least 9% of the net benefit of the reference model. You have a 95% probability with a depth of 3 and only a 70% probability with a depth of 50. Okay, so it helps you make decisions about complexity. What's the impact of noise variables? So we've got our reference model. What if we say that some people say, well, okay, that's the reference model, but I may also look at introducing 10 other predictors because I'm hopeful maybe one of them will be important. Well, maybe a conservative thing is to say, okay, but what if they're not important? What's the impact of that? The impact of that is you get more degradation. So let's say you added 10 predictors to the approaches I've just done before. They're all noise variables. And again, looking at an N of 75, use Bayesian ridge regression. The probability of getting a net benefit that's at least 90% of the true models is now just 0.14 compared to if we did not use those 10 noise variables, it was 0.68. So the impact of you throwing lots of things in which are likely to be not important is you're going to produce a model that's less clinically useful. So again, sample size has to be in this context. And lastly, threshold choice is important. So in what I've done so far, I've used a threshold of 0.5 to define when somebody's risk is high. That was based on the clinical papers. But actually, at that threshold, even when the sample size is 75, and you fit frequentist lasso, you would expect to get on average a 90% net benefit at the maximum. And there's a 95% probability that your net benefit will be at least 80% of the maximum. So, this is a little bit of a note of caution that Although small sample sizes are not recommended in general, there can still be some clinical benefit. It's just not likely to be guaranteed for one individual, but in a population setting, there would be potentially some benefits. Just, we probably wouldn't know which people would be the ones to benefit. That even makes sense. And the reason why that is the case here is that the threshold is miles away from the average risk. So the average risk is about 0.7 or something, and here the threshold is 0.5. So actually what you're doing is you're just moving to a situation where you're treating everybody, and treating everybody here already has quite a high net benefit. So in other situations, the threshold choice may have a very different implication where there's real dramatic loss in clinical results. Again, Northern Ireland are 10 minutes away from their finest victory. And there's 15 minutes to go there. Right, so coming to an end, what N should be targeted? So I've given you a platform for looking at different sample sizes and the impact that those sample sizes have in the context of the predictors, the reference model, the target population, and the modeling strategy. So what N should we use? Well, just to go back to what I've already said, the required N depends on you. Depends on your choices, your estimates, and targets. So what's your modeling approach? What's your key metrics of interest? It may well be net benefit, but it may be you want to go further and make sure precise risks are estimated for individuals, that calibration's ensured. It also depends on whether you're happy to be focusing on whether this is going to be achieved on average in expectation, or whether it's going to be achieved with assurance probabilities. So in any one model, are you going to achieve this? That's really what we should be looking at. And like I said, it also depends on the clinical context and the threshold choice. So in this example, if I'd used logistic regression, standard logistic regression with— and that PM subsample size N of 456, I would expect a calibration slope of 0.89. It's not too far away from 1, but the calibration performance is not likely to be perfectly close to 1. In, in the population. I would get an expected mean absolute prediction error of 0.05, expected C-stat of 0.75, expected degradation of C-stat of 0.01, expected net benefit of 0.4, and I would expect to get about 98% of the maximum net benefit that I could have got. And there's a probability of 1 that the net benefit will be at least 90%. So there's some good criteria here, which is why I think PM sample size still holds up. very well as a minimum. But it could be that you say, well, this is not relevant because I wouldn't fit logistic regression, which is sort of what PM sample size is based on. So I'd want to look at Bayesian ridge regression, and I can only get 75 people. So is that going to be acceptable? Well, it would only be acceptable if your target is to make sure that there's a 95% probability that the net benefit will be at least 80% of the maximum net benefit. Because they would achieve that. But other criteria are not good. So the mean absolute prediction error, you may have wanted it to be 0.05, but from this approach with 75 people, you would expect it to be 0.13. We'd need a sample size of 456 to ensure an expected mean absolute prediction error of 0.05. But again, 456 may not be sufficient if you're wanting to target a probability of 0.9 that the calibration slope is, let's say, between 0.85 and 1.15, because an N of 456 gives a probability of only 70% of achieving that. Rather, we need an N of about 1,000 to have a probability, assurance probability of 90% of achieving a calibration slope between 0.85 and 1.15. Hence, the N required is context-specific. A statistician like me would say it depends. And therefore, it's not just a case of coming to Stata, R, or Python and plugging in values quickly with 5 minutes to spare before the grant application is submitted. What you have to do is you need to sit down with clinicians, with patients, and with other methodologists to identify the targets that you want to achieve for the model in the context of a chosen modelling strategy. And how you think models perform in the field that you're working in. So it needs concerted time and shouldn't be done as a last-minute thing, but sometimes sample size calculations are done. Last 2 minutes. So Sage on R code for this example that I looked at pretty extensively is available on my GitHub page. I've talked about the fully simulation-based approach. It can be time-consuming. R is hugely faster than Stata, but can still take 15 minutes, 20 minutes, depending on which modeling method you're fitting. For example, Bayesian approaches, Bayesian modeling approaches do tend to take a long time. So the next steps are computational solutions and approximations. Keep an eye on PM for sample size because we are going to update that in the coming months. To embed these extensions and to have PM sample size will still be the place to go to for the umbrella sample size for prediction model development. But we will have other options for doing this type of work. And like I said, in the paper, we talk about approximations based on Fisher's information to try and speed things up as well. So, in summary, we propose this general sample size framework to target posterior sampling distributions. This embeds any model development approach because it covers any model development. Sorry, it covers any model development approach and it covers all existing sample size calculations. Yeah. So it covers targeting population stability, individual level stability. It covers the closed-form approximations. It covers the simulation studies we use to focus on calibration slope. All those previous things, this covers it all. And the reason why it covers it all is because you get these beautiful posterior distributions which allow you to look at any performance metric degradation, and in particular assurance probabilities. And like I said, any sample size calculation, this is about doing the best you can with the information available at the time of the grant proposal or time of doing this. So, yeah, do the best you can. Right, I'll leave you with that quote, which you will all know. Thank you very much.

Thank you, Richard, for such a brilliant talk. I will just see whether there are any Questions online first. Before we move on to this, is there anyone in the room who has any questions? Thank

you, Richard. I was just wondering, like, you know, if someone's building a model, but perhaps the model isn't necessarily a prediction model, but maybe they're interested in looking at, you know, odds ratio association between certain things and an outcome, would you suggest to use a similar framework for the sample size, or What would you suggest they should do in that scenario?

So, if you're looking at, like, prognostic effects,

covariate or

something. Yeah, you can always use a simulation-based approach to do that.

Because I've reviewed papers before and they do some— there's some weird, like, gene power thing. And I'm like, hmm, seems— doesn't— but then I've never known, like, is this appropriate if your goal isn't to do individualized predictions? Like, maybe they would argue you wouldn't need As many observations because you're not getting that individualized risk.

Yeah, so in that, in that situation, the focus is on the estimation of your prognostic factor effect. So the odds ratio, the hazard ratio. So you'd want to know what's the power to be able to detect whether that prognostic factor effect is bigger than 1. So your metric there is the single predictor. Whereas here, I'm not as bothered about the predictor effects themselves. I'm more interested in the overall model equation, what the predictions are spitting out, how they're doing. So, yeah, a lot of the calculations for prognostic factor research are very similar to randomized trial situations, but you need to adjust for correlation with other variables.

Yeah, so put a bunch of other things in the model as well. Yeah,

yeah, it's called an inflation factor. They're a closed-form solution, but simulation would apply to any method.

Thank you.

Just have one question online. So, a question online, they've asked, does this sample size calculation approach work for external validation?

No, not the specific one I've outlined here. But the premise of doing simulation for external validation sample size calculations still holds as well. So there's a paper by Kim Snell in Journal of Clinical Epidemiology, I think in about 2018, where she proposed a simulation-based approach to external validation sample size. There's also some closed-form solutions for external validation. sample sizes. If you don't know how to find them, please send me an email. I can send you the papers.

Thank you, Richard. Very, very interesting. Another follow-up question. When you do your Fisher information approximation, I guess that works just for a reduced class of variation models, right? If you want to use as reference model random forest, then you have to switch to Simulation, right?

Yeah, so the Fisher's information, obviously, that's really based on maximum likelihood estimation properties and almost sort of large sample theory. So it does break down, like, at the end of 75, it doesn't do as well as— I think the PM sample size minimum is pretty much okay. But it's— what I would say is that is a very helpful starting point because the simulation can take quite a lot of time. So you could do the approximation just to look at a lot of sample sizes quickly. And then you identify a range which you think, well, this looks like it's a good range. And then maybe use the simulation-based approach to look at a few of those which would take longer. Usually the machine learning, more black box type methods, random forest, usually they would, they would need larger sample sizes anyway. So you can still take these as a minimum when you do regression-based approximations.

And the other question is about if you always need to start from the truth, or what you believe is the best process. of the truth. So you have a good data generating process that you formulate.

Yeah.

Then if you believe that that is the true data generating process, why would you test other models, right, with, you know, different predictors? If you believe that your true data generating process is a simple logistic regression with age and sex, why should I train random forests with 20 predictors? that there might be— the only reason that I see for doing that is that actually you can prove that a misspecified model can perform better than the true model purely in terms of mean squared prediction error. I don't know, under certain conditions. I don't know how that converts to, you know, calibration of other metrics. But yes, I was curious about what you—

I think what the regression model's reference model allows you to do is you can add in lots of complexity if you wanted to. So you can put in nonlinearity. You can put in interactions, so you can sort of add in lots of complexity that would be mirrored by a random forest or something. So you could start with the default that, okay, let's be optimistic. Previous models that assume simple regression approaches have a C-statistic of 0.76. But maybe if I believe that I can improve on this, without the complexity, then I introduce that in my reference model and move the C statistic up to 0.8 with that extra complexity, and then see the sample size I would need to get that. So I think you can include the complexity in there, but I think it's difficult. I mean, I might be wrong, but it's quite difficult to specify the structure of a random forest in this sort of environment. So I'll do the complexity and the regression stuff. You could—

yeah, thank you.

Yeah, we've just got another couple of questions online first. So Warren has said, thank you for the great work. How do you suggest which metrics, posterior distributions, targets for the simulations? I understand it depends on context and model specification, but how so?

So I think usually in a clinical setting, you would have a project team. So you would have clinical experts who know what they're trying to achieve with this model. So they might be saying, okay, if we knew that somebody's risk was above 10%, we would treat them differently, we would monitor them differently. So that's already telling you that they're interested in making a difference on patient outcomes, and therefore measures such as net benefit will be very important. Because there's a threshold for making decisions and they're trying to look at differences. You may also then speak to a patient group and say, what's important to you? What's important that the model achieves? And they may say, oh, they're happy as a group, that they would just want to know that on average in the population, the model was beneficial. Or they may be saying that they want to make sure that every individual is going to benefit, or that every individual has a reliable prediction, and therefore they would have to put precision of estimated risks. So it's asking people, which statisticians, we don't— we're not always very good at asking.

Okay. And just one final question online. I think you already touched on this slightly, but can I use this general approach for sample size calculation to develop machine learning-based risk prediction model?

Yes. Yeah. So you specify your reference model. And like I said, I think the easiest way to do that is through a regression model, but it could add complexity, as much complexity as you want. But then in the simulation, when you fit the models to your sample of data that you've got from the target population, you would use your machine learning method.

I have one final question.

Yeah, I think the one from Clare is about misspecification. So I think you mentioned In the absence of not having any information, so no pilot data, no synthetic data, no real data, you assume equal effects of the predictors.

Yes, on standardized scale.

Yeah. Any comments around how robust that is to misspecification?

Not sure yet. There's— I think there's been some work by Pavel and colleagues that have shown that it's quite robust. For expected values. So, in terms of the expected calibration slope, I'm not sure it'd be as robust to individual level of disturbance. I think the more individualized you get, the more important the specification of the weights and predictor distributions is going to be. But of course, like I said, it's just pragmatic. You know, I would rather people make pragmatic assumptions and did this rather than just blindly assuming everything's going to be fine. And people spending half a million pounds to do research that ends in high assurance of crap.

Can I have one quick question? How can we get machine learners to use this, given that they come out and say that we don't need sample size?

Education, education, education. And statisticians on grant panels flagging this particular issue. Yes, we do need a lot of thought on that.

Sorry, I think you missed the biggest takeaway, right? Don't let any footballers do your accounting.

Yeah, definitely.

Okay, thank you, Richard. So just before we finish, just to wrap up, we have our next TAP seminar and our final TAP seminar of this academic year. on the 14th of July, which will be at 1 PM. And we'll be joined by Dr. Willie Hamilton from the University of Exeter, who will be talking about the ERICA trial, which is evaluating a diagnostic tool in a cluster randomised trial. So if you're already on the mailing list, you should have already received an invite, but if not, then please do send me a message and I can add you to that as well. So please can we all thank Richard again? Thank you.