The importance of sample size for clinical prediction models - NIHR RSS Research Methods Academy

Hi everybody, and welcome to today's NIHR Research Support Service seminar. This is our 2nd seminar session in our new Research Methods Academy series. My name is Lottie, and I'm the scientific coordinator for the RSS methods development work. Just a quick note before we begin, today's session is being recorded so that those who couldn't join us live will be able to catch up later. At the end of the session, we will be having a Q&A. If you have any questions for our presenters, please can you pop them in the Q&A box that you'll find at the bottom of your page. You can upvote questions that you'd most like answers, as we may not be able to get to all of them. The chat is available also for general comments. We won't be taking questions from the chat, but you're very welcome to introduce yourselves there, and members of the team and I will be on hand to offer support. As this is a new series, we're going to ask you for some feedback. We'd really appreciate it as it will help shape the future events, so you'll see some links throughout the session. Thank you all again for joining us. I'm delighted to hand over now to one of our RSS co-directors for a formal welcome, Professor Greta Raate, who is also a professor of primary care and the joint director of the Primed Clinical Trials Unit at UCL. Thank you.

Thank you, Lottie. On behalf of the NIHR Research Support Service, Victoria Cornelius and I would like to welcome you today. Today's presentation, as Lottie mentioned, follows our first seminar last week on sample size in qualitative research. I hope some of you here today had the opportunity to be at our last seminar, which is now available live to watch. The NIHR Research Support Service provides expert support and advice to applied health, public health, and social care researchers on the design and delivery of research, including post-award support. We are a national service where researchers can access the most appropriate support irrespective of where you're based. We will post a link to how to access the RSS in the chat. The RSS Research Methods Academy, based at Imperial and Partners, is coordinating this seminar series, which focuses on innovation and best practice in diverse research methodologies. A key aim is to move these methods from theory into practice. As Lottie has mentioned, we encourage you to sign up for our future seminars and to provide us feedback, as this seminar series is a new initiative. Our chair for this seminar is Professor Amana Omar. She's Professor of Medical Statistics at the Department of Statistical Science here at UCL and head of the Biostatistics Group at the Joint Research Office within the NIHR UCLH UCL Biomedical Research Centre. Romana provides statistical leadership for Primary Clinical Trials Unit at UCL, and she's also an expert advisor in statistics at the RRSS Hub here at Imperial and Partners. I'm going to hand over to Romana now. Thank you, Romana.

Hello. Good afternoon, everyone. So, the title of today's talk is the importance of sample size for for the development and evaluation of clinical prediction models. And the key learning points for today's session are to understand what clinical prediction models are, to understand key measures of a model's predictive performance, to recognize the impact of small sample sizes on prediction model research, to appreciate current and emerging approaches to undertake sample size calculations for model development and evaluation, and to recognize statistical software to implement the approaches. And it is my great pleasure to introduce today's speaker, Professor Richard Riley, who is a professor of biostatistics at the University of Birmingham, leading research in prognosis prediction models and evidence synthesis. Professor Riley is a statistics editor for the BMJ and BMJ Medicine. He's an NIHR senior investigator and a fellow of the Academy of Medical Sciences. He has published over 200 articles, co-edited 2 textbooks, and leads multiple MRC and NIHR-funded projects. And Professor Riley has been championing methodology or playing a role of champion role for methodology in healthcare research. So, I will now pass on to Professor Riley.

Thank you so much. That's very kind of you and really privileged that so many people are watching this and to be invited to give this seminar. So, thank you so much. So, yeah, so I'm talking about the importance of sample size. in relation to clinical prediction model research. And I'd like to thank my funders there at the bottom of the screen and my collaborators. So, introduction. I'm going to litter this talk with football quotes because I think they're quite funny. Footballers often don't make sense. So, Paul Gascoigne allegedly said, I never make predictions and I never will, obviously making a prediction in that statement. And we all make predictions all the time. In healthcare research, we have something called a clinical prediction model, which is looking to make predictions of future outcomes in individuals. Here's an example. You can click on this on the web, the CRASH tool, where it's looking at patients with a traumatic brain injury. You enter in various predictor values, feature values, the country, the age, Glasgow Coma Score, et cetera. And it will output a prediction here that this individual has a 14% mortality risk by 14 days. That's an example of a clinical prediction model. There are many of these in NICE guidelines, so they're a very hot topic. Lots of people are trying to do this sort of research. You may have heard of things like QRisk, the Apgar score, the diabetes— the Leicester Diabetes Risk Score, for example. So more formally, a clinical prediction model is an equation such as a regression equation or a machine learning black box that's used for estimating or predicting a particular outcome or an outcome value for an individual conditional information about that individual. So their predictor values, such as their age, sex, so forth. There are 3 key phases to prediction model research. There's a phase where we develop the model, produce the equation. We then evaluate the model, for example, in new data. And we also evaluate the impacts of using that model on patient outcomes. We also, when we evaluate it, we have multiple measures of evaluation. So we have something called discrimination. So discrimination tells us whether a model's predictions separate out those with and without the outcome. We may measure that through the area under the curve or the C-statistic. We have another measure called calibration, which is whether your observed predictions agree with the predicted— sorry, your observed values agree with your predicted values from the model. We measure that through calibration slopes and the over E statistic you may have heard of. And we also have a clinical utility measure such as net benefit, which tells us the benefit of using that model to make clinical decisions. So before we get going, I thought we'd have a poll. So this question is just trying to gather, you know, your expertise here. So do you have experience with clinical prediction model research? Fill out this poll just so we get a grasp of who's attending this and your level of experience. So we'll give about 30 seconds for this. So a couple of yeses, whether you've led a project or you've collaborated on a project, and a couple of nos. First no is, I haven't yet, but I'm looking to do this in the future. And the final no is no, and you don't have any plans. Okay, so we see that Yeah, so a good mix of individuals there. So a lot of you are collaborating on projects or planning on doing this in the future, and I think that's really where a lot of people are at at the minute. With the advent of AI and machine learning, lots of people are thinking that they need to get into this field, and they're planning to do it but want to know good practice. So this is good timing for this seminar today. However, as Steve McLaren put, when he was England football manager, we have to reduce our expectations of England, and we have the players to do it. And that is very true in prediction model research. We have to reduce our expectations of how well these models are doing. So, if you look at reviews of models, there are hundreds and hundreds of models out there in every disease area. For example, over 400 in COPD prognosis. Should we trust these models? No. Mostly we shouldn't. Why? Because very few are evaluated in new data, and there's common methodology weaknesses. For example, we did a living review in COVID models. Over 90% of them were classified at high risk of bias, so we don't trust the predictions coming out of these models. Now, a major gap in methodology quality is sample size. Now, in most areas of medical research, like trials, everyone would be familiar with a sample size calculation. In prediction model research, what happens is people just have a dataset and they go, it doesn't matter. We're just, we're just going to fit a model to this and develop a model. So if you look at reviews of sample size calculations that are used in applied articles, very few people are thinking about sample size, and they're just using the data that they've got to develop models. So this is a problem, and we've explained this in a recent article published in Lancet Digital Health. So in this talk, I'm going to showcase why small sample sizes are a problem in this field, and I'm going to showcase some potential solutions. Okay, so let's crack on. So, part 2 of the talk, impact of small sample sizes on clinical prediction models. Now, if you develop your AI fantastic method, you may end up in a situation like this where people are saying it's fantastic, but it doesn't work. This is Michael Owen commentating on a football match. That is a fantastic penalty. He'll be gutted it went wide. So very strange quote, but emphasizing that with small sample sizes, you're gonna run into problems despite your fantastic AI hype that you're giving it. So let's go through these. First of all, small data leads to data that's not representative of your target population. So that means your models are not gonna be generalizable. You can't tailor the model to certain subgroups or regions, and it leads to concerns of fairness. So underrepresentation of particular groups in your data. So there's a really lovely paper by Thomason who showed that even in a very big dataset, the effective sample size may be very small for some individuals defined by particular combinations of predictor values. For example, living in a particular region defined by particular ethnic characteristics, et cetera. Second one is small sample sizes lead to large uncertainty in predictor effects and their importance. So you're developing a model. What do you put in the model? You decide what goes in the model, and you estimate the importance of that. Well, if you've got small data, there's going to be a lot of uncertainty about what you select. If you develop another model using a different sample of the same size, you may end up with very different predictors in that model. We call this a multiverse of models in this paper, the clinical prediction models in the multiverse of model madness. Now, if you are in the AI world, you may have heard of explainable AI. We can't explain what's going on when there's all this uncertainty. So again, small sample sizes impact the ability to explain and understand the model. This also then leads to uncertainty in model predictions. So this means that when you develop a model and you give a prediction to an individual, there's a lot of uncertainty behind the scenes in that prediction. In some situations with small sample sizes, you may be outputting a prediction, but actually the uncertainty around that goes from let's say a risk of 0 to 1. This relates to large epistemic uncertainty or model-based error. Here's a very simple example. You may have heard of a logistic regression with a LASSO penalty to deal with overfitting. Here I'm just fitting a model with just 9 predictors with sample sizes ranging from 50 up to 5,000, and I'm showing you across 1,000 different models the variability in predictions for 9 individuals. So here's an individual, here's whose true risk is 0.1, we can see the scatter of estimated predictions from these 1,000 models goes from 0 up to 1 in some situations. Huge uncertainty with small sample size of 50. As you get up to 5,000, the uncertainty is narrowed. So we want to be developing models around this area and not down here. But sadly, many models in healthcare are down this area at the moment in practice. Huge instability. When you are in that small sample size situation, the impact as well is on lower discrimination performance. You can't separate out individuals with and without the outcome very well. And what does that lead to? It leads to model degradation, so worse performance of your model than if you could have had larger sample sizes. That means C-statistics are lower, but crucially, it means Worse patient outcomes. You cannot discriminate between people properly, and therefore you're not ending up giving the high-risk people the treatments or whatever it is they should be having, and that leads to lower benefits on patient outcomes. So the consequences here on healthcare are big. This is not Netflix recommendations. This is about healthcare consequences. We also in healthcare are very conscious about calibration, and if you have small sample sizes, then in new data it's likely that your predictions are going to be miscalibrated. Now, people may say it doesn't matter because we have methods to deal with that called penalization methods that deal with overfitting, and that's true. But the issue is that those methods often use the data at hand to address the overfitting problem, and that is therefore estimated with a lot of uncertainty, and it's— so it doesn't necessarily solve the problem and often won't. Here's Franz Beckenbauer. One year I played 15 months. Well, clearly that's ridiculous, but it's an example of miscalibration. Here's an example of a calibration plot where you're plotting on the x-axis your estimated risk from the model and on the y-axis your observed risk in the validation data. I've developed 1,000 models here with very big sample sizes and plotted 1,000 calibration curves, and you can see they're all very close to the 45-degree line. But when you have small data, what happens is you get this. So, in new data, you get this huge instability that calibration could be far off. So, somebody whose predicted risk is 0.8, their actual observed risk could be about 0.4. So, clearly, you don't want to be in a situation because that's going to dilute clinical utility, and you're telling patients one thing when actually the truth is something else. Here's a real example of miscalibration in the literature. I've taken away the name of the paper. This is terrible calibration. you know, somebody who's predicted risk is 0.4 has an actual observed risk, you know, close to 0.1. But they say in the paper the calibration curve showed a good agreement. Doesn't show a good agreement. There's a lot of spin in the prediction model field. And small sample sizes promote spin. So reviews generally show that evaluation studies are also too small. So not just developing the model but evaluating the model The test dataset's too small. This leads to spurious claims of a model has been validated when it hasn't been validated. It's just the confidence interval is very, very wide. Or you get spurious results. Have a look at this. Now, this is from a paper that claims that this XGBoost model conducted with these particular predictors can predict death risk accurately. So that's a strong claim about COVID-19 prediction model. got a C-statistic of 1, a confidence interval that goes from 1 to 1. That should raise a red flag. If you look closely at the paper, it's got 7 events, 7 deaths. This means you get some weird results where you clearly can't estimate enough in this data. There are also no calibration checks in this paper. So small sample sizes lead to some weird results and spin. Okay, so I've highlighted some of the issues. The next bit of this talk is to think about solutions. Michael Owen, he is back again on his quote, when they don't score, they hardly ever win. That's true of prediction model research. My main message today is, if you don't think about sample size in advance, you're hardly ever going to win. You're hardly ever going to develop a reliable and robust model. Now, the first thing to note is that sample size calculations in this field are not about power and Type 1 and Type 2 errors. So we all know Type 1 and Type 2 errors in randomized trial design. But here in prediction model research, we're not focused on hypothesis tests and p-values. Our focus is on, is the model good enough at making predictions? So are those predictions accurate, and how can we make the most accurate predictions when we develop a model? relative to the available predictors, the number of predictor parameters, and the number of events, and so forth. Okay? So not about statistical power, but about how can we estimate predictive performance precisely, and how can we develop a model that gives us as most predictive information as we can from the predictor information that we've got. So let's focus on model evaluation. So model evaluation, what I mean is this. You've got a new dataset, and you want to evaluate a previously developed model. Does it perform well in terms of those measures I introduced at the start— calibration, discrimination, clinical utility? Now, in this field, there's a lot of rules of thumb, and some of these rules of thumb were a helpful starting point. And these papers suggested that for model evaluation— you may also know that as validation or external validation— they suggest at least 100 events and 100 non-events. In particular, for calibration, you may need at least 200 events as a rule of thumb. But what we've done in recent years is try and tailor sample size calculations to be more exact. So to think about the model at hand, the setting of interest such as the outcome prevalence, how we think the model is going to perform, and tailor our sample size calculations accordingly. And we've implemented this in a package called PMValSampleSize, which is available in R, Stata, and Python. Expertly written by Joey Ensor here at Birmingham. This calculates the minimum sample size to target precise estimates, so narrow confidence intervals around calibration measures such as the O/E or calibration curve, discrimination in terms of the area under the curve or the C-statistic, and if you're interested in clinical utility, the net benefit measure. As for any sample size calculation, the user needs to enter values And so here the user would need to enter what they think the outcome prevalence, the instances in the population, the anticipated calibration slope and O over E. By default, we assume the model works well, so they are 1, and the anticipated C-statistic. We can also enter, if we've got that information, our anticipated distribution of predicted risks in the population. But if you don't have that, you can approximate it from the C-statistic that's entered, and if it's a time-to-event outcome, the mean follow-up time. We also have to specify what confidence interval widths are we targeting in expectation. So how narrow do we want them to be? We have to select that as a user. But let's have a look at this in practice. So let's think of this model. This is a sample size for external validation of a model to estimate risk of failure of a mechanical heart valve within 12 months. So it's a binary outcome. I'm going to say that we want to target— that we evaluate— when we evaluate this model, we want a confidence interval width for the C-statistic of 0.1 and a calibration slope confidence interval width of 0.3. They're arbitrary. In the papers, we talk about how you choose those values. Based on the model development data, we can see that predicted risks have a skewed normal distribution, and we're going to enter the outcome proportion in the population, which we think is going to be 0.018, so very small. C-statistic of 0.8, and again, I'm going to assume the model is well calibrated, so these values are 1. Then it's just a simple one line of code. So pmvalsumsize is the command, type binary, prevalence of the outcome, our assumed C-statistic, the distribution of the predictive risks— we don't necessarily need to specify that, we don't know it— and then the desired confidence interval widths that I've just specified. And that will very quickly, within a split second, give you the sample size required here is driven by that to estimate the calibration slope precisely is just over 4,000. So, because it's a rare outcome, it's 79 events. So again, it's lower than the 100 events rule. So it's actually helpful to understand that in this situation, you need a lower sample size. Okay, what's great about this field is there are lots of emerging papers, and it's not just our group that's working on it. Rebecca Whittle and colleagues have extended the concepts to look at precisely estimating sensitivity, specificity, F1 score, and accuracy. Some of you that work in the machine learning field will know those measures. We think they are secondary to calibration and discrimination, but they still may be of interest. And a lovely paper by Sadazafi, Mohsen Sadazafi, is looking at Bayesian approaches. And accounting for the uncertainty in what we think the model is going to do in terms of the C-statistic or the calibration performance and how you embed that. And that leads to nice properties such as getting the sample size needed to have assurance. So the probability, for example, that your calibration width for the— the width for the calibration slope is going to be less than 0.3 and making sure that you've got a probability of 90% of achieving that target. And I really like that concept. So that's just whetting your appetite for some new initiatives in that field. Okay, let's move on now to the final bit, which is about model development. This is an example of overfitting that Pele spoke about. So this is, I think that France, Germany, Spain, Holland, and England will join Brazil in the semifinals. Well, obviously, you can only have 4 teams in the semifinals. So that's an example of overfitting to the situation at hand. So, it's where you get extreme results. And sample size calculations for model development have often focused on dealing with overfitting. So, overfitting to the data at hand, which won't be generalizable. I bet everybody on this call has heard of this rule of thumb of 10 events per variable, which sadly is embedded in people's minds when they do this sort of research. But many papers have suggested that's not very good. Some say 5 is needed, some say, which I agree with, there's no rationale for any EPV rule, and also that it's not just the number of events that's important, it's also other things as well. So again, to try and address this, we've developed a module called PM_SampSize, which is available in Stata, R, and Python, thanks to Joey Ensel and Rebecca Whittle here at Birmingham. So from 2019, we proposed Based on closed-form sort of statistical equations, how to calculate the minimum sample size you need to develop a model that targets a small amount of overfitting in predictor effects. For example, you may want to target overfitting of 10% or less, which relates to this concept of a uniform shrinkage factor of 0.9 or close to 1. Or, and you want to target a small amount of overfitting in the overall risk. So you may have heard of R-squared, the overall fit of percentage variance explained, and you want to target small optimism in that measure, less than 0.05. And lastly, can we estimate— ensure that we estimate precisely the overall risk? If you can't estimate the average risk or the average value in a population, why are you using a dataset to make individualized predictions if you can't estimate the average risk precisely? Like I said, this is based on the statistical theory based on regression theory, and the focus is on the population level. And so it provides a minimum, a minimum sample size calculation. Again, we need to enter values to apply the sample size calculation, so the user must enter again the outcome proportion or the mean outcome value in the target population, the anticipated C-statistic or R-squared of the model, The number of candidate predictor parameters, so not just the number of variables but the number of parameters we're thinking of putting in, our targets for what we want in terms of the shrinkage factor, like I said, 0.9 would be a good default, and the optimism in R-squared could be 0.05. And again, what the target precision of the overall risk would be, so maybe targeting a confidence interval width of 0.1, such as if the risk was 0.5, the confidence interval will be targeted between 0.45 and 0.55. Subjective decisions go into that like any sample size calculation. Let's look at an example. Don't really need to understand the clinical context here, but the command is PM SampSize. We specify the outcome type here, it's binary. For this model, I'm going to— based on previous models in the field, I think C statistics are around 0.86. We might be hoping that our new model does better, but that's conservative based on previous models. We're going to enter potentially 11 parameters, and we are going to assume that the prevalence of the outcome is 0.5, so 50%. So this automatically, again, in a split second, looks at the 3 criteria and tells us which is the sample size that drives the final sample size calculation. Here, the criteria 3, which is about the overall risk, we need 385 individuals relating to here 17.5 events per predictor parameter. Now, just to show that this is a tailored sample size calculation, if I was to change the C-statistic to 0.66, we now have that we need a larger sample size because the C-statistic is lower. That means there's more noise in the data. Therefore, there's going to be— it's going to be harder to develop a reliable model. And now we need nearly 1,200 participants and 54 events per predictor parameter. So It's very tailored to the specific setting that you're looking at. So now 597 events compared to 193. So one line of code makes it very, very straightforward. They're using these to put in some sensible values to help with their calculation. Again, like I said before, the evaluation— there's a lot of ongoing work in this area. This is a really nice paper that came out last year by Pavlo and colleagues. They show that PM sample size is sometimes inaccurate, so that the sample size you need is bigger than the sample size that PM sample size suggests in some particular situation. So PM sample size is based on closed-form solutions, so statistical theory. What Pavlo and colleagues suggest is to use a simulation-based approach, so use simulation to generate lots of datasets and look to see what happens on average. And that's a lovely package called samplesizeDev in R. We've also recently done some work on looking at moving this beyond population-level stability to individual-level stability. This is in a package called PMStabilitySS. So this is now going beyond the original criteria and also saying, how can we target precise confidence intervals around individual risk estimates? Remember early on I showed you about those graphs where intervals for some individuals going from 0 to 1. We could maybe have sample size calculations to target interval widths that are expected so that we avoid that problem, and that's available in PNStabilitySS. Right, I'm just going to pause again, ask another question for everybody. And so this question is about machine learning, AI, and sample size. So, do you think machine learning and AI methods require the same level of sample size planning as traditional statistical approaches? clinical prediction models. So, all the approaches I've talked about for development so far, really principally based on regression approaches, but there's a lot of hype about AI. So, my question to you is, do you think AI essentially resolves this issue? Yes, sample size is equally important. No, AI methods are more robust and perform well even with small sample sizes. And the last one is unsure. So, just think about that and maybe put that down. We just have 20 more seconds. So, okay, here come the results. Okay, so yes, sample size is equally important for AI methods. 7% don't think they are, and 22% are unsure. So most of you agree with my perspective, which is that actually, yes, sample size is equally important for machine learning and AI methods. So let's have a look at that. It's interesting when I talk to people about sample size in this area, to computer scientists, people from a data science sort of background, they are very much not taught about study design at all. And they will often say, well, it doesn't matter because we have methods that can handle this. They will optimize the predictions in the context of the sample size that you've got. But remember the things I told you at the start, all those principles about model degradation, miscalibration, lower C its lower clinical utility, lack of fairness, all those things apply equally to AI models as well. Have a look at some of these quotes. So, Van der Ploeg did simulation studies to compare AI-based methods and showed that you need over 10 times as many events per variable to achieve a stable C-statistic and overfitting minimum compared to logistic regression. So, good old logistic regression You know, nowadays that would be called a super learner or something like that. But, you know, don't, don't just think that AI solves all these problems because you often need much larger datasets because of the number of parameters those models are fitting. And this quote here, penalization methods are not a carte blanche. They do not guarantee a reliable prediction model is developed. They are more unreliable when needed most. So just bear that in mind. I'd like to finish by pointing you to a new paper. I haven't got the time to go into the details of this new paper now, but just very briefly, this is a paper that we've tried to bring together all of our previous ideas and those of other people and also extend it to some new concepts for model development, which we think works for any model development approach, whether it's a regression, whether it's machine learning, whether it's AI. The premise of this paper, if you have a look at it, is that what we're trying to do is essentially anticipate what the posterior distributions, what the uncertainty certainty distributions of predictions will be for a given sample size, for a given model development approach in the particular target setting of interest. And in particular, we want to try and minimise degradation of a model compared to some reference model. So this has sort of been built in the previous calculations as well, but not explicit. And we want to try and maximise patient benefits and clinical utility. And so we're targeting in this new framework assurance probability. So the probability that when you develop a model that in new data, you will get a calibration slope that's close to 1, or the net benefit of the model will be at least 90% of the maximum net benefit that you could get. So you gain 90% of the benefit on patient outcomes that you want, you could get with that data. So have a look at that. That's exciting work that we're hoping to publish. Um, soon. Right, in summary, do you have small sample sizes for your model development or evaluation? Are you in denial? Are you in a situation where you're overfitting? You're gonna end up with a big shock. Um, and hopefully what I've done today is push back against lots of people who think the opposite and think we don't need to worry about this in this setting. So appropriate sample sizes are important for generalizability, for fairness, to maximize the quality of the model and reduce degradation, to lead to stable performance. Yet remember the issues with miscalibration and making sure confidence interval widths are narrow around individuals, and to maximize the benefits on patient outcomes. If you've got a convenience dataset, so a dataset that's just on your computer or that's in the lab, in the department, people go, let's just use this data. Really stop and think before using that data and think about the sample size calculations that I've talked about and whether that data is suitable. And consider using the software that we've developed, PM_SampleSize and PM_ValSampleSize, alongside others that are emerging or being produced by others. So I'll leave you with some selected references, which are mostly self-indulgent, and a couple of courses at Birmingham that we run on this topic and other related topics. But yeah, thank you so much for listening. I hope that's been helpful and has covered a lot of stuff, but it's given you enough— whet your appetite for reading more about this, and hopefully will lead to you thinking about sample size in more detail when developing and validating your models. Thank you. I'll pass now back to Romana.

Thank you for such an important and insightful talk. While we wait for questions to come up, I'm just— I may take the privilege of asking the first question, which is, what would be your advice to a health researcher or a health research team who wants to develop a risk model for a rare disease which has a low event rate? An example that comes to my mind is hypertrophic cardiomyopathy. And to predict sudden cardiac death. And even after pooling multiple cohorts across multiple countries, they still come up with small numbers. So should they give up?

No, they shouldn't give up. I know there are some people that would say, oh, we shouldn't be developing models at all in those situations. I don't agree with that. The reason is that, and this is particularly addressed, I think, in our new sample size paper, that you can still have benefits. Yeah, so you can still have information that's adding value towards better outcomes in this population, even though you're not getting narrow confidence intervals for everybody. There could still be population benefit. So, for example, you may be in a situation where you can develop a model and you would be able to get let's say, net benefit clinical utility that's at least 70% of the maximum you could get. Now, clearly, we would like to have 90% or 95%, but you're still getting 70%. So there's a— you're going to improve over what's currently there, and that's the key thing. It's an incremental process. So you've got to think about what's your reference point. So if your reference point is that we don't have any information at all to discriminate between individuals in this area, then if your C-statistic is 0.55 for your model, it could still have net benefits over current options. In another situation, you may have the existing models are out there, they have C-statistics of 0.9, and therefore your new model may need a C-statistic of 0.95 to make a difference. But in rare diseases, you could still make a difference even with small sample sizes, but you're not going to get the clinical utility, the precision of individual risks that would be ideal. So you've got to be aware in that situation that there's going to be some compromise. And we certainly shouldn't be throwing thousands of predictors in. We should be very sensible with how many predictors we're putting in. Otherwise, you're very quickly going to dilute any benefit that you get because you're going to degrade the models so that everything shrunk towards zero. You see statistics shrunk very closely to 0.5 or something. So there needs to be some very careful planning between the sample sizes available, the targets that you want to obtain, and the number of predictor parameters that you're willing to put in those models. So it's quite a nuanced issue. And I think my main message there is, what is the current standard of care? What's the current prediction performance of existing models in that area? And how can we do better than that to make a difference that's moving forward to improving that population in terms of the patient outcome?

Thank you. Yeah, I struggled with the number of predictors because I had to develop a risk model for HCM. So would you sort of advise the team to pre-specify the predictors and limit them, which is one way, or the other one is to do variable selection, which has got its own Problems. So,

yeah, I think I would, regardless of whether you're doing variable selection or pre-specifying what goes in the model, I would still use the same equation. So I would have a situation where I'm counting it, pre-specifying how many parameters that are going into my model, even if I'm using variable selection. So even if that variable, some of those variables will not be in the final model, I'm still counting those that I'm thinking about using. Because ultimately, I'm looking at the data to make decisions about what variables and what parameters are ending up in my model. So, yeah, if you, if you're just ending up with a subset of the variables, you still looked at a fuller subset. So I would still think about the full subset. I think one of the issues that we have in this field is that there's a jump between people looking at the average prognosis in a condition and jumping straight from there to developing an individualized prediction model. And I feel that that misses out a hugely important area of research, which I call prognostic factor research, where you're looking to identify those factors that have prognostic value. So, I think in an ideal world, we're developing models on the back of prognostic factor research, where we know from multiple studies what factors are adding prognostic value. Then you're developing models with those factors, and you're more likely to develop a much more reliable model. And you're not in a situation where you're throwing thousands of things in. The issue at the minute is that people jump very quickly to developing a model on small data, throwing thousands of rows, hoping that they'll pick up something that's been missed before. But like I said, by doing that, you're in a situation where there's so much noise, and it will dilute everything, degrade the model performance. write down.

Thank you. I think we have some questions here. Okay, so there is a question on pediatric research in rare diseases where we cannot objectively get the ideal sample size. Does this mean we can never make risk prediction modeling? Which is, you have already answered that question.

Yeah, have a look at that new preprint because I think that addresses it. You just got to accept that your target for the sample size is very different, and you're not going to achieve the individual level of precision that you'd ideally want. But it may still add net benefit. So think about that clinical utility measure. What's the probability that your net benefit is going to be higher than existing strategies, or, and so forth?

And there's a second question about that as well. So I'll move on to a different topic. Which statistical software is best for prediction model?

For developing models, that is a pretty much impossible question to answer. It depends on the package, but in R, if you look at Frank Harrell's hsMiscellaneous package, the RMS regression modeling strategies stuff, there's so much there on prediction modeling. In Stata, you've got PM suite that Joey Ensor has developed, which has got lots of tools for validating and evaluating models in terms of producing calibration plots and things like that. But really, any software like R or Stata and Python will have tools for developing models that are generalizable. So, like penalized regression models, so models such as regression, which allow for adjustment for overfitting, that they include bootstrap methods to examine and adjust for optimism. Or if you're looking at tree-based methods, you've got things like random forests that are built into the software which integrate bootstrapping into the development process. It's such a big question to answer that quickly. Yeah.

There's another question, which is, can predictive models achieve both near-perfect calibration and discrimination, or is there a trade-off between these 2 concepts?

Oh, there's definitely a trade-off. Yeah, you'll often get models that look like they've got very high discrimination performance, but when you look at the calibration, it's terrible. And often to get good calibration, you will have to dilute the discrimination. So this is a big issue with machine learning methods. Machine learning methods and machine learning applications and people who have come through, like I said, a computer science, data science route, they don't tend to be taught about calibration. I don't quite understand why that is, but it's something that's very often ignored and it really infuriates me. So their focus would often be on maximizing the C statistic at the end of the curve, but the reality with that is that you often lead to instability issues in predictions at the individual level and miscalibration in new data. So if you don't examine calibration, you will obviously hide that particular issue. So you need to have this balance, I feel, between discrimination performance and calibration. And again, I think this is where the sample size calculations can help because they're— what they're trying to do is to reduce the potential for miscalibration in new data. And like I said, they're trying to help you optimise the possible C-statistic that you could get. And then if you do both those things, that will then maximise the clinical utility that you can get. There's been some lovely papers by Ben Van Calster, for example, looking at the Achilles heels of The Achilles heel of prediction model research is about calibration. And if you worsen calibration, you worsen clinical utility. And if you worsen the C-statistic, you will worsen clinical utility. So both of those things are important to think about. This is in a healthcare setting where we're making decisions on patient outcomes.

And then there are 2 questions on split data validation in terms of development and validation. So data splitting. So if— so the question is, if they want to develop a model but they cannot find a dataset for external validation, can they use then your sample size formula for external validation to compute the sample size for the holdout fold?

Well, yes, they could. So you could have PMVal sample size, could work out what the sample size is needed for the test data. But what I would strongly say in that situation is don't split your data. If you're in a situation where you have a dataset, I would use all that data to develop the model. And the reason for that is developing a model is very difficult for the reasons that I've shown you. Why would you want to throw data away from model development when I've shown you it lowers clinical utility if you do that, that it makes miscalibration happen, that it reduces the ability to pick up predictors? Rather than doing that, it is better to use all the data and develop the model and use approaches like cross-validation, or in particular bootstrapping, to repeat your model development strategy multiple times with bootstrap data and evaluate the optimism and potentially adjust for it through that process. I haven't got time to go into those specific details. There are some machine learning methods which may not cope with that well, um, through computational reasons, or maybe they already use bootstrap to develop the model, and therefore some data splitting may be needed simply because of the method that's being used for model development. But again, I would say that it will only lead to model degradation and lower performance than if you could have used all your data to develop the model. So what I would suggest is having a look at that new paper, and you can then work out in your development datasets the sample size needed to develop a model in that development dataset. So the model development sample size That is then gonna give you a high probability of achieving good performance, however you define that, in new data in the future when you do get that new dataset to validate the model. I hope that helps.

Thank you. Then there's a question about sample size for regression outcome types that are not binary, in particular about categorical, I guess.

Yeah, so PMsumsize can handle, like I said, binary data. It can also handle time-to-event data. So with time-to-event data, you have to think about average length of follow-up as well, not just the outcome proportion, but also the length of follow-up. And it will also handle continuous outcomes. So in that situation, rather than putting in the outcome proportion, you would put in the expected mean value of the outcome in the population, and you'd have to put in the R-squared so that, you know, what you thought the percentage of variability explained would be and the variability of the outcome. So there's a slight change in the inputs. There's work being done on multinomial outcomes, so colleagues that I work with in Manchester, they've looked at sample size extensions to multinomial outcomes So just, you just have a search for that paper. You should come across that paper from Alexander Pate, P-A-T-E, Sample Size Calculations for Multinomial Outcomes.

Thank you. There's a question about, do you have any advice about the sample size section of NIHR applications for AI-based models? How would you write such a section?

Well, I think you would, you would have to justify your sample size that you would be using for your AI development approach, whichever that is. I mean, there could be a situation where it's quite difficult to write that down if you're using an AI method for imaging, for example. I'm not sure we've addressed that in the work that we've done so far, because that's a very different type of data where— and I'm not sure how you'd call it data. It's an image, you know, people are looking at that. So that's quite tricky. What I would say then is there's a huge emphasis then on what's the sample size for the evaluation part. So you can always do a sample size calculation for evaluation regardless of how the model was developed. It can be developed any way. When you get to the evaluation, it doesn't matter how it was developed. The key thing is what is the model outputting, what are the predictions, and how do we examine the performance of those predictions. What— how do we look at the accuracy, the clinical benefit of those predictions? So it doesn't matter how they're being produced, we just look at the predictions from that model. So that sample size calculation at some point in an NIHR application needs to say we have a dataset or we're going to obtain a dataset for evaluation purposes that's of a sufficient sample size to precisely estimate those predicted performance measures. So I think that's way should be the basics, even if you struggle in some way to look at sample size for development. But even then, you would still have the overall risk criteria. So if you can't estimate the average risk in the development data precisely, then, you know, how could you estimate an individual's risk precisely, for example?

There's a question which says, if What if we have tens of thousands of data? Are we still supposed to calculate sample size, or we can just use all the data we have?

Well, use all the data you've got. I'd love to be in that situation. I think that's often quite rare. Look, don't forget, you may have tens of thousands of data, but do you have tens of thousands of events? You often won't. You often may still have hundreds of events. So in that situation, think still about how many predictor parameters you're thinking of developing, using for the model development, because that's important. It's not just the total sample size. It's the number of events. It's the number of predictor parameters you're thinking of putting in. It's the anticipated performance of the model. And like I said in my talk, if you've got models with lower anticipated performance, you will need larger sample sizes. I think it's worth always doing the sample size calculation, even if the data is ridiculously big, just to say to people who are reviewing your funding application or reviewing your article, you know, we did a sample size calculation. It was— our data far exceeded the minimum required by whatever method you want to use. I think that's always helpful just to justify it.

There is, I think, one question which probably requires some clarification about the model validation sample size. So it talks about what considerations should we make regarding the validation dataset? How should we link— how should we think about the number of predictors we include versus the important predictors we may miss when developing prediction models?

Well, in the validate— I might have misunderstood the question, but in the validation data, it doesn't matter anymore about whether you've missed predictors or whatever's gone into the model development process. All you're then taking is you've got an existing model with a certain number of certain set of predictors in there. You apply that model and it will make predictions. So you're evaluating the benefit of that existing model in the context of what what predictors are in that model, what the equation is, and therefore you evaluate that model and validate it as it was. If you want to then think about, oh, I want to add in predictors, I want to tweak the model in some way, that's developing a new model. So that's called model updating, and then you'll be back to a model development sample size calculation to work out you know, in your validation data, if I'm gonna develop or update a model, what sample size to update that and update that model. I may have misunderstood that question. Sorry. But—

No, I think they just want some clarification. Yeah. And are there any— there's another question. Are there any recommendations on the sample size required to identify the optimal cutoff? For a continuous biomarker that predicts a survival outcome?

Good question. Um, what I would always recommend is that we don't use any cutoffs. Yeah. So when you've got a continuous biomarker, what I would suggest is that we analyze that on its continuous scale. We think about maybe nonlinear relationships between that biomarker and the outcome. So we think about you know, polynomial terms or splines or something like that. And of course, machine learning methods, tree-based methods would allow for nonlinearity in that setup. So, we don't think about cutoffs of the continuous biomarker. Clinically, we need cutoffs when we're making decisions for individuals based on their risk. So, we say, okay, let's give somebody this more aggressive treatment if their risk is high. So what we're talking about there is thresholds and cut points on the risk scale. So what I would recommend is that predictors that go into the model, they're not dichotomized or categorized, but at the end, the output of the model is an estimated risk for an individual. And then in conjunction with speaking to clinical experts, with patient representatives, understand what ranges of risk suggest that we would make a decision that's different to the— if the risk was lower than that risk. And that's then making decisions on the risk scale. So, make decisions and cut points on the risk scale, which is on the output of the prediction model, not while we develop the model on the scale of the predictors themselves.

Probably one last, uh,

I think I think we're nearly at time, Romana, if that's okay. Sorry.

Okay.

I just wanted to say thank you. Thank you, Romana, first of all, for doing the Q&A and welcoming everyone to the session. And then also Professor Riley, thank you so much for such an informative session and for all the questions that were provided in the chat and in the Q&A. I'm just going to quickly Share my screen. Just remember that the RSS is here to support anyone for any applications and career opportunities in health, public health, and social care research. Please just visit our website and complete a form, and you'll hear back from one of our fantastic advisors. There's a feedback form, it's literally 4 questions, if you have a chance to complete that. And then please remember that in the next couple of weeks, We have 2 sessions on stepped wedge trials, so one on the 15th of October and one on the 22nd. They're going to be jointly held with Professor Richard Hooper and Professor Carla Hemming. So once again, thank you all, Romana, Richard, Greta, and please email us if you have any follow-up questions. Thank you. Thank you.