Stability of Clinical Prediction Models Developed Using Statistical or Machine Learning Approaches

Hi everybody, this is Professor Richard Riley from the Institute of Applied Health Research at the University of Birmingham in the UK, and Gary Collins from the University of Oxford and myself have been working on the problem of stability, or should I say instability, of clinical prediction models developed using either statistical or machine learning methods. And in this talk, I'm going to present to you some of our work that's emerged from this research. In particular, I'm going to push back against people developing prediction models in small datasets, and I'm going to show you how unstable many prediction models are in terms of the predictions for individuals and suggest that many models could actually be causing harm from being developed with such instability. So some dramatic results will be shown, and I hope this is an informative lecture for you. So what do we mean by a clinical prediction model? A clinical prediction model means that we are interested in predicting the risk or the particular value of a future clinical outcome that's relevant to individuals such as patients in healthcare. And these models utilize multiple predictors, also known as features or prognostic factors, They are multiple variables such as age, body mass index, family histories, maybe some biomarkers, which are used in combination to give every individual their risk or predicted value of that future clinical outcome of interest. So well-known examples are the key risks for predicting cardiovascular disease, key COVID for predicting risk of mortality, admitted to hospital, the Apgar score in newborns, Nottingham Prognostic Index in breast cancer, Leicester Diabetes Score, and so forth. There are many models out there, and indeed many people in their career— research career— end up doing some sort of prediction modeling. Maybe they'll develop a model using a regression-based approach, but increasingly people are interested in machine learning methods such as random forests or super learners, with the idea that these are going to allow us to develop better clinical prediction models. Here's an example. This is a prognostic model for predicting mortality risk in patients with traumatic brain injury. Now this is just the interface. Behind the scenes there's a logistic regression equation, and that logistic regression equation will give the predicted risk here of 14-day mortality for an individual when you enter the values of their predictors for this model, which are the country the age, the Glasgow Coma Score, pupillary reactivity to light, extracranial injury. And plugging these in for this particular individual from Australia, aged 47, etc., we see that their risk of 14-day mortality based on this CRASH tool is 14.2%. So a simple example of a clinical prediction model. Now clinical prediction model research has 3 main phases. The first one is the model development. This is the architecture, the process to produce the model equation or the, the black box, if you like, of machine learning approach. The next phase is usually to do some sort of external validation. This is where we test the performance of the model in new data, often from different settings or countries. And the third phase is maybe to look at the performance of your prediction model in the context of changing clinical decisions and the impact that has on patient health outcomes. Now improvements are needed throughout the whole process, and in some of my other talks I, I mentioned what's needed. Today I'm just going to focus on the development phase. So this is the process to produce the model equation or model black box, like I said. Sadly, few developed models are fit for purpose. What I mean by that is that the predictions that they produce are unreliable, such that in new data they are not likely to perform well. Another related word for this is instability. So today I'm going to demonstrate what we mean by instability and how to quantify this and expose this problem, because I fear that at the moment many models are out there and published, but the actual issue of stability and the problem that most models is not— are not likely to be reproduced in new data is not exposed well enough. So when we think about developing a good model How do we view that? How do we have in our minds that a model is likely to be reliable? Well, for me, a model is more likely to be reliable when you've got a large sample size, when the model is developed using a dataset that's large. It also needs to be representative, of course, of the target population. We also need to think about how many predictors or features we're going to be putting in our model. The more we put in, the more potential instability there could be. Think of it another way: the more complexity that we add to our model development and the model equation, the more likely it is we're going to get things wrong. And so it's important to think about the number of predictors relative to the sample size. You may have heard something like events per variable. So it's not just the sample size; it's also the number of events. I think again models are more likely to be reliable if they're developed using sound statistical or machine learning methods. So approaches that recognize the complexity in the data and use approaches to deal with what we call overfitting, chance findings. So what are those? We— you may have heard of things like penalized regression like the LASSO or elastic nets. Machine learning methods like ensemble methods. We can also use internal validation processes to examine instability in our data and to maybe try and address it in some sense. Resampling methods like cross-validation and bootstrapping are fundamental parts of the model development process. So if we use those, we can examine problems in our development and issues of uncertainty and overfitting and try and estimate and adjust for those problems. A model is more likely to be unstable and unreliable when the development data is small, when your development data is a poor representation of the target population, when the number of predictors or the number of predictor parameters is large relative to the number of events and when the modeling approach doesn't adjust for optimism due to overfitting. Now my concern is that actually many models are being developed in this situation, but even more when we develop models like this, that even if the sample size is small or even maybe even moderately sized, there are still problems in many situations with the models that we develop. So let's have a look at trying to expose this problem. Instability. What do we mean by instability? So this is the idea that your developed model— so for example, your regression equation or your forest— may be different if it had been developed in a different sample of the same size from the same population. So when we think of developing a model, we've got an existing dataset that we'll use, or we may be let's say do a cohort study where we recruit new data. Now that data set is a sample from the population. Had a different sample been taken by chance, we may have ended up with a different model equation. So for, for example, in a regression, we may have ended up with a different intercept term, different betas. We may have selected different predictors, different features to go in our model. We may have different trees, different actual magnitudes of predictor effects. And the smaller the sample size, the more that the model that you developed using your sample that you happen to have could be very different than another model that could have been developed if by chance you had a different dataset of the same size. So this is instability, and instability in the model leads to instability in predictions. This means that predictions from your model that you create could well be different to predictions from another hypothetical model that you would have generated if you'd used a different sample of the same size from the same population. So the larger the instability concern, the greater the threat a model is unreliable. If there is instability, then the concern is that the model that we create is less likely to be valid even in the population that we developed our model in. I'm not even talking about external validation here, I'm just talking about internal validity for the development population. The issue of instability is rarely thought about after we've developed our model and rarely presented. So what I want to push people today is to Examine it and report it every time we develop our models. Let's think about this through a motivating example. I'm going to do a simulation study. In my simulation study, I'm going to be generating some data of size 100. In the population, I'm going to assume an overall risk of 50%. I'm going to generate individuals' logit risks by a single predictor X with a mean of 0 and a variance of 4. Like I say, I'm going to have a random sample of 100 individuals and I'm going to simulate 11 predictor values. One of those is X. That's the only predictor that's actually genuine. The other 10 are going to be noise variables. They're not predictive and they don't actually add any predictive information, but let's pretend I don't know that. And therefore I may be including them in my model. I'm going to call those Z1 to Z10. So based on this information, I'm going to simulate outcomes based on the true model, which is just X. And so with a 50% event risk, I'm going to get approximately 50 events. Then to my data of 100 participants, I'm going to fit a logistic regression model with a LASSO penalty. And LASSO is just an approach to account for potential overfitting through shrinkage of predictor effects. I'm then going to use my developed model to make predictions in the full population. Let's say the full population is of 100,000 individuals. I'm going to repeat this process as well from 1 to 5. Let's say I'm going to do it 1,000 times. So I'm going to develop essentially 1,000 different LASSO models based on 1,000 different random samples of that size 100. So this is going to lead to 1,000 models and 1,000 predictions, one for each of those 1,000 models for each of the 100,000 individuals in my population. So what I'm interested in is, okay, you know, this is reflecting the uncertainty in the model development process, but ultimately How does this propagate to uncertainty, or if you like instability, in the predicted values for each individual in my population? So let's have a look. First of all, we can think about instability at an average level. So what's the instability in the mean predicted risk? So going to get 1,000 mean risks. We've got 1,000 models applied to the whole population. We're going to get an average value that's predicted for each of the 1,000 models that I've developed. And so what I'm plotting here is the distribution of the mean risk across those 1,000 models. We see that on average it gets 0.5, as we might expect, but there's uncertainty. In some models, the mean risk is around 0.4. In others, it's around 0.6. So there's instability in the average risk, but it's quite narrow. You know, there's quite good precision in the average mean predicted risk across those different models. That's the first level. The next thing to think about is, okay, what's the instability in the distribution of predicted risks? So for each of those 1,000 models, I could plot the distribution of risks in the population. And here I've got 2 of the models and I'm plotting the distribution. So in this model here, the distribution of predicted values in the population goes from 0 up to about 1, and it seems to be sort of a normal distribution type shape. But in an— for another model, again just another model that I happen to develop using the same sample size of 100 individuals, the distribution is very different, isn't it? Very different. And this is starting to reveal instability at the individual level, where the distribution across 2 models that just happen to be developed with the same sample size is very, very different. And this is then starting to be a worry. So focusing more on the individual level, let's think about subgroups. What's the variability inpredicted values for subgroups. So let's think of a subgroup, let's say, just for individuals defined by an x value of less than -1. So across the 1,000 models, I am able to get 1,000 different predictions, one for each of those different models, for this subgroup. Let's have a look at the variability, the instability. So in this group on average I'm getting predictions around 0.2, but for some models for this subgroup I get predictions down towards about 0.05, and in other models I get predictions up to around 0.35, even maybe up to slightly up to 0.4. So that's starting to be concerning, isn't it? Just by chance I may develop a model that for this subgroup gave me a risk that's very low, about 0.05, But by chance, they may have developed another model where the predicted risk for this group is 0.35 or 0.4, much higher. This is worrying. Let's push it even further. Let's look at the 4th level, instability at the individual level. So here's an individual defined by a true risk of about 0.5, and again, I'm plotting the distribution of the predicted risks for this same individual across the 1,000 models. So for this individual, we may get predictions up to around 0.8 in some models and down to around 0.2 in others, even though on average we're getting risks of about 0.5, which is their true value. And remember, when you develop a model, you produce one model, and that's the model you take forward potentially to inform clinical decisions. So for an individual, if that one model happens to be 0.2, you may not get treated because you're considered low risk. If you had a different model, you may have got treated because you could have got predicted to have an 80% risk. This is madness, and this is instability madness that we need to recognize in this field. Let's look at just changing the sample size a bit. And on the bottom here, I've got individuals simulated individuals where I know that their true risks go from 0.1 to 0.9. So I've got 9 individuals, and here I've got a sample size of 50, here I've got a sample size of 100. And again, I'm just plotting like I did before the scatter around 1,000 models. So when the sample size is 50, this individual here whose true risk is 0.5, their predicted risks could be anywhere from 1 to 0. And indeed most individuals across the whole range of 0.1 to 0.9 have variability from about 1 to 0. This is the 100 sample size situation that we've come across just before. Again, lots of instability. Let's boost the sample size up considerably now to 1,000, and we start to see the instability, the variability in predictions across the 1,000 models, starting to narrow. And indeed, when we have 5,000 sample size, the instability is quite small. Still reasonable around 0.5, going from about, I don't know, from about 0.6 to about 0.4. But for this individual, you know, it's quite tight around their true value of 0.1. Clearly, when we develop models, they are much more likely to be reliable for individuals if if we're in this situation at the 5,000 level in this particular example. That's not a rule to always use, but clearly when the sample size is much larger than the 100 that we did to begin with, we're going to get more stability. And we need to be pushing ourselves towards this situation rather than where we currently are, where we're developing models in small sample sizes. So this is my key message: always examine quantify, and report instability after model development. We need researchers to expose this, um, otherwise we can't really tell without any further validation whether our models are likely to be reliable. Now, this is not a new idea. There are many previous papers on this topic. What I really like is Frank Harrell, Frank Harrell's language where he talks about how a developed model is just an example model. I like the idea that we need to recognize that our model that we develop is just an example. We could have easily generated a different model, and there's a nice blog post there that talks about this. Also, there's been a lot of work about how in small samples the instability is reflected by different sets of predictors, different functional forms. Lots of work by Patrick Royce and Nivelle Salbrei that you can have a look at. As is often the case, if you go back far enough, you realize that Doug Altman was ahead of the game on this. So in 1989, Altman and Anderson examined using bootstrapping the variability in predicted values for individuals and showed that they are often very wide. This was also recently looked at by Pate et al. in 2020, and really our work that we're presenting here builds on those ideas. So earlier on, I introduced a simulation study where we were able to compare predicted values with true values. In reality, of course, when you've got a particular data set and you develop a model, you don't know the truth. I mean, that's why you're developing the model— you're trying to get a prediction. So in that situation, we need to use a different approach to be able to examine instability, and what I'm going to propose is bootstrapping to following the outline by Altman and Anderson. Essentially what we do is develop, let's say, 1,000 bootstrap samples with replacements. That's going to give us 1,000 bootstrap datasets. These are the same size as the original datasets. And then in each of those bootstrap samples, we're going to first of all develop a new model, a bootstrap model if you like, and we're going to use exactly the same model development steps that we used before. So if you developed your model with the logistic regression, let's say forcing in all the predictors, then you use that approach. If you use logistic regression with backward selection, you use that approach. If it was a lasso, use that approach. If it was a random forest, use that approach. Replicate the same number of trees that you used before. Um, all of the steps that you use to produce your model, you must make sure are replicated in your model development process in the bootstrapping. If there's a split of the data, so say for example you develop on half the data and maybe recalibrate on the rest of the data, that needs to be replicated. So the random split needs to be in the bootstrap process. So this leads to 1,000 bootstrap models, each of which have been developed using the same approach as originally done. Then in the original sample, we're going to apply each of those bootstrap models back to those original patients and derive a prediction from each of those models. So just as in the simulation study, we're going to get 1,000 predicted values, estimated risks, for each individual. And this allows us to then examine the instability. For example, in the prediction instability plot, where we plot the bootstrap model predictions on the y-axis against the original model predictions on the x-axis. We can do calibration instability plots where we plot the calibration curve instability the curves for the original model predictions in the 1,000 bootstrap samples. I'll come back onto classification instability later. We can also quantify the MAPE, the mean absolute prediction error. So that's the mean absolute difference between the original and the bootstrap model predictions for each individual, and also overall we can calculate that. So this is an overall measure of the error in the absolute error in the, in the predictions compared to the original values. If you want more details on MAPE, then have a look at the work by Van Smeden and Pate. Let's look at a real example. We're going to develop a prediction model for the risk of death by 30 days after acute myocardial infarction. We can use the GUSTO-1 dataset, and we thank Duke Clinical Research Institute for making this dataset available. It's been used many times in different prediction model papers over the years. And the full dataset is 40,000 individuals, 2,800 deaths by 30 days. So the overall risk of death in this population is about 7%. In my model, I'm going to use 7 predictors, 7 candidate predictors listed here. And for reference, if it was just to fit logistic regression model and include those 7 predictors in my model, force them in, the C-statistic would be 0.8. So promising discrimination performance. So big data, strong predictors, potential here to develop a prediction model. Okay, so let's fit an unpenalized logistic regression model to that dataset, or very big dataset, 40,000 participants, 2,800 events. So that's about 400 events per predictor. So this is big data. And I'm going to plot the instability results. First of all, the average MAPE is 0.0027. So this is very small average difference between the bootstrap predictions and the original prediction across all individuals. And now I'm presenting what I call the prediction instability plot. This is the predicted or the estimated risk from the developed model for each individual against the 1,000 predicted risks from the 1,000 bootstrap models. And we can see that the variability of predictions from the bootstrap models is very narrow across the whole range. The dotted line is a 95% interval, so where 95% of the prediction model predictions are lying. So for example, if you've got a predicted risk of 0.2 in your original model, 95% of the bootstrap models are very, very closely around 0.2. If you had a prediction of around 0.5 from the original model, the variability is a little bit wider here, but again still very, very close to the 0.5. So this is a situation where it's telling me based on the information from the model development dataset, that when I apply this model in new data from the same population, it's highly likely that this model will be reliable, i.e., that the predictions from my model will be in good agreement with those observed predictions in new data. So very, very good example of a situation where prediction model development is important and useful. Let's now just randomly take 300 participants, uh, 21 events, so about 3 events per predictor. So this is now what I would consider to be a small data set, but lots of people would develop models in even smaller data sets than this. Look at the instability now. This is dramatic. So the MAPE now is on average 0.031. And when we look at the scatter, let's say for an individual with a predicted risk of about 0.4, a 95% range across the bootstrap models is, you know, getting close to about 0.8 down to about 0.1. So my model suggests that this individual has a predicted risk of 0.4, But other models could suggest up to 0.8 or down to 0.1. So this is the uncertainty— I don't mean a confidence interval, I mean the stability uncertainty range across the different example models that could have been produced. Now clearly it's slightly tighter around this end and seems to get bigger as we get further up the risk range. Would you be happy as an individual patient to use a prediction from one model knowing that other models could have given you a very, very different prediction that's up to very high and down to very low? I don't think you would, and therefore as researchers we shouldn't accept instability like this if we're going to be using this model to make individual-level decisions, for example, about when to start treatment, how often to monitor and things like that. Now let's have a look at the instability in the calibration curves. So I'm plotting my original predictions calibration in each of the bootstrap samples. Again, for the full big data, we see that the variability in calibration curves is quite narrow But in the small dataset, the calibration curves are very variable, you know, somewhere down here, somewhere up here. So this suggests that without further validation data in the small dataset, it's highly unlikely that the model is likely to be well calibrated in new data because the scatter is so broad, whereas this one we'd expect it to be reasonably well calibrated. Like I said, you know, many people would develop models with lots more predictors than 7, and even if the dataset had more predictors than number of events, this wouldn't stop people. So let's try and replicate a more noisy situation. So I'm going to add 20 noise variables. So I've got 7 predictors plus 20 noise variables, which are not predictors, but I've just added them in randomly generate— generated continuous variables that could now be selected by chance. Let's focus on a small data set, um, 750 participants, 53 deaths. So we've got about now lower than before, 2 events per predictor. Like I said, 27 predictors. And let's apply unpenalized logistic regression. Now the average MAPE is 0.038, and again we get this huge instability. So even bigger than we had in the previous slides, where let's say somebody with a predicted risk of 0.6, their instability is reflected by a range of about 0.9 down to about 0.2. Now lots of you may be thinking, ah well, we've got modern methods that resolve this. For example, we can use the LASSO. The LASSO will sort this problem, surely. Well, the lasso does improve things. So here I've applied a logistic regression with a lasso penalty. The average MAPE is now 0.029, so reduced down from 0.038, but the instability is still enormous. So again, going up this end, individuals with a risk of— predicts a risk of about 0.5. Other models may have been up to 0.8 or down to about 0.2. Even down at the lower end, where maybe that's where the clinical decision-making thresholds are of interest. You know, somebody with a predicted risk of about 0.2, other models may have been up to about 0.4. So the variability down here may actually be very important and consequential. So my message here is that LASSO, yeah, it definitely helps, but it doesn't solve the problem of small sample sizes. And if we don't present instability plots like this, we don't reveal this particular problem. And in new data, because of this instability, it's likely that models in new data— the model in new data will not work as well as we thought it was going to and is not fit for purpose. Here are the calibration plots. Again, we see that the lasso compared to the original model, which has got more variability in calibration curves, this is slightly narrower than before, again suggesting the lasso does better, but again, there's still a lot of variability, so I wouldn't be confident that calibration would be good in new data based on the instability I'm seeing here. Now, those into machine learning methods would perhaps say, well, you're looking at statistical methods and machine learning methods resolve this problem. Well, they don't. Even though machine learning methodology is often sort of reflecting this instability by the use of ensemble methods and super learners and including cross-validation, for example, embedded in the process to help look at the instability and try and address it in some sense, it doesn't mean that ultimately the problem with small sample sizes goes away. So for example, you know, random forests— this is probably one of the first machine learning methods that people learn. It's a very popular ensemble method, uses actually bootstrap aggregating to generate predictions over multiple models. So we are sort of reflecting this process that I've been talking about. It also includes a random element to feature predictor selection to limit the correlation across models and aggregates over those. So let's just have a look to see how this does. I'm going to use the RF— the R forest model in Stata just with the default options. And I'm going to use 500 trees. Oh well, now we see that it actually has made things worse. MAPE has gone up to 0.042, and we see a very worrying bizarre shape where individuals seem to have a much wider potential range across the different bootstrap models than for the LASSO or the unpenalized logistic regression. Let's have a look at these individuals with a prediction, let's say, of about 0.8. So the original model predicts 0.8, but across the bootstrap models we can have predictions close to 1 and yet also predictions close to 0. Even down at 0.2, where the original model random forest gave a prediction of 0.2, this individual You know, other models would have got 0, others would have potentially gone upwards to around 0.6 to 0.8. This is not acceptable. We're talking about individuals and patients, decision-making at the individual level. If you're producing a random forest and saying, well, this is better than some other method, well, whether that's true or not, it doesn't excuse the instability that's remaining even after doing the bootstrap aggregating and the whole process that's ultimately trying to resolve this problem of instability. It doesn't resolve the instability, and we're left with a situation where had a different example model been produced, individuals would have been allocated a very potentially a very different risk than they have done based on this particular model that we produced here. So prediction instability plots are hugely important as red flags for problematic prediction models that have been developed. Okay, now let's just look to see what happens after I apply a random forest as before, but then I use a recalibration. So sometimes random forests are very badly calibrated, and we can use a logistic regression model to recalibrate it, also known as Platt scaling. What happens when we do that is that we see there's A lot of movement in the predictions down to the lower end, but we still have instability. So, you know, for individuals now around 0.2, their instability in their predicted value is from again up to about 1 down to about 0. So recalibration doesn't resolve the problem. And that is my feeling, is that, you know, a lot of the rhetoric around machine learning methods is that, oh, if we tune them, we will solve every problem. If we do a good tuning, we, we can solve the problem. But it, it really doesn't solve the problem when you examine the instability. The problem is we need large data, and small sample sizes are not fit for purpose for prediction model development, no matter what method we use. Even though some methods do slight improvements over others, in general the instability will be still big and too big to be reliable. Another area where instability checks are important is in regard to examining algorithmic fairness. So algorithmic fairness— make it— the point with algorithmic fairness is to make sure that a model or an algorithm is performing equally well for all individuals and subgroups regardless of ethnicity, region, gender, and other characteristics. So if a model is shown to be more unstable for some subgroups than another, I think this is of a concern. For example, instability may be low for one group but high for another, and it may often be larger for those underrepresented groups in the development data. So you don't have a lot of people of one particular ethnic origin then that could well lead to the model being unstable for that group. Another way of thinking about it statistically is that if you have some predictors in the model that are estimated with a lot of uncertainty, then subgroups defined by that predictor will have a lot of instability. So it's important to examine instability for subgroups as part of algorithmic fairness checks. So going back to one of the LASSO models from earlier, a small sample size of about 4 events per predictor. Here I'm plotting the instability plots for males and for females. The MAPE is actually slightly bigger for females, but really if you look at the plots, they're very similar. So both have got some concerns about instability. And I can't see too much difference between them, which perhaps suggests that they're performing equally well, or even like equally badly, for both males and females in this example. But in other examples, that may not be the case, and really would encourage you to examine this in your datasets for different ethnicities, different regions defined by deprivation, for example, different genders, and so forth. Coming to the end now, the other area where it's important to think about instability is in regards to clinical decision-making, where we have thresholds that define some clinical actions. So for example, it may be that at a risk of 10% or above, we decide to monitor people more closely or give them a particular treatment. Therefore, whether individuals fall above or below that threshold is important, and If your model is unstable such that if you happen to develop a different model, that an individual's classification would have been below the threshold rather than above it, this raises alarm bells again. So I suggest calculating the classification instability index for every individual. This is the proportion of bootstrap samples where that prediction model gives you a different classification than the original prediction for that individual. And we can show this on a scatter plot. Where we plot each individual's classification index against the original predicted value. So again, look at a reasonably small data set and we apply a LASSO. Then here's the classification instability index on the left, and this is the predicted value on the right. And let's say a threshold of about 10% was the threshold used for clinical decision making. Now clearly, individuals with a predicted value around 0, their probability of reclassification by bootstrap samples is close to 0. But as you get close to the threshold, those individuals with values close to about 10% have a high probability across the bootstrap samples of changing from below to above or above to below. I mean, that's inevitable when you, you get close to the threshold. But it's good to see that as predictions get higher, the chances of moving classification comes down. So an individual here with a predicted risk of about 0.4, they're clearly above the threshold of 0.1, and across the bootstrap samples there's basically zero or just a tiny percentage of those bootstrap models where the classification for these individuals would have been below the threshold. If we rather look at the random forest again, where we had this concern, didn't we, about large instability, then we see that at those high risks where really that's miles away from the threshold of 10%, these high risks, we've still got a high chance, up to about 40%, that a different model would have classified these individuals as below the threshold of 10%. So again, red flag that this model is not stable at the individual level. So in summary, I hope this has raised awareness to you of the fact that many models are being developed in small sample sizes and are just not fit for purpose. Small sample sizes, too many predictors, inappropriate methodology can all lead to this issue that the model you developed is just one example model and could be very different from another example model. The issue is that when you do this, it's unlikely to validate well in new data with poor calibration. If you do not have a validation dataset, which is often the case, then we should be prepared to examine and report issues instability issues at the development stage. Instability in individual predictions, I think, is one of the most important things we can present alongside other measures like calibration, discrimination, net benefit. Plot the variability with a 95% individual interval of predicted values across the bootstrap models compared to the original model. Quantify the MAPE Plot the variability in calibration curves, classification decision curves, which I didn't show here, which may present net benefit and clinical utility. Such plots and measures really do help expose reliability concerns and I think would really help systematic reviewers. So if you're systematically reviewing articles that propose prediction models and you're having to say which are the better quality and most likely to be reliable based on the model development paper itself without any further validation, then these instability plots would be hugely revealing. And those that show— are shown to be more stable are much more likely to be reliable, i.e., have low risk of bias, if you like, than other models which have huge instability. Ultimately, without a decent sample size, you're going to be in trouble. In earlier work We have suggested minimum sample size criteria to precisely estimate the overall risk. Remember when I talked about level 1 of instability, the mean risk? Make sure that's precisely estimated. Target a small MAPE on average across all individuals. Minimize overfitting. And ideally, working towards the individual level to get stability at individual level predictions, we need precisely estimated predictor effects. These may be very hard to come by, but at the very least we should be ensuring that the overall risk is precisely estimated and that the overfitting, which is a measure of the issues of model complexity, are kept to a minimum. Thank you for listening to this talk, and I'm just going to leave you with some references. The work I presented here with Gary Collins is forthcoming. And I will leave in the notes below the link to the paper when it comes out. We'll put a preprint out first, and then the references to the papers that I've mentioned by Altman and Pate and the sample size work that we've done as well. Do check out our website prognosisresearch.com where we give some introductory material and videos about prognosis and prediction research. And also you can see details of our training courses such as Statistical Methods for Risk Prediction and Prognostic Models, where we have a really excellent faculty to talk through some of these issues through the development, validation, and impact stages of prediction modeling. But for now, thank you for listening, and I hope you can check out my other videos as well.