Scientist Stories: Gael Varoquaux, Medical AI

Welcome to the joint Alice Heidelberg AI and NCT Data Science seminar. It's my very great pleasure to have Gal Varouko here. He is a research director at INRIA where he leads the Soda team on computational and statistical methods to understand health and society with data. He also, you may know him from there, co-founded scikit-learn, one of the most widely used machine learning toolboxes. I checked actually It has more than 75,000 citations by now. And I just learned recently that he actually has a PhD in quantum physics. And today I had a very interesting, uh, conversation because he explained me his statistical perspective on quantum computing. So if you want to discuss that, uh, yeah, feel free to talk to him. It was a very interesting view. From my perspective, he's a role model also for bringing medical AI to the machine learning community. So for example, maybe you have seen his ICLR '23 paper. I'm a very, very big fan of his paper. I think I already forwarded 3 or 4 this year to my group, and I'm very much looking forward to your talk. Stage is yours.

Thank you, Lena. You're a role model for me. So, okay, so today I'd like to— so this is working, right? The mic is working. Okay, cool. So today I'd like to talk about a specific aspect of AI, which is the validation. And I'll go a bit in detail about what and why. The point being is that I'm absolutely convinced that AI, which is of course an umbrella term that means many things, can bring many great things to medicine and health. But I can tell you that not everybody is convinced it's working currently. And there are good arguments to say that most AI developments actually fail to deliver currently. And my strong belief is that we need to test what we do better. And I'll go a bit more in detail about it. And so let's start, you know, with the The dream, and I've been spending something like 15 years of my life working on AI or machine learning and brain imaging with the goal of being useful to mental health. And one of the starting points is that brain images, in particular functional brain images, are very rich and complex images. And so the hope is that machine learning, which is just very complicated statistical modeling, can be useful for this. And so one of the challenges that people brought forward is that mental disorders are extremely heterogeneous. And so if you look at population— well, populations of individuals, then you're going to have variations in cultural aspects of their behavior and maybe cultural aspects of the diagnostic criteria that are used. And if we look in particular at autism, for instance, there's a real question. You know, we see differences in prevalence of autism, but this is a diagnostic problem. Are we diagnosing people more? Is this a natural epidemics that's underlying. And so, question we asked was, can brain imaging give us universal diagnostic criteria? And so, we tackled this a few years ago, and one of the things that we showed is that if we kept increasing data, we would be able to predict as well inside the same acquisition site. So, on the same imaging device, but maybe more importantly, with the same population of doctors and patients and across other acquisition sites, sometimes overseas. And so for this, you need a lot of data. Well, here we had 1,000 brain images. Now, by machine learning standards these days, 1,000 is not a lot. But by brain imaging standards, especially 5 years ago, this was considered as massive. And so our claim is, yes, this is going to work. Heterogeneity is not going to be a roadblock, but we need more data. So we believe in this. Maybe, by the way, we fooled ourselves because we did the experiments ourselves and the incentive is to publish and say it works. So more recently, we set up a challenge where we gave people data. There was a hidden test set, and we incentivized people to predict as well as possible. And so in such a setting, it's blinded and the incentives are aligned. So, you know, people are not incentivized to overfit the train set. So I think we can trust the results better. And things work quite well. We're looking at area under the curves that are currently a bit above 0.75. And if we extrapolate, we hope that they can go all the way to 0.8, which is from a diagnostic perspective, interesting. Maybe we could consider the prevalence of the disease to know how interesting it is, but I'll focus about this later. So take a message, very large data sets, machine learning. We're going to solve the problem. We're going to improve the diagnostic. We're done. We can go home. Well, that's not what a study of the literature shows. So with Veronika Shtepnikina a few years ago, we kind of introspected ourselves and the literature on the progress that machine learning has brought to medical imaging. And so one of the things we looked at, for instance, was Alzheimer's disease because it's been one of the reasons we took Alzheimer's disease is it's been studied massively. People have written many, many papers on this. And so what we show is that from the literature, from the published paper, larger datasets in publication don't really seem to improve prediction accuracy. And across time, more years of research don't really seem to improve prediction accuracy, at least in the published papers. So, you know, it's not all that simple. And one thing I learned, because these days I work more with biostatisticians, people who are basically medical statisticians and who worry about validating things that are going to be literally used in the clinic, is that in this world, machine learning is a bad word. And one of the reasons is that we machine learners have overpromised so many times. Publications that claim that things were useful ended up not being useful so many times. And so there's a problem here. We need to tackle it. We may be convinced that our stuff is useful, but we need to go out and show it. And one of the problems is that too often our evaluations are unreliable. And so to take a specific example, a few years ago, I was looking once again in the literature at the published prediction accuracy for mental disorders as a function of the study size. And one thing that you can see is that in general, larger studies lead to less big effect sizes, so smaller published accuracy. And why is that? The two explanations. One is that those larger studies are more heterogeneous, have more heterogeneous recruitment, which actually should be something we should target because, you know, real life is not a perfect cohort. It's a cohort with a lot of diversity. But there's another explanation is that smaller sample sizes have more variations just due to sampling noise. And there is a publication threshold, which is if you get a good publication accuracy, prediction accuracy just by chance, you're more likely to be accepted. And, you know, basically small studies have more noise, and hence it's easier to get by chance a good prediction accuracy and to publish. Now, which is true, I can't tell, but I worry. Okay, let's look at our prediction challenge. So once again, the incentives were aligned. We gave people money to predict as well as possible on the private set. But we also looked at how well they were predicting on the public set, on the dataset that they had. And you can see that a fraction of people were predicting fantastically well on the public set and not very well on the private set. So, this is one-on-one overfit. And I don't have this data here, but if we look at through time, because everything was done on the server, so we have the logs of everything that people did. And so if we look at this phenomenon through time, we can see that people are, the more they work, the more they overfit the dataset that they have, even though they're not incentivized to do this. And at the end of the day, by the way, in this prediction, the best performers were really simple models. And fancy techniques did not work well. And that doesn't really align with what's published. Is that true only in academia? Well, no. If you look at Kaggle competitions, here I'm showing the blue curve is the spread of the difference between two test datasets. The public leaderboard and the private leaderboard. And it amounts basically to sampling noise in the validation, right? I'm trying the model twice on two different datasets. And the difference between those two is noise. And the red line is what I call the winner's gap. It's the difference between the winning algorithm and the one that was, 10 percentile less good. So, it's, you know, you take basically the 10% best algorithms and they're between this line and zero. And so, what you can see is that in some, in a good fraction actually, of the Kaggle challenges, the measurement noise is actually much bigger than the difference between the winner and many, many others. So basically I claim that there's no statistical, no meaningful— I don't even want to use the term statistical— there's no meaningful difference between, you know, the 10% best algorithms. And sometimes you even get an overfit, which is basically algorithms don't rank at all the same way on the public leaderboard and the private leaderboard, even though once again, the incentives are aligned and we're not in academic settings. Is this true only of medical machine learning? No. Last year we were looking at different problems. We were looking at deep learning and tabular data, and we were surveying many promising publications on deep learning and tabular data. And our experiments in the lab didn't really match what was published. And so we published at NeurIPS, which is kind of one of the deep learning conferences. We published a paper that claims that actually deep learning doesn't outperform tree-based models, and they tend to perform better. Okay, so I really worry, I'm worried literally that we tend to have unreliable evaluations in the work we do. And that's a bottleneck, an absolute bottleneck. One of the reasons is that we work with finite test sets, and finite test sets come with sampling noise. If I sample 100 individuals and if I sample another 100 individuals, I'll get different results. And the corresponding confidence intervals when we're in classification settings are huge. The reason is simple. In classification setting, one individual brings only one bit of information, you know, 0, 1, so very little information. So I need many individuals to have a proper estimation of the performance of the algorithm. Many. On top of this, I mean, you look at those confidence intervals, you know, when you have 10,000, you think, well, okay, plus and minus 1%, that's not that huge. But the thing is, we have human overfits, and we have incentives. And so these are the 95% confidence intervals, but I'm convinced that we mind the tails of that distribution because we keep trying things to get them to work. So I'm really worried. We need to address this if we want progress in real-life settings. And then there's another problem, which is that the data we work with is typically an imperfect window on the application. And one of the reasons is that we grab all the data we can because that's what we can do, and we really need a lot of data. So to give you an illustration, there was a few years ago a large dataset that was put on Kaggle on diagnosing lung cancer. And so you can see that after this dataset was put on Kaggle, the number of publications in AI or computer science looking at lung cancer just jumped. But if you look at medical publications of lung cancer diagnostic, no change. And the reason is, it's not obvious to me, and I don't know, I'm not an expert, how interesting from a clinical standpoint it is to focus on this specific cancer compared to another one. So obviously, and it's kind of natural, you know, we computer scientists work with the data we have, whether it's clinically relevant or not. And then given a given data set, we know we typically take it and then we munch it, we work on it. But it only captures a very small fraction of the application. It may come with information in the dataset that's a consequence of the diagnostic and not a cause. So people have shown that diagnosing pneumothorax algorithms would detect a chest strain that is put in, in the lungs. But this is put after the diagnostic. It's a consequence of the diagnostic, not a cause. So it's actually— so the algorithm works, the, the, the prediction is correct, but it's completely useless. And the reason is that we're, we're getting images that are down the line. They don't correspond to the images that would be useful to the clinician. Same thing with a, a dermatologist looking at malign tumors on the skin, people have shown that in the training data and in the validation data, the malign tumors sometimes were circled because they were identified as malign and the algorithm just learned to detect the circle. So that's the first problem. It's basically the data we have doesn't reflect the application.. But then the second problem is that we have sampling biases. Our populations on which we train the algorithms are not representative of the target populations, which is, by the way, epidemiology 101. Public health worries a lot about this because public health knows it's a problem. And so here we have a cultural problem in computer science is that we focus on pretty results, which may be valid statistically, but they're valid statistically in the dataset that we have. And that's a focus on what people call internal validity versus external validity. Now the question is, does it carry over to a target population and target medical practice? And this is one of the reasons why we often fail to be useful. And finally, or not finally, but more imaging. I've focused on imaging for many years, but it's only a fraction of the information that comes with a patient. No clinical decision will be made, or very little, sorry, clinical decisions will be made looking only at an image. The clinician always knows things about the individual the healthcare system. And all this comes together. And what we've seen recently focusing on mental health is that simple information such as simple sociodemographics and questionnaires were better predictors of mental health than brain imaging. Now, The reason people acquire brain imaging is to understand mechanisms, not to do diagnostic. But so many people have published diagnostic, me included, on brain imaging. So then, you know, the question is, are we, are we fighting the right battle? So I'm convinced that we're often limited more by the data that we have than by our machine learning algorithm. And as somebody who does machine learning algorithms, it's annoying, but it's a fact of life. We can't, you know, I don't think we can not worry about it. So if we zoom out, we're limited by the data, by its size, which gives us sampling noise in the test set. And I worry more about the size in the test set than the size in the train set. Which gives us, I believe, rampant overfit at the level of the literature, including in some of my published papers. And we're limited by the representativeness of the data, which gives us useless solutions. So I believe we need to shift our focus if we want to be useful, and we need to focus on integrating more and more diverse data and we need to focus on better validation. And then there is yet another problem, and I think that's the last one, is that the applications are not only about classification accuracy. Maximizing the performance doesn't always carry over to something useful for the application because it depends on how this prediction will be used in the clinic. And, you know, the practice to establish clinical utility in evidence-based medicine is a randomized controlled trial, which if you work with professionals of randomized controlled trial, the first thing that you do, well, the first thing that you do is you define a population, define an intervention, You define an outcome. So an outcome is very seldom a diagnosis. It's typically long-term health improvement. So for instance, I want a diagnosis answer, but the question is, if I intervene, will I improve long-term the population's health? And that's going to be different for different populations. Population may also mean early on or late in terms of cancer diagnosis. So randomized controlled trials define all this, and then they do an intervention. And the intervention, so they modify the way care is applied. And this is interesting because intervention actually modifies everything. When you do an intervention, you modify complex equilibriums at the level of a healthcare system. It's also very expensive, like ridiculously expensive. It's also dangerous from the ethical standpoint. You may decrease the standard of care for the people in the RCT. So this is why we don't do this, because it's expensive, because it may be dangerous from the ethical standpoint. So what we do is actually good in a sense. You know, we're not intervening, which is good. But what we do, we need to keep in mind that it's a proxy for the intervention. And it's a cheap proxy, which is good, but we need to improve it. And if we improve this proxy, I think we will have more impact. Okay, so then I'd like to talk about what are things that I see we can do to improve our validation. The first thing that we need to do is that we need to raise the bar. We need to do things such as addressing distribution shifts, and those are things we can do as machine learners. We as machine learners, most of us are not brave enough to acquire data, and we need to do better model validity. So one of the problems that keep coming up is the lack of error bars in what we do. We just don't know how precise we are. One call? No. Yes. So, and one thing that I think keeps having an impact on the error bars is the choice of hyperparameters. We In everything we do, most of the things we do, we need to choose parameters that we can't control. I can't control the number of the width of a neural network. I can't tell you, of course, it's 42. And so those are arbitrary. Those are completely uncontrolled. And they're present in all our publications. So we have those arbitrary magic numbers that are present in all our publications. That are typically done by human beings that have their own biases. And I think it's a major problem. I think this is one of the things that are explaining the fact that results may not carry over from one publication to another. So I'm convinced that we need to do automatic hyperparameter tuning. And everybody hates it because it's very computationally expensive. It's very tedious. But without this, we have absolutely no control over it. And one thing I like to do here is to do random search. Random search is one of the most stupid algorithms to do this, but the thing is it allows us also to estimate the variance due to hyperparameter selection. And that's real variance, right? Because given a new dataset, I will need other choices of hyperparameters. So I think we should be doing more random search. Data sampling noise is a major problem. And there we're limited by the size of the test set. And so I think we should be doing repeated train-test split. We shouldn't do like a single validation set. And here, of course, I'm going at odds with what has happened in the machine learning community because 20 years ago we were doing cross-validation. These days we're doing single splits. There are many reasons behind this, but I don't think this is a good thing. We can also derive confidence interval on test accuracy. I'll come back to this. And here we need to be careful that we have two different use cases. One is we're doing machine learning research, computer science research, and we want to compare two algorithms. And the other one is we're interested in clinical validation. In which case we want to evaluate something that we're going to take and put out of the lab. And those are two different use cases, and I'll come back to this. So for better validation, we need better metrics. And a lot of work on the metrics is being done here. And the challenge is that the metrics must capture utility. And utility is a function of prediction error, but it's not just prediction error. It depends on many different things. And now this is the difficult problem that we don't like as machine learners because it includes many of the things we don't understand well. But if we don't tackle the problem between the difference between a prediction error and a utility, we really risk running into diminishing errors and optimizing the wrong thing. We need to go beyond a fixed threshold. All the clinicians I talk to want some continuous measure of confidence in the prediction because they want to be able to modulate their decisions. And something I've learned through time is that there's no simple story. A single metric will not suffice. So I'll come back to this, but controlling confidence is harder than people think. People think about confid— calibration, but calibration only measures an average error given the prediction accuracy. And we would like from clinical standpoint to control individual probabilities. And this is hard because the individual probabilities are never observed unless we do test-retest. Okay, so let me dive a bit deeper in those points that are presented and go a bit more in the mechanics or the math, you name it, of these things. And I want to start with the error bars. And those error bars, we found out, are actually very subtle. There is a lot going on in, in the validation So, you know, how do we do model evaluation in general? Well, we take the data we have, we split it between train set and test set, and we move on. This is what we do. So, before we go further, I think we need to be very precise about which questions we're asking. What are we benchmarking? Are we benchmarking a prediction rule? So, we're giving a function that goes from X images to Y diagnostic, for instance. This function, by the way, could be anything. It could be designed by experts. Most clinical prognostic models these days used in practice are designed by panels of experts, not machine learning algorithms, but whatever. We're given a function. It came out of the blue. We don't know. And our question is, how reliable is this function?. And then there is another scenario, which is a training procedure. And now we're in machine learning. We're given a procedure, a training procedure, that uses training data, so pairs of x and y, to output a prediction rule. And I want to stress that those are two different settings. And they correspond to two different application scenarios. The first one is really what you want to know when you're going to put in production, quote unquote, your method. So for medical practice, in medicine, it's sometimes called external validity. The second one is what we want to do when we do machine learning research and we want to do claims on algorithms. So when we publish and we say, woohoo, my method is better than your method, we're in second scenario. First scenario, external validity. We need to ask a few questions here, such as, do I have X test that is different enough from X train? It may seem trivial, but we don't want to have repeated acquisition of the same individual in the training and the test. But in biostatistics, people are really pushing for a broader notion of external validity. They're really pushing to generalize not with like an arbitrary cutoff in the same data from training test, but to show that we're robust to some shifts that are likely to occur, such as across hospitals or across time. And, you know, a few years ago, I was like, no, this is a different problem. This is distributional shift. And standard assumptions of machine learning is IID sampling. We can take the stance, it's mathematically correct, but then people won't trust us to put our methods in production because they are afraid such shifts will occur. So if you want to convince people that what we do is useful, we need to show that it's robust through a variety of shifts. And then the question is, we also need to have our test data to be representative of our target population. And that's crucial. In medicine, there is something that's known as Table 1. It describes the statistics of your— the sociodemographics, for instance, of your study population. And the first thing people do is look at Table 1, and if Table 1 doesn't match the population they work on, they basically don't read this thing. We need to do the same thing. And then here, the important question is not the training data. I don't care what, you know, data augmentation you do on the training data. What I care is the statistics of the test data. And so here we can do things such as reweighting our test data to match statistical moments or age, things like this, or we can worry on the test data about confounding associations that we don't want to see, things that are sometimes called shortcuts. And this is something we need to do to validate our algorithms, to convince people they do work. In this scenario, The error bars are limited by the sampling noise on the test set. So it's, you know, you have a given test set. And one thing I should stress is that in this scenario, we have a fixed test set. We don't have multiple test sets. The reason is we are trying to validate a given learning, a given predictor. So if we have multiple train sets, we have multiple predictors. So we don't know which one we're validating. And suppose, you know, we retrain and we fall in terrible local minima, then we're going to put in production things that are going to kill people. So we're with a fixed test set. And when we're with a fixed test set, really the evaluation noise is the sampling noise on the test set. And this we can actually easily tackle in statistics, either with parametric models such as a binomial distribution, or just by bootstrapping our test sets. It's not very hard to do, to bootstrap the test set. And this gives us non-parametric confidence intervals on our evaluation error. And it's not even that costly because we're not retraining. So it's really something that we should do. It's often listed as part of the good practices of clinical prediction models. So scenario 2, we're benchmarking to conclude on training procedures. So here we want to evaluate a procedure that outputs a prediction rule given training data. This is what we do in most of our are machine learning publications. Even though sometimes we try not to discuss this and we try to discuss fixed test sets. But when you want to say, you know, I have this new method that works really well, I'm sorry, you're in this setting. In such settings, we have many arbitrary components, such as the random seed. They're meaningless. People have shown that you can vary hugely the prediction accuracy on a fixed test set by tweaking the random seed. And I've had discussions more than once by people telling, you know, it's fine. I take the prediction accuracy given by my best seed because it's the best prediction accuracy. Now, this is first extremely an unfair comparison to other algorithms. You know, if I really wanted to compare to other algorithms, then my procedure would be try the algorithm 10 times, pick the best seed on the validation set. Now, that's a legit algorithm, but this is my algorithm. And then second, if like given a specific seed like 3407, this will not carry over to a new training set. So while I was on sabbatical at Mila, I worked with people there to do a systematic evaluation of the different sources of variance in a machine learning evaluation pipeline. And so we compared many different things. And the important thing, by the way, is that this was done across many different tasks, including with state-of-the-art neural networks. And so this was incredibly computationally expensive, by the way. So we compared the variance while rerunning an evaluation, modifying various arbitrary elements. Such as data augmentation, data order, weight init, dropout, random noise that we don't know where it came from. We also looked at varying the splits between train and test. And maybe the take-home message here is that the split between the train and test was the dominant factor, even in large computer vision experiments. So we typically, when we do neural networks, we typically look at only the weight in it. We try a few weight in it, and this is what we report. We're neglecting the bigger part of the evaluation model. So, you know, better evaluation would be to sample all these things multiple times, which is basically getting us back to cross-validation. Multiple train-test splits. Now there's the bigger problem, which is the hyperparameter selection. And so in the paper, we looked at hyperparameter selection, but there it gets brutally expensive because you need to rerun many times hyperparameter selection algorithms. And what we showed, by the way, was that fancy hyperparameter selection methods did not decrease the imperfections due to choices of hyperparameters. Choices of hyperparameters are a statistical choice. They are by construction noisy. And so what we recommend is to do random search because at least with random search, you can, because in random search, every data point is independent, you can take subsets of data to estimate the uncertainty in your prediction accuracy that is due to hyperparameter selection. I know people won't like this, but I think this is important. And by the way, I really think this is what explains the difference between our experiments on tabular deep learning and the publications, is that people were basically optimizing their hyperparameters. And as a consequence, there was variance, maybe overfit, that was not reported. In the paper. Okay, so what's a simple procedure? In the paper, we describe a simple procedure to actually have error bars, and it's about sampling everything, the data split, all the random choices, you know, the seeds, everything, multiple times. For hyperparameters, if we do random search, it's also sampling. And then, you know, we can take the 5 best or something like this and report the standard deviations. And here, the important thing is that we're reporting the standard deviation, not the standard error. So we're not dividing by the number of splits we've done. Now, we could get in many details on what this statistically means. Basically not a p-value. It's not, it's closer to a, to a Neyman-Pearson approach to statistical testing. So basically the problem with p-values is that they go to zero. Well, they become significant if you have enough data points. So if I were to do random samples of data sets, data splits, I could make any arbitrary difference between two algorithms significant. So basically, the significance of the difference between two algorithms is above all a measure of how rich was the lab that did the experiment because the lab could sample things many, many times. So we're more interested in probability of improvement. In all those samples, how many times was algorithm A bigger than algorithm B? And ideally, this is more than half of the time. So it's like 75% of the time. So in the paper, we give a recommendation on the threshold. And I think the threshold is around 75%. So basically, we're not accepting tiny improvements. So it's really easy. Reporting the standard deviation is something everybody can do. We're not asking people to understand sophisticated statistics. Okay. I want to talk about a few other things, a few of my pet questions. One of them is meaningful metrics in imbalanced settings. One of the reasons this is important is that most pathologies have low prevalence, which is a good thing. So we have very imbalanced settings. So how can we report something that is meaningful? We can look at accuracy, but accuracy is not meaningful because if I have, you know, 90% of class 0, predicting only 90% gives me an accuracy— predicting, sorry, only class 0 gives me an accuracy of 90%. So then the standard practice in machine learning is to look at, you know, differentiate the errors in class 1, which we're going to call sensitivity, and the errors in class are zero. And then we might use things such as balanced accuracy or F1 score, which are basically a function of sensitivity and specificity. Now, sensitivity and specificity are the probability of having a predicted positive given a true positive. That's the definition of sensitivity. Well, that's what sensitivity measures. Sorry, not the definition. And specificity is the opposite for the negative class. And I claim that we're asking the wrong question. We're not interested in the probability of the prediction given the truth. We're interested in the probability of the truth given the prediction. And if we do this, we can easily do a bit of a Bayes' theorem.. And what we see when doing Bayes' theorem is that we see that the prevalence pops up in the equation. So by the way, this probability of the truth given the prediction is the positive predictive value and negative predictive value, which is used a lot in medicine and not at all in machine learning. And I used to not understand why they were using the PPV and PV. Why is there, you know, this difference? Well, the problem with the positive predictive value and the negative predictive value is that they depend on the prevalence. So you need your test data to match your target application for this to be meaningful. So they characterize not only the classifier, but also the dataset. And I think this is one of the reasons why we don't use them in machine learning. But I think this is a problem. And I fell in this trap. Trap, by the way. So odds ratio are things that we don't use a lot in machine learning, but they're the odds of an event is the probability of this event divided by the probability of not this event. So P of A divided by 1 minus P of A. And odds have interesting properties of not being sensitive to class prevalence. And it's simple math there in the appendices of the paper that I'm mentioning here. And it's really simple math. And so then we can compute something that's known in medical statistics as the likelihood ratio of the positive class. And it's the odd, so it's the post-test odds. So the odds of true positive given predicted positive divided by the pretest odds. And this is independent of the prevalence. So this is easy to interpret. It's the change in odds that we get when we get a positive detection. And it's independent of the prevalence. So I think actually we should be reporting this more. And this has been known in biostats for more than 40 years. People have developed this for blood tests. So this is really useful because it allows us to extrapolate across test sets of different prevalences. So we can communicate on this result and then reason about different prevalences. So to me, the problem with imbalanced classes is really a problem of looking at the probability of the truth given the prediction., and not the probability of the prediction given the truth. And this is how we should, you know, think about our results, and we seldom do that. So now I'd like to talk a bit about controlling the probability of an outcome. So in medicine, people are really interested in getting a number where, you know, where it's— when it's higher, it's more likely that the event is more likely to occur, whether it's diagnostic or prognostic. And the reason they're really interested in this is that the clinician is going to make the decision to balance many things, including like local harm-benefit trade-offs, which may depend on the patient's family. And when we think about this in machine learning, we immediately think about calibration. The problem with calibration is it controls an average. So the definition of calibration in people— so there, the literature has been a bit blurry. People have been sometimes using different definitions, but the prevalent definition when it's written, when people write the definition, is that a classifier is calibrated when the average error rate for all the samples with a score s is s. So if you take all the individuals that are assigned a score of 0.8— of 0.6, sorry— 60% of these individuals have class 1. This is the definition of calibration. Now the problem is that you can hit a calibrated classifier which has very inhomogeneous error on subgroups. So you can have a subgroup where it's 100% accurate and another subgroup where it's only 20% accurate, and on average on those two subgroups, it's 60% accurate. And so this is not what the application would like to control. So it does not control individual probability. Oops. So now there is a mathematical tool that controls individual probabilities, and it's known— those are what's called proper scoring rules, and one of them is the Brier score. And so it's a metric between the predicted confidence and the observed binary label. And this is kind of magic, by the way, because this metric doesn't need to have access to the true probabilities. It only needs to have access to the observed binary label, which is the difficult thing. We're never observing the true probabilities. And so proper scoring rule theory tells us that these metrics are minimal when the predicted confidence are equal to the probability of Y given. People don't really use them. And I think one of the reasons is that we don't know how to interpret them. I give you, I have a Breyer score of 0.2. We don't know if it's good enough. It cannot be interpreted as an error rate. It doesn't come with a simple scale to understand. So one way to look at this is to decompose these scoring rules as a set of different errors And so for this, we need to distinguish the classifier output, which I'm going to call S, the actual label probability, the things that we're trying to get, which I'm going to call Q, and the calibrated score, C, which in a sense is what you get if you recalibrate the classifier. And if we do this, we can decompose the proper scoring rule in 3 terms. I'll start by the last one, the irreducible error, which is due to intrinsic randomness in your application. This cannot go to 0. The best possible prediction will still have an irreducible error. And then you have a combination of 2 terms that I'll call the epistemic error. And it's the distance to the best achievable prediction. And it's made of 2 terms, the calibration error, which is the one we typically look at, and the grouping error, which is the remainder. So the grouping error can really be seen as the variance on the calibration plot. So here I'm plotting the true probability as a function of the prediction confidence, and my classifier is calibrated because on average the true probability— well, on average along a vertical line, the true probability is equal to the predicted confidence, but it's an imperfect Classifier, because it's got variance. Now, this plot I cannot plot in real data because it requires access to true probabilities, which I don't have. Okay. And so we did a paper at ICLR this year where we showed that we can estimate the grouping loss. Technically, it's a lower bound, but in practice, it's a lower bound that is close to the real value. And Just to simplify the paper a lot, because there's a lot in this paper, it relies on choosing bins well. And so for this, we use a classifier to divide the space in bins where the errors is reasonably homogeneous. And so the interesting thing here is that we can say how far a classifier is to the irreducible error. So then we can say whether we're, you know, really close to the irreducible error and it's time to stop and the data we have is not going to get us any further and we need maybe to improve our sensors, or whether we need to work more on this. So maybe the most important point that I want to make here is that a calibrated classifier does not predict the probability of the label given the data. This is a very common mistake I read all the time. And people reason on the wrong things. And the reason we need this, by the way, is because our validation is always a proxy of the area of interest. Okay, I want to talk quickly. I'm running out of time, but one last thing that has started to worry me recently is that most often Not always, but we want our prediction to support decision. And when we do this, we actually have causal concepts that creep in. And the right way to address those questions is by using causal inference. Here I'm going to look at prognostic models. So prognostic model is a model that predicts the future of an individual given their current health. And so I'm looking at my data and I can predict the outcome, what I'm interested in, given the health covariates. And so I can build a predictor on this. But in the data, some people have been given a treatment and some people have not been given a treatment. So then I can predict the outcome as a function of this intervention, the treatment. And the treatment can be many things.. So I can predict the outcome. And when I do this, if I get the perfect prediction given the treatment or given not the treatment, then I can reason on the difference between treated and non-treated. And this is going to tell me whether an intervention is beneficial or not, or how beneficial this is. And this is the basis of part of causal inference. We can predictors, we predict,, and we compare scenario A, scenario B. And so we use our predictor to make our decisions. Now the challenge, the challenge is that in most data we work on, the treatment has not been allocated at random. So here we see that people with good health have not received the treatment and people with bad health have received the treatment. Okay, so basically the two groups are not comparable. That's one-to-one treatment allocation in medicine. And if we use a predictor that just fits the average error, which is what predictors do, it may not capture well the difference because it may just shrink parts of the space shrink, for instance, the untreated to the treated. And so this is a good predictor, it minimizes prediction error, but it is not a good predictor to reason on treatment allocation. That's a problem. If we want to reason on treatment allocation, we may need to favor predictors that extrapolate better to regions of the space that are quite sparse. And their standard cross-validation prediction accuracy is not a good model selection practice because you basically need to reweight the error on treated and untreated. And I'm not going to go into details, I'm out of time, but basically we ran an empirical study recently, it's pre-printed right now, that kind of agrees with theory that we discovered by basically running this study. And there are some risks that are more complicated, but they compensate for treatment allocation and that allow you to select the best predictor if you're interested in a given intervention. I think we should be thinking more about this. Okay, I'm out of time. So maybe the one thing you can remember here is that when you want to support a decision, it's a causal question. And before I conclude, I want to say I've talked a lot about, you know, technical solutions, but a lot of this problem of validation is a social problem that we need to, as a community, raise the bar and ask the right questions. This is my team. They're awesome.

And I'm done.

Thank you. Thank you so much.