Scientist Stories: James Zou, How to Evaluate Medical AI

A ton of excitement about using and deploying medical AI in different applications, healthcare settings, clinical settings. And to sort of motivate and set the stage for the rest of the primer, I want to tell you a bit about like a puzzle, a bit of a detective story that we had to solve recently at Stanford. All right, so here's what the puzzle is, and you can help me to solve this detective story. So the context for this is that there's actually a bunch of algorithms that people have been developing, including by commercial companies, on detecting skin cancer. So the idea is that people would take photos using their smartphones of their different skin lesions, and the algorithms, which often be some sort of ConvNet in the background, will look at these photos and tell you how likely it is to be this, to be like skin cancer, right? And clinically, this is actually a really important application because there are millions of people every year who have skin cancer but are not diagnosed until it's too late. And if you actually catch it early on, then it's relatively easy to treat. But if you catch it too late, then it's very hard to treat, right? So the idea is that because these kinds of data is very easily accessible, right? Anyone with a phone can potentially take these photos, then this can potentially, you know, be transformative in catching a lot of these diagnoses early on. So super compelling application, right? So there are a bunch of these tools that were developed, and we were interested in deploying them at the hospitals at Stanford. So we were asked to do a pilot. So we looked at these initial tools. And if you look at the original publications, they all had very good performance. The AUCs or accuracies will be above 0.9. And then we did an initial pilot on a bunch of the patients at Stanford's data, right? And then the performance is actually very poor. So the accuracy was dropped to about 0.6, which is not that much better than random. So I guess the puzzle is to figure out like why did these models, right, work so poorly on the real patients? Like why was there such a big drop?

Drop-off in their performance.

So we did like a systematic audit, right? And I'll tell you a little bit more about how we do these kind of audits to figure out the goal of the audit is basically to sort of explain this drop-off, right? Do you want to basically identify what are the mechanistic factors that can really lead to this drop-off, right?

This gap in the model's performance.

So the single biggest factor that we found, right? Was actually, I think, something that people didn't— here didn't mention, which is there were actually a lot of mistakes in the original test data sets, label mistakes. So what ended up happening was that in the companies, they were, you know, they're generating these test data to evaluate these algorithms. And in order to generate a large amount of test data, they basically had dermatologists visually look at each of these images to say, is this, you know, cancer or not cancer? And visual inspection, even by trained dermatologists also have a lot of mistakes, right? And turned out that those mistakes actually led to an inflation of the model's performance, right?

Um, that—

so that's actually contributed to about half of this dropout, right?

So even just the label mistakes, which is not often something that we typically think of, uh, like label mistakes in the test data, that, um, as machine learning developers, I mean, that the Basically, the models are making the same errors as the dermatologists.

A lot of them were, yeah.

So the actual ground truth in these cases come from you take a biopsy of the skin lesion and do a pathology test, right?

And then, so that's actually the ground truth that we compare with from the pathology test. And compared to that ground truth, there were just a lot of these systematic mistakes.

So the second big factor, something that people here mentioned, is that there is a pretty substantial difference in the disease distributions, right? So the original data were mostly trained on common diseases, and then here we had both common and also less common diseases because people, the people that came into Stanford. So the third factor that we didn't find was also quite important, which is the models end up doing much worse on images that came from darker skin patients. So basically if you look at the sensitivity of the model, so sensitivity here means that the patient has skin cancer, how often would the model detect skin cancer? So sensitivity is actually really low when the models are applied to images that came from darker skin patients. And when we look more carefully into this, we found that basically the original training and test datasets for these algorithms is to have very few, in some cases no images that came from darker skin tones. So these were actually quite interesting findings. So first, they reveal that you can have a model that does very well according to standard benchmarks, but there could still be actually a lot of issues, limitations with these models. And second, many of these limitations actually come back to different issues with the data. Could be issues with the training data or biases in data or some sort of distribution shifts in the dataset.. And there's also a lot of very heterogeneous noise and misannotations in the data, especially in medicine. And this is not a unique problem just for dermatology, right? So we actually did a systematic survey of the FDA-approved medical AI algorithms.

So back when we did the survey in 2021, there were 130 FDA-approved medical AI devices, right?

So I just summarized them in this plot here. So each of these symbols corresponds to one of these FDA-approved algorithms, and they're stratified by which body part they apply to. We can just focus on a couple of main takeaways here. One is that if you look at these algorithms, only 4 of them, the 4 that's colored in the red circles, were evaluated with more of a clinician-in-the-loop evaluation.

Somebody is actually using the algorithm to see how good it is as a part of this FDA evaluation process.

The remaining algorithms, so most of these, right, 126 of the 130 words evaluated similar to how the dermatology algorithms evaluated, so just applying them on the benchmark dataset that was previously retrospectively collected. And as we saw before, like, you know, how good the algorithm works on these retrospective benchmarks can often be quite misleading, right? And the second is that there are actually a lot of limitations even in the retrospective datasets, right? So most of the time these datasets were only collected from one location, from one site. So that doesn't really test how well the model might generalize across different hospitals. So we think that there's actually quite a lot of these important limitations in how all these medical algorithms are being evaluated, which sort of motivates some of the things I'll talk about next. But any questions so far? Yeah, what does high risk here refer to? So, the FDA actually has a categorization for each medical AI algorithm, like, is it considered to be high risk or medium risk? So, basically, if the algorithm is often directly leading to clinically important diagnosis or action, that's considered to be high risk.

Algorithms doing something like segmentation, that's then it's an earlier part of the clinical workflow, it's considered to be medium or low risk.

And the evaluations are more stringent for the high-risk ones. But I guess even here, we see that even among the high-risk algorithms, there's still very limited evaluation. Yes.

What do you mean include the clinician in the evaluation pipeline, and what do they really do there?

Yeah. Yeah, so basically 126 of 130 algorithms were only evaluated in a standard machine learning approach, which is like you have a retrospective benchmark dataset, right? So it was previously collected, and you run the algorithm on that benchmark dataset to see what is the AUC. There's actually no human involvement actually using the algorithm. The 4 out of these 130 were done with more prospective study. The prospective study here, then somebody was actually collecting the data in the future, and then there's actually maybe a clinician or someone who's actually using the algorithm on the new data. Good questions. So, you know, a couple of the main challenges that we learned, right? So, you know, one is that, of course, in biomedicine, a lot of the data can be very heterogeneous, right? So we need to have some systematic ways to figure out what is maybe higher quality, more informative data, and also to filter out the noise. And the second is that often the model's performance on these retrospective benchmarks could be very misleading, right? So we need to have some ways to do much more rigorous prospective evaluations, especially involving actual real users. So I'll tell you a little bit about how we try to mitigate both of these challenges. So for the first one, I'll tell you about some of these techniques that we and our colleagues have been developing around data evaluation, which is set up systematic algorithms to figure out when you have a dataset, which could be genomics or imaging, medical records datasets, how do I actually figure out what part of my data have different levels of noise systematically. So these are things that we're calling data shaft scores. And then that's for evaluating datasets. And then for evaluating the algorithm, we've also been running a lot of work recently on thinking about how do we actually really design truly randomized clinical trials to test medical AI systems, which is very different from how these previous algorithms were evaluated.

So diving first, well, first talk about data evaluation and then second about algorithm evaluation.

So for the data evaluation, right, so taking a step back, right, so here's sort of the mathematical formulation of this problem, right, which is a pretty general problem, right? So many times, you know, in computational biology or medicine, right, so we have some datasets which are training data, right? You have your favorite learning algorithm which could be as simple as a linear model or could be a more complicated neural network, right? And then you have some performance metric that you care about, right, which could be accuracy or how well this works in a particular deployment setting, right? Like, for example, like in my example I was mentioning, right, so the training data are these different dermatology images, the learning algorithm would be this convolutional neural network, and then my performance is that I want to know, like, how does the algorithm work across these different skin tones across different populations. Thanks. And the idea of data evaluation is the following. It's very intuitive, right? So the idea is that, okay, so I have my training data and I have my performance. Ideally, I would like to be able to mathematically attribute the model's performance back to the individual training data, right? So if a model gets like 80% accuracy, ideally I want to know, okay, so which data points contributed to that 80% accuracy? Similarly, if my model exhibits some mistakes or biases, I would like to know ideally, like, oh, this is the kind of data that's— because it's maybe misannotated—

that's contributing to that mistake.

If I can actually do this end-to-end way, going back from the result to the individual training data, then I can make the whole system much more transparent and easier to debug. How to do this attribution is the problem that we call data evaluation. So, how do people think about solving this data evaluation problem? So, maybe the first idea that might come to mind is the following. It's also very intuitive. Let's do some sort of leave one data point out kind of analysis. So, in this case, if I have these three training data points, which could be data from different hospitals or from different patients or from different biobanks, And I want to figure out what is the value and the contribution of this middle point, this start point. So what I can do then is to, I mean, I'll train a model using all of the data, and then this is the shading here will indicate the decision boundary that model learns. And I can maybe just leave out this point that I'm interested in and retrain the model and see how does the decision boundary change. And then I can see this, like I can use this to quantify what is the influence or the impact of this point of interest.

Does that make sense?

Yes. Oh, I guess it's shaded here. So it just says that this is hidden here, but it just says that this is actually related to what people call it. This is called leverage score influence function. This is leave one out analysis. It's actually been done before, and this is often called by different names in different communities. It sounds quite intuitive, but for the modern large data settings, this actually ends up being very noisy. The reason why it's very noisy is that Imagine if I have like a big dataset of 100,000 genomes or a million images, right?

Like the impact of leaving out a particular sample, it's going to be sort of very relatively small, right?

Especially if there's randomness in how I train the algorithm with different random seeds. If I do this leave one out, then it's going to give like a very noisy estimate of what is the value, what the influence of this particular data point. So then this motivated this idea that we and our colleagues developed what they call the Shapley value, which is a very natural extension of leave one out. So the idea here is that, okay, so if I do this leave one out once, it's gonna be very noisy, then why don't we just do this a bunch of different times? So we're gonna create a bunch of different scenarios where each scenario corresponds to taking a different random subset of my original training data. I have these different, 3 different scenarios here. And then each of these scenario, then I can do this leave one out analysis and say, okay, in this scenario, what is the impact of leaving out this point star? And in this other scenario, what is the impact of leaving out this point? So the reason why we take these different scenarios and the benefit of doing this is first, now we actually have much more robust estimates of the impact of this particular point because we're aggregating then across these different scenarios. And the second is that since different scenarios correspond to different random subsets of my data, this actually gives me a way to also disentangle maybe different interactions between different data points. So then what we do is we basically take these different scenarios, right? In each scenario, estimate the leave-one-out score, and in the end, there's a mathematical way to average across these different scenarios to get a single estimate for the for the importance of an individual data point.

And then we can do this across different data points.

Now doing this, there are some computational challenges which I'll come back to, how to do this efficiently. But this is at least the high-level idea of what we're trying to do.

Makes sense.

I guess maybe you'll address this, but you know, with different random seeds, if you're training like a deep learning algorithm, you might get wildly different accuracies.

That's right, yeah.

Would this have to be done over many different seeds, or would it be with respect to one seed?

Yeah, so it's a good question. So here we don't have to fix the random seed, so basically Each of these scenarios then could already have a different random subset of my data and they could have their own random seed. And this is sort of the benefit of this approach is that by aggregating across these different scenarios, we can show that we actually have a much more stable estimate, statistical estimate of what is the impact of this particular data point, which also marginalizes out across different randomness in the training data. That's right, if we have enough scenarios, yeah. Okay, so no, so here's just sort of the, you know, in formula, like what I explained in words, right? So the way we're going to estimate the value or the importance of a point, it's going to be this, you know, some weighted average across different scenarios, right? Where each scenario is going to take a different random subset of my data and look at, okay, what is the contribution of adding or removing that point from that subset?

In terms of the model's performance. And I can pick any performance metric of interest, could be AUC or accuracy or F1 score, you choose that.

So let me tell you then how we can actually use this kind of data evaluation methods. One way, like the motivation we talked about before is on try to identify mislabeled data points in my training set. So we actually applied this to a bunch of publicly available datasets that people are often using, including these, chest X-ray dataset. So basically when you apply the algorithm, it's going to assign a score to each image, right? So sometimes the scores are positive, right? Indicates that those images are more informative. And you can actually have images that have negative scores. So the negative scores often be an indicator that there's something potentially, some issues with that data point. So in this case, we actually looked at the images, the chest X-rays that have negative scores. And then we just went back and had like a panel of several radiologists to reevaluate each of those images. And it turned out that basically 60% of those images that had low Shaftley scores were basically misannotated in the original dataset. So for example, maybe they were annotated as not having pneumonia, but when people were actually reevaluating it, they decided that this image does correspond to a patient that was in pneumonia. So this is actually very helpful because then it's sort of a way for us to systematically figure out what could potentially be the subset of my data where it's useful to reevaluate or reexamine and filter those out. And removing those data points ends up really improving the performance of the model.

Just quickly following up from all those questions, but here you're training on the order of 100 hundreds of models.

Is that correct? Yes. Okay.

Yeah, yeah. So here we're basically training on the orders of, uh, yeah, thousands of subsets, right? So each subset corresponds to a model. Um, but I'll tell you in 2 slides of how we actually—

there are different ways of making this very efficient computationally.

Yes, just curious about the other 40% of the images. Is there something that unifies those images, something they have in common?

Yeah, so it's a good question. So the 60% are the ones where it's pretty clear to the radiologist when they reexamine that those are mislabeled. The other 40%, oftentimes it's a bit ambiguous to the radiologist. Like, OK, is this actually correctly labeled or not correctly labeled? That's some of those.

There were also a subset of those 40% that just look like different kind of outliers.

Maybe they were correctly labeled, but there was potentially some other artifacts in the image.

Such that when the models are trained on that image, that actually degrades the model's performance.

Sorry if I missed it, but you mentioned the number of subsets. What is the number of samples per subset? Yeah, there are a few sets that would be helpful to know the total number of independent samples.

Yeah, so it's a good question. So for each subset, it has like a different cardinality into the number of samples, right? So most of the subsets are on the orders of a few hundred to like— could be from a few tens of samples to hundreds or thousands of samples.

So the subsets also have different sizes.

Okay, so just to mention one other quick application, which is Going back to this dermatology example that motivated this, right? So there, in particular, the training data can be very heterogeneous. Sometimes people even just scrape training data from the internet. And in that case, then the Shapley score, right, this data Shapley score we compute is basically going to assign a score to each of these training images, right? And as we said, so the ones with negative scores are the ones that are likely to have some issues, right? Because they're misannotated or because they just contribute artifacts, right? So one, and you know, because the model is trained on this heterogeneous data, the performance of it on the actual patients, right, has this drop-off. So a very natural thing now is once we have these scores is to sort of retrain the model, right? But just on this, basically weight the data points, right?

By their Shapley scores.

So basically that would be a natural way to encourage the model to pay more attention to data points that we've learned tends to be more informative or have higher quality annotation. And turns out that if you actually take the same dataset and then take the same model and then retrain it on this weighted dataset where the weights come from the Shapley scores, you can actually immediately get much better performance on a deployment setting.

Units are the Shapley scores in.

Yeah, so the units here are in terms of how much they contribute to the model's performance. So let's say the performance is in the units of AUC or accuracy, then the Shapley scores will also be in the same unit. Okay. So these are just a couple of applications of what one can do, right? Once we actually have a way to quantify the values of individual points, you can use them to identify mislabeled data and also improve your model's performance by just training it on the weighted data, which encourages the model to pay more attention to the higher quality data points. And there's a bunch of other applications that people have been exploring now of using this to do active learning, to use this to do synthetic data generation, So I think there's a lot of potential applications of data evaluations on how to improve your data and also improve the downstream models. Okay, so there's also been a bunch of recent research on how to kind of compute these Shapley scores efficiently. There are basically like two main types of strategies on computing data Shapley scores. So basically, if you try to do this naively, then that requires taking a lot of different subsets and then training a separate model on each different subset, which becomes very expensive if you're talking about deep learning models. So two main types of strategies that people have been exploring are, first, in many settings, in terms of that, we can actually derive analytic formulas either that exactly or approximately computes the Shapley values. So for example, for different classes of regression and classification problems, It turns out that you can actually just analytically solve what does the value look like by taking all these different subsets, assuming the subsets are randomly selected. So in those cases, it's actually, you can just compute the Shapley value almost instantaneously, right?

Because these are actually analytic formulas.

In other more general models where the analytic formulas are not available, what people are doing is basically coming up with different approximation schemes. So one natural way to approximate the Shapley value is basically by doing some sort of Monte Carlo sampling. Instead of taking that exact sum across those different subsets, you basically just randomly generate subsets where you first sample the size of the subset, which is the number of data points in that set. And then you sample, once you pick the size, then you sample how many, what are, specific data points to put in into the set, right? And it turns out that those Monte Carlo sampling schemes can also be made to converge very quickly, right?

So usually on the orders of a few hundred samples, then they basically have converged into the actual estimate.

So with those kinds of techniques, people have been able to sort of scale up these Shapley computations to compute individual values for individual data points for datasets of on orders of hundreds of thousands of data points and for deep learning, ConvNet kinds of models.

Do you know if it's possible to compute the Shapley scores using like a bootstrap sample?

Using bootstrap samples? Yeah. Yes. Yeah, so it's a good idea. So it turns out that when, for example, when you're running like a random forest, right, or XGBoost kind of algorithm, basically generating a lot of different trees where each tree corresponds to a different bootstrap sample, you're actually effectively doing the computation for computing data-shaft-based scores, right? So, and we have actually a recent paper that shows that you can basically leverage all the computation they've already done when you do the random forest to basically get shaft values for free.

How does this relate to the idea of like mining hard examples where the examples near the decision boundary are sort of most valuable for training? But I sort of imagine those examples would also have negative Shapley values because they kind of make the problem harder.

You see what I'm asking? Yeah. So I think there's a really interesting connection between the data Shapley values and these hard examples and also with active learning. So actually a lot of the data points, let's say if you have a data point that's correctly labeled but near the decision boundary, it's often like a really important data point. And also tends to have relatively high Shapley values, because if I remove that data point, then my decision boundary could actually change a lot, often in a bad way. So that actually means that those data points tend to have high Shapley scores.

And basically if I do active learning to try to recruit more data points that have the Shapley scores, often that's a good way to figure out what are the things that are near decision boundary that ends up being more critical.

Also curious if you try implement your model on an unseen data point, if there's any uncertainty associated with that. For example, if you know that, oh, my model wouldn't perform well on this specific type of image because it's not characteristic of the images that I train on. It's like basically a low-quality image.

That's right.

Yes.

Yeah. Would you know that beforehand? Could you, before getting a prediction using the model?

Yeah. So it's a good question. So for that, then often we'll have to rely on some sort of distance measure to say how close is that data point in some embedding space. To my training set. And then, yeah, if this is very large, that could be a good indication that maybe that's a hard data point for my data to extrapolate to. So we won't be able to get that exactly directly from the Shapley values, but we can use the Shapley values and then do the extrapolation and see how close is it to the Shapley values that the data points in my set that have high Shapley scores.

How does the Shapley value, I guess empirically or theoretically, depend on the model architecture? That's, I mean, in the case of some kind of ConvNet, if you were to change its depth or something like that, how does that?

That's also a good question. So here the data evaluation is model dependent, right? So basically if I Do a, let's say, a linear regression model, I can get one set of Shapley values. So the data value, maybe the data points that are most important for a linear model will be different from the data points that are more important for a deep neural network. So as the network architecture changes, the model gets bigger, different subsets of my data ends up having higher or lower value.

Okay. Good questions.

On that last, that last question. So you, your outcome depends on the space of models that you examine, and each model is a priori equally valuable, right? Equally right. So how do you choose among them, or do there's some kind of majority consensus that develops?

Yeah, so usually the practitioner, they're using a particular model, right? So for example, for the dermatology developers, so these companies, they have a particular deep learning model they're using that they decided to train on this data. Then the Shapley value would say, okay, for whatever model that they've chosen, here are the different data points that end up being the most important or least important for that model. So, you know, some of these data values could change across, of course, to which specific architecture they choose. But often what we find is that the data points that have the negative values, right, end up being pretty consistent across the different models that you use. Because those are the data points that are misannotated, right?

So those are consistent across different.

So you're really counting on domain expertise here, is that right? Because you're asking for these samples, what do the practitioners find effective?

Yeah, so yes, that's right. So it's whatever models that you want to use, right? That you're already using on your datasets, then you can overlay the Shapley values as sort of a, wrapper on that and say, okay, for that model, what are the contributions of different data points?

Hi, pardon me, I have a question. How do you use this to distinguish between data points that are outliers but actually correctly labeled versus incorrectly labeled?

Yeah, yeah. So actually a lot of the data points that are correctly labeled but outliers end up having quite high Shapley scores because first those are data points that are relatively unique, right? If you're very redundant, you also are not that important.

But if you have data points that are unique and correct, then end up being more important.

So those end up having relatively high scores. And then the incorrectly labeled ones are the ones that actually have low scores because they actually actively hurt the model's performance.

It seemed like when you presented the results that the model did better in the setting where it had incorrectly labeled data. Like I thought maybe that those labels were easier to predict than the true ground truth. Like you described that it did well in the original setting with incorrect labels, but less well at Stanford with correct labels.

Is that right?

That's right.

So then why do the incorrect labels have low Shapley scores? It seems like they're actually having a large influence on the model and the model's ability to do well.

Yeah, so I think it depends on how we do the performance evaluation. If I do my performance evaluation on a misannotated dataset, then it will be harder to detect which of my training points are misannotated. In this case, we do the performance evaluation actually on a smaller dataset where we actually know the ground truth. And then we learn that, okay, in order for the model to actually do well on the real patients, here are the data points that hurt the model's performance, and these are the misannotated ones. Okay, yeah, great questions. So this is sort of an approach that we've been using, and a lot of our colleagues are also expanding, developing for data evaluation. And then maybe I'll just also more briefly mention some of the things we're doing on the model evaluation side, which thinking about how to design actually clinician-in-the-loop randomized clinical trials to evaluate these models. So I'll do this through a particular case study where we recently did one of the first of these large-scale randomized clinical trials evaluating medical AI. So the story actually started several years ago. So 3 years ago, we developed developed this algorithm for assessing heart conditions based on echocardiograms. So, these echocardiograms are very routinely collected. They're cardiac ultrasound videos. They're routinely collected. There are over 10 million of these done every year in the US. So, it's one of the most common measurements for assessing how the heart function or what the heart is functioning in the patient. And just like a quick primer is that how people The idea with these videos is that you have a video here, usually for a few seconds. And they're basically trying to see essentially how much power is the heart pumping, is heart producing in that few seconds. And they measure this with this quantity called ejection fraction. And how they do this is actually through a pretty manual process. So if you think of the heart as like its power pump, and to estimate how much power it's producing, you want to know like how much the pump is contracting, right? So then what they do is that they'll actually go through the video manually. They'll find the frame, I think the frame where the heart is the most expanded, right? They'll trace that out and they'll find another frame where the heart is most contracted. They'll trace this left ventricle chamber out. And then based on how much the volume contracts between these two frames, then there's actually a mathematical formula for estimating this thing called ejection fraction.

It's basically a measure of how much how well the heart is functioning. So this process, you can imagine, actually requires a fair amount of manual work, right?

You have to find those frames by eye and then do the tracing by hand and then do the calculations.

And it could be quite a bit of variation across different clinicians.

So that motivated why we wanted to, you know, develop this AI algorithm, right? So the algorithm is basically mimicking that workflow where it takes its input, this ultrasound video, And then does this segmentation automatically, that's shaded in blue, and then it estimates this ejection fraction for every individual beat. So the algorithm itself, I won't go into too much detail here, but it's a particular kind of neural network that's looking across this video. So it's basically looking for both temporal patterns and also spatial patterns of this spatial-temporal convolutional network. But the main thing I want to focus on is actually now that we have this model, right? So how do we really try to evaluate it to avoid some of these issues that we mentioned before? So the first evaluation we did in the original paper was just the standard kind of retrospective benchmarking that people often do, right? So then we would take this model and then we had some retrospectively collected datasets from two different hospitals, from Cedars and from Stanford.

And then we saw that, oh, the model worked pretty well across different datasets, right?

And then people were pretty happy with that. But we saw before, right? So these kind of retrospective evaluation benchmarks, even if it does really well, can often be very misleading, right? So for the last 3 years, we've been thinking about, okay, so how can we really do, really go beyond that, right? And then do a truly prospective study to see how well does this algorithm work when it's really integrated into the clinical workflow. When someone is actually using it. So, in this more recent work, which was just published a few weeks ago, we set up this randomized clinical trial or RCT to test this algorithm. So, the idea is that you have these videos that are being collected, and then we have these two arms where the first arm is that the videos would be triaged to the AI where the AI would do the assessment, would do the tracing of the segmentations. The second arm will be these videos will be randomly triaged into the standard clinical workflow where the human, when the human is called a sonographer, will do the tracing and segmentation. The AI algorithm will be integrated into this clinical workflow. It's using the exact same vendor systems from Siemens that the clinician is using. Then afterwards, both of these arms will produce the output which the cardiologist would assess. In a standard workflow, the cardiologist would also look at the output from the sonographer and assess whether they want to modify anything. So here in this case, the cardiologist, you wouldn't know whether they're looking at the output from the AI or from the sonographer. So they can just see, okay, do they agree with the readout or do they want to change anything? And then we use the— then that becomes like a validation for us to see how much does the cardiologist actually need to change the output from the AI model or from their the sonographer, right, in this head-to-head comparison. That's the setup of the study.

Is that clear to folks?

And to do this and to make sure everything is integrated into workflow, so we actually spend a lot of time to make sure to basically to build the algorithm directly into the vendor system, into like Siemens, right? So the benefit of that is that it doesn't really change The cardiologists and operators, they don't need to change any of their workflow or behaviors. They're already familiar to using this system. And the AI is just basically a layer on top of that. And this is down for about 3,500 studies where half of these are randomized into the AI arm, the other half are randomized into the human clinical workflow. And I'll just tell you the main result of this. So this is basically the main readout. So each dot here corresponds to one of these studies, one of these videos. The x-axis corresponds to the ejection fraction that's evaluated by the AI algorithm, a number from 0 to 100, and the y-axis corresponds to the cardiologist. They get the final say so they can modify these initial readouts. That's the y-axis. This one's for the AI cardiologist and this one is basically for the stenographer versus cardiologist. One main readout that we have is basically how much do the cardiologists have to change their assessments compared to their initial assessment. So, that's measured in terms of this being absolute deviation. So, for the AI arm, it's the mean absolute deviation is about, it's less than 3%, which is quite low, right? And compared to the stenographer, the human arm, so the mean absolute deviation is closer to 4%, right? So, this suggests that when they're looking at the readouts from the AI algorithm, cardiologist ends up having to make substantially less modifications. In particular, if we look at the number of cases where the cardiologist has significant disagreements with the initial readouts, with the AI arm, it's about 17% of the time, where the human snuggler arm, it's closer to 30% of the time. These significant disagreements are the ones where that actually involves the most amount of time because then that means that in these cases, cardiologists have to actually often redo the tracings and redo the assessment.

Yeah.

If AI is rarely assessing above 75, whereas the sonographer is assessing over 75, quite a large portion of— so yeah, is that some kind of inbuilt bias into the algorithm that it doesn't do that, or—

yeah, so it's a good question. So the, I guess the key threshold for the ejection fraction is actually around like 45 or 50, right? So because if the ejection fraction is below that range, below let's say 45, that's often considered by the considered to be a high risk for heart failure. So the algorithm ends up paying most of the attention toward this lower range because if it's above 50, then it's considered to be normal. And then how much normal is it, right? Often doesn't matter, it matters less. But if it's below 50, that's like the most important range because that's where maybe you want to be prescribed a different medication.

How did you encourage the algorithm to care more about that lower range?

Yeah, yeah, so that's a good question. So the algorithm is basically, it's trying to do the segmentations for the left ventricle chamber, which is basically, and that chamber is what's used to estimate ejection fraction. And it also is doing additional tasks like trying to predict risk for heart failure. And often risk for heart failure is very much determined by whether the ejection fraction is less than the threshold.

The pictures you showed seem to be two-dimensional projections, whereas the volume is what would matter according to ejection volume. They use lots of sections or just one?

Ideally, we'll get actually 3D. In the current clinical settings, Most of these ultrasound videos are 2D videos. It's because this technology is much more readily available. So there's like tens of millions of these 2D videos that are collected. And then there's basically like a mathematical formula that looks at this particular 2D video and then use that to approximate the 3D volume contraction for this ejection fraction. It's not perfect, but this is what people, the clinicians actively use.

Could you clarify what the cardiologist and sonographer are doing differently when they're doing their assessment? Also, is it just like a single— when they're doing this calculation, is it a single contraction or expansion in the video frame? You go from the big to small thing in one cycle?

Yeah. The sonographer is basically the person who That's the initial segmentation. Often she's just looking at an individual frame to pick out the largest and the smallest within that individual beat. Then she would do these two segmentations like the ones we see here, and then that's like the initial tracings to estimate the ejection fraction. Then the current standard clinical workflow is that then the cardiologist would do a final check. Cardiologist would say, does this look reasonable? If it looks reasonable, then she would not actually modify anything. If it looks weird or looks different, then the cardiologist would actually retrace the left ventricle chamber and re-estimate the ejection fraction, right? And where the AI is coming in is basically, you know, coming in here, right? That's the initial tracing.

The cardiologist still does the final assessment.

So The last thing I want to mention is that we actually also just asked the cardiologists, like, can they tell whether they're looking at the output from the AI or from the human arm, right? Like, you know, we wanted this to be a blinded randomized clinical trial, and is it actually blinded or not, right? So the cardiologists basically for each of the videos, they make this prediction whether they think it's looking at the AI or stenographer, and they were only able to correctly guess what they were looking at, whether AI or stenographer, less than a third of the time, right? Which actually means that the blinding I think was actually quite good because the algorithm is really integrated into the clinical workflow.

Like they couldn't really tell whether, they basically didn't have to change their behavior at all because they don't know whether they're looking at the AI or the clinician, right? And here we thought the blinding is particularly important because if the doctor actually knew that they were working at the output of an AI model, then they might not have changed their behavior.

Which then would be a confounder in this randomized clinical trial.

Naturally, to want to take a temporal Fourier transform on the image data. Is that part of the algorithm or not yet?

For these videos, so people are not actually looking at the Fourier transform. They're really just looking at like individual beats and looking at the contraction for that individual beat.

So what is actually interesting is it's actually recommended for them by the American Heart Association to take multiple beats and look at the variation in the rhythm across different beats.

But because the current workflow is very manual, it's too time intensive.

People only look at one beat.

But the AI algorithm, it's essentially instantaneous.

It actually does look across multiple beats and that's improving performance.

So, yeah, exactly. That was my question. So, now that you can do multiple, I guess, cardiac cycles, presumably you could get better estimates of this fraction and also kind of estimates of the variability.

Yeah.

And do you think those types of things might be informative for whatever it is the cardiologists are trying to assess?

Yeah, so I think multiple beats is actually quite informative, especially for symptoms like arrhythmia where it's basically irregular beats, right? For these things, you actually, you don't, it's very limited if you only look at one beat. You actually do want to explicitly look at variation across different beats. But looking at different variation across different beats previously was very manual, time-intensive. Basically, you have to segment out and trace out multiple beats. But with the algorithm, we actually do see a lot of benefits from aggregating data from multiple feeds. Okay. Oh, by the way, we actually also released all of this data under the code, so it's all publicly available. So we identified, we anonymized all of these videos for over 10,000 of these patients. So if you want to actually explore this dataset yourself and play with the code, it's all publicly available. Yeah, so I'll just wrap up and summarize here. So we try to— I think we basically focus on these two main issues on the data side and also on the algorithm side. And we think that these data evaluation approaches, it's quite useful as a way to systematically figure out what are the challenges in your dataset, the misannotations. And then on the algorithm side, so I'm actually quite excited about using these randomized RCPs as a new standard of how how we think medical AI should be evaluated going forward, right? So we're really going beyond these sort of benchmark performance metrics. So I'll just stop here. So like all of the papers and the data that I mentioned are available online, and then these are the main references. And these works are all driven by amazing students and postdocs. Been fortunate to work with, in particular I want to acknowledge David Ollam and Brian Kueb, who led the cardiology work, and then Roxana and Eric Wu, who led the dermatology AI work. So excited to talk about and have a discussion around how we can actually use generative AI for a lot of these biomedical settings.

Right.

No, I'm sure a lot of you have been playing around with these different models. There's been really tremendous advances, right, in generative techniques. So for example, you can do text-to-image, right? Write something like, you know, these different captions, and it would generate all these amazing images for you. And certainly generative language models like GPTs have also been quite transformative, has a huge potential going forward. So what a lot of things that we've been interested in my group is thinking about, like, how can we leverage these kind of generative AI techniques, especially for these more foundational biomedical discovery process. And there's sort of 3 main areas we found these generative models to be quite helpful in a lot of our research work. The first direction is around really thinking about how to use these generative models to help us to expand the researchers' creativity, especially in settings where there's maybe a large number of possibilities of large space to explore. And the second one, second direction is using these generative models to help us to really balance very complex multidimensional trade-offs. So often in biomedicine, we have to trade off things like accuracy and efficacy and safety. A lot of these high-dimensional trade-offs, we actually found these generative models to be very helpful So that guide us in balancing trade-offs. The third direction is try to use these generative models to often to turn data that are maybe more readily available into much more expensive and richer kinds of data, to essentially up-level our datasets computationally. So these are sort of 3 broad directions that we're very excited about applications of these generative AI approaches. So I just want to share some of these with you through sort of very specific, and so like a specific vignette for each of these directions. All right, so for example, for the first one, I'll tell you about a recent story of how we actually use these generative models to design and to experimentally validate some of these novel drugs, novel antibiotics for pathogens that have developed resistance, right, to the existing drugs. And then the second one, I'll tell you about actually how to use these models to design new clinical trials. Often clinical trials are some of the most complex trade-offs you have to balance. So these models end up being very useful. And the third one I'll tell you about is how we actually— this idea of in silico spatial omics, right? How to generate— computationally generate synthetic data corresponds to spatial transcriptomics and spatial proteomics using these approaches. And like before, like in the primer, so feel free to interrupt and ask questions. Okay, so starting with this idea of how we use these generative approaches to design new drugs, right? So here's like a brief history of antibiotics to motivate why this is particularly, I think, a good setting for these generative models. As many of you know, so the first antibiotic was discovered by Fleming, right, in 1928. And in the first few decades after this, there's actually sort of like a gold rush of a lot of new drugs being developed and coming onto the Clinics. However, for the last 60 years, there were actually relatively few structurally novel antibiotics that people were discovering, right? And the reason why this is a big challenge is that, like, if the drugs are not very different from each other, then it's easier for the bugs, for the pathogens, to develop resistance across many similar drugs. And for— and because, right, we are not getting a lot of these new antibiotics, the existence of bacteria that have developed resistance to the existing antibiotics, right, this kind of antibiotic-resistant bacteria, is becoming like a huge public health crisis, right? So currently over a million people die every year because of drug-resistant bacteria, and that number is really estimated to go up to over tens of millions of people in the next few decades. And so this is actually a kind scary timeline. So on the top of the timeline corresponds to when a new antibiotic and when a new drug is actually discovered and put onto the market.

And on the bottom is basically when we first observe drug resistance to that drug.

You can see, actually see that in many cases there's only a few years gap between when we have a new antibiotic versus when we sort of lose efficacy due to resistance for that drug. So it's really an arms race, right, between new drug discovery and pathogens. And it's an arms race that we're currently losing because we're not actually producing many novel drugs. And so consequence, you have these bacteria like the one shown here, the Acinetobacter baumannii, which is considered by experts to be the perfect predator because it has developed resistance to many of the existing antibiotics that we have, right? So then just motivated by this, I'll tell you about, you know, how we can actually use the generative AI to help us bridge this gap, right, to quickly develop actually a lot more of new antibiotics. And in particular, we'll have a new antibiotic designed by AI that's experimentally then recently validated to be effective in killing Acinetobacter and also a bunch of other pathogens. So this work is led by Kyle Swanson, who's a really awesome PhD student in computer science working with me at Stanford. And the idea here is that we have this vast space of different potential small molecules. And chemists and drug developers have only explored a very small slice of that space. And this is a great opportunity to actually use these generative models to help us to explore maybe new parts of the small molecule space that have never been synthesized before. We're going to use these models to help us to explore those space. And also to help us to synthesize new molecules to experimentally validate them. So this whole pipeline is led by Kyle, and I'll explain a bit of how we approach this problem. So on the technical side, one of the biggest challenges with actually using generative AI for drug discovery has been the fact that the algorithms are really great at coming up with new drugs, new molecules. Right, but it's often very hard to synthesize these things, right? Uh, maybe you can design new molecules, but we don't really know how to efficiently build them. If you can't really build these new molecules, then it doesn't help anyone, and this makes it hard to do the experiments, and it's certainly hard to mass-produce. So how we want to actually solve this problem is to not just ask the generative AI to generate new molecules for us, But we want the after-generative AI to actually generate new recipes for creating these new molecules. So each recipe is going to be basically a step-by-step procedure for how to efficiently actually synthesize each of these new drug-like molecules. So it's going to tell us the recipe. And then we're going to experimentally follow those recipes to actually produce these new molecules. So here's an example of what these recipes would look like. So often for these drug discovery, we're working with these small molecules. And these small molecules, often we're starting with these building blocks, which are called the molecular fragments. So each of these building blocks would be things that are easily available. And there are a large number of these molecular fragments that are easily available on the orders of over 100,000 of these fragments. And then we have these known chemical reactions. So each of these reactions will correspond to maybe a template I can take a subset of these fragments. And it's basically the rule that says that if I put these two fragments together, then I can create a more complicated outcome, right, after I do this reaction. All right, so we have hundreds of thousands of fragments and we have hundreds of these reactions, known chemical reactions. And you can do these in combinations, right? So then you can have this massive space of potential molecules. So the idea here is that we want the generative AI to basically tell us what is the step-by-step procedure of taking the fragments and which reactions and in what combination to actually produce the outcomes. And then we want it to basically give us a very robust set of recipes for building each of the new molecules. So it will be— so the technique we use will be Monte Carlo tree search. So similar, yeah, so it'll be more of a sampling-based approach.

Hi, I have a question. Could you use this method to limit the creation of forever chemicals? Because a lot of processes you use now actually generate these for other chemicals that we're trying to get away from?

That's right. Yes. Yeah. So, so that's a good question. So we can actually build into the multicolor tree search specific constraints to steer us away from maybe undesirable parts of the chemical space.

Yeah. So it's been some years since I learned any organic chemistry, but what was most often the case is that the yields for these reactions— the reaction looks great, but the yield is just terrible.

That's right.

And I would have thought with the reaction you would want to indicate the yield because it could be vanishingly small.

Yeah, yeah. So the reaction efficiency is also something that we do consider. We want to take basically reactions and also reactants, right, that are likely to be highly efficient. Okay, okay. So, so more formally, so the, the way we try to model and solve this problem is through this Monte Carlo tree search, right? It's a very similar kind of algorithm that's, uh, if you've seen like the AlphaGo kind of papers from DeepMind, right? That's kind of similar to the deep reinforcement learning algorithm that they use for game playing. But in their case, they're basically using this very large tree search to explore like what moves of chess or Go to use. In our case, we're actually using this tree search to explore what set of fragments and reactions to put together into these new molecules. So the tree's gonna basically be exploring this vast chemical space. So here's the illustration of how this works. So in the initial levels of the tree, we're starting with the individual fragments. So there could be hundreds of thousands of initial fragments. As I move down the tree, we're going to get more and more complex molecules. So the first level will be the individual fragments. The next level, then I can have different— look at different combinations of fragments, different pairs of fragments. And then I can apply different reactions. So each reaction applied to a set of fragments, then it's going to produce an outcome, a more complex molecule. So not all reactions are compatible with different fragments. So not all parts of this tree are accessible, but we can actually build that into the tree search that is exploring only parts of the space where we know are generating valid molecules.

Well, this is amazing work.

It's just one little question. Does this have to be a tree structure?

I think it can be a DAG, right? If you say put one thing after the molecule, another one before it, can your best guess. So there could be different pathways of getting to the same molecule, but there are actually ways of grouping things together to make it simpler into like a tree search. Oh, I see, I see.

Yeah. Okay. So basically, as you go down this tree, right, so each row out for each path from the roots to the leaf then corresponds to a recipe, right? Because it tells you a step-by-step procedure of starting with fragment 1, add fragment 2, apply reaction 27 to get an outcome. So that's actually very convenient because each row often corresponds to a valid set of a recipe for generating new molecules. Of course, this is still an extremely astronomically large search space. So then we need some— still have some ways to steer and to guide the AI algorithm. And the way that we guide this is through another AI model, what we call a property predictor. So this property predictor is going to be like— think of it as another Classifier or discriminator, right? It essentially looks at one of these computationally in silico generated molecules and say, okay, so how likely is this molecule to be effective for, let's say, as an antibiotic, right?

For killing these pathogens.

So it encodes the molecule as a graph, like molecular graph, where the, you know, you have the atoms that are nodes and the bonds are the edges. And it look at this graph and then uses a graph neural network to basically kind of estimate like how likely is this to be a an effective drug. So this property predictor is going to compute then a score to say, okay, so this generated molecule is likely to be effective. Then that score can then be backpropagated through this rollout, this path, to update the scores for the individual fragments. So sort of like similar to these reinforcement learning ideas where you can actually propagate these scores as a way to make the search much more efficient. And when we're doing this search, we're trying to balance these different competing objectives. So you want the molecules we generate to have high efficacy, to be effective in killing these bacteria, killing pathogens. We also want them to be structurally novel, different from anything that people have synthesized before, safe for humans. And importantly, we want this whole process to be very highly efficient in terms of synthesizability. So again, we want molecules that are easy to produce.

Maximum depth of the, like, the number of reactions that you restrict to?

Yeah, it's a good question. So, uh, so in principle, the algorithm can, can go quite deep, right? Can look at different combinations of reactions. And in silico, we actually do look at, uh, maybe depth was, uh, 6 or 7, where we have actually multiple reactions. For the experiment I will tell you about, where we actually synthesize these molecules, we actually look at usually like depth 3 or 4.

So hopefully the overall strategy makes sense to people.

So we're going to use the tree search, and then the rollout is going to give us the recipes that are most promising.

And the score is differentiable. It's not, um, yeah, so you have that, you have something pretty differentiable.

That's right, yeah.

So the, the, the property predictor comes on, it comes from like another neural network. Okay, so, so this is all done in silico, right, at Stanford. So now the fun part begins.

Now we actually get to make these molecules and see do they actually work, right?

Uh, so, uh, so this is sort of the Interesting part of the story, so actually, because one of the best companies for producing these molecules called Endomim, right, they're based in Ukraine, in Kyiv. So we worked with them quite closely for the past year to actually produce these molecules, right? And you can imagine that they have a lot of other things on their mind for the past year beyond our collaboration, right? But amazingly, even with the war going on, they were actually still very— we can still call them on the phone, and they were able to make these things for us. So basically what we did is that we took 70 recipes created by the AI algorithm. Again, each recipe corresponds to one of these robots. We sent that to NMIM, right? And then each recipe then corresponds to the step-by-step procedure, and they basically follow the recipe to create the new drugs for us, right? And especially with all the work going on there, like the recipes have to be extremely easy and robust to follow, right? Because they're not going to optimize things.

Or try out too many different things, right?

So these have to be like so easy to follow that you can, you know, follow them and build these drugs even with electricity outages and other stuff going on. And we're very happy that actually they were able to efficiently synthesize 58 of the 70 recipes, or they were able to synthesize 58 entirely new molecules by following these recipes, right? So the hit rate is actually over 80% for generating these new molecules. And then the next step is that then they actually figure out a way to ship these molecules back to us, which itself is actually a bit of an adventure, but we managed to get these things back. And then we can then in the lab experimentally test them against these pathogens, right? And for this experiment part, we want to focus on pathogens that are the highest like the most urgent and the highest priority. So the World Health Organization has this designation of what are called escape pathogens, right? So there are 6 escape pathogens which have basically developed massive resistance to many of the existing antibiotics. And we had in the lab, our collaborator, John Stokes, we had basically 4 of these 6 escape pathogens. We tested each of the 58 new molecules against these 4 escape pathogens. And there were 6 molecules developed by the AI, the ones that are shown here on the top, that were experimentally shown to be effective in killing all 4 out of these escape pathogens, which includes the Acinetobacter baumannii, this perfect predator that we saw as the initial motivation. So these are just a bunch of experiments. The details are not so important, but these are basically experiments that how they do these experiments, essentially do these dosage escalation Right, so you want to basically show that these drugs are effective in killing the bacteria even when they're used at very low dosage.

At least lower the dosage, the less likely it is to have toxicity and side effects when it's used in patients.

So even at very low dosage, so all 6 of these novel antibiotics were shown to be effective in killing all 4 of the escape pathogens. So that's very promising. We're doing additional animal testing now to really assess the side effects and toxicity, but right now it seems to have relatively low toxicity, which is also good.

That's really cool. So—

Yeah, maybe you can go.

Yeah, I'll go and then the other person can go. What was the space of molecules that you designed that reached whatever chemical property you were optimizing for before you narrowed it down to 70, or was it just— those 70? Like, was there any kind of selection from a bigger superset that led to those final 70?

Yeah, so basically we use this, you know, this discriminator, the property predictor, to help us to guide to decide the 70, right? So we initially have like a larger set of a few thousand potential drug-like molecules, and then we use the property predictor to rank them, and we also further select based on diversity and novelty, so they're structurally different from the existing drugs.

Was the desire for low toxicity compounds built into the model some way, or was it kind of like fortuitous that it seems that these tend to be low toxicity?

Yeah, that's a good question. So that was sort of implicitly built in as a part of the discriminator. So the discriminators are trained on available drug screening data and, you know, where we prioritize drugs that have efficacy at low dosage, right? So it's part of what's learned by the discriminator is to, you know, that the low dosage is good and it's how it's able to implicitly guide the search space to look for these regions.

Do you have any methods for predicting which of these compounds are least likely, or I guess most likely to stay effective for the longest time period against these bacteria? Or are you working on testing that in real life with the already resistant bacteria?

Yeah, that's a good question. So we are testing them experimentally now. It's hard to say beyond the fact that these are molecules that are very different from any of the existing molecules that have been seen in nature before. So we're optimistic that that'll be a bigger barrier. But the other thing, so but that being said, we do expect the drugs, the bugs to develop resistance to them at some point. The nice thing about this process is that it's actually very fast, right? So the whole process from the AI drug generation, synthesis, and testing actually took about 2 months. And actually, I think a few weeks of that is just to get the molecules back from Ukraine. And so that actually, so that hopefully potentially shortens the development process so that we're actually developing much faster compared to the timescale of evolution of the bugs.

I have a quick question about the novelty along those lines. Can you comment a bit about how the novelty is related to the ability of the discriminator to generalize? It seems like it's very related.

Yeah, so it's a good question because we are basically explicitly asked the Monte Carlo tree search to extrapolate into like a novel part of the chemical space, and almost by construction that makes the problem harder for the machine learning algorithm. So basically in order to help that problem, we found two things to be quite useful. The first is that we actually build in a lot of domain knowledge into the chemical reactions, right? So the AI gives a lot more structure to the search problem that we saw that, you know, because it knows the chemical reactions, it's able to extrapolate these new molecules even though it hasn't seen them before, but it knows the sort of grammar of how the molecules come together. And the second is that we actually end up not using just a single discriminator, but we actually use an ensemble of multiple discriminators. And that also makes it more robust to extrapolate into these new novel regions.

Okay, great questions.

Yeah, so overall, still a work in progress, but we're very excited about this approach of using this Monte Carlo tree search and reinforcement learning ideas as a way to explicitly generate not just new drugs, but also easy recipes to actually synthesize these new drugs. So that makes it easy to close this entire drug development pipeline process And then we can then generate, test, and use the signal to feed back into the model and to improve the generation algorithm. So that's just one example of how we can actually use the generative AI to really help us to expand the creativity of the research, especially in searching for new parts of the molecular space. So the second vignette I want to mention is how do we use these models to help us to balance these different complex trade-offs. I will see actually behind the scenes will be a similar kind of combinatorial search algorithm, but the application, even though the application here is very different. For this part, I want to tell you maybe about how to design new clinical trials, which are maybe some of the most complex protocols in terms of having to balance many different trade-offs. Some of the initial work on this is published in an initial paper led by Rixiang Liu, who's another fantastic PhD student in my group, where Rishan and our collaborators helped to develop this platform called Trial Pathfinder, which is a way to design new clinical trial protocols, right? Computational studies protocols. And we're actually quite excited that this is now being used by the largest pharma companies like Roche and Genentech to help them to design some of the new cancer trials. It was also selected by the pharma industry as the top data science innovation last year.

So we're excited that it's getting a lot of uptake.

So the reason why we're focusing on clinical trials is because clinical trials is really the bottleneck of entire medicine, right? So you can have hundreds of great drugs, you know, like the ones we developed before, but in order to get them to patients, only one of them would make it to patients by having to go through all these very strict clinical trials. And each trial can cost tens, sometimes hundreds of millions of dollars to run, which makes the average cost of bringing actual drug per patient to be on average several billions of dollars. Right? So anything that we can do to make this clinical trial process more efficient, right, and also more diverse, more inclusive, can just be broadly beneficial to the entire medical translation process. And one of the reasons why these clinical trials are very expensive and also very complex to design is because they are highly selective. Right? They're, you know, very picky about which patients they enroll. You can have a trial that has no particular disease, right? And then the trial will actually have pages of rules to filter out, you know, which disease are eligible or which patients are eligible to participate in that trial, right? So, maybe some patients are filtered out because they have comorbidities. Other patients are filtered out because their lab values are outside of some range. So, many of these trials are basically looking for what are so-called Olympic athletes among the patients that they enroll in their study. They're really filtering out a lot of the other patients. That has several really significant consequences. The first is that a lot of patients, they are not eligible to enroll in these trials, so they can't really access these treatments. It also makes it very difficult for the drug developers. Something like 40% of cancer trials have really a hard time to enroll enough patients. I think it also means that because the trials are so highly selective, right, they're looking for these Olympic athletes of patients. So, how well the drugs work on these selected patients may not reflect how well the drugs work on average patients like us, right?

So, there's a huge distribution shift, right?

And, you know, people also recognizing recently that these clinical trials are just really not very diverse, right? The trial participants are not representative of the general population.

Which also makes it hard for the efficacy of the drug to really be validated in general.

So how do people currently design these trials? So to illustrate this, I should take a screenshot of, you know, some of the trials our collaborators are developing, right? So the details here are not important, but each row here corresponds to one of the rules, basically, for filtering patients, right? So basically, this rule says that, you know, if the patient's neutrophil count is outside of some range, then they're not eligible to participate in this trial. It's actually really interesting to actually see a discussion in the sidebar of the Google Doc. Actually, a clinician actually asked, can we actually relax this threshold to enroll more patients? Then somebody else then say, well, we don't really have enough evidence to change that threshold. Then the third person would say, then let's just keep the more conservative threshold that was used before. People do want to change, but they don't really have the good tools to make this change. And is there people recognizing this is a huge challenge, right? For example, the National Cancer Institute in the US has a mandate now that all of these clinical trials needs to be simplified and to enroll more diverse patients.

But how to actually do that in a data-driven way, that's still the open challenge.

And this is where we think this generative AI could be very helpful. Right, so the idea here that we have is actually also very intuitive. So we're going to basically use the generative AI to help us to design the eligibility rules, design these protocols for the clinical trials. And we're going to make that data-driven in that we're going to actually use the data from large-scale medical records to help to guide the AI algorithm in designing clinical trials. So that's going to essentially provide the reward signal to help inform how the AI was going to design these protocols. And to do this, we partner with Flatiron Network, which is a network of over 2 million cancer patients in the US. They have medical records data from almost 300 hospitals in the US. All right, so here's basically the high-level idea of how this approach works, right? So basically, the algorithm is going to look at all these existing clinical trials and come up with some initial set of designs. So again, think of the eligibility design for clinical trials, essentially a set of mathematical rules and statements, right? So essentially you have all these mathematical constraints and you're looking for the intersection and the intersection can give you the eligible patients. So you can have dozens of these rules, right? And then the algorithm can propose a set of rules and then it goes into the medical records of these 300 hospitals, right? And it's going to generate synthetic pull out patients that satisfy these rules, right? And also generate some natural control patients that maybe took different treatments. And then we can computationally match these distributions, right? And then see, okay, the patients that satisfy these rules compared to the patients that didn't satisfy the rules, like do they have differences in their safety or efficacy, right? So that's going to be like the signal that the algorithm gets as a sort of reward, as a way to guide the search in the rule space. And because it's all done in silico, right, so it's actually very easy for the algorithm to change these constraints, right? So maybe you can change these different thresholds to enroll more patients, can even remove some of these thresholds altogether, remove some of the rules. It makes these changes and then see, okay, so how does, you know, relaxing these rules or changing this threshold, how does that actually affect the real-world efficacy and safety of the patient. And to test this, and then goes back to the medical record, generates additional sets of patients, right? So now because these things are relaxed, now there are more patients that are eligible now, right? These are real patients, right? And then we can see, okay, do these real patients, do they have more toxic, you know, do they have more adverse events or what is their survival look like? So based on all of this information, it guides the algorithm to search through this combinatorial space, right, of these different thresholds, different rules, to see what's like a reasonable range of things to set, right?

And that becomes the recommendation. So I think I can conceive of how the constrained optimization would tell you something about the number of eligible patients, but could you say more about how you get a sense of safety? I mean, if I were running a clinical trial, the point of the clinical trial would be to tell me, well, is this safe for a certain category of patients? How can you back that up?

Yeah, so it's a good question. So for each of the patients, since we have their longitudinal medical records, so we actually have things like their survival, overall survival, progression-free survival, or if they have major toxicity events, adverse side events, Some of those are captured in medical records. So then the algorithm would see that, okay, so by enrolling these patients, maybe they're not Olympic athletes, right? But they're like a little bit sicker patients. Then follow their medical records over time and see, okay, do these patients actually have worse overall survival or do they have more adverse events?

That's right.

Yeah, so we're in silico, so we're emulating what this actual event would look like using the longitudinal medical records. So that's the signal that the algorithm learns to guide its search with. Yeah, so we don't have the new treatments. So oftentimes, we try to look for maybe a most similar treatment, like from a similar mechanism.

Maybe I don't have a new immunotherapy, but there's a lot of immunotherapies on the market, so we can use that.

I have a quick question about multiple testing in this context. How do you— because many of the tests that you do to test differences in safety are going to be somehow related to each other because they involve tweaking.

How do you control for multiple testing in this context?

Yeah, so there could be all sorts of correlations, interactions between these different thresholds and different lab values. So actually the algorithm also has to learn that, oh, maybe if I change the bilirubin level, that also changes what patients will have with hemoglobin counts, right? So the algorithm actually learns that because it sees so many medical records on these millions of patients, right?

And then in the end, you know, to finally validate the algorithm, we have like a held-out set of patients from other hospitals that hasn't seen before to validate the protocol. Ever tried synthesizing patients? So for many rare diseases, you probably can't even recruit all of the patients.

Yeah. Yeah, so that's something we're exploring next. So here we're focusing on the relatively more common cancer types, so where we have enough patients to basically do this emulation. So I think doing it for the rare diseases, it's interesting.

It's much more challenging.

Okay, so then just to summarize, right, so the algorithm then basically, you know, takes these rules, searches through this combinatorial space of rules. So similar to what we're doing with the antibiotic space, but instead of looking at molecules, we're actually looking at these different lab values, right? So it does millions of these simulations in parallel, right? And then in the end, actually summarizes all of this information together and actually uses the Shapley value I mentioned in the primer to basically summarize all of this information together into one set of data-driven recommendations for the patient, for the clinicians. So basically, for each of these clinical trials, these are different potential rules for deciding what patients to enroll based on lab values or different biomarkers. The algorithm then proposes essentially an initial design for that trial. So we applied this to several different types of cancer. Just focus on lung cancer here. So here, we applied it to 10 different trials looking at clinical trial sets, advanced non-small cell lung cancer. So each of these is one of the trials. We can just focus on the average in the interest of time. So basically, the first two numbers here would indicate if I used actually the original designs of these trials, how many patients will be able to enroll, and what is the hazard ratio of those patients? The hazard ratio here, lower the better.

So lower hazard ratio means a lot better, survival.

And then the last two numbers here indicate if I actually use the algorithms proposed designs of these trials for the eligibility rules, right?

So how many patients now we're able to enroll on the hazard of those patients, right?

So we find that across these different trials, by using the algorithms, the data-driven designs, we can more than double the number of eligible patients.

That's great.

And also interestingly, even though we're recruiting a lot a lot of the patients beyond these, you know, very narrow cohorts, maybe some of the sicker patients, we actually don't see, and then we see a similar, if not maybe a little bit lower hazard ratio, which means that many of these sicker patients actually benefit even more from the treatments compared to the Olympic athletes of the patients. And I think this is actually really important also from an access perspective, right?

From equity perspective.

It's one thing we find is that when we actually use the data-driven designs of these trials, we're now able to enroll a lot more minorities, a lot more women, and also a lot more older patients. So many of these demographic groups were basically systematically filtered out. They're not eligible to participate in the original clinical trials because they're not considered to be as healthy or their lab values are not within these very narrow ranges, right? But what we— the learning from this algorithm is that actually we could relax those ranges, enroll more patients, more diverse patients, and at the same time ensure the safety and outcome of these patients.

Hi, I have a question. Have you thought— this is a great method. Have you thought about applying it earlier in the process? For example, looking at preclinical studies? Because a lot of the preclinical studies are what give us these trial eligibility rules to begin with, but maybe the models we're using are not representative of the populations that we're interested in.

Yeah, so that's a great question. So that's something we would love to do next. So here the trials are all like Phase 2, Phase 3 trials, so we'd love to move earlier up in the clinical pipeline. Yeah, okay, but just to summarize this part, so we're excited about using these generative approaches as a data-driven way to help us to design the eligibility criteria of these trials, right? And then we can validate this using actual clinical trials and also other independent cohorts. Here we mostly focus on cancer because that's where we have the most, the best curated data, but I'm excited about expanding this to other diseases like autoimmune diseases and outcomes. And, you know, we're excited that now this is actually being used by the pharma industry to help to design their trials. There's also, you know, a big push by the industry to design more inclusive, more diverse clinical trials.

I think this aligns very well with the broader mandate, right, from the FDA and from the NCI on these more inclusive trials.

Great. Any more questions about this part?

The rules that you learn for inclusion and exclusion in the trials, are they simple and interpretable, like lab value A must be some threshold, or is it like a more complicated decision?

Yeah, so actually one of the really nice things that it ended up being very simple insights, very simple rules. For example, The algorithm actually learned that a lot of the lab values, like bilirubin counts, ends up being very narrow, too narrow in the original clinical trial. They're filtering out too many patients because these bilirubin thresholds were set based on like what people thought were like healthy patients in the '80s, right? And they filtered out too many women and too many older patients. And the algorithm actually learned that, oh, we can have a broader range for bilirubin values, right? And that would enroll a lot more patients. And it's also very easy to implement. But those are the very easy insights actually the algorithm learns but ends up being quite generalizable across different cancer types.

Sorry, could you speak more about the validated using clinical trial data piece? Is that like validating your estimates of what the effects would be in the sort of in silico? Yes, that's right.

Yeah. Yeah, so to do that, we actually leverage— there's a lot of natural experiments essentially in clinical trials. So different companies ends up just, even different teams within the same company, they end up using different thresholds for designing their clinical trials because there's not a common standard, right? Some would use a bilirubin threshold of 1, others use 2, right? And then that's sort of a natural experiment that we can then use to say, okay, so the algorithm actually said in this case threshold of 2 would enroll more patients and we'll have no more side effects. Then we find, oh, there's actually a company like Genentech that actually did a threshold of 2, and then we see that, oh, they didn't actually have any more side have better outcomes. So that would be a way to use the natural variation in clinical trials to validate the algorithm. So in the interest of time, maybe I'll just— I wouldn't really go into the third section too much, except just to say that I think one of the other ways we can use these generative models is also to help us generate synthetic data. One of the big types of data that I'm particularly excited about, I know people here in the audience have also done a lot of great work in, is more like spatial omics data, like the spatial proteomics. Here I'm showing like image from CODEX. So these kinds of data are relatively difficult and expensive to generate, right? And especially for a large number of clinical samples. So it's not one set of work that we're using generative approaches to do is to basically help us to essentially computationally generate CODEX images that have similar level of insights to the actual experimental codex data. Right, so just to— happy to discuss more details maybe in the discussion section, but just to show you what this would look like, right? So, for example, in this level of spatial proteomics, you're basically measuring 30 to 100 different antibodies in parallel over these clinical samples, right? And we show that we can actually computationally impute many of these experimentally measured antibodies, or just based on the morphologies of the cells using some combination of H&E images and maybe a small number of the measured antibodies, right? So then that means that instead of having to experimentally measure all of these other antibodies using some more expensive experiments, you can actually generate, computationally generate synthetic data that will cover these other antibodies, right? So here will be like, for example, you know, example of where you actually do like CD1c3, experimentally and also what we generate is from the computation. And we show that essentially, like, if you actually do the actual CODEX experiments versus if you take the synthetic CODEX data we computationally generate with these models, you can actually get very similar cell type identifications and also very similar microenvironments, right? These are just examples where it's taking from the 25th, the median, the 75th percentile in terms of how well the synthetic data is, right? So 25th percent power, basically the settings where the synthetic data is actually relatively poor performance. Even in that case, still able to capture many of the key cell types, the different immune subtypes and tumor microenvironments, right? So which means that now people actually, the kids makes it much easier for folks to be able to work with these spatial data, especially in settings where it's too hard or expensive to do the experiments. Maybe the last thing I'll mention is that we're very excited about these generative AI models, but I think we also need to be very clear-eyed about what are some of the challenges. One big challenge that we've been studying recently is we find that these generative AI approaches models are often very effective at basically amplifying whatever biases that we build into the model. An example of this is from a recent study that we did on looking at Stable Diffusion, and we also did the DALL-E type models. Basically, here, if you actually ask Stable Diffusion to generate images of what a software engineer or a terrorist would look like, these will be example images that the model would actually generate. You can actually see that the model actually learns a lot of stereotypes that are built into its generated images and generated data. So for example, I think 20% of software engineers actually self-identify to be females, but the output of the Stable Diffusion models basically produces essentially no female software engineers, even though they are present in the training set. So actually, that's one of the more fundamental challenges we see is that the generative models, they're not just reflecting the biases in the training data, they're actually very effective if we're not careful about amplifying the biases in the training data. It takes this 20% and then pushes it all the way to zero. That's actually one of the ongoing challenges that we and the community have to think about how to mitigate these challenges for using generative approaches in medicine and healthcare. Then I'll just summarize here. We're excited about using generative AI. I think They have a lot of potential applications in discovering new drugs, running new clinical trials, right? So both the beginning and the end stage of the entire clinical pipeline, and also things in between like discovering new biomarkers and new data types.

But of course, we need to also be careful, mindful of the potential limitations and biases in these models.