Scientist Stories: Anant Madabhushi, Using AI to Predict Immunotherapy Response

Thank you to everyone around the world for joining us for this week's Immunome Lab Meeting, and happy New Year. My name is Kristin Aboud. I am the science editor at the Human Immunome Project, and I will be your moderator today. Artificial intelligence was certainly on everyone's minds as 2022 drew to a close, particularly following the release of ChatGPT chatbot from OpenAI at the end of November. And while using AI to write poems or essays can be fun, this webinar series is focused on the overlap of machine learning and immunology in an effort to improve human health. If this is your first time joining us, welcome. We kicked off the Immunome Lab Meeting webinar series last year with two excellent talks. And if you missed those sessions, I encourage you to visit our website, humanimmunomeproject.org, to view the recordings. This week, we continue showcasing the research of leading scientists who are working at the frontiers of AI and human immunology with an exciting talk on how we can use artificial intelligence to better predict the response to cancer immunotherapy. Before we get started, I'd like you to please note that the information presented today includes pre-published data. At the request of our speaker, we ask that you do not reproduce or disseminate the data presented in today's webinar in any way. Thank you so much for your understanding. And now without further ado, it is my great pleasure to introduce today's speaker, Dr. Anant Madhubushi, a professor of biomedical engineering and a faculty member in the departments of pathology, biomedical informatics, and radiology and imaging sciences at Emory University. Dr. Madabushi is also a research health scientist at the Atlanta Veterans Administration Medical Center. He has authored more than 450 peer-reviewed publications and has more than 100 patents either issued or pending in the areas of artificial intelligence, computer-aided diagnoses, and computer vision, among other fields. His work on smart imaging computers for identifying lung cancer patients who need chemotherapy was named one of the top 10 medical breakthroughs of 2018 by Prevention Magazine. And in 2019, Nature hailed him as one of 5 scientists developing offbeat and innovative approaches for cancer research. During today's presentation, I ask that you please send any questions using the Q&A function in Zoom. I will ask our speaker a broad selection of your questions after the presentation,, and we will have some time for discussion. With that, it is my pleasure to now turn it over to Dr. Anant Madhubushi.

Thank you very much, Kristen, for that very kind introduction. All right, great. So just want to start off with a disclosure before I go into my formal disclosures. I'm not an immunologist. I know very, very little about immunology. But I've had the pleasure of working with a lot of people who do know a lot more about cancer immunology. And so this work really represents some of the work that we've done in the application of AI to cancer immunology, but more, I would propose sort of an aspirational viewpoint of what could be or what can be with AI as we look at cancer immunology. And hopefully, I'll— be able to convey and suggest opportunities outside cancer as well. So to the more formal disclosures, I am a co-founder and equity holder in Picture Health, a company that has licensed some of the technologies I'll be talking about today. I also serve on scientific advisory boards for a few companies. And we have a number of sponsored research agreements with pharmaceutical companies. I guess my true disclosure is the fact that I do have a social media account. That's my Twitter handle. For those of you on Twitter, if any of what I talk about today is remotely appealing, please feel free to give me a follow. So I want to set the context a little bit for my talk with reproducing some statistics that I'm sure a number of you are aware of. I was pretty stunned a few years ago when I found out that the probability of a man in the United States being diagnosed with some form of cancer in his lifetime is 50%. For women, it's marginally better with a statistic of 1 in 3, which effectively means that 40% of the adult population in the United States would be diagnosed with some form of cancer in their lifetime. However, when you look at the actual mortality statistics on account of cancer, and not to belittle or trivialize 600,000 deaths per year on account of some form of cancer, there clearly is a discordance between 600,000 people dying because of the disease or diseases of cancer every year vis-à-vis this diagnostic incidence rate of about 40% in the adult population. And of course, there are certain reasons why there is such a large discrepancy between these two numbers. One, the fact that we've become more aggressive about early screening. We are using imaging as well as biomarkers for early screening and early detection.. And we know that early detection of cancer helps save lives. We do know that the use of better treatments that we have in our arsenal today has dramatically improved the survivorship of a number of cancer patients, treatments like immunotherapy that I'll talk about today. But if you're completely honest, we have to acknowledge that at least part of the reason for the schism between the mortality vis-à-vis the actual diagnostic incidence is the fact that we are overdiagnosing and overtreating a number of cancers today. And perhaps a great example of this is really prostate cancer, a disease that affects 1 in 7 men in the United States annually, but only 1 in 40 will actually succumb to it. And yet we throw the proverbial kitchen sink in terms of treatment for these men with prostate cancer. We treat them with surgery, we treat them with radiation therapy, treatments that are invasive, that do have toxic side effects. Associated with them. And this is in spite of the fact that the vast majority of men with prostate cancer will die with prostate cancer as opposed to on account of prostate cancer. If we're being completely honest, we have to acknowledge that at least in the United States, apart from the toxicity on account of the treatments, there is also the very real issue of financial toxicity. These are some staggering statistics that are slightly dated now. I suspect strongly that The numbers are even worse today in 2023. But from 2018, we know that 42% of new cancer patients will lose their life savings. A number of cancer patients will lose their life savings within 1 year of that initial cancer diagnosis. And driven by the patient-centric toxicity as well as the financial toxicity, in our group, we've been looking at the use of artificial intelligence with routinely acquired data. I'll talk about radiology scans, I'll talk about pathology scans to not just help in better diagnostics, not just in helping identifying the presence of our absence of disease, but really looking at ways in which we can risk stratify disease better. Can we predict disease outcome and progression better? And then more critically from a clinical perspective, what can we do in terms of predicting response to treatment and benefit of treatment? Because at the end of the day, it's really the use of these AI-driven prognostic predictive tools that will truly help us realize the promise of precision medicine. One of the things that has become abundantly clear over the last couple of decades is the fact that there is a tremendous amount of information that is latent in routinely acquired biopsy slides, H&E slides. And with the digitization of glass slides, it has become possible because of whole slide scanners That has ushered in a true revolution in AI for digital pathology. And what that has really allowed scientists like myself and others to start to do is to use machine vision, machine learning algorithms, and start to apply to digital images of H&E glass slides to be able to start to pull out information that goes above and beyond what a pathologist or a human reader might be able to discern from these slides. And so here you see an example of a routine H&E slide of cancer. And with the use of machine vision, one can start to identify the individual lymphocytes and start to look at the spatial architecture and spatial arrangement of the individual immune cells. One can start to look at the textural appearance of the different tissue compartments, looking at the stroma, looking at the epithelium, and also starting to analyze the textural and the heterogeneity of the individual nuclei, really capturing the chromatin patterns within the individual nuclei using textural and mathematical features, really providing a series of quantitative metrics that now allow for a very deep digital description of tumor morphology in a hitherto unprecedented fashion. And so about 15 years ago, I had the good fortune of meeting and collaborating with Dr. Sridhar Ganeshan, a breast oncologist at Rutgers University. And Sridhar was looking at some of the algorithms we were developing in the context of H&E slides for cancer grading.. And she, an oncologist, looked at the work and said, you know, Nandan, it's very impressive what you're doing, but, you know, as an oncologist, my pain point is knowing how to treat people in breast cancer, where a number of breast cancer patients end up receiving chemotherapy where they actually will not benefit from chemotherapy, or they don't need the chemotherapy. We need better ways of being able to identify which women can avoid, safely avoid chemotherapy vis-à-vis those women that actually will benefit from the chemotherapy. And so we came up with this idea of the image-based risk score, where the idea was that taking the biopsies and slides associated with the tissue, that could then be digitized using whole slide scanners. And having digitized those slides, those images could then be analyzed by machine learning, machine vision algorithms. And a series of quantitative features could then be prized out from these slides. Which could then be associated with treatment benefit and treatment response. Now, one of the things I also became aware of was that there were molecular assays out there to try to address this particular issue. Oncotype DX is probably the most famous of these, a 21-gene expression-based assay to help identify which early-stage ER-positive breast cancer patients would benefit from adjuvant chemotherapy versus women who could safely avoid adjuvant chemotherapy. The challenge with these gene expression-based assays is the fact that they're expensive, they're tissue destructive, and they need physical shipping of the tissue blocks to a centralized location. And so our idea was, well, could we start to interrogate these slides with machine learning algorithms so that now you could do this image assessment of risk and benefit of therapy in a completely digital fashion? In other words, you wouldn't require physical shipping of tissue, you wouldn't need to destroy any tissue, you could do this in a non-destructive fashion. And that was the big sort of opportunity that we saw and started to work on. Now, one of the first things that we recognized that we needed to do was to go in and identify the cells of interest, lymphocytes, cancer nuclei. And around 10 years ago, deep learning, which of course uses neural networks, a concept that's been around for over 60 years, the increasing sophistication in computational hardware as well as algorithms, neural networks now could be tethered together in different unique innovative ways. And the way these neural networks were combined together and stood up resulted in different architectures, different networks. And this is sort of the burgeoning area or field of deep learning where you've got all of these neural networks stacked on top of each other. And it provides a really unique unsupervised manner of generating representations associated with the target of interest. So you see here an example of work that we published now about nearly a decade ago where one of the first applications of deep learning to digital pathology we trained a particular kind of deep learning algorithm called a stacked sparse autoencoder, provided a series of images where we annotated the individual cells. The deep learning network was then able to identify representations associated with these cells and was then able to spit out the precise location of these cells on new unseen images. So the beauty of this particular category of approaches is the fact that you don't really need a lot of domain knowledge. You don't really need to know the pathobiology of the disease, but having the annotations of the target of interest and providing a discovery or a training set to the network, the network then is able to learn in a largely unsupervised fashion. And these algorithms now become pretty efficient and relatively fast. And here's an example from some of our work on just off-the-shelf hardware where we are able to go in and identify hundreds of thousands of lymphocytes on these H&E images fairly rapidly. And this has allowed for phenotyping of these images in a very quick, rigorous manner. Now, having said that, one of the challenges with deep learning, which of course is a particular category of machine learning algorithms, is the opacity of the representations that are prized out by these networks. So the network learns a representation associated with the target of interest, but what it learns tends to be fairly abstract. It tends to be fairly opaque. And hence, a lot of these approaches often are called black box approaches. They become very widespread and extremely powerful and very useful for a number of domains. But there is a recognition, particularly in medicine, particularly in the biomedical sciences, that there is a challenge in being able to really understand how this AI works, how deep learning works, what these representations mean. And particularly when it applies to medicine and health, there is a challenge because how do you trust an algorithm where you don't really understand the representations? How do you really trust the decisions being made by this more black box, opaque class of deep learning algorithms? And so in our group, we've tried to sort of have the proverbial cake as well as eat it. Because we're looking at a blend of deep learning but also what are called handcrafted machine learning approaches where we're using the power of deep learning to do a deep phenotyping of the image. In other words, using the power of deep learning to segment out the structures of interest, the different tissue compartments, the individual cells, the lymphocytes, the cancer nuclei, etc., etc. But having done that, what we are now trying to do is to look at more interpretable patterns and interpretable features that can be then priced out from these images that really capture the morphology, but does it in an interpretable and intuitive way. So for instance, having segmented out all the individual nuclei and lymphocytes within the stroma, within the epithelium, now we can start to look at graph and network-based approaches to look at the spatial architecture and the spatial arrangement and the interplay of lymphocytes and cancer cells. You can do that in the stroma, you can do that in the epithelium, and essentially create these digital biomarkers that can then be associated with outcome and treatment response. So it's melding the power of the more opaque black box approaches to do the segmentation, but having done that, we're then extracting the more interpretable features. And so what I'll really focus on for my talk today are some exemplars of this category of more interpretable intuitive deep learning or, sorry, interpretable machine learning and machine vision-based approaches. And I'll go through some examples, but what we've shown in our work is that this class of deep phenotyping, which is a meld of the deep learning and the interpretable handcrafted machine learning approaches, it can be really beneficial across multiple different indications to predict treatment response. And I won't go through all the various different combinations that you see on the slide here, but we've really demonstrated across a plurality of different indications that you can see on the left across multiple different drug groups and drug types. And you can see that here as well as associated combinations to the right that this class of approaches applying it to just H&E images can really have a significant impact on treatment response and treatment benefit. And so I'll just pepper you with a few examples across these different conditions. Over the next few slides. So I want to start off with a breast cancer example that I referenced a couple of slides ago, where we took a more interpretable approach. First, again, using the power of deep learning to go and phenotype the image, identify individual cancer cells, identify tubules, identify mitotic figures, but then really creating an interpretable measure of nuclear pleomorphism, tubule count, and mitotic count. So essentially, by first doing the segmentation, we're then able to come up with a metric that a pathologist, an oncologist is able to understand. What is the diversity in the shape and the appearance of the individual nuclei? What is the density of the tubules in these slides? What is the density of mitotic count in these slides? We're then able to take these measurements and use it in conjunction with a machine classifier to now prognosticate outcome in ER-positive, estrogen receptor-positive breast cancers. And so this is work that's currently under review, but we've demonstrated on a plurality of sites both in the US as well as in India that this very interpretable machine classifier allows for prognosticating outcome both in lymph node-negative ER-positive breast cancer patients but also lymph node-positive ER-positive breast cancer patients. And why? That's beneficial is because again, we, you know, I talked about the Oncotype DX test, which is this molecular-based test that's tissue-destructive and has been validated in its benefit of chemotherapy for women with breast cancer. The Oncotype, the way it works is that it provides a risk score. If you have a low risk score, essentially suggests that you're not going to benefit from chemotherapy, you can avoid chemotherapy. On the other hand, if you have a high risk score,, it means that you probably will need the chemotherapy. Well, what we did was we looked at a cohort of patients who had had the Oncotype score and then applied the image-based risk score for further risk categorization within those buckets. So in other words, we looked at patients who had a low Oncotype score, applied the image-based risk score, and found interestingly that even within patients who had a low Oncotype score, there were a subset of patients that had poor outcomes, suggesting that in hindsight, these women who the IBRIS classifier identified as having high risk could have benefited from adjuvant chemotherapy. Of course, this was retrospective, and so therefore, in hindsight, these women might have benefited from chemotherapy. But what we also found is that patients with high Oncotype, again, women that Oncotype is saying should be receiving chemotherapy, we were able to identify a subset of patients that actually had really good outcome, had favorable survival characteristics, again suggesting that in hindsight, perhaps they did not need to have been offered or given chemotherapy and they could have possibly benefited just from surgery and hormonal therapy alone. So an example of where a simple image classifier working off H&E slides can add significant value to this more expensive molecular-based test. One of the critical things that we've recognized is the importance of validating these approaches on completed clinical trials. And we're working very closely with a number of cooperative groups. And this is some data with SWOG S8814 that was recently presented at the San Antonio Breast Cancer Meeting, where we demonstrated that the image classifier could significantly stratify the low-risk and the high-risk groups And SWOG S8814 was a completed clinical trial of ER-positive, lymph node-positive breast cancer patients. But it's not just invasive ductal, invasive breast cancer. DCIS is the so-called premalignancy or pre-invasive breast cancer. And here, using machine learning, we've been able to capture the immune cells on standard H&E images and essentially shown that with AI and machine learning, we can stratify DCIS into a high-tilt group as well as a low-tilt group. And what is really interesting in this data and in this publication that we're about to submit very soon for publication is that we showed not only that there was significant survival differences and risk of progression to invasive ductal carcinoma, between the high-tail group, that is those women with a larger number of lymphocytes versus women with a smaller number or lower number of lymphocytes, what we also were able to demonstrate was that there was a treatment benefit between the high-tail group and the low-tail group. In other words, we found there was significant differences in the benefit to women who had high-tail and low-tail. And that potentially women who had high tails may not really benefit from radiation therapy. And that's a big deal because that suggests that women with this particular tail phenotype in DCIS could potentially safely avoid radiation therapy. Nice thing here is again, we actually were able to work with a cooperative group in the UK and validate retrospectively this particular signature on this completed clinical trial and so demonstrate not just the prognostic benefit of this immune signature, but also the predictive benefit in terms of radiotherapy. So you can look at the TIL count and you can quantify high TILs and low TILs on an H&E image, but can you do more than that? And this was a question that we explored a few years ago. Hermann Corredor, an instructor in our group, demonstrated very nicely in a paper in Clinical Cancer Research that it wasn't just the number of TILs, but really the spatial interplay between TILs and cancer nuclei that was strongly associated with recurrence in early-stage lung cancer. And so what we did here was to use again machine learning and deep learning to identify cancer cells and lymphocytes on standard H&E images. So what you're seeing here are not quantitative immunofluorescence or immunohistochemistry images. These are standard H&E images. But the machine learning algorithm has been able to identify exactly where the cancer cells are. You can see the green dots— sorry, the red dots represent the cancer cells. The green dots represent the tumor-infiltrating lymphocytes. And having identified those individual cells, we then use clustering algorithms to create clusters of the tumor-infiltrating lymphocytes. These are the green clusters., as well as create clusters of the cancer nuclei. And these are the red clusters. And what you see here is that the spatial interplay between the green and the red clusters are quite different between patients that don't have recurrence vis-à-vis patients who do have recurrence. One of the striking things that you notice is that the lymphocytic clusters tend to be fairly large and sort of almost provide a mothering effect or an enveloping effect around the cancer cell clusters, almost suggesting that they're preventing the breakout of these cancer cells and therefore suppressing metastasis and progression. On the other hand, when you look at patients who do have recurrence, you're seeing larger dominant cancer cell clusters and smaller fragmented TIL clusters, which don't seem to have quite the same sort of mothering effect around the cancer cell clusters. And we showed that these spatial statistics, quantitative statistics that capture this interplay, was strongly associated with risk of recurrence in early-stage non-small cell lung cancer. In a follow-on paper to this study that we published last year, we wanted to further understand whether this TIL signature, the spatial TIL signature, was consistent across different subtypes of lung cancer. So we looked both at adenocarcinomas as well as squamous cell cancers. And working with Kurt Shalper and David Rimm and others at Yale University, we looked at a series of squamous cell cancers and adenocarcinomas. We looked at the H&E, but we also had corresponding quantitative immunofluorescence images, so we could do the immune subtyping of these immune cells on the corresponding QMIF. And we did a tight coregistration between the H&E images and the QMIF slides. And what we were able to do in this particular study was first to phenotype, the arrangement of the immune cells on the adenocarcinomas as well as squamous cell cancers. And one of the things that we found was that till density turned out to be more important and more prognostically relevant in adenocarcinomas, architecture and interplay between cancer cells and immune cells seem to be more important in the context of squamous cell cancers. Having co-registered this with the quantitative immunofluorescence images, we found that in terms of immune cell composition, we found that there was a higher density of CD4-positive cells in adenocarcinomas versus squamous cell cancers, where we saw that there was a higher expression of both CD4-positive as well as CD8-positive cells. So the nice thing here was we didn't just stop at the H&E. We actually took it further and connected it with the quantitative immunofluorescence, so providing more sort of insight into the immune subtypes. That we were capturing from the H&E images. So I talked about the prognostic benefit in the last couple of slides. And in a follow-on paper to the clinical cancer research paper was this paper that we published in Lancet E Biomedicine a couple of years ago where we demonstrated that the spatial till architecture was not just prognostic but also predictive for added benefit of adjuvant chemotherapy in early-stage non-small cell lung cancer. So just like I showed with the work that we've done in breast cancer, we demonstrated that in early-stage non-small cell lung cancer that the spatial TIL motif could provide benefit or could help identify those lung cancer patients that would truly receive added benefit from adjuvant chemotherapy. So looking at stage 1A, 1B, and some stage 2 patients, we were able to demonstrate in a retrospective trial, in a retrospective study, that patients who had received surgery alone, that we could identify a subset based on the H&E images that there were a subset of patients who were going to have poorer outcome and therefore in hindsight could have benefited from adjuvant chemotherapy. And then looking at patients who got surgery plus chemotherapy, looking back, we were able to identify a subset of patients that had much better outcome, much better survival advantage, and therefore in hindsight could have possibly avoided the adjuvant chemotherapy. In a more recent paper, this actually just came out over Thanksgiving, we looked at, or actually last summer, in Science Advances, we took the spatial TIL signature to try to see whether we could predict response to immunotherapy. This was a relatively small study, about 220 patients, but we did have data from 4 or 5 sites. And in all of these, we had biopsies corresponding to lung cancer patients who went on to receive immunotherapy. They received a variety of different immunotherapy drugs, including atezolizumab, pembrolizumab, as well as nivolumab. In every case, we digitized the images, looked at the spatial arrangement and spatial architecture of the immune cells, and showed a couple of things. One, that we could predict response to immunotherapy based off the spatial TILT signature from the baseline biopsies, and that this signature turned out to be fairly robust and consistent across different sites. But perhaps even more pleasing was the fact that we we were able to demonstrate the association with overall survival across different IO drugs. So looking at Pembrolizumab, Nivo, as well as Atezo. Again, relatively small numbers here, but even these small numbers you're seeing, we could separate out patients who had more favorable survival versus poorer survival across these different drug and IO agents. One other piece that we found that was really interesting in this particular study from last year was that when we looked at patients who had low PD-L1, so now these are patients who typically are either not offered immunotherapy or if they're offered immunotherapy, it's typically in combination with chemo, that we were able to risk stratify two distinct groups within the low PD-L1 setting. So in other words, we were able to identify a subset of patients that were low risk and a subset of patients that were high risk and showed very clear separation between these two risk groups. And the benefit of that is the fact that in the low risk group, these are potentially patients who even though they're low PD-L1 could possibly be offered mono-IO therapy. So in other words, it could potentially forego the more toxic chemotherapy even though they're low PD-L1. A few more examples from our work in the context of this sort of immune characterization for cancer therapy. This is work that's not been published yet, but it's a variant on the spatial architecture of TILs that I previously showed, where we've actually been able to identify individual clusters of the immune lymphocytes. Now we know, for instance, that not all lymphocytes are the same. You've got the activated lymphocytes, You got the exhausted cells, you got the bystander cells, and the idea here on this particular project was, could we identify on the HMEs alone these different clusters that might have different levels of activation? And what we were able to do was using features related to the appearance, the shape, the size of the, of the tails, excuse me, on the H&E images, we were able to identify individual clusters. We were able to link specific clusters with outcomes. In other words, instead of using all the lymphocytes, we were able to identify partitions and clusters of lymphocytes that were more strongly associated with outcome in lung cancer patients. And in some very initial experiments where we've again looked at the association between these clusters with the corresponding immune clusters on quantitative immunofluorescence images, we've identified that what we seem to be picking up are the more activated cells, whereas the sort of the bystander cells, the exhausted cells, immune cells tend to sort of form their own distinct cluster. And we've demonstrated not just the prognostic benefit, but also predictive benefit. And this was a collaboration with Bristol Myers Squibb where we've shown that this signature seems to be associated with overall survival in lung cancer patients treated with nivolumab. So this was the CHECKMATE-057 trial where the signature has now been validated on. Just wanna keep moving along. This is work that we published last year with Gerd Schalper and his group where we now actually switched from looking just in H&E to quantitative immunofluorescence. Essentially what the idea here was to try to come up with metrics of heterogeneity of the different immune cell types. So we looked at the CD4 positive, CD8 positive, CD20, as well as the tumor cells, and came up with a way of capturing the disorder or heterogeneity of the immune composition and the tumor cell composition, not on the H&E, but directly on the quantitative immunofluorescence. And so we used a spatial statistic called the Rao's index to capture the diversity of the immune cells and the species diversity and showed that there was a strong association between the Rao index, which captures this diversity of the different immune cell types, with outcome in patients treated with immunotherapy. So the question then was, could you try to recreate in an automated way what a quantitative immunofluorescence image might look like. And this is unpublished data, but just want to share with you some, again, aspirational views of what could be with artificial intelligence. This is work that's being led by Chris Barretta, a graduate student in my group, who's trying to train an AI algorithm to try to predict what a CD4 positive, a CD8 positive, cell might look like directly from an H&E. What we're doing is a careful co-registration of H&E images and quantitative immunofluorescence. So we're able to map the labels from the quantitative immunofluorescence images directly onto the H&E images. And now on a cell-by-cell basis, we've got the ability to identify what that cell— what a CD4 or CD8 positive cell looks like on the H&E.. And so we're using this to train the algorithm to directly predict immune cell composition from an H&E without necessarily having the corresponding quantitative immunofluorescence image. Not quite there yet, don't have too much data to show, but just wanted to again show you what this could look like and what the opportunities are in cancer immunology with AI. All right, a few other examples and other indications. This is work that we published last year in JNCI where we demonstrated that the spatial TIL architecture was associated with outcome in the context of head and neck cancers. We've also demonstrated not just in the context of lung cancer, but that the spatial TIL architecture signature that we had was also associated with response to immunotherapy in gynecologic cancers. This is work that we published last year. Showed again that there was an association in patients who had been treated with nivolumab in the context of endometrial, ovarian, as well as cervical cancer. There was an association between this spatial tail signature and progression-free survival. One of the other questions that we've been asking is, now, are there potentially morphologic differences in the immune signature between different populations? And this is work that was presented at the ASCO meeting a couple of years ago. A manuscript based on this is about to go out. And what was the question that we were trying to understand was, given the fact that African-American women tend to be diagnosed with endometrial cancer and uterine cancer at a rate that's about 3 times observed in Caucasian American women, are there potentially morphologic differences between these populations? And while I will be the first to submit that race is a sociopolitical legal construct, the use of race is helpful in terms of defining populations. And so even though we use the construct of race to define populations, what was interesting in the study was we found that there were differences in the immune cell architecture and the TIL architecture between African-American and Caucasian-American women. That in turn allowed us to create more tailored models for predicting outcome specifically in African-American women. We actually showed that if you used a TIL signature that was population agnostic, it didn't actually prognosticate risk of recurrence in African-American women. However, if you created a population-specific model for African-American women based on that particular TIL signature, then that actually turned out to be highly prognostic of risk of recurrence in African-American women.. So this is something that we are continuing to explore, but it suggests that AI might need to get more nuanced in the kind of differences that might manifest across different populations. And it's not just oncology. So I want to give you a quick example. This is work we've been doing with Elliot Paister and Ken Margalis at the University of Pennsylvania. This was published a couple of years ago in the European Heart Journal, where we showed on 2,300 patients who had undergone endomyocardial biopsies for cardiac transplants, that the spatial arrangement of the immune cells with respect to cardiomyocytes was strongly implicated in the risk of rejection. We actually showed that this spatial interplay between the immune cells and cardiomyocytes actually allowed for prediction of cardiac rejection grade in a way that was actually better or superior compared to cardiac pathologists. So very, very promising findings in terms of patients undergoing cardiac transplant and shows further the applications of this kind of approach outside just oncology. And this is work that a couple of years ago, I think just like everybody else, we were working on COVID. And we looked at just trying to understand whether with AI, with machine vision, we could try to characterize just from H&E images what the immune architecture looked like. And so we looked at a series of patients who died because of H1N1 versus COVID-19 and just compared quantitatively the immune architecture and the spatial aggregates of immune cells from autopsy images or images of patients who have been autopsied and found that there were significant differences. The H1N1 patients tended to have larger aggregates of lymphocytes. Compared to COVID-19 patients. So in the last few minutes, I wanted to very quickly transition over from pathology to radiology. And I think the title of my talk was AI and cancer immunotherapy across length scales. And so we've spent a lot of time looking at tissue and looking at immune cells at a microscopic scale, but I wanted to move out or zoom out and look at the macro length scale with radiology images. One of the big opportunities with radiology images is that there is macro-level textures that can still be very beneficial in being able to distinguish different similar-appearing pathologies. And here's an example of similar-appearing pathologies. One of the projects that we've been working on, which is funded through an NCI R01 with Dr. Pallavi Tiwari at the University of Wisconsin-Madison, is trying to distinguish between radiation necrosis and recurrent tumors in the context of patients with GBM or glioblastoma multiforme. And the challenge is that these patients undergo chemoradiation, they get an MRI scan, and when they get that follow-up MRI scan, something shows up on the scan, and it's not immediately clear whether one is looking at radiation necrosis or one is looking at tumor recurrence. And so we developed an approach called Collage, and essentially the idea was to analyze the MRI scans and identify the dominant edge at every voxel or every pixel in the image. Having done that, we could then compute a quantitative measure of the disorder with respect to these gradient orientations. In other words, we were able to capture a measure of entropy associated with these gradient orientations. And if you recall your high school physics, entropy really captures the degree of disorder in the system. So if you see a high level of entropy, it essentially reflects a great deal of disorder in these gradient orientations. That is, they're discordant with respect to each other. On the other hand, if you see a lower level of entropy, it reflects a more contiguity, more homogeneity in terms of the orientation. And so when we applied the entropy measurements on top of these gradient vectors, we actually were able to generate color maps like this, where the left panel panel, you can see, has a lot more entropy, a lot more heterogeneity. The right panel has a lot less entropy, a lot more homogeneity. And of course, it turns out that the more entropy, the more heterogeneity is associated with tumor recurrence, and the radiation necrosis is associated with a lower degree of heterogeneity. We've taken this approach and applied it to prognosticating outcome and also predicting benefit of therapy. Therapy in early-stage non-small cell lung cancer. What we found is that this entropy feature is really implicated with treatment response and treatment benefit, not just within the tumor, but in fact outside the tumor in the peritumoral environment. And in a publication from a couple of years ago, we showed that the immune sort of milieu around the tumor as captured by these entropy features allowed us to not just prognosticate recurrence, but really specifically identify patients who would truly benefit from adjuvant chemotherapy. This concept of entropy and looking at the peritumoral milieu, we found, has had implications in multiple other domains as well. This is work that we published a few years ago in the context of breast cancer patients who were receiving neoadjuvant anthracycline therapy and showed that the entropy features around the tumor, again, were more strongly associated with treatment response. So in other words, the higher the entropy, the more likely these patients were to respond to neoadjuvant anthracycline therapy. On the other hand, patients who tended to have much lower degree of entropy, the less likely they were to have pathologic complete response. To neoadjuvant anthracycline therapy. So the logical question is, what is going on outside the tumor? What are these entropy features capturing? And so in a paper from a few years ago in JAMA Network Open, we actually tried to study that question. We got a series of peritumoral biopsies in these women who were undergoing chemotherapy for breast cancer, got the biopsies, digitized the biopsies, and interrogated the immune milieu. And what was really stunning was that the entropy features that we were capturing from an MRI scan, right, this is a completely different scale compared to the biopsies. The entropy features tended to mirror the spatial architecture of the tumor-infiltrating lymphocytes on the biopsies. This was really, really interesting because what we're seeing here is that at the macro length scale on the MRI, the entropy patterns are being modulated by what's happening at a microscopic level. So that immune infiltrate and the architecture of the immune infiltrate seems to really have been modulating the entropy patterns and the radiomic textural patterns that we're capturing from the MRI. So again, sort of this crosstalk between the different length scales was really interesting. And it, again, sort of gets towards trying to open up the black box, really trying to get at a more deeper underpinning and understanding of what these features mean. We've taken this entropy idea and applied it to a series of other indications. This is a study from a few years ago where we looked at patients treated with immunotherapy across multiple different sites. It was a multi-site, multi-agent study, very similar to the study that we published last year of the biopsies. In this case, we were looking at CT scans. Of patients and showed that the entropy features both of the tumor but also the peritumoral environment was strongly associated with response to immunotherapy. Like the previous study, we looked at atezolizumab, nivolumab, as well as pembrolizumab and showed a strong association between the entropy features and outcome and response. But also, just like with the breast cancer study, we showed an association with the lymphocytic infiltration patterns of the baseline biopsies. Again, with the focus, with the intent of really trying to get at interpretability of these radiomic features. In a follow-on paper from last year, we looked at patients undergoing combination therapy, specifically in the context of stage 3 non-resectable non-small cell lung cancer patients who get chemo radiation followed by durvalumab. And in this study, we actually showed that radiomic features, the entropy features of the tumor and the peritumoral environment from a baseline CT scan was strongly associated with outcome, in this case, progression-free survival in these patients. And what was really interesting was that that association held up independent of the PD-L1 expression. So regardless of whether we had low or high PD-L1 expression, you would see a real value add and risk stratification that came about on account of these radiomic features between the low-risk and the high-risk groups. I sort of keep coming back to the importance and the necessity of validating these findings on clinical trial datasets. And so just recently at the SITC meeting, we showcased the validation of the signature on CP1108. This was a collaboration with AstraZeneca where we got access in a blinded fashion to the scans from AstraZeneca. We ran our signature on the CT scans and demonstrated that this feature was associated with response as well as survival. And a publication based on this is being written up. We talked about entropy. We talked about texture patterns. But what about tumor-associated vasculature? Been very, very interested in looking to understand how tumor-associated vasculature is associated with treatment response. In a paper that we published last summer in Clinical Cancer Research, we showed that the twistedness of the tumor-associated vasculature, not just the amount of vasculature, the twistedness of the vasculature was associated with pathologic complete response in breast cancer patients undergoing neoadjuvant chemotherapy. In other words, the higher the degree of twistedness, the more likely it was that these patients were not going to respond to neoadjuvant chemotherapy. On the other hand, the smoother the vasculature, the more likely these patients were to respond to chemotherapy. In a paper that just came out in Science Advances over Thanksgiving, so just about 2 months ago, we showed that the twistedness of the tumor-associated vasculature was also implicated in response to checkpoint inhibitors. And so when we looked at patients who did or did not respond to immunotherapy, we found there were significant differences in the twistedness and the convolutedness of the tumor-associated vasculature. We found that patients who tend to respond had a much smoother vasculature compared to patients who didn't respond, who had a much more twisted, convoluted vasculature. And that not only was this twistedness associated with treatment response, it was also associated with overall survival and progression-free survival. In fact, there was also an association between the vessel twistedness and PD-L1 expression. And then finally, just to end, I talked about pathology, I talked about radiology, what about putting those together? And so one of the things that our group has been really focusing on, and this is an example from some work we're doing in small cell lung cancer, is how can we combine these feature patterns the immune milieu and immune architecture from the pathology images along with the entropy features both in the tumor as well as the peritumoral environment from the CT scans. Can we combine these features and show that we can better prognosticate outcome, better predict treatment response? So that's exactly where we've been focusing on. We've demonstrated that the combination actually is much stronger in terms of the signal and being able to predict outcome and treatment response. And we believe that we could really take this approach, this paradigm of combining RAD and PATH, and apply it to a whole plurality of different indications. So with that, I'd like to conclude. So hopefully what I've conveyed is that computational analytics with routine imaging can help address questions in cancer immunology, and I focus primarily on prognosis and predicting treatment response. Regardless of the choice of AI approach, I think there's No question that we need to establish multiscale disease associations. Really being able to establish the morphologic and molecular underpinning of these features is going to be critical. There's no question we have to extensively and rigorously validate these approaches, as well as evaluate the reproducibility of approaches and do that across multiple different populations. I want to finally just end by thanking the amazing team of scientists, students, postdocs, faculty colleagues that I have the pleasure and honor of working with.

And of course, just a quick shout out to the funding agencies that put food on the table. Thank you very much. Thank you so much, Dr. Manooshie. It's a tremendous amount of interesting work. So really appreciate your walking us through all that. We'll try to run quickly through some of the questions that have been coming in. And we have a couple of questions focused on, you know, things that you may have tried to improve the accuracy of prediction. And so the first one was whether you tried to combine image classifier with Oncotype DX to see whether that improved the accuracy.

Yeah, thank you for that question. We actually did do some of that work back in, I think, 2017 or 2018. We had an abstract at ASCO where we did exactly that. We combined the image classifier with Oncotype DX and actually demonstrated, particularly for the low Oncotype DX group, that we could— that the combination actually improved the performance of Oncotype alone by about 20%. In other words, we could identify 20% more truly low-risk patients who could successfully avoid adjuvant chemotherapy.

Thank you for that question. Yeah. Did you ever get phenotype of both innate and adaptive immune cells in TME and combine it with image classifier and see whether that would be more informative?

That's a great question, and I think that's exactly where we want to go. We have not yet done that phenotyping. We've just started to, as I showed, you know, really just very, very preliminary data with looking at sort of that, you know, immune phenotyping, looking at things like quantitative immunofluorescence. We just started a project now looking at, you know, spatial transcriptomics. So we'll start to get to that finer, more granular immune phenotyping and then start to combine it with other kinds of image features. And I think this is what I wanted to put out there. This, you know, we've really just scratched the surface. I think there's a tremendous opportunity to really bring in the more deeper granular sort of immunology information and really combine it with a lot of these features from standard imaging, standard data.

Yeah, yeah, absolutely. And I know you talked about the component of race. We had a question on whether You also looked at other patient characteristics like age, time of onset of cancer, the cancer diagnosis, sex, etc.?

Yeah, that's a great question. Yeah. So everything that we do, I didn't really have time to get into the details, but in every study we always do a very rigorous multivariate analysis to look at the codependency of clinical pathologic factors. And what was interesting in these studies, I mean, I showed the data with the uterine cancer.. But we have similar data in breast cancer. We have similar data in prostate cancer that we published on a few years ago in Clinical Cancer Research. These, these morphologic differences that we're seeing across populations, you know, turn out to be independently prognostic and independently associated with outcome in a multivariate setting. So we've tried to account the best we can for a number of those factors that were just mentioned. And found consistently that these morphologic differences persist even in that multivariable setting. So there seems to be something there. Of course, we've got to continue to explore. We've only found these differences morphologically. The question, of course, is molecularly, what does it mean? And we clearly need to get more sophisticated and use things like spatial transcriptomics and single cell to really probe, you know, the underpinnings of these differences.

Right. Okay.

We had a question about what the tissue was that you looked at for both COVID-19 and H1N1 patients. So those were patients who had been autopsied and we looked at lung tissue.

Lung tissue. Okay. So then we had a question from a PhD student who's interested in translational research and would like to know your thoughts on opportunities and challenges of integrating these high-end computational tools in the clinical settings of developing countries?

Great question. So we've, so I see my program officer from NCI is actually on the call, Dr. Didi Rao. So it gives me an opportunity to talk about a grant from the NCI that we recently funded on where Dr. Rao is actually the, is working with us. And this is a partnership with India, with the Tata Cancer Center. And specifically as part of this 5-year U01, we are working with the Tata Cancer Center to deploy these AI tools from routine pathology images to be able to create population-tailored models for breast cancer, oral cavity cancer, as well as prostate cancer. The main idea in low-middle-income countries in sort of resource-scarce settings is to be able to identify which patients truly will benefit from the treatment resources. Because, you know, there you're not just talking about patient toxicity, you're talking about a qualified amount of resources that are available. So you've got to make sure that the, that the patients who are truly going to benefit from the therapy are the ones who benefit from the therapy. And so very exciting project that we've just kicked off.

I hope in the coming days to be able to report on some of the initial findings from that study. That's fantastic. Well, thank you so much. I really— such fascinating and innovative work, and we really appreciate your taking the time to both present and then answer questions from our audience. Thank you also to all the attendees for participating in today's webinar and for submitting questions. We're so glad, as always, to have you join us. And we look forward to many more engaged discussions. I'll just let you know that the next Immunome Lab meeting will be on Wednesday, February 8th at 10:00 AM Eastern time. Please note it's a Wednesday, not a Thursday for next month. And we will be hosting that day Dr. Jian Tang from the Mila-Québec AI Institute. There's no need to register for the webinar in advance. Please just visit our website at humanimmunomeproject.org. Immunome.org to obtain the link. And while you're there, I invite you to please sign up for the newly launched Immunome Report, which is another platform for sharing the research and voices of scientists working at the frontiers of human immunology and AI. And finally, please do visit our website and follow us on social media where we will post a recording of today's webinar. Thank you again for participating today. And we look forward to seeing you next month at the Immunome Lab Meeting.

Thank you so much, Dr. Madabushi. Pleasure. Thank you.