Scientist Stories: Anirban Mukhopadhyay, Medical Computing

Welcome to a new NCT Data Science seminar. Today we will hear about how to reverse engineer the doctor's mind, presented by Anirban Mukhopadhyay. A warm welcome, and I also welcome Lena Meyerhein, which will give the introduction. Lena, the stage is yours.

Yeah, good morning everyone. It's our pleasure to have Anirban here. When we just met in the Zoom room, Doreen said, could you please help me pronouncing your last name? And he just, he said, oh well, just call me Anirban. So it's It's not an easy thing to do. Yeah, it's great to have you here. Anirban did his PhD at the University of Georgia. He did a postdoc in Italy and Berlin and now is an independent group leader at the Technical University of Darmstadt. His group name is interestingly the Medical and Environmental Computing Lab. Very interesting. And I personally know Anirban mainly Well, I think he always posts very interesting things on Twitter and also probably hosts the most popular podcast in the context of medical image computing, which is AI Ready Healthcare. You should maybe take the time to listen. It's very interesting. And yeah, we're very, very happy to have you here with this interesting title, Reverse Engineering the Doctor's Mind. Stage is yours.

Thank you for the very nice introduction and kind invitation, Lena. So yeah, I mean, I took the liberty to be rather provocative, and some might consider, I don't know, downright crazy with the title. Well, that might be true, but I duly believe that crazy things are still worth pursuing. So, but really, let's first try to understand why we are trying to do this reverse engineering. So this is really what the world looks like when you use each country's access to doctors as the area of the landmass. So that's Africa, those of you who are wondering. So that's really the problem that we have. So of course, this is data from 2004 by World Health Organization, but things didn't really change much from that time. And really, if you talk about the combination of economic recession, time and cost of training healthcare professionals and the tendency of healthcare professionals to be located in the major urban centers. So together you have this perfect scaling up problem that simply cannot be solved with the old approach of training more professionals. Now fortunately, health tech has already shown its potential in a broad range of application. You can see this sort of an EMBS poster of the different applications where healthcare can really help, where the health tech can help making better, better access to healthcare. But what also really happened is that during pandemic, we got an acceleration in terms of the trust in technology for access to care at the societal level. So we are more or less ready to envision an alternative world where like combining all pervasive mobile networks with AI can make basic diagnostic screening as well as maybe some of the simple surgical cares much more accessible to a global health context. But really, if we are talking about technical assistance, we can break it down into two types. One is the passive and another is the active assistance. So passive assistance would be basically sensing, providing numbers, this type of assistance, and what is really currently available. And if we are talking about the majority of the research that is going towards the explainable AI, human-in-the-loop AI, what we are trying to do is going from current situation of passive assistance to the active assistance at the local scale. Now, on the other side, we have this global scale of going from the local to really remote location. And pandemic has a clear understanding that we can use video conferencing-based telehealth for solving many of the basic problems. But really what we are relying on there is this old model of one-to-one consultation. But what we should be really talking about rather is sort of a model that is time parallel, and basically time parallel in the sense of one-to-many. And the easiest way to achieve this is by basically prioritization. So the idea what we are talking about here is to start off a framework where the patient history is taken, basic sensing is performed, most relevant information is prioritized and stored beforehand. And this can be done, for example, in collaboration with the unskilled or understaffed health workforce at the remote location.. But also we have an AI that is really clinic ready so that it ensures effective communication with the expert while best utilizing their time. The really, the alternative of going through one remote teleconsultation after another is like not really realistic as it would put more burden on an already stretched healthcare system. So let's take a look at one of the success stories of this one-to-many prioritization paradigm. And this comes from US Veterans Affairs diabetic foot ulcer framework. So what happened there is that basically diabetic foot ulcer is a problem that leads to amputation of the extremities. And that's really a major concern if you are not detecting this early. So this particular company, Podimetric, I don't have any association with them, I'm just using their example. What they came up with, they showed a multicenter study of their smart mat where they showed that you can correctly identify 97% of the observed diabetic foot ulcer on an average of lead time of 37 days before the typical procedure in which you diagnose. And this is really a super efficacy in terms of a multicenter trial. And they really are doing great, let's say, innovation in that space. But I really wanted to point out in their press release to the fact that this company's nursing team connects with the patient and the patient's provider in as close a real time as possible. So basically they have to send their nursing team on the physical location to actually bring things forward once the diagnosis has happened. Now, if we are really talking about global health, sending professionals to remote location has many logistical issues, often impossible to scale up if you are really talking about sending to them in real remote location. So a much better option is to come up with an AI that can augment the unskilled or understaffed health workforce at that remote location. And really, my vision is to build such an AI that would enable better recording of patient history, prioritization, and then efficient communication with the expert who is sitting somewhere else. And for successful collaboration, with such an export. Just an AI that recognizes patterns from images are probably not good enough. Along with pattern recognition capabilities, these sort of clinic-ready AIs should be intelligent enough to model the clinical environment where these are operating. And unlike supervised learning that are trained for this very particular task and often fails to generalize, we need clinical intelligence that is ready to perform neural complicated clinical rootlets. Now, while such clinic-ready AI might be attainable in many ways, no doubt, it is unlikely that doing more and more deep learning for pattern recognition will lead us there. I think reverse engineering the mind of a doctor is our best bet to attain clinic-ready AI, since we know for sure This is the only intelligence that eventually develops into a board-certified clinician. And I plan to realize this vision in 3 steps. So I would like to build an AI that starts like a medical intern, learns like a resident, and practices like a doctor. So how we are going to be doing about it. So Let's break it down, the rest of the talk, into these three distinct parts. So let's talk about starting like a medical intern. So when the medical internship starts, the doctors, they are looking at humans. They are not really seeing humans as a bunch of pixels, right? So they have a structural and functional understanding of the anatomies, their shapes, material properties, spatial configurations, all these things. So we really need to encode this domain knowledge as the inductive bias of our learning models. And this basically, not doing it has major consequence. I will give just one example from our research where we looked at a rather common inductive bias, shape as a constraint. And we will also talk about performance metric. This is something where I must thank Lena for bringing this attention into metric and how important it is simply because when we were doing it back in 2017, '18, we didn't have much reference, but we sort of had an understanding that there is something of a problem that we are facing. So I will tell you the example where we worked on, and this particular project happened in collaboration with Uniklinik Düsseldorf for inner ear surgeries. And we are trying to bring high precision robotic surgery for sort of cochlear implant or tumor removal. And when we are talking about such high-precision robotic surgery, it needs to be planned early from preoperative images, maybe CT, maybe MR. And the problem really is that unlike typical medical imaging setting, now it's getting more and more data intensive. We don't really have access to massive annotated data when we are doing surgery, surgical data science. And really what we have to still do is we still have to come up with segmentation models that generates an anatomically plausible segmentation on which we can do planning. So this basically brings us to the second problem of really functional evaluations. I think like in around 2017, '18, Lena's work was probably starting, but we didn't really have the reference that we right now have. But the intuition really was clear that surgeons don't care about Dice accuracy, but they do care whether they can perform a surgery while ensuring that they are not going to hurt the patient. Because for these high-precision surgeries, if you touch one of the knobs just to solve someone's hearing problem, that person might come out of the anesthesia and you see one side of the face has got paralyzed. So you really don't want that to happen. So our solution there was simply a shape-regularized U-Net, a rather naive approach, I would say, because at that time, of course, U-Net was already quite famous. Really see the problem of U-Net in such a thing that if you don't have a lot of training data, but you have to segment a lot of really tiny risk structures, you end up with such fragmented structures. Now you will still get pretty high dice accuracy, but it's simply not usable for planning. On the other hand, we had these old statistical shape models that typically generate anatomically plausible shapes, but they have initialization issues. So we sort of merged these two ideas to sort of start with the U-Net and then correct the segmentation with statistical shape model. But the different than typical approach of evaluating segmentation on segmentation accuracy, we actually evaluated it on the downstream tasks. And here the functional evaluation was about performing linear or nonlinear access for this particular different types of operations. So here you can see we are trying to reach a particular tumor through a nonlinear axis through this sort of semicircular canal. Whereas here we are trying to really not touch any of these nerves while trying to reach the cochlea. And the segmented model, like completely automatic planning with our approach generated the blue, which is rather similar to the green that the manually generated one that the surgeon considered is good enough. So yeah, so that was really a work pretty like around 2018 or so. And we got some community recognition, but we also realized at that point that this shape constraints bringing that in the segmentation framework is becoming sort of a norm. And there were quite a lot of works coming out there where people are trying to do either pre-hoc, post-hoc, or an end-to-end framework of bringing shape constraint into U-Net and other forms of deep learning. So we wrote a survey around that, and there it was already clear by 2020 that we are really seeing a shift towards more end-to-end models. And now we have basically a conference that is organized focusing only on geometric deep learning. I'm not organizing the conference, but just trying to see that the entire community has realized the benefit of bringing inductive bias into the learning models. The next part of the entire process is about the actual learning itself. And the learning of resident physician is far more complex than the typical static supervised training regime that we use for deep learning. One main difference is that they have a sort of situational awareness. So the residents acknowledge the dynamic environment where they are learning., and they ask for supervision when they detect a case that is out of their comfort zone. So at its core, the equivalent would be looking at a continual learning setup where we acknowledge the fact that we have an open-set dynamic environment. We can't predict where the future will be, and we have to really detect the drift and come up with robustness. But majority of the research that we see, for example, in a MECI kind of setting, is based on this sort out of this naive worldview that we will have all data at one central location and data characteristics will remain static over time. The only problem is that that's not the case simply because patient privacy reasons and staff mix. It's really difficult that you can't store all the data in one central location. And even if you store all data in central location, data distribution actually shifts, data characteristic drifts. And that really means the results that you are getting today might not reflect what you will get 1 year down the line. One example of it will be this clinical data that is dynamic. So for example, here is the ground truth of an MRI segmentation of the hippocampus when you are using the segmentation model trained on data that is this. Of its type. But as you see that often, like, as the data characteristic drifts, you see the segmentation model deteriorates. And we really have this problem that outdated models are unreliable. So we have to come up with a framework of learning which respects the dynamic reality of clinic, that data will arrive sequentially, old data won't be available forever, and then you have to still detect reliably when the model is not really certain anymore, and you have to start learning that. So this is the entire learning setup that we are trying to solve. Now, the first part of this technical challenge is to detect when the model is unreliable. If you are looking at this, you can detect at 3 different ways from 3 different sources. You can look at the out-of-distribution detection, at the sample level, you can use the uncertainty of the model itself, or you can look at the final product that is coming out of the model and try to assure certain quality of the model. We worked at all three of these directions, but for this talk, I will focus only on out-of-distribution detection part. I guess a lot of you are already familiar with what out-of-distribution is, but for those who are not, this is The mandatory cow picture from an Indian guy, I guess. So what you are looking at really is a model that is trained on cow pictures where the cows were in pastures or in green fields. It's fairly certain to detect cows that are in green fields in Alps maybe, or in Indian greens. But when the cow is for some reason at the beach, it certainly is not sure anymore and it it's detecting something else but not the cow itself. And this is something that a problem that appears also in medical imaging where deteriorating performance happens in conjunction with silent failures. So one of these examples would be our time, which is the pandemic time. And there basically the problem really is that Many COVID-19 lesion segmentation solutions exist, but there is also sort of a meta-understanding, the fact that majority of the research that we did around pandemic AI is basically sort of— is a lost opportunity simply because I guess there were many factors that were involved, but one of the problem was really external validation. So typically when we are talking about a setup like this in the modern time of, let's say, challenge, and then there, the idea here is that we have a very heterogeneous training data that we train with the awesome NNUnit, and the dataset is large and heterogeneous, so it should generalize. And the thing really is that that's in fact true. So I think, Klaus at one point tweeted that the competition they had for COVID-19 lesion segmentation, out of the top 10 performers, 9 of these were basically using NNUnit. So that's really the standard of the research. But the problem really is that when you are talking about external validation, the performance degrades, right? So here what we are seeing is that when you are validating the result on the challenge data, we are pretty good. But as we are going from challenge data to our in-house data to two other datasets which are publicly available, we see a degradation in performance. Now, the thing though is that out-of-distribution detection is always a secondary goal for any of these algorithms. So a lot of research that focuses on let's train our out-of-distribution model on data and show that it's a great out-of-distribution model. That's not really realistic in the sense that primarily we want a better segmenter. Out-of-distribution is just something to ensure that we are detecting when the models fail silently. But really, the problem is that outputs always show high confidence. But you might ask that, okay, but if it is really happening in a realistic setting with a person like a radiologist on board, That's not a problem, right? They can detect that the model is failing. It really is like the answer always is depends. So for example, if you are look at this, this is a prostate tumor. And when you are looking at the multi-parametric MRI of prostate tumor, which is the typical way they are detecting it, you can use some simple thresholding and get an understanding of where the tumor's. Unfortunately, COVID-19 was different. The lung lesions there had very complex geometric patterns. They were very different sizes. Often the telltale sign of COVID-19, which is this ground glass opacity that had overlaps with consolidations— we are involved in this Raccoon Project where we have seen that when the radiologists are finally talking about what ground what lung glass opacity versus what consolidation is. Often there is disagreement of how they are looking at this. So that really creates the major problem that you need really advanced radiologists to detect that the model is failing. It's not often very easy or intuitive to say, okay, now the model is failing, so we should not trust it anymore. Now, the traditional approach to out-of-distribution detection in this post-hoc manner without doing any training is to consider a probabilistic perspective perspective and really see the change in network outputs for different perturbations. What we did, we tried to look at rather at a distance, at the latent space perspective. And quite simply, if we are talking about it, so we have these sort of volumes that are being represented by an NN unit or a unit in the latent representation. And what we are really looking at it is that at latent representation how the points are located. And when a new dataset comes in, we are trying to see whether the dataset data point is far from the typical distribution that the network has seen during training. If the data is too far away from the typical distribution, then we have no reason to believe that the network will do well for this particular data point, simply because it has seen similar datasets before. And we really, how to understand how reliable this method is. So we basically looked at the multiple methods. I will immediately show the tables, et cetera, but let's try to understand these graphs for a second. So what we are looking at is a graph where we have in the X-axis, the uncertainty and the Y-axis is the performance of the segmentation in terms of DICE. So this green is basically the region where it's low in terms of uncertainty and high DICE. So it's all good. In the blue region, it's bad performance, but we don't really care too much simply because it's correctly flagged that bad performance. This red, however, is the like quadrant of silent failure where you basically are having low DICE, but the uncertainty is quite low as well. So you are quite certain that you are doing well even though you are actually doing bad. And what you are looking here is the second performing method in this, uh, in this setting for post-hoc Monte Carlo dropout. And you see quite a lot of data points here in this quadrant which is thrown away for our particular method. So we, we are pretty reliable in terms of detecting these silent failures. And then of course we have this massive dataset, a massive table with many other methods to compare that typically people use in the post-hoc setting. And we are showing that, okay, we are doing good. But of course we also have problems, right? The problem comes here from this yellow quadrant where we have good predictions, but we are simply not using it because we are saying that that, okay, the model is not very confident that it's doing well, even though it's actually doing well. So that's something where we really are underutilizing the model's performance. Let's take a look at some of these qualitative results. So just to show you here, it's a ground truth segmentation, and the model did pretty well in terms of the prediction. We have high dice, and we can see here the uncertainty for all methods is quite low. Now compared to another one, this is really tiny lesions where methods didn't really do well. You can see the dice is pretty low, but all these methods are giving rather similar numbers in terms of uncertainty, whereas ours is the only one that really gave a high number in terms of uncertainty. That is, don't rely on what we are talking about. You should you should really talk to a radiologist. So that's sort of the first part of actually detecting when the network is not doing well. But let's really talk about the second problem, which is medical continual learning. So now that's a really nascent field, and we have again two problems of medical continual learning. The first problem is that the algorithm itself— when you are trying to design an algorithm, here for continual learning, we have this sort of orthogonal directions in terms of performance, which are basically contradict with each other, and we have to develop an algorithm there. So the thing is that you can't really forget the old data point. So that becomes too plastic, that you are really focusing too much on the new data that you are getting and forgot how the old like how the old data looked like. On the other hand, you can't be too rigid, as in you have to still learn new information. You can't be too rigid to the old ways. You have to learn the new ways, and you have to do all these while maintaining a certain memory buffer and training times, etc. So that's really the problem that we have to deal with in terms of the algorithmic design. The second problem, of course, I guess, is pretty close to the other problem of validation. It's simply just that also along with the metric, the problem comes with the setting in which you are validating your method. And there, the problem really is that often continual learning in settings such as computer vision is validated with rigid task boundaries where we know about the task labels. This assumption is something like you expect somebody to tell you that now there is a data drift happened, so you start learning it. But that's rather unrealistic. Nobody in medical will ever tell you that there is clear-cut boundaries and that, like, even if there is a clear-cut boundary, there is no one to tell you. But often there are also drifts that happen continually over time because of the certain change in the small protocol of the MRIs and stuff like that. So really how performance degrades, Professor Suchi Saria, she did a lot of work in Johns Hopkins on the electronic health record. But we here are focusing on the image data itself. And here our idea is basically we are still trying to use the distance that we calculated in the feature space. To sort of have an understanding of the data drift. And we are basically accumulating the mean and covariance of the data as it's arriving over time and calculating the distance. We sort of have a certain threshold. If things go beyond that, then we say, okay, so now the data characteristic is different and we have to start the training again. And we basically showed that in this sort of elastic-plastic scenario, we are basically maintaining enough plasticity without really forgetting the old information as well. So it's a sort of, compared to the other methods that is out there, this is probably something that does quite well. What we didn't solve here is the fact that our model grows. We don't yet have a solution where we keep the model size similar. Yeah, and that brings us to the last part of really practicing like a doctor. So imagine for a second that a doctor who is practicing at a primary care center, he or she calls the colleague at a tertiary care university hospital to refer a patient. Now communication is rather crucial here. Most important information will be the soft signals about the diagnosis from the patient history that needs to be communicated to the patient, to the doctor who will look at the patient in this tertiary care. Let's take radiology as a case study simply because it's most advanced in terms of the digitization. And really, let's try to understand the information flow of this setup. So it's rather simple. So you have this radiology reading room where the radiologist is looking at the images. Scrolling it with one finger, uh, and dictating on the dictaphone from which you generate certain radiology report. The report goes to the both the patient and the doctor. Um, so what we are trying to do is we are trying to create a virtual radiology reading room, which is not really any different from the idea of teleradiology. The only thing we are trying to do here is trying to replace a tele-radiologist, the performance with the person who is performing the radiology, the reporting, the interpretation with an AI, just to train and practice the AI in such a closed model. So if you think of such a model with such an information flow, the success criteria here is rather obvious, right? It's the Turing test. So if those doctors who are receiving the reports can't see the difference of these reports between what is generated by a radiologist versus what is generated by a clinic-ready AI, we are— we can call the clinic-ready AI is fairly successful in imitating the radiologist performance. But really, of course, when we are talking about a learning setup, we have to have a feedback loop, right? So if we sort of like close the information flow loop in the radiology setup. What we really have here is the radiologist who is looking at the images, creating a report, and the doctor is looking at the report. He or she is prescribing things at the patient, and maybe at some later time they are getting some feedback about the patient. So this entire feedback is of course delayed for the radiologist. But we need really immediate feedback for training our models, right? So there we can think of an alternative information flow, which is of course simplistic, but rather keeps that essential. So the images go through the neural network, the neural network generates the report. That's the report that is read by the radiologist or other doctor who changes the report and the neural network learns continually. Or they say that's fine. And that's also an information, the feedback loop from which the neural network learns. Again, fairly simple, but we have a century-old radiology practice that is standing in our way. To understand the problem, let's talk a little bit about the history. So radiology sort of happened quite in parallel with photography. So around late 1800s, we had photography and radiology both happening, both happening in, in films. And then we took almost 100 years, and then we saw that both practices became digital. And with becoming digital, we had these softwares which are basically workflow management tools with superior manual image processing capabilities like Photoshop, the PACS systems, all of these appear. And then we saw some of the portability issues, some of these feedbacks in-camera AI, quality checking AI. These sort of things popped up. That's all fine. The only problem is that this is the focus where we are focusing the export, the photographer, the radiographer, the radiologist, but we have a massive consumer base, right? So now if we are talking about the consumer focus, uh, in both cases there are of course people, but in case of radiology, there is also other departments which are really using the radiology report to, to to make certain decisions. Now, if you are thinking of the people, uh, the product they consume from photography is photos. But for radiology, it's much more important to have the product which is the radiology report. Now, this is apparently the first ever radiology report written back in 1896. You can see that's a report that's like an unstructured text. It's almost like a letter written from one doctor to another. Now, this is what '22 report would look like. That's unstructured text written from one doctor to another. Of course, it's digitized, but not much else has changed. So if our technology hasn't changed for 100+ years, either it's perfect like our coffee cup, or there is a certain problem there, right? Now, if we are really talking about consumer focus, AI in photography that's completely changed how we consume photographs. You can talk about Instagram, Flickr, Google Photos, Facebook. We, we completely have a different understanding of how to consume photos, but we really don't have an equivalent thing for radiology. And that basically means we end up with these unstructured reports which are extremely difficult to interpret for a computer. So you really end up like all of those who— all of you who work with natural language processing for these reports, you would agree with me that you can infer many things and anything from these reports without getting the particular types of supervision, the feedback that you need to continually train your neural networks. So our solution is of course not perfect, but we just relied on structured report to solve the problem to complete the loop. And here, German Ministry of Health was really, really nice. They funded us with generous support so that what we could do is we collaborated with clinicians, but of course also with a company called Smart Reporting to close this information loop. So what we are doing here is that that we have these images that are interpreted by the neural network, and then we generate structured reports. But then we can also do is a prioritization, and this prioritized interpretation can go to the radiologist who is checking. In case they are happy, that's fine. Otherwise, they will make certain adjustments, and those adjustments will be stored in a system which is machine-readable so that the network itself can get the real feedback. So it's rather idealistic, I would agree. But what really this framework allows us is it allows us to ask all sorts of questions about the situational awareness of the AI that we are building, its communication capabilities, and the sort of unobvious ways in which things can go wrong when you deploy such a human-in-the-loop system. And I guess there are no equivalent ways of detecting how an AI model will perform when it's being deployed with a human in the loop if you are really building isolated AI systems. So that's really our focus. So we so far have built the sort of this vertical integration. We'll focus on asking the communication questions. We are a publicly funded research group, so we make our software frameworks publicly available. There are two examples. One is called Frodo. That's a long history of why we end up calling it Frodo, but there is Twitter post to check about. But basically, in simply what it's doing, it's a sort of, again, post-hoc framework. Framework to detect out-of-distribution data for whole slide images, and we are planning to extend it to other modalities as well. So also radiology, also maybe some of the endoscopic datasets. So that's really our hope. The other framework, again, very much thanks to Klaus for providing NNUnit, so we sort of piggybacked on it and came up with this framework which we call Lifelong NNUnit. And simply, the idea here is that we can develop continual learning framework as much as we want, but it's a real environment where people need to have certain performance assurance. If you are really building on continual learning and talk about forgetting or doing better where your basic performance is really low, then nobody cares, right? How good or bad you are doing continually. You are bad, so you are bad. So we took the state-of-the-art medical image segmentation framework and then built in the continual learning framework so that you have the assurance of the NNUnit performance. And then you are seeing how, if you are training things continually, how the performance would evolve in the later parts. So do check it out. Yeah. Basically, in summary, we are really talking about accelerating equity of access at the global scale, where we really need to reverse engineer the doctor's mind so that we have clinic-ready AI supporting the really understaffed, underskilled people at the remote location to better prioritize, take the history, communicate that with the people who are experts and sitting in the urban centers. And I proposed a 3-step plan for attaining the audacious goal. Yeah, this involves starting like a medical intern, learning like a resident, and practicing like a doctor. I also talked about the core technical challenges rather, and the performance metric that we can look at to see how good or bad we are doing. Yeah, I would have settled, I guess, with something more realistic for sure, unless I was surrounded by such an amazing group of PhD researchers. We also have many more students at the bachelor's level, master's level, interns who are working on the group, but I only had limited space in the slides, so only these people. Lena, at the introduction, mentioned, but I guess I will unashamedly talk about the the podcast that we do. It's called AI Ready Healthcare. Lena has already been there talking about the challenge research, not so much the metrics, but maybe I will invite you again for the metrics. And then yeah, I tweet regularly about other research and stuff that we are doing there. So yeah, thanks again for your attention, and I'm looking forward to an engaging discussion.

Thank you so much, Aníban. Yeah, so Doreen, actually you're doing the moderation, right? Sorry, I hand over to Doreen.

Yes, so you can ask your questions directly if you want to or just paste it in the chat. So we are happy to get them all answered today. So Lena, would you start with the first question?

Well, I have one very technical question, but one very broad question that just came into my mind initially when you showed the— this Africa slide initially, because I'm thinking about it all the time. I mean, we speak a lot about fairness and how to bring AI to other countries, but we also have this whole, you know, climate change problem. And I was wondering if you have thought about the implications for our research. You know, we train all these deep learning models, but we use a lot of energy for that. I mean, I know this is a bit of an out-of-distribution question now, given your talk, but I was really interested in your opinion because I know you think about such issues in general.

That's— yeah, that's indeed an out-of-distribution question. So I guess there is one particular researcher, Raghav, from Copenhagen, who is really into using the carbon footprint tech track, like tracking the carbon footprints of these, these models that we train. And I guess that's really an important issue to deal with. The other sort of climate change issue that I often think about is basically, again, fast triaging, bringing attention to things that has happened, uh, and like basically bringing fast, uh, access to at least the fact that, okay, somebody is in trouble. And I will give you one example, probably, that you will realize. So, uh, what happens here is that, for example, with the change of global temperature, we see a significant more number of morbidity and mortality in the Northern European population. And that's not gonna decrease anytime soon if the, if the things going on like this. So that way we really have to bring in some of the technical, let's say, advances in the way so that we have a faster diagnosis. Um, by doing that, will we make the climate more hot? I guess yes. Of course. But I guess we use technology to go from a bigger problem to a rather smaller problem, like from not having enough food to obesity. So that way, I would say probably it's, it's much more relevant that we bring attention to people who are now in need and really also think about, of course, how to make green AI. But I guess in context, I don't know if I answered your question properly.

But yeah. No, thank you. That made me the more technical question. So I appreciate that you also spoke about problems. So when you talked about the out-of-distribution, you spoke about those samples that we basically neglect because we think we cannot make a qualified decision. And I was wondering, so you took the decision to phrase your problem as an out-of-distribution problem. Did you also explore more, let's say, domain generalization techniques where you train the networks in the way that they become independent of maybe future changes? Have you looked into this yet? And if so, can you share your experience? Because I'm asking because we always get these beautiful latent representations when we do this, but then we don't really get gets a performance boost on the downstream task. I was wondering if you have similar experiences.

So thanks for the question. It's a very important question about the generalization. I guess we sort of— to answer the first question, did we look at it? Yes, we looked at it in two different directions. One is the sort of disentanglement approach, and the other one is the sort of, how to say, regularization-based approach. So the disentanglement approach is more about the content versus style, and we keep the content representation similar. We had, I would say, some early success depending on what sort of data and problem you are talking about. But really, I won't call it how to say, a major success. The bigger— but that's not really the big problem. The big problem really is that the other direction where we are using the regularization, so invariant risk minimization and methods like that. So we were very interested in that, and then I put one of the people you can see here, Antoine. So Antoine here working closely with Camila to see whether we can bring in for segmentation problems. And we could not make it work for any of the medical image segmentation problems. And frankly, basically, we had to write a paper saying that these things do not work in medical image segmentation, and we had to publish it in GCPR simply because nobody else will take such a paper, right? Because it's so much I guess awesome to say things work rather than say report that things do not work.

But yeah, yeah, totally. But it's interesting that we're making the same experiences. I mean, yeah, okay, yeah, thanks. Any questions from the audience? The audience is so silent after the, um, fire attack. Oh, there's one, and I— Doreen, I think. Took your part again.

Yes, yes, Stefan, you can directly ask your question.

Yeah, thanks.

Thanks for a nice talk as well. I also saw your paper about detecting when a pretrained end-to-end fails silently, and I read it, and I found it really interesting. And one question, like, I was wondering that you, yeah, like you were training on one dataset, right? And then you tried to generalize to 3 different datasets, right? Did you actually find out why this generalization did not work or why it was detected as out-of-distribution? Like, were there different scanners used or did you analyze the metadata there? That would be very interesting to me.

Yes. So I guess, first of all, thanks so much for reading the paper. So I guess these days people stopped reading papers, right? So people look at Twitter threads. So thanks for reading, first of all. So I guess the second question is why? The major problem we had is exactly what you said, that it's coming from different types of scanners and they have different spatial resolution. That's one of the problems. And then you hope for that, let's say, NNUnit, when you trained on part of a self-configuring network in a meta-learning approach and it has seen a very heterogeneous dataset, it should be able to interpolate. But we often end up with the problem of sort of extrapolation, as in, for example, the MOSMED dataset. If you look at it, the spacing is so far off that, like, that's where we got the lowest performance, but it's sort of explainable in that way. The other thing that also depends on the patient characteristics itself. So for example, if you are looking at, the, let's say, the Western world, the American College of Radiology said that don't use CT or X-ray for diagnosing COVID-19. So it's primarily for patient management in the, in the, in the, um, how to say, in the ICU, right? Intensive care units. Now those are really, really sick patients. Now So if we are talking about them, you will often see a massive amount of infection there in their lungs. So a lot of that lung is really filled. Now, if you are talking about some other parts where maybe CT is used more commonly for the diagnostic purposes in the earlier days, there are really tiny portion of the lung being infected. So if you are really looking at dataset where the distribution has a lot of really sick patients, maybe really old patients, and trying to bring it to more of a diverse population where they are not that sick, things will go wrong, of course. But yeah, I guess there are, I mean, I guess if anything, I would say the biggest lesson from this is that we cannot completely predict the out-of-distribution scenarios. There will always be black swans. There will always be unknown unknowns. So developing all the algorithms and all the, let's say, evaluation setup, keeping that in mind is probably our best bet.

Excellent. So we also have a question in the chat. Anyone, would you like to directly comment on it or should I read it because it's really, really long?

I can read it, but I— sorry, can you hear me?

Yes.

So I can read it, but I guess I have to scroll through it. Um, but if you have a summary, maybe you can say during—

I'll try to— yeah. Augustine is referring to the comparison between photography development and radiology development. And he states, I think that the different outcome regarding the implementation in society is due to the different regulation demands between TikTok, Facebook, and so on. And that an structured system to integrate the images in an informative way for the doctors. And one is the different regulatory demand, and second is the liability. In the first platforms, the liability relies on the user and consumer. That is the same in the platform that you propose. The liability will rely on the manufacturer. So this might be one of the major differences that impact so badly the different outcomes of photography and radiology.

Yes, I totally agree with, with your point that when we are talking about liability. It's really, and the entire, basically the regulatory framework that is currently in place for US FDA, the medical device regulations and other countries that makes development of technology slower simply because technology needs to happen many more chances of failures before it can mature in healthcare. That's not really the case totally. So I guess one thing probably to note that though, that the structured reporting itself is now a validated framework. And for example, this company Smart Reporting we do, we work with, they have these structured reports for many different types of diagnostics that are being used rather widely. The problem though, of course, that filling up these structured reports with an AI framework and learning from it, both of these are dubious. So one example that I can give is that if— so it's funny how things happen. So let's say more or less everybody who works with medical data know that data is heterogeneous. And data is the heterogeneity, the continuity comes from both spatial heterogeneity spatial heterogeneity and temporal heterogeneity. Now, when you talk about spatial heterogeneity and we will come with federated learning distributed setup, everyone like, yes, that's the way to go about it. If you talk about temporal heterogeneity, everyone is like, no, no, no, we don't do it. And it's absolutely crazy in a way to think that we recognize that there is like heterogeneity and the entire change of of like things are happening, but we sort of are so happy with the spatial but not with the temporal. One problem of that is simply, I guess, this old mindset of mainly approval people approved drugs, right? And then the idea really was that once this drug is approved with this clinical composition that was validated with type of people, it's all RCT resulted in blah, blah, blah, reducing whatever effect. That's great. We don't change it anymore. Now, drug development and intelligence are not the same thing, right? Like you can imagine, like I have a 2-year-old. If we say like, we stop his learning now and Then we hope that over the next 5 years, he still performs the same. That, that's absolutely crazy. We don't stop the learning of those who are looking at the images, be it pathologists, be it radiologists, be it surgeons. They look at these things and recognize that, okay, things have changed and we have to really adjust. People move from one city to another. They adjust with the new department's framework. Saying that, okay, we can't let it do is a bit of a rich thing, I guess. But people will take time to change it. I guess sort of I tried to— I totally agree with you, Augustine, about the question though. But I guess that's sort of my take right now at how things are.

Okay, thank you. So do we have any questions? A further question? So now you have the last possibility to, otherwise, yeah, we are ready for lunch. So thank you very much for this really, really interesting talk, also for being our guest today. And Lena, you can—

yeah, thank you so much.

See you around, hopefully. Absolutely, another Mikai or something. But thank you so much for the invitation. It was a real pleasure. Yeah, have a nice day, everyone.