Scientist Stories: Julia Schnabel, AI-enabled imaging

Welcome to the NCT Data Science Seminar. This event is something special as it is hosted by Heeda, HITS4Health, Alice Life, and the NCT Data Science Seminar communities. A warm welcome to all the representatives. A warm welcome also to Julia Schnabel, our speaker today. We are really happy to have you here. Before we start, I have to announce that the seminar today will be recorded and the video record will be published later on our YouTube channel. If you have questions, please use the chat function or raise your hand. As last time, there will be a short HEADA presentation, and afterwards, Linda Meyerhein will take over as the host of this event. Victoria, now the stage is yours.

Yeah, welcome everybody, also from our HEADA side. We are the Helmholtz Information and Data Science Academy. And for those who are not joining from the Helmholtz cosmos today, Helmholtz is Germany's largest scientific research organization. And HeDa is part of it and connects as a roof of 6 data science research schools. And with that, it's Germany's largest postgraduate training network. Here you can see our 6 Helmholtz Information and Data Science schools throughout Germany, and the HITS4Health is of course one of our schools. So if you're interested in the other schools or want to have more information, I will share later on some links in the chat. What are we doing as HeDa apart from the research our research schools are doing? We are promoting and providing financial support to all Helmholtz institutions with different training and network opportunities. And we offer, of course, exchange and visiting programs, which you can also find on our website. We are doing a lot of HIDA networking events such as career days. We will soon have a career day on health as well, and I will share the invitation link later on also in the chat. And of course, we have on our website very interesting opportunities for jobs in the field of data science from all our centers. And if you want to know more about HeDa, you can follow us on Twitter, on LinkedIn, and subscribe to our newsletter, and of course, explore our website. So with that said, I wish you a lot of fun today with our lecture, and yeah, follow us and keep you posted.

Thanks, Victoria. Now let's move on to the actual scientific content. It's my great pleasure to introduce Julia Schnabel to you. Many of you will know that she's actually one of the most famous medical imaging researchers worldwide. She's actually— I have to look at my notes— a professor of computational imaging and AI in medicine at Technical University of Munich, also director of the new Institute of Machine Learning and Biomedical Imaging at Helmholtz Center Munich, and has a secondary appointment as chair in computational imaging at King's College London. She graduated in computer science from Technical University of Berlin. Then she left Germany for a while, did a PhD in computer science at UCL in the UK, then joined the University of Oxford where she became full professor. Actually, I hope I get it all right, Julia. And then finally went to King's College London, and now we're very, very glad to have her back in Germany because she's very famous. She's elected fellow of IEEE, of ELLIS, of the MECAI Society, won many awards Actually, in Alice, I have to mention that she's one of the about a dozen founding fellows for AI in Health. So basically, she's one of the few or was one of the few representatives of AI in Health in Europe. Yeah, I personally got to know Duya on the MECAI board where we both serve. She's now the executive secretary. And I was always very impressed with several aspects. One was that she always fought for students, always wanted to give students more money for traveling, more rides, more whatever. Also for diversity, not only in gender, also in, in other aspects. And that's probably also one of the reasons that she was one of those that actually now brought MICAI to Africa. This is very special. You have to know that, yeah, MICAI is the main medical imaging society worldwide, and the bidding process is very competitive, like in the Olympics. You have to submit a proposal, and that year— and I was on the board, so I really know— was so, so competitive. But yeah, she and her team, they made it, and Mikae goes Africa. So Julia is really a person who wants to include minorities and, yeah, who wants to bring everybody together. And also, she's never afraid, I have to say that, to bring up delicate issues that are controversial and that that just needs saying. And I really appreciated that always about you. I wanted her to become president, but she rejected. Maybe I will succeed sometime in the future. Anyway, in German, you would say she has her heart on the right flag. I don't know how to say it in English, but I'm very glad Julia is here with us today. And yeah, the stage is yours.

Well, thank you very much, Lena. I'm duly humbled and embarrassed. I could just say right back at you, I think it's been a fantastic time working with with you on the, on the MECA board. And well, I miss you there. Let me just see whether that works. Well, some of these issues will come up. I had some background on me. So this is the full story. But I think, you know, you got everything right. I'm getting confused these days myself. I still have one foot in the UK despite now being here in Munich. And I've got all these things. One thing I just wanted to point out, I've joined HIDA's steering committee of HeDa. So I get to see a lot of the different graduate schools. In fact, this is the third one I'm talking to. I talked to MADATA and to MADS, the Munich School of Data Science, of course. And for me, it's a great opportunity to connect with lots of researchers all across Germany. So for me, this is a real pleasure. And Lena is one of my prime people in Germany anyway. So this is great to be here today. Just where I am, so Munich isn't such a big town after London, but it's big enough that I still get in some good cycling distances in. So I'm at TUM, I'm out in the research campus in Garching in the informatics department, which is a bit of a ride. It's 45, 50 minutes. The Helmholtz Centre is located right next to the Bayern Munich training campus. So just to give you some orientation. I still have my satellite group in London and most of the work, or actually all of the work I'm presenting today is still from that group. As I'm still building up my lab in Munich, but it's growing on the right-hand side. You see more and more faces coming. I'm supervising, co-supervising new PhDs, and my first postdocs are finally starting next month. So here are some of my disclosures. The work I'm presenting today was funded by the Engineering and Physical Sciences Research Council in the UK, and the grant is called SmartHeart. I'll talk a little bit about at the end who's involved in that. What I want to talk today about is what I call AI-enabled imaging. So in medical imaging, we usually fall into some of these categories. We either acquire images, so we generate some raw data using some imaging sensor, an MRI scanner, or an ultrasound probe. Then we construct images, so we transform the raw data into some image suitable for viewing by a radiologist. Or I actually come more from the image post-processing group. It's actually a name I don't like so much. It's more image analysis actually now these days, but it's kind of a mix of filtering, segmenting, registering images, but also doing some more complicated things like constructing models for detection classification of diseases. And then finally, of course, there's the image interpretation by clinicians. Now you've got this whole pipeline here of imaging from sensor to the clinician. And machine learning is great. It can be applied quite modularly along at all these different components on the imaging pipeline. But it's not really linking across that much yet. And this is what I'm really excited about because machine learning really has the power to bridge those gaps and do things more jointly and even end to end. So the background, I've met some of the students at HITS4Health because I'm also on the advisory board of of it. So it was really great to get to know, even though on Zoom and the pandemic, and again now on Zoom. But some of you I know don't have that background. Some of you are a bit more experimental. So you're quite nice mix of people. But the focus, of course, is on data science. So you have to acquire data. Machine learning was really, I think, hit it off most with computer vision. That's still the case. First, it was artificial intelligence, machine learning, deep learning. Actually, deep learning is not a new concept. Actually, I did it in my undergraduate degree. Only that computers were still a bit slow and we didn't have that much annotated data, but we already worked on MNIST data and backpropagation, all these things. Convolutional neural networks were just starting at the time. But ImageNet also came in the 2010s only. That actually enabled a lot of new things. And of course, GPUs came and people could detect strawberries and buckets and traffic lights all of a sudden. The network architectures are, of course, quite varied. The simplest one would be like this simple encoder-style architecture for a simple classification task, detecting dogs and cats or regressing things as well. Of course, you can then go into an encoder-decoder architecture. So you kind of compress everything into a bottleneck, have this kind of lower-dimensional space, and then you bubble all the way up again to for example, to pixel-wise classification for segmentation, use this for restoration and other tasks. So this is very, very simple, basic, just illustrative deep learning for you. In medical imaging, and here I'm showing cardiac imaging, of course we need to be a bit more considerate and respectful of the data. It's not just photography, even though photography is complicated on its own. It's about patient anatomy, patient physiology, patient pathology. And of course, the actual acquisition is quite, quite different. There's imaging physics involved as well, which we need to think about. But in principle, we can use a lot of the same things. So in medical imaging, we're very good at stealing ideas from computer vision. So we do all these things. We restore images. We reconstruct images. We classify images, do regression tasks, and of course segmentation. This is where U-Net, the one I'm showing you, was really lifting off. And that's actually a MICAI paper, which is quite nice. I think over 32,000 citations from Olaf Ronneberger. I think it was mentioned before, if I'm too fast or skipping over things, please make yourself known, raise your hand, or feel free to interrupt. So why do we want to do AI-enabled imaging? Well, we've got lots of challenges in medical imaging, which makes machine learning quite difficult to apply. We've got sometimes very poor quality data. There are artifacts. There's patient promotion. We don't have these large, well-curated, annotated datasets. There's also a lot of legal, logistical frameworks which don't allow us to get easy access to hospital data, and for good reason as well, but it's frustrating at times. Clinical data is just really very varied. We've got lots of differences across different scanning systems. We've got domain shift. I mentioned that a bit towards the end. And of course, there's this black box approach. We've got this lack of interpretability, so it's hard to sell this to radiologists. Times. So here I want to talk about applying machine learning along the imaging pipeline from acquisition to interpretation. And these approaches, as I mentioned, can be modular or increasingly integrated and even end-to-end. And cardiac MR, because of the SMART-Heart grant, was our main starting point here. And my focus in that grant is on image quality control. So really thinking about how good are the images. Are they good enough? Can we restore the quality? Can we still rescue them? Do we have to discard them? And then that actually opens up a lot of new avenues in image reconstruction of MRI, in image segmentation and analysis, and also the domain shift. So, senior cardiac MRI in particular is routinely acquired for the assessment of cardiac health. It's great. You've got these videos of— almost videos of the heart beating. But here the quality, if you just think about quality issues, you've got the underlying MRI physics to take care of. There's this trade-off between signal to noise. So the better the images get in terms of actual quality, the lower the temporal resolution will get, and the other way around. So you have to trade off these two things. Then there's also the patient physiology to think about. There's the beating heart, of course, that's what we want to image. Patients breathe, which is good, but not helpful in terms of imaging. They are patients, so they struggle having a consistent breath hold. And of course, they're lying in the scanner for 20, 30 minutes, and so they will be just moving about as well, which means that images of poor quality, then they need to be discarded or the annotations will be impacted and wrong. So the diagnosis actually might be impacted as well, or we need to recall patients, which affects not only the hospital workflow, but it also delays the diagnosis. So If we want to apply AI in this domain, we usually want some really good high-quality data and annotations, large, well-curated databases, which is the tricky bit. But thankfully, there was this initiative in the UK a couple of years back and is still ongoing where they're scanning 100,000 volunteers for brain, heart, and body imaging across dedicated centers in the UK. It's very tight quality control and scan consistency. And it's really a very nice dataset. Everybody can get access to that. Now they charge some money, but it's not too bad. But I think it's good value for money, actually. You get really good access and you can really see high-quality images here. There's a German equivalent called NACO, the National Court, which is only 30,000. I think access might be a bit more tricky there, but I think we are getting here in Germany as well. One thing to bear in mind Nina talked about diversity. This is across the UK cohort. So, it's basically UK residents between 40 and 70 years old, I think. Predominantly healthy. Some of them, of course, might have some conditions. Of course, the UK population is a little bit more diverse than the German population, so that's good. So, it's a bit more heterogeneous. But it's not all from London, the data, it's also from North England, so then becomes a bit less diverse. So the problem, of course, is yeah, we can do great stuff on high-quality data, but in the end, we want to go to real patient data where we've got all these problems: inconsistent breath holds, hospital workflows which don't work that well, scanners of different quality, and so on. So we have to bridge that gap to make our work useful for clinicians in the end. So the first step in that pipeline would be to think about image quality assessment. So is the image good enough? Do we need to rescan the patient? We could also use that step to actually establish, curate a training database so they actually can go through hospital data and say good, bad, you know, a bit like Cinderella sorting peas from— I don't know what she was sorting things from actually. So basically just putting things into different databases, the ones you can use and the ones which are not useful unfortunately. Next step, of course, would be a bit more ambitious and say, well, having established which datasets are not very good, could we still rescue them? Could we avoid patients having to be recalled? Could we not delay diagnosis, still get something out of those data? And actually, could we even improve further downstream tasks? So segment, classify diseases, and so on. And then, of course, for me, the ultimate goal— and we're not quite there yet, but I think we're working towards that— is to accelerate imaging, to stop imaging when the image quality is good enough for the diagnosis that we need, which would actually help to scan more patients in the same amount of time, or not only pushing through more patients, but allow more dynamic time for patients who actually need extra scans, identify those more quickly and say we need a bigger scan time for those. And for those where we actually can exclude whatever we wanted to exclude, we're done already. We don't have to go through these extra protocols. So it's a bit more like doing a bit more like a patient-specific scanning setup. And the first thing is, as I showed in the beginning, is classifying the good from the bad. So we can show networks good and bad quality images and train it in a supervised fashion and get a usually quite good answer. And that's the same as classifying dogs or cats in an image, so there's not much different there. The first thing my postdoc at the time, Ilkay Oksuz, who's now a lecturer in Istanbul, was to think about scan planning. So that's, that's a big problem. If you want to do a 4-chamber view in cardiac MRI, you have to do 2 other scans before you even get there. So you have to do a 2-chamber view as shown here, and then a short axis view. So actually just going perpendicular through that, then you have to put these lines through, and only with those lines you can actually plan the 4-chamber view. So you need to have this appropriate angle. You need to exclude the aorta as well. And it does go wrong. It actually goes wrong more often than you would think in a normal hospital setting. And if it does go wrong, you've got these off-axis images, these kind of false 5-chamber look images, which where the left ventricular outflow tract is visible. And that then gives results and difficulties in atrial analysis. So in these cases, the patient would need to be called back in, the diagnosis would be delayed. That's not good. You want to actually ideally detect that at scan time and say, actually, we have to repeat the scan immediately. But so we did this retrospectively instead. There's a lot of things to consider. Excuse me. It's not only about pushing images through, it's also locating where the heart is in the images. So we actually do some object detection. You can use your, again, your favourite network to do that, or you can do just template matching as we did in the good old times to get your 2D dataset, and then you train a very, very simple network. In this case, it's actually almost the original LeNet from Yann LeCun where you just train for LVOT present or not, so poor quality or bad quality. Now we were faced with a problem though. UK Biobank is a really good high-quality dataset that we had very few of those poor quality images, very few of of the LVOT images, false 5-chamber look images. But we found 123. So we did just a balanced set of 123 good quality images. We would have had 80,000 more or so, and those few poor quality images. Of course, we did the usual data augmentation, just jiggle around images just to get a bit more variation in the training. We had 5 temporal frames of each sequence, kept them of course separate either in the training or in the test set. That gave us 615 images for each class and did, as you stretch for data, tenfold cross-validation just to boost the performance a bit. And what we also did is, because that was a time when I was still not convinced by deep learning, I was still a skeptic at the time, and we said, well, let's start with machine learning first. Let's see how good we can get with things like random forests or support vector machines and so on. But I was very soon convinced actually with this little trick, the bit of data augmentation, we got an accuracy of 80%. Adding data augmentation, 82%, and precision and recall were also quite good. So as I was still skeptic, Vilke Oxius had to kind of convince me a little bit and said, well, let's look a little bit inside of the network. What happens with these attention maps on which the classification is based? So this is an LVRT. Image, a bad one. This is a good quality image. Just looking at the attention maps, you could see actually the LVOT attention map focuses directly where the LVOT is present, where the good quality attention focuses more on the separation between the heart chambers. This was kind of just a bit of a glimpse, an attempt, or first attempt at explainability, but it's important to actually question what your network does. The next step was to think, well, if you've got poor quality images, can we go from poor quality to good quality? With LVOT, we can't, of course, because the scan has been acquired in the wrong way in the first place. So we can't undo time there. We can't just, you know, virtually put the patient back in the scanner because we would really have to recall them. But at least we can detect it and maybe detect it in real time and keep the patient in the scanner. But if there is a motion in the data set like this, could we actually go from poor quality to high quality image? So actually, radiologists can just view those images. This is just some example of more general motion here. But with motion, we've been working for ages on respiratory motion models, also in image navigation. I'm sure Lena has done contributions to that field as well because the problem is, of course, you want to do guidance and you want to compensate for motion in real time ideally. The problem is here in this case, it's motion which is within an image. It's not just between images, so that registration doesn't help. It's motion within an image. Could we actually understand where this motion actually comes from, and could we do something about it? So you see a good quality image on the left, really nice crisp myocardium, bright blood pool. You see the papillary muscles and so on. And then on the right-hand side, you see a motion artifact image. Now, first, the first view, if you are not used to those images, you might just think, It's just a bit poorer contrast, bit smaller heart, and looks a bit fuzzy. Yeah, it does. It looks a bit blurry, which means that what you actually see is several phases of the heart at the same time, almost superimposed. So you don't see end systole or end diastole. You see some mash-up in between, which is not what you want if you want to calculate, for example, the end ejection fraction or some other parameter, clinical parameter of interest. And the reason for this is to understand actually how these images are acquired. So you do ECG triggering normally, and you acquire not just one cardiac frame per frame, you acquire several lines of k-space, basically the raw data over several heartbeats and over different parts of the cardiac cycle at the same time. So you've got these these lines for frame i, time i, basically. But that's also done over several breathing cycles and over several heart cycles, but always at the same part of the heart cycle, approximately, until you're done with all frames and have done, I don't know, a couple of breathing cycles and cardiac cycles. And of course, if everything goes well, you would just do your inverse Fourier transform and get to your nice high-quality images. But if you do— if something goes wrong with the trigger, you're not quite sure at which part of the cardiac phase you are at any one time, which means some of those lines can get mixed up. And then if you then do your inverse Fourier transform, you will get these kind of mushed-up images where you see this kind of different several cardiac phases at the same time. Now that we understand how that works,— well, we've got tons and tons of high-quality data from UK Biobank. We can actually make them look worse. We're really good at making really good images look not very nice if we wanted to. The other direction is the difficult one. So we've got these tens of thousands, almost 100,000 high-quality images of volunteers. We are more likely to have these poor-quality images due to patients being imaged in the clinic. But to bridge that gap going from UK Biobank algorithms to algorithms we can deploy in the clinic, we actually need to take into account that poor-quality or recover that poor quality and make it look high quality. Now, if we start mixing up lines of k-space, that's easy. We can do that. We can just go back into k-space. Technically, I mean, there might be some MR physicists here, we don't have the phase images, we only have the magnitude. So we have to do some tricks there, but ideally we would have the raw k-space data. We should not throw it away, but it's usually thrown away, unfortunately. You have only the reconstructed images on UK Biobank, but we can apply different levels of corruption. We can mix up different cycles, different, across different frames, we can deliberately corrupt. So we can go from really high nice quality images to poor quality images. Now we know this way is basically a physics, MR physics motivated explainable data augmentation, which we can then exploit to augment the data in our restoration methods. This is basically analogous to data augmentation and computer vision where we would wiggle around some cats and rotate them and zoom in, right? Only here it's really physics-based, it's not just random translations, rotations. We can do that in addition, of course. So the idea is now to say, okay, we know lines of k-space are wrong. We're not just training models to just against good, you know, going from bad images to good images because we can deteriorate them and can no can go back with a simple encoder-decoder architecture back to the good image. We can actually even exploit that further. If you think about how MR images are acquired, when you want to be fast, you would do undersampling. You would not acquire dense k-space anymore. You would apply some undersampling trajectory and just acquire a percentage of lines, maybe going down to 25% or 10% even. And then there's lots of works which focused on that where you use an undersampling trajectory, acquire an undersampled case space, get a poor quality image, but at the same time do a data consistency too because you can again simulate an undersampled case space from a fully sampled case space, apply some reconstruction network and get a high-resolution fully sampled reconstruction. Yeah, there will be some loss. The trick here is to really overcome the violation of the Nyquist frequency theorem. And that's when neural networks are really nice. They try to overcome that for you. If you would just do undersampled inverse Fourier transform, you would get these horrid artifacts. But we can apply this now because we now have a fully sampled acquisition, only that some of those lines in k-space are wrong. They're just sorted in the wrong way. So we have a fully sampled acquisition, but we can detect which lines— we can train networks to detect which lines are wrong. Once we've detected those wrong lines, we can throw them away, discard them, and then formulate basically this motion reconstruction problem in an undersampling reconstruction problem. By having thrown away all the corrupt lines of case space, we basically just have an uncorrupted undersampled case space and then actually can do our MR reconstruction and get a fully sampled reconstruction artifact corrected. It's actually an undersampled reconstruction now because we've thrown away lines of k-space, but you get really good quality images. Now one thing is of course nice, we've got nice images now. So as I mentioned in the beginning, the next step is actually to do maybe something useful with those images. And one thing is just to say, okay, we compensate for the motion, we get really nice quality images, and then we can segment them, for example. So we can actually see what happens. So if you've got a high-quality image, we get a really nice U-Net, NN-U-Net segmentation out of that. That's not a problem normally. If we use here in this case an artificially degraded image the subsequent segmentation will fail. We can measure that. That's very easy. We can measure failure quite well. But we could actually do better than that. We could actually say, well, we don't only want to rescue images, go from bad images to good images, and then apply segmentation. We could actually do both at the same time. We can have deliberately complex networks these days. So we can even in this case, I'm not just putting in one 2D image, I'm putting in one whole time series of 2D images as a 3D input. We actually then use a reconstruction network with the artifact detection as before. Only this time it's over the whole time series, so we detect all wrong phase baselines across the time series. In an ideal world, I would reshuffle those lines and put them and slot them back into the right place. But we never got there. And some lines might just be missing in a real scenario anyway. And you can do this. I'm not putting these technical details here. We do use a recurrent network approach here. So we actually couple with that the time series together and exploit redundancy between the different case spaces. So we get it as a 3D output, our undersampling trajectory, then can do our reconstruction. And then we can put a segmentation network in there as well. Your favorite one, U-Net, NN-U-Net, whatever you want. And we just put this all into one joint training objective. If you've been around for so long as I have, you will see these loss functions, they look like basically like our cost functions which we used in segmentation ages ago. The problem is then to balance these hyperparameters, weighting the different terms. In the end. But you get with this both a reconstructed image and a segmentation. You might even skip the reconstruction step if you wanted to because you're just interested in the segmentation. That would be an end-to-end learning approach. But here you see some examples on the bottom. You see some deliberately corrupted image, the difference from the original uncorrupted image and not a very good segmentation. On the right-hand side, you see a recovered reconstruction, much better difference image. Still, there's some differences you see there, especially at the segmentation boundaries and then the segmentation. So there was this issue, should we just do sequential, you know, one after the other, or should we do it training, or should we do it end-to-end? And of course, you can look at this, you can do it in a retrospective study, you can create artifact images, look at the difference from the original one, segment the images, and it looks bad. If you do motion reconstruction just followed by segmentation, it's middle column, you get quite nice segmentation. Reconstruction doesn't look that much better, I find. But if you do it jointly, and that was for me really some nice finding, not only is the segmentation better, but also the reconstruction looks nice. Because now the reconstruction has focused on the segmentation objective. So actually, it looks to me almost like some nonlinear blurring has happened, which helps the segmentation. So the reconstruction is now more suitable for the segmentation task. If your objective would have been something different, like a classification task, the reconstruction probably would look different again. So it's actually, you have to think about for what purpose do you want to do these different steps. And so you might have to train networks with different objectives. But we also— one reviewer actually, I think, insisted, reviewer 2, I'm sure, to say, why can't you do it prospectively? So can you show it in real data? We had shown on retrospective data that we get some better results by really taking real corrupted data where we did not know the case-based trajectory, but still ran it through our network, and we got some nicer results. Whether they're correct or not, of course, we don't know. They look more plausible, certainly. But here, we put a volunteer in the scanner and did deliberate ECG mis-triggering. But we also acquired a really high-quality scan, you know, young clinical fellow who can hold their breath for ages. So that actually gives us both real good quality and real poor quality data. So here are just some examples of this series. So we've got on the left a deliberate mis-triggering result. Is the actual acquisition which looks that bad, and the segmentation result which still does surprisingly well but not perfect. Then we did the motion reconstruction just followed by segmentation which looks nice. Then we did the joint motion reconstruction segmentation which both look better, both the reconstruction and the segmentation. And on the right column you see the corresponding good quality image. Of course not the same image as we would have obtained in our joint sequential model because different acquisition in the first place, But you see they look actually quite similar. Actually, I think ours looks better, but I'm biased. So in a way, this method actually has also potential to be a generic image reconstructor because we used also on good quality images and actually improved signal-to-noise ratio and other things. But of course, with the segmentation in mind, so it's maybe better if you want to do a segmentation. If you want to do something else, you may have to train on a different objective. We went a little bit further just to think about image quality control, how it works in a clinical workflow. Let's see whether that movie works. Yeah. So we first acquire images, then we accept or reject having a trained network. Those rejections, those would be because retrospective, those patients would have to be rescanned. Of course, we can now go one step step back, in fact, and actually redo those. Then we do a full cycle segmentation. So do a full cardiac sinus segmentation. Then we compute parameters of interest like ventricular volume curves. Then we do another quality control step where we look at the profiles of the volume curves for LV/RV consistency, filling and ejection points. So we can again accept or reject, right, because there segmentation quality may not have been good enough, and only then we would accept the ventricular function parameters. So we compared this to what a clinician would do, and we reached with that for our clinical fellow who is really excellent, human-level accuracy. So very, very good agreement in total. So this was done on 700 cases, 500 healthy and 200 with ischemia. And based on these ventricular parameters. So this is now starting to be put into clinical workflow. This is something we can actually run side by side with experienced cardiologists as well and actually starts becoming a really helpful protocol. And then the most recent bit, just a few minutes left, I hope, was to actually also think about active imaging. So when to stop scanning? So we know people use undersampled case space, we know there's also motion-corrupted case space which we make sparse by removing the bad lines, but maybe even when we just start acquiring, could we stop earlier? Could we decide when do we have enough information just to make time maybe for more relevant protocols for patients? So the aim there is to accelerate the scanning while still having enough image quality for the downstream task we're interested in. So, again, we started with Biobank subjects here, 200 healthy and 70 with cardiomyopathy. Just a cross-section through the normal population in the UK reveals, of course, people at that age have already some health problems. That's one of the key research questions actually from UK Biobank. Can we detect those? And then we performed retrospective, in this case, radial undersampling, because it's a bit more relevant in cardiac. So we went from Cartesian sampling before to radial one and did several quality checks. Again, look first at the reconstruction quality, so good/bad quality check, and then the second check was on segmentation quality again. And then we did clinical function assessment. And we basically were able to reduce the scanning time per slice from 12 to 4 seconds. On average. So we can apply this as generic reduction time by staying within a 5% error. So this work from Ines Machado, which we presented at a MICAI workshop last year, and you see here the whole overview. We start from the left with the MRI position, we do the undersampling, then comes the first quality control step where we actually classify images in good and bad quality. So at what level of undersampling would it still pass that check is the question. Then we do the full cycle segmentation. Then we do a segmentation quality check directly on the segmentation maps. Then we only— then we do the volume curve analysis and go through the puncture parameters and diagnosis. So the final thing, just 2 minutes or so, I want to spend on domain shift. So this is when things go wrong. So basically when you go from one hospital to the other, from UK Biobank Siemens scanning systems to Philips Philips, different field strength scanning systems in your hospital. So basically, the problem there is that machine learning models do not have such a great generalization capability if you don't have enough data. But it's also, if it's on the same quality data, 1.5 Tesla on Philips or Siemens, or good quality on either, does not agree very well with our algorithms. So we looked here at cardiac MRI. We looked at different scenarios, different scanner types, different field strengths. We stayed away from the scanning protocol, which was close enough. Of course, there's a problem of different annotations done by different people. We always have that problem. So it's also a domain shift if you think about it. And of course, patient demographics, disease status, and so on is an issue there. So here we just looked at 4 different domains: Siemens 1.5-tesla scanner, Philips 1.5, Philips 3-tesla, and then the Siemens PET PET MR, 3 Tesla, where we also do some cardiac scans sometimes, not the PET, just the, just the MR. We had different numbers of subjects, so there's a class imbalance problem. So, we only had 42 on the last one. But we started also splitting the domains, so our number is getting smaller and smaller. So, we looked at looking for each scanner type for healthy, for two kinds of cardiomyopathy, dilated and hypertensive, and then also hypertrophic cardiomyopathy. So actually, our smallest group is only 8 left. So that's not very good. I'm surprised how much you can do with so little data as well. We also did some mixed domains. So the first mixed domain is just all the healthy subjects. And the second mixed domain is then the ones with these two types of cardiomyopathy as well. And then we had the third domain, which just is mixing healthy and some myopathy together on one scanning system. So lots of stuff to do, lots of numbers. You would expect lots of good things to happen along the diagonal in this matrix, saying if you scan on a— if you train on A1, whatever domain that is, I think it was just Siemens 1.5 healthy against itself, it would do best, but not necessarily. So we actually found some inconsistency and we also found some asymmetry as well. So for example, the mixed domain performs quite well on 3 of the 9 domains and second on the best performance on 4. So mixing data, having more heterogeneous, more diverse data in the first place usually helps. Yeah, that's a nice finding, I think. The other thing is that the cross-domain performance wasn't symmetric. On this domain, C2, trained on C2, tested on A1, fails. Whereas if we train on A1 and test on C2, it works well. We don't know. I think my argumentation for that is that if you train on diseased patients, that the healthy and the normal variation of anatomy is so varied that it's not captured anymore because it's occluded almost by the pathological variation. And that is an issue. So you need to think about that when you design experiments. I think I'll skip those. Although one thing we found that intrascanner volume overlaps turned out to be higher than cross-scanner volume overlaps. That's for both left and right ventricle and both end-systolic and end-diastolic. Frames, but the scanner difference was still to be the dominant factor contributing to the cross-domain performance drop. This is still active work we're working on, and we're looking now at more clever kinds of data augmentation to overcome some of these issues. Also, what helped for mixed-scanner training, it reaches in-domain performance when we just have enough subjects. So basically, you're having more variation of subjects even though it's all over the place. The data actually helps. And then we get, which with that also better generalization performance on unseen domains. This gap closes as the training sets become larger. So let me just finish here. So AI-enabled imaging has the potential to be really transformative in healthcare. And the nice thing about machine learning AI AI, as people call it, but it's actually deep learning what we do, is it can operate along the entire imaging pipeline, not just on reconstructed images, but you can look at all the individual steps. And the more you understand about the individual steps, the more you can actually exploit that as well for diagnostics and prognostics. So now my— ah, it's frozen. Okay. So we've looked at image quality control here mainly for reconstruction segmentation analysis. I think it's really helpful just for very basic things, just curate your dataset, just sort out the good from the bad from the ugly. Ideally to restore the images, but maybe restore with a certain downstream task in mind already. For example, the joint reconstruction segmentation, but you can think about regression or classification there as well. It's really nice to think about now prospective quality control checks so we can really move into active imaging. While we're imaging, we can discard case baselines. Obviously, we have to keep on scanning because it's not good enough for a segmentation task. And then domain generalization is a big issue, but I think we can get a handle with that. And hopefully we get more data. I mean, that's the main problem with us. And then hopefully we can actually not only work across all these different things, but we can actually also think about more end-to-end approaches. Do we need images at all? I annoy radiologists when I say that, but the data is already there in the sensor data, right? This is where the information already sits. And every other step, if you just concatenate all these other steps, you introduce an error at each individual step. So you actually accumulate errors if you do it one by one. So if you could do it straight, it might actually be more accurate in the end. So I want to acknowledge all these wonderful people from the SmartHeart grant. One of the nice one of the biggest grants I've worked on, and we're still going, and we're just thinking about round 2 right now. And then I mentioned a few of these things. Mika goes to Africa. I'm super excited about that. So we've just, we're just launching in the Afrikai network. It's a network of African researchers, graduate students joining. We have already, I think, 200 people joining, and we only did the call 2 weeks ago. So I think it's really lifting off. That's my main work for the next 2 years, at least. We are running a workshop on biomedical image registration. There's a call for short abstracts soon. If you like motion problems, that's the way to go. And Munich is lovely. It's a nice place to be. We also started, like Lena did, challenges, special interest group. We've just launched a special interest group on biomedical image registration. So hopefully great things will come of that. And my other baby here is the Melbourne Journal. It's a free, open access, peer-reviewed archive overlay journal. From the community for the community, and there'll be lots of nice events happening in conjunction with that. So with that, I'd like to thank you all. I'm really looking forward to your questions.

Thank you very much, Julia. Yeah, for those of you who don't know the Melbourne Journal, it's pretty cool because you get the review directly for your archive paper.

I really like that concept.

Yeah, very impressive. Also great to see that you're putting your, your work actually into the clinics. It always sounds so easy, but I think it's half of the work. So yeah, that's, that's great. So for questions, you can use the chat. I already see a question. You can also use the raise hand button. So Albrecht, do you want to pose the question yourself?

Yeah, sure. So I was wondering, you spend so much work on trying to get these errors out of the images, if there maybe was a way to circumvent the generation of these corrupted images in the first place by looking at the ECG triggering and maybe look into where these faults false images, these false triggers coming from? Yeah, I mean, the big problem is getting this data retrospectively because I think the way it's stored, it's printed out and stored as a PDF or something. So you don't get the raw numbers anymore from the ECG triggers. So we can't train retrospectively classification networks on the ECGs anymore. So that's a setup problem. I think it's not an unsolvable problem, but I think if you move into active scanning, you might want to monitor the ECG signal as much as you want to monitor the scanning itself. And then you could maybe run even faster detection classification network on just the ECG trigger as well. So it's a good point, but we don't have that data. We don't get it, unfortunately. We haven't trained. We have simulated it for a while. We just looked at the simulation of that. And then you can actually Once you combine it, there's nothing that prevents you from throwing it into your network as well as an additional aid. So you could use it to stop the scan and say we have to restart or something in principle, but it's just because you don't have access to the data. Where do the false triggers come from? Well, that's— these are cardiac patients. I mean, they have arrhythmia and other things. And this is exactly their problem almost, right? So it's almost a chicken and egg problem that the ECG is not regular. And then things fail. So it's more common with patients to have these kind of data than with volunteers. That's why in UK Biobank we didn't have that many data to look at. And maybe as a follow-up question, would it then be possible to use the ECG of a patient with arrhythmia to predict the false triggers that are coming from that? Well, these are these variable devices, right? This is the stuff which people have to wear for a couple of days just to monitor the ECG signal. And that's really more as a precaution to monitor them over time when they are actually active or at rest. So this is just normal cardiac care, actually. Here, the ECG trigger is just just applied to get perfect images, ideally. Okay, thank you. So one is a diagnostic tool and the other one is more like a, it's a bit like a navigator tool, if you want.

Yep. There's actually a question on Mikae. Mihiretu, do you want to speak up yourself?

Lovely question. Sana Chum, are you asking yourself? So first of all, what we are going to do— oh, it's okay. What we're going to do, so we've already did a survey for people to join the network, but africai.org is a website that's just being built, should be up in the next 2 or 3 weeks, I hope. So this is where you can also sign up. The MICCA Society is— you can always sign up. There's a student discount. In Africa, even that discount might be too expensive. So this is what Lena said, I'm pushing for, you know, making things cheaper for students because even $10 might be difficult. Even the transfer of money sometimes might be difficult just because there are no public funds available. So yeah, but as a student, you can always join MICAI if you've got the funds and hits for help. Should be rich, should be able to afford it to you. But you can, if you recommend to your colleagues, africai.org would be the first stage. And what we're trying to do in the run-up to Mekai, to run already lots of workshops, summer schools in Marrakesh. We want to run the virtual summer school or winter school, not quite sure yet, and a real one next year, like almost like a test run for Mekai. And then we really want to connect to the community there and we want to help people to actually raise awareness of MECAI in Africa, but also raise awareness in MECAI for African healthcare problems. Because we're working on luxurious UK Biobank data, but that's not data you might find outside of Cape Town, maybe even, or Cairo, maybe. So it's really, if you go to poorer countries, you might not have that. You might have more X-ray or ultrasound, more affordable imaging devices. Which is also an exciting area. And we want to promote those imaging methods in MICAI as well. So not work on super duper 7-tesla brain data, but actually on acute care data, on data where you just have to deal with not experts even acquiring the data and trying to make sense of it later.

Do you have a more technical question? So you mentioned the end-to-end training. And it makes sense to me that maybe we don't even have to reconstruct the images. Let's work on the raw data in order not to introduce errors, et cetera. But when you build these, say, complex architectures, how do you decide where the bottleneck is? So I mean, you get an initial performance. And there's so many things that you can play around, the individual architectures or the how do you weigh the loss function components, et cetera. So in such a complex system, how do you decide where to start the optimization?

Well, first of all, the system we're using are not complex. We try to keep them as simple as possible. They're much more complex systems.

I mean, more generally speaking.

Yeah, yeah, yeah, more generally. I mean, well, in the end, you do empirical testing. And in the end, you have to do trials, right? You have to put it to the test. You have to do blinded studies where you compare your methods against clinicians' work, where you actually are blinded to the answers as well. So that's the ultimate test to get regulatory approval. In the meantime, you do benchmarking, as we all do, as we work on challenges and so on, like M&M's and SACOM challenges and so on. You do hyperparameter tuning. So you start learning maybe the weighting factors and the loss functions. Ideally, I'd like to move us into the direction that we also learn the loss function and not just put our mean squared error terms in there or structural similarity index or whatever. But actually, there might be an optimal loss function for a problem. So could we move there? So everything could be trainable in the end. But then the explainability will be more and more difficult. But yeah, it's an ongoing problem. We compare, we have more data, swap between sites, train on one side, test on the other. That's how you make progress, right? But the actual question, I think, how can you do it if you do end-to-end training? How can you make sure that things in between don't go wrong? I think it's always good to have still this modular approach so you can test each individual component separately and justify it to the clinicians. So it's a bit more like a systems analysis approach or systems engineering approach that you actually are sure about input/output at each stage. But actually, I start thinking more and more that even if you just reconstruct data, we're losing, right? We're losing information. And that ripples through. At every step, we will lose something or we introduce an error. And so even if you take the systems approach, you will get a different answer because you might have had a bias. You might drift in a direction. But the end-to-end ones, they have to be very specific to a problem. I think you can't just have one general problem solver there. You just have to say to detect that kind of condition, then it might work.

Thanks. And with that, I think we're actually right on time. So thanks everyone for coming, and most of all, thank you, thanks to you, Julia, for speaking here on your very interesting work.

And yeah, Doreen, I leave you the concluding words. Yes, so thank you very much for this great talk and also Lena for chairing this session. Yeah, we are at the end, so everyone, thank you for coming. Stay healthy in this corona time and see you next time. Goodbye, everyone. Have a good day. Thank you. Bye.