Scientist Stories: Chen Chen, Deep Learning & Medical Image Segmentation

Welcome to a new data science seminar. It's my great pleasure to welcome Qian Chen from Imperial College London and the University of Oxford. And also welcome to Lena Meyer-Hein, who will take over the introduction. The stage is yours, Lena.

Yeah, welcome everyone. It's my pleasure to introduce Qian Chen today. She is a postdoc at the University of Oxford and also an honorary research fellow at Imperial College London. She received her PhD from the Department of Computing at Imperial College London in January last year, actually, and also worked as a part-time research scientist at HeartFlow. Her doctoral thesis was entitled Improving the Domain Generalization and Robustness of Neural Networks for Medical Imaging, very timely topic, and was featured in Computer Vision News magazine. Very impressive— impressively, she also won several challenges, including the multi-sequence cardiac MR segment Segmentation Challenge in 2019 and very recently the Fetal Brain Tissue Annotation and Segmentation Challenge, FETA, last year. I invited her when I saw her last year actually as an invited speaker at a Mika workshop dedicated to the topic of augmentation. So, Chenchen, looking forward to your talk. The stage is yours.

Okay, great. Thanks, Lena, for the great introduction about myself. So today I will be giving a talk on the topic of advancing deep medical image segmentation with adversarial data augmentation, which basically summarizes what I have been doing in my past PhD study. And so without further ado, let's get started. Okay. So the talk today will be divided into 5 parts. I will first give a brief introduction and directly dive into the problem behind the AI for medical imaging, the domain shift problem. And we will discuss our solutions on two aspects. One is focusing on the input space adversarial data augmentation, and the other one is focusing on the feature space adversarial data augmentation. And hopefully, we will end up with some thoughtful discussions and some takeaway messages. Okay. So before I'm going to talk about my research, let me first talk about, give you a picture about how AI can actually improve healthcare, particularly how it can improve the entire workflow of medical imaging. So here is a typical workflow shown here where actually AI can help to speed up image acquisition, reconstruction, and image analysis, as well as some follow-up studies such as diagnosis and treatment planning, which are of great clinical value. So it is expected that actually AI can remove or minimize the time spent on routine, for example, some administrative tasks, some tedious works, which can take up to 70% of a healthcare practitioner's time. So with that, clinicians can actually have more time to care and interact with patients, which is the ultimate goal of our AI for healthcare, which we're trying to improve the quality of healthcare. And my research is mainly focusing on the AI for medical image analysis, which takes lots of time if you do it manually. So here, give a cardiac image segmentation as an example. We can actually quickly perform medical image segmentation from different imaging modalities. And for radiologists, it usually can take roughly, for example, 10 minutes or even half an hour to annotate the contour of the anatomical structures for measurement and to spot any abnormal tissues. But with AI, which it can just take a few seconds to get a full picture of clinical useful information such as the shapes, volumes, et cetera. So because of its great efficiency, now we can use it actually to analyze large population data for clinical study as well as some phenotype discovery. So AI has great potential to speed up and transform healthcare services. Okay. But however, we found that— so there is news published in MIT Technology Review saying that for the hundreds of AI tools which have been built during the COVID time, actually none of them are really help. And the problem behind it is so-called the domain shift problem. So here I just using a sausage example. So imagine you have an AI system trained using Chinese sausages, okay, which looks like this. And then if you want to deploy it to German market, which, you know, they have really large varieties of sausages, white sausages, the colors, the shapes, can have great variability. So you know that actually we need to measure the gap between the training images and test images, okay? So in medical domain, the dimension of the problem is actually very common. So normally what we are facing that we have very limited training dataset collected from a very limited population, and the models can actually perform pretty good on this. Type of images. However, if we directly test it on the test domain, which basically has a larger variability, comes from like different scanners, different medical image acquisition protocols, or different populations, such kind of variations may not be handled well in the current AI models. And another challenge behind it is so-called the different image quality due to different image, the scanner configurations are like some patients, they may have some disease, they cannot hold their breath properly so that the image quality can also be affected. And that will introduce some artifacts into these images, which will also pose the, or for the AI model to predict nicely on that kind of images. Okay. So, Here is a real example where we're trying to train our AI models to predict the cardiac shapes from the MRI images. And we trained using, which might be the most, the largest dataset in the world from the UK Biobank data. So basically it consists of 5,000 subjects from single scanner. So we found that it can actually perform pretty well in the intradomain UK Bellbound dataset. However, if we directly deployed to our France, to AC/DC, which is a public dataset collected from a French hospital, we found it can still fail even the model has been trained on a very large-scale population. So this highlights the issues behind the AI models. Another example is from the recent SAM models you may heard from that is developed by Facebook trying to develop universal segmentation models. So, we tested it on these models using our very simple original image. So, it performed pretty well on these original images where basically the left ventricle blood pool has been annotated. And if we're using our developed some augmentation tools to introduce some domain shifts into this image. So basically, it has— it basically introduced some morphological transformation and some intensity transformations. We found that some can actually produce not stable results. So basically, trying to segment some non-useful information such as the background and the contour of this subject. Okay. So how could we develop robust models using limited data? One of the solutions is maybe we could just collect lots of data from multiple different training sets from different hospitals and cover lots of different populations. But this is not that easy to do. Why? There are several reasons behind it. Firstly, it has a very high labeling cost. Because unlike natural images, medical images typically require people to have such kind of medical expertise or some medical knowledge to identify the region and then perform the segmentation. It usually requires more money or pays more money to do it. And the second problem is about the label inconsistent issues. So normally, because we have large multiple datasets from different hospitals, It might be labeled by different labelers or different group of experts, and they may have some kind of inconsistent labeling protocols which can hidden the model to generalize well from such a chaotic dataset. And the last issue is the most important one, that is the data sharing and the data privacy issues as posed by the recent, for example, GDPR. And I think different countries, they have different privacy policies that basically prevent people to directly sharing data across different countries. And even for different institutions in the same country, sharing data can be also very not easy job because it might involve lots of some legislation. Routines or works to do. So a more practical thing in that we wonder if we could directly learn a robust domain-generalized segmentation model using a single domain data. That is how we face this scenario every day, right? So normally, we have collaborators from one hospital, and they provide us a clinical dataset, and we're trying to learn some AI models to help them. And we're trying to maximizing the value if it can be directly developed or deployed into different unseen test datasets. So how could we do that if we only have access to single-domain data? So our approach is so-called adversarial data augmentation. And the idea behind it is that we're trying to synthesize a virtual domain which contains some realistic and diverse and challenging image variations to enhance the training dataset. We added into the original training dataset and then use it as a data augmentation to improve the model training. With the hope that finally the performance on those unseen datasets can be improved. How could we do that? In this talk, I will try to investigate the two roads. To do the adversarial data augmentation. And the first one is trying to do it directly from the image space where we're trying to simulate some challenging transformations in order to augment data. And the second part will be about the feature space. We're trying to learn some unknown variations in the image itself. Okay, so let's first talk about the first approach of image space adversarial data augmentation. So for the input image space data augmentation, I guess the most popular examples are— everyone might already know well is about this adversarial noises, which is initially developed by Ian Goodfellow in 2015, where you're trying to generate some noises that can be— when we add this noises into this original panda images, the AI models can actually classify it from panda to gibbon, which is totally a different species. And people found that we can actually add in this type of images into the training and can improve the robustness against such a type of noises. So in medical domain, there are also some very similar phenomenon has been found in some publications. For example, they're trying to add in some adversarial noises into this skin lesion patient images. And as you see, the AI systems can actually misclassify it as malignant rather than benign. And this also poses the issues or concerns about the AI safety. And also, people may use it to get some money from the insurance company. So it's very important for us to incorporate such kind of data into the training in order to improve the model. Safety. However, while this can be very useful, there are also some limitations behind these approaches. So, first of all, is that it can be very limited in terms of the effectiveness because the adversarial noises may not necessarily exist in medical world, right? The second one is about its limited diversity. As you see from these two images, the appearance differences is very minimal. So basically, we didn't really consider about some real image variations that may consider in the medical world. So one of the work we developed is trying to, instead of simulating some adversarial noises to attack the model, we're trying to model some realistic physical world attack that really exist in the medical world to improve the model robustness and generalization. And the one type of perturbation comes to our mind is so-called the bias field, which basically introduced some realistic intensity inhomogeneities in the MRI images. So in real world, actually, MRI images are often corrupted with this kind of bias field, which is basically caused by the non-uniformity of the coils. And so such kind of low-frequency signal, which smoothly varies across images, can actually produce some challenges for the model to generalize well. And as you see that here, if we're trying to attack, perturb the image with some bias field, it can affect its grayscale and it can affect the intensity distributions even within the same tissue. As a result, the AI models can fail to recognize the shape of this myocardium. So we would like to model such kind of challenging bias field with the hope that network can learn some more robust features against potential signal perturbations. So in order to model this type of bias field, we introduce a very simple bias field generator, which is a physics-based transformation model parameterized by a set of uniformly distributed control points. So specifically, given a set of control points, we basically randomly sample these values over these control points, and then we apply some B-spline interpolation kernel to generate some positive bias field at the full resolution. And then we multiply it with the image to get augmented images. So normally we can do it just treat it as a random data augmentation each time, but we are not limited to do that. We're trying to learn or trying to identify the model weakness by generating some adversarial bias field. How could we do that? In order to learn some adversarial bias field, we developed a framework called ADVBias, adversarial bias, which is based on some consistent regularization framework. The idea here is pretty simple. Here, given input image X, We first randomly sample our control points to generate some random bias field, and then we get our augmented images. We send these two images into the network to get the original prediction and the perturbed prediction. And we then calculate the consistency regularization loss term. So essentially, it can be just a mean squared error loss. And then we're trying to optimizing the underlying values of these control points. With the hope that it can fool the network to basically mispredict the myocardium region. So this is the first step. With that, once we found such kind of adversarial bias field, we can then add it into the training of this network. Well, here, because we did not use any ground truth to calculate the consistent regularization term, it can be fit well into the current existing semi-supervised learning framework using consistent regularization. So the final training loss for the network consists of supervised loss and unsupervised consistency regularization loss computed on the labeled data points and unlabeled data points. So in that, we can actually not only utilize existing labeled points, but also a large number of unlabeled data points in the dataset. Okay? So now let's look at some examples about the adversarial bias field generation. Okay? So here is the input images and the ground truth. Ground truth. We can basically first randomly sample some bias field, as I previously mentioned, to get some bias field that looks like this, which basically you can see some regions are highlighted, which means that the value is more than 1. So part of the region will be brighter than before. And we found that actually the AI models, if you look at the predictions, show at these white contours, the network can actually perform pretty well, although it introduced some some arrows in the middle region of this black hole. If we do one-step optimization to the underlying bias field, we can actually find that the network can be very sensitive to such an adversarial bias field. If you compare these two bias fields, you can see that it basically remains very— the two are actually very similar and they just shift a little bit to to the other regions, but the network can be very sensitive to such kind of modification. So we already show you how we could generate some bias field. Now we wonder if we could apply it to simulate some realistic anatomical variations that is also very common in medical world. So for example, different individuals, they may have different anatomical variations. Even for the same patient, for the cardiac region, it can have different morphology due to the cardiac motion.

Okay.

So in this work, we're trying to replace the previous bias field generator with this adversarial deformation generator model where we're trying to simulate, which is based on the static velocity field-based deformation model. I will give you more details about this region. But the idea is basically very similar to previous one. We get the input images and augmented images, and we're trying to optimize the underlying velocity field so that the network can perform very inconsistent predictions compared to the original one by optimizing the underlying velocity field. And the one thing we needed to take care of is about the consistent regularization because now when we do some geometric transformation to the input images, the prediction geometric space has also been changed. So in order to measure the difference between the perturbed one and the original, we have to actually warp this prediction back to the original image space so that they are in the same coordinates. So how could we do that? So here we highly rely on a specific type of deformation, which is so-called the diffeomorphic deformation, which allows us to generate some invertible deformation. Here is more details about such kind of diffeomorphic transformation, where we assume that the deformation is controlled by a static velocity field. And we can basically— one of the nice property of such kind of diffeomorphic transformation is that it's invertible and it allows one-to-one mapping. So we can easily compute its forward and backward transformations that can easily transform the augmented images back to the original space. So now let's look at some examples about the adversarial deformation generation. So the first step is still some GAN denoising augmentation, which basically can be viewed as some elastic velocity field deformation. And we found that if we directly add in some deformation, the network actually performs well if we warp it back to the original space and then compare to the original prediction. But if we do one-step optimization over the velocity field, we can see actually the network can be very sensitive, which produces some false positives. Near the cardiac region, which it looks at here. And here, if we look into the details of this deformation, we found that network will be more focusing on— the adversarial morphological attack will be more focusing on attacking the local regions of this cardiac region rather than the overall torso. Whereas if you look at this random deformation, you can see part of the deformation mainly happen on those non-cardiac task-specific regions, for example, this torso here. But here, basically, it remains the same. The major deformation happens around this cardiac region.

Okay?

So now I have introduced you about 2, 3 types of adversarial attack, the noise bias field one and the velocity-based deformation one. And we further consider adding like one more family, one more transformation into the family, which is affine transformation base. And this also corresponding to a very realistic variations caused by, for example, different patients, they might have different orientations or positions when they are being imaged. And also the scale of the cardiac, the shape can be differ if you're using different field of view. So we're trying to incorporate this into this adversarial data augmentation framework. And in order to further enhance the diversity, we're introducing ADV-Train, which basically trying to each time we just randomly sample a train of different compositions of transformation families. It can be adversarial bias field plus adversarial noise. It can be adversarial morphological plus adversarial affine transformation. So the underlying space of transformation has been largely improved using ADV-Train. And then the overall framework remains almost the same. The only difference is that every time we just randomly sample a train, which corresponding to some wide range of variations that exist in real world. So here is an example about ADV-Train. We are given an input image x. We're trying to first, for example, sample a train of transformation. So here we include adversarial noise, bias field, morphological transformation, and some affine transformation. As you see, if we plot the network predictions, basically the network can be very robust actually. This may be because we already include such kind of random data augmentation as the default pipeline when we train our models. However, if we apply one more step of optimization to optimize the underlying transformations or the data augmentation parameters, we see that the network can be very fragile to such kind of adversarial transformation where it basically mispredicts the cardiac region. Also, if we look into the details about the adversarial noise and the morphological, as I previously mentioned, you will find that basically the adversarial augmentation takes knowledge about the image and as well as the current state of the network, which is smartly trying to locate the region of the cardiac and trying to attack the local region. Okay? So here comes some results that we're trying to compare ADV-Train, which is adversarial data augmentation framework, to Random Augment, which is a very powerful data augmentation framework developed by Google. And we found that actually ADV-Train can outperform the Random Augment. I show these blue boxes are the ADV-Train results, whereas The red boxes are the results from the random augment. So normally, our model, the ADV-Train, performs better than augment in this cardiac segmentation task. Interestingly, when we're trying to plot the generation or the generated images using GAN augment, we actually found a wider variability of this image intensity change, whereas for the ADV-Train, we're basically introducing some mild augmentations, but it consists of powerful morphological attacks, which is very common in medical world. And especially if you look at this image and this image, we found that actually it's very realistic simulations where the overall region or the overall structure of this patient remain almost the same. The major difference is the shape of the cardiac region. But by contrast, if you look at the random augment, so basically mainly focusing on generating some intensity attacks, some can be very unrealistic. For example, where the intensity has been inverted in this type of change. Okay, so we further compare our method to a set of semi-supervised learning methods powered by different types of data augmentation. For example, the VAT is using some adversarial noise-based augmentation and the FIX applies some adversarial noises, whereas ours is powered by the ADV-Train. You see that out of all— Now this concludes the first part where we're trying to generate some adversarial data augmentation by taking consideration into a wide range of image space transformation functions into consideration. We're trying to first optimizing the underlying transformation parameters and then optimizing the segmentation network. So here is the QR code for the paper and the code. And we are very happy to see that actually ADV-Train has been adopted in the winner algorithms by another research team in the CML Motion Challenge. So this verifies verifies the practical value of ADV-Train for promoting or improving the model robustness. Okay. I will stop here to see if there is any questions for the first part. Any questions?

Any questions from the audience? You can post them in the chat or raise your hand. Actually, I have a question. So, I mean, I'm not familiar with the exact pipeline that you presented, but from, let's say, our work with the augmentations that we use, of course you have a lot of hyperparameters in the pipeline. And to what extent do you think when you, let's say, you have a running augmentation pipeline on one problem or on many problems, and then you have a new problem, To what extent do you think you can transfer those augmentation hyperparameters or starting points using the concept of meta-learning, for example?

Yeah, I think this is a good question. So the question is about because it involves so many transformations and also it breaks the optimization into two objectives, it can have lots of hyperparameters and how could you How could I scale it to different approaches, right? Different applications. Did I understand correctly?

Yeah, yeah, exactly. So basically, how can you make sure you don't start from scratch, but you also have some knowledge on best type of parameters on a range of problems? So how can you leverage that knowledge for a new task, maybe?

Yeah, so I think ADV-Train is very kind of like easy to use is a plugin module. And actually, we found that we can— some hyperparameters, we did not require very exhaustive search procedure. Actually, we're directly taking the parameters from existing random data augmentation pipeline, which has been verified in lots of existing literature review. We're just taking that values. So this is focusing on the transformation parameters. Regarding for the optimization one, Here, we're just using like one-step optimization and using some fixed lengths. And actually, we're using the same set of parameters for different task loads. For example, in our paper, we use the same parameters for the cardiac and prostate, and both works pretty well. And this basically verifies how easily it can be used. And in the GitHub, we also provide some Jupyter notebook And our inspiration is that if you can make sure that the variation is realistic, then it should be beneficial for the downstream task. So that's why we provide our Jupyter Notebook for people to navigate the space of transformations, to set up the magnitude constraint, to make sure that the transformation is realistic. Yeah, I hope I answer your question.

Yeah, thank you. Jan has a question as well.

Yeah, I find it super interesting. Thanks. And my question would be if you kind of tried to look into which of these transformations is the most effective, if you try to take out, for example, some of the etch chain and seeing which attacks the model the best.

So I think this is a good question. So the question is about do we investigate into effect of a particular image transformation function for the performance boost? And actually, in our paper, we did investigate by changing different compositions of the transformation. And we found that the result can be quite varied. It depends on specific task or the specific dataset we used for training. So there is not a clear conclusion, say, which one outperformed the other three or which contributed the most. It really depends on the training dataset composition as well as the test set.

Okay, thank you.

Thank you for your question. Okay, any other questions? If not, I will go to maybe the second part. So the second part is about adversarial data augmentation via feature space perturbation. I think previously we have discussed a little bit about the limitation highlighted by Elena, as we are very thankful for that question. So basically, as previously discussed, it requires you to specify a particular transformation function and some constraints, some hyperparameters to constrain the magnitude of this attack. So this all requires some expertise or to, for example, to first define a specific transformation function, for example, the bias field, and then you needed to set up the hyperparameters, for example, the bias field, the magnitude. So to generate more complex and some more unknown variations, We wonder if we could do it more smartly because for the image space one, we can only generate or introduce some known variations. What about those unknowns? So our idea is pretty simple. So we're trying to let the model itself to figure out those variations. And in order to do this, we're trying to first project this image space into our image latent code via some projections. This can be done via encoder function. And then we can do some augmentation or perturbation directly in this latent code space, say. And then we can generate an augmented latent code, which can then be projected back into this image space. So this can be implemented using a decoder function. And one of the work we've done for falling into this scheme is latent code masking-based. I will first give you an overview about this framework where we're trying to introduce auxiliary decoder branch here, show are these gray blocks where we're trying to model the image generation process by first encode this image into the latent code and then use a decoder to map this latent code into the image space. And then we extract some task-specific code from this content code into this shape code. And then there is a task-specific decoder or segmentation decoder that can interpret this back into the segmentation space. Okay? So in order to simulate some corruptions or variations in real world, we're trying to looking at this content code and trying to perform some masking-based augmentation in the latent space so that we can simulate some corruptions. Into these images. And in particular, we're introducing 3 different types of perturbation, including the very famous or very popular random dropout one, which basically randomly drops out a particular channel at a probability of p. The other 2 are more aggressive, yet more challenging ones, which basically is so-called the targeted channel-wise dropout or targeted spatial-wise dropout, where we're trying to utilize the gradient information to identify those salient features to mask. And I will give you an example about how we do it in the next slides. Overall, we generate these cropped images by randomly choosing one of the schemes each time so that we can get a set of cropped images into the training. And we feed it back into the training of this segment, this dual-branch framework in order to enhance the training. Okay, so now let's look at one of these targeted masking schemes. So here is an example about how we generate some targeted masking by utilizing the gradient information, where we first calculate the gradient information of the loss function with respect to a particular latent code. And then we're trying to summarize the information in order to get a channel-wise map where we selected the top p% entries to mask. So here we consider to, instead of directly multiplying it with all zeros, we're trying to multiply with a soft masking scheme using a random value sampled from 0 to 0.5. And this gives us much more diversity, but also helps us to preserve some information in the images. I will give you an example later. So basically, with that, we can generate our masked latent code. And then we can project it back to the image space by sending it to our decoder function. So here, this slide compares the generated images using different masking schemes. The first one is the random-based one. The second and the third row shows results from the targeting-based one, which basically are more aggressive. Compared to the random-based one. So in practice, we're using the soft masking-based one, which helps us to simulate those strong image artifacts, but also it can preserve the larger structures well. One of the limitations of this masking-based method is that it cannot well preserve local structures or minor structures. So last year, Mikhail, we proposed an enhanced version one based on the same network architecture. So the overall is still a dual-branch framework. The only difference is that this time, we are not going to mask the latent code, but instead, we similarly introduce some or insert some style augmentation layers, which is called the max style. And with max style, we can actually generate images with the same anatomical structure, but overall, the image style has been changed. So as you see, with that, we can generate some augmented images back into the image space and use it as a synthesized domain. So the overall loss function remains the same. So we basically have dual task losses. One is the segmentation loss. The other one is the reconstruction loss. And we compute the losses on the source domain and the synthesized domain. So here it gives you more details about how this style augmentation work. We are here given input images. We send this into the encoder and the decoder to get some feature code. We then augmented the style of these features in the latent space by performing some style mixing between different instances. We further introduce more style noises into this latent style code in order to enhance the diversity of the features. And this will give us an image like this. But we are not limited to this grand scheme. So similarly, we also apply the adversarial training approach to enhance the effectiveness of generated samples for model robustness. And what we do is trying to finding optimum values of the style noises and the level of style mixing by searching in the direction of maximizing this segmentation loss, the task-specific loss. So here is more details about this max-still layer implementation. For the time constraint, I will skip this part, but we can go back to these slides if any people has any interesting about to know this type of max-still. Perturbation layer. So here is a result about these max-style generated images. We are given input images and some ground truths. We can get some generated images using some random sampled style mixing parameters and style noise-based parameters. But we found that the network can still be very good at predicting such kind of style augmented images. With like 5-step optimization, we can find that the image style, for example, the brightness has been changed, and the network is very sensitive to such kind of a type of change, which basically produces you poor segmentation results. So we found that MaxStyle can actually preserve well in terms of this structural similarity. If you compare the generated images and the original images' structural similarity, So here is more results about how we evaluate our method on different test domains. So the first one is the easiest one. We're trying to directly test it on the intradomain test dataset. This is the baseline method where we're just using some random data augmentation and with a U-Net segmentation model to get some results with a dual-branch architecture. We also tested on 3 challenging out-of-domain test datasets, including the ACDC-C, which basically we're trying to augment the intradomain test set with 4 existing or 4 very challenging image artifacts so that the signal has been corrupted in these images. And the middle one is the cross-set where we directly collect a large number of datasets from public challenge datasets from multiple different domains. And the last one is the most challenging one because for the other three, they are all using the same image protocol. They are all BSSFP image acquisition protocol acquired, whereas the last one is a contrast-enhanced one. So you can see that the appearance is very different from these other three. If you look at the results of this baseline, you can see that while this baseline can perform pretty well on intradomain, it can totally fail on the other three, especially the last one, okay? If you look at the MaxStar results, we are very happy to see that actually it can produce results very close to the ground truth. So we further compare our method to a set of input space augmentation and feature space augmentation. Including the previously mentioned adversarial noise, adversarial bias field, as well as some feature space augmentation. So here we compared it to MixStyle, which is a downgraded version of MaxStyle using only style mixing rather than the style noise perturbation and adversarial training. The LSM is the latent space masking-based, which I introduced in our earlier work focusing on attacking the latent code by some random masking or target masking. So overall, the LSM is the most competitive one, but MaxSTEL can actually outperform the LSM, which basically highlighted the advantage of preserving the local structures while maximizing the style diversity. We further compared— here is more like quantitative results where we found that MaxSTEL can give you 20 more percent performance improvements in terms of out-of-domain generalization performance, while the inter-domain performance remains at the similar level as the baseline. You can find more results for different tasks. We also compare our method on the prostate segmentation task, and we found a similar performance boost is roughly also 20%. So here comes to the conclusion. So we talk about how important data augmentation for model generalization and robustness and how it can be very ineffective if we just randomly sample those values. In this talk, we show you how we could advance the data augmentation by thinking about the domain knowledge to simulate some image-level variations and how we could model in the image generation from the latent feature space by using our dual branch framework. And lastly, for both roads, we can actually apply the adversarial training for improved data augmentation effectiveness. And one thing, or one limitation about adversarial data augmentation that I should mention here is that it actually requires additional training time and additional memory because it requires at least one forward and one backward pass. So that's something that you need to keep in mind. Here are some takeaway messages which are taken from the London Underground, which you may hear every day. It's called the say it, say it, sort it. So basically, in order to improve— I'm trying to relate it to model robustness— where in order to improve the model robustness, you need to first identify the model weakness, why it performs so bad on particular test set and trying to simulate or replicate such kind of feelings by using some adversarial augmentation approach in order to search it in an efficient way. And then you can basically try to— hopefully, you can optimize the model for improve the model robustness. And here is— I would like to take 1 minute to do some advertisement for the DALL-E workshop where I met Lana in this workshop. So we are going to organize the whole days again this year, MICAI. And the deadline for the submission is June 25th. And we are very happy to get some support from our industry collaborators, which will give us some award money for the best paper. So I would encourage everyone who working on the data augmentation or some related domains to submit your papers to the DALL·E workshop. Okay. With that, I would like to thank you and I'm happy to answer any questions.

Yeah. Thank you for this great talk. Are there any questions? So as before, you can raise your hand or write them into the chat.

Maybe Lena? Yeah, I can start. So I have a question. It's more a strategic question. I think it's so difficult, right, to decide what to do when you have a new problem. So I'm back to that question. Because let's say you have a good budget and then you have to decide basically how to organize your training data. So do you spend budget on more annotations, better data augmentations, or especially when it comes to style, Do you instead try to basically organize your latent space such that the domain cannot be recovered anymore and so on? So there's a lot of decisions to be made, right? So what if you, let's say, were hired by a company with an entirely new problem?

How would you approach that problem of, yeah, Yeah, I think this is a good question. So the question is about how could we start with a very new problem so that the company might reach you with a set of data, but how could we start? I think nowadays people are thinking about AI from model-driven approach to data-driven approach, right? Because the AI models really requires massive datasets. And I guess one of the things came to my mind to tackle this problem is build a cycle of the AI models. You can first train a model using maybe not that perfect data, right? And then you can use that model to identify those data points that have very poor performance. And then in that way, you can either identify whether this is the data problem or the model problem. And then you can basically recycle that model by enhancing it using some adversarial data augmentation or using some, for example, some knowledge distillation models to distill the previous knowledge into a large expanded model in order to incorporate such kind of challenging in the data. So I would say it's kind of like an iterative process, but it's important to first have a not so good model at your hand to start with to help you identify the models. Yeah. And also, I'm very happy to see nowadays we have some universal models already, which help us maybe to bridge the gap for the preparation. Maybe you can directly take in some existing trained models to help us to identify the problem. And then using our domain knowledge, for example, we are all working on the medical domain. We know better than people working on general CV, right? So we can come up with better solutions to tackle a specific challenge behind the medical domain and working close with clinician and specific domain experts to come up with the solution.

And do you think we should have something like data-centric challenges on topics where the model is fixed and you and maybe even the annotated input data is fixed and you just have to come up with smart ways to handle your data, to do augmentations, etc., to move that field forward?

I guess this is a tricky question. The question is whether we should fix the data and try to optimize the model itself.

No, the other way around, right? So normally you fix the data and everybody's working on the model architecture.

I'm asking Okay.

Sorry. Now we fix the model, let's say U-Net, NN-U-Net, whatever. And now we have to basically organize the preprocessing of the available data in a smart way.

Yeah, I guess I think there are already some areas have actually touched that aspect. For example, active labeling. So the model, it may be fixed. I'm trying to select or like smart data selection, trying to identify those valuable data in order to train better our model. And alternatively, I think people can also think about some test-time adaptation methods, right? The model, actually we're trying to optimizing the model to accommodate the particular variations from the test image itself. I think it's very promising area actually. Yeah, great.

Further questions by the audience?

Jan? Yeah, hey, so my question would be, you showed us today a lot of of heart MRI images. What about other images? And have you tried also on other kind of data types, your method? Yeah. Let's say microscopy would be my interest, for example.

Yeah, yeah, yeah. We didn't try on microscopy, but we did try on prostate MRI dataset, and we found that it works pretty well. And we found that for the, especially for the feature space data augmentation, It can actually work on even non-imaging data. For example, I have seen people use— by applying the MixStyle for the ECG data, which basically is non-image data and perform pretty well. So I would say there is a huge opportunity, especially for the feature-based data augmentation can be used to boost the general pipeline. And therefore, maybe it's useful for your microscopy as long as there is space for us to do.

Yeah. Okay.

Thanks.

Maybe very last question because I'm always interested about problems. Is there something that you can point out that you struggled with in particular that was really challenging in your work that maybe you could share with the community?

This is a very good question. One thing I would like to share is that in In my talk, I've been showing you two approaches, ADV training and the feature space. So I'm trying to combine them to see whether it can give you like a 1+1 larger than 2 effect. But however, I found that it's really difficult for me to replicate, to maximizing the combination effect for the model robustness improvement. So I'm thinking about what's the problem behind it. And maybe this is because the model capacity is limited and you have larger variations of this data augmentation that might not be useful anymore or might be limited to improve the model performance. So that's one of the problems I'm struggling currently, trying to produce higher results if I can combine them two. Because ADV-Train is mainly focusing on not only image intensity level but also geometric level, whereas the feature augmentation basically is more focusing on some intensity level variations that we are not so sure about. And yeah, it's very difficult to combine them too. And this is the problem behind it. But I'm thinking it's a very promising approach.

Yeah. Very interesting. Thank you so much. Thanks for coming as well. If you still agree, the talk will be online. I think that would be great. It was a very nice talk, I think. And yeah, I hand over to Doreen for concluding words.

Yes. So yeah, thank you again for this great talk. Thank you to the audience, also to all the questions. And yeah, see you all in 2 weeks when we have Christoph Lippert. So you can see all the infos on upcoming seminars and also the recordings we have on YouTube. And yeah, feel free also to join our mailing list. See you all in 2 weeks.

Goodbye.