Scientist Stories: Paul Jäger, Deep Learning for Biomedical Images

Good morning, everyone. First of all, thanks for joining. This is our first test with Zoom. Let us know how it feels. So we tried BlueJeans, we tried various variants, now we're in Zoom, and we would be happy for some informal feedback. Today it's my great pleasure to introduce Paul. This is one of the rare cases in which we actually have an internal DKFZ speaker, but for a very good reason. Paul is a trained physicist, and I actually remember his application very well. This was, I think, 4 years back in 2016 when he applied, and I have to admit that he did a really good job. He was a very convincing candidate, and then he did his PhD in Klaus' group, and one of the first things that I remember during his PhD was that he came up to me and he said that he thinks that was together with Simon Koh that he found a way to trick the metric of an international challenge. And that was actually the case. So the organizers had to change the design of the challenge. So Paul is very rigorous in the assessment in general. He did a very, very good PhD. I also remember that, not the exact numbers, but his paper on breast cancer Early detection had a lot of attention also in social media. So yeah, overall big, big success. And on top of this, Paul is also a founder of Heidelberg AI. You all know this because we sometimes have joint events. I think there's now more than 1,500 registered participants. So a great move also for for Heidelberg and the topic of AI. Yeah, so today we invited Paul, actually it would be Paul and Fabian Isensee, for a very exciting publication which is now accepted with Nature Methods actually. And I don't want to, I don't want to say too much. I can just say that this was a very say, controversially discussed paper at first. It had a very provocative title, No New Net, and the paper was actually rejected at MICAI. And so now it's a bit of a triumph that it's accepted with Nature Methods. But Paul can tell you the whole story on behalf of Fabian and himself.

All right, thank you, Lena, for this introduction and for the invitation. So my name is Paul Jaeger. I'm a postdoc at the Division of Medical Image Computing here at DKFZ, and today I want to talk about NNUnet, or Automated Design of Deep Learning Methods for Biomedical Image Segmentation. And this project, as Lena mentioned, originated in Fabian Isenzee's PhD project. He was working on image segmentation for many years And in this recent publication, which was accepted at Nature Methods, there was a team effort between Fabian and myself and also Simon, Jens, and Klaus. And also Fabian is here today, so if later on there are very technical questions, he might want to help us, or he will help us with some firsthand technical insights. So thanks for that. All right, so semantic segmentation is the de facto status quo approach in our field. It's featured in 70% of all biomedical image analysis competitions in recent years, and this information is taken from a recent study by Lena and her team. And so how does segmentation work? We have an input image and we do want to assign every individual pixel in this image to a predefined set of foreground or background classes. So it's a pixel-level classification, and this is typically done with a U-Net architecture proposed by Olaf Ronneberger in 2015. It's basically an encoder-decoder architecture with skip connections, and I guess you're all somewhat familiar with the U-Net, so I will not go into detail here. What is important for now is that the rise of deep learning comes with a promise, and it's the promise of full automation. So as opposed to earlier approaches, you just put your input into the model, forward process it through the model, and out comes the segmentation map. This is great, right? So end users like radiologists in the clinics or researchers from other domains like biology can just use it without expert knowledge on their datasets. Can they though? So one of the core messages of the work I present today is that no, they cannot. There is in fact a lot of manual intervention required, and choosing the network architecture to be a U-Net is only a fraction of the things that need to be configured in a deep learning-based segmentation pipeline. But because this configuration process is not very fancy and publishable, as Lena mentioned before, it got rejected at MICAI, It is often underconsidered, so let's walk through it together. So everyone who wants to perform deep learning-based biomedical segmentation on the dataset has to go through the following process. Based on your input data, you need to decide for a preprocessing strategy that includes intensity normalization or image resampling. Next, you need to configure the exact topology of your architecture. What is the input patch size going to be and the number of pooling layers, and which receptive field will your model have? And then you need to choose the right loss function and determine other hyperparameters such as batch size, learning rate, batch sampling strategies, etc. And once you start your training, then you will find out if your configuration even fits in your GPU memory. So there's additional external hardware factors coming into the equation. And when your training finished, then you— which in our domain may take up to 3 to 5 days— you need to manually decide if you are happy with the performance on your validation set, or if not, you need to give the whole thing another iteration. So I hope I convinced you that currently we are very far from full automation, and instead you need to manually configure your method, which is a time-consuming trial and error process. The success depends heavily on experience of the researcher, and it's inaccessible to non-experts. And we are not the only ones to make this observation. So there's a well-known, well-known survey on medical image analysis from 2017 with over 4,000 citations, and they state that disappointingly no clear recipe can be given to obtain the best set of hyperparameters, as it is a highly empirical exercise and most researchers fall back to an intuition-based random search. Okay, now you might say, I believe you that configuration is a lot of effort and there's no standardized way of doing things, but does it really make such a difference on the final performance of my model? So let's get some empirical evidence on that. What we see here is the leaderboard ranking of the 2019 KITS Challenge. So the task here was to perform kidney and kidney tumor segmentation. And public challenges are a nice way of getting insights into the current state of research because they provide a standardized environment where methods are compared against each other. And we take this challenge as an example because it was the largest challenge in 2019 at MICHI with the most participants. And now this figure shows the 100 entries to the challenge ranked after the final ranking scheme. And what we do here is we pick out one exact architecture type. In this case, we choose the winning architecture type of the challenge, a 3D unit with residual connections. And now we mark all of the 100 submissions in green that use this architecture type. And our first observation is, despite using the same winning architecture type, the entries can be found all across the leaderboard. And now let's look closer. For 5 example contributions, we visualize the method configuration they chose in terms of 10 key properties of their segmentation pipeline, such as batch size, patch size, or how they resample their input data. And we call this representation of a method configuration the pipeline fingerprint. And then we plot the values of each property for each fingerprint as a relative score with respect to the mean of all fingerprints. So that's these histograms that you see. And we see that these fingerprints reveal vastly differing method configurations among submissions despite them all using the exact same architecture type. And this tells us that the process of method configuration is complex and non-standardized, and it further indicates that method configuration has a huge impact on your final performance, with suboptimal design choices leading to substantial performance drops. And again, we're not the only ones to observe this. The review I quoted before also states that the exact architecture is not the most important determinant in getting a good solution, and many researchers use the exact same architectures but have widely varying results. Okay, so now we have established that method configuration is important for your performance and that is a highly manual and non-standardized process. But now you can ask, Okay, but this is the same in all deep learning pipelines. Where is your point about this being specific bottleneck for biomedical segmentation? And the answer looks like this. Datasets in the biomedical domain are extremely diverse. Here we see the 19 datasets that we used in our publication, and the blue boxes, by the way, cover 4 additional datasets that got added during the review process at Nature Methods, and we cannot show them yet. So to visualize the diversity of datasets, we defined data fingerprints. They work very similar to the pipeline fingerprints we have seen before, only that now they are made up of 8 key properties of datasets, such as the median number of voxels in the dataset, the size and spacing of anisotropies, or which modalities are used, etc. And again, the fingerprints show a widely varying landscape of properties among datasets. So let's say you go through the manual and cumbersome process of method configuration and find a nice setting on dataset 1. It is unlikely that your setting will perform well on dataset 2 because of all of these different properties. And as a consequence, you need to go through the whole configuration process again for each dataset. And this is what makes the biomedical domain special compared to classic computer vision working with natural images, for example, where there are no different dimensions, modalities, or spacing anisotropies. So if you make a list of all problems that the current status of manual and unsystematic process of method configuration causes, then let's add to the list that it needs to be repeated on every dataset. And now let's take a step back and think. We made some critical observations here, and I already mentioned that these issues are often underconsidered in our community because there's nothing fancy or publishable about them. So what does this imply for the publication landscape in biomedical image segmentation? So the current status is there's a lot of literature. Since 2015, the U-Net has gathered over 20,000 citations, and many of them propose architectural advancements over the U-Net, and that means they also feature experiments where they claim to outperform a U-Net baseline. And this is kind of counterintuitive, right, given what we just saw, the relevance and brittleness of method configuration and the vast dataset diversity that makes it so hard to translate performance from one dataset to another. Okay, so then let's start by using our observations to derive some properties that we think should be required in publications of general methodological claims in order for them to be convincing. So given what we've just seen, the claim should be evaluated on a large variety of datasets. Only then can we expect it to generalize to unseen data during application. And also, it's crucial that the proposed method, as well as all utilized baseline implementations, receive extensive optimization of their configuration on all of these datasets. And this is where our manual and non-standardized process of method configuration comes in, because it constitutes a major bottleneck for sound evaluation of proposed methods in these publications. In practice, these requirements that we set are just not feasible. They're exceeding the effort, computation power, and also often the training data in typical research scenarios in our field. And this is why in practice we observe that publications typically feature only one or two datasets, and even then it's not guaranteed that all utilized baselines received sufficient optimization budget for their configuration. So again, let's try to get some empirical evidence on this. Let's again look at the KITS leaderboard, and the green row you should already be familiar with. But now we want to distinguish different architecture types. So first we do a coarse analysis where we distinguish all 3D unit-like architectures from non-3D unit-like architectures, and this is the purple row on top. And in the 6 bottom rows, we distinguish the ones who are 3D U-Net-like further into their exact architectural modifications. So here we chose 6 architectural modifications that have been proposed and published on top of the U-Net since 2015, such as residual connections, dense connections, or additional attention modules. Now, it's hard to draw conclusions from the rare of the leaderboard because, as we have established before, suboptimal design choices during configuration can lead to large performance drops irrespective of the architecture type that's used. So let's look at the top 20 then. Looking at the purple row, we can see that all entries except one on rank 18 are 3D unit-like architectures. So we can conclude that in order to get state-of-the-art performance on the Kids task, using a 3D unit-like architecture is a necessary condition. And so that's a clear signal. But now when we look at the 6 architecture types we distinguish, there's no clear pattern or signal, and it seems very noisy. So we make the observation that none of the commonly used architecture modifications represents a necessary condition for good performance on the KIDS task. And by the way, this observation goes beyond this one dataset. We will later see that we see similar observations on 18 further datasets in our publication. So we could say that methods published as advancements in literature might not necessarily translate to new datasets. Which, as stated before, we relate to insufficient evaluation, which in return is limited by the manual and non-standardized process of method configuration. And as a side note, irrespective of the work presented today, I think this should make us reconsider how we review publications in our fields on conferences or journals. There seems to be a bias towards new fancy-sounding methods and maybe not enough focus on sound evaluation. But let's get back to our list of problems. That are caused by manual method configuration, where we can now add the two implications it has on our research and publication landscape. That is, it reduces method evaluation to one or few datasets in practice, and it might cause support baseline re-implementations. So in our work, we propose a method that addresses all of these challenges. And I'm going to show you how it in fact is able to alleviate these prevalent bottlenecks in our field. And this method is called NNUnit. And it's a segmentation method that automatically configures itself for arbitrary biomedical datasets without requiring expert knowledge nor computing resources beyond standard network training. And by the way, the name NNUnet comes from No-Net because it only relies on simple U-Net architectures without any fancy modifications. And given my previous comments on current literature bias, they might give you a hint on why we chose this name. Okay, now you might say, wait, automating configuration of deep learning methods? That's not new. There's an entire research field around it called AutoML. So yes, there is AutoML, which is a term that describes any effort to automate machine learning. And you might have heard of neural architecture search, AutoAugment, or also simpler methods like grid search, random search, or Bayesian optimization of hyperparameters. Now, one thing about these methods is that they are all entirely data-driven. So with few assumptions regarding constraints of search spaces, These methods run purely empirical optimization from scratch, and crucially, this process needs to be repeated on every dataset. So this is why they typically cover only a fraction of the deep learning pipeline, such as the architecture or the data augmentation, while NNUnet is a holistic approach that automatically configures the entire pipeline with zero manual decisions remaining. And thus, NNUnet requires zero expert knowledge upon application to a new dataset whereas AutoML methods require sensible definitions of search spaces and constraints. And empirical learning from scratch on each dataset takes its toll. AutoML methods typically extend the computation and time requirements by orders of magnitude and are thus not practical in our domain, where standard 3D segmentation training already takes up to 5 days. NNUnit, on the other hand, features automatic configuration that is performed in a matter of minutes and does not require any hardware beyond standard network training. And the same efficiency issue goes for data. In order to find a good configuration empirically from scratch on a new dataset, you need some training data, right? Or to put it differently, if your training set is small, your configuration is not likely to generalize to your own test split of the data. NNUnit, on the other hand, features task-agnostic assumptions or rules that transfer knowledge onto datasets without learning everything from the data at hand, and it's thus much more data efficient. And finally, as we see later, NNUnits achieve state-of-the-art performance, while AutoML methods are not relevant in our domain at the moment. So looking again at the KIDS leaderboard, we found no submission used AutoML except for one non-competitive submission who reported to have optimized a few hyperparameters via grid search. Okay, so now the question is, how does this work? We formalized the development process behind NNUnit in the following recipe that you see here. It consists of 3 steps and divides all design choices that need to be made during configuration into 3 parameter groups: the fixed parameters, the rule parameters, and the empirical parameters. Before going into the detail, I would like to mention an important fact. So since we make claims about configuration generalizing to new datasets, it's important that we do the development of our method on a restricted pool of development datasets and then fix everything and apply the fixed method to new application datasets. Sort of like a clean train-test split, but on a meta-dataset level. And in our case, the development pool was strictly constrained to be the 10 datasets of the Medical Segmentation Decathlon Challenge that was hosted at MICAI 2018. Okay, so now let's get— let's walk through our recipe. We start with the fixed parameters, that is, collect design decisions that do not require adaptation between datasets and identify a robust common configuration. And these are our fixed parameters. It might look kind of simple now, but actually years of Fabian's experience in biomedical segmentation go into these design choices, and it should be mentioned that these choices have interdependencies. So the task is to find a joint configuration of all of them that generalizes well across datasets. And because, as the name fixed parameters suggests, all of these choices will be untouched when applied to new datasets. So you can imagine that this constitutes a substantial shortcut comparing against AutoML, which basically needs to learn all these choices from scratch again on each new dataset. So the rule-based parameters The second step, so for as many of the remaining decisions as possible, formulate explicit dependencies between specific dataset properties, so the dataset fingerprint we saw, and design choices, so the pipeline fingerprint, in the form of heuristic rules to allow for almost instant adaptation on application. So imagine it like this. You start with a fixed value like the batch size is 5, but then you realize this value does not work well on all datasets. Now what AutoML would do is to just learn the optimal value for batch size on each new dataset from the data. But this might not be the wisest thing to do, right? Because if you do know the factors and interdependencies that lead to this design choice or should affect it, like the batch size, then you should use this knowledge. So this knowledge we formalize into heuristic rules that can then simply and quickly be executed upon application, just like if x in the data, then do y. And it looks like this at application. So when applying NNUnit to a new dataset, a data fingerprint is generated to quantify the key properties of the dataset, such as spacing, shapes, and modalities. And based on this information, all design choices that depend on the data at hand are adapted to the dataset by following our set of heuristic rules. So each arrow you see here visualizes a dependency between two design choices, and this is how such rules look like. So many of them are just simple if-else statements with some input, where some input goes in to determine a design choice, and there are more complex ones involving iterative adaptation. So let's look at one example, which is the configuration of network topology, patch size, and batch size. And what I also want to talk about is the process of coming up with these rules. It starts with a general domain knowledge about segmentation, which in our case was mostly driven by Fabian's experience in biomedical segmentation. And so you could have 3 pieces of knowledge that go like this: 1, larger batch sizes allow for more accurate gradient estimates and thus preferable, but in practice any batch size larger than 1 already results in robust training. Or 2, larger patch size during training increases the contextual information seen by the network and is thus crucial for performance. And 3, the topology of a network should be deep enough to guarantee an effective receptive field size at least as large as the patch size in order to not discard contextual information. Um, and now when developing NNUnet, we condensed this knowledge into a heuristic rule which says Start with an initial patch size, which is the median image size. Configure the architecture accordingly, which roughly means choose the number of pooling stages such that the feature maps do not become smaller than 4 voxels in each dimension. And based on this topology, a GPU memory estimation can be made. And if this fits into the GPU memory, your architecture, we max out the batch size and are done. And if not, we reduce the patch size by a little bit and repeat the whole process. So that was one example for how such a rule can look like. And now the third group of parameters are the empirical parameters. So learn only the remaining decisions empirically from the data. So this is the group of parameters that we want to keep as small as possible because empirical optimization is costly with respect to time, computation, and data. But in NNUnit, we managed to only perform two design choices empirically. And both of them happen after training, so they do not add any intensive GPU work. One of them is the decision on whether to not suppress all segmented components but the largest one, and the other is which of the 3 networks that are trained should be chosen as the final model. So, or ensembling of 2 models is also possible, and both these design choices are empirically determined based on evaluating performance on the training data. And this is how NNUnit looks as a whole then. So given a new dataset on application, the data fingerprint is generated and 3 rule-based parameters are— and the rule-based parameters are determined. They are then combined with the fixed parameters and the training is started. So by default, 3 models are trained: the 2D-Unit, a 3D-Unit, and the 3D-Cascade. But the 3D-Cascade is only triggered on datasets with large images. So when the patch size resulting from a configuration is less than 12.5% of the image size, because then we expect the model to suffer from not getting enough context information during training. So in the cascade, an extra low-resolution model is trained that captures all the context, and the context is then concatenated to the input of the standard 3D-UNIT. And once the training is done, the empirical parameters are determined, and then NN-UNIT is ready for predicting some segmentations on unseen data. But importantly, as an end user, none of that is relevant to you because all processes are entirely automated. So NNUnet can readily be used as an out-of-the-box tool to generate segmentations. And now let's see how good these segmentations are that NNUnet produces. So to evaluate NNUnet, we participated in 19 international competitions that featured 49 segmentation tasks in total. So each panel you see here is one task where NNUnet performance is a red dot and all competing submissions of this challenge are blue dots, and higher is better in this score. And the orange box shows you the task we used for development of NNUnet, and the others are applied without any manual changes. And we can see that NNUnet takes the first place in 29 out of 49 tasks. And importantly, in the remaining 20 tasks, NN-UNIT is on par or close to the winning submissions. So it shows top performance in an extremely robust manner. And to get you an idea of how surprising these results are, you would never expect a decathlon athlete at Olympic Games to outperform the specialized athletes in their respective disciplines, right? But this is exactly what happens here. NN-UNIT is applied without any extra tuning or manual intervention. And of course, the credits here go to Fabian, who implemented this method, because having good ideas in theory is one thing, but getting a concept on the street is another story. And it's totally thanks to Fabian's excellent engineering skills. And by the way, the blue box again covers results that we cannot show yet until our paper is published. These are additional experiments on biological data. Spoiler alert: these experiments also went pretty well. And to show you some qualitative results, I hope this gets you an idea of how diverse the application scenarios are. They range across different modalities, organs, organ substructures, tumors, lesions, and cellular structures in 2D as well as 3D. So now we have seen how NNUnit is able to automate the otherwise manual and cumbersome process of manual method configuration. And how it can be used as an out-of-the-box tool for state-of-the-art segmentations. But this addresses the first 4 bottlenecks we identified at the beginning of this talk. But what about the implications for our research field and the publication landscape? In this last step, I would like to show you how NNUnet can be used as an experimental framework where you integrate your ideas in and use it to evaluate them on a large amount of datasets without further effort, and how NNUnet can be used as a standardized baseline to compare your new ideas against. Without having to tune the baseline to every new dataset. And outperforming this standardized baseline will make your results look very convincing, and we expect this to be a helpful application of NNUnet to advance the field of biomedical image segmentation. So we do a kind of exemplary study here where we modify some of NNUnet's fixed parameters, such as using additional residual connections in the architecture or changing the loss function or optimizer. And this is to show two things: one, as an ablation of our fixed parameters, testing if our configuration is robustly outperforming others; and two, as an example for how new ideas can easily be integrated and evaluated on many datasets using our original NNUnit configuration as the baseline. So here you see the 10 datasets of the Medical Segmentation Decathlon, and the panel on the bottom right next to the legend shows you aggregated results over all 10 datasets.. Now, looking at individual datasets, you see vastly varying results, and only when aggregating the 10 datasets you get a clear picture. So this confirms our previous intuition that evaluation of methodological claims on one or two datasets is not sufficient. So we see that evaluation of methodological claims requires many datasets. There's strong ablation results of our fixed parameters, and that NNUnit can be used as a framework for large-scale evaluation and as a standardized baseline. So to sum up, before NNUnet automated the configuration of methods, it was either handled purely expert-driven in a manual trial and error process, or there existed a possibility to use AutoML methods for purely data-driven empirical optimization. Both had their drawbacks, as we heard. And now NNUnet outlines a new path beyond these two. It automates or systematizes the expert-driven process. So before NNUnet, there were no common settings in our field, or at least not joint configurations of settings that work well robustly across many datasets. And NNUnet also shortcuts the purely data-driven process by distilling domain knowledge into inductive biases or assumptions that can be executed instantly on new datasets. And as mentioned before, especially on small datasets, this domain knowledge is beneficial because the configuration settings you get from grid search on a small training set might not generalize to the test set, while our fixed or rule-based parameters in NNUnet transfer knowledge from a large and diverse data pool onto the dataset at hand. So that makes NNUnet truly automated, fast, and data efficient. And just two last points as a little outlook. We think that NNUnet can be used as a starting point for further optimization. So just recently at MICAI 2020, we were able to win two challenges. One is with cardiac segmentation and one with brain tumor segmentation. And in both cases, we used NNUnit as a starting point and did some modifications to specialize it or to the task at hand. And the second thing is that we're really serious about using NNUnit as a standardized baseline. To advance the field and the publication landscape and all the bottlenecks that we've seen before. So for that matter, we plan to collaborate with other initiatives for standardization such as MonAI. And we hope that we can— so MonAI is working already on standardized datasets, standardized evaluations and metrics, and vouching for reproducibility reproducibility and transparency in research. And we hope to collaborate with them and to integrate NNUnit there as a baseline so that it can really have maximum impact and move the field forward. And with that, I thank you for your attention. And the code is on GitHub if you want to have a look at it. Thank you.

Thanks a lot, Paul. That was an excellent talk. And I have to say, for me, this was one of the most impressive recent papers in the field. And I have to share one thing with you because I find, I found it quite courageous. You know that NNUnet first started with a decathlon that Paul mentioned. And for some reason, the decathlon allowed for basically having multiple test set submissions, which I don't like either. And I remember Fabian saying, no, I'm just gonna submit once. I don't wanna overfit on the test data. And he did that, and basically the team won, won with this approach. I found this really courageous and cool. And so I will start with one question, and then I will open the floor for the audience. And Paul, so you, you showed very clearly that the information provided in challenge papers, and probably also in general papers, is not sufficient to describe what people have actually done. And this is the problem that we face in the context of challenges assessment. So we want to, we want to learn as much as we can from international competitions, but it seems like currently the only thing we learn is roughly the accuracy that we can get with the method, methods around, and we don't understand, as you showed, basically why one method is better than another method. And I think this is a very tricky question, but do you have any ideas how we could address this question? I mean, what should be reported? How can we learn from the reporting better?

So, so the reason we showed everything on the Kids, on the Kids Challenge is because they were basically the only one who required proper technological reports of all leaderboard submissions. So you could kind of click through the leaderboard there was a paper and they had to describe which method they used and then their parameters and things like that. I mean, there are also substantial holes in that information. Not everyone did it as thoroughly as you could see in our graphic, but I think this is, this is a good way to go. No, like, um, if, if someone's interested in it, he can still scrape through it. I mean, of course, for people who want to do analysis on it, it would be nice if there would be a standardized way of just looking at things, but, but I think this would be asking too much from challenge organizers and participants. So I think that that's maybe a compromise, a common ground that we could move to. Yes, thank you. Can you hear me? Hi, yeah, Jonas. Hi. Okay, so yeah, just I asked this question because it's like very relevant, for example, for lung cancer, where they try to distinguish several lung nodes which can appear like a small or bigger and sometimes it's very challenging to really detect them and also to consider metastasis when they are in the very, very starting phase. And my question is more considering like, because the sensitivity of this segmentation is sometimes very important also for lung cancer patients or patients in the early stages, how sensitive it is or do you have any comparison based on this special challenge? Yeah, maybe you'd like me to respond?

Yeah.

So ML-UNIT really is not specifically well-tuned to work with large or small segmentation targets. So as long as you have a training dataset that has an appropriate number of training cases, it will be able to solve this task. And we have also applied MNUNet to some datasets that have these characteristics. For example, in the Medical Segmentation Decathlon, there is a dataset with lung nodules, and we have also applied MNUNet to MS lesion segmentation. And those are also very small spots in the brain that it needs to find.

Okay, and how— so do you have also comparison to other tools? Was there a significant difference or Was it?

Yes, we do, in fact, because these two datasets that I just mentioned are part of international challenges and they were also represented in the graph that Paul showed earlier. In the lung segmentation task, NNUnet took the first place, and on the MS segmentation task, it was among the top performing methods. It should be noted on the MS segmentation task that there was a lot of test set overfitting going on because some participants submitted 20+ times to the test set. So that's always a bit difficult to beat that.

Yes, of course. Okay, cool.

Thank you very much. Hi everyone.

It's fine.

I was just as an outsider very interested with how general or generalizable is this approach.

So I'm representing genomics and epigenomics. And in fact, we have pretty similar tasks. To image segmentation, but we have other types of data. So it would be interesting to know whether at which level can this be transferred as a concept which would require newer implementation, or one can reuse some of your work directly and so on.

So a one-to-one correspondence is obviously not given because that's an entirely different domain. NNU-Net was designed with semantic segmentation in mind, but as you said, many of the concepts that we used in NNU-Net can be transferred. For example, the way we approach the whole thing, which parameters we put into what category, how we are planning to optimize them, these things can certainly be transferred, and then the specific implementation of the specific set of rules or values that would need to be reoptimized optimized on the specific task that you have.

Yeah, so I agree. So I guess that the recipe that we propose is kind of on a domain agnostic level in a sense that you just have to find a sensible way between purely expert-driven and purely data-driven approaches. And then, I mean, as I said before, implementing the recipe is not just simply following steps, but in our case required years of experience in the field to find really these parameters that you can fix, for example, to shortcut. Okay, thank you very much.

Thanks. So Johannes asked a question that Fabian already replied to. Johannes, are you happy with the result, with the response, or do you want to discuss this more? The question was whether—

can you hear me?

Yes.

Yeah, go ahead. Yeah, so thanks for the response. I was more wondering about like light microscopy data referring to large histological sections, like on a scale of 10,000 by 10,000 pixels, such as common H&E slices or immunofluorescent stainings of whole organs, for instance.

So I know that some people within DKFZ are actually using NNUnet to segment these slices. And we have also participated— one of our group members has also participated in a competition in this area, and he actually compared NNUnet with other segmentation tools, for example by FastAI and other things they could find. And his— in his results, NNUnet was by far providing the best segmentation results on these kinds of images.

Cool.

Uh, can you, can you tell me the publication so I can look it up for Unfortunately, there is no publication.

The tool that is used within DKFZ is more something that some research group has built for themselves. I don't think that there is an associated publication, but I can try to figure out who that is and maybe bring you in contact. I know that they are segmenting vessels in these stains, and this is probably not the standard application for these images. I think you would probably be more interested in detecting tumor cells or other specifically stained cells, right?

Segmenting vessels or tracking vessels in 3D image comes pretty close.

I can try to figure out again who that was. I don't quite remember it right now, but if you have a specific segmentation problem, you can also just come talk to us and we can see if we can help you out. Out there.

Yeah. Okay, cool. May come back to it.

Thanks. Right. So, Santosh, how do you pronounce your name?

Yeah, yeah, that's correct.

So my question was almost similar to what Paul asked—

Paolo asked. It's like, I saw that the autotuning algorithm which outperformed the random grid hyperparameter search.

So would that be—

can we use that for any other machine learning models like deep learning or tree-based models for hyperparameter search? For some other domain as well, or else it's highly customizable for the medical imaging data?

I think it's kind of answered already, I think. Yeah, yeah, I would say again, so that the concept behind it of how you approach it and where you look for shortcuts, how do you weigh domain knowledge against data-driven learning, this concept I think here generalizes well, or we haven't tested it, but I would intuitively say it would generalize well. But implementing it requires then your expert knowledge in this field, and then that's currently only done for biomedical image segmentation.

He's having problems with his microphone, but I can maybe read the question and then answer to it. So Leo asked, you showed in the Kids task that you were at the top or close to it in all tasks. Did you check if there's another approach that is also close to the top on several tasks, or is there not enough information to do that? So to answer this question, I'd first like to highlight that we are very involved in semantic segmentation in the biomedical domain. So we closely keep track on what happens in the competitions and what kind of approaches are successful there. And what's really important to remember here is what Paul said earlier, is that these datasets from those different competitions are extremely different from each other. If somebody participates in a competition and they don't have a system like we do with NNUnet, they have to redo their entire pipeline for participating in a different competition. And what that leads to is that even if the same people submit to several different competitions, they will do so with very different approaches. What I know, for example, is that the person who won the BRATS 2018 challenge, that's the brain tumor challenge, brain tumor segmentation challenge also participated in the Kids Challenge and ranked in 9th place, but that was with totally different approaches. What I should highlight here is that what we have with NNUnet as a method that can just be applied to many different datasets is something that has never been attempted before and never been realized before. So there is really nothing you can compare it to on multiple tasks.

Great. Hi, Fabian and Paul. Great talk. Thanks. I was curious if you consider semantic segmentation a solved problem by now, at least in the biomedical domain.

Solved is maybe going a bit far, but I would say that as of right now, with the methods that we currently have at our disposal, at our disposal in combination with the small datasets we are dealing with, I don't think we will see a lot of progress in the coming years. But if we at some point get larger datasets where we can really show also minute differences between methods or where different types of methods can show their benefit over existing U-Net-like architectures, I think we can see a lot of progress again. And I'm also hoping for someone to come up with a really fantastic idea that kind of replaces the U-Net as the standard architecture because we have been optimizing and iterating through U-Net architectures for many years now and it would be just refreshing to have something new. So of course, my response now is regarding supervised segmentation. If you go beyond supervised learning and look into self-supervised pre-training or unsupervised learning approaches, I think a lot of success stories are to come, and I think this is also a very exciting area of research.

Yeah, and maybe just to add one thing. So currently, we train NNUnet from scratch at every new task, right? And maybe this is not the most smart approach. Maybe we can transfer knowledge between tasks, and this is also something that we want to look into.

I mean, I actually had a similar question in mind because I know you're very ambitious. So you want to win competition. So the question would be, now that you've released NNUnet, how can you— what's the strategy for winning the next challenges if everybody has access to this fully automatic method?

Well, yeah, we've seen already in MICAI 2019 and 2020, there was a progress of the leaderboards filling up with NNUnet implementations. So this year, I think many, many of the segmentation challenges featured a lot of NNUnets in the top entries. And so far, as I showed before, in this year we were able to win 2 challenges, and that is by, yeah, still having some domain knowledge where you can use NNUnit as a starting point and then optimize certain pipeline components to the task at hand. Thanks to, yeah, in one case Peter and his domain knowledge on cardiac segmentation, but also Fabian and his, his just very intuitive way of handling NNUnit, which is still ahead of other people using it out of the box.

Sorry if I, if I'm coming late. I didn't quite get the question, but from Paul's response, I can kind of imagine what it was.

Yeah. Yeah, now that everyone uses NNUnet, how do we still win challenges?

Yeah, so we don't participate in challenges where we don't have an idea on how we can improve on top of NNUnet. So if we see a challenge where, uh, the data is not just allowing for any improvements on top of what we already have with NNUnet, then I at least would not be very motivated in participating. So we really— we are looking at all the competitions, we are thinking about things that we can maybe do on top of NNUnet that would improve it. And if we find something, we participate, and if not, we let other people have it. Because in the end, if the test set is only 10 cases or so and there's 10 people submitting NNUnet, then one of them will win, and that's more of a random process than actual methodological contribution. So things we can, for example, do is, as in the case of the Bratz Challenge, which we won this year, is look at the way the segmentations are evaluated, and if there's something that doesn't match the way NNUnet selects for example, the best models, then we can try to optimize it towards the specific ranking of the specific metrics or combination of labels that are interesting for the challenge.

And also, maybe I forgot to mention, maybe it came across a little bit as AutoML has no future in my slides. And I would like to mention that we see AutoML as a very interesting research direction in the future. And we also see it complementary to NNUnit. You could also, for AutoML, since it is not holistic due to all the computation efforts, you could use NNUnit as a starting point and then just look at very, like, defined and clearly constrained pipeline components and optimize them empirically further. Because of course the configurations we find are not optimal at the moment, they're just very good and robust.

Thanks for being with us today. So we will have full house. So we are nearly 100 people joining and registering. So, um, yeah, I hope to get feedback from you. You can send me an email at d.hackmann@dkfz.de or subscribe to our mailing list.

It's data-science-seminar-join@dkfz-heidelberg.de.

And yeah, we would be happy to see you in 2 weeks. Again for the next talk. Everything would be then announced via email and also at our website. So thanks for being here and see you the week after next week.

So goodbye everyone, thanks for having us. Bye, thank you, bye-bye.