Scientist Stories: David Kuegler, Medical Imaging and Pose Estimation

Good morning everyone, this is the Data Science Seminar. So it's my great pleasure today to welcome David Kügler. He's a researcher at the DZNE in Bonn. There he works on deep learning for neuroimaging with the aim to improve MRI acquisitions. His work is ranging from anatomical segmentation to reconstruction tasks. In his PhD, David addressed the translation of AI on computer-assisted interventions to close the gap between deep learning method development and the reality of CRI application in datasets. His talk today discusses how the gap can be closed for instrument pose estimation, which is needed so robotic control can be used in further high-precision minimally invasive surgeries. David, the stage is yours.

Thank you. Yeah, thank you, and thank you for the invitation and opportunity to present my PhD work here at this point. So this is work that I've done in my PhD at the TU Darmstadt. I've now moved on to DZNE in Bonn, and yeah, I'm happy for the opportunity here. The talk itself is titled What to Learn in Instrument Pose Estimation, and yeah, I want to discuss a bit what— also what I learned, kind of. So I believe that AI is one of the pivotal technologies in medical imaging or the medical field and healthcare of the century and will change everything, will be a game changer. And that's why I'm really interested in moving this forward in a safe and secure way that is actually good for everybody, and that is for patients but also for doctors and people that are interested in helping patients. Generally, I'm looking at computer-assisted interventions, and computer-assisted interventions are the field where we are looking at surgery and the surgery room itself. And I guess most of you are familiar with that, but just to get everybody on the same level, so we typically have some imaging modalities in the operating room that range from CT to ultrasound, sometimes even MR, or oftentimes X-ray imaging, which I will be working, or I was working with in my PhD. Present on this work. But crucially, what often happens is that these tasks are very varied and have a lot of variety to them as compared to other tasks where the tasks are much more well-defined, much more clear-cut. So if we look at learning in a general context, so step back from the CHI perspective and look at learning in the general broad view, we can see that there's this kind of this field or this span between the application-centric view and a learning-centric view. And on the span, there is different ingredients that really help define the problem and work on this problem. It goes from dataset, which is on a related questions, which is on an application domain-centric view, all the way to the parameters of the learning and the training setup, which is on a learning-centric kind of scale of this view. And in between that, we have the mathematical model, the function that we use in our our learning setup to actually predict the outcomes or the value of interest. And these— so this spans a field, and learning specialists will focus on the learning side and will be good at the learning-centric things. But as CHI researchers that are using learning methods, we are in the advantage of of knowing that this is not everything, that we have to know more than that. So with CHI, as CHI researchers or as domain and applied researchers, we have to take more attention to the dataset-centric or the application-centric side, which would introduce aspects like annotations or images or how to evaluate. But it goes even further because really we're not interested in those direct outputs, but more in the bedside application. So in reality, we are interested in a surgical outcome in a CHI setting. And what we really have to now do is we have to span the whole spectrum of this learning site. And this is one of the key things which I think as researchers with learning background, but with a good knowledge of the application, which here is computer-assisted interventions. And we have to introduce the problem and to the question, and, and there, there our strength arises. And so that is something which formulates this problem of can we introduce learning methods into CHAI applications? And where are gaps that we need to close in this whole spectrum to actually connect learning methods with applications on the other side and make them work in this setup? So like I said before, CHAI has a variety of tasks that are part of the CHI field, and it goes from surgical workflow analysis to video analysis to subtasks like instrument pose estimation, which I'm showing on the very right here, where we're trying to find the position of an instrument such as this small instrument outlined here on an X-ray image, so we can then use this information to steer this instrument, may it be manual steering or in a more prospective case, robotic steering. So specifically, robots need to know where the instrument would be in order to act on the region of interest. And here, this can be obviously some robotic instrument that we could track, but it's also drills that we could want to track and get the pose of, or fiducials such as screws. And then we look at all these different assets, exemplary types of instruments. So if we If we look at how we would develop such a method, this process comes in multiple steps. And I think it's worthwhile looking at this process in order to understand where contributions can be made and also to kind of put contributions that I will be presenting in this talk in a bit of context. So the general process of creating a new method for learning in CHI, or in any field really, kind of does translate, is that we first have to define the tasks that we want to follow. So this might be like a segmentation task or the post-estimation task. But this definition does not only include things like segmentation, but it includes things like how is the general layout of this task. So we could do multiple networks that are cascaded, or we could do a preprocessing step and then some network and then a post-processing step. So it's a Is it end-to-end or is it in a different setup, basically? And that's the first kind of consideration that we have to do. And obviously, which losses are we using and things like that would also kind of follow into this field. The second step then is that we generate a dataset and we come up with augmentation schemes to, outside of the learning setup, help our dataset and help the kind of outside learning questions. Then the third step is that we need to find a neural architecture which will fit into those rough blocks that we defined in the task design And we have to find an architecture that is fitting for this and that will perform well, or best even, with regards to this task that we have defined. And finally, obviously, there comes the training step where oftentimes we're just kind of letting it run, but that obviously has to be considered as well. That is a very learning-centric task if you if we look at the scale between application-centric and learning-centric tasks, subproblems. So in this talk, I will talk about the work that I have done with i3posNet, and there I will talk about the subfields of design of the task and the dataset and documentation. Then I will talk about a second work that was submitted or— submitted not, was presented at and published at MICAI 2020. And that will look a bit at the task design, but in a different context and at the neural architecture. And finally, I will just give a kind of an overview of where I'm going now. But this will be very short. So with that kind of introduced, I want to start showing our i3pos network. So generally I've said that we're doing instrument pose estimation, but for which kind of problem are we doing instrument pose estimation? Estimation. And this is for temporal bone surgery, and that is the bone that houses the sense of equilibrium, the ear, and obviously some nerves and arteries. And this is a prospective robotic surgery location or region of interest, and we were interested in introducing a robotic surgery scenario for that. And this was supposed to be a minimally invasive surgery, and we're requiring extreme precision because, um, in the, the dimensions are just very, very small. And for this, um, scenario, we would then have, um, the instrument and a screw used as fiducial And we would want to find the position of these with respect to each other. And what we're now doing, wanted to do, is achieve it as well, and is find the position of these objects on the X-rays. And if we look at this problem, we can see that we're trying to go to the cochlea, which is now pointed at in this green arrow, or we are trying to go through the labyrinth, so it's kind of through this circular thing here. And one of the kind of most critical points to go through is this facial recess, which is this gap between those two nerves. And this is a crucial bit that actually leads to high precision requirements, and the kind of published requirement is better than 0.5mm in accuracy, otherwise you would harm these nerves which could lead to facial paresis. So in the end, the task is to find the drilling robot or the screw on such an X-ray like it is shown on So the challenges in this is that obviously it's surgery, so it's life at stake. And then there is no real dataset for this task available. Images are typically very noisy in X-ray imaging. Annotation is extremely difficult because there is no real way to do non-manual annotation. In the end, if we simulate, we have a domain transfer problem. We have extremely high performance requirements, and there is no standard established method ready and off the shelf. And overall, this is a very multidisciplinary approach. Together, we can see that at this point, this task isn't just AI ready. We can't just use a learning method and just plug it in and we're basically done. But there are so many questions and gaps and challenges that stop us from directly implementing something that is quite a gap to fill. And I believe that this is common for many CHI problems. So the traditional approach would obviously be to just do a registration. Where we're basically doing on-the-fly X-ray calculation from our model. We would compare that in registration loop, and then we would just minimize the error of the images and in that way find the pose of the instruments. And we have done that and published that in a 2018 paper that shows this can be done, but the actual quality is not actually fulfilling the submillimeter accuracy requirements. So with that all said, we have to define our own problem here, and what we're doing now is we're saying, okay, so we look at this whole process and separate it in some substeps, and those substeps are First we do a localization, then we do the pose estimation, and then finally we do a reprojection. So if we separate this, this process into these subtasks, then we have like the initial X-ray image in the very beginning, and then we get a localized position which is like 2mm accurate position, and this is kind of If we're 2mm away, then we're already in a very bad situation. So kind of, this is really a kind of a safe estimate that we're 2mm away. And from this 2mm accuracy position, we're then estimating a much, much higher precision pose, and that then can be reprojected to the patient coordinate system in the setup. And the idea here is that we want to look at this middle step in our i3posnet work and really improve this middle step because we believe that this has the highest leverage in the whole chain. So this is where we introduce our i3posnet methodology to really improve this. And there's localization methods already out there on on the market that we could just use as a plug-and-play method. And the region prediction problem is also kind of a mathematically solved issue, so that those are already kind of there. But this middle thing would have been done by registration before, and that just isn't good enough. So this is why we choose this to really improve this. And so what we're actually then looking at is we're looking at having kind of an initial position on the X-ray, and what we want to generate from this is full pose of this instrument. This full pose includes the X and Y position on the X-ray plane, and it includes things like the forward angle, which is this alpha angle, which is orientation and the tilt of the instrument with respect to the imaging direction, and then the depth as well. And for this, we have to generate a dataset. And in this work, we generate a 3-step dataset, and this is consistent of an A-type, which is, um, which is consisting of synthetic images And we can see that we have this screw which is kind of well visible in this left image here. And then we have the robotic and drill to see if it generalizes to other types of instruments. And finally, we have the dataset that we acquired using a real X-ray setup and that also use the same screw as the dataset A. But while the synthetic images can be annotated from the geometry of our simulated world, the real X-ray we have to manually annotate. And this is not trivial, as I've indicated before. I mean, the challenge is really where is the instrument, and we can see that that's the instrument. This is a zoomed-in kind of view. I mean, then it becomes much, much more difficult to find the instrument in the full view as well. And yes, we can find it like there. So the challenge here really is to— the quality and the the type of X-ray imaging really leads to some really difficult-to-work-with data in that sense. And we publish this dataset on our webpage as well, so this is open science-wise published and free to use for everybody, all three datasets actually. So going back to what we looking at in the terms of notation. And obviously in the end we want to know the position with respect to the patient, but this is really difficult to do in many ways. So the best way to actually work with this is to do it in the image space we found. So, and that, or more likely the projection space, which then includes the dimensions of the depth and the tilt with respect to the projection axes. And if we introduce these in projection space, again, for the synthetic and in annotation, we have the perfect knowledge, but for real X-rays, we really— it's difficult to find these values. And we have to contend with non-perfect annotation there. So this non-perfect annotation really makes the learning problem quite very difficult as well. And we'll go into this in the future as well. So with this method that we this dataset we have, we now in step 1, which is we have the problem design kind of task, and we want to identify a problem setup that we can then use to solve this problem. And obviously we could just put it in, the numbers in and images, and then hope that the right thing comes out. But it turns out that this is actually too effective. So in our solution then, we say that we know an initial pose. So let's use this information to make the problem more streamlined and more accurate, actually. So we start with an initial pose estimate, and then we crop around the initial pose that we have, which is kind of marked here in blue. So this is kind of the error that we have in the beginning, and with that cropping we have now an initial position that we can correct for. And when we correct properly in this scenario, then we can iteratively find the position to be more accurate. And this is exactly what we're doing. We now have a network here which is predicting the should-be positions of these landmarks, which I'll go into in a second, what landmarks. But we were using those landmarks now to predict where it should be. Then we can recalculate or reconstruct what the post should have been originally. And based on that, we can then iterate over this and iteratively find a very good estimate of where the instrument actually was. And this iterative setup actually allows us to really achieve very, very high accuracy. So the general setup of knowing initial position and then being able to refine this initial position really allows us to iteratively improve the position here. And that's the general method here. So when we actually evaluate this, and this is in comparison evaluated on pixels now, on the accuracy here, and these two right ones are the registration methods that were used prior to this. And like I've said, there is no direct comparison in a deep learning sense. Then we can see when we introduce our our learning-based approaches that we are significantly beating the prior registration approaches. And in specific, this iteration scheme also gives us additional performance. So the i2-PosNet is just doing the first guess, and the i3-PosNet then is to actually iterate over this first guess and refine the position. And this actually leads us to some very nice and clearly sub-millimeter accuracy requirements, so that would be good enough for this specific task. There are some limitations to this. Unfortunately, this top bar doesn't kind of go away here, so I don't know if this is hopefully not too much in the way. So if we analyze these, these errors now with respect to the projection angle— and remember, the projection angle is, is this tilt angle, and this kind of figure up there is showing how the tau changes, so that's the changing projection angle— we can see that if this projection angle is close to 90 degrees, which this is the case here now, then the instrument axis and the projection axis almost align, and that makes it very difficult even for us to figure out this angle alpha here because we don't really have anything to kind of follow up on. And our method actually observes the same problems. So, um, with, um, this, this approach, what we can see is that for large absolute value values of tau, the estimation error of the forward angle is actually kind of really declining and really struggling. And again, if we look at some of the sample images, we can also appreciate why that would be the case. Here, this is the, the drill bit that that we're projecting through the drill, but basically imagine this is, we have detached the machine that is the motor that is behind the drill. We have detached that and we're now looking at the image as if this was just the drill bit and looking down the drill. And obviously in these cases it would be difficult to see which direction this drill is now looking in. You could kind of argue maybe it's looking down right, but it could also be looking left more. So it's kind of these— the direction really is unclear to a large extent here. So it's not completely unexpected that this would fail in these extreme angles. So we exclude these extreme angles in our analysis because this really biases the results quite a bit towards this extreme scenario, which we can just exclude as as part of our analysis here. But— no, it's not moving forward. OK. So now I want to look at the question of why is landmarks actually the way to go. And I've just— up to now, I just said that this is the way to go. But really understanding why this is the case goes towards why the problem design is actually crucial and can actually help quite a bit. So in these graphs, on the left we have an analysis of the forward angle using landmarks, and on the right we have one that is using the prediction of the angle directly. So on the left we have just these 6 landmarks that are individually predicted, and on the right we have the central landmark and then the angle predicted directly. And we can see that while at the center this works nicely and the errors are okay, when we actually go towards the more extreme values, we tend to have a lot more errors. And what we can understand here is that it's kind of always almost more predicting towards the average of the angle. So if we predict the angle to be 0, although it should not be 0, then we would expect the error to be down here. And so this is what is happening. So it can't really discriminate in all cases what the angle should be, whereas if we do it through landmarks, it's much, much, much better. So the overall angle error is much, much lower. So, um, so it's crucial what we use as the output of the network, and landmarks actually outperforms specifically because of these features. And if we now look at like— this is the forward angle over the initial forward error, the initial error. So this is kind of the setup that we give in. If you use this, this in the, in the same setup for the projection angle, this is the same angle tau again, and we see that this is exactly the same. So if we directly predict it, we get really large errors for larger errors. So it's predicting, or it tends to predict, the average value here. And if we go through landmarks, then this is actually solved. And to understand this better, we actually did a toy experiment with a toy dataset where it's just one— the toy dataset is just a rectangle with one side rounded, so it has 360 degrees of rotation. And we put it in a random position in these small images, and we draw a random position and a random forward angle and draw this as a toy setup. I don't know how this goes away, but, um, so if we do this, then we, um, we define the task to predict this forward angle, which is the orientation of this, this small box. Now here we can compare the approach of doing— going through landmarks or going through angles in a direct or indirect way as well. So this is the angle alpha again, right? And so number one is what we can see is that there is this discontinuity at 0 to 360 degrees. That's kind of expected that maybe we're not hitting, we're not actually accurate between 0 and 360 degrees, but in our evaluation we actually correct for this.. So basically, if the predicted value is 1 degree and the correct value would be 395 degrees, then the correct evaluation here would say it's 2 errors difference or 2 degrees error. So this phenomenon is not actually what is explaining those errors, although it explains a lot of the of the errors here in this bar in the way that it's difficult for the network to handle this discontinuity in the data. It's not evaluation, this is just the difficulty of the network to handle this discontinuity. What we can even see is that if we compare this direct landmark approach, a direct angle approach, and indirect landmark approach, that the average error is even worse, and significantly so. So we can see that it's not just a problem of just this specific discontinuity, but it's actually a problem that is observed throughout the scenario. And so we can see that it's really crucial that we understand our problem and we find the right description of the data that we want to predict. And then with that, I want to go to the next part. So this is the i3posnet first, and this is now AutoSnap. AutoSnap, again, here we are looking at the task design and neural architecture. Task design is in a bit of a different way because we're not changing the i3posnet design, but we're doing architecture search, and I'll go into that in a second. And in the continuum between application and and learning. This is towards learning, trying to find the right architecture, but also it's problem design in how to do that, actually. So i3PosNet again was this setup where we had initial pose, we cropped on the X-ray, and then we reconstructed the pose from the landmark prediction, and from that we updated our position. But this is a network, but which network should we use? Up to now I have kind of left this out, and up to now this was just the VGG network, but I want to now go into which network we should use here in a bit more depth. And we look at kind of what the input here is, then this is again this image that we start with, and here it's the, the pose that we have the output. And again, remembering real quick that it's predicting landmarks, um, here as well. And if we look at typical, um, problems in, in recent times, then we can see that there is a clear search towards learning methods in, in CHI, but there seem to be also some approaches that find their own specific architectures and their own specific variations of established architectures. So here are just two examples. So with the clear majority of papers in CHI, at MICHI 2020, using deep learning, I believe there is a need for finding your best architecture and helping you find the architecture that is the best suited to your problem. Because, well, almost everybody is using deep learning, so there should be help for researchers to see which architecture would be best. So I'm looking back at this overall design now. We're kind of— for a general CHI problem, we are in this field, which the architecture, neural architecture question, which architecture should we use? We're trying to frame it in a way that we can actually get a method to help us with that. And looking back at I3PosNet, this method of finding this architecture was pretty simple. It was doing a big matrix and just trying out different variations of the architecture. And this is just variations of the VGG architecture. And if we look at time that I spent trying to figure out which architecture to use, this represents about 50 GPU days of manual search and manual finding different options here. So this is a considerable investment trying to find the numbers, and it has impact in terms of performance as well. But obviously that's a very inefficient way of going about the problem. So what could be an intuitive idea to improve this approach? Well, the intuitive idea is that we could have a neural network. We could then use a description of a neural network. So this first neural network is describing the kind of network we would want to use, and we could have a value estimator neural network that is just a value estimator that that would predict the performance for such a network. And then if we trained this value estimator, we might actually assume that this generalizes and just plug in different new networks and wouldn't have to train this and could then just predict performance of unseen network architectures in a standard neural deep learning way. So the intuitive idea here is to predict the performance. The question obviously is then, how do you benchmark? How do you actually say this is an architecture? Like, how do you discriminate between architectures and describe architectures? And then how do you find architectures efficiently? And these are the problems with this kind of setup here. But I guess we can just introduce a benchmarking task or a thing that we could do. And then we can just benchmark with data and using the architecture from the left side. We can build, train, evaluate that architecture. Then we get performance estimates. So that is doable, I guess. But it has to be efficient, right? So otherwise— will have hundreds and hundreds of hours of GPU training done and no actual benefit. So the question is, how do we do these steps? And how do we actually encode this architecture as well and do this through a block here? So the idea is that we'll just have a kind of framework, a general framework that represents the architecture. And in the ResNet kind of way where we have a repeating block, we'll look at which block we should use. So this generally is called neural architecture search, and there are established approaches that we could use. For example, there's the discrete approach, which basically is in a scenario where you test and then improve on the architecture. So this is the basic block design, or the basic block element, that is used in the Nussnet approach. And we test one, and then we see how we can improve in a recurrent network or in a reinforcement learning scheme. Reinforcement learning scheme, but it is a recurrent design that is behind this to predict what to use as the as this block. This is repeated B times because this block can be repeated B times. But the NASNet takes 200 GPU— 2,000 GPU days. There is an improvement over NASNet which is called Nownet, which takes 200 GPU days. That's basically infeasible for many wide applications. Then we have a continuous approach which is the DART approach, where we basically learn all possibilities. Basically, we build the network tree and we superimpose all operations that could be in any of those nodes. And superimposing them, then doing a sum, a weighted sum after that, to really break it down after that again. And this weighted sum then will actually lead to, to set up where through choosing the element that had the highest weight in the weighted sum, we discretize this architecture and get one architecture instead of this sum of all possible architectures. And this is the the continuous approach where we train the continuous model that includes all possibilities and then discretize the best version of that. And that takes roughly 4 GPU days, so that's much, much quicker. But it requires— since all operations need to be in memory so the gradients can be propagated, it requires extremely high video memory numbers, so just because all the values need to be stored in memory. So our method then has two contributions here, the AutoSnap method, and that is one, this how do you represent the architecture, and two, how do you search better architecture. And How do we represent architectures? And we start with a symbolic dictionary, and each of these symbols can be— this is like a language of describing neural architectures, and we use multiple symbols, and these symbols then will describe the architecture. And just some examples how this would work out: if we have this the blank language, we actually just have implicit merging. So what's important here to realize is that we don't just give the one input, we also give the previous block output, so that the block before it also has an additional input to this block. So it has a bit more ability to form skip connections that way. Um, so generally it's just an implicit merge add, so it's one output. Well, this is an almost in ResNet, um, how we would describe it as a branch, uh, convolution 3D, convolution, uh, 3D convolution 2D in a 3x3 kernel. And if we do that, this is how the network looks like. And so we can represent this, but we can also do some random architecture where we do a depthwise convolution, then some topological statements, and then some more convolutions. So topological statements, that's, that's one of the, the key additions in, in this language that previous things haven't done so explicitly, is that we introduce 3 symbols with branch, merge, and switch, which actually change the topology of this block here. So branch introduces this branch operation here it's just doing this. And then merge is merging to the two branches, and they are the most recent, to again with this operation. And then switch is, it's changing the active operation. So this is the operation that we would add the next block to or apply the next block to. In practice, what we do is we We have a stack-based language behind this where all these operations will operate on the stack of tensors. So these will always be working on the top element of the stack, and these will change how the stack is looked. This duplicates the top element of the stack, this merges the two top elements of the stack, and this switch operation then flips the top element of the stack. And with that, we can actually effectively describe in an 8-symbol string most architectures. And so with this set scenario, we could then train the neural network as a value estimator, at least so that that would be at least fixed. But we don't want to stop there. We want to go forward and actually introduce ways to actually search in this space as well. And what we can do without problem is just split this value estimator into two parts and have a latent space. And with that latent space, then we can tie— kind of go around the circle, go around and go back to the snap representation. This will allow us to actually search effectively. Also How would this work? Well, the point here is that we can actually look at an architecture in the z-space and generate the corresponding actual model from it using the decoder and then building it in the benchmark. So first, let's look at how we would train this, and I'll have to move this bar again. It's not, unfortunately, not going away. So how we would train this is, well, I've already introduced this MSE loss here for the performance, but we also have to contend with this autoencoder up at the top to actually train that. And there we have first the reconstruction loss, which is cross-entropy loss here to reconstruct the snap description, snap string that we put in into this design up here. But also what we can do is we can enforce a cycle consistency where the specific z value will always lead to the same Z value if we go the other way around, so decoder-encoder instead of encoder-decoder. And that actually improves the reconstruction and the search possibilities a bit as well in the later setups. So this makes sure that the same, like, one Z value will always lead to the same Z value after going through decoder and encoder. Um, so the, the— now we have this, this whole design of how we actually learn the setup. And the, um, the kind of last question is, so, so now how do we search for new architectures? And now this z-space comes into play a bit more prominently, um, because what we'll actually do is we will benchmark a specific architecture. So we have kind of a known good architecture, and if this network is trained in this way that we've just shown, what actually happens is that we can, in this optimization space which is in between, we can actually use a gradient descent on that values and that way get better values. So what we're doing is we're using the performance estimate as kind of the height here, and the x and y directions of this mesh are two directions of the z vector here, basically. So the— we're doing— we're using the gradients of the value estimator to find the direction of best improving performance in z-space. And since we can then use this updated z-value to update— we'll update it multiple times, multiple steps— we'll actually use this updated z-value then to generate a new SNAP block. And then we can use this new SNAP block to actually benchmark it again to bolster our number of different architectures and designs that we have. And that actually is then the method. So this is the best SNAP that we have found in the end. So we've constricted ourselves to 8 symbols, and we can see that this is somewhat similar to a ResNet or a double ResNet in a way, but it repeats this block here twice. So, and if we see what the the— we compare this with DARTS, which is, I should say, both take about the same time for training, or we constrict both of them to take the same time for training. And we can compare the different performance metrics that we've had for the I3-PosNet design here as well. And we've normalized these bars because they're all in different scales, and we've normalized them with respect to the I3-PosNet results. And we can see that in the position and in the synthetic benchmark, DARTS was actually significantly worse than the I3-PosNet architecture. But our architectures that we found are actually significantly better than— significant improvement over the I3-PosNet, which is the VGG backbone network. And what we can then see as well is that these two different architectures here, A and B, are different sizes of architectures. So there are different number of filters and feature maps. That's the difference between those two architectures here. Just show how that scales with the number. And there's still— both of them are actually less parameters than the i3-posNet design. So now I want to look at this, the search results, once more and just get the search history in here as well. This value metric is just the log of the mean squared reconstruction error. And it gives us a bit of a resolution of how good we are. And we have compared this with our search with the random search, which is a typical example of just dumb search in this field. And because this lets us more easily see that the search— and what we can see is that basically after 8 hours roughly, we are already saturated in our performance. So this also does show us that this is a very effective approach in the beginning. So these first 100 iterations are initialization, just are randomly sampled. And after that, we actually start our method and we can see that immediately after we have a steep increase in performance And this is the top 5 performance, so we can see this a bit more smooth. And you can see that this is significantly improving quite quickly. And we can also see that after about 6, about 8 hours, the improvement really doesn't go much further. So what we can actually gather from this is that our search is so effective that it kind of comes to a, optimum quite quickly. Obviously, and this could be still a problem of our— or limitation of our method— is, are we in a local optimum? And that is difficult to say because, well, if we sample for a very, very long time random samples, that really doesn't we don't get a better result really. So when we just randomly sample dumbly, we don't get a better result that quickly. So that leads me to believe that this is already a very, very good result. And again, just to illustrate that this is not a random one outlier, this is the top 5 and this is the top 25 different architectures. So we can see that there are other architectures that actually improve as well. Um, right, um, I don't remember what I want to say here. Um, so finally, um, I want to just give a big glimpse of a thing what I'm kind of moving towards now. This will be very quick, but, um, just as an overview and The group that I'm at now is very focused on segmentation for anatomical neural structures, and FastSurfer is one of the methods that we have introduced this year that is doing quite nice results, and it's well received by the community. So I will follow up on this and try to introduce some of the ideas that I've presented here as well into this work. And then we have done some learning on cortical surfaces, and there we are comparing learning in the data structures of the meshes with learning on the data structure of actually projecting these meshes onto an image and how that actually performs. Or We have worked with segmentation in diffusion imaging and diffusion data and have actually achieved some very nice results here and have analyzed how different changes in what the data for the diffusion imaging is used, how that influences the performance of the segmentation network. So that's like, how many directions do I use in my diffusion imaging protocol? Or give it— do I give the diffusion directions directly as a raw image, or do I compute tensors from it, or do I do an FFA analysis of it? And how does that actually translate to segmentation performance in a network after that? So concluding, I want to come back to this learning method overview. And I want to ask, so what is there to learn in instrument pose estimation? And answer that with the field of task design. There is— I learned that we can include domain and should include domain knowledge in the task design, and we can do that by data representations and by losses. And later representations, that's like how— what's the output? The standard for segmentation and classification here is softmax and cross-entropy. And for the post-estimation question, this was landmarks instead of directly predicting, as one example. And for the AutoSNAP design, the idea even extended further, as we didn't just formulate a representation for input and output for the snap, but also for the middle part of the network. And we enforce this by losses, and this really helped us in designing this method. And without that optimization state space, this wouldn't have worked. Then in the field of dataset augmentation, I learned that normalization is significant and important to use and has the highest return on investment in terms of time. So normalization, for example, on the output coordinates, normalizing them, the values here, to normalization interval, or increasing data augmentation is significantly as this next step. And as the final step, since data annotation is just so damn expensive, obviously helps but has the least actual effect on return on investment. And finally, if we look at the neural architecture search and comparing those 50 GPU hours that I manually invested and comparing that with this 4— sorry, 50 GPU days that I manually invested, and then the 4 GPU days that the machine ran, and significantly outperforming my results, I come to the conclusion that doing this manually is futile. So in this work, I contributed many things among the problem design for instrument pose estimation and a framework for automatically learning neural architectures for CHI. I3PosNet dataset, I contributed that that we realized that learning end-to-end is not always the best way. We came up with ways to infuse knowledge into the learning, and we introduced a new architecture representation, that's the SnapLock, and the efficient search and optimization. And finally, all that led to the sought-for submillimeter performance that we wanted for this task. Thank you for your attention, and I'm now open to questions.

Yeah, so thank you for this great talk about the MaileDI 3-PostNet, and this is also the paper for which you got the award for the best machine learning paper at the IPKEI in this year. So many thanks. For an introduction in this. So the question to the audience, yeah, you can directly post your questions to David, so feel free to also switch on your video that we can see you. And the first question will be asked by Tim Adler.

Hi, can you hear me? I can. Okay, great. So thanks for the nice talk. So my question is more of an outlook kind of question. So what steps would be still necessary to get i3-posNet into a medical product or to actually let it be used? So I think the most significant gap is the domain training. Transfer problem, really. And I think we— I don't know if these results are in the— I don't know which slide this was. Here we can see this in the slide that where the errors are really kind of listed in a more— because all these errors have different kind of scales, so it's difficult to bring them in one graphic. And what we can see is that from dataset A to dataset C, we have a significant loss in performance. So this is the generalization from simulated data or synthetic data to actual real-world measurements or real-world X-rays. And while for i3PosNet, we didn't quite break the 0.5mm threshold here, For AutoSnap, that was actually better, and it's about half of these numbers. So it's significantly better already. But still, I think to really bring this into application, this still needs to improve in this data, this domain transfer question of really getting the performance translate from the synthetic data to the the real-world data. Thank you. Maybe as a quick follow-up, do you also have, I don't know, time constraints or real-time constraints? So do you know anything of that? So we take about 50 milliseconds for one evaluation. So I don't think that's really an issue. OK, thank you. For the neural architecture search, do you have any intuition why the architectures you found are so good as they are. So we started out with the idea of, we'll just, with the intuitive idea, we'll just understand better what the architecture, which kind of architectures are good, and then went a step further and just said, well, we can just do this automatically, and came up with this idea of propagating the gradients and using the gradients for that. But that kind of removes the direct understanding of why these architectures are better or worse. And I think that's one of the crucial conclusions of this work as well, that that's not really the point. If you can get the computer to understand what's good and what's not good, just let the computer do that and learn that. Let that be part of the learning architecture. And I think this is common across all the neural architecture search approaches, that they don't really care anymore why a specific architecture is good. They just know it's good and therefore use it. So it's just empirical. Yeah, that's true, but it's a bit contrary to this whole, I don't know, explainability movement, I would say, right? This is completely contrary to explainability in the sense that you cannot actually explain why a specific architecture is used. Sure, but then again, we wouldn't want to explain— like, that's— I don't think we are really interested in explaining the architecture. We're interested in explaining the diagnosis for sure, but I don't think we're interested in explaining architectures. I mean, sure, you could probably do that and then get some interesting insights, but I've never thought of it this way.

Thank you. So we are now at the end of our seminar today. Thank you very much, David. Bye.