Scientist Stories: Mathias Unberath, Tracking Endoscopic Surgery

Welcome to a new edition of the NCT Data Science Seminar. Today we cordially welcome Matthias Umberath. Matthias is an assistant professor at the Department of Computer Science at Johns Hopkins University, core faculty in the Laboratory for Computational Sensing and Robotics, and fellow of the Malone Center for Engineering in Healthcare. He holds a bachelor in physics, master in optical technologies, and a PhD in computer science from Universität Erlangen-Nürnberg, where he graduated summa cum laude in 2017. Matthias has also broad experience as a student board officer responsible for different tasks at several MICAI conferences. Within his group, the Advanced Robotics and Computationally Augmented Environments— see his t-shirt, ARCADELEP— he advances healthcare by creating collaborative intelligent systems that support clinical workflows. Today, Matthias will talk about advances in scene reconstruction and tracking for endoscopic surgery. It's a great pleasure to have you here. Well, Matthias, the stage is now yours.

Thank you so much for the kind introduction and having me here. It's always a pleasure to speak in Germany because those are usually the talks where people get my name right, so that's wonderful. And I have decided to talk a bit about the recent advances that my group has been working on, on computer-aided interventions and computer vision-based surgical assistance systems. And this, let me see, here we go. So this is my group. It's the Arcade Lab. I have right now 1.5 postdocs and 13 PhD students that work with me. We have a, here at Hopkins, one thing that I think is, well, I don't know if it's so special, but I personally appreciate it as being special is the fact that our research happens not so much in departments, it happens at centers. And so what that means is that the people that come to us, they are are co-advised by people with multidisciplinary backgrounds. So most of the students that you see here have a co-advisor either in mechanical engineering, if their work is more on robotic systems, they might have co-advisors even in medicine if their work is more clinically and translationally oriented, or other people in computer science with completely different backgrounds so that they receive a more broad training and broader experiences. And so that's why, well, this endeavor scales to the size that you can see here. And today I will be mostly presenting work from two of my students, Max Li and Hao Ding, who have been working with me for 2 to 3 years. And then this is work in collaboration with two skull base surgeons in otolaryngology, Masaru Ishii and Pete Creighton. They are both interested in skull base. Masaru through frontal approaches through the sinus and Pete with lateral skull base surgery, so essentially from, from the back of the skull. All right, so everybody here probably has, has already heard that the health system performance has gaping divides in outcome, and that can be due to all sorts of reasons. In the specific case of surgery, those reasons mostly correlate with experience of the operating surgeon and the procedural volume that essentially affect the outcomes of the patients. That's not necessarily because providers and surgeons with a lower procedural volume cannot perform procedures. It's just that they perform certain procedures much less often, which means that the variability in their outcome, the variability in their approach, and so on is much, much higher. And so one idea and one technological branch that has been designed to reduce this variability in approach and therefore also the variability in outcomes is the introduction of surgical navigation systems and intelligent assistance systems that somehow try to dampen and reduce the variability in the surgical approach and outcomes, and therefore dampen this impact that experience and procedural volume has on patient outcomes. Now, this is certainly an exciting avenue. The problem, however, is that the conventional surgical navigation systems like the one that you can see here at the back with an optical tracker markers that are attached to the instruments as well as the patient anatomy have all sorts of problems. They are expensive. They are obtrusive because of all this marker assembly that I've just talked about. And as a consequence, because they introduce complicated calibration workflows, they introduce additional costs, they are not really widely adopted. They are particularly not widely adopted at the places that would need them most because those places that have a lower enough procedural volume for specialized cases for which you would use such a system, they can't really justify affording and buying such a system simply because they don't see sufficiently many patients. In addition, these conventional surgical navigation systems, again, based on optical tracking or electromagnetic tracking, they have limits in their accuracy. And that's for multiple reasons. The first reason is that, as you can see in the picture here in the back, the markers are usually attached somewhere far away from where the action actually happens. In this case, this is an endoscope, so the camera center is very different— is at a location very different from where the optical marker is attached. And so that introduces long lever arms that even if the calibration and the tracking of those spheres is precisely accurate, is very accurate, due to the long lever arm, error propagation results in pretty large errors in those systems. On top of that, most of the calibration here, closing this calibration loop so that we can track the tools relative to the anatomy happens once for the surgery at the very beginning. And so over time, all of these calibration transformations may in fact drift, which limits the accuracy of such systems. Further, there are additional issues with these systems, which is the fact that if we talk about laparoscopic surgery and endoscopic surgery, These systems are fundamentally limited in that they cannot handle deformable soft tissue, which is what we manipulate a lot of the time. And so that is particularly due to the fact that all of the systems that I talked about earlier work by attaching rigid bodies to the objects of interest. And so if something is deformable, we can't really attach a rigid body to it. Now, this is where image-based surgical navigation comes in. Because images are already used for many of these procedures that I talked about, and they provide dense information that is good enough to inform surgeons and providers about the actions that they should be taking next. So if we could essentially use this information to analyze it in sophisticated ways and derive the information that a surgical navigation system otherwise would derive using specified hardware, then really we would have means of overcoming the challenges that I was talking about. Now, this would then be the end of the story, would this be easy, but unfortunately developing systems based on images that somehow reliably over a long period of time in surgery reliably track the tool-to-tissue relationships is incredibly complicated. And so today I will be talking about some of the many open challenges that image-based surgical navigation faces. And I will try to highlight some of the approaches that my group here has taken of trying to address them. And for the purposes of this talk, I have these 3 larger topics. The first one is where I will be talking about generating 3-dimensional representations of the surgical scene from images and videos. This will then inform the second step that I will be talking about, which is the attempt of recovering relative spatial transformations between anatomy and tools solely from images and videos. And then towards the very end, I will talk about something that is emerging, emerging research topic in my group, which is semantic data analysis. In this case, it's multimodal for robotic surgery where we talk about segmentation, but not only from images. So that's something that I am excited about. There are, of course, so these are by far not the only challenges that exist if we want to talk about image-based surgical navigation. There are many more such as deformable tissue tracking, advanced imaging, visual feedback, mixed reality, and all sorts of other things that I will not have the time to talk about here. But if you are interested in those, I'll be happy to discuss them with you after. Okay. So let's first talk about 3D scene reconstruction, and I'll talk about a recent approach that we have put forward that uses transformers. So when we think about this endoscopic and laparoscopic surgery, most of the time the, we don't really have 3D vision. The endoscopes that we're using, they are either monocular or in some cases, for example, in da Vinci type surgery or also for microscopy, we have stereo cameras. Stereo cameras are nice because if we have a stereo camera assembly, then when we calibrate it, we essentially have this nice geometric relationships that you can see here at the bottom line where Using these geometric relationships between these two planes after calibration of our two cameras that observe the same scene, we essentially can reconstruct three-dimensional relationships. And that's because there is essentially this epipolar plane, this concept of epipolar planes, which is a plane that connects the points in 3D space that we are interested in and the two camera centers. And so we can derive certain geometric constraints of where points that lie on this plane or in 3D space then can can lie in these individual images. So really what that means is that the plane defines this epipolar line, which means that all the points that are in this one line need to be— need to lie also on the exact same corresponding line in the other image. And so what we need to do essentially for 3D reconstruction is then just identify which point on the line in the left image corresponds to which point on the line in the right image. And once we have understood where these two points are, we can essentially triangulate in 3D space. Now, one concept that is important to understand is this concept of disparity, which essentially just means that if I were looking at my scene using exactly the same camera, then essentially the corresponding point would map onto exactly the same pixel. So I would have a disparity of 0. Now, because I move my camera, that specific point will map onto a different pixel in that space.. But if I have the same lines now, what I'm going to describe as disparity is essentially a pixel shift between the pixel in the one image to the pixel in the other. So this is essentially just a difference in where on the epipolar line I observe corresponding points. And information about disparity essentially allows me to triangulate in 3D space. So that just means that if I tell you I have a left, a right image, and a disparity image, then from that information I can reconstruct 3D space. And that will be important to remember as I talk about some of the derived algorithms later on. So this sounds relatively easy, right? I have a left image. I have a right image. I match points from left and right, and then I have 3D space. Nice. Well, unfortunately, it turns out that densely matching these points is incredibly complicated. That's because natural scenes turn out to be piecewise constant. So you have some edges, and you have corners, and they carry a lot of information. But in between, you don't really have anything, right? It's texture-scarce. There's nothing there for you to precisely identify which points correspond to one another. And so densely— doing this densely for every single point on your line turns out to be complicated. Usually what we do then for this matching is we match points independently. So we look at points on the left line, and we go and we try to identify matches on the right point. And that turns out to work relatively well. It works particularly well if we use a deep learning approach to do it. But there are challenges with it, which is the fact that because of this very rigid assembly of how the cameras look at the scene, there are certain constraints that our solution must obey if we want to make sure that it's geometrically consistent. So that means the order that objects will always be on one side of the other if I have a certain assembly. There's a uniqueness of the matches that we establish up to resolution, of course. And then there might be occlusion of certain regions due to, again, the geometric constraint. If we solve this for every pixel independently, we cannot enforce any of these things simply because we're computing an arc max, and that's our answer. And so the idea that we're trying to put forward here is the idea that rather than matching points independently, really what we should be doing is we should consider this matching progress as a sequence-to-sequence matching task where we have our epipolar line in the left image, we have the epipolar line in the right image. And so really what we need to find is we need to find the best possible matching of one sequence to the other. And as we do that, because we now match the whole sequence sequentially, what we can do is we can enforce all these geometric constraints that we were talking about earlier. Now, this sounds great. Turns out that this was already great in 1985, where people started to look at this from this specific perspective early on because of exactly the benefits that I just talked about. Now, really what happened is there were a few papers on this, but it fell out of favor relatively quickly because of the challenges that come from trying to establish the sequence-to-sequence mapping using only the very local information that they had available at the time, right? Because you were computing SIFT or SURF features, they have a very local support in the images, and so you can't really resolve higher-level features if you try to do this mapping. So what we decided to do in 2020-ish is we tried to look at this again, now using feature descriptors that are derived from neural networks because they can acquire this global context very effectively. And then essentially develop an attention-based architecture, which we now know as transformers, that essentially are able to capture long-range associations between these features on the epipolar line and hopefully perform the sequence-to-sequence matching more effectively.. So let's see how that goes. So what we start with is, again, based on the fact that we do stereo matching, we start with an image pair, a right and a left image. And on that, we run a feature extractor. That's just your favorite fully convolutional neural network. I think in your group, it's the NNUnit. But in this case, I think we use something that is more DenseNet-like that we use to extract for every single pixel a feature representation. And so based on this feature representation, then we feed it into this transformer layer that consists of 6 self and cross-attention modules where really we update the representation at every pixel by computing the attention over first the same image itself. And then in a cross-attention layer, we compute the new representation based on computing the attention to all the pixels on the layer in the other image. And so we alternate that.. And we also include the relative position encoding, which is— well, position encoding is already relatively standard in Transformers. Here, we modify this attention term for position encoding just a little bit to drop all the terms that are solely awaiting between position in the left and position in the right, because essentially the absolute position in the image doesn't matter for our use case. It only matters the relative position of the two points. All right. Then towards the end, what we have done here after the end of this matching, essentially we have gotten soft assignments between what the likely correspondences are. And so what we can do now is we can use this differentiable Sinkhorn algorithm that also has dustbins, which allows us to identify occlusions in order to find the best possible matching and do this optimal run an optimal transport algorithm that is also fully differentiable to come up with our final answer. And then at the end, what we did is we essentially solved the sequence-to-sequence mapping based on individual epipolar lines. Now, in two images, what that means is after rectification, right, we solved it on every single pixel line independently. And what that might do essentially is give you something that looks a little bit rugged. So what we do in order to get a better estimate is we just do this context adjustment layer here at the very end that also has access to the feature maps of the very beginning. You can see this, this forward path over here, and this essentially just allows us to smooth out and align our estimates over the individual epipolar lines and get something that is more globally scene consistent. Okay, and at the end, what we get out of this algorithm is we get a final disparity, remember, that allows us to do 3D reconstruction of our scene based on the two images that we have.. And not only do we get that, we also get occlusion, which essentially allows us to understand which of our pixels are not in fact reconstructible because we don't know where they match. And that turns out to be useful— or, well, we actually haven't used it quite yet for that, but we believe that this will end up being useful for downstream tasks, like if you perform registration for augmented reality or other things, to actually understand which information do you observe and which information do you not have access to? Okay, so it turns out that if we have an image of 960 pixels by 540 pixels, which isn't really high definition yet, and we compute this attention, then a single attention layer here would have around 260 gigabytes of memory requirement, which is sad. Because we don't have that type of memory. And so there are a few tricks that we need to adopt here in order to make this thing trainable. And those tricks include not computing your attention with a stride of 1, but using a stride of 3, using gradient checkpointing, and so on. And what that allows us is it allows us to reduce the size and memory requirement of these attention layers to around 8 gigabytes, right? So per layer. So that then becomes feasible, and that's nice. One thing that is pretty cool is if we observe what these layers actually do and how they start building the information for matching these points and matching disparity is that at the very beginning, when we don't really have reliable representations yet, what we observe is that along these two epipolar lines, you can see that, right? So these are the true correspondences here, red and blue crosses. That we start very broad. So at the beginning, we attend and we get relatively high attention to pretty much every pixel on the epipolar line just to understand where this pixel is currently located. But then as we go deeper in the network and compute better cross-attention, you can see that the attention very quickly focuses around the point that we're interested in. And not only to that point, but also you can see to the adjacent features. And that turns out to be interesting because of this relative position encoding, where now you can even perform relatively precise matching on featureless surfaces like in the middle of the table that you see here simply because this attention mechanism allows you to attend to interesting points in the area that are somewhere there on the epipolar line and give you relative distances, which essentially benefits this dense matching even if there is feature scarceness. And you can see this exactly what I just said emerge down here where essentially what we do is an RGB embedding of the features that are being learned at every single point, where essentially what you what you can see emerge is this pattern on feature-scarred surfaces that essentially are parallel to the information-rich edges, right? And just propel them forward. And these can then be used as a guidance to essentially perform this mapping. All right. This mapping— this algorithm that I just talked about, we call it Stereo Transformer, STTR, works relatively well on SceneFlow. It performs good. We're happy. So what is even more important, I think, for us is not only that it performs well on the synthetic dataset that we trained it on, because in a way that's artificial, but we care about more is whether or not we can use this for the applications that we care about, which primarily lie in surgery where we don't really have datasets that are big enough to actually train such a big model.. And so what we really are interested in is seeing whether we have learned a generalizable way of computing this disparity. Because essentially, if you think about how we did that previously using SIFT and SURF, all of that was essentially domain agnostic. And so what we find is that the model that we have here has a very satisfactory out-of-domain generalization performance where without retraining, we perform really well on other datasets. Including medical ones like these ones here. And let me see whether the videos will start as I press. Yeah, so you can see, right, this is essentially out-of-domain generalization immediately applied to medically related data with submillimeter errors in reconstruction without any retraining. So it's very exciting. If you are interested in using this model, it is on GitHub. You can just download it with the pre-trained weights and you can run it, and it will work relatively nicely. We have used it together with colleagues, Douqi at the Chinese University University of Hong Kong to perform the scene reconstruction. This is some work that was presented last year where essentially we reconstruct the scene, we track the soft tissue, we segment the instruments, and using a lighter version of the algorithm that I just showed you that can run at around 15 Hz, we reconstruct the scene in more or less video frame rate at the same time. There is a short remark that I should be that I wanted to make here, because I know of your work in evaluation and challenge and metric design. And that's the comment on performance, which the problem really is, right? So STR, the algorithm that I just talked about, is not state of the art, right? It did not outperform the best possible methods that exist out there. But the question is, should we care, right? Is that an interesting metric to think about? And I just want to highlight that the state of the art is defined by evaluation on KITTI 2015, which has 200 images, SCARED, which has 300, and Middlebury, which has a stunning 15. And so the question really becomes, right, if I can claim state-of-the-art performance on a total of 530 images, is that truly what I should be caring about? And we have decided that we don't want to care about that. Luckily, with that specific paper, we were able to convince reviewers I hope you will be able to convince reviewers as well if you have a case to be made. Because what we care about here is the satisfying generalization performance that I showed you where we train on something synthetic and then we bring it to our use cases. We have since used it also for mixed reality guidance in total shoulder arthroplasty and so on simply because the model generalizes well. And even if it performs a little bit worse than other models on those datasets, for us, it doesn't matter. Okay, so now as we have understood 3D scene reconstruction, let's talk a little bit about image-based tracking of tools where now what we want to do is not only do we want to reconstruct what the scene looks like, we also want to understand what the true tool position is relative to the anatomy, which is very important if we want to understand surgical guidance. And so The way how we do this is using a method that again uses stereo video frames, the one that we have here. But now what we run is we run a recurrent, essentially an optimization-based method to between frame 1 and frame 2 in time establish the motion that has happened in between them, both for the background as well as for the foreground. And so the way how we do this is by using motion features that are— these motion features are then fed in the recurrent neural network that estimates spatial transformations for every pixel. But then that's not really what we want. Really what we want is we want one transformation for the instrument and one transformation for the background. So we run a fully differentiable geometric optimization to enforce exactly this constraint. And then we do this multiple times over time for every frame so that we can get updates. And at the end, what we will get is we will get the two transformations that we care about. And so the motion features that we use to do this here is that we will use disparity, like for example from your favorite 3D reconstruction algorithm like STTR that we talked about earlier. We will use a segmentation that we can get from another one of your favorite algorithms, right, that tell us for the two classes in this case that we care about, the background and the tool, the likelihood of every pixel belonging to that specific class. And we get that, of course, for both frames. And then we get cross-frame estimates, which is the scene flow, which essentially means how does a specific pixel in my image move in 3D space to the next time point. And this is essentially what we want to be updating. So initially our guess is 0, right? No scene flow at all. And then there's feature correlation, which means that after we have extracted Well, for disparity and segmentation, we get feature representations before we've performed that mapping. And between all of those things is that we end up computing this feature correlation with every feature in the left image to every feature in the right. So this is a huge volume that is correlation between every single pixel that you have there. Okay. So now, from all these features that I just talked about, how do we get better scene flow estimates? And that is essentially this recurrent neural network that takes the features that I just talked about and then regresses for every pixel in your initial frame one scene flow estimate delta, which essentially then establishes a corresponding point in the other image. So essentially it does that by indexing your correlation volume, by indexing your disparity volumes, and then optimizing for the match probability across them. And at the end, after you run this recurrent unit, it gives you for every pixel in the initial image, it tells you where this pixel should be mapping in the other image. And so once you do that, right, you have another match probability so that you can reevaluate the match probability of your new match. You run the GRU again, and again you can decide on how often you will do that. But at the end, once it has converged, you have for every single pixel, you have now a match in the other image that is based on these match probabilities. But every pixel here is solved more or less in independently, which is not what we want because what we want is we want two transformations, one for the background, one for the tool. If you had another tool, maybe you would want another one. But really what we want is those objects to move consistently. And so this is by using this geometric optimization here at the end where essentially what you can see is that we force within specific regions. So here within the segmentation region of the tool and within the segmentation region of the background, essentially the patient, what we want is we want to find the transformation given the current matches that best explains all these independent pixel relations that we have just observed. So essentially what we do is we take the mapping that we have gotten from our GRU, which doesn't know anything about this specific— the semantic association of these pixels to independent classes, and we project it onto the most likely solution that explains the independent matches using one single transformation. Okay, so that essentially gives us then one transformation for the object, one for the tool, one transformation for the background, and we run that again, and essentially we end up with our transformations. Okay, it's a complicated algorithm, so in order to train that, we need quite sophisticated data that doesn't really exist currently out there. So we ended up generating it ourselves. We have multiple ways of generating this. So this first thing that I show you here is a recent AEKAI paper that we have had. It's based on the physics-based simulation platform AMBF that is integrated in virtual reality and with haptic devices. So what we can do with this specific platform is we can go in and we can simulate skull-based surgery. So you can see this is the lateral skull base where we drill next to the cochlea. And because all of this is done in simulation, well, actuated by a surgeon, but essentially in simulation, all the data that we get here is perfectly controlled and we can use that to train very sophisticated neural networks. Realism suffers a bit, but it's a good enough sandbox to try and understand whether the algorithm that I just talked about has any merit to get close enough to what we are interested in doing. But of course, that's not really what we care about. That's, as I said, it's a sandbox, but it's very far, far away from what we want to be doing later on. So what we also have is in our mock operating room here on Homewood campus, we have this microscope, we have optical tracking solutions. So what we ended up doing is we attached these rigid body markers to the microscope, also here to the Phantom platform, as well as to the drill.. And so all of this then is calibrated. It's quite a sophisticated calibration procedure that involves micrometer stages in order to make sure that we can get this right with the hand-eye calibrations. And at the end, what this gives us is it gives us quite reliable baseline measurements of these tool-to-tissue relationships that we will also try to estimate purely in a purely image-based manner. And so we look at this using the data that we have calibrated and acquired in the way that I just talked about, which identifies the ground truth that we're after, which is the transformation for the tool. This is translation, this here is rotation, the same also for the patient, both in synthetic data and then for the surgical phantom data. And we benchmark our method to other approaches like a keypoint-based approach that uses orb features, which is a standard algorithm if you do if you're interested in SLAM, and also a scene flow-based approach that essentially established these features and then estimates scene flow based on as rigid as possible assumption, as well as 3D reconstruction followed by matching with the iterative closest point algorithm. And so what we find is that our method performs better than all of these other approaches, but we're not quite there yet. And the reason why we're not quite there is because right now— so first of all, for the tool, we still observe somewhat larger errors. We need to look a little bit closer into what the source of this error is. Currently, this is the real data. So essentially, there are lots of error sources that can happen, including the long baselines that I was talking about earlier in our ground truth setup, actually. But it's not only that. It's also the fact that we do this for one frame to the other. And so when you do this over time, what happens is that error starts compounding. And so if you compare the track— if you compare the trajectories that you see here from the ground truth, so we start here and then the ground truth is this dashed line, and then what we predict is the blue line. We're relatively close, but really over time from tool-to-tissue relationships, we're at 3.5 millimeters Very, very, very little at 0.28 degrees, but around 3.5 millimeters. So not quite clinically relevant yet, but I think, at least I hope I made you excited that technology like this can get us much closer to what we want solely from images. And the reason why we believe that this works relatively well is what I show you here where Essentially, using this dense approach to estimate scene flow and then getting these dense estimates of the transformation that we care about essentially works better because the features that we can use to estimate the transformations are much better spread throughout our image so that overall the correspondences are much better conditioned in order to solve for a relative pose, right, where if I have very scarce correspondences like the ones here, I may end up in ill-posed configurations, which essentially just doesn't give me the solution that I'm looking for. Okay, I see that I have paced myself reasonably so that I can briefly talk about something else that I am excited about, which is quite new work that we've been doing on robot tool segmentation. I think everybody and their brother brother is interested in robot tool segmentation these days. So I hope that I can provide a slightly different perspective onto it. At least that's why we were excited about this. And the way I'm going to start talking about this is using language from causal reasoning to understand a little bit of what is going on in the background. So really what this is, this is in a way, ways of understanding the data generating process and what should and shouldn't affect changes, in this case, in segmentation. And so the contemporary way of how people look at this is using a causal model that looks like this, where essentially there is T, which we will refer to as the kinematics. So that is essentially the robot kinematic parameters, as well as the camera that observes the scene. And then there is E, which is the environment, which is the patient, all sorts of things. Well, these two interact. The camera looks at the environment. The robot ends up manipulating the tissue. The environment has resistance, so it somehow has a feedback back to the kinematic vector and so on. And together, these two cause an image to be formed, right? The environmental parameters, the kinematics with the robot, and then ta-da, image. And then based on this image, what we do is we say, well, this image then causes our segmentation map. And so what everybody essentially tries to do is they try to regress this probability here, right? The segmentation given an image. And so if the image changes, the segmentation would also change. And this is a very prevalent approach, right? There are tons of challenges that address this specific problem that we can use. In my deep learning course, we do it exactly this way, right? It's a homework assignment. And so this is a very established view on how we solve segmentation problems. In general, but also in robot-assisted surgery where this may or may not be the best way of looking at it. And so while there has been tremendous progress using all sorts of models, including transformers, really the problem is that if we identify— if we posit the problem this way, then our model needs to generalize from the image space. And we all know that deep learning models are very vulnerable to domain shift and are not particularly robust in presence of unseen corruptions to our image, which in surgery unfortunately happen because there might be smoke, there might be bleeding, there might be all sorts of other things that otherwise we haven't necessarily observed in the clean training datasets that we have curated. And so the question that we set out to answer was, well, can we try to come up with a different way of looking at this problem in order to bake in this robustness from the get-go? And this is what we came up with, where we say that, well, Really, the segmentation mask isn't at all caused by the image. But really what it means is that once I know where my camera is and once I know where my kinematics are, right, because I have a robot, I understand all of this pretty precisely, I can in fact create a segmentation mask, right? So the image is in a way artificial in there. I don't really need it in order to understand. So you can see that there is no causal relationship between the image and the segmentation mask at all. But the segmentation mask simply arises from knowing where the camera is and where my tool is. And so this turns out to be useful because as we do this, the image doesn't play any role in how we actually achieve segmentation. So there is a little bit of a caveat here because this specific causal model that now identifies the counterfactual p of s and t Well, also has the challenges. There are two sources of error. The first one is confounding, which is the fact that the environment might essentially introduce occlusion, right? So it may occlude part of our tool, which essentially means that then the segmentation mask that we derive solely based on kinematics would be invalid, right? Because part of the tissue or part of the tool would not be visible due to occlusion. Now, if we assume that there is no occlusion for the time being, then this beautiful error here doesn't exist and we don't have a problem. That's very nice. But then there is a second problem, which is the fact that our robots, surgical robots, are designed to be compliant, which means that whatever we get from the system is not the true kinematic vector p, but actually something that has errors. So we get something that is tm, the measurement. Measured version of this. And what really— what that means is that if I use the measured kinematic vector and the camera pose and I use that to generate a segmentation map, then this will not be correct, right? Because my measurement will be imprecise due to, say, cable tension, all sorts of other mismatches in my forward model, and so on and so forth. And so what we need to do is we need to somehow account for this.. And this is now where the image comes in because the image is not affected at all by these measurement errors. Okay. So the image is directly caused by the environment and our true kinematics that we don't have, that we do not have access to. And so what we will try to do is we will try to estimate this measurement error based on, based on the image so that we can, we can align and regress this true kinematic position over time. And so this right here is the algorithm that, that arises. When we look at the problem in this specific way where this looks maybe complicated, but really it's relatively easy. The algorithm turns out to be very similar to 2D-3D registration if you have done something like that before where we observe the image i of the system, we observe the measured kinematic vector t_m, and so what we're going to be doing now is we're going to apply our error correction term over here. This gives us the current estimate of true robot kinematic parameters. We use that, this current estimate, together with a tool model of the robotic tool that is currently being used in a differentiable rendering module. This allows us then to synthesize a synthetic version of the really observed image. You can see that here. Then run a a Siamese U-Net between the truly observed image and the image that we have generated using differentiable rendering. We then compare the two feature maps using a cosine loss function. And then what we do is we backpropagate, essentially, this cosine loss function within our estimated segmentation map all the way back into our error term so that we maximize— we adjust the robot kinematic kinematic parameters such that we maximize this overlap between the two models in feature space. Okay. And once we have converged, essentially we use this observed— this updated final kinematic model to then generate an estimated segmentation map S, which we then can use for all sorts of downstream purposes. Right? So that's how segmentation emerges. So segmentation emerges through this alignment task by backpropagating into the robot kinematic parameters, which turns out to also not only give us segmentation masks, but it also gives us robot kinematics, which somehow is nice. Now, as I said earlier, one of the problems with these different ways of looking at problems is that there is no data available usually. So we went in and created some. We used AMBF, which is our physics-based simulator. We also went in and used the DVRK and we created this counterfactual dataset where essentially all the conditions are the same. So it's the same tools, it's sometimes the same background, it's the same motion, it's the same everything, but one condition is altered. So you can see, right? So the robot, we do this by action replay. So we go in, we record the sequence. Once we have that sequence, we store it, we go in and we replay it over the same tissue type, for example, again and again and again. But rather than say, instead of having everything stay the same, we, for example, insert smoke using a fog gun, or we insert bleeding over the places, or we swap out the background and so on. And this allows us to create quite large amounts of data that are counterfactual and that they are the same other than one specific condition here. And I should mention, this is a dataset that has these things available. So we have stereo video, we have kinematics, we have the segmentation that we get from running this also once in front of a green screen. So automatically for free, we get all the ground truth segmentations by simply segmenting out the green background. And we get the domain label because we know what we have changed. And in fact, we're currently rerunning this data curation on a much larger scale where we will also install a short-throw LiDAR sensor on top so that we get ground truth depth estimation as well for all sorts of purposes later. And so we benchmark this to other methods that exist in the space. So there's HRNet, and we use— I should say we use smoke augmentation. So we have the simulated smoke that we use as an additional augmentation. We use a SWIN transformer also as a baseline method, and another one that comes from, I think, UCL that also tries to use kinematics and then perform a a learning-based correction to this initial estimate. And so what we find is that if we train these models on the regular domain data, the one that doesn't have corruption and with smoke augmentation, what we're observing is, yes, our method performs relatively well. There are other methods that do equally well, if not even better. So all of these methods work really well if we're not evaluating outside of the domain. But what really happens is that as we go in and we change out this real domain with additional unseen corruptions like bleeding or seen corruptions like smoke, the accuracy and segmentation performance of these models deteriorate notably. And so this is an effect that is not really observed if you look at the problem through the lens that I was trying to convince you of here, where essentially we're not performing image to model,, but we're performing this differentiable optimization pipeline to align this, where by design we can retain some of the accuracy immediately because we know what we're looking for in the image. And so we don't have to solve a pixel-based problem, but we're solving a problem in much lower dimensional, much better conditioned space, which gives us better solutions from the get-go, right? And so what you can see here is that, well, this is how we initialize, right? This second to the bottom line, and then We optimize for the parameters, which increases the overlap and gives us the quantitative results that I showed you on the line before. And really what you can see here, though, if we're quite honest, is that this is a quite complicated optimization problem because of the robot joint kinematics, and that some joints have very large errors. And so with this conventional backpropagation algorithm, you end up not necessarily finding the best possible solution. You find something that has a lot of overlap, but then you reach, in a way, the limits of your optimization, where really what you would need to do is you would need to poke it out of that local minimum, which, based on these gradient methods, we can't really do it yet. So in our next steps for this work, as I said, this is very, very new, we're working on improving this backbone neural network. Right now, we use a very simple U-Net trained for a segmentation task. In order to try and solve this issue, but also looking at it from a different perspective and trying to enhance the optimization that we're using, try to use something a bit more sophisticated than simple backpropagation using stochastic gradient descent in order to optimize for the joint kinematics parameters. Because ultimately what we hope, right, is being able to quantify this mismatch between observed kinematics that are wrong due to compliance and estimating the true the true robot state might potentially allow us to do other very interesting things, right? Like for example, force estimation, other things that we have done here at Hopkins before using quite similar measurements. But for example, using torque. Okay. So here are some closing thoughts that I wanted to share with you, which I hope that I have convinced you that advanced computer vision algorithms like the ones that I talked about stereo reconstruction, scene flow estimation, and combined with geometric optimization, differentiable rendering like the one in the last sector are really contributing to making surgical navigation more capable when we talk about accuracy, when we talk about the possibility to estimate deformation densely, and also more available because we don't introduce additional hardware. So it keeps the cost low and it works immediately based on the data that we are acquire anyway, which means that we don't really have obtrusiveness. But I hope I also convinced you that there is— well, there is a lot of progress there. There are equally, if not more, challenges that we need to be addressing in the future. And so one thing I would like to encourage you is to think hard about whether or not we really— sometimes we're not even solving problems that are harder than the ones that we must. For example, in the last example that I was giving you about robot tool segmentation, pretty much everybody looks at robotic tool segmentation by taking an image and regressing the robotic tool. And that may be an interesting problem to solve, but the question is, why don't we use all the data that the robot gives us? And sometimes the answer is as simple as, well, I don't have a robot. But really what that means is that I think as a community, we need to try and and work hard to making data like that available, which we'll try to do with the data that I just talked about in the longer run in order to enable different perspectives onto solving these problems from slightly different angles, like the ones that we have committed to for quite some time now. And then there are lots of different challenges as well that I think all these advances in machine learning, that all these advances in machine learning have brought to image-guided surgery and image analysis in computer-aided interventions, that the challenges that still prevail are the robustness, building systems that are reliable and can be used in surgery, right? I think there are some successes here and there. I think at DKFZ, I think you have some. There are some initial advances now also at the IHU in Strasbourg where some of these advances go closer to the operating room. But in general, we are very far away of building systems that are reliable enough to achieve this. And part of that also is pertaining to human factors where really it's not just about whether or not the system can solve a specific task. It is also whether or not the system solves a task in a way that enables people to act on whatever output that you get. And so this is something that I think is wildly underrepresented in the current thinking about Well, both in CHAI and in MIC in fact, right? And how we make any of our recommendations actionable. Because that opens a whole different can of worms that we need to think about hard if we really want to drive these systems towards application and success in application. All right. This was all that I had for you. Thank you very much for your attention and for having me.

And I'll be open for for questions. Thank you so much, Matthias.

This was excellent. You were able to transfer your excitement to me, especially with the last contribution. Yeah, really great work.