Scientist Stories: Fred Hamprecht, Imaging Processing and Active Learning

Good morning, everyone. This is the Data Science Seminar, and today it's my pleasure to host also a joint event with Heidelberg AI. And we have a speaker, it's Fred Hamrecht today, and very welcome. And I hand over to Lena Meyerhein for a short introduction.

Yeah, good morning, everyone. It's my pleasure to have Fred Hamrecht here. He's a professor at the University of Heidelberg who's actually been working in the field of machine learning long before it became a hype. To my surprise, I only learned this now, he originally studied chemistry at the ETH Zurich. He also spent some time abroad in various places: Lausanne, Cambridge, London, Boston. And very impressively, in 2002 he actually became, if I'm correctly informed, the youngest professor at the age of 26 at the oldest German university. Heidelberg. He's also founder of the Heidelberg Collaboratory for Image Processing, HCI, and I really enjoyed reading his biosketch which says Fred is a professor at the Heidelberg University, still thinks of science as the greatest profession on earth. That's very motivating, and in this spirit, the stage is yours.

Wow, thanks very much, Lena, and all.

I really like the series. I've listened to a few very good talks here.

So it's nice to be able to share our own work.

Actually, this is— so as scientific endeavors go, this is now, I would say, a pretty mature one. We started in this around, depending on what you count, 2007, '08. I realized this morning that's when many of the people in the audience probably were in high school.

So for example, Björn Andres was one of the really important early figures in this.

And well, today I want to, I have two things to relate. So one is I want to talk about signed graphs.

What are they? How are they useful in computer vision in general?

And then how can we partition them efficiently. And for the second part, I will mostly present recent work by Alberto Bailoni, who's going to defend his PhD soon. So let's fix terminology first. This is a kind of image that we deal with often.

So some blurry microscopy image of a few cells.

And there are different kinds of questions you can ask from that kind of image. You may want to know which pixel is foreground or background.

That would— oh, and I have not shared screen.

That doesn't help.

All right.

Sorry about this. And you can— you see some cells? Great.

Okay, good.

So a couple of cells. We can either ask for each cell, is it a foreground, background?

That's what in this talk I call semantic segmentation. Or we can We can ask, well, which cell or which pixels foreground or background, and in addition, which pixels belong to one entity that would be semantic instance segmentation. And finally, there's pure instance segmentation or image partitioning where we just ask which pixels belong to the same entity.

So with that in mind, let's see if we can map these problems to these terms. So the bottom one is an image from the famous Cityscapes benchmark where, well, you see we delineate individual people here and we somehow separate them from cars, which are also delineated one from the other. Or there is a problem here, digital pathology problem, where we want to find out which regions belong to tumor, which ones do not.

Or here, a problem I will talk a lot about today, where we try and decide which pixels belong to the same cell. Now, which is which? My question to you. And so the cityscapes here, this would be an example of semantic instance segmentation, because we try and distinguish between pedestrians and cars, but also between one pedestrian and the next. This one here, tumor or not, would be an example of semantic instance segmentation. And this one finally, of most interest today in the second half of talk, will be image partitioning. All right, and I want to argue that all of these can be modeled as instances of signed graph partitioning. And for something like semantic instance segmentation in particular, there are like two broad schools of thought. And tens of thousands of papers written, but essentially all can be mapped to one of the two schools. One school would be to first detect and then figure out which pixels exactly belong to one instance. So first detect means we would first put a bounding box or put a blob inside each of these cells and then try and find out what exactly is its spatial extent.

And this other approach that I will follow today works without a prior detection step. So we directly somehow group pixels or superpixels into resulting objects.

All right, but now comes a tutorial-lecture part.

And this is the image I will use, a model of 6 pixels, but easy for me to draw.

And I want to explain what this has to do with graphs and cuts and with signed graphs and what is a signed graph in the first place. All right. So let's look at semantic instance segmentation. If I just want to distinguish between two classes, and then in your mind, you can connect each pixel to one of these two special nodes. So there's one for the, what is this purple class, and there's one node for the green class.

And then we can have a classifier that tells us how strongly we associate each pixel with each node. So typically, this would be a neural network, which tells you that with a strength of 2, it believes that this pixel belongs here. And with a strength of 4, it believes this pixel belongs to the blue class.

And with a strength of only 1, it believes that this pixel belongs to the green class.

And we can do that for all pixels. Because that's a mess, let's just concentrate on these 2 here.

Well, and then there's not—

you don't need much algorithm to make that kind of decision.

You just ask which belief is stronger, and then you assign the pixel to that meta node. So sorry for starting very basic. I think it'll get more complicated in a moment. So here, well, this pixel, it more strongly believes to belong to the green class than to the blue. And so I make this decision on a pixel-by-pixel level. If you do this on a pixel-by-pixel level, depending on how good your neural network is in semantic segmentation, the result may or may not may not look noisy. So perhaps sometimes you want to explicitly couple these decisions. That's not something that one would do, I think, in 2021, but it's something that was important for a long time in computer vision. So if we introduce these extra edges here, which we would need to cut and whose— and the cost of cutting them, we would consider that would be a mark of randomness. Random field. And as long as I have only two terminal nodes, and as long as all edges are zero or have positive weight, then I can solve this efficiently using— efficiently and exactly using the algorithm called ST-MinCut or GraphCut.

Now sometimes I may have evidence that these two pixels probably belong to the same class.

So I make some of these edges stronger than others. In that case, that would be a conditional random field and is not any harder to solve. As soon as you have more than two classes, the problem will be called a multi-way cut problem or multi-terminal cut problem.

And even with only attractive interactions, already at this point, this becomes an NP-hard problem.

And there are good approximate algorithms that one can use to solve it, but you cannot do this exactly except under very special circumstances that I don't want to discuss here in detail. So that would be semantic multi-class segmentation. Now if we remember this pedestrians and cars example, for example, if these two pixels here, they belong to different pedestrians, well, this kind of approach could only tell you, as long as we only have attractive edges, that both pics are pedestrian, but we cannot model that one pedestrian ends and the next one starts. To do this, we actually need repulsive edges. So we need edges in our graph with negative edge weights, because in that case we can say, all right, there's one instance of the blue class, another instance of the blue class, and here some other class. But importantly, we can now start delineating objects from each other, even if they belong to the same class, like two pedestrians that overlap or two cars that overlap in the image. Image. So these repulsive edges, they are fundamentally important. Without them, you cannot have instance segmentation at all.

All right.

Now, if you remember this problem where we don't have semantics, so pure image partitioning or pure instance segmentation, in that case, these extra nodes, they don't have informative edge weights.

These edges are all of the same cost, and then I can just as well omit them.

I can concentrate on the simple model here on the right-hand side.

And this is actually a problem that I'm interested in, and this is called, or can be modeled as a multi-cut operation clustering problem. Why have I been interested in this for a long time?

Because it happens to be a good model, and I think it's Björn Andres who found out first that this is a good model for this problem which is called connectomics.

And I'm showing you here a really old movie, but I think it has aged well because it shows you a little bit what this raw data looks like. So here, this is a serial sectioning EM data, so serial sectioning electron microscopy data of the tiniest part of human retina. And what we want to find out is where each neuron goes.

And at the time that we made this movie, which was in 2008, CNNs were not around, and, you know, it was the day of random forests and such.

And this is why at the time the biologists, they stained the sample in a very special way so that you would not see the interior of the cells.

You would not see the organelles at all. You would only see the outline of those neurons.

And well, nowadays, you know, with neural networks, of course, we find the insides of the cells, we find it actually helpful and we're happy to have it. But you see that this is a very intricate network work, the neuropill, of all these dendritic arbors of different neurons intermingled.

And the goal in connectomics is to find out where each of these neurons goes and to which others it is connected and so on. So you want to carve out not just one neuron at a time, but you want to carve out all at the time. And well, I have to say that I've really enjoyed working on this data for such a long time of my life because I still find it very, very pretty.

So this was the scale at which one would do these things in 2008.

And in 2020, this is now an image from the FlyE-M team at Janelia Farm. You can do this at the scale of most parts of the fly brain, and people are seriously working on doing this on the smallest mammal that they can possibly find.

And even for the smallest mammal that you can possibly find, you know, this is actually a lot of data and a lot of pixels, which is why actually, you know, it's a Google-esque kind of effort.

And indeed, it's Google together with FlyEM which produced this image on these, I don't know how many trillion voxels.

So the problem is both difficult and it's also a very large problem actually.

And I don't understand much of biology, but it is actually quite humbling if you, you know, at how much complexity there is in the brain of a fruit fly which sits on your banana. Yeah, so it's It's bananas. All right. So back to the modeling of this. I here showed this pretty image, 6 pixels, some edges attractive. So I modeled this as an unsigned graph here where I have an orange edge, it means it has a weight which is larger than 0. It does not want to be cut. Those pixels want to be together. They want to belong to the same object. Those pixels want to belong to the same object. And I have purple edges, pixels associated with edge weights smaller than 0, it means these pixels want to be in different objects. Well, and if you want to partition this graph, it is obvious how you would do it. So the pixels that like each other, you put them in the same cluster, the pixels that don't like each other, you put them in different clusters and everybody's happy.

Now, this becomes harder in real life because unfortunately, there are often conflicting situations like so.

So what, what to do with this image here? Well, I have proposed here 3 different solutions. So here I'm grouping into 2 regions, but we have cut an attractive edge. So this isn't, you know, not making everybody happy. It's not making this edge happy. Or we can group pixels like so, and now we're making this edge unhappy because those pixels really want to be together. Or we can group them all together and then we make the two repulsive edges unhappy and so on.

And you can easily see that especially as your image is larger and your number of conflicts is larger, you have this exponential set of possible solutions. And indeed the problem is fundamentally combinatorial in nature. So it's not a smooth problem where you do a bit of gradient descent and you end up in a global optimum.

Neural networks also don't have a global optimum, but it was really an enumeration kind of problem.

And obviously there are smart ways of enumerating, you're not going to brute force this, but it is fundamentally a hard problem.

So for people familiar with integer linear programming, this is how you can write it down. This is your objective function here. We have the sum over all edges E, And then we multiply the weight associated with each edge with an indicator variable. The weights are given typically by your neural network estimated from the image.

And these are what you want to fix.

And well, these edge indicators, you only allow them to be 0 or 1, where 1 means I want to cut the edge, 0 means I do not want to cut the edge.

And if we only had this top line here and this bottom line, then it would be easy.

We would just look at each edge and decide to cut it or not. Problem is, if we do this one edge at a time, we will get out inconsistent overall solutions. We're not going to get proper clusters out.

And this is why we need something like the triangle inequality in addition. And well, So altogether, with these integrality constraints here, it makes the whole problem nonconvex and NP-hard to solve.

You can solve small instances exactly, and you can solve larger instances pretty well, but no longer exactly. And of course, in response, there has been many algorithms wondering how to do that approximately. One of the famous old ones is called Kernighan-Lin, which was originally used to lay out electronic devices.

Then our lab developed a few, which for their problem domain were the best, at least at the time.

And Björn Andres has kept working on this field and developed a very nice suite of algorithms. Systems also.

A word on the modeling here. So given an input image like so, we can ask for each pixel if or not it is part of the same segment with its adjacent pixel. And so if we have a ground truth partitioning of the image, then we can train a neural network to give us this kind of prediction.

So here, for example, I think we're actually asking for a virtual edge in this, for a vertical edge in this image here. Or I can ask for a longer range and I will get a response like so. Or I can ask for a longer range in another direction, I will get a response like so. So at least if you have dense ground truth, then you can very conveniently train in your network to give you this bunch of output images. And this bunch of output images are the weights that you're going to use your graph partitioning problem. So summarizing part 1 here, I've been talking about signed graphs, undirected graphs where you have both positive, attractive, and negative, repulsive edge weights. I've argued that such signed graph partitioning can model some of the most fundamental low-level computer vision tasks. I've also tried to say why, or I've sort of argued that multi-cut is a natural objective if we want to formulate signed graph partitioning here. I've said that solving exactly is very expensive and there's a host of approximate solvers and that CNNs, at least if you have dense ground truth CNNs or whatever attention-based networks, they're good at estimating such attractive and repulsive interactions, which you can then use to model your low-level vision problems. Now, in the second part of the talk, I will talk about a very simple class of algorithms that does something useful on these signed graphs. But before I do, I wanted to ask if you have any questions at this point.

Fabian is raising his hand, so I think, Fabian, you can speak up.

Alright, yeah. So first of all, thank you so much. This has been a fantastic lecture and I've already learned so much. I was just wondering a bit about how robust that is. So if I'm thinking in terms of this neuron segmentation problem, I could envision a situation where neuron branches relatively far away, but two different dendrites or axons end up within proximity to each other, but then in that case, the CNN that would be looking at the edge weights would classify these two as belonging to separate objects and wanting the partitioning problem to separate them, whereas very far away, these two actually belong to the same problem. So can— is this something that can be resolved robustly, or does this cause errors in these types of approaches?

So let me try to see if I should correct you what you're saying.

Especially when we estimate these long-range interactions here, I might look at two pixels which are some distance away and there is lots of stuff going on in between, lots of boundaries, so it looks like they are from distinct objects. And yet if I go a few microns or nanometers, so if I go a little bit on in the data, I see that actually they are merged.

Yes.

And yeah, of course that is a fundamental conflict. And so what you need is that the transitive, we want to be together, wins against the direct, we do not want to be together. And that is a balancing question of how many how many of the attractive interactions do you have and how many of the long-range ones?

And indeed, we use a sparse pattern of long-range interactions.

So we have, here close by, we have 4 immediate neighbors on the Cartesian lattice, and we're looking at all of them. But at the distance of 10 pixels, we have, let's say, of the order of 100 pixels. But we're only looking at, let's say, 10 out of those. And I think if one made a very bad choice here, then indeed this could cause problems. That said, I think the real problem— so where do really problems happen in this kind of data? It's more because we're working a lot with boundary detection here, when you have a mitochondrion, which is one type of cell organelle, which is touching a cell boundary because the mitochondrion you want to ignore. And when you sort of, when your neural network cuts out the mitochondrion, it may cut out also the cell wall to which the mitochondrion is attached. And then it may look like things are connected even though they are not. And the other kind of problem Now there are different modalities on how to acquire these electron microscopy images, and I said serial sectioning earlier.

There's actually something called focused ion beam scanning where you have practically isotropic resolution and absolutely amazing data quality, and then there is proper serial sectioning where you take one image and then you perhaps well, different techniques. You might either cut the layer and then image, or you might image and then cut. But anyway, they are bigger to produce, they're better to produce big field of view images, but they're also more prone to really causing quite drastic artifacts.

Like when we first cut off a layer which is only a few nanometers thick and then image it, it may always tear, or you may lose one of these ultra, ultra thin slices, or something else might go wrong. And so actually these gross errors are the biggest problem where you're continuing nicely or you're following a neuron and then suddenly you're missing one or even more slices. And this is where you need then really very long range context. And this is where not AI is needed, but where the most amount of, let's say, cleverness or understanding of the data is needed.

Klaus?

I don't hear you.

Double mute, classic. Thank you very much, Fred, for this first part of the talk. I would be interested in how difficult it is to configure this solution. I mean, basically you have a learned part, the CNN part and then you have the graph part. And if you now consider variations of basically you could apply different modality for imaging or you could vary the problem that you're basically trying to solve in the image. So how much expert knowledge do you need to then configure this interplay between what do I ask from the CNN and what do I actually do in the graph and Is that anything that can be automatically optimized, or is it something that you really need a lot of expert knowledge in order to configure your weights into the interplay and all that?

Yeah, so what I think the— so with the kind of post-processing or the method to partition your lattice that I want to talk about in the second part, actually this has Let's say, I think you have essentially one hyperparameter in there, namely which linkage criterion do I want to use? But I would claim that this is pretty much the only choice that you need to make sort of arbitrarily. And so indeed, it means that you put most of the modeling decisions then on the side of the neural network. And there in particular, I think the hardest decision is the one here with this connectivity, which also implicitly Fabian asked about. So this pattern of what close-range pixels and what long-range interactions do you want to measure? So this determines, of course, the topology of your graph. So here for simplicity, I always only showed edges between adjacent pixels, but by these long-range interactions, we have the long-range ones. And that's, I think, the more interesting choice than asking in my U-Net, do I want the blocks to be residual or not, or do I do something transformer-based? So sort of all of the CNN part, you can as you know, completely plug and play, or let's say replace one by the other. And so bottom line, the real modeling choice is in the connectivity here.

Thanks for the good question. Any other questions?

I may add to the answer of Fred because I was also part of this research.

Olli-Pyörte speaking.

Maybe you can update me on the latest incarnation. So I mean, the algorithm contains three major modules. First, the neural networks to detect these membranes, and at the end, the graph partitioning algorithm. But in between, there was a superpixel creation algorithm. That defined the graph. And I think this was the part that required most expert knowledge to get good superpixels and to define the features of the superpixel edges.

Yes, exactly.

Good question, Ulrich, and the answer is no longer. And this is the perfect introduction to my part 2. So maybe I should start with part 2 now. Now that the questions get really clever.

Thanks, Urvish.

Wait, sorry, there's a question in the chat, a basic one. Maybe you can look at it.

All right, the question is if the CNN is run per image or if it's a 3D CNN and what ground truth exactly means.

So yes, we like to use 3D neural networks. So depending on the modality, your data may—

the resolution of your data may not be isotropic. It may be much worse in one dimension than in the other two, but still it's obviously a good idea to use a 3D neural network. And a ground truth then means that ideally you want a full partitioning, and that is of course, a tremendous amount of work to generate. And when I said Googlesque, I meant both the effort on the compute, but also the effort in generating the training data. And now, of course, you can try and do clever things and learn from sparse ground truth and so on.

But it's not really something we focused on because dense ground truth was available for All right, so dense ground truth was available for sufficiently large volumes to train our neural networks.

Okay, so I'll go on to part 2. So now, yes, we have a neural network, we know how to estimate these edge weights, but now what do we do with the signed graph. And now I would like to talk about a family of algorithms, some old, some new, very simple algorithms, namely the ones that come from agglomerative clustering. I think probably Torsten Beyer was the first, at least in our lab, who figured that they could do something interesting. But Alberto Bailoni, who is going to defend his PhD soon, has really looked into this in much depth.

Now.

And this is essentially his work. So first, what is this algorithm?

If we focus here on the top left, we have a loop where we have some current set of segments. So Su and Sv are segments that belong to partitioning pi here. And between each pair of segments, we measure their interaction strength.

And I will argue in a moment that how exactly we measure this interaction strength, that this is absolutely crucial and gives you very different behavior and accuracy and performance and all. Now, if this biggest interaction, if it is an attractive one, then we merge these clusters.

And then we update the interactions, and that's it.

And now I've omitted a few words. There is something with constraints that if you want to know about in more detail, you should, you know, read up in the— there's one preprint out which has half of the stuff that I'm showing here, and we'll put out a new one in a few days with a lot more theory that Alberto has worked on since. All right, now the linkage criteria are crucial.

So here's a tiny example: a cluster, another cluster, a couple of edges between the two.

And now how do you aggregate those interactions? And there are a couple of obvious choices.

So why not sum them up?

And if we sum them, then here we have 7 7 14 10 is a sum.

If you sum them up, it gives you a strength of +4. Or you could look at the single strongest interaction, and that will give you a -10 here.

And so coming back to Fabian Isensee's question, so if we take this single strongest interaction, yeah, that may be perhaps more prone to provoking the kind of problem that he evoked, where a single strong repulsive interaction can overrule the many, many attractive ones that we do have. Or instead of sum, one could take the average.

So we had the sum was +4 and we have 4 edges, so the average interaction strength is 1.

And well, initially I thought that sum or average would be pretty much the same thing, but it has very different behavior.

Indeed, and then you can ask for the biggest attractive or the biggest repulsive edge. And certainly you could ask for many others.

So this is already a family with 5 members, and if we say, okay, we do or do not use cannot link constraints, then this is 10 members or 9 because 2 of the members happen to be identical. You can invent other things here and they may be meaningful, like the Felsenstein-Pudenhöfer algorithm, for example, is not covered here. But already these 5, okay, we understand them better than others and we're already pretty happy with the accuracy. So just to show that this innocuous change, so going from the, changing from the sum of interactions to the average of interactions, here's just a toy example to show you that indeed this can give you into different clusters.

So here we have the same graph. And then, well, you look for, if we go to some part here, if we look for the strongest interaction, the strongest one here is +7, so these two are merged. We have a new node CD.

And now we look for the strongest interaction, we have +8, +6, or -10, it's this +8, so these two are merged.. And at that point, we are left with only one more net interaction, which happens to be overall negative.

So that's when we would stop the aggregation. And so you see this dendrogram here, and if you stop after the last sort of attractive, net attractive interaction, you would cut here.

So you've clustered A, C, D, and B is in a separate cluster..

If on the other hand you use the average criterion, well, then you would have a cluster of CDB versus A, so different results. All right, and so you can, well, try this out for other criteria, some of which are that I've mentioned earlier, and you can do this either on unsigned graphs or unsigned graphs or unsigned graphs also using cannot link constraints.

And well then if you naively take the intersection of these, you get this whole matrix or table of algorithms.

And some are really old school ones. So on unsigned graphs, this is what people have been doing since 1950 or so.

But on the signed graphs, the literature becomes much more patchy. So the sum linkage is something that Björn Andres and Thorsten Bayer and Jörg Kappers found at almost the same time independently. And the mutex watershed we published not long ago, but it was Alberto who understood that this can also be understood as an agglomerative clustering algorithm.

And so we have these family members here and we can study them one by one.

The MUTIX I've just mentioned is a favorite one of mine.

Why?

Because we understand it maybe best of all of these. So I have a, you know, this slide is just to show off, no time to explain it now, But we understand in which limiting case this mutex watershed actually solves the multicut problem exactly.

And this is described in detail in a PAMI paper from last year.

All right, and so the mutex watershed, we, I think we invented and we studied, but before understanding it's one instance of the agglomerative family. And then there are other algorithms which as far as as we know, have not been studied previously on signed graphs with the exception of this paper here that has come out a few months ago. And now, something that I don't have time to talk about in detail today is how we can characterize them more theoretically. So Alberto could prove that for this subclass shown here, These algorithms define an ultra-metric, which normally is discussed on unsigned graphs and here generalized to signed graphs.

And this subclass of algorithms here have the property of shift invariance, which means that if you give me a signed graph and if I then add any constant to all edges, for example, a positive constant large enough such that it makes all edges attractive, then, well, on signed graphs you have a natural stopping point, right? You always stop agglomerating when your net interaction becomes negative. But if you disregard this and went on clustering until you're left with a single cluster comprising all of your nodes, then here, these family members framed here, they give you the same dendrogram no matter what additive constant you use.

And I think that has some deep consequences in the sense that these algorithms are probably less novel than they first seem because they would give you the same dendrogram as their brethren working on unsigned graphs. All right, now I mentioned earlier how different the behavior is.

So let's look at this, the same data, and I'm stopping here.

So what we've seen is the aggregation.

And for example, this sum criterion has the effect of, or it tends to build one cluster at a time.

So it builds one cluster and that one in the process becomes stronger. And that sort of reinforces the interactions, the positive interactions with its neighbors, and then it's like on a lucky streak and this cluster is built until completion and then the next one starts. On the other hand, look at this one here. This is much more reminiscent maybe of what Kruskal's algorithm would do in a minimum spanning tree problem.

It starts assembling these tiny fragments of clusters anywhere in the image..

And on the right-hand—

so on the left-hand side, we have what, 10 segments? On the right-hand side, we have what, 10,000 segments after the same number of merges?

So extremely different behavior.

And Mutex here for this kind of graph is somewhere in between.

And if we now let them all go on, well, given the nice outputs of our Deep Neural Network, they all come up with good and pretty similar segmentations.

But what has happened in between was extremely dissimilar.

So which one is best? And I have to give you the frustrating lawyer's answer that it depends.

It depends.

All right, so on pixel grid graphs, most of biggest interest here to computer vision people, we find on average across the problem We've studied that this average criterion gives us the highest accuracy. And so then coming back to Klaus's question, all right, you have your raw data, you have your— maybe I should delete the C— anyway, you have your neural network predictions. And then as long as you settle on, yeah, I want to use the average criterion, then this has zero parameters and you get out your segmentation. However, you could use the mutex instead, which is fastest, or you could use the sum criterion instead, which in our experience works best on dense graphs such as social networks and so on.

So that's what I meant by the one hyperparameter that you do have is, do you want to use some criterion or average criterion or this absolute maximum criterion.

In principle, you have the choice of do you want to use the constraints or not. But so far in our experiments, we found that the constraints do not really help. So I would say the choice really is between these three. And well, the nice thing is all algorithms are extremely simple and not terribly expensive.

So you can just try them out. Them all. And now this, I have two results here on, so this ISB benchmark, it's really old and it's on small data, but literally hundreds of labs in the world have participated.

And this is the current leaderboard. So right now the winner is a very nice new network called PatchPerPix from Dagmar Keilmüller's lab. And then she uses MuTEx-Watershed on top of her neural network predictions. And this is what Alberto had. This is a good but normal CCN, and again with MuTEx-Watershed on top. So both use the same post-processing.

And on the CREAMY challenge, This is a much bigger dataset and more ground truth, and it is harder, but a smaller number of labs have taken part in it.

And again, there is the current leaderboard. And the leader for quite a while now has been Konstantin Pappe from Anna Kreschuk's lab, where he uses highly engineered, so a good CNN, our favorite superpixels and then the lifted multi-cut on top. And the next entries currently are Alberto with the same CNN and importantly no superpixels and then, you know, these average algorithms, once with constraints, once without constraints. And then by the way, we can also run it of course on the our favorite superpixels, but here they don't even help. So the superpixels that we put a lot of—

that Ulrich asked about—

that we put a lot of love and effort into, they are now gone.

So we have the raw data, we have the choice of stencil, you have to choose your neural network architecture obviously, but once you have those images here, you just run your GASP with your favorite merging criterion and no other parameters, which is something we really like.

Okay, so summary of that part. GASP is a family of very simple agglomerative algorithms for signed graphs, which Alberto has now studied as a family. And for instance segmentation and computer vision problems, We find that in most cases, average linkage gives you the highest accuracy, but mutex watershed is the one that's fastest. So empirically, this is an n log n. So not in theory, but empirically we see an n log n scaling with problem size and it's still very, very good. And so my overall summary here is that both parts together. Signed graph partitioning can be used to model a very broad range of computer vision problems. And no matter what progress there is on the neural network side, in the end, you always have this final inference step where you need to cluster or aggregate or cut or whatever.

And this is the part that I'm interested in this talk here. And simple agglomerative algorithms can be defined not only on unsigned graphs but also on signed graphs. And we find that they do pretty well for these signed graph partitioning or clustering problems.

That's all. I'm looking forward to your questions.

And if you're interested in any of these questions, then please reach out to me afterwards. Thank you very much.

All right, questions? Well, thanks, Fred, for the really nice talk. I have one question. At the beginning, you distinguished two main types of algorithms, the detection-based one and the graph-based ones that you presented. So how would they relate, and why don't we see the detection ones on the leaderboards? Is it a matter of of inductive bias and training data or?

The reason is that the data is huge.

So these instances or these single neurons, they have a length of tens of thousands of pixels. And so the Mask R-CNN and some kind of family, they were great when your objects, let's say your car or your pedestrian or whatever, whatever, when it fits into the field of view of your neural network. Because the important thing, of course, for the detection-based ones is to make precisely one detection per object. Now here, and how do you do that on the cars and pedestrians?

Well, you feed in your image at a couple of scales. And if you downsample enough, your car will fit into, or even your truck will fit into the field of view.

If you downsample this enough, each single neuron will completely utterly just be gone. The only thing you're left with are, the cell bodies. And I mean, yes, you could argue then, right, then we have one seed or one detection per cell body and then we want to propagate from there, which works as long as you have all cell bodies in your field of view, which is usually not the case in connectomics.

So there are good reasons why the detection-based ones must fail on these kinds of data where you have very, very long but not very thick objects.

And to be clear, I mean, if I had to work with these and if I want something out of the box, yeah, I'd probably use one of the detection-based methods.

Okay, thanks. That's very interesting.

Ulli is raising his hand. Yeah, I think I missed it in the talk. So usually the output of a CNN is a response per pixel, so like membrane strength per pixel, but for your algorithm you need edge weights. Do you transform the pixel responses of the network to edge weights in the graph, or do you learn edge weights directly in the CNN?

The edge weights are learned directly. So the ground truth image, you can— let's see, I don't have the picture here, I think. The ground truth you can translate into images by asking, you know, these two pixels at a distance of 10, are they in the same segment or not?

And you do that for each pair of pixels at a distance of 10 in the horizontal direction, let's say. So you get out a binary image, and this is what you use as ground truth directly.

So I would say you're learning edge weights directly.

So like in the UTX watershed paper?

Yes.

I think Lena has a question. Yeah, sorry, I don't have a raised hand, but the button is co-host. Yeah, many thanks, Fred. I have a meta science question. So you showed all these rankings from the challenges, and you basically described the methods by X plus Y plus Z, you know, core method, post-processing, etc. How do you decide what makes a winner the winner? So I mean, how do you decide which component components were crucial which you would want to adapt compared to what didn't make a difference.

Yeah, I think it's pretty clean here because in this case it's two different neural networks, but it's the same post-processing. So mutix watershed in both cases. Here, this neural network is the same as that neural network and as that neural network because we were very friendly with Konstantin and his lab So we know it's the same neural network and that all differences in performance then come from the choice of post-processing.

Yeah, I mean, I see it in this case, but do you have a general strategy?

I'm just interested because— Oh, no, I don't. I think the general observation is that you're not going to save a bad neural network with a fancy output and vice versa. With a great neural network, you will be able to get something reasonable with a very simplistic inference or post-processing. And I think the, The trend of recent years is that we— so I'm thinking of tracking now, which is always on my mind even though we don't work on it all that much. So for example, in tracking, you could have pretty simple or pretty weak priors of what should be assigned to what.

And then you could do a very fancy afterwards conflict resolution inference and so on.

Or you could try and give the neural network more space-time context and then maybe even train it end-to-end with a very simple post-processing.

And I think given enough training data, this latter strategy will win.

I'm not sure if that is your question, Lena. Well, that was also interesting. I was a bit more general. And yeah, I just find it so hard that basically papers basically publish a whole pipeline. And then I always wonder what really makes a method perform well. But it's a hard problem. But you gave a specific answer to a specific problem.

So this is also interesting.

I think Klaus has an answer.

No answer, sorry.

Okay, then let me just, you know, answer quickly. I mean, when you say good paper, of a good paper, of course I expect a decent lesion study, but by this criterion, many papers are not good, of course.

All right, Klaus. Yeah, just Maybe similar direction but a little different. So maybe you could provide me with an intuition how difficult it is to come up with that Cremie score here. So how, I mean, you always have to ask yourself, okay, what is a metric that in the end represents a really nice result? And is it easy in this case to measure that? Because I mean, I know in other areas we are really discussing hard on how to measure success.

Now this is also one area where this is discussed a lot and where what you, where it is not obvious what is the right thing to do. Now, CREAMY is something simple. It's whatever, you know, it's a split score plus the merge score, you know, something simple and it's in the repo open source.

So you can reproduce it, you can understand what it does.

But some people here in this field in particular, they have been developing other scores that try and look more at the number of topological errors or— and sometimes they were more or less principled. I think they all have a point. They all have merit.

And the community has not really settled on any single one of them.

So I would say unsolved problem, in short. Probably it also depends on the scientific question that you want to ask later on the data. So like the biological question. So for example, you could measure how many synapses are wrongly assigned to the wrong neuron, or whether you get the correct subnetworks or whatever, right?

When we participated in KRAEMI, one big difference that we found was that the annotations are really sparse and really only available on very few images. Is that something that's being addressed in the community at the moment, that's like also trying to come up with more labels and more dense, dense labels?

Um, so the HEMI-Brain, which I cited earlier, they have made the, uh, the Google segmentation available. So you can, with an API and all, so you can get, you can stream terabytes and terabytes of not flawless, but pretty good segmentations that you could learn from. I think a few years back, this is where Google and us sort of bifurcated. So this is Vera and James' team here. One of the first things his team did when it was established was to collect all the training data from all the different modalities and all the different labs.

There were bits and pieces of training data out there in the world, freely accessible.

And of course it was a good idea to bring it all together to train, to let the neural network learn from them all. And it's completely meaningful and the right thing to do. And it's something that we did not do because I'm not so interested in this. Ultimately, I am more interested in methods. And yes, of course, it's nice to win benchmarks, but I wasn't really, really interested in pushing it all the way in the industrial you know, sort of in the big scale and so on. And unfortunately, this flat-filling algorithm which the HemiBrain team has used has not been evaluated on these public benchmarks. So we think the accuracy is probably about the same. But again, what I said earlier, if you put more training data and make your network deeper and so on, this will always help. So that's the right thing to do.

Just scientifically, I don't find that part so appealing.

So I'm more interested in what can we do on inference side or on the neural network side. We are interested, so a few years back, 2018, we published one of the first papers steerable neural network, so on equivariant networks.

And nowadays we're using that to think about, you know, can we use that to address quantum chemical problems, such equivariant networks.

So I, yeah, somehow I like this conceptual things more. Cool, yeah, sounds interesting. Thank you. All right, I think there are no more questions.

So I think I hand back to Lena and Doreen. Yeah, thanks to you all, especially to our speaker today, and also for the interactive discussion. It was quite fun to listen. And yeah, for more data science videos, you can join our YouTube channel and also look for the new announcements on our website of the Data Science Seminar and also for Heidelberg AI. So thanks for watching and listening, and goodbye to you all.