Scientist Stories: Andrea Vedaldi, Actionable Interpretations from 2D to 3D

Welcome to the joint event hosted by Alice Live Heidelberg and the NCT Data Science Seminar. It's my great pleasure to welcome our speaker today, Andrea Vidali. He is a professor of computer vision and machine learning and a co-lead of the Visual Geometry Group at the Engineering Science Department at the University of Oxford, and he will talk today about discovering actionable interpretations from raw visual data from 2D clustering to 3D reconstruction. As always, you will have the opportunity to ask plenty of questions by using the chat function after the talk. But before we start, I hand over to Fred Hamblecht for some introducing words.

All right, many thanks, Doreen and Govind, for organizing all this. We're very happy to have Andrea Vidali with us today. I'll briefly introduce what he's done in the last few years. He graduated from Padua, earned his master's and PhD from UCLA under mentoring of Stefano Soatto. His committee is a who's who in computer vision. One person in there is Alan Yu, who some of you heard in our excellence cluster talk, Vofix, just a few weeks back. And he, Andrea, then moved on to Oxford where he went through the gears, postdoctoral assistant, then eventually associate and now full professor. He shares his time between uni and Facebook. And Andrea is the recipient of many awards. So outstanding Master of Science, outstanding PhD thesis, He was a winner of the Pascal Visual Object Classification Challenge in the pre-deep learning era. Then he went from strength to strength in the deep learning era. He has received multiple outstanding and best reviewer awards, won best paper awards at the MBC and CVPR, and it goes on and on. Won the Test of Time Award from SIGMM-ACM, and In the last few years, Andrea has been making really interesting contributions to self-supervised, to unsupervised learning. And to us here in Alice Life Heidelberg and at DKFZ, we often work with these quickly changing biological datasets for which there is no, no or no extensive ground truth, no labels to learn from. And this is why all this work of learning from few or from no labels is so relevant to us. And so we are very happy, Andrea, that you took the time to talk to us today, and we're looking forward to your presentation.

OK, thank you very much for the introduction.

So without wasting time, I'm just going to get started then.

All right, so the talk today is going to be about discovering auxiliary interpretation of images without using manually provided labels. Okay, so that's the idea, and maybe we can look at it here from this slide.

So here I have an image, and what do I mean by actionable information?

Well, I mean information that a computer can extract for me, but it makes sense to me so that as a human user of this information, I can do something with it. For example, here there is a labeling of the ship with all the different parts, and there is an explanation what the parts actually do and are for. So that's actionable in the sense that I look at that image, that information, and can understand what the image is about and do something about that. Now, the normal way in which in computer vision we try to get at this result of extracting actionable or interpretable information is through a process which is based on supervised learning.

So what does that mean?

So it means that we have images as input to our algorithm, and we have also labels for those images.

And the labels, they explain to the algorithm what it's supposed to find in those images. For example, here you see a segmentation of those images in different parts which correspond to different semantic regions.

It could be a building, an animal, a person, etc.

And the algorithm doesn't know about those things a priori, right? So the algorithm is given the images and is given also the labels which tell it essentially what's there, right?

So this is supervised learning, of course. Now, we can further formalize that in the case of an image classifier. So in this case, we have an input which is an image x. We have a function phi that we want to learn that generally is going to be a deep neural network, so it will comprise several layers, if you like, the nature of which doesn't matter too much, but those are learnable functions. As a whole, they form my model. My model will produce some high-dimensional vector which is a representation of the image.

So just a vector that somehow contains the information which is in the image, which is useful to me. And then I have another block there, a classifier, that takes the vector and maps it to something which is interpretable.

For example, a label for the image. This is a particular bird.

This is a goldfinch.

That's what I want to know. And so how do we learn that?

Well, we learn that with labels. It means that in order to get this function, What we need to do is to provide a dataset to the algorithm, dataset D, which contains pairs x and y, where x is of course the image and y is the label of that image.

And through a process of learning, which amounts to optimizing the function to minimize loss, the prediction error, I'm going to get a model that is able to predict the label that is consistent with input image.

Of course, after learning, I have the function that I can apply to new images, and the model is now able to tell me what is the content automatically. But for learning that model, I need to provide these datasets which contain these pairs.

That is to say, I have to have supervised data, images and labels. Now, that's fine, but it's also very expensive and not very scalable.

And it's also not very intelligent, if you like.

So as humans, yes, we are taught by teachers.

We can read books.

We can read the information off the internet. So we can receive labels, in a way, from an external source.

But a lot of the stuff that we do, especially when it comes to processing images, is— I wouldn't say innate, but is a capability that we develop automatically just by operating in the real world. So what that means is that we can look at stuff and come up with interpretations which are interesting and useful automatically. So that's exactly what we would like to be able to do with machine learning. So we would like to move to a regime in which machines are effectively smarter than they are at the moment.

And just given raw data or raw data and very little supervision are able to develop not just an ability of interpreting images but also to develop the nature of the interpretation itself. So being able, for example, to come up with a classification of those images without me telling the machine what are the classes. So how does this look like in practice?

Well, one simple way of formulating this problem in a way that is actually approachable is to look at that as a clustering problem.

So on the left here, what you see is my standard neural network.

I get an image, I put that into this function which I want to learn, and the function produces as output some sort of vector or other kind of representation of the image. And then on the right, I have some machinery that is able to take those representations and cluster them, group them. It could be something very simple, for example, just k-means clustering. And now the hope is that those clusters would actually mean something.. And so what's the difficulty here?

Well, the difficulty is that I would like to come up with this clustering without having to have external labels that tell me what the classes actually are. That's why I wrote there k-means. k-means is an algorithm that basically is able to do that.

But if you set up this problem, and the problem is to optimize your k-means energy function together with optimizing your representation, your function phi, what happens is this. So what happens is that this collapses. So if you think about it, what k-means tries to do is to place these centers, which are this sort of cluster representative, spread over the space in such a way that you minimize the distance of the cluster center to your data points. But now you have an ability of modifying those data points because you're learning representation itself in such a way that they can coincide with a single class that you estimate. So the most easy— the easiest thing you can do to optimize a clustering function or clustering energy like k-means is just to take your representation, make it useless by mapping every single input image to the same vector. And then at that point, you can easily minimize the k-means energy function by just having a single cluster where all the data points coincide with that single representative you have. Okay, so that doesn't really work.

So now this would be also called deep clustering. Deep clustering means that you run in parallel or together a clustering algorithm like k-means together with learning the function phi, which is your deep network. And if you do it in a naive way, that doesn't—

just doesn't work. So how can you fix that? Well, this is an argument that we have introduced maybe 2 years ago now, well, 1.5, and that was addressing this problem.

And we did 2 very simple things here to try to make it work well for at least the case of image clustering. So the first thing was actually a bit controversial, but actually makes sense.

So what we did is, okay, we don't want this cluster to collapse, so what we're going to do is we're going to force these clusters to have equal mass.

So that you see here, the picture on the right is the same as before, but now we have something in the energy or the objective that we are optimizing that encourages this cluster to have more or less the same mass.

So that's the first thing that fixes this degeneracy, prevents the algorithm from collapsing everything into a single cluster.

And two, well, we have structure in the data. So these are images, so these are not random points, and we do have a priori knowledge of what we expect those clusters to look like a bit.

In particular, if those clusters are semantics that correspond to concepts that exist in the images, they should be invariant to the transformations that you apply to those images and they leave the meaning of the image unchanged.

For example, I can do these things to my cat image: I can zoom in, I can take out the color, I can partially occlude the image, or I can rotate it. And all these different operations that I can apply to my image should really not change the cluster the image is assigned to. Because the meaning of the image should not be changed by those transformations.

Of course, this is just on average. So that's the other constraint we have here, is that if you apply transformations to your data, then the clustering function, the function that maps each image to its own label, should be invariant to that.

Now, you can develop this formally, and it comes out to be a nice algorithm. So you have to learn two things.

One is the representation phi that's on the left, The other thing you have to learn is the label assignment Q on the right here. That's just a table that tells you for each data point i, each image, which label you should associate to that.

So what you're going to do is you're going to alternate two steps, if you like. The first one, it goes through energy—

so it amounts to cross-entropy minimization.

So what you do is you learn a function phi that, given your assignment, tries to predict them. And that's just the standard label prediction task.

Which amounts to, or you can formulate as minimizing the cross-entropy loss. But the other way around, so you want to do these associations in a way that is compatible with this loss function you are optimizing, this sort of cross-entropy loss. You want to do it in such a way that the clusters have equal mass. So that turns out to be an optimal transport problem. And of course, it's a very large one because typically you're processing millions of images and thousands of labels, so you need to be a little bit smart how you solve that.

But there are ways of solving optimal transport really, really fast nowadays, and some of those are actually differentiable too. So in principle, you can use that in your learning. And, you know, put this thing together and you get a nice formulation that actually works. So you can learn to label your images as well as cluster them, or you can figure out the labels automatically with this. Now, there is one more thing I wanted to say before I show you some examples of the results. So this works for individual, so single images.

So you put—

you apply this to a collection of images, like for example ImageNet without labels. You run this and you get some classes which are actually meaningful. But you can do something more.

So, and here again it goes back to exploiting the structure that exists in the data. Suppose now my data is videos, it's not just images. Then what I know is that those videos, they should cluster semantically in the same sort of space, and they should be associated with the same labels regardless of whether I look at the visual part of the video or I listen to the audio part.

So now this is another type of invariance that I can use to try to make my cluster more semantic, more meaningful.

And this is similar to the transformation I suggested before, but now the transformation, if you like, is a sort of transformation, but it's a very special one that takes the video and projects that down onto the visual part.

So here on the top, or projects it down on the audio part on the bottom. Now, what I can do is I can train a pair of clustering functions, one for the visual part, the one for the other part, audio part, and encourage this versions of views of the data to map to the same clusters.

Now, why does that make sense? Well, again, so there is very little in common between an image and an audio track, except the fact that very often the visual appearance and the sound of something are characteristic of the actual nature of the object which is portrayed in the video.

So by forcing the clustering to be invariant to this kind of very strong kind of transformational projection that really separates the two streams of information completely, then you're really encouraging your clusters to latch on what's actually the content of the image, because that's the only thing that can explain why the video and the audio go together.

So this is great for abstraction, And in fact, it works pretty well.

That's just an update on the previous slide where you can see how the energy function changes if you do this sort of modality invariant clustering.

This is a very small change to the algorithm, and then it works just as before.

We have tested this on a dataset of videos, which is called VTG Sound.

VTG is the lab I'm coming from, so that's why VTG.

And VTG Sound is just a collection of videos that have a characteristic appearance and sound, if you like.

Built into them.

So you run this algorithm on these videos without providing supervision, and then what you can do is you can check how well your clusters that are discovered automatically align with the clusters that you get instead by using a human labeler.

So you can see whether your machine is able to reproduce clusters which are close to the clusters that a human would actually produce.

So this is an example of the results.

Of course, there are plenty of other results in the papers, but just I want to highlight one or two results which are important.

So what you see here is what happens. So the measure we're using here is NMI, just normalized mutual information. It's just a measure of how well two clusterings of the same data align.

And we're comparing two things here, one using only visual information, just the images, and the other one is to use this cross-modal clustering.

What you can see here is that there is really a big jump when you move to just clustering things by their appearance to clustering things based on this cross-model.

Kind of invariant clustering. And really, this shows that because of this big jump, it really shows how important it is to get the right kind of inductive biases into your clustering so you can get a meaningful answer out of your system. There's another bar here which is not really very important, but this is just to say that we have a version of this algorithm that does not require your cluster to have exactly equal mass.

Gonna estimate a mask which is more sensible, and if you do that, you get a tiny little bit of improvement further on top of what you have already, but not very much.

Okay, so that's really not crucial.

The crucial thing here is really how you build, which is the inductive biases or the transformation that you use to form these clusters. There's a little demo here that I'm going to skip in just the interest of time, but if you're interested, you can go to this webpage and you can look at some of the clusters that have been found by the algorithm and have a a bit of fun to see what happens. And you're going to see that some clusters are more clearly identified by the visual appearance of the object, and some others are more clearly identified by the sound of the object.

So that also shows that these cues are somehow complementary.

Okay, so that's about clustering, but in general, if you look at the recent machine vision literature, you're going to see that there's been a tremendous amount of work on self-supervised representation learning. So here I'm listing a few methods.

Famous ones are BERT, Monco-Simpleer, and Suave, etc. So what these do is—

well, they don't come up with clusters directly, but what they do is they learn a function phi, the representations that map your image onto a vector. Now, this vector is trained in a way to optimize a certain learning criterion, of course, and is done in a way that does not require you to provide labels to your algorithm, meaning that this is also unsupervised.

And then you get a representation, and this representation turns out to be actually very, very good. So what does it mean for the representation to be good in this context? Well, it means that now if you take it and use it with supervision to learn, for example, a classifier on top of this representation, then you do very well. So it means that you can train the representation itself without labels, and then you need a little bit of labels to just train the last layer of your network, if you like, your classifier. And that performs well. But this still requires labels. All right, so it still requires some supervision to take your representation and map it to interpretable concepts.

But here we are interested in checking whether these algorithms are able to find automatically an interpretation which is meaningful to a human.

And so how can we check?

So we have these vectors that they don't mean anything per se. How can we check whether there is some information which is close to what a human might actually find to be useful in them or not?

Well, we can probe them, and there are two ways of doing that.

One is called direct probing. So what I can try to do is I can try to take these vectors and map them to concepts that have been labeled in a dataset by a human.

And maybe in the process, I'm going to try to use a very, very simple classifier or predictor, for example, just looking at an individual dimension of your representation to try to perform this mapping.

And if that's successful, I can say that that particular dimension of the representation is, for example, looking for cows or dogs or something like that.

That's fine, but a shortcoming of this is that this only works to check whether those concepts that you have labeled manually exist or are somehow represented in your vector. But this is an unsupervised representation, so maybe the representation is learning something which you didn't think about labeling, or maybe something that you didn't label directly.

For example, you could find— there could be a dimension which is sensitive to not dogs in general, but maybe only to furry dogs or large dogs.

And, you know, then this direct probing would actually fail.

So you can do another thing which is called inverse probing, which is something that we have developed more recently, which goes the other way around. So what you do here is you collect a large set of complementary human annotations for an image, and with those you try to predict the clusters that you get by grouping images based on your representation. So essentially, you sweep the role of human labels and representation, and you try to do the mapping the other way around. If you do that, you can get a figure like this.

So what you see here is how much clusters that you get by grouping representations together using something as simple as k-means look if you try to explain them using a combination of labels that have been produced by humans on a given dataset.. For example, you see here that if you take as a dataset ImageNet and use a representation which is trained in a supervised manner to fit ImageNet labels, then if you do this grouping, most of those groups that you find automatically using k-means are explained by the ImageNet labels themselves. That is not surprising because that's your training signal, and so the network has just learned it. But what happens if you remove the ImageNet supervision? You just have the ImageNet images but without the labels. And then you run one of these other algorithms to try to come up with this representation that forms groups automatically. Well, it turns out that in this case, ImageNet labels are still pretty strong—

well, can still provide a strong explanation of the groups you get.

But then other kind of labels, for example, the presence of other kind of objects, the presence of given types of scenes, are actually complementary, and they can explain better the sort of groups that you get without supervision.

So the idea here is that these machines, even if you provide to them a dataset like ImageNet, which is strongly biased towards finding the 1,000 groups that are represented in ImageNet labels, just because those are—

you have example images of those groups—

nevertheless, because this is unsupervised, are going to find concepts which are interpretable which go beyond ImageNet itself.

Okay, so now it's a bit of a problem how you can determine which these concepts are.

And inverse probing provides a way of trying to do that. The other message here is that there is still quite a bit of a gap.

So the groups that you get are partially explainable by human-like concepts, but in part they are not.

So there is still a gap of, if you like, something in this representation that is being found out automatically that either is not meaningful or it is meaningful in a way which we do not understand. Now, for the remaining of the talk, I'm going to shift the gear here a little bit and move to a complementary topic.

So far I discussed the problem of clustering, and the idea was to try to come up with labels for images that are close to the sort of high-level labels that myself as a human, or you in fact, could actually associate to those images, right?

Things like, here's an image of a bird or a particular species of bird.

Now, that's very high-level information in a certain sense, and some of that information only makes sense for humans because they are a byproduct of human culture, if you like. So it's kind of unreasonable to expect a machine that is not a human, it does not live like a human, doesn't have human interaction, to really discover a representation of visual data which is perfectly aligned with the representation that I have in my head, what visual data actually means, at least at the high level.. But if I look at something like geometry, if I look at the shape of this object, I don't really know what it is, but I can build a model in my head of the shape of that.

And if you move the level of geometry and physics, well, this is a layer of representation of data that is much more likely to be in common between a machine, a human, an animal, and maybe even an alien, right? So this is much more close to the metal, if you like, or close to the physics or the natural properties of of matter and something that an agent that operates in a physical space might actually require to learn. That suggests that this kind of interpretation is actually much more likely to be something that it can find automatically and to get something that actually aligns with my human interpretation of geometry, just because it's lower level.

So I'm going to show you a few results that we got in this space of learning geometric interpretations without supervision.

So what do I mean by that?

Well, the most easy thing you can try to do in terms of 3D interpretation is to perform 3D reconstruction.

So it means I look at an image and I want to get a sense of the shape of the object I'm looking at, or perhaps if I'm looking at an object which can animate or deform, I want to have a notion of the deformation of the object to be able to understand the motion of it.

And if you look classically at what people do to try to reconstruct an object or learn to reconstruct an object is to use tons of supervision, actually.

So for example, you can learn a deep network that tries to do reconstruction using as ground truth during training the shape of the object that you actually see in an image.

Or perhaps you can use some weaker form of supervision.

For example, you might want to know the viewpoint you're looking at or the viewpoint of the object you're looking at.

Or maybe you might have the depth map, which is not a full 3D reconstruction, but it gives you additional sort of partial information about the shape of your object in that particular view, you might want to use silhouettes, or maybe you have labeled keypoints, or at least you have the camera viewpoints annotated even though you don't have multiple views of the object. Okay, so these are typical assumptions that people have made throughout the years in order to be able to learn to reconstruct objects in 3D.

So what we did was to try to get rid of all of this stuff and ask ourselves, so how far can we get if we only are given a collection of images of a single object type, category, and how well can we do the reconstruction for that. All right, so we're not going to have information about the 3D shape, the viewpoints, we're not going to have silhouettes, we're not going to have keypoints, depth maps, and we're not going to have multiple views.

We're just going to have a collection of raw images like this.

So these are images of cats. Of course, it's not completely unsupervised, if you like, because I know that these are all images of a consistent object category, in this case cat faces.

So that's still supervision. But other than that, I just plug—

put in my algorithm just a collection of images like those.

And I have a network that learns automatically to build a reconstruction like that, which is fairly convincing. So how did we do that? Well, we're going to exploit one type of structure that exists in most objects that you have in nature, which is symmetry.

So these are examples of animals.

Animals have bilateral symmetry. It doesn't mean that each individual instance of an animal is symmetric, but it means that the model of an animal is symmetric, if you like.

But many other objects exist in nature that are also symmetric.

So it's a restriction to think about symmetric object categories, but it's not so bad.

So then how do we get a network to reconstruct those images? Well, so we're gonna follow a process of what I call here photogeometric auto-encoding.

So what it means is that I take my input image here, I, and I'm gonna pass this to essentially what are 3 networks.

The first network here is going to give me the viewpoint of the object. The second network is going to give me the depth of the object.

And the third is going to reconstruct the texture, or be the color of the object.

And then I'm going to have a box here, which is designed by hand, by the way.

So that's an inductive bias that takes these 3 bits of information, the viewpoint, depth, and the texture, and tries to get me the image back. And eventually, I just close this into a loop. Because the idea is that if this works well, then the input image here should be reproduced by the image I reconstruct here, starting from this decomposition of the image into the 3D geometry.

Okay, so that's why this is called an auto-encoding process. It's just an image gets encoded and then the image gets reconstructed back. Now, if you do that, it doesn't work.

So what happens is that you simply get a copy of the input image into the texture, the depth becomes flat, and the viewpoint becomes trivial, and then you get a perfect reconstruction here. So you get a collapse. So what do we do? The key to make this not collapse is to use the fact that it expects your object to be symmetric. So what we did is to enforce the fact that the texture here should be symmetric and the depth here should be symmetric as well. And you can see that if you don't make that, then reconstruction degenerates. But the minute you add this symmetry constraint, actually you start to see something appearing here. It means that you're going you're going to go towards a solution which is actually not degenerate. Now, there are a bunch of interesting tricks, I think, that we developed to actually make this work well. So the first one is, of course, you want to make some intermediate representation to be symmetric. You can do that through a loss. So you can take that quantity here, your depth, for instance. You can flip it and check that the flipped version and the original version, they match, for example, in L2 loss. You can do it, but that's actually annoying because you need to add another loss term to your energy function, which means tuning another weight parameter, is just not a very good way of doing it. So what we did instead was simply to randomize the— just toss a coin essentially, and with a certain probability take this depth and flip it, but without flipping the viewpoint.

Okay, so now this works. So you're going to get the correct reconstruction here regardless of how you flip the coin, only effectively if the depth and the depth-flipped are identical. Okay, so this is an indirect way of enforcing a constraint, in this particular case a symmetry constraint, which works by, you know, breaking the method unless the quantity that you want to be symmetric in fact is symmetric.

All right, okay, so you get that, and so now you get symmetric texture, symmetric depth, and you get the reconstruction.

There is something more though, so So we have lighting.

So lighting is something that's actually not symmetric in a face or, you know, in an object you want to reconstruct, because it generally comes from a side of the object.

For example, here you see that it comes—

it's coming from the right, I think. And so in order to compensate for that and make the texture actually symmetric and so becoming—

so what we want to do is to really estimate the albedo of the object, not actually the actual texture, which also reflects the light you have. So what we did was to, well, essentially factor the light effect as well.

So we have another network here which estimates the light source What we do is estimating essentially a light map combining the depth with the light source, so we get the shading, and then we combine the shading with the albedo to get a canonical reconstruction. We apply the viewpoints because we also have the geometry from the depth, and we can reconstruct the image like that in the bottom left. And that works better.

There's a final thing that I need to tell you about. So the fact is that these objects are actually not exactly symmetric. They're probably symmetric, For example, you see here that the face has hair, so that's not an asymmetry we can explain just by messing around with lighting. That's just an intrinsic asymmetry of the geometry or the appearance of the face.

So how do we tackle that?

Well, we add the ability of the model—

well, we modify the method in such a way that you can estimate a probability that each individual pixel in your image belongs to a region in your object which is symmetric or not. So what this means is that here we have two versions of a sort of confidence map, and if this is bright, it means that the algorithm is really confident that that point is symmetric in this case, and if it is dark, the algorithm thinks that that point is actually not symmetric, for example, in this case in the hair region.

And now you can combine these two confidence maps into your loss, and you can learn them automatically with some formulation that I don't have time to discuss. All right, okay. And so, you know, then you train that and you get your results, which I'm going to show you a few examples of that now. Yep, okay. All right, so these are some results that we got. So you can see that from a single image, after we train the method, we can get pretty decent reconstructions. The important thing here is that you can actually recognize the person, I guess, from the reconstructed geometry, which really means this is pretty faithful. Okay, but because this is just using a network now to perform the prediction, the image itself doesn't have to be a real image anymore.

So the network is able to abstract away from details of real images and is able, for example, to reconstruct paintings in a way which is pretty realistic, although, you know, convincing.

And it actually can push this even further. So you can take even abstract art, you can put that into your network, and you're still gonna get— because network has now acquired this bias of what a real face looks like—

you're still going to get something which is in between the painting and the real face.

It's pretty interesting that a model which is only trained without labels on a dataset of raw images can actually do that, I think. It also works on videos pretty smoothly.

Again, this is not optimized for videos just yet, but because the reconstruction is stable, it doesn't flicker when you reconstruct a video.

And of course, this doesn't work just for human faces. It also works for animal faces because, again, you don't need labels, so you don't need the 3D ground truth for your face.

You just need a collection of, in this case, cat faces, and you can run it and you're going to get some reasonable results. Okay, so this is a visualization of the asymmetry which has been discovered by the method. And so you see here, for example, in this cat face that the method knows that this part of the cat face is not symmetric because of the texture. But now I also want to briefly mention some work we have done beyond that, which is actually more recent and tries to tackle the problem of adversarial videos.

So lots of the research we have been doing in computer vision so far has been on still images. And to some extent, I think that's a mistake.

Humans don't learn from still images.

In fact, not biological agent, I guess, learns from still images. We learn from an evolving reality.

And an approximation to that is, of course, videos.

So it makes sense to extract information from videos.

Those are much, much richer than images.

And in particular, when you observe a video and there is an object in the video, like for example a bird which is moving around, then you know that the basic geometry and appearance of the object don't change. So the change that occurs in the video is explained by some limited parameter variation.

For example, the viewpoint might differ. Okay, and by using this knowledge, you can then extract a lot of information from those videos that is actually much, much harder to extract if you only have single images.

So here, what we did was to try to learn to reconstruct, in this case, birds, but other object categories could also be in principle done like that, given as input videos and then developing as output a network that can take a single frame and reconstruct the bird shape and appearance.

Now, this is not completely unsupervised.

It's unsupervised in the sense that we do not provide supervision for the 3D geometry of the bird, but we are assuming two things.

So we are assuming that we have a mask extractor, so we have a detector for the bird which is also able to get your segmentation, for example Mask R-CNN, and then we also have an optical flow method that can give us correspondence between video frames.

So this is something I really want to get rid of in the future, but for now we are using it.

Okay, so what we do is we take the videos, we extract the mask, we extract the optical flow, And with that, we can learn another photogeometric autoencoding network that tries to explain those measurements starting from 3 bits of information: the rigid pose of the object, a sort of deformation of the shape of the object, which is going to be done through a little skeleton that explains how the object deforms in a video, and then also the construction of the texture. Then you combine those things together and you can get an explanation of the image appearance, of the mask of the image, and the optical flow of the image. That you can then compare with what you have measured to try to close this sort of auto-encoding loop.

So this is just a few more details about how the method works.

So the prior shape for the object is just a sphere, so it's not committed to look like a bird.

And then as learning progresses, what we're learning is a sort of category-level shape.

So you see here that we learn that on average, this object looks like a bird. And then for each particular video, we have an ability of further deforming that sort of genetic template shape into a more specific template for that bird.

I also have frame articulation.

That means that I have a little skeleton, which in this case is a very simple structure which is linear, that can explain how the bird deforms, for example, to allow the bird to turn its head. And then I have a rigid transformation that tells me how the bird is posed in the image, but that explains the overall viewpoint of the bird, if you like. And given that, I can do the rendering and then compare my rendered results to the input that they get into the algorithm and try to optimize it. So these are some examples of the results. So you see, you see here input images, these are the 3D shapes that are estimated by the method.

Now you can extract them, you can look around them, and you know, they don't look perfect, but they look, I think, kind of reasonable.

You can add texture on top, and maybe here I can briefly switch to if I can, to my browser so I can show you. Yeah, this is what this looks like in action.

I don't know how well you can see this through the connection, but you should see a video of a bird on the left, and on the right is the reconstruction. And this is interactive, so I can, as the bird animates, I can rotate around and look what the method has done. All right, so that's the kind of reconstruction I can get with this. So you'll notice that, you know, it does deform in a way which is somehow meaningful with respect to the input video here.

So it's tracking the motion okay-ish, but you know, it's still not perfect, of course. The jump is too smooth, the penis reconstruction is too smooth as well.

So there is still some work to be done before we can declare victory here, but that's the sort of thing we want to—

we are now developing.

Okay, all right, so switching back to the presentation, I would like to conclude the presentation by showing you a bit of the work we have been doing in Facebook about scaling up 3D understanding. So what I showed you so far are two works that I have done in Oxford, actually. And those are about reconstructing a single category at a time.

That's already hard. But truly, the variety of stuff that exists in the natural world is much, much larger than just a few categories.

So how are we going to go about scaling up 3D understanding to the complexity of the stuff that's out there? Well, one thing I have contributed recently is a new dataset. By the way, this is not so much my work but the work of my collaborators in Facebook, just to be clear.

So this is a new dataset which is useful for researching 3D reconstruction of particular object categories. So if you look at the data that exists out there, you have synthetic datasets like ShapeNet that are interesting and useful and large, but they are synthetic, so they're not very realistic.

So if you train on this data, sometimes what you learn, the models you learn, they don't really transfer nicely to the real case.

On the other hand, you have lots of real datasets, but they are mostly single-view datasets.

And most of them don't even come with the 3D geometric ground truth.

So you cannot really use them easily to learn about this thousands of categories that exist in the world, but maybe probably hundreds of thousands.

So what we did here was to develop a new dataset which we called CO3D, which stands for Common Objects in 3D. That's a play on word on COCO, which is Common Object, the Common Object version of Microsoft that only has 2D information. So this is going to cover some of the same categories in COCO, but it's going to do that in a way which is more suitable for research in 3D reconstruction. Now, this is going to be less challenging than working on single images because we're going to have multiple views, and from that we can actually get rather easily the 3D reconstruction of at least some of the points in those images, but it's much more realistic than something like ShapeNet just because it's actually real data.

All right, so this is a little video that shows the content of this dataset. So what you see here on the left are the videos that are being captured, and on the right is a 3D reconstruction we got from those videos.

And the way we collected these datasets was, well, the opposite of labeling data using Mechanical Turk.

We still use Mechanical Turk, but what we gave to the mechanical turkers was not a video or an image to annotate, but was a label, and their task was to collect a video of the object.

So this is collected on a fairly large scale. We have roughly 20,000 videos of, I think there's 50 object categories.

And so you can see here a sort of panoramic view of what you might expect to find in this data. So you're going to get lots of videos specific to one object at a time, and the camera is going to go around them. So you're going to see the different views.

Now with that, so that's again, these are some of the example videos we have collected using Mechanical Turk.

And then on top of that, we have done for you some preliminary reconstruction using standard techniques like COLMAP, Structure from Motion essentially.

That can reconstruct the camera viewpoint change and can also give you a partial reconstruction of the scene itself.

And this dataset is available now at that address if you're interested.

Now, once you have this data, what can you do with it? So one thing you can do with it is to learn to perform 3D reconstruction from a single view or perhaps a small number of views. So what we did here is to take datasets for an object category like in this case, a fire hydrant. And we use those videos for training. Then at test time, we have a machine that is able to take 1 to 10 views of a new instance of the object category and spit out immediately a 3D reconstruction in a sort of feedforward manner. So this is similar to what I showed before, the ability of the method to reconstruct, say, a face or a bird. But now this is much more generic because it works for many different object categories.

And arguably these categories are more complex.

So how do we do this reconstruction? So this is based on neural rendering.

I don't know whether you're familiar with that or not, but neural rendering is an increasingly popular way of representing 3D geometry in deep learning. It works in a very simple manner. So what we do is you want to reconstruct an image of an object like an apple. So the way you're going to represent your scene is through well, what is just a multilayer perceptron, MLP. And this MLP is able, for each 3-dimensional point, to give you 2 bits of information. One is the density, so whether that point in space is occupied or not by your object, and the other one is the color, so if the point is occupied, what color it has. Now, given that information, which is available for every 3D point in space, what you can do is you can shoot a ray from the camera into 3D space. You can look up the colors of those 3D points look at where your ray hits the first occupied point in the volume, and with that you can get the reconstruction of the color. If you repeat that for all possible pixels, then you can get the reconstruction of the image. But now the entirety of the 3D geometry and appearance of your object is stored into a multilinear perceptron, MLP.

There is no further structure, there is no convolutional network here. Now, how do you learn something like this? Well, you start from a bunch of training images from which you expect to know the viewpoints, Okay, so that's given to the method. Then you're going to get an input image, you're going to get your MLP, which is a representation of the geometry and appearance. You're going to shoot a ray, you're going to try to do the reconstructions as explained, and you're going to compare your reconstructed image to the ground truth image you have here.

You're also going to do that for a mask that is available for those objects, and you're going to try to make the mask and the appearance of the object as close as possible to the ground truth versions of these two bits of information. And with that, you just optimize the parameters of your multilayer perceptron until this works. Okay, it's extremely straightforward, at least conceptually.

This works well if you have only one object to reconstruct, but what if you have multiple objects? So suppose you have all apples in the world that you want to reconstruct.

Well, in this case, a single MLP is not enough, or at least it's not enough unless you have something that changes in specific for each object you want to reconstruct. So how are you going to get— one way to do that is to modify your MLP so that it takes as input also a vector z, which is a code that specifies the geometry and appearance of that particular version of the apple, say.

Okay, so now we have a space of models that can reconstruct not just a single instance, but can reconstruct any apple.

And we call this latent code because that's what it is.

And one way to get that is to get the code so you can optimize the code for each individual video you have together with the rest of the parameters of the MLPs.

But there is a more efficient way of doing that from a practical perspective, which is to say to build an autoencoder. So we're going to develop— we can develop or think about developing another network like a CNN that given a single input image here directly gives you the code z that you need to use in order to get the 3D shape and appearance of that object that is seen in that particular image.

Okay, now if you do that, actually it doesn't work. It's actually a very, very difficult problem to, given an image, come up with the right code that gives you the reconstruction you want.

So one contribution that we had this year was to develop new ways of taking an image, or maybe a small number of images, and extracting a code that would actually give you the reconstruction.

So we call that WCE, which stands for word conditional ray embedding.

So you see here the comparison between the model explained before, which I called here NeRF-like. NeRF is maybe the most famous version of this class of models. And then we have this VCE version, which is very similar except that what you input in your MLP is not the 3D points directly, but is an encoding of these 3D points, which is called z(x). The z(x) is obtained by taking an input camera you have, so an input view, an example view, shooting the ray from the viewpoint of the camera you want to reconstruct, considering each point in 3D as before, looking at where they project onto the other views, and collecting some features that are being computed on the 2-dimensional image, for example using a ResNet, so that you can get an initial value for z(x).

Okay, so again I'm going to repeat.

So what you're going to do is you want to reconstruct the view from a target camera, you're going to shoot a ray, you're going to observe individual 3D points, for each one of those you're going to look at where they land in your images, which you can do because you have the relative camera motion. And you're going to pick features, 2D features, in correspondence of those points where they land. Okay, and that's going to be your z. All right, so now this is a little video that illustrates that, that was done by the intern who actually developed this.

And I would never be able to generate an animation like this, so I don't know how he did it.

But okay, so these are input views, what you see here on the screen.

On the left, and on the right you get another example. And now you have a target view. So you have, from input views, you have extracted features using a CNN like a ResNet. Now you get a target view, you want to reconstruct that.

So you shoot rays for these points on the 3D rays, you're going to look up where they land on the other views, you're going to pull the features that you get from those views, okay?

And you're going to collect information, you're going to fuse it using an aggregation function which just collects information from different views, and that's based on the viewing angle.

So you're gonna weight more information that comes from views that are close to the view you want to generate.

That makes sense.

Okay, and yeah, that then is fed to this MLP to actually do the 3D reconstruction or, you know, estimate the occupancy and color.

And these are some of the results that you get here, and, you know, they're okay, but they're not great yet. So why are they—

so there are two messages here. First, it's not great, but they are much, much better than what you can get if you don't use these techniques to actually encode information.

So you see here some performance comparisons using NeRF without this way of pulling information using this word-conditioned array embedding and NeRF that actually does that.

And you can see that there is a big gap in performance.

So this is actually very, very important. But then, and this is again not my work, we revisited this story and tried to come up with a better architecture, which we did using transformers.

So you probably know that transformers are now all the rage in computer vision, so we decided to give it a try, and it actually worked well for this task as well. So we use them in two ways.

The first one is, so when you want to paint your points, you need to collect information across along it, right? So you need to collect information across the different 3D points that you have sampled and then put it together in order to get the color of the pixel that you want to render.

So that can be done by a transformer in a very natural way, transformer architecture. So we do that.

The other thing is that we have to pull information across multiple views. So you see here that information about these 3D points is collected from different user objects.

That's another pooling operation. It turns out that you can also implement that using a transformer.

So the architecture that we get looks something like this. So we have a view dimension in which we change the view of the camera that is looking at the object. We have a ray dimension, which looks at pulling information across a ray.

And then we alternate transformers that operate along a ray and along view interactions to aggregate information and eventually come up with the color of the pixel.

And yeah, so if you do that, everything works better. So you see here a comparison between this NeRF-Former, which is this NeRF plus transformer, and other methods. And all of them, you will notice, they use this forward-conditioned re-embedding that I explained before, because if you don't use it, nothing really works. Okay, and the former tends to work better in terms of image quality and is about the same or just slightly worse in terms of geometry reconstruction. So it's a little bit overfitting that, as opposed to another technique, which is called SRN, which is also being modified to use or work conditional re-embedding.

And these are some of the results you can get.

So you can see the comparisons between the original NeRF, SRN, and the NetFormer.

And clearly, these results are much more sharp and more pleasing.

They just look better.

And here's another example. And again, these have been obtained by using as input only a few views of the object, and everything else is done in a feedforward manner, so it's actually very fast.

Okay, I would have a last chapter, but I don't know that I have enough time to discuss it in any detail.

So it would be great if we could leave time for questions.

Do you want to? Exactly.

So I'm going to skip over this and move to the conclusions then, if that's okay. All right.

Okay.

All right, so I mean, the first part of the talk was about clustering, right? And I explained this method for self-labeling, which actually turns out to work fairly well in terms of being able to find clusters in images which are reasonably aligned to human-provided clusters.

But Anna suggested that, in my view, probably we should also look, or maybe first look at reconstructing geometry and physics. That's just because at a lower level there is a much better chance for machines to actually understand that type of information as opposed to high-level information that may require some human-like experience to develop. And I presented two lines of work there. One is reconstructing objects in 3D using no supervision, one category at a time. And then I discussed a little bit this problem of scaling up this level, this type of understanding, to in this case, dozens of different categories and eventually to thousands of them.

And that's the work which is close to what we do in Facebook instead.

And of course, there is a little advertisement there.

We are looking for postdocs and research engineers, just in case someone is interested.

OK, so thank you. Thanks, Andrea. Very, very nice talk and lots of food for thought. On which of these concepts we can adapt to our problems. Yeah, everybody's free to ask questions. Doreen, I think, prefers if you type in the chat. So if you have a question, just type it there or raise a hand and we'll unmute you. Let me start, Andrea, on the clustering part. So you had a slide where you said you learned this matrix Q of the assignments, and then you learn the representation itself. And without any other measures, the algorithm will take the easy way out and just put everything into a single cluster. So do you optimize over— in this picture, do you optimize over cluster locations also, or are your cluster centers always at the corners of a simplex?

No, they are not.

They are— I mean, the figure is a bit approximate.

So what we're learning is a distribution Q. Let's see what we can highlight it here. Yeah, so this guy here, Q.

So you see here two quantities. One is phi, which is my network that does maps the image onto the label, and Q is a probability distribution that for each index of an image gives a distribution of possible labels. So that's what this picture here—

and so, you know, this is a matrix which has dimension k by n, where k is number of classes, n is the number of images you want to cluster.

And of course now this is like, yeah, of course is a binary matrix in this example, but in reality that would have a certain number of non-zero entries.

So there is a degree of membership of each data point to one or more clusters.

Which is you need in order for this stuff to be easily learnable, of course. In this Q, is that predicted by a network itself?

No, Q is just a table.

Okay.

Yes, so the parameters of the network are just all in phi, and Q is just a table.

So you can think of that as a soft version of assigning labels to your data.

So it's a bit like—

yeah, it's a form of auto-labeling, effectively. Yes.

And then where does the— in your equations, where does the desire that clusters be compact, where does that enter?

Compact in the sense of, well, there isn't really any force that encourages clusters to be compact as such, except that the network itself is a continuous function, right?

So it will not like to map classes to— so it prefers to map similar images to similar labels, say.

But that's the only constraint that there is.

Otherwise, it is possible to have classes that are not compact.

Yes.

Okay, thanks very much. I'm going through the questions in chronology. Sebastian asks about clusters that are not of equal size, because presumably in videos some categories are more frequent than others and so on.

What do you do there? Indeed.

So I briefly mentioned that.

So I have to say that the motivation for this method initially was not to get good clusters, but was to get good features.

So we were still working on the representation learning, and then the idea is that once you have a representation, use that for some other class using a little bit of supervised information. Then we tested this in terms of how good the classes are directly, and we found out they're actually pretty good and better than the sort of classes you get by classing other kinds of representations.

And this was even using the constraint that the classes have about equal mass.

Now, part of the reason could be that the data we use for testing, they tend to be relatively uniform, but VGG-Sound is actually not, so that definitely has long tails.

So what we did when we moved this into the—

when we applied this to videos was also to modify the method in such a way that your label assignment would actually follow a sort of Zipf law, one of these geometric distributions that is closer to the sort of distribution of labels you can expect in real data.

And so we have done that, but the performance in terms of classifier quality doesn't really change.

Actually, I'll show you a graph that was just illustrating this point.

So actually, this is what happens if you relax this notion of uniformity and you move to clusters which are non-uniform.

You can gain a little bit, but it's just marginal. So I wouldn't even— yeah, this is barely significant, I guess.

So I think the short answer is that even if you allow for the clusters to be non-uniform, we're still sufficiently far off from perfect clusters that this doesn't really matter.

Thank you very much. Next is Ollie, who is multitasking.

I was wondering, Andrea, this is really nice, the invariances, you know, and then you use those for the semantic clustering. I mean, with the 3D data that you collected at the very end of your talk, is there an opportunity to also learn these invariances from data and then sort of go back to the first problem? I mean, you mentioned, you know, using the smoothness of images and videos, right, to sort of create the semanticness, but And obviously having the 3D view of these objects, is there a way to sort of solve this completely data-driven without using prior information?

Okay, so I think—

I don't know what this is going to answer your question specifically, but the advantage of having a video is that throughout the video you're going to expect most of this stuff not to change, right? So for example, the meaning of your objects, the class of objects, is not going to change. In fact, even the instance of your object is not going to change.

Which means that, sure, by looking at the video, you can make some educated guesses about what is invariant, and therefore everything that changes is probably something you want to be invariant to anyway, in the sense it's not semantically relevant, right? For example, the viewpoint.

So yes, videos can provide you with fairly direct cues about some of these factors that are— in vision, we call them nuisance factors sometimes, like a viewpoint. That make recognition of an object or reconstruction of an object difficult. And so having access to video can give you direct evidence about the effect of those factors because the other factor is actually constant. So to that extent, certainly videos can help a lot. In fact, I think if you want to learn automatically about categories that exist in the world, for example, similar to this testing problem, you're probably going to be better off if you can use videos at least initially to build your viewpoint invariant representations first. So yeah, sure, the videos can help. Discover invariances automatically— well, depends what you mean by that. So I think we know that we want to be invariant to viewpoint in terms of recognizing an object regardless of the position of the camera.. And learning how to do that, certainly something we can do from the videos, but the notion that we want to be viewpoint invariant, that's so basic that you can just encode it as an inductive bias in how you design your model, I think.

So the concept of viewpoint itself, I suppose you could discover it from videos automatically, but it's just easier to put it there in the model as a design choice.

Yeah, no, it makes sense. Yeah, I thought this morning, you know, many of these invariances you also showed when you rotate an image, you know, you change the, you know, the color brightness and these elements, you know, these are all to some extent also technical invariances, right, that are not, you know, not necessarily linked, you know, you basically create synthetic invariances as well. And I was just wondering, you know, how these two worlds would come together.

Yeah.

Yeah, I mean, sure. I mean, so if you're a little bit familiar with how we do contrastive learning, for example, now you synthesize some of this nuisance effect by changing, simulating a change in illumination of your image by changing the colors quite dramatically.

So sure, these are things that exist naturally in videos, so you don't need to add them artificially anymore.

So sure, we can get them for free in a way.. But these are probably relatively minor effects compared to the change in viewpoint, which is the dominant thing that happens in a video. The other thing which is really nice that happens in a video is the fact that objects can actually move and articulate, which is something which is really, really difficult otherwise to learn from still images. For example, human motion.

So it can give you cues about that, about what it means for— it can reveal the articulated nature of some of those objects that it would be very, very hard to grasp otherwise.

Thank you very much. Okay, Sebastian has one more and then we should close because our hour is up. In the chat he writes, it seems that deep clustering methods that use transformations to prevent collapse outperform methods that use reconstruction. When learning geometry, you do use reconstruction.

Any comments? It's a very interesting debate, actually. So in fact, it's something that we keep discussing, whether generation is necessary for unsupervised learning or not.

I mean, it's not necessary, of course, but if you want, is that optimal, or is that what you should really—

is that the best way of doing it or not? When it comes to geometry, the answer is probably yes, because Well, it depends on the granularity of reconstruction, but you really want to be able to get information which is accurate at the level of individual pixels because that's enough to really change the shape of an object. So if you move part of an object just a few pixels in your image, then the 3D location of the object can actually change quite a lot as a result. So in that sense, you want to reconstruct to explain information at the level that is comparable to the accuracy of individual pixels.

Whether that amounts to reconstructing the pixel color exactly is debatable, but It's pretty close, I guess. When it comes to learning something like high-level concept of clusters, it's much less clear that you need to be generative or not.

There's plenty of work on generation, and I think it's definitely needed. It's worth keeping in mind and try out.

But again, I'm not completely convinced it's necessary. Let's put it this way. For everything. Okay, so yeah, um, having said that, some people might argue that k-means is a way—

is a form of— it's a very simple generative method, right?

Because your cluster centers are an approximation of your data. So you can think that, you know, sampling clusters amounts to sampling data points to a very, very crude approximation.

So go figure. Okay, sounds like that is, you know, to be continued. I mean, not right now, but unresolved question.

Good one.

Okay, then thanks so much, Andrea. Thanks so much, everybody.

Thank you.

We wish everybody a great summer vacation.

Yeah, so also from my side, Andrea, thank you very much for being with us. It has been a pleasure.

Thank you.

Very good.

Thank you very much then.

All right.

Yeah, goodbye everyone.