Ishan Misra: Self-Supervised Deep Learning in Computer Vision | Lex Fridman Podcast #206

The following is a conversation with Ishan Misra, Research Scientist at Facebook AI Research who works on self supervised machine learning in the domain of computer vision or in other words, making AI systems understand the visual world with minimal help from us humans. Transformers and Self Attention has been successfully used by OpenAI's GPT3 and other language models to do self supervised learning in the domain of language. Ishan, together with Yann Lecun and others is trying to achieve the same success in the domain of images and video. The goal is to leave a robot watching YouTube videos all night and in the morning come back to a much smarter robot. I read the blog post Self Supervised Learning the Dark Matter of Intelligence by Ishaan and Yann Lecun and then listened to Ishaan's appearance on the excellent Machine Learning Street Talk podcast and I knew I had to talk to him. By the way, if you're interested in machine learning and AI, I cannot recommend the ML Street Talk podcast highly enough. Those guys are great. Quick mention of our sponsors Onnit, the information Grammarly and athletic greens. Check them out in the description to support this podcast. As a side note, let me say that for those of you who may have been listening for quite a while, this podcast used to be called Artificial Intelligence Podcast because my life passion has always been will always be artificial intelligence, both narrowly and broadly defined. My goal with this podcast is still to have many conversations with world class researchers in AI, math, physics, biology and all the other sciences. But I also want to talk to historians, musicians, athletes and of course occasionally comedians. In fact, I'm trying out doing this podcast three times a week now to give me more freedom with guest selection and maybe get a chance to have a bit more fun. Speaking of fun, in this conversation I challenge the listener to count the number of times the word banana is mentioned. Ishan and I use the word banana as the canonical example at the core of the hard problem of computer vision and maybe the hard problem of consciousness. This is the Lex Friedman podcast and here is my conversation with Ishan Misra. What is self supervised learning? And maybe even give the bigger basics of what is supervised and semi supervised learning? And maybe why is self supervised learning a better term than unsupervised learning?

Let's start with supervised learning. So typically for machine learning systems, the way they're trained is you get a bunch of humans, the humans point out particular concepts. So if it's in the case of images you want the humans to come and tell you what is present in the image, draw boxes around Them draw masks of pixels which are of particular categories or not. For nlp, again, there are lots of these particular tasks say about sentiment analysis, about entailment and so on. So typically for supervised learning, we get a big corpus of such annotated or labeled data and then we feed that to a system and the system is really trying to mimic, so it's taking this input of the data and then trying to mimic the output. So it looks at an image and the human has tagged that this image contains a banana. And now the system is basically trying to mimic that. So that's its learning signal. And so for supervised learning, we try to gather lots of such data and we train these machine learning models to imitate the input output. And the hope is basically by doing so now on unseen or like new kinds of data, this model can automatically learn to predict these concepts. So this is a standard sort of supervised setting. For semi supervised setting, the idea typically is that you have of course, all of the supervised data, but you have lots of other data which is unsupervised or which is like not labeled. Now the problem basically with supervised learning and why you actually have all of these alternate sort of learning paradigms is supervised learning just does not scale. So if you look at, for computer vision, the sort of largest, one of the most popular data sets is ImageNet. So the entire ImageNet data set has about 22,000 concepts and about 14 million images. So these concepts are basically just nouns and they're annotated on images. And this entire data set was a mammoth data collection effort. It actually gave rise to a lot of powerful learning algorithms, is credited with the rise of deep learning as well. But this data set took about 22 human years to collect, to annotate. And it's not even that many concepts, right? It's not even that many images. 14 million is nothing really. Like you have about, I think 400 million images or so, or even more than that, uploaded to most of the popular sort of social media websites today. So now supervised learning just doesn't scale. If I want to now annotate more concepts, if I want to have various types of fine grained concepts, then it won't really scale. So now you come up to these sort of different learning paradigms. For example, semi supervised learning, where the idea is, of course, you have this annotated corpus of supervised data and you have lots of these unlabeled images. And the idea is that the algorithm should basically try to measure some kind of consistency or really try to measure some kind of signal. On this sort of unlabeled data to make itself more confident about what it's really trying to predict. So by access to this lots of unlabeled data, the idea is that the algorithm actually learns to be more confident and actually gets better at predicting these concepts. And now we come to the other extreme, which is like self supervised learning. The idea basically is that the machine or the algorithm should really discover concepts or discover things about the world, or learn representations about the world which are useful without access to explicit human supervision.

So the word supervision is still in the term self supervised. So what is the supervision signal? And maybe that perhaps is when Yann Lecun and you argue that unsupervised is the incorrect in terminology here. So what is the supervision signal when the humans aren't part of the picture or not a big part of the picture.

Right. So self supervised, the reason it has the term supervised in itself is because you're using the data itself as supervision. So because the data serves as its own source of supervision, it's self supervised in that way. Now the reason a lot of people, I mean we did it in that blog post Vidyan, but a lot of other people have also argued for using this term self supervised. So starting from like 94 from Virginia Dessal's group at I think UCSD, now she's at UCSD. Jeetendra Malik has said this a bunch of times as well. So you have supervised and then unsupervised basically means everything which is not supervised. But that includes stuff like semi supervised, that includes other transductive learning, lots of other sort of settings. So that's the reason now people are preferring this term self supervised, because it explicitly says what's happening. The data itself is the source of supervision. And any sort of learning algorithm which tries to extract just sort of data supervision signals from the data itself is a self supervised learning algorithm.

But there is within the data a set of tricks which unlock the supervision.

Right?

So can you give maybe some examples? And there's innovation ingenuity required to unlock that supervision. The data doesn't just speak to you some ground truth, you have to do some kind of trick. So I don't know what your favorite domain is, so you specifically specialize in visual learning, but is there favorite examples maybe in language or other domains?

Perhaps the most successful applications have been in NLP language processing. So the idea basically being that you can train models that can you have a sentence and you mask out certain words and now these models learn to predict the masked out words. So if you have like the cat jumped over the dog, so you can basically mask out cat. And now you are essentially asking the model to predict what was missing. What did I mask out? So the model is going to predict basically a distribution over all the possible words that it knows. And probably it has like if it's a well trained model, it has a sort of higher probability density for this word cat. For vision, I would say the sort of more, I mean the easier example which is not as widely used these days is basically say for example, video prediction. So video is again a sequence of things. So you can ask the model. So if you have a video of say 10 seconds, you can feed in the first nine seconds to a model and then ask it, hey, what happens basically in the 10 second, can you predict what's going to happen? And the idea basically is because the model is predicting something about the data itself, of course you didn't need any human to tell you what was happening because the 10 second video was naturally captured. Because the model is predicting what's happening there, it's going to automatically learn something about the structure of the world, how objects move, object permanence and these kinds of things. So like if I have something at the edge of the table, it'll fall down. Things like these, which you really don't have to sit and annotate. In a supervised learning setting I would have to sit and annotate. This is a cup. Now I move this cup, this is still a cup. And now I move this cup, it's still a cup and then it falls down. And this is a fallen down cup. So I won't have to annotate all of these things in a self supervised setting.

Isn't that kind of a brilliant little trick of taking a series of data that is consistent and removing one element in that series and then teaching the algorithm to predict that element? Isn't that first of all, that's quite brilliant. It seems to be applicable in anything that has the constraint of being a sequence that is consistent with the physical reality. The question is, are there other tricks like this that can generate the self supervision signal?

So sequence is possibly the most widely used one in NLP for vision. The one that is actually used for images, which is very popular these days, is basically taking an image and now taking different crops of that image. So you can basically decide to crop, say the top left corner and you crop say the bottom right corner and asking a network to basically present it with a choice saying that, okay, now you have this image, you have this image Are these the same or not? And so the idea basically is that because different, like in an image, different parts of the image are going to be related. So for example, if you have a chair and a table, basically these things are going to be close by versus if you take, again if you have like a zoomed in picture of a chair, if you're taking different crops, it's going to be different parts of the chair. So the idea basically is that different crops of the image are related and so the features or the representations that you get from these different crops should also be related. So this is possibly the most widely used trick these days for self supervised learning and computer vision.

So again, using the consistency that's inherent to physical reality in visual domain that's, you know, parts of an image are consistent and then in the language domain or anything that has sequences like language or something that's like a time series, then you can chop off parts in time. It's similar to the story of RNMs and CNNs, of RNNs and Convnets. You and Yann LeCun wrote the blog post in March 2021 titled Self Supervised Learning the Dark Matter of Intelligence. Can you summarize this blog post and maybe explain the main idea or set of ideas?

The blog post was mainly about sort of just telling. I mean this is really an accepted fact. I would say for a lot of people now that self supervised learning is something that is going to play an important role for machine learning algorithms that come in the future and even now.

Let me just comment that we don't yet have a good understanding what dark matter is.

That's true.

The idea, maybe the metaphor doesn't exactly transfer, but maybe it's actually perfectly transfers that we don't know. We have an inkling that it'll be a big part of whatever solving intelligence looks like.

Right. So I think self supervised learning, the way it's done right now is I would say like the first step towards what it probably should end up learning or what it should enable us to do. So the idea for that particular piece was self supervised learning is going to be a very powerful way to learn common sense about the world or like stuff that is really hard to label. For example, is this piece over here heavier than the cup? Now for all these kinds of things, you'll have to sit and label these things. So supervised learning is clearly not going to scale. So what is the thing that's actually going to scale? It's probably going to be an agent that can either actually Interact with it. So lift it up or observe me doing it. So if I am basically lifting these things up, it can probably reason about, hey, this is taking him more time to lift up. Or the velocity is different, whereas the velocity for this is different, probably this one is heavier. So essentially by observations of the data, you should be able to infer a lot of things about the world without someone explicitly telling you, this is heavy. This is not. This is something that can pour. This is something that cannot pour. This is somewhere that you can sit. This is not somewhere that you can sit.

But you just mentioned ability to interact with the world. There's so many questions that are yet to be, that are still open, which is how do you select the set of data over which the self supervised learning process works? How much interactivity, like in the active learning or the machine teaching context is there? What are the reward signals, like how much actual interaction there is with the physical world? That kind of thing. So that could be a huge question. And then on top of that, which I have a million questions about which we don't know the answers to, but it's worth talking about is how much reasoning is involved, how much accumulation of knowledge versus something that's more akin to learning or whether that's the same thing. But so we're like, it is truly dark matter.

We don't know how exactly to do it. Yeah, but we are, I mean, a lot of us are actually convinced that it's going to be a sort of major thing in machine learning.

Let me reframe it then. That human supervision cannot be at large scale the source of the solution to intelligence.

Right.

So there has. The machines have to discover the supervision in the natural signal of the world.

Right. I mean, the other thing is also that humans are not particularly good labelers. They're not very consistent. For example, what's the difference between a dining table and a table? Is it just the fact that one, if you just look at a particular table, what makes us say one is dining table and the other is not? Humans are not particularly consistent. They're not very good sources of supervision for a lot of these kind of edge cases. So it may be also the fact that if we want an algorithm or want a machine to solve a particular task for us, we can maybe just specify the end goal and like the stuff in between, we really probably should not be specifying because we're not maybe going to confuse it a lot actually.

Well, humans can't even answer the meaning of life. So I'm not sure if we're good supervisors of the end goal either. So let me ask you about categories. Humans are not very good at telling the difference between what is and isn't a table, like you mentioned. Do you think it's possible, let me ask you, like a. A pretend you're Plato. Is it possible to create a pretty good taxonomy of objects in the world? It seems like a lot of approaches in machine learning kind of assume a hopeful vision that it's possible to construct a perfect taxonomy. Or it exists perhaps out of our reach, but we can always get closer and closer to it. Or is that a hopeless pursuit?

I think it's hopeless in some way. So the thing is, for any particular categorization that you create, if you have a discrete sort of categorization, I can always take the nearest two concepts or I can take a third concept and I can blend it in and I can create a new category. So if you were to enumerate N categories, I will always find an N+1 category for you that's not going to be in the N categories. And I can actually create, not just n plus one. I can very easily create far more than N categories. The thing is, a lot of things we talk about are actually compositional. So it's really hard for us to come and sit and enumerate all of these out. And they compose in various weird ways, right? Like you have like a croissant and a donut come together to form a cronut. So if you were to like enumerate all the foods up until, I don't know, whenever the cronut was about 10 years ago or 15 years ago, then this entire thing called cronut would not exist.

Yeah, I remember there was the most awesome video of a cat wearing a monkey costume. People should look it up. It's great. So is that a monkey or is that a cat? It's a very difficult philosophical question. So there is a concept of similarity between objects. So you think that can take us very far, just kind of getting a good function, a good way to tell which parts of things are similar and which parts of things are very different?

I think so, yeah. So you don't necessarily need to name everything or assign a name to everything to be able to use it. Right. So there are like, lots of Shakespeare.

Said that what's in a name?

What's in a name?

Yeah.

Okay. And I mean a lots of like, for example, animals, Right. They don't have necessarily a well formed, like syntactic language, but they're able to go about their day perfectly. The same thing happens for us. So I mean, we probably look at things and we figure out, oh, this is similar to something else that I've seen before and then I can probably learn how to use it. So I haven't seen all the possible doorknobs in the world, but if you show me like I was able to get into this particular place fairly easily, I've never seen that particular doorknob. So I of course relate it to all the doorknobs that I've seen and I know exactly how it's going to open or I have a pretty good idea of how it's going to open. And I think this kind of translation between experiences only happens because of similarity, because I'm able to relate it to a doorknob. If I related it to a hairdryer, I would probably be stuck still outside, not able to get in.

Again, a bit of a philosophical question, but can similarity take us all the way to understanding a thing? Can having a good function that compares objects get us to understand something profound about singular objects?

I think I'll ask you a question back. What does it mean to understand objects?

Well, let me tell you what that's similar to. No, I. So there's an idea of sort of reasoning by analogy kind of thing. I think understanding is the process of placing that thing in some kind of network of knowledge that you have that it perhaps is fundamentally related to other concepts. So it's not like understanding is fundamentally related by composition of other concepts and maybe in relation to other concepts and maybe like deeper and deeper understanding is maybe just adding more edges to that, to that graph somehow. So maybe it is a composition of similarities. I mean, ultimately I suppose it is a kind of embedding in that wisdom space.

Yeah, okay. Wisdom space is good, I think. Do think. Right. So similarity does get you very, very far. Is it the answer to everything? I mean, I don't even know what everything is, but it's going to take us really far. And I think the thing is things are similar in very different contexts. Right. So an elephant is similar to, I don't know, another sort of wild animal. Let's just pick, I don't know, lion in a different way because they're both four legged creatures. They're also land animals, but of course they're very different in a lot of different ways. So elephants are like herbivores, lions are not. So similarity. Does similarity and particularly dissimilarity also actually helps us understand a lot about things? And so that's actually why I think Discrete categorization is very hard, just like forming this particular category of elephant and a particular category of lion, maybe it's good for just taxonomy, biological taxonomies. But when it comes to other things which are not as maybe for example, like grilled cheese. I have a grilled cheese, I dip it in tomato and I keep it outside. Now is that still a grilled cheese or is that something else?

Right, so categorization is still very useful for solving problems. But is your intuition then sort of the self supervised should be the, to borrow Yann Lecun's terminology, should be the cake. And then categorization, the classification. Maybe the supervised like layer should be just like the thing on top, the cherry or the icing or whatever. So if you make it the cake, it gets in the way of learning.

If you make it the cake, then you don't. We won't be able to sit and annotate everything that's as simple as it is. Like, that's my very practical view on it. It's just, I mean in my Ph.D. i sat down and annotated like a bunch of cars for one of my projects. And very quickly I was just like it was in a video and I was basically drawing boxes around all these cars. And I think I spent about a week doing all of that and I barely got anything done. And basically this was I think my first year of my PhD or like second year of my master's. And then by the end of it I'm like, okay, this is just hopeless, I can keep doing it. And when I'd done that, someone came up to me and they basically told me, oh, this is a pickup truck, this is not a car. And that's like, aha. This actually makes sense because a pickup truck is not really like what was I annotating? Was I annotating anything that is mobile? Or was I annotating particular sedans or Was I annotating SUVs?

What was I doing, by the way? The annotation was bounding boxes.

Bounding boxes.

There's so many deep profound questions here that you, you're almost cheating your way out of by doing self supervised learning, by the way, which is like what makes for an object? I suppose to solve intelligence. Maybe you don't ever need to answer that question. I mean this is the question that anyone that's ever done an annotation, because it's so painful, gets to ask, like, why am I doing a drawing? Very careful line around this object, like, what, what, what is the value? I remember when I first saw semantic segmentation where you have like instant segmentation where you have a very exact line around the object in a 2D plane of a fundamentally 3D object projected on a 2D plane. So you're drawing a line around a car that might be occluded, there might be another thing in front of it, but you're still drawing the line of, of the part of the car that you see. How is that the car? Why? Why is that the car? Like I had like an existential crisis every time. Like how is that going to help us understand, solve computer vision? I'm not sure I have a good answer to what's better and I'm not sure I share the confidence that you have that self supervised learning can take us far. I think I'm more and more convinced that it's a very important component. But I still feel like we need to understand what makes like this, this like dream of, maybe what it's called, like symbolic AI of, of arriving. Like once you have this common sense base, be able to play with these concepts and build graphs or hierarchies of concepts on top in order to then form a deep sense of this three dimensional world or four dimensional world and be able to reason and then project that onto a 2D plane in order to interpret a 2D image. Can I ask you just an out there question? I remember, I think Andre Karpathy had a blog post about computer vision, like being really hard. I forgot what the title was, but it's many, many years ago and they had I think President Obama stepping on a scale and there was humor and there was a bunch of people laughing and whatever. And the interesting, there's a lot of interesting things about that image and I think Andre highlighted a bunch of things about the image that us humans are able to immediately understand. Like the idea, I think of gravity and that you can, you have the concept of a weight, you have a, you immediately project. Because of our knowledge of pose and how human bodies are constructed, you understand how the forces are being applied with the human body. The really interesting other thing that you're able to understand is multiple people looking at each other in the image. You're able to have a mental model of what the people are thinking about. You're able to infer like, oh, this person is probably thinks like is laughing at how humorous the situation is and, and this person is confused about what the situation is because they're looking this way. We're able to infer all of that. So that's human vision. How Difficult is computer vision, like, in order to achieve that level of understanding and maybe how big of a part does self supervised learning play in that, do you think? And do you still, you know, back. That was like over a decade ago. I think Andrej and I think a lot of people agreed computer vision is really hard. Do you still think computer vision is really hard?

I think it is, yes. And getting to that kind of understanding, I mean, it's really out there. So if you ask me to solve just that particular problem, I can do it. The supervised learning route. I can always construct a data set and basically predict, oh, is there humor in this or not? And of course I can do it.

Actually, that's a good question. Do you think you can. Okay, okay. Do you think, do you think you can do human supervised annotation of humor?

To some extent, yes, I'm sure it'll work. I mean, it won't be as bad as like randomly guessing. I'm sure it can still predict whether it's humorous or not in some way.

Yeah, maybe like Reddit upvotes is the signal. I don't know.

I mean, it won't do a great job, but it'll do something. It may actually be like, it may find certain things which are not humorous. Humorous as well, which is going to be bad for us. But I mean, it'll do a pretty. It won't be random.

Yeah, kind of like my sense of humor. Okay, so fine, so you can. That particular problem. Yes. But the general problem you're saying, is hard.

The general problem is hard. And self supervised learning is not the answer to everything. Of course it's not. I think if you have machines that are going to communicate with humans, at the end of it, you want to understand what the algorithm is doing. Right. You want it to be able to produce an output that you can decipher that you can understand, or it's actually useful for something else, which again is a human. At some point in this sort of entire loop, a human steps in and now this human needs to understand what's going on. And at that point this entire notion of language or semantics really comes in. If the machine just spits out something and if we can't understand it, then it's not really that useful for us. So self supervised learning is probably going to be useful for a lot of the things before that part. Before the machine really needs to communicate a particular kind of output with a human, because, I mean, otherwise how is it going to do that without language.

Or some kind of communication? But you're saying that it's possible to build a big base of understanding or whatever of what's better concepts like common sense concepts. Supervised learning in the context of computer vision is something you focused on, but that's a really hard domain and it's kind of the cutting edge of what we're as a community working on today. Can we take a little bit of a step back and look at language? Can you summarize the history of success of self supervised learning in natural language processing, language modeling? What are transformers? What is the masking, the sentence completion that you mentioned before? How does it lead us to understand anything? Semantic meaning of words, syntactic role of words and sentences?

So I'm of course not the expert in nlp. I kind of follow it a little bit from the sides. So the main sort of reason why all of this masking stuff works is I think it's called the distributional hypothesis in nlp. The idea basically being that words that occur in the same context should have similar meaning. So if you have the blank jumped over the blank, it basically whatever is like in the first blank is basically an object that can actually jump is going to be something that can jump. So a cat or a dog or, I don't know, sheep something, all of these things can basically be in that particular context. And now so essentially the idea is that if you have words that are in the same context and you predict them, you're going to learn lots of useful things about how words are related because you're predicting by looking at their context what the word is going to be. So in this particular case, the blank jumped over the fence. So now if it's a sheep, the sheep jumped over the fence, the dog jumped over the fence. So essentially the algorithm or the representation basically puts together these two concepts together. So it says, okay, dogs are going to be kind of related to sheep because both of them occur in the same context. Of course, now you can decide, depending on your particular application downstream, you can say that dogs are absolutely not related to sheep because, well, I really care about dog food, for example, I'm a dog food person and I really want to give this dog food to this particular animal. So depending on what your downstream application is, of course this notion of similarity or this notion or this common sense that you've learned may not be applicable. But the point is basically that this just predicting what the blanks are is going to take you really, really far.

So there's a nice feature of language that the number of words in a particular language is very large. But it's finite and it's actually not that large in the grand scheme of things. I still got it because we take it for granted. So first of all, when you say masking, you're talking about this very process of the blank removing words from a sentence and then having the knowledge of what word went there in the initial data set. That's the ground truth that you're training on. And then you're asking the neural network to predict what goes there. That, that's, that's like a little trick. It's a really powerful trick. The question is how far that takes us. And the other question is, is there other tricks? Because it, to me it's very possible. There's other very fascinating tricks. I'll give you an example. In autonomous driving, there's a bunch of tricks that give you the self supervised signal back, for example, very similar to sentences, but not really, which is you have signals from humans driving the car because a lot of us drive cars to places. And, and so you can ask the neural network to predict what's going to happen in the next two seconds for a safe navigation through the environment. And the signal comes from the fact that you also have knowledge of what happened in the next two seconds because you have video of the data. The question, in autonomous driving, as it is in language, can we learn how to drive autonomously based on that kind of self supervision? Probably the answer is no. The question is how good can we get? And the same with language, how good can we get? And are there other tricks like we get sometimes super excited by this trick that works really well. But I wonder, it's almost like mining for gold. I wonder how many signals there are in the data that could be leveraged that are like there is that. I just want to kind of linger on that because sometimes it's easy to think that maybe this masking process is self supervised learning. No, it's only one method. So there could be many, many other methods, many tricky methods, maybe interesting ways to leverage human computation in very interesting ways that might actually border on semi supervised learning, something like that. Obviously the Internet is generated by humans at the end of the day. So all that to say is what's your sense in this particular context of language, how far can that masking process take us?

So it has stood the test of time, right? I mean, so Word two vec, the initial sort of NLP technique that was using this to now, for example, like all the Bert and all these big models that we get, Bert and Roberta, for example, all of them are still sort of based on the same principle of masking. It's taken us really far. I mean, you can actually do things like, oh, these two sentences are similar or not, whether this particular sentence follows this other sentence in terms of logic. So entailment, you can do a lot of these things with just this masking trick. So I'm not sure if I can predict how far it can take us because when it first came out, when word 2 vec was out, I don't think a lot of us would have imagined that this would actually help us do some kind of entailment problems. And really that will. And so just the fact that by just scaling up the amount of data that we're training on and using better and more powerful neural network architectures has taken us from that to this is just showing you how maybe poor predictors we are as humans, how poor we are at predicting how successful a particular technique is going to be. So I think I can say something now, but 10 years from now I look completely stupid.

Basically predicting this in the language domain, is there something in your work that you find useful and insightful and transferable to computer vision, but also just, I don't know, beautiful and profound that I think carries through to the vision domain.

I mean, the idea of masking has been very powerful. It has been used in vision as well for predicting, like you say the next sort of. If you have n sort of frames and you predict what's going to happen in the next frame. So that's been very powerful in terms of modeling, like in just terms, in terms of architecture. I think you would have asked about Transformers a while back. That has really become like, it has become super exciting for computer vision now. Like in the past, I would say year and a half, it's become really powerful.

What's a transformer?

Right? I mean the core part of a transformer is something called the self attention model. So it came out of Google. And the idea basically is that if you have N elements, what you're creating is a way for all of these N elements to talk to each other. So the idea basically is that you are paying attention, each element is paying attention to each of the other element. And basically by doing this, it's really trying to figure out you're basically getting a much better view of the data. So for example, if you have a sentence of like four words, the point is if you get a representation or a feature for this entire sentence, it's constructed in a way such that each word has paid attention to everything else. Now, the reason it's like different from say what you would do in a convnet is basically that in the convnet you would only pay attention to a local window. So. So each word would only pay attention to its next neighbor or like one neighbor after that. And the same thing goes for images. In images, you would basically pay attention to pixels in a 3x3 or a 7x7 neighborhood, and that's it. Whereas with the transformer, that self attention, mainly the sort of idea is that each element needs to pay attention to each other element.

And when you say attention, maybe another way to phrase that is you're considering a context, wide context in terms of the wide context of the sentence in understanding the meaning of a particular word. And in computer vision, that's understanding a larger context to understand the local pattern of a particular local part of an image.

Right. So basically, if you have, say again, a banana in the image, you're looking at the full image first. So whether it's like, you know, you're looking at all the pixels that are off a kitchen of a dining table and so on, and then you're basically looking at the banana also.

Yeah. By the way, in terms of if we were to train the funny classifier, there's something funny about the word banana. Just wanted to anticipate I am wearing a banana shirt.

So.

Yeah, is there bananas on it? Okay, so masking has worked for the vision context as well, and so this.

Transformer idea has worked as well. So basically looking at all the elements to understand a particular element has been really powerful in vision. The reason is like a lot of things, when you're looking at them in isolation. So if you look at just a blob of pixels. So Antonio Torralba at MIT used to have this really famous image, which I looked at when I was a PhD student, where he would basically have a blob of pixels and he would ask you, hey, what is this? And it looked basically like a shoe, or like it could look like a TV remote. It could look like anything. And it turns out it was a beer bottle. But I'm not sure it was one of these three things. But basically he showed you the full picture and then it was very obvious what it was. But the point is, just by looking at that particular local window, you couldn't figure out, because of resolution, because of other things. It's just not easy always to just figure out by looking at just the neighborhood of pixels, what these pixels are. Yeah, and the same thing happens for.

Language as well, for the Parameters that have to learn something about the data. You need to give it the capacity to learn the essential things. Like if it's not actually able to receive the signal at all, then it's not going to be able to learn that signal. And in order to understand images, to understand language, you have to be able to see words in their full context. Okay, what is harder to solve? Vision or language? Visual intelligence or linguistic intelligence?

So I'm going to say computer vision is harder. My reason for this is basically that language of course has a big structure to it because we developed it. Whereas vision is something that is common in a lot of animals. Everyone is able to get by. A lot of these animals on Earth are actually able to get by without language. And a lot of these animals we also deem to be intelligent. So clearly intelligence does have like a visual component to it. And yes, of course in the case of humans, it of course also has a linguistic component, but it means that there is something far more fundamental about vision than there is about language. And I'm sorry to anyone who disagrees, but yes, this is what I feel.

So that's being a little bit reflected in the challenges that have to do with the. The progress of self supervised learning, would you say? Or is that just the peculiar accidents of the progress of the AI community that we focused on Link or we discovered self attention and transformers in the context of language first.

So like the self supervised learning success was actually for vision has not much to do with the transformers part. I would say it's actually been independent a little bit. I think it's just that the signal was a little bit different for vision than there was for like NLP and probably NLP folks discovered it before. So for vision the main success has basically been this crops so far, like taking different crops of images. Whereas for NLP it was this masking thing.

But also the level of success is still much higher for language.

It has. So that has a lot to do with, I mean I can get into a lot of details for this particular question. Let's go for it. Okay, so the first thing is language is very structured. So you are going to produce a distribution over a finite vocabulary. English has a finite number of words, it's actually not that large. And you need to produce basically when you're doing this masking thing, all you need to do is basically tell me which one of these like 50,000 words it is. Yeah, that's it. Now for vision, let's imagine doing the same thing. Okay. We are basically going to blank out a particular part of the image. And we ask the network or this neural network to predict what is present in this missing patch. It's combinatorially large, right? You have 256 pixel values. If you're even producing basically a 7 cross 7 or a 14 cross 14 window of pixels at each of these 169 or each of these 49 locations, you have 256 values to predict. And so it's really, really large. And very quickly, the kind of prediction problems that we are setting up are going to be extremely intractable for us. And so the thing is, for nlp, it has been really successful because we are very good at predicting, doing this distribution over a finite set. And the problem is, when this set set becomes really large, we are going to become really, really bad at making these predictions and at solving basically this particular set of problems. So if you were to do it exactly in the same way as nlp, for vision, there is very limited success. The way stuff is working right now is actually not by predicting these masks, it's basically by saying that you take these two crops from the image, you get a feature representation from it, and just saying that these two features this. So they're like vectors, just saying that the distance between these vectors should be small. And so it's a very different way of learning from the visual signal than there is from nlp. Okay, the other reason is the distributional hypothesis that we talked about for nlp, right? So a word given its context, basically the context actually supplies a lot of meaning to the word. Now, because there are just finite number of words and there is a finite way in which we compose them. Of course, the same thing holds for pixels, but in language there's a lot of structure. So I always say, whatever. The dash jumped over the fence, for example, there are lots of these sentences that you'll get. And from this you can actually look at this particular sentence might occur in a lot of different contexts as well. This exact same sentence might occur in a different context. So the sheep jumped over the fence, the cat jumped over the fence, the dog jumped over the fence. So you immediately get a lot of these words which are, because this particular token itself has so much meaning, you get a lot of these tokens or these words which are actually going to have sort of this related meaning across, given this context. Whereas for vision, it's much harder because just by pure the way we capture images, lighting can be different. There might be different noise in the sensor. So the thing is, you're capturing a physical phenomenon. And then you're basically going through a very complicated pipeline of image processing and then you're translating that into some kind of like digital signal. Whereas with language you write it down and you transfer it to a digital signal, almost like it's a lossless like transfer. And each of these tokens are very, very well defined.

There could be a little bit of an argument there because language as written down is a projection of thought. This is one of the open questions is if you perfectly can solve language, are you getting close to being able to solve easily with flying colors, pass the Turing test kind of thing? So it's similar but different than the computer vision problem is in the 2D plane is a projection of a three dimensional world. So perhaps there are similar problems there. Maybe this.

I think what I'm saying is NLP is not easy. Of course, don't get me wrong, abstract thought expressed in knowledge, or knowledge basically expressed in language is really hard to understand. Right? I mean we've been communicating with language for so long and it is of course a very complicated concept. The thing is at least getting somewhat reasonable, being able to solve some kind of reasonable tasks with language, I would say slightly easier than it is with computer vision.

Yeah, I would say, yeah. So that's well put. I would say getting impressive performance on language is easier. I feel like for both language and computer vision, there's going to be this wall of like this hump you have to overcome to achieve superhuman level performance or human level performance. And I feel like for language that wall is farther away. So you can get pretty nice, you can do a lot of tricks, you can show really impressive performance. You can even fool people that you're tweeting or your blog post writing or your question answering has intelligence behind it. But to truly demonstrate understanding of dialogue, of continuous long form dialogue, that would require perhaps big breakthroughs in the same way in computer vision. I think the big breakthroughs need to happen earlier to, to achieve impressive performance. This might be a good place to. You already mentioned it, but what is contrastive learning and what are energy based models?

Contrastive learning is sort of a paradigm of learning where the idea is that you are learning this embedding space or so you're learning this sort of vector space of all your concepts. And the way you learn that is basically by contrasting. So the idea is that you have a sample, you have another sample that's related to it. So, so that's called the positive and you have another sample that's not related to it. So that's Negative. So for example, let's just take an NLP or in a simple example in computer vision, so you have an image of a cat, you have an image of a dog. And for whatever application that you're doing, say you're trying to figure out what pets are, you're saying that these two images are related. So image of a cat and dog are related, but now you have another third image of a banana, because you don't like that word. So, so now you basically have this banana.

Thank you for speaking to the crowd.

And so you take both of these images and you take the image from the cat, the image from the dog, you get a feature from both of them. And now what you're training the network to do is basically pull both of these features together while pushing them away from the feature of a banana. So this is the contrastive part. So you're contrasting against the banana. So there's always this notion of a negative and a positive. Now, energy based models are like one way that Yann sort of explains a lot of these methods. So Yann basically, I think a couple of years or more than that, when I joined Facebook, Jan used to keep mentioning this word, energy based models. And of course I had no idea what he was talking about. So then one day I caught him in one of the conference rooms and I'm like, can you please tell me what this is? So then very patiently, he sat down with a marker and a whiteboard. And his idea basically is that rather than talking about probability distributions, you can talk about energies of models. So models are trying to minimize certain energies in certain space, or they're trying to maximize a certain kind of energy. And the idea basically is that you can explain a lot of the contrastive models, GANs, for example, which are like generative adversarial networks, a lot of these modern Learning methods, or VAEs, which are variational autoencoders, you can really explain them very nicely in terms of an energy function that they're trying to minimize or maximize. And so by putting this common sort of language for all of these models, what looks very different in machine learning, that, oh, VAEs are very different from what Gans are, very different from what contrastive models are. You actually get a sense of like, oh, these are actually very, very related. It's just that the way or the mechanism in which they're sort of maximizing or minimizing this energy function is slightly different.

It's revealing the commonalities between all these approaches and putting a sexy word on top of it, like Energy. And so similarities, two things that are similar have low energy. Like the low energy signifying similarity, right?

Exactly. So basically the idea is that if you were to imagine like these embedding as a manifold, a 2D manifold, you would get a hill or like a high sort of peak in the energy manifold wherever two things are not related. And basically you would have like a dip where two things are related. So you'd get a dip in the.

Manual and in the self supervised context, how do you know two things are related and two things are not related?

Right, so this is where all the sort of ingenuity or tricks comes in. Right. So for example, you can take the fill in the blank problem or you can take in the context problem and what you can say is two words that are in the same context are related. Two words that are in different contexts are not related. For images, basically two crops from the same image are related. And whereas a third image is not related at all for a video, it can be two frames from that video related because they're likely to contain the same sort of concepts in them, whereas a third frame from a different video is not related. So it basically is, it's a very general term. Contrastive learning has nothing really to do with self supervised learning. It actually is very popular in for example any kind of metric learning or any kind of embedding learning. So it's also used in supervised learning. And the thing is, because we are not really using labels to get these positive or negative pairs, it can basically also be used for self supervised learning.

So you mentioned one of the ideas in the vision context that works is to have different crops. So you could think of that as a way to sort of manipulating the data to generate examples that are similar. Obviously there's a bunch of other techniques. You mentioned lighting as a very, you know, in images, lighting is something that varies a lot and you can artificially change those kinds of things. There's the whole broad field of data augmentation which manipulates images in order to increase arbitrarily the size of the data set. First of all, what is data augmentation? And second of all, what's the role of data augmentation in self supervised learning and contrastive learning?

So data augmentation is just a way, like you said, it's basically a way to augment the data. So you have say, n samples. And what you do is you basically define some kind of transforms for the sample. So you take your say, image and then you define a transform where you can just increase, say the Colors like colors or the brightness of the image, or increase or decrease the contrast of the image, for example, or take different crops of it. So data augmentation is just a process to basically perturb the data or augment the data. And so it has played a fundamental role for computer vision, for self supervised learning. Especially the way most of the current methods work, contrastive or otherwise, is by taking an image. In the case of images, is by taking an image and then computing basically two perturbations of it. So these can be two different crops of the image with different types of lighting or different contrast or different colors. So you jitter the colors a little bit and so on. And now the idea is basically because it's the same object or because it's like related concepts, in both of these perturbations, you want the features from both of these perturbations to be similar. So now you can use a variety of different ways to enforce this constraint. Like these features being similar, you can do this by contrastive learning. So basically both of these things are positives. A third sort of image is negative. You can do this basically by like clustering, for example. You can say that both of these images should, the features from both of these images should belong in the same cluster because they're related, whereas image like another image should belong to a different cluster. So there's a variety of different ways to basically enforce this particular constraint.

By the way, when you say features it means there's a very large neural network that extracting patterns from the image. And the kind of patterns that extracts should be either identical or very similar. That's what that means.

So the neural network basically takes in the image and then outputs a set of basically a vector of numbers. And that's the feature. And you want this feature for both of these different crops that you computed to be similar. So you, so you want this vector to be identical in its entries.

For example, be like literally close in this multidimensional space to each other. And like you said, close can mean part of the same cluster or something like that in this large space. First of all, I wonder if there is connection to the way humans learn to this. Almost like maybe subconsciously in order to understand a thing, you kind of have to see it from two, three, multiple angles. I wonder, I have a lot of friends who are neuroscientists maybe and cognitive scientists. I wonder if that's in there somewhere. Like in order for us to place a concept in its proper place, we have to basically crop it in all kinds of ways do basic data augmentation on it in whatever very clever ways that the brain likes to do.

Right?

Like spinning around in our mind somehow. That is very effective.

So I think for some of them we need to do it. So like babies, for example, pick up objects like move them, put them close to their eye and whatnot. But for certain other things actually we are good at imagining it as well. Right. So if you. I have never seen for example, an elephant from the top. I've never basically looked at it from like top down. But if you showed me a picture of it, I could very well tell you that that's an elephant. So I think some of it will just like we naturally build it or transfer it from other objects that we've seen to imagine what it's going to look like.

Has anyone done that with the augmentation? Imagine all the possible things that are occluded or not there, but not just like normal things, like wild things, but that are nevertheless physically consistent.

So I mean people do kind of like occlusion based augmentation as well. So you place in like a random like box, gray box to sort of mask out a certain part of the image. And the thing is basically you're kind of occluding it. For example, you'll place it say on half of a person's face. So basically saying that, you know, something below their nose is occluded because it's grayed out. So.

No, I meant like you have like, what is it? A table and you can't see behind the table and you imagine there's a bunch of elves with bananas behind the table. Like I wonder if there's useful to have a wild imagination for the network because that's possible. Well, maybe not elves but like puppies and kittens or something like that. Just have a wild imagination and like constantly be generating that wild imagination. Because in terms of data augmentation that's currently applied, it's super ultra very boring. It's very basic data augmentation. I wonder if there's a benefit to being wildly imaginable while trying to be consistent with physical reality.

I think it's a kind of a chicken and egg problem. Right? Because to have amazing data augmentation you need to understand what the scene is and what we're trying to do data augmentation to learn what a scene is. Anyway, so it's basically it just keeps.

Going on before you understand it. Just put elves with bananas until you know it's not to be true. Just like children have A wild imagination until the adults ruin it all. Okay, so what are the different kinds of data augmentation that you've seen to be effective in visual intelligence?

For like vision? It's a lot of these image filtering operations. So like blurring the image, you know, all the kind of Instagram filters that you can think of. So like arbitrarily, like make the red super red, make the green super greens, like saturate the image.

Rotation cropping.

Rotation cropping, Exactly. All of these kind of things.

Lighting is a really interesting one to me. That feels like really complicated to do.

So I mean they don't. The augmentations that we work on aren't like that involved. So they're not going to be like physically realistic versions of lighting. It's not that you're assuming that there's a light source up and then you're moving it to the right and then what does the thing look like? It's really more about like brightness of the image, overall brightness of the image, or overall contrast of the image and so on.

But this is a really important point to me. I always thought that data augmentation holds an important key to big improvements in machine learning. And it seems that it is an important aspect of self supervised learning. So I wonder if there's big improvements to be achieved on much more intelligent kinds of data augmentation. For example, currently, maybe you can correct me if I'm wrong. Data augmentation is not parameterized. You're not learning, you're not learning. To me it seems like data augmentation potentially should involve more learning than the learning process itself.

Right.

You're almost like thinking of like generative kind of, it's the elves with bananas. You're trying to. It's like very active imagination of messing with the world and teaching that mechanism for messing with the world to be realistic. Because that feels like, I mean it's imagination, just as you said. It feels like us humans are able to maybe sometimes subconsciously imagine before we see the thing, imagine what we're expecting to see. Like maybe several options and especially we probably forgot, but when we were younger, probably the possibilities were wilder, more numerous. And then as we get older, we become to understand the world and the possibilities of what we might see becomes less and less and less. So I wonder if you think there's a lot of breakthroughs yet to be had in data augmentation and maybe also can you just comment on the stuff we have? Is that a big part of self supervised learning?

Yes. So data augmentation is key to self supervised learning that has the kind of augmentation that we're using and basically the fact that we are trying to learn these neural networks that are predicting these features from images that are robust under data augmentation has been the key for visual self supervised learning and they play a fairly fundamental role to it. Now the irony of all of this is that for deep learning purists will say the entire point of deep learning is that you feed in the pixels to the network, neural network and it should figure out the patterns on its own. So if it really wants to look at edges, it should look at edges. You shouldn't really go and handcraft these features, right? You shouldn't go tell it that look at edges. So data augmentation should basically be in the same category, right? Why should we tell the network or tell this entire learning paradigm what kinds of data augmentation that we are looking for? We are encoding a very sort of human specific bias there that we know things are like if you change the contrast.