Connecting Images with Natural Language

Okay, so, uh, thank you. So yeah, my name is Andrej Karpathy. Until recently, I was a PhD student at Stanford working with Fei-Fei Li, and I'm currently transitioning to a new position as a research scientist at OpenAI. So in this talk, I'll try to get you excited about problems in the rapidly growing area of connecting images and natural language. So what do I mean by this exactly? Well, consider this— consider first the domain of all visual information that contains all the possible images, all the possible regions, really any array of pixels. And then consider the domain of all possible natural language utterances. When I talk about connecting images and natural language, I'm really talking about finding mappings between these two domains. That is, given any, any array of pixels, we would like to find natural language utterances that describe that array of pixels, or conversely, given any utterance, we would like to either retrieve visual information that depicts the described concept, or if we're feeling ambitious, maybe we want to also try to generate images images de novo based on the description. So just to make this more concrete, consider an example. Humans are very good at this mapping. So when you and I look at this image, it feels very effortless to describe all of its aspects. So say there's a man sitting on the red bouncing ball, or there's a person sitting on the chair and eating a sandwich, or we see that there's a black bag on the floor, or that there's a door handle, or something like that. In this talk, I'll try to argue that we should strive to endow computers with the same ability. And if you're like Jackie Chan here, you might be wondering, why is this a good idea? So in fact, I believe that there's an entire pile of reasons for why we would want to do this, and for the purposes of this talk, I'll cherry-pick a few that I find particularly compelling. So as a first reason, I would like to argue that the ambition to connect these two modalities actually goes back all the way to the beginning when the field of artificial intelligence was born. Arguably, this was around 1950 when Alan Turing published this document where he ponders whether a machine can think. And this is also the source of the famous Turing test. Now, in this document— this document is very interesting, but there's a paragraph in the conclusion that I find particularly interesting. And there's a sentence there that— in this paragraph, basically, Turing is pondering how we can achieve such a machine that can think. And as one of the possible approaches towards this problem, he suggests that we should build a machine and that we should endow it with sense organs so that it can perceive the visual world around us. And that we teach it to understand and speak English. So presumably we can talk to it in natural language, we can point out things in the visual world and teach machines similar to maybe how we teach children what the world looks like and what it's made up of and so on. And these kinds of sentiments of seeing the future of AI as machines that we can talk to in natural language is echoed many times throughout artificial intelligence research. Maybe another pointer that I really like is from the 1970s. This is work of Terry Winograd at MIT, Andrej Karpathy, who worked on a program called Trudlu. I encourage all of you to look at this, at his PhD thesis, because it's really inspiring. Here we have a very capable machine intelligence, but in a very narrow domain of blocks world. And so this is a blocks world. You can manipulate blocks around, you can move these pyramids and blocks, and you talk to this machine through this dialogue system where you can give it commands to execute in this world. You can ask it questions about this world. It manipulates this world. It can introspect and explain why it did certain things. You can also teach it about new concepts, so you can define new concepts. You can tell that a steeple is made up of two blocks and a triangle, and then it will understand that, and so on. So you can really have very sophisticated dialogue with this machine through natural language interface, and it's all grounded in this visual— in this particular physical instantiation of the world. So in that sense, I would argue that connecting images and language is really a stepping stone towards AI as we had imagined it for many decades. Second, due to the recent availability of data, due to the huge amount of compute that has recently become available, due to algorithms that we've developed, especially convolutional neural networks, and the huge amount of infrastructure that we built for ourselves, we have managed to push visual recognition performance down to quite low levels. So here we're showing ImageNet top 5 error over time, and we see that we're doing really well. In fact, we're doing, uh, so well that we're basically in the vicinity of human performance. I know something about this comparison. And so we're roughly at human-level performance for recognizing these categories in images, and in some cases even surpassing human performance on some of the fine-grained categories like species of dogs and so on. And so we've really done extremely well over the last few years. But I would like to argue that if you are considering visual recognition as a, say, question on a test, then really most of what we're doing so far is either asking true or false questions when we train binary classifiers, or asking multiple choice questions when we train a softmax classifier to categorize images into categories. And so I'd like to suggest that the next natural step in this evolution is to convert visual recognition into a short-answer question, just describe what you see. Okay? So in that sense, I would like to argue that connecting these two modalities is a yet unsolved problem and that it's a critical next frontier in computer vision as we try to push ourselves into new domains and become more ambitious about what we can achieve in visual recognition. Third, I would like to argue that natural language is actually quite a beautiful label space to work with. First, it's extremely rich. You can express a lot of things in natural language. You can talk about objects, about their attributes, about actions, about relationships. Not only do you have a lot of primitives available to you in the form of words, but you can compose them arbitrarily in these recursive structures. It's a beautiful, rich label space. Second, people are already fluent in natural language. And so if we were to build our models, um, such that they can consume and operate over this API of natural language, then there would be no need for arbitrary translation pipelines that go back and forth between languages. So for example, if I have a query that I'd like to search in my photo collection for something specific, then every one of us is already fluent in natural language. It would be beautiful if we could just consume that instead of having to process it into some number of classifiers that we have to run over some images to get the answer. So instead, we'd like to just treat this as a first-class citizen and really build models that can consume and work with this input directly, or treat it as an output and produce these kinds of descriptions directly and interact in very natural ways with humans. Third, natural language is pervasive. You and I, we all talk in natural language, and so we've actually encoded a huge amount of knowledge about the world inside this encoding, and we put all of it up on the internet. So we have Wikipedia, books, and really the internet is a huge repository, a treasure trove of knowledge encoded in natural language. And so if our models were capable of dealing with this domain as, again, a first-class citizen, then a lot of interesting applications would unlock themselves. And we would really be connecting these two huge domains of knowledge. One domain of knowledge, which is the physical world of when I push things, how do they move? I can observe that in vision. And really the second huge domain of knowledge, which is more semantic, that we encode in natural language. And so we can map between them. As a concrete example of this, perhaps, consider reading a description of a camel in Wikipedia, and if you read that description, again stored in internet, you can figure out that a camel is some kind of an animal, a horse-like creature with a hump on its back. And if you're good at understanding the relationship between the visual world and the domain of language, then you should be able to figure out that that image might be a camel. So in that sense, I would like to argue that natural language is a beautiful label space to use in our models. Now, the next part of my talk is organized along the selection of 3 papers that I worked on during my PhD. That built up towards this goal of connecting natural images and natural language. So first, I'll tell you about matching images and sentences, then about generating sentences from images, and then about dense captioning. This is a paper that we've introduced in this CVPR 2016, where we're not just generating a single sentence, but we're both detecting and describing everything in a complex visual scene. And so I'll tell you about these three projects briefly next. So first, Matching images and sentences. Here's a concrete problem statement. We have a finite collection of sentences and a finite collection of images, and then what we would like to do is we would like to select a sentence and rank images based on how compatible they are with that sentence. Or conversely, we would like to be able to pick any image and rank the sentences based on how compatible they are with that image. And the way we're going to hook up— kind of design this architecture is that we're going to think about taking a pair of an image and a sentence and computing a score of how well that image matches that sentence. And so we have to figure out kind of 3 basic blocks here: how do we process the images, how do we process sentences, and how do we compare objects across these 2 modalities? So to process the images, um, oh, and just a brief mention that since we are in a deep learning workshop, of course the way we approach this problem is we think about end-to-end learning, and we're trying to design a single differentiable function that maps an image and a sentence to a single number telling you how compatible they are. So to process the image, of course, we're going to use a convolutional neural network which looks at the image and gives us a vector V describing the appearance of that image on a high level. Now, to process the sentence is a slightly bit more tricky because sentences are variably length, unlike images. So sentences can have some different number of words in them, and we'd like to encode these sentences into a fixed vector S similar to what we did with images so that we can later easily compare objects across these two modalities. So the problem is, how do we encode variably length sentences into fixed size vectors that summarize their content? So you can think about several simple approaches. For example, in the bag-of-words approach, we would assign every single word some kind of a vector that we're going to learn, a word vector. And then to encode the entire sentence, we could, for example, do a sum or an average across the sentence. And that's not ideal, of course, because we're losing some of the long-term relationships in the sentence. You could extend this and use a bag-of— use n-grams so we can encode two words at a time. But again, you're losing some of the long-term relationships in this sentence. So in this particular paper from 2013 where I work— that I worked on with Richard Socher, we developed a recursive tensor neural network that processes a sentence encoding in a hierarchical fashion based on a deep— based on a dependency tree parse of that sentence. So we start off at the leaves and we have word vectors for all of our nodes. We use dependency tree parse to extract the tree. And then we have a recursive merging function that has some parameters w that recursively merges the representations until at the very end you have a root. At the root you have a single vector that is a function of all the words and also all the relationships of words in that sentence. So this is one of only many possible approaches of encoding sentences. In practice, you'll see, for example, recurrent neural networks could just go linearly over the line— over the sentence and just go chunk chunk chunk. And at the end you could take the hidden state vector of the RNN and that's your sentence description. Or you could average all the hidden states of the RNN and that could be your sentence description. There are many ways of encoding a sentence in a, in a single vector. So at this point, we have a vector V summarizing the image and a vector S summarizing the sentence. And now we'd like to compare these two objects across these two modalities. The simplest way that also works well is to just simply take an inner product. And so the way to interpret this is, or at least one of the ways, is that we're embedding both images and sentences into a common multimodal embedding space. And we're thinking about, um, arranging this space so that images and sentences that correspond point in similar directions. One concrete way of seeing how this might work is that, for example, one of the dimensions of V could correspond to whether or not there are fur textures present in the image, and the corresponding, um, dimension of S could correspond to whether or not dog or some other fur-like creature was mentioned somewhere in the sentence. And so therefore, if both of those dimensions are either both positive or both negative, then they will interact positively in the inner product and increase the score of your match. And, uh, conversely, if only one of them is present, then you will get a negative interaction in a— in product, and you'll decrease the score. Now, of course, during training, we don't just consider a single image-sentence pair. We'll actually consider an entire batch of images, an entire batch of sentences, and we're going to pairwise evaluate our network on every one of these. And so we end up with this 2D matrix of scores between images and sentences. And of course, we'd like the numbers along the diagonal here in green to be high because these are image-sentence pairs that correspond, and the numbers off-diagonal to be low because they are ones that do not. And so we can encode this into a loss function, and I'm not going to go into full details due to time constraints, but we can encode this intuition in the loss function. And once we optimize this function, uh, only the diagonal numbers will be high effectively around the training set. So we train this end-to-end, uh, on image-sentence datasets Just to give you an idea about image sentence datasets, these are some of the ones that people use. Especially MS-COCO has become very popular recently. So in MS-COCO, you have on the order of 100,000 images, and every image is described with 5 sentences, and these sentences are written by Amazon Mechanical Turk workers. So notice, by the way, that Microsoft COCO has about 100,000. ImageNet has 1 million images. So we're data-starved when we work with images and sentences right now, unfortunately, and we need more data, I believe. To actually get nice results in these domains. Here are some examples from this particular model that I presented. I'm not going to dive too deep into them. Here we're taking an image, for example, a test image, and we're looking at some finite set of sentences and testing their score and sorting by it. And I'm not going to go into details here because this is actually 2013, so this is pre-AlexNet computer vision, where here we use, I think, the autoencoder from Quocly that they trained at Google at the time. But presumably now with state-of-the-art residual networks, this would work much more— much better. I'd like to briefly point out that I worked on top of this framework with some other— with some more recent work from last year. So in particular, this is a project from CVPR 2015 where we are still in the same setting, but the way we compute image-sentence score match is slightly different. We're not forcing every image to be a single vector and every sentence to be a single vector. Instead, we're encoding both of these objects as a set of vectors, and I refer to these, um, as fragments. So we break down the images into fragments, in this case object detections. We break out sentences into fragments, in this case words. But these words are all enriched by context, and that context is mediated by these recurrent connections in a bidirectional recurrent neural network. And so we end up embedding fragments instead of entire images and sentences, and then we end up comparing fragments instead. And we compute the image-sentence score as a function of these fragment scores. What's nice about this is that this model works much better. It also makes much more sense, and it also gives us interpretability. And of course, I jumped over many details of this architecture, but I just want you to get the high-level idea. So if this model says that this image is compatible with that sentence, what's nice about this framework is that we can actually see— we can inspect the individual fragments and see which ones aligned or pointed in the same directions. So we can see that the model thought that this was a good match because that's the dog and that's the fence, and that's the white ball, and so on. Here's another example. Here we see that the model thought that that is a girl standing. That's very likely to be a table. That's a toothpaste or a LEGO vehicle. And we can also see that the toothbrush had a very low score. So unfortunately, the model was not able to find the toothbrush in that image. So I talked about ranking so far. Ranking is very nice and interesting image sentence kind of setting to work with. Unfortunately, there are many limitations. Still. So for example, if you'd like to describe images with sentences, it feels kind of limiting that you are constrained to a finite collection of sentences that you have to check one by one for compatibility. This is also inefficient because we have to evaluate every single, single sentence candidate one by one. And it's unsatisfying because when you and I describe images, it feels as though we're kind of just generating these descriptions. It doesn't feel like we're internally matching across all the sentences we've ever heard in the past and checking their compatibility or something like that. So in the second project, I'll talk to you very briefly about directly generating these sentence descriptions for images. This is image captioning, and I assume many of you might be also familiar with this. So I'm going to skip through this relatively— well, I'm not going to skip through it, but I'll work through it rather quickly, I suppose. So in this case, we're trying to go from an image to a sentence, and we'd like to generate this description. Now, several pioneering approaches have been developed in this case before last year. But a lot of these approaches rely on fixed templates, such as this one, where you would do object detection or attribute classification, and you would do a fill-in-the-blanks kind of task. So they were very explicit methods. And so as of last year, we looked at this problem, and we were thinking much more in terms of end-to-end learning and how can we define a differentiable function from the image, the raw input, to the sentence, the raw output. And so of course we're going to use a convolutional neural network to process the image, but the latter part was not as obvious. In the case of image classification, you just have a finite collection of categories and you can just have weights for them and you predict the one that is most likely. But if you'd like to predict the sentence, it's not quite clear how you can have a single kind of differentiable architecture that can predict a variably sized output in number of words. And so we looked to— it turns out that we can use to— we can look to natural language processing for inspiration on how to tackle this problem, where language modeling is a very common task. And so really what we're trying to do here is we're trying to have a distribution over sentences. And since you can factor a joint into a product of marginals, we see that we can represent distributions over sentences by modeling the probability of a word conditioned on all the words before it. So that's a probability distribution that we'd like to model. One of the state-of-the-art approaches to modeling this distribution is to use a recurrent neural network. So, um, in a recurrent neural network, we'd be putting— we'd be feeding in these words and we'd be predicting vectors at every single timestep that give us the distribution over the next word in the sequence. So now instead of having this simple setup where we predict a single category, we instead have a language model from which we can sample sentences. And now the only problem is that we have to condition it on the image. And there are many ways that you can actually condition this. For example, one of the simplest ones is that the hidden state of the RNN is usually initialized with zeros— with a zero vector. Instead, we can initialize it with the vector v that comes out from the ConvNet. And then we've conditioned the generative process of this recurrent neural network that predicts next word in a sequence based on that image. We treat— we train these models on the same image-sentence datasets that I've already shown you, and we get some pretty results that I'm not going to go too deep into. I'd like to cherry-pick on one result because I think I've looked at many of these results before, and I'm kind of— I have some kind of a summary statistic of how this works. So just to give you an idea, this image is captioned as "A woman in bikini is jumping over a hurdle." So it's a very funny failure case. As you might imagine, this sentence does not occur anywhere in the training data. The model has, uh, generated it and it has never seen this sentence before. And this kind of highlights how these models work, at least according to what I've seen so far, is that these models have these kinds of templates that they've learned, like a noun and noun is verb and noun. And then they look at the detections in the image and they kind of slot them in. And so here the model has identified a woman in bikini jumping and it has identified what it thinks was a hurdle, which of course isn't that, and then it slotted these things into its template. And so it learns these few templates and it slots in these words, and it does that in a differentiable way. Now, so image captioning is fun. It's fun to look at these results. It's fun to play with. Unfortunately, it's also quite limiting in some cases, especially when you put in complex images like this one. In an effort to describe everything in this image with one sentence, the model may have to fall back on quite generic descriptions like a group of people in an office or something like that. These are my labmates, by the way, at Stanford. So in this next project, in our most recent work, we were thinking of if you have these complex scenes, instead of rela— we're going to relax this assumption of having to describe an image with a single sentence, and we're instead going to try to both detect and describe everything in a complex visual scene. We refer to this task as dense captioning, and it combines the challenges of both object detection and image captioning. So we'd like to do dense prediction so that we actually propose detections, but we'd also like to use a complex label space of natural language instead of just categories. Now, our approach again is to use end-to-end learning to address this problem. So we're thinking about how do we decide— how do we design a single differentiable function that goes from the raw input, the image, to the raw output, which in this case is a set of sequences, these descriptions that, um, and also they're grounded in regions of that image. So as this is actually a fully supervised task, so we have dataset from Visual Genome that was collected by my colleagues at Stanford, Ranjay Krishna et al., also working with Fei-Fei. And we collected this dataset we called Visual Genome where we asked people to perform the dense captioning task. So they went into images and they dragged a bounding box and they described it in natural language. And we have on the order of 100,000 images. These are subsets of MSCOCO and on the order of 5 million region descriptions. So a large dataset to to work with. So we know exactly where's— what's where and how it's described. Now, in the image captioning architecture, we went from an image to a vector v through a ConvNet that went into the recurrent neural network language model that generated a sentence for that vector. Now, intuitively, what we'd like to do is we'd like to actually caption many different regions of this image. So we have many vectors v, and we'd like to kind of— that we'd like to obtain for all the interesting parts of that image, and then we'd like to caption those. So the way we designed this architecture is building on top of work from object detection, especially Faster R-CNN. So we think about a convolutional neural network processing the image, and then we introduce a localization layer in the middle before the captioning. And this localization layer will basically work as follows: we're doing training— we're at training time— this image goes through the convolutional network, and the localization layer will perform region proposal. So at this stage, we will predict, say for example, 300 scored boxes. Every scored box is 5 numbers: the x, x1, y1, x2, y2, indicating the coordinates, and a score which is telling us the confidence of the model's prediction. So for example, at this stage we would be predicting 1,500 numbers that describe these boxes. Now we're during training time, so we actually know the correct boxes for this image according to our training data. So we know that those green boxes are what people selected as interesting in this image. So what we'll do now is we have these 300 predictions and we'll iterate over all the true boxes and we'll snap them to their closest predicted box, and those are shown now in red. And then what we'll do is every one of these red boxes that align to something in the ground truth will nudge it slightly towards its aligned ground truth because we wish the model had predicted it slightly in that direction. And that's of course encoded in the gradient signal of the loss function. And, um, in addition to that, all the red boxes that aligned to something are marked as positives and will increase their confidence and all the yellow boxes will be marked as negatives and will decrease their confidence. Now, in addition to this, every one of these green boxes was also associated with a description. So we'd like all these red boxes to also produce that description. So for that, we're going to take every one of these red regions and we'll perform a differentiable cropping operation on the feature maps of a convolutional network. And this differentiable cropping can be achieved using a spatial transformer, which was a paper presented recently also from DeepMind. So we differentiably crop the regions corresponding to those predicted boxes and we'll send off those features into captioning. And so effectively we're describing those regions of interest. So I'll give you a sense of what this works like on this image that I took of my labmates. In this case, DenseCap will look here, it will— you do a forward pass, you detect everything, you describe everything, and then I'm ordering by the confidence of those predictions and I'm going to go from highest confidence to lowest. First, the model finds that there's a black computer monitor. It loves to select computer monitors, by the way, because they are both very common in the training data and they're also very easily identifiable. So the scores are always really high for monitors. Then it finds a man with black hair. Then it says that there's a man wearing a blue shirt sitting on the chair. Then it says that right there, there's people in the— people are in the background. Then it says that there's another computer monitor on the desk. Then it says that there's a red and brown chair. I find this one quite interesting because the only way you can tell that that's a chair is by the affordance of the person sitting on top of it. Okay? Then it says that there's a silver handle on the wall. That's of course incorrect because it's not on the wall, but it's relatively close. And there's a black bag on the floor, and then there's a wall is white, and this goes on for 300 different descriptions. We also evaluated this method quantitatively compared to my work from last year, where we did image captioning on regions, but these regions came from image proposals, and every single region was independently captioned. And so basically what we end up with is we're much more— our model works much better because it's fully end-to-end, but also significantly faster. So 13 times faster and better. So that's great. I'd like to show you DenseCap because this is very fun to look at. So let me try to start this video. Okay, so this video I realize is a bit seizure-inducing, but this app is very quick so we can run it quite quickly. And this is me, I'm holding a mug. This video is going to pause in a bit, so let me try to pause this. Oh, I can't pause this. How do I pause this video? Okay, my apologies. I don't think I can. Sometimes it allows me to pause, and today it decided that there's no pause button on my screen, so We're just going to— it pauses once in a while, but I don't have explicit control over it. So here, what's interesting is that it— I have my mug in front of me, and it finds everything. It says that there's a man in a blue shirt, and it says that there's a red and white logo on a shirt. And so it thinks that my mug, which I'm holding near me, is a logo on my shirt, which I thought was really interesting. Here it says that this is a red and brown bear, which is quite close to a red panda. Silver door handle. We're going to pan around and find my labmate here. So it says that this is a man in sunglasses. It says that the door is open. Here, this is my messy desk. It finds all the cups, blue plastic containers, white keyboard, blue jeans, Basically, we have, I think, this video on the website, so you can also look at it and go through it slowly. White paper towel, hand of a person. Here I show it a phone, et cetera. So it's really fun to play with this demo. As I show in the next slide— oh, here it is. It says, "Jacek's mouth of a—" Yeah, it says "head of a man" and then it says "eyes of a woman," which I thought was hilarious. Hehe. So DenseCap is available on GitHub. This is written in Torch. All the code for training is available. The pretrained models are available, and this webcam demo that I showed you is available, so you can run this for yourself and you can really play with it and debug it. I'd like to also point out that we can do captioning and it's fun to look at, but also something very important and interesting application-wise is to do the converse task, of you give me a region— sorry, you give me a description like head of a giraffe, and I go off into a large collection of images and I find regions that, uh, this— that could be heads of giraffes in this case. So this is a retrieval setting. For example, if we go into a test set of MSCOCO and we look out for head of a giraffe, the way we process these images is we take every one of these test images, we forward it through, we get our proposals. But instead of captioning, we instead take our query, head of a giraffe, and we go over all our proposals and we test how likely each would have been to generate head of a giraffe. And so in particular, DenseCap thinks that this region would have been very likely to generate head of a giraffe, and it comes from that small region, etc. These are white tennis shoes, so these are all correct in this case. These are the top, I believe, 8 from the test set. Here it's hands holding a phone. In this case, we have 2 mistakes. We have— this is some kind of a controller, and this is a digital camera. And then we try to debug it quite a bit because, uh, so we can test for arbitrary regions, arbitrary descriptions. So we tried something like a front wheel of the bus, which we thought would be difficult, and it turns out that the model has no idea about what's a front or back or middle. So it just retrieves wheels of a bus, but it really does not understand front or back of a bus, unfortunately. Okay, so I told you about 3 projects that I worked on that built towards connecting images and language. So I told you about matching images and sentences, about generating sentences, and I told you about dense image captioning. What I'd like to do now with the remainder of my talk is try to distill some takeaways of how this area is evolving more generally. What are the kinds of interesting directions? Where are people doing— where are some interesting modeling paradigms that people are adopting for tackling these problems? So first of all, I would like to talk about evaluation innovations, or actually lack thereof. This is kind of a problem. So in image captioning, we're producing the sentence, and that's a candidate description of that image. Now, we are given also 5 reference descriptions written by humans, And what we have to do now is we have to compare our prediction to the sentences that were written by humans. It turns out that this comparison is actually highly non-trivial to do automatically. And so we have different automatic— automatic methods for this, like Meteor, SIDER, and BLEU scores, but unfortunately they don't work very well. And so one way you can see this is that you can look at the leaderboard of the COCO image captioning challenge, and you can see that lots of methods do extremely well according to our according to our automatic evaluation metrics. And humans, which you think should be really good at captioning, are actually at number 16, all the way roughly half— roughly halfway down the list. So, but when you take human descriptions, when you basically do a human evaluation where you ask people which descriptions they prefer, by a large margin they prefer descriptions written by humans. So what tells— that tells us not any of these methods that we're currently working with. So what that tells us is that our automatic comparison methods, evaluation methods are broken. We need to fix them. And humans ideally should be very near the top of this list. So one possible avenue for future work here that I haven't seen anyone try, but it seems like this should be doable, is to learn this evaluation metric. There are several ways you could actually obtain supervision. For example— and the reason I say learn the evaluation metric is that all of these metrics that we're using right now, Meteor, SIDER, and BLEU, they're actually fairly simple. They look at n-gram overlaps. They don't treat, uh, things like synonyms very well. And so I think there's a lot of opportunity for training this metric, either, for example, saying that all these descriptions written for a single image are very close— they should be close compared to all the other sentences— or we can use maybe some human judgments as direct supervision. I'm not 100% sure, but it's definitely something we have to address and think about going forward. I'd like to also talk about dataset and task innovations of how— what people are doing in this space. So first, as I briefly mentioned, there's the Visual Genome dataset, which has descriptions not just for the full image but also for regions. So we have nice— so we can do things like dense captioning. So that's one dataset or a task that we can use going forward. There's also an interesting project called Visual Mad Libs. This is similar to image captioning, but it's a more structured form of image captioning. You're not getting the model to just say arbitrary things about images. You're constraining it with these templates where the model has to fill in the blanks. And what that allows is for easier evaluation, more fair evaluation perhaps. And so you're still dealing with interesting image sentence problems, but the evaluation perhaps is a bit easier. Um, people are looking at unambiguous descriptions. The idea here is if I'm given an image in a specific region I'd like to describe, uh, a specific region, then I'd like to generate the description not just for the entire image, but a description for that specific region, uh, of that image discriminatively within that image. Or conversely, if we'd like to take an, um, an image and a description, we'd like to identify what that description is referring to. So again, an interesting problem that combines images and language. People are looking, of course, at video captioning. So instead of captioning just static scenes, how do we deal with motion? And how do we express these higher-level concepts that happen also over time? People are looking at visual question answering, also partly with the motivation of addressing the evaluation problem. So some of these datasets are here. What's nicer about these is that you, again, still have images and language problem because you're given an image and a query about that image, and you try to answer it. And they have both— tasks where you answer just using language, or you have tasks so that you are given multiple choice answers that you have to pick from. And so the evaluation, again, is maybe a bit easier to do. And you still have this difficult image language aspect to it. People are looking at visual question answering also in videos, so asking questions about movies. Another interesting one in this space is going backwards. Going backwards. So the idea here is that we'd like to actually take text descriptions and we'd like to generate images that depict those descriptions. So we're not just doing retrieval, we're actually generating images. And so we have some interesting work recently building on top of the Draw generative model developed by DeepMind roughly a year ago, where we are basically conditioning this generative model on, on these descriptions and generating these small images. Right now these are fairly small and it doesn't work extremely well, but I think there's a generative models are evolving quite quickly and they're becoming much, much better very quickly. This is also building on top of a generative adversarial network, so you're given a description and you generate these images. And again, this is early, early results, but I think this is very encouraging. And it's definitely possible that within the next few years we'll be able to take arbitrary descriptions and generate indistinguishable images or even entire movies. I'd like to think that you could have entire movies or TV shows on demand just by describing what you want or something like that. Why just retrieve things and build them by hand? Okay. Now I'd like to talk about modeling innovations in terms of how people address these problems and what are the kinds of paradigms that people work within. So first we're seeing that sequence-to-sequence paradigm is quite successful. Sequence-to-sequence was first developed in the area of machine translation in Sutskever et al., where they think about machine translation as just a sequence-to-sequence problem. For example, we take a French sentence and we read it one word by one word. With an LSTM encoder, and we generate a single vector at the end of this, which is the hidden state of that LSTM encoder that summarizes that source sentence. And now we have another decoder that is trying to produce its translation in English starting from that vector. And so this pipeline, this kind of a model has been adopted in both VQA and also in image captioning. So Oriol Vinyals' paper of image captioning from last year is actually a sequence-to-sequence model. In fact, a fun story about this is that this codebase— this was done at Google. So the codebase they used for neural machine translation was just ported over to image captioning. And instead of having French to English, they swapped out the French and they piped in this new funny language of images. And really, for the purposes of that model, images are just another language that happens to always— all of its sentences are just a single word that somehow represents the image. And that works. And Oriole's model is actually one of the state-of-the-art models in image captioning. It's interesting that you're able to repurpose these models so easily to very different tasks. And we've seen these in BQA. People are playing with a lot of attention mechanisms, especially soft attention. This again comes from machine translation literature where we are— have this French sentence going to English sentence. With a soft attention mechanism, you're allowed to, while you're decoding the English sentence, so your LSTM is producing one word at a time, you're allowed to at every single timestep reference back to your source sentence, to the French sentence. Through this attentional mechanism where we can pay attention selectively to different parts of the source sentence. This allows you to not have to remember the source sentence in the— in your hidden state, and it allows you to focus on generation because you can always refer back to the source sentence. So this is quite powerful and has been used in image captioning as well. So in the image captioning, uh, this was a paper from last year where at every single timestamp where they're decoding a word of the description, they refer back to the image and they have these soft attention maps over the image while they're of what the model is paying attention to as it's saying different things about the image. So effectively, the model looks around the image as it's describing it. So we've seen a lot of visual soft attention also in VQA space. Also in video captioning, we're seeing very similar architectures, also building on both sequence-to-sequence paradigm and also on sequence-to-sequence with soft attention. There's another paradigm of hard attention that hasn't seen as much adoption yet. Part of it is because it's much more tricky to actually optimize these models. So in hard attention, instead of having a soft distribution over what you're paying attention to, say in the source sentence as you're transcribing it to English, you instead— or say in the image captioning scenario, instead of having a soft attention mask over where you look while you're describing, you actually commit yourself to specific regions. And the problem with that is that that's not a differentiable operation because you don't know what would have happened if you had paid attention somewhere else. And so you can actually optimize this using policy gradients. Effectively, what that amounts to is you do it through sampling. You think about having a policy that emits where you look next, and then effectively what happens is that you have a stochastic policy that samples, that samples where you look next, and then whatever ends up working well in terms of describing images will be encouraged later. I had a blog post on this recently where I tried to give a gentle introduction to policy gradients, so if you'd like to know more about this, perhaps check out my blog there. One other paradigm I wanted to mention very briefly is— and this one I find particularly interesting and is maybe the most recent— is this work from Jacob Andreas and colleagues at Berkeley on dynamic neural network module— dynamic neural module networks paradigm. Okay, and basically what we're doing here is— this is— I find this very interesting. It's— we're in a VQA setting and we're given basically an image and a query and we'd like to answer that query. And this query is given in natural language, and what this approach does is that it reads this query, and then it effectively compiles a program for answering that question. So there's one network that does this compilation step of coming up with a structured program that needs to execute in order to compute the answer. So it's a very interesting, interpretable kind of approach. And these execution graphs are made up of these nodes, these modules, and every one of these modules is a small neural network that we all train. So there's basically— we have this execution— we have this compilation step and an execution step, they involve neural networks and we train everything. And so in this case, as a specific example, we have the question, what is in the sheep's ear? And, uh, this model decides that it has to find a sheep, it has to find an ear, it has to apply the end operation to the attention masks received by running those two modules, and then it has to use the describe module, which is also a neural network, on the output of that attention mask that comes from the end operation of those two attention masks. So you really think about it in these terms of programming languages, and it's quite, quite fascinating. And I think this is really exciting work for going forward. And the last kind of paradigms I wanted to talk about is in the context of generating images based on textual queries. We're looking in generative models. There are basically 3 rough paradigms that people are playing with. This is in the context of variational autoencoders, in context of autoregressive models like pixelRNNs, and in context of generative adversarial networks like GANs. And so all of these are approaches that are currently being developed for generating images, and once we— once these work well, we can condition them on sentence descriptions and generate really nice results. So in summary, I'd like to— I wanted to argue in this talk that connecting images and natural language is an exciting area. We should fix evaluation. I talked briefly about the major paradigms that people use to address problems in this area. And one interesting, maybe fun takeaway is that if you want to do well in computer vision, you should pay attention to what's happening in machine translation. It's kind of interesting that we've seen this convergence really of approaches in many different areas of artificial intelligence. It used to be the case that I was not able to read speech recognition papers or neural machine translation— or machine translation papers. But over the last few years, everything has kind of become large neural networks, and now we're able to read each other's work and build on top of it. So that's quite exciting. And as a last slide, just to self-promote here very briefly, this is a webpage that I built called archivescience.com. And basically there's a huge amount of work out there on arXiv, and it's kind of hard to find, go through, and sort through. So in a lot of— when I try to kind of distill some of these paradigms and so on, I find this website very helpful. I built it originally for myself and then I shared it with others. It looks at all the arXiv papers, it pulls them in, and then it allows you to search similar papers based on nearest kind of a You can search for similar papers based on any paper that you like. You can also create an account and you can create a library of papers that you like, and arXiv Sanity will train an SVM for you. So it will recommend papers. And this is the only way that I explore arXiv recently, because I can go and I can see what are the arXiv papers over the last week that it thinks are relevant to me based on my library. And so I find this quite useful, and I hope you will as well. And so thank you, and I can take questions.