Recurrent Neural Networks & Long Short-Term Memory - Andrej Karpathy, Research Scientist, OpenAI

Andrej Karpathy. Okay, so thank you for the introduction. So I'll be talking about recurrent neural network language models, especially in the context of dense image captioning, and I'll show you what that means by the end of the talk. Now, recurrent neural networks are wonderful models to work with. Because they offer a lot of flexibility in how you can design your recurrent neural network architectures. So normally when you're working with neural networks, you're in the case on the very left where you have some kind of a fixed-sized input vector shown in red, then you process it with some hidden layers shown in green, and then you process— and then you predict a fixed-sized output vector in blue. And so an example of this setting is, say, image classification. So with recurrent neural networks, you can set up a whole other kinds of architectures. For example, you can operate over sequences in in input, output, or both. And so in the case of image captioning— oops. In the case of image captioning, we're taking a single fixed-sized image and we're producing a sequence of words that are part of the caption of that image. In a setting of sentiment classification, we are taking a sequence of words and we're trying to predict whether or not the sentiment is positive or negative. And in the case of image— in machine translation, we can take a sequence of words in English and we can produce a sequence of words in French that consists of the translation. Or we can be in the setting of video classification where we have a sequence of frames, of video frames, and we're trying to classify every one of them, but crucially we want every single frame's predictions to not be just a function of that frame, but every frame that has come before it in the video. Now, it turns out that recurrent neural networks actually have a very simple API. They basically consist of a single step function, and the step function takes a vector x and it produces a vector y. But crucially, the RNN has a state that it gets to update every time it sees a vector x x come in. And so one specific implementation of this in a vanilla recurrent neural network might look as follows. In this case, the recurrent neural network state is made up of a single hidden state vector, and the step function takes a vector x, and then it gets to update its state vector according to the recurrence formula shown in both code and in LaTeX form, and in mathematical form below. And so we get to update our hidden state based on the previous hidden state and based on the current input x that has fed into the RNN, and then we get to predict the vector y based on the hidden state h. And so these matrices here, W and— W_hh, W_xh, and W_hy— those are the parameters of a recurrent neural network. And as you change these parameters, of course, you're going to get a different behavior of the recurrent neural network. And now we'd like some specific kind of behavior from the RNN, so we feed it a lot of data and we tune these parameters over time. So for example, one way to endow semantics on top of the vectors x and y is to work in the context of character-level language models. And before I dive into the example, I'd just like to point out that in my own work on these models, I've worked on top of a lot of previous work from Alex Graves, Ilya Sutskever, Tomas Mikolov, and Yoshua Bengio. And the way this will work is in a character-level RNN model, we are going to consume a sequence of characters, and then we're trying to predict the next character in the sequence. So suppose we're working with this simple example of a string "hello" that we'd like to train on. What we're going to do is we're going to feed the characters H, E, L, and L into the RNN one step at a time. So we have 4 distinct time steps here, and we're feeding the characters into the RNN. And the RNN can only process vectors, so we have to encode them in some way, such as, for example, one-of-k encoding. And at every single time step, the RNN will update its hidden state based on the recurrence formula that I've shown you. So we have 3 numbers, 3 numbers in the hidden state in this case, and we're using that formula to compute the hidden vector at every single time step. And now the RNN is trying to predict the next character in a sequence at every single time step. Since we have 4 characters in our example vocabulary here, the RNN has to produce at every time step 4 numbers that say— that give scores or log probabilities for every single next character in a sequence. So for example, in the very first time step, we fed in the letter H, and with the current setting of weights in the RNN right now, the RNN assigns the letters H, E, L, and O probabilities, or log probabilities, 1, 2.2, -3, and 4.1 respectively. And so in particular, we know that in this training sequence, the letter H— the letter E should follow H, and so we know that the score 2.2, we'd like that to be higher, and the scores of all the other characters in red should be lower. And so at every single timestep, what this amounts to is we want the green numbers to be higher and the red numbers to be lower, That's what you encode in your gradient signal and backpropagate through the network. And the backpropagation signal then tells you the gradients on the weights, and that tells you how to adjust your weights so that the predictions are more consistent with your training data. And so what we can do now is we can take an entire set of documents and we can feed it into the recurrent neural network, and it will learn how to predict these sequences. So for example, as a specific example, we can feed in poems of William Shakespeare, and for our purposes, it's just a giant string of characters that we plug into the RNN one at a time and that we train on. And then we train the recurrent neural network on this, and what we can do then is use the RNN to generate or sample or hallucinate these kinds of sequences that it has seen in the training data one character at a time. Now when you train the RNN, what you'll see is that in the beginning we just start off with random weights, right? So the predictions that it produces are just complete gibberish in the very first box there. But then when you train a bit, the RNN starts to discover that, okay, characters are these— this input data is actually made up of words. It starts to use spaces, and It starts to use quotes and periods at the end of sentences, and so on. And as you train more and more, it discovers more and more long-term statistical structure in your data, and eventually it's producing samples of poems that are nearly indistinguishable, at least on a low-level statistical level, to poems from Shakespeare. And so you can generate entire poetry, infinite poetry from Shakespeare. So I'll just read one paragraph for example. Alas, I think he shall become approached, and the day when little strain would be attained into being never fed, and who is but a chain and subject of his death, I should not sleep. And so you can generate infinite poetry like this. Of course, we can feed in any kind of character sequence that we like, so— and we have. So for example, we found this open source book on algebraic geometry. So we just take all of LaTeX source code for this book, and we feed it one character at a time into the RNN, and then the RNN learns to generate mathematics. And so it generates LaTeX that almost compiles. We have to fix it a tiny bit. But what it comes up with is basically it starts to create its own lemmas, it starts to prove them, it puts little rectangles at the ends of its proofs, and it just kind of samples all of this, and yeah, I can barely distinguish it from the actual data, right? Sometimes it tries to create diagrams, so in this case it has almost succeeded. My favorite part is on the top left, the proof is omitted for some lemma. So you can feed in any kind of character data. We also tried feeding in code, so I went on the Linux repository and I downloaded all of source code of Linux. This turns out to be about 16 megabytes, or was it 600? A lot of megabytes of text, and this is C code, and then you feed it into the RNN and then it can just create code. And so this is a function that was generated by the RNN, and you can see that it takes some arguments, it has some variables, it creates for loops, if statements, It creates its own bogus comments, and syntactically— syntactically, this is— there's very few mistakes syntactically in what the RNN does. Some of the mistakes you'll notice, by the way, is that it uses variables that were not declared, or it declares the variables and never uses them. Those are the kinds of mistakes you'll see. Sometimes the RNN decides that it's time to start a new C file, and what it does then is it samples It recites character by character the GNU GPL license. It has learned it by heart, and then it creates some includes and then samples some macros and then dives into code. And so it has a very good idea of these statistical patterns in the data and how they should just be created there. Everything I've described so far, by the way, can be implemented in about 100 lines of Python code, where the only dependency is NumPy. And so this is a gist that you can look up on GitHub, and this is an entire character-level language RNN model in these 100 lines. Now, of course, if you'd like to actually be more serious and you'd like to use GPUs and so on, then you might be interested in my other project called CharRNN, and this is a more serious implementation, but it's basically the exact same thing. Now, many people have taken this code and they've kind of run it on very different things, so we can sample cooking recipes. This one is for basic cheese wings, and, you know, there's a whole bunch of ingredients. You preheat the oven to 350°F and so on. And so forth. So you can look at these. People have also created all kinds of crit— Twitter bots. So this CharNN was trained on the Bible and now it's sampling verses. You can feed it music. You can— it can generate also Magic cards and so on. So basically you can just— it's quite powerful and it's a very simple codebase. Now, in my own work at Stanford with my equal co-author on this paper, Justin Johnson, and my advisor, Fei-Fei Li, we were more interested in kind of studying just exactly how this works. So we went through character-level RNNs and we're studying the statistical patterns, how this works, what are the kinds of errors that it's still making, and how they change as you scale up the model. And so one of the experiments I'd like to highlight for you is we were looking at interpretable cells in the RNNs. And so in these experiments, we get to pretend that we are neuroscientists and we get the RNN to read some text, in this case some piece of code, and we're coloring— we're looking at a particular cell in the RNN, a particular hidden state vector, and and we're looking at what excites it as it's reading this text. And so this hidden state neuron is not really— does not really have an interpretable sequence that it responds to. But as you go through many neurons, you find several that are quite interpretable. So for example, we found a quote detection cell in the RNN. It basically is off, but then when it sees a quote, it turns on, and then it stays on until the quote ends. And so what's remarkable about this is that we, of course, didn't specify this, but in the process of the RNN trying to predict the next character, it has decided that it's useful to allocate some space in its hidden state to keeping track of the quotes. And so it just developed this little algorithm internally, you know, through backpropagation. We also found some cells that keep track of line lengths. So in this dataset here, there's a newline character after roughly every 80 characters. There's a newline character that starts a new line. And the RNN has therefore decided that it's a good idea to actually create a small counter for how far along you are in the sequence. And so there's a cell that basically turns on and then slowly decays over time, and this might help you predict the next newline character. There are these cells that only respond in if statements in code, or cells that, um, only respond inside comments or strings, or cells that get more excited as you indent your code and you're deeper in the nested expression. And so basically the RNN just chunkates your data in all these different patterns so that it can perform this prediction. Now in my own work roughly a year ago, I realized that these RNN models were extremely powerful, and I was specifically interested in applying them in a case of visual recognition. And so we're going to look at image captioning, where we take an image and then we're using an RNN to predict a sentence for it. And this model is made up of two components. There's a convolutional neural network, which we know is very good at seeing images, and then there's a recurrent neural network, which we know is very good at modeling sequences, in this case sequences sequences of words that make up a sentence. And then I like to use the analogy of playing with neural networks as kind of playing with Lego blocks, and so what we do is we just kind of stick these two components together. That corresponds to the arrow in between the two modules. So just to give you an idea about how this would work and how we could caption a test image, we take a test image and we feed it through a convolutional neural network. Now normally at the end of a convolutional network you have a classifier that would predict one of 1,000 categories for this image. But in this case, we're going to get rid of that classifier, and instead we're going to redirect the representation on the top layer of the convolutional network into the recurrent neural network. So we begin the generation with a special start token, and we forward the recurrent neural network to give us the probability distribution over the first word in the description, in y0. And critically, this prediction is also a function of the image through that added interaction into the RNN. And so you might imagine, for example, that the presence of the straw textures in this man's hat can positively influence the probability of sampling the word straw in this context at the first timestep. So we sample from that distribution at the very first timestep. Suppose that the word that was sampled is indeed straw, and we feed it back into the RNN, and we forward it again to get a probability distribution over the second word in the sequence, and we sample from it again, and suppose that it gives us the word hat, and we continue in this fashion just plugging in the words and it's generating new and new word for this image until it samples a special end token, and that's the period at the end of the sentence. And at this point, the RNN would have described that image as a straw hat period, one word at a time. So we train these data— we train these models end-to-end on image sentence datasets. One popular example right now that people are using often is Microsoft COCO. It consists of roughly 100,000 images and they are described with 5 sentences each, and this is data from Amazon Mechanical Turk. And so there's an example of a training image with some of its captions. And when we train this, the RNN learns to describe these images. And so here are some examples. So in the top row, for example, we have a man in black shirt playing guitar, or a construction worker in orange safety vest working on the road, and so on. On the bottom row, I have some of my favorite failure cases. We have a young boy holding a baseball bat, or a cat sitting on the couch with a remote control, or a woman holding a teddy bear in front of a mirror. I'm guessing it's the texture on the man's sweater. Or we have just a complete failure case of a horse in the middle of a road, but I can't see any horse, so I'm not sure what happened there. Now, in my most recent work, again with Justin Johnson, my equal co-author on this paper, and my advisor Fei-Fei Li, we've extended this basic model into this monster on the bottom. And effectively what this amounts to is what we have right now, the image sentence captioning model, is it takes a full image and it describes it with a single sentence. What we're doing now in this new model is we're tackling the joint task of both detection and description. So we can take an image and forward it through this model, and in a single forward pass of a network, we can both detect and describe everything in the image. And so here in this example, for example, you have detected a trunk of an elephant, or that whole elephant, or the ball is white, or there are large green trees in the background, or the roof of a building, the leg of an elephant, and so on. So everything is described in the image. Here's another example. We have light on a wall or sign on a wall, man sitting on a table, you know, laptop, white laptop on the table, and so on. And so there are hundreds of descriptions that this RNN produces for this image, but I'm only showing some of the top 20 because it's way too dense otherwise. The model also has a very good idea of object parts. For example, it knows that there's a tail of a plane or an engine of a plane or the nose of a plane and so on. We can also run this task conversely, and what we can do is we can query with a small piece of text, and then we can get the model to look for, through a database of images, for regions in your images that are compatible with that text snippet. So for example, when the query is head of a giraffe, we go off and we find all these little heads of giraffes, or legs of zebra, or red and white signs, or white tennis shoes, and so on. So we can search for arbitrary little text snippets in large collections of datasets, and it's relatively efficient. Now if you'd like to learn much more about these topics, we are teaching currently a class at Stanford called CS231N, and Justin and Fei-Fei are my co-instructors. And lecture videos, slides, notes, and assignments, everything is available so you can learn much more about the topic. And I think this is the last slide, so I'm done. Thank you. [APPLAUSE]