C5W3L07 Attention Model Intuition

For most of this week, you've been using a encoder decoder architecture for machine translation, where one RNN reads in a sentence and then a different one outputs a sentence. There's a modification to this called the attention model that makes all this work much better. The attention algorithm, the attention idea has been one of the most influential ideas in deep learning. Let's take a look at how that works. So given a very long French sentence like this, what we are asking this green encoder neural network to do is to read in the whole sentence and then memorize the whole sentence and store it in the activations conveyed here and then for the purple network, the decoding network to then generate the English translation. Jane went to Africa last September and enjoyed the culture. Many wonderful people. She came back raving about how wonderful trip was and so tempting me to go to. Now, the way a human translator would translate the sentence is not to first read the whole French sentence and then memorize the whole thing and then regurgitate an English sentence from scratch. Instead, what a human translator would do is read the first part of it, maybe generate part of the translation, you know, look at the second part, generate a few more words, look at a few more words, generate a few more words, and so on. You kind of work part by part through the sentence because it's just really difficult to memorize the whole long sentence like that. And so what you see for the encoder decoder architecture above is that it works quite well for short sentences. So you might achieve a relatively high bleu score. But for very long sentences, maybe longer than 30 or 40 words, you know, the performance comes down. So the bleu score might look like this as the sentence length varies and short sentences are just hard to translate, hard to get all the words right. And long sentences, it doesn't do well on because it's just difficult to get a neural network to memorize a super long sentence. So in this and the next video, you see the attention model, which translates maybe a bit more like humans might, looking at part of a sentence at a time. And with an attention model, machine translation systems performance can look like this because by working one part of a sentence at a time, you don't see this huge dip, which is really measuring the ability of a neural network to memorize a long sentence, which maybe isn't what we most badly need a neural network to do. So in this video, I want to just give you some intuition about how attention works, and then we'll flesh out the details in the Next video. The attention model was due to Dmitri Badenu, Kyung Kyun Cho and Yosha Venzhou. And even though it was obviously developed for machine translation, it spread to many other application areas as well. And this is really a very influential, I think, very seminal paper in the deep learning literature. So let's illustrate this with a short sentence, even though these ideas were maybe developed more for long sentences, but it'll be easier to illustrate these ideas with a simpler example. We have our usual sentence, j' ain vesit vrique en sa temps. And let's say that we use a rnn. And in this case I'm going to use a bidirectional RNN in order to compute some set of features for each of the input words. And here I've drawn the standard bidirectional rnn with outputs y1, y2, y3 and so on up to y5. But we're not doing a word for word translation. So let me get rid of the Y's on top. But using a bidirectional rnn, what we've done is for each of the words, or really for each of the five positions in the sentence, you can compute a very rich set of features about the words in the sentence and maybe surrounding words in every position. Now let's go ahead and generate the English translation. We're going to use another RNN to generate the English translation. So here's my RNN node as usual. And instead of using A to denote the activation, in order to avoid confusion with the activ activations down here, I'm just going to use a different notation. I'm going to use S to denote the hidden state in this RNN up here. And so instead of writing a one, I'm going to write S one. And so we hope in this model that the first word it generates will be Jane. Right, to generate Jane visits Africa in September. Now the question is, when you're trying to generate this first first word, this output, what part of the input French sentence should you be looking at? Seems like you should be looking primarily at this first word, maybe at a few other words close by, but you don't need to be looking way at the end of the sentence. So what the attention model would be computing is a set of attention weights. And we're going to use alpha to denote when you're generating the first word, how much should you be paying attention to this first piece of information here? And then we'll also come up with a second, let's call it attention weight alpha 12 which tells us when we're trying to compute the first word, hopefully, Jane, how much attention, how much attention should be paying to this second word from the input and so on. And we need Alpha 1, 3 and so on. And together this will tell us what is exactly the context denote as C that we should be paying attention to. Um, and that is input to this RNN unit to then try to generate the first word. So that's one step of the rnn. And we'll flesh out a lot of these details in video. For the second step of this RNN, we're going to have a new hidden state S2 and we're going to have a new set of attention weights. So we're going to have Alpha 21 to tell us when we're generating the second word. I guess this would be visits, maybe that would be the ground truth label. How much should we pay attention to the first word in the French input? And also Alpha 2, 2 and so on. How much are we paying attention to the word visits? How much should we pay attention to the freak? And so on. And of course the first word we generated, Jane, is also an input to this and we have some context that we're paying attention to. In the second step there's also an input and that together will generate the second word. And that leads us to the third step, sv, where this is an input and we have some new context C that depends on the various alpha ZV for the different time steps. That tells us how much we paying attention to the different words from the input fringe sentence and so on. So some things I haven't specified yet, but that will go further into detail in the next video is how exactly this context defined and the goal of the context is for the third word. It really should capture that. Maybe you should be looking around this part of the sentence and the formula you use to do that will defer to the next video as well as how do you compute these attention weights. And you see in the next video that Alpha 3T, which is when you're trying to generate the third word. This would be Africa if it's getting the right output. The amount that this RNN step should be paying attention to the French word at time T that depends on the activations of the bidirectional RNN at time T cycles will depend on the forward activations and backward activations at time T. And and it will depend on the state from the previous steps. It will depend on S2 and these things together will influence how much you pay attention to a specific word in the input French sentence. But we'll fetch out all these details in the next video. But the key intuition to take away is that this way the RNN marches forward, generating one word at a time, until eventually it generates maybe the EOs. And at every step there are these attention ways alpha T T prime that tells it when you're trying to generate the t English word, how much should you be paying attention to the t prime French word? And this allows it on every time step to look only maybe within a local window of the French sentence to pay attention to when generating a specific English word. So I hope this video conveyed some intuition about the attention model and that you now have a rough sense of maybe how the algorithm works. Let's go on to the next video to flesh out the details of the attention model.