Andrew Ng explains how encoder-decoder models work: an encoder RNN reads an input sequence and compresses it to a fixed vector, then a decoder RNN generates output one token at a time. The same approach applies to image captioning using a CNN to extract features and an RNN to write captions.
Hello and welcome to this final week of this course, as well as to the final week of this sequence of five courses in the deep learning specialization. You're nearly at the finish line. In this week you hear about sequence to sequence models, which are useful for everything from machine translation to speech recognition. Let's start with the basic models, and then later this week you hear about Beam Search, the attention model, and we'll wrap up a discussion of models for audio data like speech. Let's get started. Let's say you want to input a French sentence like Jean Vicique Le Fric in September, and you want to translate it to the English sentence. Jane is visiting Africa in September, as usual. Let's use x1 through x in this case5 to represent the words in the input sequence. And we'll use Y1 through Y6 represent the words in the output sequence. So how can you train a neural network to input the sequence X and output the sequence Y? Well, here's something you could do. And the ideas I'm about to present are mainly from these two papers due to Ilya Suskava, Oyo Vigneaus and Kwak Ler, and that one by Kyung Hyun, Cho Baad Van Marien Bohr, Kegler Gausser, Dimitri Badenil, Feki Bogueras, Holger Strank and Josh Benjamin. First, let's have a network, which we're going to call the encoder network, be built as a rnn, and this could be a GRU or an LSTM feed in the input French words, one word at a time. And after ingesting the input sequence, the RNN then outputs a vector that represents the input sentence. After that you can build a decoder network, which I'm going to draw here, which takes as input the encoding output by the encoder network shown in black on the left, and then can be trained to output the translation one word at a time, until eventually it outputs, say, the end of sequence or end of sentence token upon which the decoder stops. And as usual, we could take the generated tokens and feed them to the next set of hidden states in the sequence, like we're doing before when synthesizing text using the language model. One of the most remarkable recent results in deep learning is that this model works given enough pairs of French and English sentences. If you train a model to input a French sentence and output the corresponding English translation, this will actually work decently well. And this model simply uses an encoding network whose job it is to find an encoding of the encoding input French sentence and then use a decoded network to then generate the corresponding English translation. An architecture very similar to this also works for image captioning. So given an image like the one shown here, maybe you want it to be captioned automatically as a cat sitting on a chair. So how do you train a neural network to input an image and output a category caption like that phrase up there? Here's what you can do from the earlier course on confnets. You've seen how you can input an image into a convolutional network, maybe a pre trained Alexnet, and have that learn an encoding or learn a set of features of the input image. So this is actually the Alexnet architecture. And if we get rid of this final softmax unit, the pre trained Alexnet can give you a 4096 dimensional feature vector of which to represent this picture of a cat. And so this pre trained network can be the encoder network for the image. And you now have a 4,000 or 4,096 dimensional vector that represents the image. You can then take this and and feed it to an RNN whose job it is to generate the caption one word at a time. So similar to what we saw with machine translation translating from French to English, you can now input a feature vector describing the input and then have it generate an output sequence or output set of words one word at a time. And this actually works pretty well for image captioning, especially if the caption you want to generate is not too long. As far as I know. This model, this type of model was first proposed by Junhua Mao, Wei Shu Yi Jiang, Wang Zhiheng Huang and Alan Yu. Although it turns out there are multiple groups coming up with very similar models independently and at about the same time. So two other groups that had done very similar work at about the same time, and I think independently of Mao et al, were audio vignales Alexander Toshev, Sammy Banjo and Dmitri Erhan, as well as Andrej Karpathy and Fei Fei Li. So you've now seen how a basic sequence to sequence model works, or how a basic image to sequence or image captioning model works. But there's some differences between how you would run a model like this to generate a sequence compared to how you were synthesizing novel text using a language model. One of the key differences is you don't want a randomly chosen translation. You maybe want the most likely translation, or you don't want a randomly chosen caption. Maybe not, but you might want the best caption. The most likely caption. So let's see in the next video how you go about generating that.
