Sequence to Sequence Deep Learning (Quoc Le, Google)

I will divide it in two parts. So number one, I will work with you and develop the sequence to sequence learning. And then the second part, I will place sequence to sequence in a broader context or a lot of exciting work in this area. So let's motivate this by an example. So a week ago I came back from vacation and in my Inbox I have 508 emails, unreplied emails. And a lot of emails basically just require just yes and no answer. So let's try to see whether we can build a system that can automatically reply these emails to say yes or no. And for example, some of the email would be from my friend aunt, she said hi in the subject and she said are you visiting Vietnam for the New Year Quok. That would be her content. And then my probable trip live would be yes. So you can gather another set like this and then you have some input content. So for now let's ignore the author of the email and the subject, but let's focus on the content. So let's suppose that you gather some email and some input would be something like are you visiting Vietnam for the New Year Quark? And the answer will be yes. And then another email would be are you hanging out with us tonight? The answer is no, because I'm quite busy. So the third email would be did you read the cool paper on Resnet? The answer is yes, because I like it. Now let's do a little bit of processing where basically in the previous slide we have year and comma and then quark and then question mark and so on. So let's do a little bit of processing and then put the comma, a space between year and comma and then and so on. So this step a lot of people call tokenization and normalization. So let's do that with our emails now. So, and then the second step I would do would be to do feature representation. So in this step what I'm going to do is the following. I'm going to construct a 2000 dimensional vector, 2000 represent the size of English vocabulary. And then I'm going to go through email, I'm going to count how many times a particular word occur in my email. For example, for example the word R occur one in my email. So I increase the counter and then u occur once or I increase another counter and S, et cetera. And then I will reserve at the end a token to just count all the words that just out of vocabulary. Okay? And then now if you do this process, you're going to convert all of your email from input to output pairs where the input would be fixed length representation of 20,000 dimensional vector and output would be either year or one. Okay, any questions so far? Okay, good. Okay, so I will get. So as you said, somebody in the audience said the order of the words don't matter and the answer is yes. So I'm going to get back to that issue later. Now so that X and Y and now your job. My job now is to try to find some W such that W times X can approximate Y. Y is the output right and Y here is yes and no. So because of this problem has two categories, you can think of it as a logistic regression problem. Now if anybody follow the great CS210 29 class by Andrew probably can formulate this very quickly. But in very short, the outcome comes as follow. You try to come up with a vector for every email. Your W is a two column matrix. The first column will find the probability for whether the email have to be answered as yes. Second column will be answered as no. And then you basically take the dot product between the first column. Algorithm is called the stochastic gradient descent. So you run for iteration one to like a million. You run for a long, long time. You sample a random email X and there's some reply. And then if the reply is yes, then you want to update your W1 and W2 so such that you increase the probability that the answer is yes. So you increase the first probability. Now if the correct reply is no, then you're going to update W1 and W2 so that you can increase the probability of the email to be answered as no. So the second probability, so, so let's call those P1 and P2. So because I said to update the increase of probability, what does that mean? What that means is that you find the gradient of the partial gradient of the objective function with respect to some parameter. So now you have to pick some alpha, which is the learning rate. And then you say W1 is equal to W1 plus some alpha, the partial derivative of log of P1 with respect to D of W1. Now I cheated a little bit here because I used the log function. It turns out because the log function is a monotonic increasing function, so increasing P1 is equivalent to increasing the log of P1. And usually with this formulation, stochastic gradient descent works better. Any questions so far? And then you can also Update, you know, W2 if the email is to be replied is yes. And you can, you can have different way to update if the reply is no. So what's. And then if you have a new email coming in, then you take X and then you convert into the vector. Then you compute the first probability w 1 times x divided by exponential w 1 times x plus exponential w 2 times x. And if that probability is larger than 0.5, then you say yes. And if that probability is less than 0.5, then you say no. So that's how you do prediction with this. Now, now there's a problem with this representation is that there's some information loss. So somebody in the audience just said that the order of the words don't matter. And that's true. Now let's fix this problem by using something called the recurrent network. And I think Richard Soldier already talked about recurrent networks and some part of it yesterday, and Andre as well. Now the idea of a recurrent network is basically you have also have fixed length representation for your input, but it actually preserves some sort of ordering information. And the way that you compute the hidden units are the following. So the function H of 0 is basically hyperbolic tangent of some matrix U times the word vector for the word R. So Richard also talked about word vectors yesterday. So you can take word vectors coming out of word 2vec, or you can just actually randomly initialize them if you want to. So let's suppose that that's h of 0. Now h of 1 would be a function of h 0 and the vector for U which is A times h of 0 plus u times v of vector U. And then you can keep going with that. This is one of my three most complicated slides, so you should ask questions. No questions. So everybody familiar with recurrent nets, huh? Wow. Okay, so to make prediction with this, you tag on the label at the last step and then you say, try to predict Y for me. Now how do you do that? Now here basically you, you went the way you did before and basically you make update on the W matrix, which is the classifier at the top, like what I said earlier. But you also have to update all the relevant matrices, which is the matrix U, the matrix A and some work vectors. So this is basically you have to compute the partial derivative of the loss function with respect to those parameters. Now that's going to be very complicated and usually when I do that myself, I get that wrong. But there's a lot of toolkits out there that you can use, which is you can use auto differentiation in TensorFlow, or you can call Torch, or you can call Teano to actually compute the derivatives. And once you have the Derivatives, you can just make the updates. Yeah, yeah. So u, the matrix U are share. So I'm going to go back to one side. So this matrix U are share for all vertical matrices. And the size you have to determine ahead of time. For example, the number of column would be the size of the word vectors, but the number of rows must be like 1000 if you want, or maybe 255 if you want. So this is model selection and it depends on whether you underfitting or overfitting to choose a bigger model or a smaller model and your compute power so that you can train a larger model or smaller model. The matrix U. Yeah. So the word vectors. The word vectors, the number of word vectors that you use are the size of vocabulary. Right? Which is. So you're going to tend to end up with 20,000 word vectors. Right? But the size of. So that means you have 20,000 rows in matrix U. But the number of column you can, Sorry, the number of columns is 20,000, but the number of row would be. You have to determine just yourself. Okay, any other questions. Now? Okay, so what's the big picture? So the big picture is I started with bag of words representations. And then I talk about I and N as a new way to represent variable size input that can capture some sort of ordering information. Then I talk about auto differentiation so that you can compute the partial derivatives and you can find auto diff in TensorFlow or Tiano or Torch. Now then I talk about stochastic gradient descent as a way to train the neural networks. Any questions so far? Ok, you have a question. How long is the waiting time? If you have. Oh, that also depends on how big your training set and how big is your computer and so on. Right. But usually if you use RNN and if you use like a hidden state of 100, it should take like a couple of hours. Yeah, but it depends, largely, largely depends on, you know, size of training data because you want to iterate for a lot of. You sample a lot of emails. Right. And you want your algorithm to see as many emails as possible. Right? So, okay, so if you use such algorithms to just say yes, no and yes, no, then you might end up losing a lot of friends. Because. We don't just say yes. No, because when you say when, for example, my friend asked me, are you visiting Vietnam for the New Year walk? Then maybe the better answer would be yes. See you soon. That's a better, nicer way to approach this. And then if my friends ask me, are you hanging out with us tonight? So instead, just Say no. I would say no, I'm too busy. Or did you read the cool token? So let's see how we're going to fix this. So before I going to tell you the solution, I would say this is the. This problem basically requires you to map between variable size input to some variable size output. And if you can do something like this, then there's a lot of applications because you can do auto reply, which is what we've been working on so far, but we can also use it to do translation translate between English, French. You can do image captioning. So input would be a fixed length vector representation coming from confnet. And then output would be the cat set on the mat. Or you can do summarization, the input will be a document and output would be some summary of it. Or you can do speech transcription where you can have input would be speech frames and output would be words. Or you can do conversation. So basically the input would be the conversation so far and the output could be multiply. Or you can do Q and A, et cetera, et cetera. So we can keep going on now. So how do we solve this problem? So this is hard. So let's check out what Andre Capathi has to say about recurrent networks. Okay, so Andrei said that there's more than one way that you can configure neural networks to do things. So you can use neural network to map recurrent network to map one to one, right? So at the bottom that's the input. The green would be the hidden state and the output would be what you want to predict. Now one to one is not what we want because we have many to many. So it's probably more like the last to the right. But we arrived as the solution that I said in the red box. And the reason why that's a better solution is because the size of the input and the size of output can vary a lot. Sometimes you have smaller input but larger output. But sometimes you have larger input and smaller output. So if you do the one in the red circle, you can be very flexible. If you do the one to the extreme right, then maybe the output has to be smaller or at least the same with the input, which that's what we don't want. So let's construct a solution that look like that. Okay, so here's the solution. So the input would be something like hi, how are you. And then let's put a special token. Let's say the token is and. And then you're going to predict the first token which is M and then you predict the second token. Fine. And then you predict the third token. Thanks. And then you keep going on until you predict the word and. And then you stop. Now I want to mention that in the previous set of slides I was just talking about yes and no. And in yes and no you have only two choices. Now you have more than two choices, you have actually 20,000 choices and you can actually use the algorithm that are the logistic regression and you can extend it to cover more than two choices. You can have a lot of choices and then the algorithm will just follow the same way. So this is my first solution when I work to seq. To seq. But it turns out it didn't work very well. And the reason why it didn't work very well is the model never know what it actually predicted in the last step. So it keep going and it keeps synthesizing output. But it didn't know what it said, it didn't know what decision it committed in the previous step. So a better solution would look like this. A better solution is basically you feed what the model predicts in the previous step as input to the next step. So for example, in this case I'm going to take M, I'm going to feed it in to the next step so that I'm going to predict the second world, which is fine, and et cetera. So a lot of people call this concept autoregressive. So you eat your own output and make it as your input. Any questions so far? How do you know when to stop? Oh, whenever it produce end then you stop, there's a special token and. Yeah, now, okay, so relevant architecture here would be the people also call the encoder as the recurrent network in the input and the decoder would be the recurrent network in the output. Okay, so how do you train this? So again, so you basically you run for a million steps, you see all your emails and then you say you sample, for each iteration you sample an email X and a reply Y would be I'm fine, thanks. And then you sample a random word YT in Y and then you update the inn encoder and decoder parameters so that you can increase the probability that Y of T is correct, given all what you've seen before, which is your YT minus 1, YT minus 2, et cetera and also all the axes. Right. And then you have to compute the partial derivatives to make it work. So the computing partial derivatives is very difficult. So again I recommend you to use something like auto differentiation in TensorFlow. Or Torch or Tiana. OK, you have a question. So the number of nodes are changing dynamically for each sample. Yeah. But the recurrent. The number of parameters didn't change because you have U and V. U, V and A are fixed. Right. Okay. So the question in the audience is that if the R and N are different for different example and the answer is yes, so the number of steps are different. I have a question there. How do you evaluate that it's doing the right thing in the test set? Because it's in. Yeah, I'm going to get to that in the next slide. Yeah. Okay, So the question is, in practice, how long would I go for the RNN? I would say if you usually stop at like 400 steps or something like that, because outside of that is going to be too long to make the update and compute. It's very expensive to compute, but you can go more if you want to. Yeah, I have a question. Yeah, Yeah, yeah. So that's a problem. So if at prediction, I'm going to talk about the prediction next. So let me go to the prediction and then you can ask questions. So how do you do prediction? So the first algorithm that you can do is called greedy decoding. In greedy decoding is for any incoming email X, I'm going to predict the first word and then you find the most likely word and then you feed back in and then you find the next most likely word and then you feed back in and. And et cetera. So you keep going, you keep going until you see the word end and then stop. Or if it exceeds a certain length, you stop. Now that's too greedy. So let's do a little bit less greedy. So it turns out that given X you can predict more than one candidate. So let's say you can predict K candidates. Let's say three. So they take three candidates and then for each candidate you're going to fit in the next step and then you arrive at three. So the next step you're going to have nine candidates and then you're going to end up going that way. So here's a picture. So given the input X, I'm going to predict the first tokens that will be high. Yes. And please. And given every first token like this, I'm going to feed back into the network and the network will produce another three and et cetera. So you're going to end up with a lot of, a lot of candidates. So how do you select the best candidate? Well, you can traverse each beam and then you compute the joint probability at Each step and then you find the sequence that have the highest probability to be the sequence of choice. What is your reply? Any question? This is the most complicated slide in meta. Oh yeah? Yes. So the question is, what do you do with the out vocabulary words? Now it turns out in this algorithm what you do is that for any word that is out vocabulary, you create a token called unknown and you map everything to unknown or anything that our vocabulary to be unknown. So it doesn't seem a very nice thing, but usually it works. Well, there's a bunch of algorithms to address these issues. For example, they break it into characters and things like that and then you could fix this problem. Yeah, yeah. What is the cost function? The cost function is that. So I go back one slide. So the cost function. One more slide. So the cost function is that you sample a random word yt Here, let's suppose that here, this is my input so far input and I'm sample yt. Let's say t is equal to 2. So which means the word find. Okay, I'm at the word find. I want to increase the probability of the model to predict word fine. So every time the model will make a lot of predictions, a lot of them will be incorrect. Right. So you have a lot of probabilities, you have probability for the word and then probably and et cetera. And then probably for zzzzzzz. Right. And you have a lot of probabilities. You want the probability for the worst, for the fine to be as high as possible, you increase the probability. Does that make sense? Yeah, but we don't care. Oh, you condition on im. So when I'm at fine, my input would be hi, how are you? And I'm ok. That's all I see. And then I need to make a prediction. I have to make that prediction. Right? Right. And if I'm at the word thanks, my input will be hi, how are you? And I'm fine. And I gotta get my thanks. Probability. Right. Okay. Yeah. I have a question here. Oh, I haven't thought about it yet. So the question is, how do you personalize so. Well, one way to do it is basically embed a user as a vector. So let's suppose that you have a lot of users and you embed a user as a vector. That's one way to do it. Yeah, I have a question here. So basically if all the sequences were the same length number paths on the tree is K to the N, right? Yeah. So that's a lot. Yeah. So the question is that let's suppose that my beam search is 10, then you go to from 10, like 100 and then 1000 and suddenly it grows very quickly. Right. It go through. Like if your sequence is long, then you end up with K to the N or something like that. Well, one way to do it is basically you do truncated beam search where any sequence with very low probability, you just kick it out. You don't use it anymore. So you can do this, you can do 3, 9, and then you 27 and then you go back up to 9 and then you keep going. So that way you don't end up with a huge beam. And usually in practice, using a beam size of 3 or 10 would work just fine. It works great. Yeah, I have a question there. Okay, so because it's an inn, we don't have to pad the input now to be fast. Sometimes we have to pad the input because we want to make sure that bash processing works very well. So we pad. But we pad with only like zero tokens. Okay, you do change the graph from batch to batch. Yeah. So let's suppose that you have a sequence of 10, then you have a graph of 10, then you have a sequence, a batch of all 20, you have make a graph for 20 and et cetera. Yeah, that will make the GPU very happy. I have a question there. Where you would connect that embedding in this network as the initial state or. Oh, so. So are you asking. So the. So my interpretation of your question is how do you insert the word embedding into the model? Is that correct? User embedding. User embedding. Oh, if you want to personalize the thing, then at the beginning you have a vector, and that's a vector for quark with the ID 1, 2, 3, 4, 5. And then if it's Peter, then the vector would be 5, 6, 7, 8. So Peter would be the initial vector for the encoder, like the initial state. Yeah. Yeah, that's one way to do it. Yeah. Well, there's more than one way. You can do it at the end or you can do it at the beginning. You can insert at every prediction steps. But my proposal is just put it at the beginning. That's simpler. Okay, I have a question there. Yeah, you. That's a very good question. The question is, what if the model derails? Right. If you make a prediction and then that's a bad prediction and then your model never see and then it keeps derailing and it will produce garbage. Yeah, that's a good question. So I'm going to get to that. So. Well, so this is a slide. So there's an algorithm called scheduled sampling. So in scheduled sampling what you do is instead of feeding the truth during training, you can feed what sample from the submax, so what generated by the model and then feed in as input so that the model understands that if it produce something bad, it would actually recover from it. So that's one way to address this issue. Does that make sense? Yeah. Any question? There's a question here. Okay, Yeah, yeah, yeah. So in this algorithm the question is how large is the size of the decoder? Well, my answer is that try to be as large as possible, but is it going to be very slow? And in this algorithm what happens is that you, you use the same, you use fixed length embedding to represent very much the long term dependency, like a huge input, that's going to be a problem. So I'm going to come back to that issue with the attention model in a second. Any question? Ok, here's a question. Scoring based on a single word, doesn't that make you learn away from synonyms? You're unlikely to use them in the same sentence for the answer. So does the model learn synonyms? Is that the question or what's the question? Aren't you biasing it against synonyms? So if your answer is fine and you're scoring based on that answer. Oh, I see. Well, yeah, it turns out that if you learn it turns out that it map good and if you visualize embedding, the good and fine and so on are mapped very closely to the embedding space. But in the output we don't know what else to do. The other approach is basically to train the world embeddings using world2vec and then try to ask the model to regress to the world embeddings. So that's one way to address the issue. We tried something like that, did not work very well. So whatever we had in here was pretty good. Okay, I have to keep going. But anyway, the algorithm that you've seen so far, turns out actually answers some emails. So if you use the Smart Reply feature in inbox, it's already used this system in production now. For example, in this email, my colleague Greg Corrado got an email from his friend saying that hey, we wanted to invite you to join us for an early Thanksgiving on November 22nd. Beginning around 2pm Please bring your favorite dish and reserve by next week and then it will propose three answers. For example, the first answer would be towers in. Second answer would be we'd be there and the first answer is, sorry, we won't be able to make it. Now, where do these three answers come from? Those are the beams. Now there's an algorithm to actually figure out the diversity as well of the beams so that you don't end up with very similar answers. So there's an algorithm like a heuristic that make these beams a little bit more diverse and then they pick the best three to present to you. Okay. Any question? Yeah, I have a question here. How do you make sure that. Yeah, there's no guarantees. So the question is how do I guarantee that the beam will terminate an end. Now there's no guarantee it can go on forever. Indeed there are certain cases like that if you don't train the model very well, but if you train the model well with very good accuracy, then the model usually terminates. I hardly see any cases that it doesn't terminate. Yeah, but there are certain corner cases that it will do funny things. But you can stop the model after like 1000 or 100 or something like that so that you make sure that the model doesn't do that, doesn't go on crazy. Right. I have a question. Yeah. There's nothing in the email which says that they're inviting multiple people, but the reply seems. That's very interesting. Yeah, it just comes out because there's a lot of emails and if you invite someone there's more than one person and then maybe it learns about Thanksgiving. It just means inviting the whole family, things like that. Yeah, it just learn from statistics. Yeah. Or maybe there's something like that. Yeah. Ok. Oh, in this algorithm. So the question is do I do any post processing to correct the grammar of the beams in this algorithm? We did not have to do it. Yeah. Okay. I have another question. So how contextual these mouse are? So, okay, so the question is how contextual? So I would say we don't have any user embedded in this. So it's pretty general. The input would be the previous emails and the output would be the prediction, the reply. That's all we have. So it sees the context which is the threat so far. Okay. Did I answer your question? Okay. Yeah, you can catch me up after the talk. Yeah. Oh, it run on the server? Yeah, slow question I guess. There's definitely some email which is not suitable for auto smart reply. How do you use. Oh, I see. So the question is there's certain emails that are not relevant for smart reply. Maybe they're too long or you should not reply or something like that. So in fact we have two algorithms. So one Algorithm to say yes or no to reply. Then after it passes, sees the threshold, there's an algorithm to run to produce the threshold. So it's a combined of two algorithms that I presented earlier. I have to get going, but you can get back to the question. So there's a lot of more interesting stuff coming along. Okay, so what's the big picture so far? So the big picture is that we have an inn encoder that it's on the input and then we have an in decoder trying to predict one token at a time in the output. Now everything else falls as the same way. So you can use stochastic gradient descent to train the algorithm. And then you do beam search decoding. Usually you do a beam search of three and then you should be able to find good would be with the highest probability. Now, someone in the audience brought up the issue that we use fixed length representation. So just before you make a prediction, the H of N, the H of N, the white thing right before you go to the decoder, okay, that is a fixed length representation and you can think of it as a vector that capture all everything in the, in the input. It could be 1,000 words or it could be five words. And you use a fixed length representation for variable length input, which is kind of not so nice. So we want to fix that issue. So there's an algorithm coming along and it's actually invented at University of Montreal. Joshua, he's here. So the idea is to use an attention. So how does an attention work? So in principle, what do you want is something like this every time before you make a prediction. Let's say you predict the word M, you kind of want to look again at all the hidden states so far. You want to look at all what you see in the input so far. Now same when you do find, you also want to see all the hidden state of the input so far and on. Now how do you do that as a program? So well, you can do this. So at H of M you predict a vector C. Let's say that vector is the same dimension with all the H. So if your H of 1 is dimension of 100, then C also have a dimension of 100. Okay, and then you take C and then you dot product with all the H and then you have coefficients a a 0, a 1, blah blah blah, a to the N. Okay, and those are scalars. Okay? And then after you have those scalars you compute something called the beta, which is basically a submax of all the alpha. So to compute that you take the exponent bi is an exponential AI divided by the sum of exponentials. And then you take those bi and then multiply by H of I. And then you take the weighted average and then you take the sum and then you send it to as additional signal to predict the world M. And then you keep going with that. So in the next step you also predict another C. And then you take that C to compute the dot products. You compute the a, and then you can compute the B, you can take the B, you do the weighted average and then you send it to the next step to send it to the prediction. And then you use stochastic gradient descent to train everything. Okay, and this algorithm is implemented in TensorFlow. Okay, so intuitively, what is going on here? So let's suppose that you want to use this for translation. So in translation you want to, for example, the input would be hi, how are you? And the output is hola como stats or something like that. Okay? And then when you predict the first word, you want hola to correspond to the word high. Okay? Because there's a one to one mapping between the word high and hola. So if you use the attention model, the betas that you learn will put a strong weight for the words ola for the word high and then it has a smaller weight for all the stuff. And then if you keep going, then when you say comus, then it will focus on ha and et cetera. Okay. So it move that coefficient, it put a strong emphasis on the relevant words. And especially for translation, it's extremely useful because, you know, the one to one mapping between the input and output. Any questions so far? This is definitely very complicated. Yeah, I have a question. How do you deal with languages by word or there's. Oh, right now the A and B are learned, so I don't. So the question is, how do I deal with languages where the order like reverse, for example, English to Japanese. Right. So some of the verbs get moved and things like that. Well, I did not hard code A or B. They are learned. So by virtue of learning, they will figure out what beta to put right to weight the input. And those are basically computed by gradient set. Right. So they just keep on learning. Okay, I have a question here. Okay. Yeah. So the question is, are there any work on putting attention in the output? Yeah, I think you can do that. I'm not too familiar with any work in here, but I think it's possible to do it. I think some people explore something like that. Yeah. Any question? Oh, I have a question. Another question. So with the capitalization of your first words in the. Does that imply that if you have a vocabulary. Yeah, yeah, yeah. So the question is, let's suppose, because right now the word high is capitalized at the first character. It doesn't mean I'm using two n or n vocabulary size. So in practice we should do some normalization. If you have a small data set, what you should do is you normalize the text. So high will be like lowercase and et cetera. Now if you have a huge data set, doesn't matter, we just learn. Okay, yeah, I have a question there. Right, yeah, so yeah, so the question is, in a sense is capture the positional information in the input. Yeah, I agree. I have a question there. Button punctuation. So the question is, what do I do with punctuation? Well, right now I just present the algorithm as if it's a very simple implementation, like the very basic. But one thing that you can do is before you train the algorithm, you put a space between the word and the punctuation so that you do some. That step is called tokenization or normalization in language processing. So you can use any Stanford NLP package or something like that to normalize your text so that it's easy to train. Now if you have infinite data, then it will just learn itself. Okay, so I should get going because there's a lot of other interesting stuff. Okay, so it turns out that that's the basic implementation. But if you want to get good results and if you have big data sets. So one thing that you can do is to make the network deep. And one way to make deep is in the following way. So you stack your recurrent network on top of each other in the first sequence. To sequence paper, we use a network of four, but people are gradually increasing to six, eight and so on right now. And they getting better and better result. Like in imagenet, if you make a network deeper, you also get better results. Okay, so if you want to train sequence to sequence with attention, then a couple of years ago when we, like many labs working on this problem were behind the state of the art. But right now in translation, many translation tasks, basically this model already achieved state of the art result in a lot of these WMT data sets. So to train this model. So number one is that as I said, you might end up with a lot of vocabulary, our vocabulary issues. So Barack Obama would be just an unknown, right? Hillary Clinton and is an unknown. Now you might use something like word segments, right? So you segment the words out. For example, Barack Wama would be ba and rak and et cetera. Or you can use all the smart algorithms. For example, word character split. You can split words that have unknown to be into characters and then you treat them as a character. There's some work at Stanford and they prove that it works very well. So that's one way to do it. Now tip number two is that when you train this algorithm, because when you do back propagation or forward propagation, you essentially multiply a matrix many, many times. So you have explosion of function value or the gradient or implosion as well. Now one thing that you can do is you clip the gradient a certain value. So you say that if the gradient magnitude of the gradient is larger than 10, set it to 10. Then tip number three is to use GRU. Or in our work we use long short term memory. So I want to revisit this long short term memory business a little bit. So what's long short term memory? So if you use an in cell, basically you concatenate your input and the hidden state and then you multiply by some theta and then you apply with some activation function. Let's say that's a hyperbarbolic tangent. Okay, now that's the simple function for inn. Now in lstm you basically you multiply the input and H by a huge, a big matrix, let's call that theta. That theta is four times bigger than the theta I said in the inn cell. And then, and you're going to take that Z that coming out, you split it into four blocks, each block you can compute the gates and then you use the value of something called the cell. And then you keep adding the newly computed values to the cell. So there's a part here that I say the integral of C. What it does is basically it keep a hidden state which, where it keep adding information to it. So it doesn't multiply information, but it keeps adding information. You don't need to know a lot of this if you want to just apply LSTM because it's already implemented in TensorFlow. Any questions so far? Okay, so in terms of applications, you can use this thing to do summarization. So I've seen, I've started seeing work in summarization, pretty exciting. You can do image captioning and the input in that case would just be the representation of an image coming out from VGG or coming out from Google. Net and et cetera. And then you send it to the rnn. The ANN will do the decoding for you. Or you can use it for speech recognition or transcription or you can use it for Q and A. So the next part of the talk, I will talk a little bit about speech recognition. Well, in speech recognition the input would be maybe some waveforms and then the output would be some words. Hi, how's it? Well, one thing that you can do is you crop your input into windows, that's the green boxes there. And then you crop a lot of them and then you send a lot of them to an inn, and then you convert it into MFCC before you send to MFCC or spectrogram or something like that. And then you use the algorithms that I said earlier, and then with attention, and then you do the transcription, you predict one word at a time in the output. Now the problem with this algorithm is that when it comes to speech, you end up with a lot of inputs. You can end up with thousands and thousand steps. So back propagating in time, even with attention, can be difficult. Now one thing that you can do is basically you do some kind of a pyramid to map the input. So if you do enough layers, you can divide your input into factor of 8 or 16, if you do enough layers and then you produce the output. So we're working on an implementation where the output is actually characters like in the Baidu's work, where they have the ctc. Now I have to say that the strength of this algorithm is actually have an implicit language model in the output. So when I have the word how it's actually conditioned on high and stop before and including the input. So there's an implicit language model already. But the problem with this is that actually you have to wait until the end of the input to do decoding. So the decoding has to be done offline. Okay. So if you use this for voice search, it might not be too nice because people want to see some output right away. So in that case, there's an algorithm that can use this and do it in an online fashion block by block. Now also I have to mention that in translation, the sequence, sequence with attention works great, it's among the state of the art. But when it comes to speech, it doesn't work as well as the ctc, at least in published results we're not as good as ctc, which is what Adam talked earlier, or some of the, hmm, DNN hybrid, which is the most widely speech system currently. So I want to pause there and then I can take questions. Any questions? I have a question at the back. Yeah, yeah, Yeah. Oh. So how does it work in translation? Well, in translation what we do is Basically we have pairs of sentences. So for example, hi, how are you? And then. Hola. Right. And then we have pairs of sentences like this. And then we just feed it into the sequence to sequence attention at every step. Again, we're going to predict one word at a time. But before we make a prediction, the model has the attention so it actually see the input once more before it makes a prediction. That's how it works. Now, what is the issue with the model again, please? Yeah, the first word itself in the sequence is actually. The prediction is actually just 100 words. I see. Well, I, I don't, I don't, I can't quite follow the question. But let, let's take it offline. Is that okay? Yeah, yeah. And then we can do some paper. Okay. Together. I have a question. Yeah. So I have a question. For example, what if you actually compose emails in different languages like Vietnamese and English? Yeah. Yeah. Okay. So the model, the inbox thing that I presented, it was on in English, but there's no limitation in the model in terms of language. So let's suppose that in your inbox, sometimes you write in English and sometimes you write in Vietnamese or sometimes you write it in Spanish, whatever, and you personalize by user embedding. Then I would say that it will just learn your behavior and then it will basically predict the world that you want. But make sure that your output vocabulary is large enough so that it covers not only the English words, but also the Spanish word and et cetera, like Vietnamese and so on. So your vocabulary going to be not going to be 20,000, it's going to be like 100,000 because you have more choices. And then you have to train your model on those examples. Yeah, it's a matter of training data, that's all. Okay, I have a question there. It cannot do translation on the fly. Yeah. Is it possible to change your model a little bit so that it doesn't have to start predicting at the end? Yeah, yeah, yeah. So the question is that in the case of voice search, right now you have to wait to the end to make a prediction. Is there any other ways? Yeah, yeah. The answer is yes. You can make a prediction block by block. So you can actually figure out like an algorithm, a simple algorithm to actually segment the speech and then make a prediction and then take the prediction and feed it as input at the next block. So you can keep going like that. So in theory you can actually do online decoding, but I'm saying that the work on you can do online decoding, but that work is Currently work in progress. How about that? Okay, I have a question there. So the question is regarding the Google's auto reply. Yeah. So if I have to build a system like that, I mean, I was wondering how do you guys set up with a training data set? Oh yeah. So. So we have some input email and then some output email where expert written emails reply and then you can just train it that way. So you have like a trained data set that you. To put these. Yeah. Okay. I have a couple questions. You. Okay. It seems like with sequence to sequence there's not much of a constraint on how the output aligns with the input while the CTC there is sort of constrain that goes left to right. Is there any way to add this constraint to sequence? Sequence that it works better on something like speech. Yeah. The question is that in speech recognition the CTC seems to be a very nice framework because it match if there's a monotonic increasement in the output and the input. But CTC make this independent assumption. It doesn't have a language model in it. Maybe the sequence to sequence, we can address this. Yeah, I think that's a great idea. Maybe we should write a paper together. Okay. I think I haven't seen it, but I think that's a very good idea question. I see. Okay, great. So the question is that because right now we predict one step at a time, is there any way to actually look globally at the output and maybe use some kind of reinforcement learning to adjust the output? And the answer is yes. So there's a recent paper at Facebook called I think sequence level training or something like that where they don't optimize for one step at a time, but they look at the globally and then they try to improve world error rate or they try to improve bleu score or things like that for translation. And it seems to be making some improvement in the metrics that they care about. Now if you show it to humans though, people still prefer the output from this model. So some of the metrics that we use in translation and so on might not be what the metrics that we optimize and the next step prediction seem to be what people like a lot in translation. Yeah. So the question is, can we add the GAN loss? Like the GAN loss? Yeah, I think that's a great idea. Yeah. I have a question here. Yeah. Based on the human input. Yeah. Is there a way to dynamically change the encoder also change once you have the model? Like is there a way based on the human input to influence the Encoder. Yeah, yeah. So let's suppose that you type the first word hola. Then you can actually start the beam from there. So the question is, is there any way to incorporate user input? So I say yeah, let's suppose that you want to. You say hola, sorry, hi, how are you? Right. And then as soon as the person type ola, that actually restrict your beam, so you can actually condition your beam on the first word ola and your beam will be better. Yeah, I think that's a good idea. I have a question. Oh, so how much data did we use? So in translation, for example, we use several WMT corpuses and the WMT corpora usually have tens of millions pairs of sentences, something like that, and every sentence have like 20 words on average. 20, 30 words on average. I can't remember. But that's something like that order of magnitude. Yeah, yeah. I have a question there. I can't really hear. Oh. So how does it compare to Google Search auto completion? Honestly, I don't know what's used underneath Google Search Auto Completion, but I think they should use something like this because it's. Okay. I have still lots of interesting stuff coming along. So what's the big picture? So the big picture is so far I talk about sequence to sequence learning and yesterday Andrew was talking about most of the big trends in deep learning and he was talking about the second trend was basically doing end to end deep learning. So you can characterize sequence to sequence learning as end to end deep learning as well. So the framework is very general. So it should work from a lot of NLP related tasks because a lot of them you would have input sequence and output sequence. I have still lots of interesting stuff coming along. So what's the big picture? So the big picture is so far I talk about sequence to sequence learning and yesterday Andrew was talking about most of the big trends in deep learning and he was in nlp. It could be input would be some text and output would be some passing trees. That's also possible, but it works great when you have a lot of data. Now, when you don't have enough data, then maybe you want to consider dividing your problems into smaller components and then train your sequence to sequence in the sub components and then merge them. Now if you don't have a lot of data but you have a lot of related tasks, then it's also possible to actually merge all these tasks by combining the data and then have an indicator bit to say this is translation, this is summarization, this is email reply, and then change it only. And that should improve your output too. Talking about the second trend was basically doing end to end deep learning. So you can characterize sequence to sequence learning as end to end deep learning as well. So the framework is very general. So it should work from a lot of NLP related tasks because a lot of them you would have input sequence and output sequence. Now this basically concludes the parts about sequence to sequence and then the next part, I'm going to place sequence to sequence in a big picture of the active ongoing work in neural nets for nlp. So if you have any question you can ask now, I take maybe two questions because I think I running out of time. So I have a question. Yeah. Does your model handle emoji? So the question is, does the model handle emoji? I don't know. But emoji is like a piece of text too, right? So you can just like feed it in as another extra token. If you make your vocabulary 200,000, then you should be able to cover emoji as well. Yeah. I have a question. In nlp, it could be input would be some text and output would be some passing trees. That's also possible. But it works great when you have a lot of data. Now, when you don't have enough data, then maybe you want to consider dividing your problems into smaller components and then train your sequence to sequence in the sub components. If you have new data coming in. So should I retrain the model? Well, I think towards the end we lower the learning rate. So if you add new data, it just, it will not make a lot of good updates. So usually you can add new data, increase the learning rate and then continue to train. Yeah, that should work. Okay, so I already took two questions. Let's keep going. So there's an active area that actually is very exciting, which is in the area of automate and then merge them. Now, if you don't have a lot of data, but you have a lot of related tasks, then it's also possible to actually merge all these tasks by combining the data and then have an indicator bit to say, this is translation, this is summarization, this is email reply and then genetic Q and A. So you can think that maybe the setup would be can you read a Wikipedia page and then answer a question? Or can you read a book and answer a question? Now in theory, you can use sequence to sequence with attention to do this task. So it's going to look like this. You're going to read the book one token at a time with the book, then read the question. And then you're going to use the attention to look at all the pages and then you make a prediction of the tokens. Right. So that's kind of. That's kind of. Sometimes we do answer this question that way. Sometimes we don't have knowledge about the fact. So we actually read the bullet journal. And that should improve your output too. Now this basically concludes the parts about sequence to sequence and then the next part I'm going to play sequence to sequence in a bigger picture of the active ongoing work in book again to answer the fact. But a lot of the time, if you ask me, is Barack Obama the President of the United States? I would say yes, because it's already in my memory. So maybe it's better to actually augment the ANN with some kind of memory. Okay. So that it will not do this look back again. It's kind of annoying. Look back again. So there's an active area of this research. I'm not a definite expert, but I'm very aware. So I can place you in the right context here. So work in this area would be memory networks by Western and folks at Facebook. There will be neural tutorial machines at Deep Mind neural nets for nlp. So if you have any question you can ask now. I take maybe two questions because I think I running out of time. So I have a question. Yeah. Dynamic memory networks would be Richard Soldier, presented yesterday and then stuck augmented RNNs by Facebook again and et cetera. Now. Well, let's. So I want to show you high level, what is this? So the question is augmented memory means. So let's think about the attention. So the attention look like this. So in the encoder you're going to look at some input and then you have a controller which is your H variable. And then you keep updating your height variable. But. But along the side you're going to write down into memory your H1, H2, H3 and et cetera. You store it into a memory. Clear? Right. And then the decoder, what you're going to do is you continue producing some output. You're going to update your controller G. Does the model handle mog? I don't know, but it's mog. But you're going to read from memory your H. Okay. So again in the input you write to memory and then in the output you read from memory. Now let's try to be a little bit more general. And the general would be at any point in time you can read and write. You have a controller and you can read and write. Read and write all the time. Now to do that you have to follow an architecture. So you have some memory bank. Emoji is like a piece of text too. Right. So you can just like feed it into as another extra token. If you make your vocabulary 200,000, then you should be able to cover emoji as well. Yeah. I have a question. A big memory bank. Okay. And then you can use the write. You can decide to write some information into it by a combination of the memory bank in the previous step and the hidden variable in the previous step. And then you also read into the hidden state too. And then you keep making a memory update and then you can keep going forever like that. So this concept is called RNN with augmented memory. Okay, Is that somewhat clear? Any question? You have a question? Oh, so if you have new data coming in. So should I retrain the model? I think towards the end we lower the learning rate. So if you add new data, it just. The question is when you read, do you read the entire memory bank? A lot of these algorithms are actually soft attention. So yes, it will look the entire memory. You can actually predict where to look and then read that. Only that block. Now the problem with that is you end up with very. It's not differentiable anymore. Right. Because the thing that you don't read don't contribute to the gradient. So it's going to be hard to train. But you can use reinforce and so on to train it. So there's a recent paper reinforcement learning neural Turing make a lot of good updates. So usually you can add new data, increase the learning rate and then continue to train. Yeah, that should work. Okay, so I already took two questions. Let's keep going. So there's an active area that actually is very exciting which is in the area of automations that actually server does something like this. Not exactly, but it will deal with discrete actions. Okay. Any question? No question. Wow. Okay, so another extension that a lot of people talk about is using ANN with augmented operations. So you want to augment Q and A. So you can think that maybe the setup would be can you read a Wikipedia page and then answer a question or can you read a book and answer a question? Now in theory you can use sequence to sequence with attention and then to do this task. So it's going to look like this, the neural network with some kind of operations like addition, subtraction, multiplication, the sine function, et cetera, a lot of functions. So to motivate you, you can think about Q and A can fall into this. For example, Here's a context. The building was constructed in the year 2000. And then it was then later on people say, well, or it was then destroyed in the year 2010. And then the question would be, how long did the building survive? You're going to read the book one token at a time with the book, then read the question and then you're going to use the attention to look at all the and the answer would be 10 years. Now, how would you answer this question? Pages and then you make a prediction of the tokens, right? So that's kind of. Sometimes we do where you say 2010, subtract 2010 years. Now, neural nets, if you can train with a lot of sample, it can do that too. It can learn to subtract numbers and things like that, but it requires a lot of data to do. So maybe it's better to augment them with functions like addition and subtraction. So the way you can do it is that the neural network will answer this question that way. Sometimes we don't have knowledge about the fact, so we actually read the book again to answer the fact. But a lot of the time, if you ask me, is Barack Obama the President of the United States? I would say yes, because it's already in my memory. So maybe it's better to actually augment the. Read all the tokens so far and will push the numbers into a stack. And then you get. The neural net is augmented by a subtraction and a addition function. And then you assign this probability for these two functions. So green, the more dark that means the higher probability. Okay, so you assign two probability and you compute the weighted average of the values coming out of these two functions in N with some kind of memory so that it will not do this loop. And then you take that and then you pop it and you push it into the stack in the next step. And then in the next step you will call the addition and subtraction again and et cetera, that back again. It's kind of annoying. Look back again. So there's an active area of this research. I'm not a definite expert, but I'm very aware, so I can place you in the right context here. So work in this area would be memory networks by the principle of something called neural programmers or neural programmer interpreters. So there are two papers last year from Google Brain and DeepMind was talking about this. So that's some of the related work in the area of augmenting recurrent networks with operations with memory, et cetera. Now, what's the big Western and folks at Facebook there will be neural tutorial machines at DeepMind Picture. Okay so the big picture I want to revisit and I say so what I've talked to today is sequence to sequence learning and it's an end to end deep learning task. So it's one of the big trends happening in natural language. It's very general. So dynamic memory networks would be. Richard Soldier presented yesterday and then stuck augmented RNNs by Facebook again and et cetera. Now well let's. So I want to show you like high level what is it you can do if you have a lot and lot of supervised data? It's a supervised learning algorithm. So if you have a lot of data it should work great. But if you don't have enough supervised data then you consider dividing your problem and then training in different components. Or you can train jointly in a multitask settings and people also train it jointly with Autoencoder. Namely you read the input sentence and then predict the output. Augmented memory means okay, so and that's also. And then you train jointly with all the tasks and that works. Let's think about the attention. So the attention look like this. So in the encoder you're going to look at some input. Okay. And then you have a controller which is your H variable as well. If you go home and then you want to make impact at your work tomorrow, then so far that's so far so good that can make. And then you keep updating your height variable but along the side you're going to write down into memory your hash impact. Now if you want to do some research and I think things with memory operation augmentation are some of the exciting areas. But it seems like still work in progress. But expect a lot of advances in this area in the near future. So one H2, H3 and et cetera, you store it into a memory clear right. And in the decoder what you're going to do is you going to continue producing some output. You're going to update your controller G. If you want to know more you can take a look at Chris Ola blog. He talk about attention and augmented recurrent networks. I also wrote some tutorials. Pretty simple but you're going to read from memory your H. Okay right. This. The sequence to sequence with attention for translation is implemented in TensorFlow. So you can download, you can actually download TensorFlow and train it. What I said today. Now there's a lot of work going on in this area. Many. So again so in the input you write to Memory of these are not mine. So as you can see, you can even read the words and then in the output you read from memory. Now let's try to be a little bit more general. And the general would be at any point in time how many papers come along in this area. So you can read and write. You have a controller and you can read and write. Read and write all the time. I can pause there and I have five minutes to answer questions. I have a question there. Yeah, you were talking about reading a book and giving an answer from that book. Yeah. If you have a corpus of books context. Now to do that you have to follow an architecture. So you have some memory bank, a big memory bank. And then you can use the write. You can decide to write some information into it by a combination of the memory bank in the previous step. Yeah, Yeah, I see. Can you speak to the microphone? Because I can't hear very well that microphone and then I think people can hear that as well. When you're training a Q and a network. So you're taking the example of the hidden variable in the previous step and then you also read training from a book to answer questions. Yeah. So if let's say Harry Potter, who was Harry Potter's father, There could be many books that have a character to the hidden state too. And then you could make update and then you can keep going forever like that. So there's a context resolution issue, which is which Harry, should I answer the question for? Yeah. How do you solve the context context problem when you're training this kind of Q and A type network? I think that's a great question. So I think one thing is that you can always personalize, for example, you know that the guy when. So this concept is called RNN with augmented memory. Okay, is that somewhat clear? Any question you have a question it talks about you can have a representation for the user. And then you know that when he say Harry he's because he actually been reading a lot of books about Harry Potter. So. It's more likely to be Harry Potter. But I think with the algorithm I said I just want to make sure that it's as simple as possible. So if you do the question is when you read do you read the entire memory bank? A lot of these algorithms are actually soft attention. So yes, it will look so has to ask the question Harry Potter rather than Harry. But I'm saying if you represent user vectors and then you inject more additional knowledge about the users about the context into additional token in the input of the net the Net configured out the entire memory. You can actually predict where to look and then read that. Only that block. Now the problem with that is you end up with very by itself. Yeah. So that's one way to do it. Yeah. Okay. I have a question there. Yeah. You did some work on Doc 2 Vec. Yeah. Do you have an idea what the state of the art in generalizing Word2Vec is to more than one word? Oh, I see. I think it's not differentiable anymore. Right. Because the things that you don't read don't contribute to the gradient. So it's going to be hard to train. But you can use reinforce and so on. Skip Thoughts are interested in directions to train it. So there's a recent paper, reinforcement learning neural Turing machines that actually saw it does something like this, right? Not exactly, but it will. So Dr. Vec is one way, but Skip Thoughts. So the idea of Skip Thoughts was actually Ruslan. So Salakush Dino was the author on this. And his idea is basically using sequence to sequence to predict the next sentence. So the input would be the current sentence, the output would be the previous sentence or the next sentence. And then you could train a model like that. The model is filled with discrete actions. Okay. Any question? No question. Wow. Skip thought. And I've heard a lot of good things about Skip Thoughts where you can take the embedding at the end and then you can do document classification and things like that. Okay. So another extension that a lot of people talk about is using ANN with augmented operations. So you want to augment that, and it works very well. So that's probably one place that you can go. My colleague at Google is also working on something called Autoencoder. So instead of predicting the next sentence, he predict the current sentence. So trying to repeat the current sentence. And that's kind of work well too. Yeah, yeah. The neural network with some kind of operations like addition, subtraction, multiplication, the sine function, et cetera, a lot of functions. What's your thoughts on how to solve the common sense reasoning problem? Oh, common sense. I'm deeply interested in common sense, but I gotta say, I have no idea. I think maybe you can do something like. I think common sense is about a lot of. First of all, there's a lot of knowledge about the world. So to motivate you, you can think about Q and A can fall into this. For example, here's a context. The building was constructed in the year 2000. It's not captured in text, for example, gravity and things like that. So maybe you really need to actually combine a lot of modality. That's one way to think about it all. The thing is make sure that unsupervised learning work. And then it was then later on people say, oh, it was then destroyed in the year 2010. And then the question would be, how long did the building survive? And the answer would be 10 years. Now how would you answer this question where you say 2010, subtract 2010 years. Now, neural nets, if you can train with a lot example, it can do that too. It can learn to subtract numbers and things like that, but it requires a lot of data to do so. So maybe it's better to augment them with functions like addition and subtraction. So the way you can do it is that the neural network will read all the tokens so far and will push the numbers into a stack. And then you get the neural net is augmented by a subtraction and a addition function. And then you assign these a probability for these two functions. So green, that's another approach. But I think this research area, I think I'm just making guesses right now. Is there a good way to represent all these rules and using some software? Yes, yes. So the question is, how do you represent rules? So if you think about this network, the neural programmer network that it actually augmented by addition and subtraction, then these are rules. You can augment it with a table of rules and then ask the network to actually attend into that root table. People have looked into this direction. So that's one way to do it. Okay, you're saying basically I'm this to do some logical reasoning. Yeah, yeah, yeah. Hey, great talk. Yeah, thank you. Is there like a practical rule of thumb for how many sequence pairs you need to train such a model successfully? And is there, are there any tips to reduce how many pairs you need if you don't have this data? I see, okay, so usually the bigger data set the better. But the corpus that people train this on, translation, for example English to German, it has only about 3,5 million. The more dark this means, the higher probability. So you assign two probability and you compute the weighted average of the values coming out of these two functions, sentences or something like that. So that's kind of small. 3 million. And still people are able to make it to the state of the art. So that's pretty encouraging. Now if you don't, and then you take that and then you pop it and you push it into the stack in the next step, and then in the next step you will Call the addition and subtraction again and et cetera. Even though I have a lot of data then I would say things like pre train your word vectors with language models or Word 2 VEC. That's one area that you have a lot of parameters. You can. That's the principle of something called neural programmers or neural programmer interpreters. So there are two papers last year from Google Brain and Dietmain was talking about this. Pre train your model with some kind of language model and then you reduce the softmax. That's another area that you have a lot of parameters or use dropout in the input embedding or go. So that's, that's some of the related work in the area of augmenting some random word in the input sentence so those things can improve the regularization when you don't have a lot of data. Okay, yeah, thank you. Recurrent networks with operations with memory, et cetera. Now what's the big. Okay, yeah, thank you. Thank you Kwok. So we'll reconvene at 6 o' clock for Yoshi Bengel's closing keynote picture. Okay, so the big picture I want to revisit and I say so what I've talked to today is sequence to sequence learning and it's an end to end deep learning task. So it's one of the big trends happening in natural language. It's very general so you can use if you have a lot and lot of supervised data. It's a supervised learning algorithm. So if you have a lot of data it should work great. But if you don't have enough supervised data then you consider dividing your problem and then training in different components. Or you can train jointly in a multitask settings and people also train it jointly with autoencoder namely you read the input sentence and then predict the output sentence again and that's also. And then you train jointly with all the tasks and that works as well. If you go home and then you want to make impact at your work tomorrow, then that's so far so good that can make some impact. Now if you want to do some research and I think like things with memory, operation augmentation are some of the exciting areas but seems like still work in progress but expect a lot of advances in this area in the near future. So if you want to know more you can take a look at Chris Ola blog. He talk about attention and augmented recurrent networks. I also wrote some tutorials. Pretty simple this. The sequence to sequence with attention for translation is implemented in TensorFlow so you can download. You can actually download TensorFlow and train it. What I said today. Now there's a lot of work going on in this area. Many of these are not mine. So as you can see, you can even read the words. That means how many papers come along in this. This area? So I can pause there and I have five minutes to answer questions. I have a question there. Yeah, Yeah, I see. Can you speak to the microphone? Because I can't hear very well that microphone. And then I think people can hear that as well. When you're training a Q and a network. So you're taking the example of training from a book to answer questions. Yeah. So if, let's say Harry Potter, who was Harry Potter's father, there could be many books that have a character Harry. So there's a context resolution issue, which is which Harry. Should I answer the question for? Yeah. How do you solve the context context problem when you're training this kind of Q type network? I think that's a great question. So I think one thing is that you can always personalize, for example, you know that the guy, when he talk about you can have a representation for the user. And then you know that when he say Harry, because he actually been reading a lot of books about Harry Potter. So it's more likely to be Harry Potter. But I think with the outcome, I said, I just want to make sure that it's as simple as possible. So if you do, the user has to ask the question Harry Potter rather than Harry. But I'm saying if you represent user vectors and then you inject more additional knowledge about the users about the context into additional token in the input of the net, the net can figure it out by itself. So that's one way to do it. Yeah. Okay. I have a question here. Yeah. You did some work on Doc 2 Vec. Yeah. Do you have an idea what the state of the art in generalizing Word2Vec is to more than one word? Oh, I see. I think Skip Thoughts. Interesting directions here. So Dr. Veck is one way, but Skip thoughts. So the idea of Skip Thoughts was actually Ruslan Salakushdinov was author on this. And his idea is basically using sequence to sequence to predict the next sentence. So the input would be the current sentence, the output would be the previous sentence or the next sentence. And then you could train a model like that. The model is called Skip Thought. And I've heard a lot of good things about Skip Thoughts where you can take the embedding at the end and then you can do document classification. And things like that. And it works very well. So that's probably one place that you can go. My colleague at Google is also working on something called Autoencoder. So he instead of predicting the next sentence, he predict the current sentence. So trying to repeat the current sentence. And that's got to work well too. Yeah, yeah. What's your thoughts on how to solve the common sense reasoning problem? Oh, common sense. I'm deeply interested in common sense, but I gotta say, I have no idea. I think maybe you can do something like. I think common sense is about a lot of. First of all, there's a lot of knowledge about the world that is not captured in text, for example, gravity and things like that. So maybe you really need to actually combine a lot of modality. That's one way to think about it. The other thing is make sure that unsupervised learning work. That's another approach. But I think this research area, I think I'm just making guesses right now. Is there a good way to represent all these rules and using some software? Yes, yes. So the question is, how do you represent rules? So if you think about this network, the neural programmer network that it actually augmented by addition and subtraction, then these are rules. You can augment it with a table of rules and then ask the network to actually attend into that root table. People have looked into this direction. So that's one way to do it. Okay, you're saying basically I command this to do some logical reasoning. Yeah, yeah, yeah. Hey, great talk. Yeah, thank you. Is there like a practical rule of thumb for how many sequence pairs you need to train such a model successfully? And is there, are there any tips to reduce how many pairs you need if you don't have? Okay, so usually the bigger data set the better. But like the corpus that people train this on, translation, for example English to German, it has only about 3,5 million pairs of sentences or something like that. So that's kind of small. 3 million. Right. And still people are able to make it to the state of the art. So that's pretty encouraging. Now if you don't even have a lot of data, then I would say things like pre train your world vectors with language models or Word 2 VEC. That's one area that you have a lot of parameters. You can pre train your model with some kind of language model and then you reduce the submax. That's another area that you have a lot of parameters. Or use dropout in the input embedding or drop out some random word in the input sentence. So those things can improve the regularization when you don't have a lot of data. Okay. Yeah. Thank you. Okay. Yeah. Thank. You. Thank you, Bob. So we'll reconvene at 6 o' clock for Yosha Benjo's closing keynote.