MIT 6.S094: Recurrent Neural Networks for Steering Through Time

All right, so we've talked about regular neural networks, fully connected neural networks. We've talked about convolutional neural networks that work with images. We've talked about reinforcement, deep reinforcement learning, where we plug in a neural network into a reinforcement learning algorithm. When an agent, when a system has to not only perceive the world but also act in it, collect reward. And today we'll talk about perhaps the least understood but the most exciting neural network out there. Flavor of neural network is recurrent neural networks. So first, administrative stuff. There's a website, I don't know if you heard cars, mit Edu, where you should create an account if you're a registered student. That's one of the requirements. You need to have an account if you want to get credit for this. You need to submit code for Deep traffic, JS and deepteslajs. And for deep traffic, you have to have a neural network that drives faster than 65 miles an hour. If you need help to achieve that speed, please email us. We can drop. We can give you some hints. For those of you who are old school SNL fans, there's a deep thought thoughts section now in the profile page where we encourage you to talk about the kinds of things you tried in Deep Traffic or any of the other deep Tesla or any of the work you've done as part of this class for deep learning. Okay, We've talked about the vanilla neural networks on the left. The vanilla neural network is the one where it's computing. It's approximating a function that maps from one input to one output. An example is mapping images to the number that's shown in the image. For ImageNet, it's mapping an image to what's the object in the image. It could be anything. Now in fact, convolutional neural networks can operate on audio. You could give it a chunk of audio 5 second audio clip that still counts as one input because it's fixed size. As long as the size of the input is fixed, that's one chunk of input. And as long as you have ground truth that maps that chunk of input to some output, ground truth, that's a vanilla neural network. Whether that's a fully connected neural network or a convolutional neural network. Today we'll talk about the amazing, the mysterious recurrent neural networks. They compute functions from one to many, from many to one, from many to many. Also bidirectional. What does that mean? They take as input sequences time series, audio, video, whenever there's a sequence of data, and that temporal dynamics that connects the data is more important than the spatial content of each individual frame. So when there's a lot of information being conveyed in the sequence, in the, in the temporal change of whatever that type of data is, that's when you want to use recurrent neural networks like speech, natural language, audio. And the power of this is that for many of them, for a recurrent neural network, where they really shine is when the size of the input is variable. So you don't have a fixed chunk of data that you're putting in, it's variable input. And the same goes for the output. So you can have, you could give it a sequence of space speech, several seconds of speech, and then the output is a single label of whether the speaker is male or female. That's many to one. You can also do many to many translation. You can have natural language put into the network in Spanish and the output is in English. Machine translation, that's many to many. And that many to many doesn't have to be mapped directly into same size sequences. So for video, the sequence size might be the same. You're labeling every single frame. You put in a five second clip of somebody playing basketball and you can label every single frame, counting the number of people in every single frame. That's many to many. When the size of the input size of the output is the same. Yes, question. The question was are there any models where there's feedback from output and input? And that's exactly what recurrent neural networks are, is produces output and it copies that output and loops it back in. That's exactly, that's exactly almost the definition of a recurrent neural network. There's a loop in there that produces the output and also takes that output as input once again. And so there's also many, many where the sequences don't align. Like machine translation, the size of the output sequence might be totally different than the input sequence. We'll look at a lot of cool applications like you can start a song, so learn on the audio of a particular song and, and have the recurrent neural network continue that song after a certain period of time. So it can learn to generate sequences of audio of natural language, of video. Ok, I know I promised not many equations, but this is so beautifully simple that we have to cover back propagation. It's also the thing that if you're a little bit lazy and you go on the Internet and start using the basic tutorials for TensorFlow, you ignore how back propagation work at your peril. You kind of assume it just works. I give it some inputs Some outputs, and it's like Lego pieces. I can assemble them like you might have done with deep traffic. A bunch of layers, put them together, and then just press train and back propagation is the mechanism that neural networks, currently the best mechanism we know of that is used for training. So we need to understand the simple power of back propagation, but also the dangers. Summary. Say up at the top of the slide, there's an input for the network is an image. There's a bunch of neurons, all with differentiable signs, smooth activation functions on each neuron. And then as you pass through those activation functions, taking the input, pass it through this net of differentiable compute nodes, you produce an output. And that output, you also have a ground truth, the correct, the truth that you hope you expect the network to produce. And then you can look at the difference between what the network actually produced and what you hoped it would produce. And that's an error. And then you back or propagate that error, punishing or rewarding the weights, the parameters of the network that resulted in that output. Let's start with a really simple example. There's a function that takes as input up on top three variables X, Y and Z. The function does two things. It adds X and Y, and then it multiplies that sum by Z. And then we can formulate that as a circuit circuit of gates where there's a plus gate and a multiplication gate. And let's take some inputs shown in blue. Let's say X is negative 2, Y is 5, Z is negative 4. And let's do a forward pass through this circuit to produce the output. So negative 2 +5 equals 3. Q is that intermediate value 3. This is so simple and so important to understand that I just want to take my time through this because everything else about neural networks just builds on these concepts, right? Okay, so the, the AD gate produces Q. In this case, it's 3. And then 3 times negative 4 is 12. That's the output. The output of the circuit of this network, if you think of it as such, is negative 12. And so the forward pass is shown in blue. The backward pass will be shown in red in a second here. So what we want to do is what would make us happy, what would make F happy is for the output to be as high as possible. Negative 12 is so. So we could do better. So how do we teach it? How do we adjust X, Y and Z such that it produces a higher F, makes us happier. Okay, let's start backward. The backward pass, we make the gradient on the output one, meaning we want this to increase, we want F to increase. That's how we encode our happiness. We want it to go up but one. And in order to then propagate that fact that we want the F to go up by one, we have to look at the gradient on each one of the gates. Now, what's a gradient? It's a partial derivative with respect to its inputs. The partial derivative of the output of a gate with respect to its inputs. If you don't know what that means, it's just the. How much does the output change when I change the inputs a little bit, what is the slope of that change? If I increase X For the first function of addition, F of X, Y equals X plus Y. If I increase X by a little bit, what happens to F? If I increase Y by a little bit, what happens to F? So taking a partial derivative of those with respect to X and Y, you just get a slope of 1. So when you increase X, F increases linearly. Same with Y. Multiplication is a little trickier. When you increase X, F increases by Y. So the partial derivative of F with respect to X is Y. The partial derivative of F with respect to Y is X. So if you think about it, what happens is the gradients when you change X, the gradient of change doesn't care about X, it cares about Y. So it's flipped. So we can back propagate that one, the indication of what makes us happy backwards. And that's done by computing the local gradient for Q. So the partial derivative of F with respect to Q, that intermediate value, that gradient will be negative 4. It will take the value of Z. As I said, it's the multiplication gate. It'll take the value of Z and assign it to the gradient. And the same for the partial derivative of F with respect to Z, it will assign that to Q, the value of the forward pass from the Q. So there's three and a negative four in the forward pass in blue and then that's flipped negative 4 and 3 on the backward pass. That's the gradient. And then we continue in the same exact process. But wait. So what makes all of this work is the chain rule. It's magical. So what it allows us to do is to compute the gradient, the gradient of an F with respect to the inputs xyz. We don't need to construct the giant function that is the partial derivative of F with respect to X and Y and Z analytically. We can do it step by step back, propagating the gradients, we can multiply the gradients together, as opposed to doing partial derivative of F with respect to X, we have just the intermediate the local gradient of F with respect to Q and of Q with respect to X and multiply them together. So instead of computing the gradient of that giant function, x + y equals times Z, in this case, it's not that giant, but it gets pretty giant with neural networks. We just go step by step. We'll look at the first function, simple addition. Q equals X plus Y. And the second function, multiplication, F equals Q times Z. So the gradient on X and Y, the partial derivative of F with respect to X and Y is computed by multiplying the gradient on the output negative 4 times the gradient on the inputs, which, as we talked about when the operation is addition, that's just one. So it's negative four times one. That means. What does that mean? Let's interpret those numbers. You now have gradients on X, Y and Z. The gradient of the partial derivative of F with respect to X, Y, Z, that means. So for X and y, it's negative 4, for z is 3. That means in order to make F happy, we have to decrease the inputs that have a negative gradient and increase the inputs that have a positive gradient. The negative ones are X and Y, the positive is Z. Hopefully, I don't say the word beautiful too many times in this presentation, but this is very simple, beautifully simple. Because this gradient is a local worker, it propagates for you. It has no knowledge of the broader happiness of F, it just propagates. It computes the gradient between the output and the input. And you can propagate this gradient based on, in this case, F, a gradient of one, but also just the error instead of one, we could have on the output. The error is the measure of happiness. And then we could propagate that error backwards. These gates are important because you can break down almost every operation we could think of that we work with in neural networks into one of several gates like this. And the most popular are three, which is addition, multiplication and the max operation. So for addition, what you do is you ignore the. Okay, the process is you take a forward pass to the network so we have a value on every single gate. And then you take a backward pass, and through the backward pass you compute those gradients. For an AD gate, you equally distribute the gradients on the output to the input. So when the the gradient on the output is negative 4, you equally distribute it to negative to negative 4 and you ignore the forward pass value so that 3 is ignored when you back propagate it on the multiplication gate. On the multiply gate, it's trickier. You switch the forward pass values. So if you look at F, that's a multiply gate, the forward pass values are switched and multiplied by the value of the gradient in the output. If it's confusing, go through the slides slowly. It'll make a lot more sense, hopefully. One more gate, there's the match gate, which takes the inputs and produces as output the value that is larger. And when computing the gradient of the max gate, it distributes the gradient similar to the AD gate, but to only one, To only one of the inputs, the largest one. So it, unlike the AD gate, pays attention to the input values on the forward pass. All right, lots of numbers. But the whole point here is it's really simple. What a neural neural network is just a simple collection of these gates. And you take a forward pass, you calculate some kind of function on the end, the gradient at the very end, and you propagate that back. So usually for neural networks, that's an error function, a loss function, objective function, cost function, all the same word. So that's the sigmoid function there, F. When you have three weights, W0, W1, W2 and X2 inputs, X0, X1, that's going to be the sigmoid function. That's how you compute the output of the neuron. But then you could decompose that neuron, you could separate it all into just a set of gates like this. Addition, multiplication, there's exponential in there, and division, they're all very similar. And you repeat the exact same process. If the there's five inputs, there's three weights and two inputs, x0, x1. You take a forward pass through this circuit. In this case, again, you want it to increase so that the gradient on the output is 1, and you back propagate that gradient of 1 to the inputs. Now, with neural networks, there's a bunch of parameters that you're trying to, through this process, modify. And you don't get to modify the inputs, you get to modify the weights along the way. And the biases, the inputs are fixed, the outputs are fixed, the outputs that you hope the network will produce. What you're modifying is the weights. So I get to try to adjust those weights in the direction of the gradient. That's the task of backpropagation. The main way that neural networks learn as we update the weights and the biases to decrease the loss function. The lower the loss function, the better. So in this case, you have three inputs on the top Left, the simple network, three inputs, three weights. On each of the inputs, there's a bias on the node B. It produces an output A. And that little symbol, this is indicating a sigmoid function. And loss is computed as Y minus A squared divided by 2, where y is the ground truth, the output that you want the network to produce. And that loss function is back propagated in exactly the same way that we described before. All right, so the subtasks that involved in this update of weights and biases is that the forward pass computes the network output at every neuron. And then finally the output layer computes the error, the difference between A and B. And then backward propagates the gradients. Instead of one on the output, it'll be the error on the output and you back propagate it. And then once you know the gradient, you adjust the weights and the biases in the direction of the gradient, or actually the opposite of the direction of the gradient, because you want the loss to decrease. And the amount by which you make that adjustment is called the learning rate. The learning rate could be the same across the entire network, or it could be individual to every weight. And the process of adjusting the weights and biases is just optimization. Learning is an optimization problem. You have an objective function and you're trying to minimize it. And your variables are the parameters, the weights and biases. And neural networks just happen to have tens, hundreds of thousands, millions of those parameters. So the space, the function that you're trying to minimize is highly nonlinear. But it boils down to something like this. You have two weights here, two plots, or actually one weight, sorry, one weight. And then as you adjust it, the cost, you adjust in such a way that minimizes the output cost. And there's a bunch of optimization methods for doing this. This is a convex function. So you can find the minimum, the local minimum, if you know about these kind of terminologies, the local minimum is the same as the global minimum. So there's not. It's not a weirdly hilly terrain where you can get stuck in. So your goal is to get to the bottom of this thing. And if it's really complex terrain, it'll be hard to get to the bottom of it. So there is a lot of different. This general approach is gradient descent. And there's a lot of different ways to do gradient descent, some adding in various ways of adding randomness into the process so you don't get stuck into the weird crevices of the terrain. All right? But it's messy. You have to Be really careful. This is the part you have to be aware of when you're designing a network for deep traffic and nothing is happening. This might be what's happening. Vanishing gradients or exploding gradients when the partial derivative is small. So if you take as a sigmoid function, the most popular for a while activation function, the derivative is, is zero at the tails. So when the input to the sigmoid function is really high or really low, that derivative is going to be zero. So the gradient that you compute gradient tells you how much I want to adjust the weights. The gradient might be zero. And so you back propagate that zero a very low number and it gets less and less as you back propagate. And so the result is that you don't, you think that you don't need to adjust the weights at all. And when a large fraction of the network thinks that weights don't need to be adjusted, then they don't adjust the weights and you're not doing any learning. So the learning is slow. There's some fixes to this. There's different types of functions. There's a piecewise the Rayleigh function, which is the most popular activation function. But again it suffers if the neurons are initialized poorly. It might not, this function might not. It might be zero gradient for the entire data set. Nothing that you produces input. You know, you run all your thousands of images of cats and none of them fire at all. So that's the danger here. So you have to pick these, both the, the optimization engine, the solver that you use and the activation functions carefully. You can't just play it plug and play like, like they're Legos. You have to be aware of the function. SGD Stochastic gradient descent is the vanilla optimization algorithm for gradient descent for optimizing the loss function over the gradients. And so what's visualized here is again, if you've done any numerical optimization, non linear optimization, there's the famous saddle point. That's tricky for these algorithms to deal with. What happens is it's easy for them to oscillate, get stuck in that saddle and oscillate back and forth as opposed to what they want to do, which is go down into. You get so happy that you found this low point, then you forget that there's a much lower point. And so you get stuck with the gradient. The momentum of the gradient keeps rocking you back and forth without you going to a much greater global minimum. And there's a bunch of clever ways of solving that. The atom Optimizer is one of those. But in this case, as long as the gradients don't vanish, sgd, the stochastic gradient descent, one of these algorithms will get you there. They might take a little while, but they'll get you there. And that's the main question. The question was you're dealing with a function that's not non convex and how do we ensure anything about it converging to anything that's reasonably good? The local optima, it converges to. And the answer is you can't. This isn't only a nonlinear function, it's a highly nonlinear function. The power and the beauty of neural networks is that it can represent these like arbitrarily complex functions. It's incredible, right? And you can learn those functions from data. But the reason people refer to neural networks training as art is you're trying to play with parameters that don't get stuck in these local optima for stupid reasons and for clever reasons. Yes, question. So the question. Yeah, so continue on the same thread. So the thing is, we're dealing with functions where we don't know what the global optimal is. That's sort of the crux of it. Everything we talk about, interpreting text, interpreting video, you know, even driving, like what is the optimal for driving? Never crashing. You know, it sounds easy to say that, but you actually have to formulate the world under which you define all of those things and that becomes really nonlinear objective function for which you don't know what the optimal is. It's just that's why you just keep trying and get impressed every time it gets better. It's essentially the process. And you can also compare, you can compare to human level performance. So for imagenet we can tell the difference in cats and dogs and top five categories. And 90 shoot 96% of the time, whatever accuracy. And then you get impressed when a machine can do better than that, but you don't know what the best is. These videos can be watched for hours. I won't play it until I explain this slide. So let's pause to reflect on back propagation before I go on to recurrent neural networks. Yes, question. As a practical manner, how can you tell when you're actually training a net whether you're facing the vanishing gradient problem or you need to change your optimizer or you need to reach some local minimum? The question was how do you practically know when you've hit the vanishing gradient problem? So the vanishing gradient could be so that that. The derivative being zero on the gradient happens when the activation is exploding so like really high values and really low values. The really high values is easy because they're like your network is just going crazy producing very large values. And you can fix a lot of those things by just capping the activations. The, the values being really low, resulting in a vanishing gradient are really hard to detect. This is, I mean there's a lot of research in trying to figure out how to, how to detect these things, but it's. If you're not careful, it's oftentimes you can, you can find that. And this isn't hard to do where like 40, 50% of the network of the neurons are dead. Where they're called like for Raylu, they're dead RELU nodes, they're not firing at all. How do you detect that? That's part of learning. So if they never fire, you can detect that by running it through the entire training set. I mean there's a lot of tricks, but it's, that's, that's the problem is you try to learn and then the, you look at the loss function and it's not converging to anything reasonable. It's either going all over the place or just converging very slowly. And that's an indication of something is wrong. That something could be the loss function is bad. That something could be that you've already found the optimal or that something could be the vanishing gradient. And again, that's why it's an art. But certainly at least some fraction of the neurons need to be firing, otherwise the initialization is really poorly done. Ok, so to reflect on the simplicity of backpropagation and the power of it. So this is this kind of step of back propagating the loss function through the gradients locally is the way neural networks learn. We don't have. It's really the only way that we've effectively been able to train a neural network to learn a function to adjusting the weights and biases. The huge number of weights and biases, the parameters is just through this optimization is back propagating the error where you have the supervised ground truth. So the question is whether this process is just like fitting. Adjusting the parameters of a highly nonlinear function to minimize a single objective is the way you achieve intelligence, human level intelligence. And that's something to think about. You have to think about the, for driving purposes, what is the limitation of this approach? So what's not happening? The neural network design, the architecture is not being adjusted. You're not evolving any of the edges, the layers, nothing is being evolved. And so there are other optimization approaches that I think are more interesting and inspiring than effective. So for example, this is using soft cubes to. So this, this is falling out of the field of evolutionary computation, evolutionary robotics, where you evolve the dynamics of a robot using genetic algorithms. And that's. So you can think of. So these robots are being taught to in simulation, obviously to walk and to swim, so that one is swimming. But you could. The nice thing here is the dynamics, that highly nonlinear space as well that controls the dynamics of this weird shaped robot with a lot of degrees of freedom is the same kind of thing as the neural network. And in fact people have applied genetic algorithms and colony optimization, all kinds of sort of nature inspired algorithms for optimizing the weights and the biases, but they don't seem to currently work that well. But it's kind of, it's a cool idea to be using nature type evolutionary algorithms to evolve something that's already nature inspired, which is neural networks. But something to think about is, you know, the back propagation, while really simple, is kind of dumb. And the question is whether general intelligence reasoning could be achieved with this process. All right, recurring neural networks. So on the left there, there's an input X with weights on the input U. There's a hidden state, hidden layer S with weights on the, on the edge, connecting the hidden states to each other and then more weights V on the output O. It's a really simple network. There's inputs, there is hidden states, the memory of this network and there's outputs. But the fact that there is this loop where the hidden states are connected to each other means that as opposed to producing a single input, the network takes arbitrary number of inputs. It just keeps taking x one at a time and produces a sequence of x's through time. And so depending on the duration of the sequences you're interested in, you can think of this network in its unrolled state. So you can unroll this neural network where the inputs are on the bottom. Xt minus 1, Xt Xt plus 1 and same with the outputs 0, sorry, ot minus 1 ot ot plus 1. And it becomes like a regular neural network unrolled some arbitrary number of times. The parameters again, there's weights, there's biases. It's similar to CNN's convolutional neural networks in that it's just like convolutional neural networks make certain spatial consistency assumptions. The recurrent neural networks assume temporal consistency amongst the parameters, so it shares the parameters that W, that U, that V is the same for every single Time step. So you're learning the same parameter no matter the duration of the sequence. And that allows you to look at arbitrarily long sequences without having an explosion of parameters. And this process is the same exact process that's repeated based on the different variants that we talked about before in terms of inputs and outputs, one to many, many to one, many to many. And the backpropagation process is exactly the same as for regular neural networks. Has a fancy name of back propagation through time bptt. But it's just backpropagation through an unrolled recurrent neural network where the errors are computed on the outputs, the gradients are computed back propagated and computed on the inputs, again suffering from the same exact problem now of vanishing gradients. Now the problem is that the depth of these networks can be arbitrary long, right? So if at any point the gradient hits a low number zero, it becomes that neuron becomes saturated. That gradient, let's call it saturated, that gradient drives all the earlier layers to zero. So it's easy to run to a problem where you're really ignoring majority of the sequence. This is just another Python way to the pseudo code way to look at it is you have the same W. Remember you're sharing the weights and all the parameters from time to time. So if the weights are such whh, if the weights are such that they produce, they have either they have a nagging value that results in a gradient that goes to zero that propagates through the rest. So that's the pseudocode for backpropagation, the backward pass through the RNN that WHH propagates back. And so you get these things with exploding and vanishing gradients where this, for example, a error surface for a single hidden unit rnn. So these visualizing the gradient, the value of the weight, the value of the bias and the error. So the error could be really flat or could explode. And both are going to lead to you not making the gra, either making steps that are too gradual or too big. It's the geometric interpretation. Okay, what other variants that we'll look at a little bit are there for RNNs? It doesn't have to be only one way. It can be bidirectional. So there could be edges going forward and edges going back. What that's needed for is things like filling in, missing whatever the data is filling in missing elements of that data, whether that's images or words or audio. And generally, as always is the case with neural networks, the deeper you go the better. So this is that Deep referring to the number of layers in a single temporal instance. So on the right of the slide is is we're stacking in the not in the temporal domain. Each of those layers has its own set of weights and its own sets of biases. These things are awesome, but they need a lot of data. When you add extra layers in this way. Okay, so the problem is while recurring neural networks in theory are supposed to be able to learn any kind of sequence, the reality is they're not really good at remembering what happened a while ago. The long term dependency. So here's a silly example. Let's think of a story about Bob. Bob is eating an apple. So the apple part is generated by the recurrent neural network. The recurrent neural networks can learn to generate apple because they've seen a lot of sentences with Bob and eating and they can generate the word apple for a longer sentence, like Bob likes apples, he's hungry and decided to have a snack, so now he's eating an apple. You have to maintain the state that we're talking about BOB and we're talking about apples through several discrete semantic sentences. And that kind of long term memory is not because of the van, because of different effects, but vanishing gradients. It's difficult to propagate the important stuff that happened a while ago in order to maintain that context in generating apple or classifying some concept that happened way down the line. So when people talk about recurrent neural networks these days, they're talking about LSTMs, long short term memory networks. So all the impressive results on time series, on audio, on video, all of that, that requires LSTMs. And so again, vanilla RNNs up on top of the slide is each cell is simple. There's some hidden units, there's an input and there's an output. Here we'll use 10h as the activation function. It's just another popular sigmoid type activation function. LSTMS are more complicated or they look more complicated, but in some ways they're more intuitive for us to understand. There's a bunch of gates in each cell we'll go through. Those in yellow are different neural network layers with sigma and TANH are different types of activation functions. TANH is an activation function that squishes the input to the range of negative 1 to 1. Sigmoid function squishes it between 0 and 1 and that serves different purposes. There is some point wise operations, addition, multiplication, and there is connections. So data being passed from layer to layer, shown by the arrows. There's concatenation and there's a copy operation on the output. So we copy the output of each cell is copied to the next cell and to the output. Okay, let me try to make it clarify. Clarify a little bit. So there's this conveyor belt going through inside each individual cell. And they all have. There's really three steps in the conveyor belt. The first is there is a sigmoid function that's responsible for deciding what to forget and what to ignore. So it's responsible for taking in the input, the new input xt, taking in the state of the previous, the output of the previous cell, previous time step, and deciding do I want to keep that in my memory or not, and do I want to integrate the new input into my memory or not? So this allows you to be selective about the information which you learn. So, for example, the sentence Bob and Alice are having lunch. Bob likes apples. Alice likes oranges. She's eating an orange. So the way. So Bob and Alice are having lunch, Bob likes apples. Right? Now, if you say you had a hidden state keeping track of the gender of the person we're talking about, so you might say that there's both genders in the first sentence. There's male in the second sentence, female in the third sentence. And that way when you have to generate a sentence about who's eating what, you'll keep the gender information in order to make an accurate generation of text corresponding to the proper person. So you have to forget certain things, like forget that Bob existed at that moment, and you have to forget Bob likes apples, but you have to remember that Alice likes oranges. So you have to selectively remember and forget certain things. That's LSTM in a nutshell. So you decide what to forget, decide what to remember, and decide what to output at that cell. All right, so zoom in a little bit because this is pretty cool. There is a state running through the cell, this conveyor belt, previous state, like what, you know, the gender that we're currently talking about, that that's the state that you're keeping track of, and that's running through the cell. And then there is three sigmoid layers outputting a one, you know, a number between zero and one, but it's one when you want that information to go through and zero when you don't want it to go through the conveyor belt that maintains the state. And so first sigmoid function is we decide what to forget and what to ignore. That's the first one. You take the inputs from the previous time step, the input to the network from the current time step and decide, do I want to forget? Do I want to ignore those? Then we decide which part of the state to update, what part of our memory do we update with this information and what values to insert in that update. Third step is we perform the actual update and perform the actual forgetting. So that's where you have the sigmoid function, is you just multiply it. When it's zero, it's forgetting. When it's one, that information passes through. And finally we produce an output from the cell. So if it's translation, it's producing an output in the English language where the input was in the Spanish language, and then that same output is copied to the next cell. Okay, so what can we get done with this kind of approach? We can look at machine translation. I guess what I'm trying to question, what is the representation of the state? Is it like a floating point or is it like a vector or what is it? The state is the activation multiplied, by the way. So it's the outputs of the sigmoid or the 10H activations. So there's a bunch of neurons and they're firing a number between negative 1 and 1 or between 0 and 1. And that holds a state. It's just calling its state is sort of simplifying, but the point is that there's a bunch of numbers being constantly modified by the weights and the biases, and those numbers hold the state and the modification of those numbers is controlled by the weights. And then once all that is done, the resulting output of the recurrent neural network is compared to the desired output and the error is then back propagated through the weights. Hopefully that made sense. So machine translation is one popular application and all of it is the same. All of these networks that I'll talk about, they're really similar constructs. You have some inputs, whatever language that is again, German maybe, I think everything is German. And the output, so the inputs are in one language. A set of characters that compose a word in one language, there's a state being propagated and once that sentence is over, you start, as opposed to collecting inputs, you start producing outputs and you can output in the English language. There's a ton of great work on machine translation. That's what Google is mostly using for their translator. Same thing I showed this previously, but now you know how it works. Same exact thing. LSTMs generating handwritten characters, so handwriting and arbitrary styles, so controlling the drawing where the input is text and the output is handwriting, and it's again the Same kind of network with some depth. Here. The inputs is the text. The output is the control of the writing, character level, text generation. This is the thing that told us about life, the meaning of life, literary recognition, and the tradition of ancient human reproduction. That's again the same process, input one character at a time. Where you see there is an encoding of the characters on the input layer, there's a hidden state, hidden layer that's keeping track of those activations, the outputs of the, of the activation functions. And every single time it's outputting its best prediction of the next character that follows. Now, in a lot of these applications, you want to ignore the output until the sentences, the input sentence is over, and then you start listening to the output. But the point is it just keeps generating text whether it's given input or not. So you producing input is just adding. Steering the recurrent neural network, you can answer questions about an image. So the input there. So you could almost arbitrarily stack things together. So you take an image as an input, bottom left there, put it into convolution neural network, and take the question. There's something called word embeddings. It's to broaden the representative meaning of the words. So how many books is the question? So you want to take the word embeddings and the image and produce your best estimate of the answer. So for a question of what color is the cat? It could be gray or black. There's the different LSTM flavors producing that answer. Same with counting chairs. You can give an image of a chair and ask the question, how many chairs are there? And it can produce an answer of three. So I should say that this is really hard, right? It's an arbitrary question ask of an arbitrary image. So you're both interpreting, you're doing natural language processing and you're doing computer vision all in one network. Same thing with image caption generation. You can detect the different objects in the scene, generate those words, stitch them together in syntactically correct sentences, and re rank the sentences. All of those are LSTMs. The second and the third step, the first is computer vision, detecting the objects, segmenting the image and detecting the objects in that way, you can generate a caption that says a man is sitting in a chair with a dog in his lap. Again, LSTMS for video caption generation. For video, the input at every frame is an image that goes into the lstm. The input is an image and the output is a set of characters. First you load in the video. In this case the output is on top of. So you encode the video into a representation inside the network and then you start generating words about that video. So first comes the input, the encoding stage, then the decoding stage. Take in the video, say a man is taking, talking, whatever. And because the input and the output is arbitrary, there also has to be indicators of the beginnings and the ends of a sentence. So in this case end of sentence. So you want to know when you stop in order to generate syntactically correct sentences. You want to be also be able to generate a period that indicates the end of a sentence. So you can also again, recurrent neural networks LSTMs here controlling the steering of a sliding window on an image that's used to classify what's contained in that image. So here a CNN being steered by a recurrent neural network in order to convert this image into the number that's associated with the house number. It's called visual attention. And that visual attention can be used to steer for the perception side and it can be used to steer network for the generation on the right we can generate an image as so the output of the net. It's a LSTM where the output at every time step is visual. In this way you can draw numbers. Here I mentioned this before, is taking in as input silent video sequence of images and producing audio. So this is an LSTM that takes convolutional, that has convolutional layers for every single frame, takes images as input and produces a spectrogram audio as output. The training set is a person hitting an object with a drumstick. And your task is to generate, given a silent video, generate the sound that a drumstick would make when in contact with that, with that object. Okay, there medical diagnosis that's actually. So I've listed some places where it's been really successful and pretty cool. But it's also beginning to be applied in places where can actually really help civilization, right? In medical applications. So for medical diagnosis, there is highly sparse and variable length sequence of information in the form of for example patient electronic health records. So every time you visit a doctor, there's some test being done and that information is there and you can look at it as a sequence over a period of time. And then given that data, that's the input, the output is a diagnosis, a medical diagnosis. So in this, in this case we can look at predicting diabetes, scoliosis, asthma, so on with pretty good accuracy. There's something that all of us wish we could do is stock market prediction. So you can input for example? Well first of all you can input the raw stock data right the order books and so on financial data. But you can also look at news articles from all over the web and take those as input as shown here on the x axis is time. So articles from different days LSTM once again and produce an output of your prediction binary prediction whether the stock will go up or down. And nobody's been able to really successfully do this but there is a bunch of results and trying to perform above random which is how you make money right Significantly above random on the prediction of is it going up or down. So you could buy or sell and especially when there's a in the cases when there was crashes it's easier to predict so you can predict an encroaching crash. These are shown in the table the error rates for different stocks automotive stocks. You can also generate audio this exact same process.