Foundations of Deep Learning (Hugo Larochelle, Twitter)

That's good. All right, cool. So, yes, I was asked to give this presentation on the foundations of deep learning, which is mostly going over basic feed forward neural networks and motivating a little bit deep learning and some of the more recent developments and some of the topics that you'll see across the next two days. So as Andrew mentioned, I have just an hour, so I'm going to go fairly quickly on a lot of these things, which I think will mostly be fine if you're familiar and enough with some machine learning and a little bit about neural nets. But if you'd like to go into some of the more specific details, you can go check out my online lectures on YouTube. It's now taught by a much younger version of myself. So just search for Hugo La Rochelle. And I am not the guy doing a bunch of skateboarding. I'm the geek teaching about neural nets. So go check those out if you want more details. So what I'll cover today is I'll start with just describing and laying out the notation on feed for all neural networks, that is models that take an input vector X that might be an image or some text and produces an output F of X. So I'll just describe forward propagation and the different types of units and the type of functions we can represent with those. And then I'll talk about how we actually train neural nets, describing things like loss functions, back propagation that allows us to get a gradient for training with stochastic gradient descent and mention a few tricks of the trade. So some of the things we do in practice to successfully, successfully trained neural nets. And then I'll end by talking about some developments that are specifically useful in the context of deep learning, that is neural networks with several hidden layers that came out at the very after the beginning of deep learning, say in 2006, that is things like dropout, batch normalization, and if I have some time, unsupervised pre training. So let's get started and just talk about assuming we have some neural network. How do they actually functions? How do they make predictions? So let me lay down the notation. So a multi layer neural feed forward neural network is a model that takes as input some vector X which I'm representing here with a different node for each of the dimensions in my input vector. So each dimension is essentially a unit in that neural network and then it eventually produces at its output layer a an output. And we'll focus on classification mostly. So you'd have multiple units here and each unit would correspond to one of the potential classes in which we would want to classify our input. So if we're identifying digits in handwritten character images and say we're focusing on digits, you'd have 10 digits. So you would have sort of zero from zero to nine. So you'd have 10 output units. And to produce an output the neural net will go through a series of hidden layers. Um, and those will be essentially the components that introduce non linearity that allows us to capture and perform very sophisticated types of classification functions. So if we have L hidden layers, the way we compute all the layers in our neural net is as follows. Uh, we first start by computing what I'm gonna call a pre activation. I'm gonna note that A and I'm go, I'm going to index the layers by K. So, so AK is just the pre activation at layer K and that is only simply going to be linear transformation of the previous layer. So I'm going to note HK as the activation and the layer. And by default I'll assume that layer zero is going to be the input. And so using that notation, the preactivation at layer K is going to correspond to taking the activation at the previous layer k minus 1, multiplying it by a matrix wk. Those are the parameters of the layer. Uh, those essentially corresponds to the connections between the units between adjacent layers. And I'm going to add a bias vector, that's another parameter in my layer. So that gives me the pre activation. And then next I'm gonna get a hidden layer activation by applying an activation function. This will introduce some non linearity in the model. So I'm going to call that function G and we'll go over a few choices that we have for common choices for deactivation function. And so I do this from layer 1 to layer L. And when it comes to the output layer, I'll also compute a preactivation by performing a linear transformation. But then I'll usually apply a different activation function depending on the problem I'm trying to solve. So having said that, let's go to some of the choices for the activation function. So some of the activation functions, you'll see one common one is the sigmoid activation function. It's this function here. It's just one divided by one plus the exponential of minus the preactivation. The shape of this function you can focus on that is this here it takes the preactivation which can vary from minus infinite to plus infinite and it squashes this between 0 and 1. So it's bounded below and above below by 0 and above by 1 so it's a function that saturates if you have very large or very large magnitude, positive or negative pre activations. Another common choice is the hyperbolic tangent or tang activation function on this picture here. So it squashes everything, but instead of being between 0 and 1, it's between minus 1 and 1 and 1. That's become quite popular in neural nets is what's known as the rectified linear activation function. Or in papers you will see the RELU unit that refers to the use of this activation function. So this one is different from the others in that it's not bounded above, but it is bounded below. And it's actually, it will output zeros exactly if the preactivation is is negative. So those are the choices of activation functions for the hidden layers and for the output layer, if we're performing classification, as I said, in our output layer, we will have as many units as there are classes in which an input could belong. And what we'd like is potentially, and what we often do is interpret each unit's activation as the probability according to the neural network that that the input belongs to the corresponding class that it's labeled Y is the corresponding class C. So C would be like the index of that unit in the output layer. So we need an activation function that produces probabilities, produces a multinomial distribution over all the different classes. And the activation function we use for that is known as the softmax activation function. It is simply as follows. You take your preactivations and you exponentiate them. So, so that's going to give us positive numbers. And then we divide each of the exponentiated preactivations by the sum of all the exponentiated preactivations. So because I'm normalizing this way, it means that all my values in my output layer are going to sum to one, and they're positive because I took the exponential. So I can interpret that as a multinomial distribution over the choice of all the C different classes. Okay, so that's what I'll use as the activation function at the output layer. And now beyond the math, in terms of conceptually and also in the way we're going to program neural networks, often what we'll do is that all these different operations, the linear transformations, the different types of activation functions, will essentially implement all of them as an object and object that take arguments, and the arguments would essentially be what are the things that are being combined to produce the next value? So for instance, we would have an object that might correspond to the computation of preactivation, which would take as argument what is the weight matrix and the bias vector for that layer and take some layer to transform. And this object would sort of compute its value by applying the linear activation, the linear transformation. And then we might have objects that correspond the specific, you know, activation functions, or like a sigmoid object or a tench object or relu object. And we just combine these objects together, chain them into what ends up being a graph, which I refer to as a flow graph that represents the computation done when you do a forward pass in your neural network up until you reach the output layer. So I mention it now because you'll see the different softwares that we presented over a weekend will essentially sort of, you know, exploit some of that representation of the computation in neural nets. And it'll also be handy for computing gradients, which I'll talk about in a few minutes. And so that's how we perform predictions in neural networks. So we get an input, we eventually reach an output layer that gives us a distribution over classes if we're performing classification. If I want to actually classify, I would just assign the class corresponding to the unit that has the highest activation. That would correspond to classifying to the class that has the highest probability according to the neural net. But then you might ask the question, okay, what kind of problems can we solve with neural networks? Or more technically, what kind of functions can we represent mapping from some input X into some arbitrary output. And so if you look at, if you go look at my videos, I try to give more intuition as to why we have this result here. But essentially, if we have a single hidden layer neural network, it's been shown that with a linear output, we can approximate any continuous function arbitrarily well, as long as we have enough hidden units. So that is, there's a value for these biases and these weights such that any continuous function, I can actually represent it as well as I want. I just need to add enough hidden units. Um, so this result applies if you use activation functions, nonlinear activation functions like sigmoid and tanh. So as I said in my video, if you want a bit more intuition as to why that would be, you can go check that out. But that's a really nice result. It means that by focusing on this family of machine learning models that are neural networks, I can pretty much potentially represent any kind of classification function. However, this result does not tell us how do we actually find the weights and the bias values such that I can represent a given function. It doesn't essentially tell us how do we train a neural network? And so that's what we'll discuss next. So let's talk about that. How do we actually, from a data set, train a neural network to perform good classification for that problem? So what we'll typically do is use a framework that's very generic in machine learning, known as empirical risk minimization or structural risk minimization, if you're using regularization. So this framework essentially transforms the problem of learning as a problem of optimizing. So what we'll do is that we'll first choose a loss function that I'm noting as L. And the loss function, it compares the output of my model, so the output layer of my neural network with the actual target. So I'm indexing with an exponent here with T to essentially as the index over all my different examples in my training set. And so my loss function will tell me is this output good or bad, given that the label is actually y? And what I'll do, I'll also define a regularizer. So theta here is you can think of it as just the concatenation of all my biases and all of my weights in my neural net. So those are all the parameters of my neural network. And the regularizer will essentially penalize certain values of these weights. So as I'll talk more specifically later on, for instance, you might want to have your weights not be too far from zero. That's a frequent intuition that we implement with regularizer. And so the optimization problem that we'll try to solve when learning is to minimize the average loss of my neural network over my training example. So summing over all training examples, I have capital T examples plus some weight here, that's known as the weight decay, some hyperparameter lambda times my regularizer. So in other words, I'm going to try to have my loss on my training set as small as possible over all the training example, and also try to satisfy my regularizer as much as possible. And so now we have this optimization problem and learning will just correspond to trying to solve this problem. So performing this, finding this argmin here for over my weights and my biases, and if I want to do this, I can just invoke some optimization procedure from the optimization community. And the one algorithm that you'll see constantly in deep learning is stochastic gradient descent. This is the optimization algorithm that we'll often use for training neural networks. So SGD stochastic gradient descent functions as follows. You first initialize all of your parameters that is finding initial values for all my weight matrices and all of my biases, and then for a certain number of epochs. So an epoch will be a full pass over all my examples. That's what I'll call an epoch. So for a certain number of full iterations over my training set, I'll draw each training example so a pair X, input X, target Y. And then I'll compute what is the gradient of my loss with respect to my parameters, all of my parameters, all my weights and all my biases. This is what this notation here. So nabla, for the gradient of the loss function, and here I'm indexing with respect to which parameter I want the gradient. So I'm going to compute what is the gradient of my loss function with respect to my parameters. And plus lambda times the gradient of my regularizer as well. And then I'm going to get a direction in which I should move my parameters. Since the gradient tells me how to increase the loss, I want to go in the opposite direction and decrease it, so my direction will be the opposite. So that's why I have a minus here. And so this delta is going to be the direction in which I'll move my parameters by, by taking a step. And the step is just a step size alpha, which is often referred to as a learning rate, times my direction, which I just add to my current values of my parameters, my biases and my weights. And that's going to give me my new value for all of my parameters. And I iterate like that, going over all pairs, XY's computing my gradient, taking a step side in the opposite direction and then doing that several times. Okay, so that's how stochastic gradient descent works. And that's essentially the learning procedure. It's represented by this procedure. So in this algorithm there are a few things we need to specify. To be able to implement it and execute it, we need a loss function, a choice for the loss function. We need a procedure that's efficient for computing the gradient of the loss with respect to my parameters. We need to choose a regularizer if we want one, and we need a way of initializing my parameters. So next what I'll do is I'll go through each of these four different things we need to choose before actually being able to execute stochastic gradient descent. So first the loss function. So as I said, we will interpret the output layer as assigning probabilities to each potential class in which I can classify my input X. Well, in this case, something that would be natural is to Try to maximize the probability of the correct class, the the actual class in which my example XT belongs to. I'd like to increase the value of the probability assigned by computed by my neural network. And so because we set up the problem in which we have a loss that we minimize, instead of maximizing the probability, what we'll actually do is minimize the negative and the actual log probability. So the log likelihood of assigning X to the correct class Y. So this is represented here. So given my output layer and the true label Y, my loss will be minus the log of the probability of Y for my neural according to my neural net. And that would be, well take my output layer and look at the unit. So index the unit corresponding to the correct class. So that's why I'm indexing by Y here. We take the log because numerically it's turns out to be more stable, we get nicer looking gradients. Uh, and sometimes in certain softwares you'll see instead of talking about the negative log likelihood or log probability, you'll see it referred as the cross entropy. Uh, and that's because you can think of this as performing a sum over all possible classes and then for each class checking, well, is this potential class the target class? So I have an indicator function that that is one if Y is equal to C. So if my iterator class C is actually equal to the real class, I'm going to multiply that by the log of the probability actually assigned to that class C and this function here. So this expression here is like a cross entropy between the empirical distribution which assigns zero probability to all the other classes, but a probability of 1 to the correct class and the actual distribution over classes that my neural net is computing, which is F of X. Okay, that's just a technical detail. You can just think about this here. I only mention it because in certain libraries it's actually mentioned as the cross entropy loss. So that's for the loss. Then we need also a procedure for computing what is the gradient of my loss with respect to all of my parameters in my neural net. So the biases and the weights, um, you can go look at my videos if you want the actual derivation of all the details for all of these different expressions. I don't have time for that. So all I'll do, and presumably a lot of you actually have seen, you know, these derivations. If you haven't, just go check out the videos. In any case, I'm going to go through what the algorithm is. I'm going to highlight some of the key points that will come up later in understanding our actually backpropagation functions. So the basic idea is that we'll compute gradients by exploiting the chain rule and we'll go from the top layer all the way to the bottom, computing gradients for layers that are closer and closer to the input as we go, and exploiting the chain rule to exploit or reuse previous computations we made at upper layers to compute the gradients at the layers below. So we usually start by computing what is the gradient at the output layer. So what's the gradient of my loss with respect to my output layer? It actually it's more convenient to compute the loss with respect to the pre activation. It's actually a very simple expression. So that's why I have the gradient of this vector Al +1. That's the preactivation at the very last layer of the loss function, which is minus the log F of X Y. And it turns out this gradient is super simple. It's minus E of Y. So that's the one hot vector for class Y. So what this means is E of Y is just a vector filled with a bunch of zeros and then the one at the correct class. So if Y was the fourth class, then in this case it would be this vector where I have a one at the fourth dimension. So E of Y is just a vector. It's, we call it the one hot vector full of zeros and the single one at the position corresponding to the correct class. So what this part of the gradient is essentially saying is that I'm going to increase, I want to increase the probability of the correct class. I want to increase the preactivation which will increase the probability of the correct class. And I'm going to subtract what is the current probabilities assigned by my neural net to all of the classes. So F of X, that's my output layer and that's the current beliefs of the neural net as to in which class. What's the probability of assigning the input to each class? So what this is doing is essentially trying to decrease the probability of everything and specifically decrease it as much as I. The neural net currently believes that the input belongs to it. And so if you think about the subtraction of these two things, well, for the class that's the correct class, I'm going to have one minus some number between zero and one because it's a probability. So that's going to be positive. So I'm going to increase the probability of the correct class and for everything else it's gonna be zero minus a positive number, so it's going to be negative. So I'm actually going to decrease the probability of everything else. So intuitively it makes sense. This gradient has the right behavior. And I'm gonna take that preactivation gradient, I'm going to propagate it from the top to the bottom and essentially iterating from the last layer, which is the output layer L +1, all the way down to the first layer. And as I'm going down, I'm going to compute the gradients with respect to my parameters and then compute what's the gradient for the preactivation at the layer below and then iterate like that. So at each iteration of that loop I take what is the current gradient of the loss function with respect to the preactivation at the current layer and I can compute the gradient of the loss function of with respect to my weight matrix. So not doing the derivation here, it, it's actually simply this vector. So my, in my notation I assume that all the vectors are column vectors. So this preactivation gradient vector and I multiply it by the transpose of the activations. So the value of the layer right below the layer K minus 1. So because I take the transpose, that's a multiplication like this and you can see if I do the outer product essentially between these two vectors, I'm going to get a matrix of the same size as my weight matrix. So it all checks out. That makes sense. Turns out that the gradient of the loss with respect to the bias is exactly the gradient of the loss with respect to the preactivation. So that's very simple. So that gives me now my gradients for my parameters and I need to compute ok, what is going to be the gradient of the preactivations at the layer below? Well, first I'm going to get the gradient of the loss function with respect to the activation at the layer below. Well that's just taking my preactivation gradient vector and multiplying it by for some reason does it show here but and multiplying by the transpose of my weight matrix. Super simple operation. Just a linear transformation of my gradients at layer Kit, linearly transformed to get my gradients of the activation at the layer K minus 1 and then to get the gradients of the preactivation. So before the activation function I'm going to, I'm gonna take this gradient here which is the gradient of the activation function of the layer K minus 1 and then I apply the gradient corresponding to the partial derivative of my nonlinear activation function. So this here, this refers to an element wise product. So I'm taking these two vectors, this vector here and this vector here. I'm gonna do an element wise product between the two. And this vector here is just the partial derivative of the activation function for each unit individually that I've put together into a vector. Okay, this is what this corresponds to. Now the key things to know this is first that this pass, computing all the gradients and doing all these iterations is actually fairly cheap. It's complexity is essentially the same as the one that's doing forward pass. So all I'm doing are linear transformations multiplying by matrices, in this case the transpose of my weight matrix. And then I'm also doing this sort of nonlinear operation where I'm multiplying by the gradient of the activation function. So that's the first thing to notice. And the second thing to notice is that here I'm doing this element wise product. So if any of these terms here for a unit is very close to zero, then the preactivation gradient is going to be zero for the next layer. And I highlight this point because essentially whenever, that's something to think about a lot when you're training neural nets, whenever this gradient here, these partial derivatives come close to zero, that it means the gradient will not propagate well to the next layer, which means that you're not gonna get a good gradient to update your parameters. Now when does that happen? When will you see these terms here being close to zero? Well, that's gonna be when the partial derivatives of these nonlinear activation functions are close to zero or zero. So we can look at the partial derivative, say of the sigmoid function. It turns out it's super easy to compute. It's just the sigmoid itself times one minus the sigmoid itself. So that means that whenever the activation of the unit for a sigmoid unit is close to one or close to zero, I essentially get a partial derivative that's close to zero. You can kind of see it here. The slope here is essentially flat, and the slope here is flat. That's the, the value of the partial derivative. So in other words, if my preactivations are very negative or very positive, so if my unit is very saturated, then gradients will have a hard time propagating to the next layer. That's the key insight here. Same thing for the tan function. So it turns out the partial derivative is also easy to compute. You just take the Tange value square it and, and going to subtract it to one. And indeed, if it's close to minus one or close to one, you can see that the slope is flat. So again, if the unit is saturating, gradients will propagate, have a hard time propagating to the next layers. And for the ReLU, the rectified linear activation function, the gradient is even simpler. You just check whether the preactivation is greater than zero. If it is, the partial derivative is one. If it's not, it's zero. So actually either going to multiply by one or zero, you essentially get a binary mask when you're performing the propagation through the relu. And you can see it, the slope here is flat and otherwise you have a linear function. So actually here the shrinking of the gradient towards zero is even harder. It's exactly multiplying by zero. If you have a unit that's saturating below and beyond all the math in terms of actually using those in practice during the weekend, you'll see three different libraries that essentially allows you to compute these gradients for you. You actually usually don't write down backprop, you just use all of these modules that you've implemented. And it turns out there's a way of automatically differentiating your loss function and getting gradients for free in terms of effort, in terms of programming effort with respect to your parameters. So conceptually, the way you do this, and you'll see essentially three different libraries doing it.