Andrew Ng explains how to implement gradient descent for a neural network with a single hidden layer. The video gives the forward-propagation equations (Z1=W1X+B1, A1=g(Z1), Z2=W2A1+B2, A2=sigmoid(Z2) for binary classification) and the backpropagation derivatives (DZ2=A2−Y, DW2, DB2, DZ1, DW1, DB1) with practical numpy implementation notes (np.sum axis, keepdims) and a reminder to randomly initialize parameters.
All right, I think this will be an exciting video. In this video you see how to implement gradient descent for your neural network with one hidden layer. In this video I'm gonna just give you the equations you need to implement in order to get back propagation or to get gradient descent working. And then in the video after this one, I'll give some more intuition about why these particular equations are the accurate equations or the correct equations for computing the gradients you need for your neural network. So your neural network with a single hidden layer for now will have parameters W1, B1, W2 and B2. And so as a reminder, if you have NX or alternatively N0 input features and N1 hidden units and N2 output units. In our example so far I've only had n2 equals 1, then the matrix w1 will be n1 by n0. B1 will be an n1 dimensional vector. So we can write that as a n1 by 1 dimensional matrix, really a column vector. The dimensions of w2 will be n2 by n1 and the dimension of b2 will be n2 by 1. Right? Where again, so far we've only seen examples where N2 is equal to 1, where you have just one single hidden unit. So you also have a cost function for your neural network. And for now I'm just going to assume that you're doing binary classification. So in that case, the cost of your parameters as follows, it's going to be 1 over M of the average of that loss function. And so l here is the loss when your neural Network predicts y hat, right? This is really a 2 when the ground truth label is equal to y. And if you're doing binary classification, the loss function can be exactly what you use for logistic regression earlier. So to train the parameters of your algorithm, you need to perform gradient descent. When training a neural network, it's important to initialize the parameters randomly rather than to all zeros. We'll see later why that's the case. But after initializing the parameter to something, each loop of gradient descent would compute the predictions. So you basically compute your Y hat, I for I equals 1 through M, say, and then you need to compute the derivative. So you need to compute DW1, and that's basically the derivative of the cost function with respect to the parameter W1. You need to compute another variable, which I'm gonna call DB1, which is the derivative or the slope of your cost function with respect to the variable B1, and so on. Similarly, for the other parameters W2 and B2 and and then finally, the gradient descent update would be to update W1 as W1 minus alpha. The learning rate times D W1, B1 gets updated as B1 minus, the learning rate times DB1. And similarly for W2 and B2. And sometimes I use colon equals and sometimes equals as either, either notation works fine. And so this would be one iteration of gradient descent. And then you repeat this some number of times until your parameters look like they're converging. So in previous videos we talked about how to compute the predictions, how to compute the outputs, and we saw how to do that in a vectorized way as well. So the key is to know how to compute these partial derivative terms, the DW1, DB1, as well as the derivatives DW and DB2. So what I'd like to do is just give you the equations you need in order to compute these derivatives. So, and I'll defer to the next video, which is an optional video to go greater into depth about how we came up with those formulas. So let me just summarize again the equations for forward propagation. So you have z1 equals w1x plus b1, and then a1 equals the activation function in that layer applied element wise to Z1. And then Z2 equals W2A1 plus B2. And then finally, this is all vectorized across your training set, right? A2 is equal to G2 of Z2. Again, for now, if we assume you're doing binary classification, then this activation function really should be the sigmoid function. So I'm just fill that in here. So that's the forward propagation or the left to right forward computation for your neural network. Next, let's compute the derivatives. So this is the back propagation step, gonna compute DZ2 equals A2 minus the ground truth y. And just as a reminder, all this is vectorized across examples. So the matrix Y is this one by m matrix that lists all of your m examples stacked horizontally. Then it turns out DW2 is equal to this. In fact, these first three equations are very similar to gradient descent for logistic regression. Comma x is equals 1 comma, keep dims equals true. And just a little detail, this NP sum is a Python numpy command for summing across one dimension of a matrix, in this case summing horizontally. And what KeepDims does is it prevents Python from outputting one of those funny rank one arrays, right where the dimensions was n comma. So by having keepdims equals true, this ensures that python outputs for db2 a vector that is n by 1. In fact, technically this will be, I guess, n 2 by 1. In this case it's just a one by one number. So maybe it doesn't matter, but later on we'll see when it really matters. So so far what we've done is very similar to logistic regression. But now as you compu continue to run back propagation, you would compute this so dz2 times g1 prime of z1. So this quantity g1 prime is the derivative of whatever was the activation function you use for the hidden layer and for the output layer. I assume that you're doing binary classification with the sigmoid function. So that's already baked into that formula for D. And this times is an element wise product. So this here is going to be an N1 by M matrix. And this here, this element wise derivative thing is also going to be an N1 by N matrix. And so this times there is an element wise product of two matrices. Then finally DW1 is equal to that and DB1 is equal to this NP sum DZ1 axis equals 1, keep dimms equals true. So whereas previously the keep dims may be matted less if N2 is equal to 1. So this is just a one by one thing, it's just a real number here, DB1 will be a N1 by one vector. And so you want Python, you want NP sum to output something of this dimension rather than a funny rank one array of that dimension, which could end up messing up some of your later calculations. Um, the other way would be to not have to keep them parameters, but to explicitly call, you know, reshape, to reshape the output of NP sum into this dimension, which you would like DB to have. So that was forward propagation in I guess, four equations and back propagation in, I guess, six equations. I know, I just wrote down these equations. But in the next optional video, let's go over some intuitions for how the six equations for the backpropagation algorithm were derived. Please feel free to watch that or not, but either way, if you implement these algorithms, you will have a correct implementation of forward prop and back prop, um, and you'll be able to compute the derivatives you need in order to apply gradient descent to learn the parameters of your neural network. It is possible to implement this algorithm and get it to work without deeply understanding the calculus. A lot of successful deep learning practitioners do so. Um, but, but if you want, you can also watch the next video just to get a bit more intuition about the derivation of these equations.
