Backpropagation Intuition (C1W3L10)

In the last video, you saw the equations for backpropagation. In this video, let's go over some intuition using the computation graph for how those equations were derived. This video is completely optional, so feel free to watch it or not. You should be able to do the homework either way. So recall that when we talked about logistic regression, we had this forward pass where we compute Z, then A and then the loss. And then to take derivatives, we had this backward pass where we could first compute DA and then go on to compute DZ and then go on to compute DW and db. So the definition for the loss was L of a comma Y equals negative y log A minus 1 minus Y times log 1 minus A. So if you're familiar with calculus and you take the derivative of this with respect to A, that would give you the formula for da. So DA is equal to that. And if you actually figure out the calculus, you could show that this is y over a negative y over a plus one minus Y over one minus A. Let me just kind of derive that from calculus by taking derivatives of this. It turns out when you take another step backwards to compute dz, we then worked out that DZ is equal to A minus Y. I didn't explain why previously, but it turns out that from the chain rule of calculus, DZ is equal to DA times G prime of Z, where here, G of Z equals sigmoid of Z is our activation function for this output unit in logistic regression, right? So just remember, this is still logistic regression where we have x1, x2, x3 and then just one sigmoid unit, and then that gives us a, gives us y hat. So here the activation function was a sigmoid function. And as an aside, only for those of you familiar with the chain rule of calculus, the reason for this is because A is equal to sigmoid of Z and so partial of L with respect to Z is equal to partial of L with respect to A times DA dz. But since A is equal to sigmoid of Z, this is equal to D DZ G of Z, right? Which is equal to G prime of Z. So that's why this expression which is DZ in our code is equal to this expression which is DA in our code times G prime of z. And so this expression is just that. So that last derivation would have made sense only if you're familiar with calculus and specifically the chamber from calculus. But if not, don't worry about it. I'll try to explain the intuition wherever is needed. And then finally Having computed DZ for Leucci's regression, we would compute dw, which it turns out was DZ times X and DB was, which is just DZ when you have a single training example. So that was logistic regression. So what we're going to do when computing backpropagation for a neural network is a calculation a lot like this, but only we'll do it twice, because now we have not X going to an output unit, but X going to a hidden layer and then going to an output unit. And so instead of, instead of this computation being so the one step as we have here, we'll have, you know, two steps here, right? In, in, in this kind of a neural network with two layers. So in this two layer neural network, that is with an input layer, a hidden layer and an output layer, remember the steps of a computation. First you compute z1 using this equation and then compute a1, and then you compute z2. And notice z2 also depends on the parameters w2 and b2. And then based on z2, you compute a2 and finally that gives you the loss. So what backpropagation does is it will go backward to compute DA2 and then DZ2 and then go back to compute DW2 and DB2 go back to compute DA1, DZ1 and so on. Okay? We don't need to take derivatives with respect to the input x, since the input X for supervised learning is fixed. So we're not trying to optimize X, so we won't bother to take derivatives, at least for supervised learning with respect to X. And so I'm going to Skip explicitly computing DA2. If you want, you can actually compute DA2 and then use that to compute DZ2. But in practice you could collapse both of these steps into one, into one step. So you end up that DZ2 is equal to a 2 minus Y, same as before. And you have also I'm going to write DW2 and DB2 down here. Below you have that DW2 is equal to DZ2 times A1 transpose and DB2 equals DZ2. So this step is quite similar to for logistic regression where we had that dw was equal to dz times x, except that now a1 plays the row of X and there's an extra transpose there because the relationship between the capital matrix W and our individual parameters W does a transpose there, right? Because W is equal to a row vector. In the case of logistic regression with a single output DW is like that, whereas W Here was a column vector. So that's why there's an extra transpose for A one, whereas we didn't for X here for logistic regression. So this completes half of backpropagation. And then again you can compute the DA1 if you wish. Although in practice the computation for DA1 and DZ1 are usually collapsed into one step. And so what you'd actually implement is that DZ1 is equal to W2 transpose times DZ2 and then times an element wise product of G1 prime of Z1. And just to do a check on the dimensions, right? If you have a neural network that looks like this, right, outputs Y. So if you have n0, nx equals n0 input features, n1 hidden units and n2 so far nice, and n2 in our case just one output unit, then the matrix w2 is n2 by n1 dimensional z2, and therefore dz2 are going to be n2 by one dimensional. It's really going to be a one by one when we're doing binary classification and z1 and therefore also dz1 are going to be n1 by one dimensional. Note that for any variable foo, foo and D foo always have the same dimension. So that's why W&DW always have the same dimension. And similarly for B&DB and Z and DZ and so on. So to make sure that the dimensions of this all match up, we have that DZ1 is equal to W2 transpose times DZ2 and then this is an element wise product times G1 prime of Z1. So matching the dimensions from above, this is going to be N1 by 1 is equal to W2 transpose with transpose of this. So this is going to be N1 by N2 dimensional, DZ2 is going to be N2 by one dimensional. And then this, this is the same dimension as z1. So this is also n1 by one dimensional. So elementalized product. So the dimensions do make sense, right? N1 by one dimensional vector can be obtained by N1 by N2 dimensional matrix times N2 by N1. Because the product of these two things gives you an N1 by one dimensional matrix. And so this becomes the element wise product of 2n 1 by one dimensional vectors. And so the dimensions do match up. One tip when implementing a backprop, if you just make sure that the dimensions of your matrices match up. So if you think through what are the dimensions of your various matrices including W1, W2, Z1, Z2, A1, A2 and so on and just make sure that the dimensions of these matrix operations may match up sometimes. That will already eliminate quite a lot of bugs and backprop. All right, so this gives us DZ1. And then finally, just to wrap up DW1 and DB1, we should write them here, I guess. But since I'm running out of space, I'll write them on the right of the slide. DW1 and DB1 are given by the following formulas. The this is going to be equal to dz 1 times x transpose. And this is going to be equal to dz. And you might notice a similarity between these equations and these equations, which is really no coincidence because X plays the row of a zero. So X transpose is a zero transpose. So those equations are actually very similar. All right, so that gives a sense for how backpropagation derived. And we have six key equations here for DZ2, D, W2, DB2, DZ1, DW1, and DB1. So let me just take these six equations and copy them over to the next slide. Here they are. And so far we've derived back propagation for if you are training on a single training example at a time. But it should come as no surprise that rather than working on a single example at a time, we would like to vectorize across different training examples. So we remember that for forward propagation, when we're operating on one example at a time, we had equations like this as well as say A1 equals P1 of Z1. And in order to vectorize, we took, say, the Z's and stacked them up in columns like this, Z1M, and called this capital Z. And then we found that by stacking things up in columns and defining the capital uppercase version of these, we then just had Z1 equals W1x plus B and A1 equals G1 of Z1, right? And we defined the notation very carefully in this course to make sure that stacking examples into different columns of a matrix makes all this work out. So it turns out that if you go through the math carefully, the same trick also works for backpropagation. So the vectorized equations are as follows. First, if you take these DZs for different training examples and stack them as the different columns of a matrix, and same for this and same for this, then this is the vectorized implementation. And then here's the definition for or here's how you can compute DW2, there is this extra one over M, because, you know the cost function j is this 1/m of sum from I equals 1 through m of the losses. And so when computing derivatives, we have that extra one over m term, just as we did when we were computing the weight updates for logistic regression. Um, and then that's the update you get for db2 again, sum of the dzs and then with a 1 over m. And then dz1 is computed as follows. Um, once again, this is an element wise product only, whereas previously this was. We saw on the previous slide that this was an N1 product by one dimensional vector. Now this is a N1 by M dimensional matrix and both of These are also N1 by M dimensional. And so that's why that asterisk is a element wise product. Right? And then finally, the remaining two updates perhaps shouldn't look too surprising. So I hope that gives you some intuition for how the back propagation algorithm is derived in all of machine learning. I think the derivation of the back propagation algorithm is actually one of the most complicated pieces of math I've seen, and it requires knowing both linear algebra as well as the derivative matrices to really derive it from scratch from first principles. If you are an expert in matrices matrix calculus using this process, you might be able to derive the algorithm yourself. But I think that there are actually plenty of deep learning practitioners that have seen the derivation at about the level you've seen in this video and are already able to have all the right intuitions and be able to implement this algorithm very effectively. So if you are an expert in calculus, do see if you can derive the whole thing from scratch. It is one of the very hardest pieces of math, one of the very hardest derivations that I've seen in all of machine learning. But either way, if you implement this, you know this will work, and I think you have enough intuitions to tune it and get it to work. So with that, there's just one last detail I want to share with you before you implement your neural network, which is how to initialize the weights of your neural network. And it turns out that initializing your parameters not to zero, but randomly turns out to be very important for training your neural network. In the next video, you'll see why.