Vectorizing Logistic Regression's Gradient Computation (C1W2L14)

In the previous video, you saw how you can use vectorization to compute the predictions, the lowercase a's for an entire training set, all sort of at the same time. In this video, you see how you can use vectorization to also perform the gradient computations for all M training examples, again all sort of at the same time. And then at the end of this video, we'll put it all together and show how you can derive a very efficient implementation of logistic regression. So you will remember that for the gradient computation, what we did was we computed dz1 for the first example was going to be a 1 minus y1, and then dz2 equals a 2 minus y2, and so on. So, and so on for all M training examples. So what we're going to do is define a new variable D. Capital Z is going to be DZ1, DZ2, right? DZM. Again, all the D lowercase z variables stacked horizontally. So this would be a one by m matrix or alternatively a M dimensional row vector. Now, recall that from the previous slide, we'd already figured out how to compute capital A, which was this A1 through AM, and we had defined capital Y as Y1 through YM also, you know, stacked horizontally. So based on these definitions, maybe you can see for yourself that DZ can be computed as just A minus Y, because this is going to be equal to a one minus y one is going to be the first element, a two minus y two is going to be the second element, and so on. And so, and so this first element, a one minus y one, is exactly the definition of dz1. The second element is exactly the definition of dz2, and so on. So with just one line of code, you can compute all of this at the same time. Now, in the previous implementation, we've gotten rid of 1 for loop already, but we still had this second for loop over training examples. So we initialized DW to 0 to a vector of zeros. But then we still had to loop over training examples where we have dw x1 times dz1. For the first training example, dw x2, dz2, and so on. So we do the M times and then, you know, dw divide equals by m. And similarly for b, right db was initialized as 0, and then db plus equals dz 1, db plus equals dz 2 down to, you know, dzm and then db divide equals m. So that's what we had in the previous implementation. We'd already gotten rid of one for loop, so at least now DW is a vector and we weren't separately updating DW1, DW2 and so on. So we've gotten rid of that already. But we still had a for loop over the M examples in the training set. So let's take these operations and vectorize them. Here's what we can do for the vectorized implementation of db. What it's doing is basically summing up all of these DZs and then dividing by m. So db is basically 1 over m sum from I equals 1 through m of Dzi and where all the DZs are in that row vector. And so in Python, what you do is implement 1 over m times np.sum of dz, right? So just take this variable and call the np sum function on it and that will give you db. How about dw? I'll just write out the correct equations. You can verify it's the right thing to do. DW turns out to be 1 over M times the matrix X times DZ transpose. And to kind of see why that's the case, this is equal to 1 over m, then the matrix x is x1 through xm, right? Stacked up in columns like that. And DZ transpose is going to be DZ1 down to DZM, like so. And so if you figure out what this matrix times this vector works out to be, this turns out to be 1 over m times x1dz1 dot dot dot plus xm dzm, right? And so this is a n by one vector. And this is what you actually end up with with dw, because DW was taking these, you know, XI Dzi and adding them. And so that's what exactly this matrix vector multiplication is doing. And so again, with one line of code, you can compute dw. So the vectorized implementation of the derivative calculations is just this. You use this line to implement db, and use this line to implement dw. And notice that without a for loop over the training set, you can now compute the updates using you want to your parameters. So now let's put all together into how you would actually implement logistic regression. So this is our original highly inefficient, non vectorized implementation where so the first thing we'd done in the previous video was get rid of this volume, right? So instead of looping over DW1, DW2 and so on, we had replaced this with a vector value dw, which say this is DW xi, which is now a vector times dzi. But now we'll see that we could also get rid of not just the for loop below, but also get rid of this for loop. So here's how you do it. So using what we had from the previous slides, you would say capital zone is equal to W transpose x plus B. And the code you write is capital Z equals NP dot W transpose x plus B. And then A equals sigmoid of capital z. So you've now computed all of this and all of this for all the values of I. Next, on the previous slide, we said you would compute DZ equals capital A minus capital Y. So now you computed all of this for all the values of I. Then finally dw equals 1 over m x dz transpose and db equals 1 over m of, you know, np sum dz. So you've just done forward propagation and back propagation, really computing the predictions and computing the derivatives on all m training examples without using a for loop. And so the gradient descent update then would be, you know, W gets updated as W minus the learning rate times dw, which you just compute to the power of np B gets updated as B minus the learning rate times db. Because sometimes it's putting colons there to denote this is an assignment. But I guess I haven't been totally consistent with that notation. But with this, you have just implemented a single iteration of gradient descent for logistic regression. Now, I know I said that we should get rid of explicit for loops whenever you can, but if you want to implement multiple iterations of gradient descent, then you still need a for loop over the number of iterations. So you, if you want to have a thousand iterations of gradient descent, you might still need a for loop over the iteration number as an utmost for loop like that. And I don't think there's any way to get rid of that for loop, but I do think it's incredibly cool that you can implement at least one iteration of gradient descent without needing to use a for loop. So that's it. You now have a highly vectorized and highly efficient implementation of gradient descent for logistic regression. There's just one more detail that I want to talk about in the next video, which is in our description here. I briefly alluded to this technique called broadcasting. Broadcasting turns out to be a technique that Python and Numpy allows you to use to make certain parts of your code also much more efficient. So let's see some more details of broadcasting in the next video.