How to compute the cost and gradients for logistic regression across m training examples by averaging per-example losses and gradients. The lecture walks through pseudocode with nested loops over examples and features, then explains why vectorization matters for speed on large datasets.
In a previous video you saw how to compute derivatives and implement gradient descent with respect to just one training example for logistic regression. Now we want to do it for m training examples. To get started, let's remind ourselves of the definition of the cost function j cost function WB, which you care about is this average, right? 1 over m sum from I equals 1 through m. You know the loss when your algorithm output AI on the example Y, where you know, AI is the prediction on the I've trained example, which is sigma of zi, which is equal to sigma of w transpose xi plus b. Okay, so what we show in the previous slide is for any single training example, how to compute the derivatives, um, when you have just one training example, right? So D W1, D W2 and DB with now the superscript I to denote the corresponding values you get if you were doing what we did on the previous slide, but just using the one training example, Xi yi. Excuse me, missing an I there as well. So now you notice the overall cost function is a sum or is really the average because of the 1 over m term of the individual losses. So it turns out that the derivative with respect to say W1 of the overall cost function is also going to be the average of derivatives respect to W1 of the individual loss terms. But previously we had already shown how to compute this term as say D W1, I, right? Which we, you know, on the previous slide show how to compute this on a single training example. So what you need to do is really compute these derivatives as we showed on the previous training example and average them. And this will give you the overall gradient that you can use to implement gradient descent. So I know that was a lot of details, but let's take all of this up and wrap this up into a concrete algorithms and what you should implement to get logistic regression with gradient descent working. So here's what you can do. Let's initialize j equals 0, dw 1 equals 0, dw 2 equals 0, db equals 0. And what we're going to do is use A for loop over the training set and compute the derivatives with respect to each training example and then add them up. So here's what we do For I equals 1 through M. So M is the number of training examples we compute zi equals w transpose xi plus b, the prediction AI is equal to sigma of zi and then let's add up j j plus equals yi log AI plus 1 minus yi log 1 minus AI and then square put a negative sign in front of the whole thing. And then as we saw earlier, we have dzi, right is equal to AI minus yi and dw gets plus equals x1i dzi dw2 plus equals xi2 dzi. Oh, and I'm doing this calculation assuming that you have just the two features, so the n is equal to 2. Otherwise you do this for dw1, dw2, dw3 and so on. Then db plus equals dzi and I guess that's the end of the for loop. And then finally, having done this for all M training examples, you would still need to divide by M because we're computing averages. So DW1 divide equals M, DW2 divide equals M DB divide equals M in order to compute averages. And so with all of these calculations, you've just computed the derivatives of the cost function J with respect to each of your parameters, W1, W2 and B. Just a couple of details of what we're doing. We're using DW1 and DW2 and DB as accumulators, right? So that after this computation, you know DW1 is equal to dwell derivative of your overall cost function with respect to W1, and similarly for DW2 and DB. So notice that DW1 and DW2 do not have a super strip I because we're using them in this code as accumulators to sum over the entire training set. Whereas in contrast, DZI here, this was DZ with respect to just one single training exam. So that's why that had a super strip I to refer to the one training example I that that's computed on. And so, having finished all these calculations, to implement one step gradient descent, you would implement w 1 gets updated as w 1 minus the learning rate times d w 1, w 2 gets updated as w 2 minus learning rate times dw 2, and b gives updated as B minus Learning rate times db, where dw1, dw2 and db were, you know, as computed. And finally, J here would also be a correct value for your cost function. So everything on this slide implements just one single step of gradient descent. And so you'd have to repeat everything on this slide multiple times in order to take multiple steps of gradient descent. In case these details seem too complicated again, don't worry too much about it for now. Hopefully all this will be clearer when you go and implement this in the programming assignment. But it turns out there are two weaknesses with the calculation as we've implemented it here, which is that to implement logistic regression this way, you need to write two for loops. The first for loop is this for loop over the M training examples. And the second for loop is a for loop over all the features over here. So in this example we just had two features. So n is equal to 2 or nx equals 2. But if you have more features, you end up writing DW1, DW2, and you have similar computations for DW3 and so on down to DWN. So it seems like you need to have a for loop over the features over all N features. When you're implementing deep learning algorithms, you find that having explicit for loops in your code makes your algorithm run less efficiently. And so in the deep learning era, we've moved to bigger and bigger data sets. And so being able to implement your algorithms without using explicit for loops is really important and will help you to scale to much bigger data sets. So it turns out that there are a set of techniques called vectorization techniques that allow you to get rid of these explicit for loops in your code. I think in the pre deep learning era, that is before the rise of deep learning, vectorization was a nice to have, you could sometimes do it to speed up your code, and sometimes not. But in the deep learning era, vectorization that is getting rid of for loops like this and like this has become really important because we're more and more training on very large data sets. And so you really need your code to be very efficient. So in the next few videos we'll talk about vectorization and how to implement all this without using even a single for loop. So with this, I hope you have a sense of how to implement logistic regression or gradient descent for logistic regression. Things will be clearer when you implement the programming exercise. But before actually doing the programming exercise, let's first talk about vectorization so that you can implement this whole thing, implement a single iteration of gradient descent without using any for loops.
