Logistic Regression Gradient Descent (C1W2L09)

Welcome back. In this video, we'll talk about how to compute derivatives for you to implement gradient descent for logistic regression. The key takeaways will be what you need to implement, that is the key equations you need in order to implement gradient descent for logistic regression. But in this video, I want to do this computation using the computation graph. I have to admit using the computation graph is a little bit of an overkill for deriving gradient descent for logistic regression. But I want to start explaining things this way to get you familiar with these ideas so that hopefully it will make a bit more sense when we talk about full fledged neural networks. But so with that, let's dive into gradient descent for logistic regression. To recap, we had set up logistic regression as follows. Your predictions Y hat is defined as.

Follows, where Z is that.

And if we focus on just one.

Example for now, then the loss with.

Respect to that one example is defined as follows.

Where A is the output of logistic regression and Y is the ground truth label.

So let's write this out as a computation graph. And for this example, let's say we.

Have only two features, X1 and X2.

So in order to compute Z, we'll.

Need to input W1, W2 and B in addition to the feature values X1 and X2. So these things in a computation graph get used to compute z, which is W1, x1 plus W2 x2 plus B. Draw a rectangular box around that, and then we compute Y hat or A equals sigma of Z. That's the next step in a computation graph. And then finally we compute L, A, Y and I won't copy the formula again.

So in logistic regression, what we want to do is to modify the parameters.

W and B in order to reduce this loss.

We've described the forward propagation steps of.

How you actually compute the loss on.

A single training example. Now let's talk about how you can.

Go backwards to compute the derivatives.

Here's a cleaned up version of the diagram. Because what we want to do is compute derivatives with respect to this loss. The first thing we want to do going backwards is to compute the derivative.

Of this loss with respect to script L over there with respect to this variable A.

And so in the code you just.

Use DA, right to denote this variable.

And it turns out that if you are familiar with calculus, you could show.

That this ends up being negative Y over A plus 1 minus Y over 1 minus A.

And the way you do that is you take the formula for the loss.

And if you're familiar with calculus, you can compute the derivative with respect to the variable lowercase a, and you get this formula.

But if you're not familiar with calculus, don't worry about it.

We'll provide the derivative formulas you need throughout this course.

So if you're an expert in calculus.

You know, I'd encourage you to look up the formula for the loss from.

The previous slide and try taking the.

Derivative with respect to a using you know, calculus. But if you don't know enough calculus to do that, don't worry about it.

Now, having computed this quantity, or da.

The derivative of your final output variable with respect to A, you can then go backwards and it turns out that you can show that dz, which this is the Python code variable name, this is going to be, you know, the derivative of the loss with respect to Z. Or for l, you could really write the loss including A and y explicitly as parameters, or not, right? Either type of notation is equally acceptable. We could show that this is equal to a minus Y.

Just a couple comments, only for those.

Of you that are experts in calculus. If you're not expert in calculus, don't worry about it. But it turns out that this DL dz, this can be expressed as DL da times DA dz and it turns out that dadz, this turns out to be A times one minus A and DL DA we had previously worked out over here.

And so if you take these two.

Quantities, you know, DL da, which is this term, together with dadz, which is this term, and just take these two things and multiply them, you could show that you the equation simplifies to a minus Y.

So that's how you derive it.

And this is really the chain rule that I briefly alluded to before.

Okay? So feel free to go through that.

Calculation yourself if you are knowledgeable about calculus. But if you aren't, all you need to know is that you can compute DZ as a minus Y. And I've already done that calculus for you.

And then the final step in backpropagation.

Is to go back to compute how much you need to change W and B. So in particular, you can show that the derivative with respect to w1, and in code we'll call this dw1, that this is equal to x1 times dz and then similarly dw2, which is how much you want to change w2 is x times dz and b, excuse me, db is equal to dz.

So if you want to do gradient descent with respect to just this one.

Example, what you would do is the following. You would use this formula to compute DZ, and then use these formulas to compute DW1, DW2 and DB. And then you perform these updates. W1 gets updated as W1 minus learning rate alpha times DW1, W2 gets updated similarly, and B gets set as B minus the learning rate times db. And so this would be one step of gradient with respect to a single example.

So you've seen how to compute derivatives and implement gradient descent for logistic regression with respect to a single training example. But to train your logistic regression model, you have not just one training example, you have an entire training set of m training examples. So in the next video, let's see how you can take these ideas and apply them to learning not just from one example, but from an entire training set.