Andrew Ng explains the loss function for logistic regression: L(y,ŷ) = -y log(ŷ) - (1-y) log(1-ŷ). When y=1, the loss penalizes predictions below 1; when y=0, it penalizes predictions above 0. The cost function averages these losses across all examples, and because it's convex, gradient descent can find the optimal parameters w and b.
Optimum. If you didn't understand the last couple of comments, don't worry about it. We'll get to it in a later video. But the intuition to take away is that this function L, called the loss function, is a function we'll need to define to measure how good our output Y hat is when the true label is Y. And squared error seems like it might be a reasonable choice, except that it makes gradient descent not work well. So in logistic regression, we will actually define a different loss function that plays a similar role as squared error, but will give us an optimization problem that is convex. And so we'll see in a later video, becomes much easier to optimize. So what we use in logistic regression is actually the following loss function, which I'm just going to write out here, is negative Y log y hat plus one minus Y log one minus Y hat. Here's some intuition for why this loss function makes sense. Keep in mind that if we're using squared error, then you want the squared error to be as small as possible. And with this logistic regression loss function will also want this to be as small as possible. To understand why this makes sense, let's look at the two cases. In the first case, let's say Y is equal to one. Then the loss function Y hat comma Y is just this first term, right? And there's negative sign. So it's negative log Y hat if Y is equal to 1. Because if y equals 1, then the second term, 1 minus y, is equal to 0. So this says if y equals 1, you want negative log Y hat to be as big as possible. So that means you want log Y hat to be large, to be as big as possible. And that means you want Y hat to be large. But because Y hat is, you know, the sigmoid function, it can never be bigger than one. So this is saying that if Y is equal to 1, you want y hat to be as big as possible, but it can't ever be bigger than one. So saying you want Y hat to be close to one as well. The other case is if Y equals zero, if Y equals zero, then this first term in the loss function is equal to zero, because Y, you know, zero. And then the second term defines the loss function, so the loss becomes negative log one minus Y hat. And so if in your learning procedure you try to make the loss function small, what this means is that you want log 1 minus y hat to be large. And because there's a negative sign there. And then through a similar piece of reasoning, you can conclude that this loss function is trying to make y hat as small as possible. And again, because y hat has to be between 0 and 1, this is saying that if y is equal to 0, then your loss function will push the parameters to make y hat as close to zero as possible. Now, there are a lot of functions with roughly this effect that if y is equal to one, it try to make y hat large and y is equal to zero, it tried to make y hat small. We just gave here in green a somewhat informal justification for this particular loss function. We'll provide an optional video later to give a more formal justification for why in logistic regression we like to use the loss function with this particular form. Finally, the loss function was defined with respect to a single training example. It measures how well you're doing on a single, single training example. I'm now going to define something called the cost function, which measures how well you're doing on the entire training set. So the cost function j, which is applied to your parameters W and B is going to be the average, really 1 over m of the sum of the loss function applied to each of the training examples in turn, where here y hat is, of course, the prediction output by your logistic regression algorithm using, you know, a particular set of parameters W and B. And so, just to expand this out, this is equal to negative 1 over m. Sum from I equals 1 through m of the definition of the loss function above. So this is yi log y hat I plus 1 minus yi log 1 minus y hat I. I guess I can put square brackets here. So the minus sign is outside everything else. So the terminology I'm going to use is that the loss function is applied to just a single training example, like so, and the cost function is the cost of your parameters. So in training your logistic regression model, we're going to try to find parameters W and B that minimize the overall function j written at the bottom. So you've just seen the setup for the logistic regression algorithm, the loss function for a training example, and the overall cost function for the parameters of your algorithm. It turns out that logistic regression can be viewed as a very, very small neural network. In the next video, we'll go over that so you can start gaining intuition about what neural networks do. So with that, let's go on to the next video about how to view logistic regression as a very small neural network.
