Andrew Ng derives the logistic regression loss function, Loss(F,Y) = -Y log(F) - (1-Y) log(1-F), and shows why it works whether Y is 0 or 1. The cost function J(θ) averages these losses across training examples and connects to maximum likelihood estimation.
In the last video, you saw the loss function and the cost function for logistic regression. In this video, you'll see a slightly simpler way to write out the loss and cost functions so that the implementation can be a bit simpler when we get to gradient descent for fitting the parameters of a logistic regression model. Let's take a look. As a reminder, here's the loss function that we had defined in the previous video for logistic regression. Now, because we're still working on a binary classification problem, y is either 0 or 1. Because y is either 0 or 1 and cannot take on any value other than 0 or 1, we'll be able to come up with a simpler way to write this loss function. You can write the loss function as follows. Given a prediction F and the target label Y, the loss equals negative Y times log of F minus 1 minus Y times log of 1 minus F. And it turns out this equation, which we just wrote in one line, is completely equivalent to this more complex formula up here. Let's see why this is the case. Now, remember, Y can only take on the values of either 1 or 0. In the first case, let's say y equals 1. This first y over here is 1, and this 1 minus y is 1 minus 1, which is therefore equal to 0. And so the loss becomes negative 1 times log of f of x minus 0 times a bunch of stuff that becomes 0 and goes away. And so when Y is equal to one, the loss is indeed the first term on top, negative log of F of X. Let's look at the second case when Y is equal to 0. In this case, this y here is equal to 0. So this first term goes away, and the second term is 1 minus 0 times that logarithmic term. So the loss becomes this negative 1 times log of 1 minus f of x. And that's just equal to this second term up here. And so in the case of Y equals zero, we also get back the original loss function as defined above. So what you see is that Whether Y is 1 or 0, this single expression here is equivalent to the more complex expression up here, which is why this gives us a simpler way to write the loss with just one equation without separating out Y2 cases like we did on top. Using this simplified loss function, let's go back and write out the cost function for logistic regression. So here again is the simplified loss function. And recall that the cost J is just the average loss average across the entire training set of m examples. So it's 1 over m times the sum of the Loss From I equals 1 to M. Now if you plug in the definition for the simplified loss from above, then it looks like this one over m times the sum of this term above. And if you bring the negative signs and move them outside, then you end up with this expression over here. And this is the cost function, the cost function that pretty much everyone uses to train logistic regression. Now you might be wondering, why did we choose this particular function when there could be tons of other cost functions we could have chosen. Although we won't have time to go into great detail on this in this class, I'd just like to mention that this particular cost function is derived from statistics using a statistical principle called maximum likelihood estimation, which is an idea from statistics on how to efficiently find parameters for different models. And this cost function has the nice property that it is convex. But don't worry about learning the details of maximum likelihood. It's just a deeper rationale and justification behind this particular cost function. The upcoming optional lab will show you how the logistic cost function is implemented in code. I recommend taking a look at it because you implement this later in the practice lab at the end of the week. This upcoming optional lab also shows you how two different choices of the parameters will lead to different cost calculations. So you can see in the plot that the better fitting blue decision boundary has a lower cost relative to the magenta decision boundary. So with the simplified cost function, we're now ready to jump into applying gradient descent to logistic regression. Let's go see that in the next video.
