A brief instructional video explaining why logistic regression uses the commonly seen cost function. The speaker shows that treating the model output ŷ as P(Y=1|x) leads to a compact expression P(Y|x)=ŷ^y(1−ŷ)^{1−y}, takes logs to derive the single-example loss L(ŷ,y)=−[y log ŷ + (1−y) log(1−ŷ)], and shows that summing/logging under an IID training set gives the cost J(W,b) (with a 1/m scaling). The derivation frames minimizing J as maximum likelihood estimation for logistic regression.
Chapters
In an earlier video, I've written down a form for the cost function for logistic regression. In this optional video, I want to give you a quick justification for why we like to use that cost function for logistic regression. To quickly recap, in logistic regression, we have that the prediction Y hat is sigmoid of W transpose x plus B, where sigmoid is this familiar function. And we said that we want to interpret Y hat as the probability that Y is equal to one given x. So we want our algorithm to output Y hat as the chance that Y is equal to one for a given set of input features x. So another way to say this is that if Y is equal to 1, then the chance of Y given X is equal to Y hat. And conversely, if Y is equal to 0, then the chance that y was 0 was 1 minus y hat, right? So if Y hat was a chance that Y is equal to 1, then 1 minus y hat is a chance that Y is equal to zero. So let me take these last two equations and just copy them to the next slide. So what I'm going to do is take these two equations which basically define P of Y given x, you know, for the two cases of y equals 0 or y equals 1, and then take these two equations and summarize them into a single equation. And just to point out, Y has to be either 0 or 1, because we're in binary classification. So y equals 0, 1 are the only two possible cases, right? But I'm going to take these two equations and summarize them as follows. The Let me just write out what it looks like, then we'll explain why it looks like that. It's one minus Y hat to the power of one minus Y. So it turns out this one line summarizes the two equations on top. Um, let me explain why. So in the first case, um, suppose Y is equal to 1, right? So if y is equal to 1, then this term ends up being Y hat, because that's Y hat to the power of one. This term ends up being one minus Y hat to the power of one minus one. So that's the power of zero. But anything to the power of zero is equal to one, so that, you know, goes away. And so this equation just says P of Y given X equals Y hat when. When Y is equal to one. So, you know, that's exactly what we wanted. Now, how about the second case? What if y equals 0? If y is equal to 0, then this equation above is P of Y given X equals Y hat to the zero. But anything to the power of zero is equal to one. So, you know, that's just equal to one times one minus y hat to the power of one minus Y. So one minus Y is one minus zero. So this is just one. And so this is equal to one times one minus Y hat, which just equals to one minus Y hat. And so here we have that y equals 0p of y given x is equal to 1 minus y hat, which is exactly what we wanted above. So what we've just shown is that this equation is a, you know, correct definition for p of y given x. Now, finally, because the log function is a strictly monotonically increasing function, you know, maximizing log of p of y given x should give you a similar result as optimizing P of Y given X. And if you compute log of p of y given X, that's equal to log of Y hat to the power of y1 minus y hat to the power of 1 minus y. And so that simplifies to y log y hat plus 1 minus y times log 1 minus y hat. Right. And so this is actually negative of the loss function that we had defined previously. And there's a negative sign there because usually if you're training a learning algorithm, you want to make probabilities large, whereas in logistic regression, we're expressing this. Yeah, we want to minimize the loss function. So minimizing the loss corresponds to maximizing the log of the probability. So this is what the loss function on a single example looks like. How about the cost function? The overall cost function on the entire training set on m examples? Let's figure that out. So the probability of all the labels in the training set. Right. I'm writing this out a little bit informally, is if you assume that the training examples are drawn independently or drawn iid identically independently distributed, then the probability of the examples is the product of probabilities. So the product From I equals 1 through M of P of Yi given Xi. And so if you want to carry out maximum likelihood estimation, right, then you want to maximize the find the parameters that maximizes the chance of your observations and training set. Um, but maximizing this is the same as maximizing the log. So I'm just put logs on both sides. So log of the probability of, you know, the labels in the training set is equal to log of a product is the sum of a log. So that sum From I equals 1 through M of log p of y I given xi. And we had previously figured out on the previous slide that this is negative l of y hat I yi. And so in statistics, there's a principle called the principle of maximum likelihood estimation, which just means to choose the parameters that maximizes this thing, or in other words, that maximizes, you know, this thing, right? Negative sum From I equals 1 through M L of y hat I yi and I just move the negative sign outside the summation. So this justifies the cost we had for logistic regression, which was j of WB of this. And because we now want to minimize the cost, instead of maximize the likelihood, we've gotten rid of the minus sign. And then finally, for convenience or to make sure that our quantities are better scaled, we just add a 1 over m extra scaling factor there. But so to summarize, by minimizing this cost function J of wb, we're really carrying out maximum likelihood estimation when with the logistic regression model under the assumption that our training examples were IID or identically and independently distributed. So, thank you for watching this video, even though this is optional. I hope this gives you a sense of why we use the cost function we do for literary regression. And with that, I hope you go on to the exercises, the primary exercise, and the quiz questions for this week. Best of luck with both the quizzes and their programming exercise.
