Learn why accuracy fails for imbalanced data like spam detection, and how log loss penalizes confident wrong predictions instead. The tutorial walks through the formula with examples: a confident wrong prediction scores 2.3, while an uncertain correct one scores 0.69, then shows Python code with clipping to avoid infinite values.
Chapters
The next step is to decide how we measure if our algorithm works. Choosing how to evaluate your machine learning model is one of the most important decisions an analyst makes. The decision balances the real world use of the algorithm, the mathematical properties of the evaluation function, and the interpretability of the measure. Often we hear the question, how accurate is your model? Accuracy is a simple measure that tells us what percentage of rows we got right. However, sometimes accuracy doesn't tell the whole story. Consider the case of identifying spam emails. Let's say that only 1% of the emails I receive are spam. The other 99% are legitimate emails. I can build a classifier that's 99% accurate just just by assuming every message is legitimate and never marking a message as spam. But this model isn't useful at all because every message, even the spam, ends up in my inbox. The metric we use for this problem is called log loss. Log loss is what is generally called a loss function and it is a measure of error. We want our error to be as small as possible, which is the opposite of a metric like accuracy, where we want to maximize the value. Let's look at how log loss is calculated. It takes the actual value 1 or 0, and it takes our prediction, which is a probability between 0 and 1. The Greek letter sigma, which looks like an uppercase e below indicates that we're taking the sum of log loss measures for each row in the data set. We then multiply this sum by negative 1/ N the number of rows to get a single value for log loss. We will unpack this math a little more by looking at an example. Consider the case where the true label is 0, but we predict confidently that the label is 1. In this case, because y is 0, the first term becomes 0. This means that the log loss is calculated by 1 minus y times log of 1 minus p. This simplifies to log 1 minus 0.9 or log 0.1, which is 2.3. Now consider the case that the correct label is 1, but our model is not sure and our prediction is right in the middle. At a value of 0.5, our log loss is 0.69. Since we are trying to minimize log loss, we can see that it is better to be less confident than it is to be confident and wrong. Here is an implementation of log loss. The most important detail is is the clip function, which sets a maximum and a minimum value for the elements in an array. Since log of 0 is negative infinity, we want to offset our predictions ever so slightly from being exactly one or exactly zero so that our score remains a real number. In this example, we use the EPS variable to be 0.00013 zeros and then a one which is close enough to zero to not affect our overall scores. After adjusting the prediction slightly with clip, we calculate log loss using the formula. If we call this function on the examples we looked at earlier, we can see that the confident and wrong item returns the expected value of 2.3 and the prediction that is right in the middle returns 0.69. We have implemented it here to demonstrate how to take a mathematical equation and turn it into a function to use for evaluation, just like you may need to do if you were participating in a machine learning competition. Now let's develop some intuition for how the log loss metric performs with a few examples.
