Andrew Ng explains why accuracy fails for imbalanced datasets, like a wake-word detector that's 96.7% negative examples. He introduces precision, recall, and the F1 score (harmonic mean of both) to evaluate rare events, then shows how factories prioritize recall to catch defects they can't afford to miss.
F1 ≈ 83.4% (harmonic mean of the reported precision and recall).
96.7% negative examples and 3.3% positive examples.
An example stating 99.7% of phones have no defect (y=0) and a small fraction are y=1 (defective).
Chapters
Data sets where the ratio of positive to negative examples is very far from 5050 are called skewed data sets. Let's look at some special techniques for handling them. Let me start with a manufacturing example. If a manufacturing company makes smartphones, hopefully the vast majority of them are not defective. So if 99.7% have no defect and are labeled y equals 0 and only a small fraction is labeled y equals 1, then print 0, which is not a very impressive learning algorithm, will achieve 99.7% accuracy. Or medical diagnosis, which was the example we went through in an earlier video. If 99% of patients don't have a disease, than an algorithm that predicts no one ever has a disease will have 99% accuracy. Or speech recognition. If you're building a system for wake word detection, sometimes also called trigger word detection, these are systems that listen and see if you say a special word, like Alexa or OK Google or Hey Siri, most of the time that special wake word or trigger word is not being spoken by anyone at that moment in time. So. So when I had built wake word detection systems, the datasets were actually quite skewed. One of the datasets I use had 96.7% negative examples and 3.3% positive examples. When you have a very skewed data set like this, raw accuracy is not that useful a metric to look at because print zero can get very high accuracy. Instead, it's more useful to build something called a confusion matrix. A confusion matrix is a matrix where one axis is labeled with the actual label. So is the ground truth label y equals 0 or y equals 1 and whose other axis is labeled with the prediction. So was your learning algorithms prediction y equals 0 or y equals 1? So if you're building a confusion matrix, you fill in with each of these four cells the total number of examples, say the number of examples in your dev set in your development set that fell into each of these four buckets. Let's say that 905 examples in your development set had a ground truth label of Y equals zero and your algorithm got it right. Then you might write 900. These examples are called true negatives because they were actually negative and your algorithm predicted it was negative. Next, let's fill in the true positives, which are the examples where the actual ground truth label is 1 and the prediction is 1. Maybe there are 68 of them true positives. The false negatives are the examples where your algorithm thought it was negative, but it was wrong. The actual label is positive. So these are false negatives submit the 18 of that. And lastly, false positives are the ones where your algorithm thought it was positive, but that turned out to be false. So nine false positives. The precision of the learning algorithm, if I sum up over the columns, 905 plus 9 is 914 and 18 plus 68 is 86. So this is indeed a pretty skewed data set where out of 1,000 examples, there were 914 negative examples and just 86 positive examples. So 8.6% positive, 91.4% negative. The precision of your learning algorithm is defined as follows. It asks, of all the examples that the algorithm thought were positive examples, what fraction did it get right? So precision as is defined as true positives divided by true positives plus false positives. In other words, it looks at this row. So of all the examples that your algorithm thought had a label of 1, which is 68 +9 of them, 68 of them were actually right. So the precision is 68/68 +9, which is 88.3. In contrast, the recall asks of all the examples that were actually positive, what fraction did your algorithm get right? So recall is defined as true positives divided by true positives plus false negatives, which in this case is 68 over 68 plus negative 18, which is 79.1%. And the metrics of precision and recall are more useful than raw accuracy when it comes to evaluating the performance of learning algorithms on very skewed data sets. Let's see what happens. If your learning algorithm outputs zero all the time, it turns out it won't do very well on recall. Taking this example of where we had 914 negative examples and 86 positive examples, if the algorithm outputs zero all the time, this is what the confusion matrix would look like. 914 times it outputs 0 with a ground truth of 0, and 86 times it outputs 0 with a ground Truth of 1. So precision is true positives divided by true positives plus false positives, which in this case turns out to be 0 over 0 plus 0, which is not defined. And unless your algorithm actually output no positive labels at all, you get some of the number that hopefully isn't zero over zero. But more importantly, if you look at recall, which is true positives over true positives plus false negatives, this turns out to be zero over zero plus 86, which is zero percent. And so the print zero algorithm achieves zero percent recall, which gives you an easy way to flag that this is not detecting any useful positive examples. And the learning algorithm with Some precision, even the high value of precision is not that useful, usually if its recall is so low. So the standard metrics when I look at when comparing different models on skewed data sets are precision and recall, where looking at these numbers helps you figure out. And of all the examples that are truly positive examples, what fraction is did the algorithm manage to catch? Sometimes you have one model with a better recall and a different model with a better precision. So how do you compare two different models? There's a common way of combining precision and recall using this formula, which is called the F1 score. One intuition behind the F1 score is that you want an algorithm to do well on both precision and recall. And if it does worse on either precision or recall, that's pretty bad. And so F1 is a way of combining precision and recall that emphasizes whichever of P or R precision or recall is worse. In mathematics, this is technically called a harmonic mean between precision and recall, which is like taking an average, but placing more emphasis on whichever is the lower number. So if you compute the F1 score of these two models, it turns out it to be 83.4% using the formula below here. And model two has a very bad recall, so its F1 score is actually quite low as well. And this lets us tell maybe more clearly that Model 1 appears to be a superior model than Model 2. For your application, you may have a different weighting between precision and recall. And so F1 isn't the only way to combine precision and recall. It's just one metric that's commonly used for many applications. Let me step through one more example where precision and recall is useful. So far we've talked about a binary classification problem with skewed data sets. It turns out to also frequently be useful for multi class classification problems. If you are detecting defects in smartphones, you may want to detect scratches on them, or dents or pit marks. This is what it looks like if someone took a screwdriver and poked a cell phone or discoloration of the cell phone's LCD screen or other material. Maybe all four of these defects are actually quite rare. But you might want to develop an algorithm that can detect all four of them. One way to evaluate how your algorithm is doing on all four of these defects, each of which can be quite rare, would be to look at precision and recall of each of these four types of defects individually. In this example, the learning algorithm has 82.1% precision on finding scratches and 99.2% recall. You find in manufacturing that many factories will want high recall because you really don't want to let the phone go out that is defective. But if an algorithm has slightly lower precision, that's okay because through a human re examining the phone, they will hopefully figure out that the phone is actually okay. So many factories will emphasize high recall and by combining Precision recall using F1 as follows. This gives you a single number evaluation metric for how well your algorithm is doing on the four different types of defects and can also help you benchmark to human level performance and also prioritize what to work on next. So instead of accuracy on scratches, dense pitmarks and discolorations, using F1 score can help you to prioritize the most fruitful type of defect to try to work on. And the reason we use F1 is because maybe all four defects are very rare and so accuracy would be very high even if the algorithm was missing a lot of these defects. So I hope that these tools will help you both evaluate your algorithm as well as prioritize what to work on, both in problems with skewed data sets and for problems with multiple rare classes. Now to wrap up this section on error analysis, there's one final concept I hope to go over with you, which is performance auditing. I found for many projects this is a key step to make sure that your learning algorithm is working well enough before you push it out to a production deployment. Let's take a look at performance auditing.
