Andrew Ng explains why machine learning teams need one number to evaluate progress, not multiple metrics. He shows how the F1 score combines precision and recall for a cat classifier, and how averaging error rates across geographies helps pick the best model fast.
Questions this video answers
Precision 95% and recall 90%.
F1 is the harmonic mean: F1 = 2 / (1/precision + 1/recall).
U.S., China, India, and the rest of the world.
3% error on U.S. pictures.
Chapters
Whether you're tuning hyper parameters, or trying out different ideas for learning algorithms, or just trying out different options for building a machine learning system, you find that your progress will be much faster if you have a single real number evaluation metric that lets you quickly tell if the new thing you just tried is working better or worse than your last idea. So when teams are starting on the machine learning project, I often recommend that you set up a single real number evaluation metric for your problem. Let's look at an example. You've heard me say before that applied machine learning is a very empirical process where you often have an idea, code it up, run the experiment to see how it did, and then use the outcome of the experiment to refine your ideas and then keep going around this loop as you keep on improving your algorithm. So let's say for your CAD classifier you had previously built some classifier A, and by changing the hyperparameters or the training sets or some other thing, you you've now trained a new classifier B. So one reasonable way to evaluate the performance of your classifiers is to look at this precision and recall. The exact details of what's precision and recall don't matter too much for this example. But briefly, the definition precision is of the examples that your classifier recognized as cats. Um, what percentage actually are cats? So if classifier A has 95% precision, this means that when classifier A says something as a cat, there's a 95% chance it really is a cat. And recall is of all the images that really are cats, what percentage were correctly recognized by your class file? So what percentage of actual cats are correctly recognized? So if classifier A has 90% recall, this means that of all the images in say, your dev set that really are cats, classifier A accurately pulled out 90% of them. So don't worry too much about the definitions of precision and recall. But it turns out that there's often a trade off between precision and recall, and you care about both. You want that when a classifier says something is a cat, there's a high chance it really is a cat. But of all the images that are cats, you also wanted to pull a large fraction of them as cats. So it might be reasonable to try to evaluate the Class 5 in terms of its precision and its recall. The problem with using precision recall as you evaluation metric is that if classify A does does better on recall, which it does here, but classifier B does better on precision, then you're not sure which classifier is better. And if you're Trying out a lot of different ideas, a lot of different hyperparameters. You want to quickly try out not just two classifiers, but maybe a dozen classifiers and quickly pick out the quote best ones so you can keep on iterating from there. And with two evaluation metrics, it's difficult to know how to quickly pick one of the two or quickly pick one of the ten or one. So what I recommend is rather than using two numbers, precision and recall, to pick a classifier, you instead define a new evaluation metric that combines precision and recall. In the machine learning literature, the standard way to combine precision and recall is something called an F1 score. And the details of F1 score aren't too important. But informally you can think of this as the average of precision P and recall r. Formally, the F1 score is defined by this formula as 2 over 1 over precision plus 1 over recall. And in mathematics this is called the. This function is called the harmonic mean of precision P and recall R. But less formally, you can think of this as you know some way that averages precision and recall. Only instead of taking the arithmetic mean, you take the harmonic mean, which is defined via this formula. And it has some advantages in terms of trading off precision and recall. But in this example you can then see right away that class phi A has a better F1 score. And assuming F1 score is a reasonable way to combine precision recall, you can then quickly select classifier A over classifier B. So what I found for a lot of machine learning teams is that having a well defined dev set, which is how you're measuring precision recall plus a single number evaluation metric, sometimes I'll call it single row number evaluation metric, allows you to quickly tell if classifier A or classifier B is better. And therefore having a dev set plus single number evaluation metric, this tends to speed up iterating. It speeds up this iterative process of improving your machine learning algorithm. Let's look at another example. Let's say you're building a cat app for cat lovers in four major geographies. You know, the U.S. china, India and other the rest of the world. And let's say that your two classifiers achieve different errors in data from these four different geographies. So Algorand A achieves 3% error on pictures submitted by US users and so on. So it might be reasonable to keep track of how well your classifiers do in these different markers or these different geographies. But by tracking four numbers, it's very difficult to look at these numbers and quickly decide if algorithm A or algorithm B is superior. And if you're testing a lot of different classifiers, then it's just difficult to look at all these numbers and quickly pick one. So what I recommend in this example is in addition to tracking your performance in the four different geographies, to also compute the average and assuming that average performance is a reasonable single row number evaluation metric. By computing the average, you can quickly tell that it looks like algorithm C has the lowest average error. And we might then go ahead with that one if you have to pick an algorithm to keep on iterating from. So your workflow in machine learning is often you have an idea, you implement it and try it out and you want to know whether your idea helped. So what we've seen in this video is that having a single number evaluation metric can really improve your efficiency or the efficiency of your team in making those decisions that. Now we're not yet done with the discussion on how to effectively set up evaluation metrics. In the next video, I'm going to share with you how to set up optimizing as well as satisfying metrics. So let's take a look at the next video.
