Andrew Ng explains how to handle multiple metrics by picking one to maximize and setting thresholds for the rest. Example: maximize accuracy while keeping runtime under 100 ms, or optimize wake-word detection accuracy while allowing no more than one false positive per 24 hours.
It's not always easy to combine all the things you care about into a single real number evaluation metric. In those cases, I found it sometimes useful to set up satisficing as well as optimizing metrics. Let me show you what I mean. Let's say that you've decided you care about the classification accuracy of your CAT classifier. This could have been F1 score or some other measure of accuracy. But let's say that in addition to accuracy, you also care about the running times. So how long it takes to classify an image and classifier, A takes 80 milliseconds, B takes 95 milliseconds, and C takes 1500 milliseconds. That's 1.5 seconds to classify an image. So one thing you could do is combine accuracy and running time into an overall evaluation metric into the cost, such as maybe the overall cost is accuracy minus 0.5 times running time. But maybe it seems a bit artificial to combine accuracy and running time using a formula like this, like a linear weighted sum of these two things. So here's something else you could do instead, which is that you might want to choose a classifier that maximizes accuracy, but subject to that, the running time, that is the time it takes to classify an image that that has to be less than or equal to 100 milliseconds. So in this case, we would say that accuracy is an optimizing metric because you want to maximize accuracy. You want to do as well as possible on accuracy. But that running time is what we call a satisficing metric, meaning that it just has to be good enough. It just needs to be less than 100 milliseconds. And beyond that, you don't really care, or at least you don't care that much. So this would be a pretty reasonable way to trade off or to put together accuracy as well as running time. And it may be the case that so long as the running time is less than 100 milliseconds, your users won't care that much whether it's 100 milliseconds or milliseconds or even faster. And by defining optimizing as well as satisficing metrics, this gives you a clear way to pick the best classifier, which in this case would be classifier B. Because of all the ones with a running time better than 100 milliseconds, it has the best accuracy. So more generally, if you have N metrics that you care about, is sometimes reasonable to pick one of them to be optimizing. So you want to do as well as is possible on that one, and then n minus one to be satisficing, meaning that so long as they reach some threshold, such as running times faster than 100 milliseconds, but as long as they've reached some threshold, you don't care how much better it is in that threshold, but they have to reach that threshold. Here's another example. Let's say you're building a system to detect wake words, also called trigger words. So this refers to the voice control devices like the Amazon Echo which you wake up by saying Alexa, or some Google devices which you wake up by saying okay, Google. Or some Apple devices which wake up by saying hey Siri. Or some baidu devices which you wake up by saying you, you know, ni hao baidu. Or I guess you only write the Chinese, that's ni hao baidu, right? So these are the wake words you use to tell one of these voice control devices to wake up and listen to something you want to say. And by the way, these are the Chinese characters for ni hao baidu. So you might care about the accuracy of your trigger word detection system. So when someone says one of these trigger words, how likely are you to actually wake up your device? And you might also care about the number of false positives. So when no one actually said this trigger word, how often does it randomly wake up? So in this case, maybe one reasonable way of combining these two evaluation metrics might be to maximize accuracy. So when someone says one of the trigger words, maximize the chance that your device wakes up. And subject to that, you have at most one false positive, you know, every 24 hours of operation, right? So that your device randomly wakes up only once per day on average, when no one is actually talking to it. So in this case, accuracy is the optimizing metric, and the number of false positives every 24 hours is the satisficing metric where you'd be satisfied so long as there's at most one false positive every 24 hours. To summarize, if there are multiple things you care about, but I said that one as the optimizing metric that you want to do as well as possible on, and one or more as satisficing metrics, will you be satisfied so long as it does better than some threshold, you, you can now have a automatic way, almost automatic way of quickly looking at multiple classifiers and picking the quote best one. Now, these evaluation metrics must be evaluated or calculated on a training set or a development set, or maybe on a test set. So One of the things you also need to do is set up training, dev or development as well as test sets. In the next video, I want to share with you some guidelines for how to set up training, dev and test sets. So let's go on to the next video.
