When to Change Dev/Test Sets (C3W1L07)

You've seen how setting up a dev set and evaluation metric is like placing a target somewhere for your team to aim at. But sometimes partway through a project, you might realize you put your target in the wrong place. In that case, you should move your target. Let's take a look at an example. Let's say you built a cat classifier to try to find lots of pictures of cats to show to your cat loving users. And the metric you've decided to use is classification error. So algorithms A and b have respectively 3% error and 5% error. So it seems like algorithm A is doing better. But let's say you try out these algorithms, you look at these algorithms, and algorithm A for some reason is letting through a lot of pornographic images. So if you ship algorithm A, the users would see more cat images because, you know, 3% error at identifying cats. But it also shows the users some pornographic images, which is totally unacceptable, but both for your company as well as for your users. And in contrast, Algorithm B has 5% error. So it misclassifies fewer images, but it doesn't have pornographic images. So from your company's point of view, as well as from a user acceptance point of view, algorithm B is actually a much better algorithm because you know it's not letting through any pornographic images. So what's happened in this example is that algorithm A is doing better on the evaluation metric, is getting 3% error, but there's actually a worse algorithm. So in this case the evaluation metric plus the dev set, it prefers algorithm A because it's saying, look, algorithm A has lower error, which is the metric you're using. But your U and your users prefer algorithm B because it's not letting through pornographic images. So when this happens, we when your evaluation metric is no longer correctly rank ordering preferences between algorithms in this case is mispredicting that algorithm A is a better algorithm, then that's a sign that you should change your evaluation metric or perhaps your development set or test set. So in this case, the misclassification error metric that you're using can be written as follows. It's 1 over M are number of examples in your development set of sum from I equals 1 to m m def number of examples in development set of indicator of whether or not the prediction on example I in development set is not equal to the actual label I, where this is to use this notation to denote the predicted value, right? So either zero and this indicator function notation counts up the number of examples on which you know this thing inside is true. So this formula just counts up the number of misclassified examples. So the problem with this evaluation metric is that it treats pornographic and non pornographic images equally. But you really want your classifier to not not mislabel pornographic images, like maybe recognize a pornographic image as a cat image and therefore show it to an unsuspecting user that's therefore very unhappy with unexpectedly seeing porn. So one way to change this evaluation metric would be if you add a weight term. Here we call this WI where WI is going to be equal to 1 if xi is non pawn and maybe 10 or maybe even larger number like 100 if xi is porn. So this way you're giving a much larger weight to examples that are pornographic so that the error term goes up much more if the algorithm this makes a mistake on classifying a pornographic image as a cat image. And in this example you're giving a 10 times bigger weight to classify pornographic images correctly. And if you want this normalization constant to still be right, technically this becomes sum over I of wi. So then this error will still be between 0 and 1. So the details of this weighting aren't important. And to actually implement this weighting, you need to actually go through your dev and test sets to label the pornographic images in your dev in test set so you can implement this weighting function. But a high level takeaway is if you find that your evaluation metric is not giving the correct rank order preference for what is actually a better algorithm, then that's the time to think about defining a new evaluation metric that you know. And this is just one possible way that you could define a new validation metric. But the goal of the evaluation metric is to accurately tell you, given two class I's, which one is better for your application. For the purpose of this video, don't worry too much about the details of how we defined a new error metric. The point is that if you're not satisfied with your own error metric, then don't keep coasting with an error metric you're unsatisfied with. Instead, try to define a new one that you think better captures your preferences in terms of what's actually a better algorithm. One thing you might notice is that so far we've only talked about how to define a metric to evaluate classifiers. That is, we've defined an evaluation metric that helps us better rank order classifiers when they are performing at varying levels in terms of screening out porn. And this is actually an example of a flow organization where I think you should take a machine learning problem and break it into distinct steps. So one knob or one step is to figure out how to define a metric that captures what you want to do. And I would worry separately about how to actually do well on this metric. So I think of the machine learning task as two distinct steps. To use the target analogy, the first step is to place the target. So define where you want to aim, and then it's a completely separate step. You so this is one knob you could tune, which is how do you place the target? And it's a completely separate problem. Think of it as a separate knob to tune in terms of how to do well at this target, or how to, you know, aim accurately or how to shoot at the target. And so defining the metric is step one, and you do something else for step two. So in terms of shooting at a target, maybe your learning algorithm is optimizing some cost function that looks like this, where you're minimizing this, you know, sum of losses on your training set. So one thing you could do is to also modify this in order to incorporate these ways, and maybe you end up changing this normalization constant as well as, I guess, one over sum of wi. Again, the details of how you define J aren't important. But the point was, with the philosophy of orthogonalization, think of placing the target as one step and aiming and shooting the target as a distinct step, which you do separately. So in other words, I encourage you to think of defining the metric as one step and then you can, and only after you've defined the metric, figure out how to do well on that metric, which might be changing the cost function J that your neural network is optimizing. Before going on, let's look at just one more example. Let's say that your two CAD classifiers, A and B have respectively, 3% error and 5% error as evaluated on your dev set, or maybe even on your test set, which are images downloaded off the Internet. So high quality, well, frame images. But maybe when you deploy your Algorand product, you find that Alvorum B actually looks like it's performing better, even though it's doing better on your dev set, and you find that you've been training off very nice, high quality images downloaded off the Internet. But when you deploy this on the mobile app, users are uploading all sorts of pictures that are much less framed. You have only part of the cat, the cat has funny facial expressions, maybe the images are much blurrier. And when you test out your algorithms, you find that algorithm B is actually doing better. So this would be another example of your metric and depth test set falling down. And the problem is that you're evaluating on a dev and test set a very nice high resolution, well frame images. But what your users really care about is the algorithm doing well on images they're uploading, which are maybe less professionally shot and blurrier and less well framed. So the guideline is if doing well on your metric and your current dev set or depth and test set distribution, if that does not correspond to doing well on the application you actually care about, then change your metric and or your dev test set. So in other words, if you discover that your dev test set has these very high quality images, but evaluating on this dev test set is not predictive of how well your app actually performs because your app needs to deal with lower quality images, then that's a good time to change your depth test set so that your data better reflects the type of data you actually need to do well on. But the overall guideline is really if your current metric and data are you evaluating on doesn't correspond to doing well and what you actually care about, then change your metric and or your depth test set to better capture what you need the algorithm to actually do well on. Having an evaluation metric and a dev set allows you to much more quickly make decisions about is algorithm A or algorithm B better? So it really speeds up how quickly you or your team can iterate. So my recommendation is even if you can't define find the perfect evaluation metric and dev set, just set something up quickly and use that to drive the speed of your team iterating. And if later down the line you find out that it wasn't a good one, you have a better idea, change it at that time. It's perfectly okay. But what I recommend against for most teams is to run for too long without an evaluation metric and dev set because that can slow down the efficiency with which your team can iterate in the middle improve your algorithm. So that's it on when to change your evaluation metric and or depth and test sets. Hope that these guidelines help you set up your whole team to have a well defined target that you can iterate efficiently toward improving performance on.