Avoidable Bias (C3W1L09)

We've talked about how you want your learning algorithm to do well in a training set, but sometimes you don't actually want to do too well. And knowing what human level performance is can tell you exactly how well, but not too well, you want your algorithm to do on a training set. Let me show you what I mean. We've used CAT classification a lot and given a picture, let's say humans have near perfect accuracy. So the human level error is 1%. In that case, if your learning algorithm achieves 8% training error and 10% dev error, then maybe you wanted to do better on the training set. So the fact that there's a huge gap between how well your algorithm does on your training set versus how well humans do shows that your algorithm isn't even fitting the training set well. So in terms of tools to reduce bias or variance, in this case, I would say focus on reducing bias. So you want to do things like find a train, a bigger neural network, or run gradient descent longer. Just try to do better on the training set. But now let's look at the same training error and depth error and imagine that human level performance was not 1%. So let me just copy this over, but in a different application or maybe on a different data set, let's say that human level error is actually 7.5%. Maybe the images in your data set are so blurry that even humans can't tell whether there's a cat in this picture. This is. This example is maybe slightly contrived because humans are actually very good at looking at pictures and telling if there's a cat in it or not. But for the sake of this example, let's say your data sets images are so blurry or so low resolution that Even humans get 7.5% error. In this case, even though your training error and dev error are the same as the earlier example, you see that maybe you're actually doing just fine on the training set. It's doing only a little bit worse than human level performance. And in this second example, you would maybe want to focus on reducing the this component, reducing the variance in your learning algorithm. So you might try regularization to try to bring your death error closer to your training error, for example. So in the earlier courses discussion on bias and variance, we were mainly assuming that there were tasks where Bayes error is nearly zero. So to explain what just happened here, for our CAD classification example, think of human level error as a proxy or as an estimate for Bayes error, or for Bayes optimal error. And for computer vision tasks, this is a pretty reasonable proxy because humans are actually very good at computer vision. And so whatever a human can do is, is maybe not too far from Bayes error. By definition, human level error is worse than Bayes error, because nothing can be better than Bayes error, But human level error might not be too far from Bayes error. So the surprising thing we saw here is that depending on what human level error is, or really this is really approximately Bayes error, or so we assume it to be, but depending on what we think is achievable with the same training error and dev error in these two cases, we decided to focus on bias reduction tactics or on variance reduction tactics. And what happened is, in the example on the left, 8% training error is really high when you think you could get it down to 1%. And so bias reduction tactics could help you do that. Whereas in the example on the right, if you think that Bayes error is 7.5%, and here we're using human level error as an estimate or as a proxy for Bayes error, but you think that Bayes error is close to 7.5%, then you know there's not that much headroom for reducing your training error further down. You don't really want it to be that much better than 7.5%, because you could achieve that only by maybe starting to overfit the training set. And instead there's much more room for improvement in terms of taking this 2% gap and trying to reduce that by using variance reduction techniques such as regularization or maybe getting more training data. So, to give these things a couple names, this is not widely used terminology, but I found this useful terminology and a useful way of thinking about it, which is I'm gonna call the difference between Bayes error or approximation of Bayes error and the training error to be the avoidable bias. So what you want is to maybe keep improving your training performance until you get down to Bayes error, But you don't actually want to do better than Bayes error. You can't actually do better than Bayes error unless you're overfitting. And this the difference between your training error and the depth error, that's a measure still of the variance problem of your algorithm. And the term avoidable bias acknowledges that there's some bias or some minimum level of error that you just cannot get below, which is that if Bayes error is 7.5%, you don't actually want to get below that level of error. So rather than saying that if your training error is 8%, then the 8% is a measure of bias. In this example you're saying that the avoidable bias is maybe 0.5% or 0.5% is a measure of the avoidable bias whereas 2% is a measure of the variance. And so there's much more room in reducing this 2% than in reducing the 0.5%. Whereas in contrast in the example on the left, this 7% is a measure of the avoidable bias whereas 2% is a measure measure of how much variance you have. And so in this example on the left there's much more potential in focusing on reducing that avoidable bias. So in this example, understanding human level error, understanding your estimate of Bayes error really causes you in different scenarios to focus on different tactics, whether bias avoidance tactics or variance avoidance tactics. There's quite a lot more nuance than how you factor in human level performance into how you make decisions. It's in choosing what to focus on. Let's in the next video go deeper into our understanding of what human level performance really means.