Bias/Variance (C2W1L02)

I've noticed that almost all the really good machine learning practitioners tend to have a very sophisticated understanding of bias and variance. Bias and variance is one of those concepts that's easy to learn but difficult to master. Even if you think you've seen the basic concepts of bias and variance, there's often more nuanced to it than you'd expect in the deep learning era. Another trend is that there's been less discussion of what's called the bias variance trade off. You might have heard this thing called the bias variance trade off, but in a deep learning era, there's less of a trade off. So it's slow talk about bias. We still talk about variance, but we just talk less about the bias variance trade off. Let's see what this means. Let's say you have a data set that looks like this. If you fit a straight line to the data, maybe you get a logistic regression fit to that. This is not a very good fit to the data. And so there's a classifier of high bias. Or we say that this is underfitting the data. On the opposite end, if you fit an incredibly complex classifier, maybe a deep neural network or a neural network with a lot of hidden units, maybe you can fit the data perfectly. But that doesn't look like a great fit either. So this is a classifier with high variance, and this is overfitting the data. And there might be some classifier in between with a medium level of complexity that, you know, maybe fits a curve like that, that looks like a much more reasonable fit to the data. And so that's the, we call that, you know, just right. Right, somewhere in between. So in a 2D example like this, with just two features, X1 and X2, you can plot the data and visualize bias and variance. In high dimensional problems, you can't plot the data and visualize the decision boundary instead. There are a couple different metrics that we'll look at to try to understand bias and variance. So continuing our example of CAT picture classification, where that's a positive example and that's a negative example, the two key numbers to look at to understand bias and variance will be the training set error and the dev set or the development set error. So for the sake of argument, let's say that, you know, recognizing cats in pictures is something that people can do nearly perfectly, right? And so let's say your training set error is, um, 1% and your dev set error is, uh, for the sake of argument, less. Let's say it's 11%. So in this example, you're doing very well on the training set, but you're doing, um, relatively poorly on the development set. So this looks like you might have overfit the training set that somehow you're not generalizing well to this holdout cross validation set or the development set. And so if you have an example like this, we would say this has high variance. So by looking at the training set error and the development set error, you know, you would be able to render a diagnosis of your algorithm having high variance. Now let's say that you measure your training set and your dev set error and you get a different result. Let's say that your training set error is 15%. I'm writing your training set error in the top row and your dev set error is 16%. In this case, assuming that humans achieve, you know, roughly 0% error, that humans can look at these pictures and just tell if it's CAT or not, then it looks like the algorithm's not even doing very well on the training set. So if it's not even fitting the training data scene that well, then this is underfitting the data. And so this algorithm has high bias. But in contrast, this is actually generalizing at a reasonable level to the dev set, whereas performance of the dev set is only 1% worse than its performance on the training set. So this algorithm has a problem of high bias, because what's not even training, it's not even fitting the training set well. This is similar to the leftmost plot we had on the previous slide. Now here's another example. Let's say that you have 15% trading set error. So that's pretty high bias. But when you evaluate on a dev set, it does even worse. Maybe it does, you know, 30% in this case. I would diagnose this algorithm as having high bias because it's not doing that well on the training set and high variance. So this is, you know, really the worst of both worlds. Um, and one last example, if you have, you know, 0.5 training set error and 1% dev set error, then, well, maybe your users are quite happy that you have a cat class 5 with only 1% error, then this would have, you know, low bias and low variance. One subtlety that I'll just briefly mention, but we'll leave to a later video to discuss in detail, is that this analysis is predicated on the assumption that human level performance gets nearly 0% error, or more generally, that the optimal error, sometimes called Bayes error, for the so the Bayesian optimal error is nearly 0% I don't want to go into detail on this in this particular video, but it turns out that if the optimal error or the Bayes error were much higher, say it were 15%, then if you look at this classifier, um, 15% is actually perfectly reasonable for training set, and you wouldn't say there's high bias and also a pretty low variance. So the case of how to analyze bias and variance when no classifier can do very well, for example, if you have really blurry images so that you know, even a human or just no system could possibly do very well, then maybe Bayes error is much higher. And then there's some details to how this analysis would change. But leaving aside this subtlety for now, the takeaway is that by looking at your training set error, you can get a sense of how well you're fitting, at least the training data. And so that tells you if you have a bias problem, and then looking at how much higher your error goes when you go from the training set to the dev set, that should give you a sense of how bad is the variance problem. So are you doing a good job generalizing from the training set to the dev set that gives you a sense of your variance? Um, all this is under the assumption that the Bayes error is quite small and that your train and your death sets are drawn from the same distribution. If. If those assumptions are violated, there's a more sophisticated analysis you could do, which we'll talk about in the later video. Now, on the previous slide, you saw what high bias, high variance looks like, and I guess you had a sense of what a good classified looks like. What does high bias and high variance looks like is kind of the worst of both worlds. So you remember we said that a classifier like this, a linear classifier, has high bias because it underfits the data. So this would be a classifier that is mostly linear and therefore underfits the data. I'm gonna drawing this in purple. But if somehow your classifier does some weird things, then it's actually overfitting parts of the data as well. So. So the classifier that I drew in purple has both high bias and high variance, right? It has high bias because by being a mostly linear classifier, it's just not fitting, you know, this quadratic light shape that well. But by having too much flexibility in the middle, it somehow gets this example. And this example, um, overfits those two examples as well. So this classifier kind of has high bias because it was mostly linear, but you need maybe a curve function according to quadratic function and it has high variance because it had too much flexibility to fit, you know, those two mislabeled or those outlier examples in the middle as well. Um, in case this seems contrived. Well, it is. This example is a little bit contrived in two dimensions, but with very high dimensional inputs, you actually do get things with high bias in some regions and high variance in some regions. And so it is possible to get crosswise like this in high dimensional inputs that seem less contrived. So, to summarize, you've seen how, by looking at your algorithm's error on the training set and your algorithm's error on the dev set, you can try to diagnose whether it has problem of high bias or high variance, or maybe both. Or maybe neither. And depending on whether your algorithm suffers from bias or variance, it turns out that there are different things you could try. So in the next video I want to present to you a, what I call a basic recipe for machine learning. So that lets you more systematically try to improve your algorithm depending on whether it has high bias or high variance issues. So let's go on to the next video.