Basic Recipe for Machine Learning (C2W1L03)

In a previous video, you saw how looking at training error and dev error can help you diagnose whether your algorithm has a bias or a variance problem or maybe both. It turns out that this information that lets you much more systematically using what I call a basic recipe for machine learning, it lets you much more systematically go about improving your algorithm's performance. Let's take a look. When training a neural network, here's a basic recipe I would use. After having trained an initial model, I would first ask, does your algorithm have high bias? And so to try to evaluate if there's high bias, you, you should look at um, really the training set or the training data performance. Right? Um, and so if it does have high bias, it's not even fitting the training set that well. Some things you could try would be to try bigger network, such as more hidden layers or more hidden units. Or you could train it longer, uh, you know, maybe run gradients and longer or try some more advanced optimization algorithms, which we'll talk about later in this course. Or you could also try this is kind of a, maybe it'll work, maybe it won't, but we'll see later that there are a lot of different neural network architectures. And maybe you can find a neural network architecture that's better suited for this problem. Putting this in parenthesis, because one of those things that, you know, you just have to try, maybe you can make it work, maybe not. Whereas, uh, getting a bigger network almost always helps and training longer well, doesn't always help, but it certainly never hurts. But so when training a learning algorithm, I would try these things until I can at least get rid of the bias problems, as if go back after I've tried this until, um, and keep doing that until I can fit at least fit the training set pretty well. And usually if you have a big enough network, you could, you should usually be able to fit the, the training data well, so long as there's a problem that is possible for someone to do right. If an image is very blurry, it may be impossible to fit it. But if at least a human can do well on the task. If you think Bayes error is not too high, then by training a bigger network, you should be able to hopefully do well, at least on the training set, to at least fit or overfit the training set. Once you've reduced bias to a acceptable amounts, I would then ask do you have a variance problem? And so to evaluate that I would look at the dev set performance. Where are you able to generalize from a pretty good training Set performance to having a pretty good dev set performance. And if you have high variance, well, best way to solve a high variance problem is to get more data if you can get it. This, you know, can only help. Um, but, but sometimes you can't get more data. Or you could try regularization, which we'll talk about in the next video, to try to reduce overfitting. And then also again this is, sometimes you just have to try it. Um, but if you can find a more appropriate neural network architecture, sometimes that can reduce your variance problem as well, as well as reduce your v, your bias problem. But how to do that? It's harder to be totally systematic how you do that. But so I try these things and kind of keep going back until hopefully you find something with both low bias and low variance, whereupon you would be done so. Couple points to notice. First is that depending on whether you have high bias or high variance, the set of things you should try could be quite different. So I'll usually use the training dev set to try to diagnose if you have a bias or variance problem and then use that to select the appropriate subset of things to try. So for example, if you actually have a high bias problem, getting more training data is actually not gonna help, or at least it's not the most efficient thing to do. Right? So being clear on how much of a bias problem or variance problem or both can help you focus on selecting the most useful things to try. Second, in the earlier era of machine learning there used to be a lot of discussion on was called the bias variance trade off. And the reason for that was that for a lot of the things you could try, you could increase bias and reduce variance, or reduce bias and increase variance. But back in the pre deep learning era we didn't have many tools, we didn't have as many tools that just reduce bias or that just reduce variance without hurting the other one. But in the modern deep learning big data era, um, so long as you can keep training a bigger network and so long as you keep getting more data, which isn't always the case for either of these. But if that's the case, then getting a bigger network almost always just reduces your bias without necessarily hurting a variance, so long as uh, you regularize appropriately, um, and getting more data pretty much always reduces your variance and, and doesn't hurt your bias much. So what's really happened is that with these two steps, the ability to train bigger network or get more data, we now have tools to drive down bias and just drive down bias or drive down variance and just drive down variance without really hurting the other thing that much. And I think this has been one of the big reasons that deep learning has been so useful for supervised learning that, that there's much less of this trade off where you have to carefully balance bias and variance. But sometimes you just have more options for reducing bias and reduce or reducing variance without necessarily increasing the other one. And in fact, so as you have a well, regularized network, we'll talk about regularization starting from the next video. Training a bigger network almost never hurts. And the main cost to training a neural network that's too big is just computational time, so long as you're regularizing. So I hope this gives you a sense of the basic structure of how to organize your machine learning problem to diagnose bias invariance and then try to select the right operation for you to make progress on your problem. One of the things I mentioned several times in the video is regularization. Uh, it's a very useful technique for reducing variance. There is a little bit of a bias variance trade off when you use regularization. It might increase the bias a little bit. It although often not too much if you have a huge enough network. But let's dive into more details in the next video so we can better understand how to apply regularization to your neural network.