#38 Machine Learning Specialization [Course 1, Week 3, Lesson 4]
Andrew Ng covers three ways to fix overfitting: collect more training data, reduce features, or use regularization to shrink parameters. Regularization is gentler than cutting features because it keeps all of them but makes their weights smaller.
Chapters
Later in this specialization, we'll talk about debugging and diagnosing things that can go wrong with learning algorithms. You also learn about specific tools to recognize when overfitting and underfitting may be occurring. But for now, when you think overfitting has occurred, let's talk about what you can do to address it. Let's say you fit a model and it has high variance. It's overfit. Here's our overfit house price prediction model. One way to address this problem is to collect more training data. So that's one option. If you're able to get more data, that is More training examples on sizes and prices of houses, then with the larger training set, the learning algorithm will learn to fit a function that is less wiggly. So you can continue to fit a high order polynomial or some other function with a lot of features. And if you have enough training examples, it will still do okay. So to summarize, the number one tool you can use against overfitting is to get more training data. Now, getting more data isn't always an option. Maybe only so many houses have been sold in this location, so maybe there just isn't more data to be had. But when the data is available, this can work really well. A second option for addressing overfitting is to see if you can use fewer features. In the previous video, our model's features included the size X as well as the size squared, that is x squared and x cubed and x circumflex 4, and so on. These were a lot of polynomial features. So in that case, one way to reduce overfitting is to just not use so many of these polynomial features. But now let's look at a different example. Maybe you have a lot of different features of a house with which to try to predict its price, ranging from its size, number of bedrooms, number of floors, the age, average income of the neighborhood, and so on and so forth to the distance to the nearest coffee shop. It turns out that if you have a lot of features like these, but don't have enough training data, then your learning algorithm may also overfit to your training set. Now, instead of using all 100 features, if we were to pick just a subset of the most useful ones, maybe size bedrooms and the age of the halls. If you think those are the most relevant features, then using just that smaller subset of features, you may find that your model no longer overfits as badly. Choosing the most appropriate set of features to use is sometimes also called feature selection. One way you could do so is to use your intuition to choose what you think is the best set of features. What's most relevant for predicting the price? Now, one disadvantage of feature selection is that by using only a subset of the features, the algorithm is throwing away some of the information that you have about the hoses. For example, maybe all of These features, or 100 of them, are actually useful for predicting the price of a hose. So maybe you don't want to throw away some of the information by throwing away some of the features. Later in course two, you also see some algorithms for automatically choosing the most appropriate. So the features you use for prediction task. Now, this takes us to the third option for reducing overfitting. This technique, which we'll look at in even greater depth in the next video, is called regularization. If you look at an overfit model, here's a model using polynomial features xx2, x cubed, and so on, you find that the parameters are often relatively large. Now, if you were to eliminate some of these features, say if you were to eliminate the feature x4 that corresponds to setting this parameter to zero. So setting a parameter to zero is equivalent to eliminating a feature, which is what we saw on the previous slide. It turns out that regularization is a way to more gently reduce the impacts of some of the features without doing something as harsh as eliminating it outright. What regularization does is encourage the learning algorithm to shrink the values of the parameters without necessarily demanding that the parameter is set to exactly zero. And it turns out that even if you fit a higher order polynomial like this, so long as you can get the algorithm to use smaller parameter values, w1, w2, w3, w4, you end up with a curve that ends up fitting the training data much better. So what regularization does is it lets you keep all of your features, but it just prevents the features from having an overly large effect, which is what sometimes can cause overfitting. By the way, by convention, we normally just reduce the size of the WJ parameters, I.e. w1 through WN. It kind of doesn't make a huge difference whether you regularize the parameter B as well. You could do so if you want or not. If you don't, I usually don't. And it's just fine to regularize W1, W2 all the way to WN, but not really encourage B to become smaller. In practice, it should make very little difference whether you also regularize B or not. So to recap, These are the three ways you saw in this video for adjusting overfitting. 1. Collect more data. If you can get more data. This can really help reduce overfitting. Sometimes that's not possible, in which case some of the options are 2. Try selecting and using only a subset of the features. You learn more about feature selection in course two. Three would be to reduce the size of the parameters using regularization. This will be the subject of the next video as well. Just for myself, I use regularization all the time, so this is a very useful technique for training learning algorithms, including neural networks specifically, which you see later in the specialization as well. I hope you also check out the optional lab on overfitting. In the lab you be able to see different examples of overfitting and adjust those examples by clicking on options in the plots. You also be able to add your own data points by clicking on the plot and see how that changes the curve that is fit. You can also try examples for both regression and classification and you really change the degree of the polynomial to be x, x squared x cubed X and so on. The lab also lets you play with two different options for addressing overfitting. You can add additional training data to reduce overfitting and you can also select which features to include or to exclude as another way to try to reduce overfitting. So please take a look at the lab, which I hope will help you build your intuition about overfitting as well as some methods for addressing it. In this video you also saw the idea of regularization at a relatively high level. I realized that all of these details on regularization may not fully make sense to you yet. But in the next video we'll start to formulate exactly how to apply regularization and exactly what regularization means. And then we'll start to figure out how to make this work with our learning algorithms. So to make linear regression and logistic regression and in the future other algorithms as well, avoid overfitting. Let's take a look at that in the next video.
