Other Regularization Methods (C2W1L08)

In addition to L2 regularization and draw file regularization, there are a few other techniques for reducing overfitting in your neural network. Let's take a look. Let's say you're fitting a CAD classifier. If you're overfitting, getting more training data can help, but getting more training data can be expensive and you sometimes just can't get more data. But what you can do is augment your training set by taking an image like this and for example, flipping it horizontally and adding that also to your training set. So now, instead of just this one example in your training set, you can add this to your training example. So by flipping your images horizontally, you could, you know, maybe double the size of your training set because your training set is now a bit redundant. This isn't as good as if you had collected an additional M set of brand new independent examples, but you could do this without needing to pay the expense of going out to take more pictures of cats. And then, other than flipping horizontally, you can also take random crops of the image. So here we've rotated and sort of randomly zoom into the image, and this still looks like a cat, but so by taking random distortions and transformations of the image, you could augment your data set and make additional fake training examples. Again, these extra fake training examples, they don't add as much information as if you were to go out and get a brand new independent example of a cat. But because you could do this, you know, almost for free, other than for some computational cost, uh, this can be an inexpensive way to give your data. This can be an inexpensive way to give your algorithm more data and therefore, you know, sort of regularize it and reduce overfitting. And by synthesizing examples like this, what you're really telling your algorithm is that if something is a cat, then flipping it horizontally is still a cat. Notice I didn't flip it vertically because maybe we don't want upside down cats, right? And then also maybe randomly zooming in to part of the image is probably still a cat. For optical character recognition, you can also augment your dataset by taking a digit and imposing random rotations and distortions to it. So if you add these things to your training set, you know, these are also still digit fours. For illustration, um, I applied a very strong distortion, so this looks very wavy. 4. In practice, you don't need to distort the fall quite as aggressively, but just a more subtle distortion than what I'm showing here to make this example clearer for you. Right, but a more subtle distortion is is usually used in practice because this looks like really warped force. So data augmentation can be used as a regularization technique. There's effects similar to regularization. There's one other technique that is often used called early stopping. So what you're gonna do is as you run gradient descent, you're going to plot your either the Training error, your, your 01 Classification error on the training set, or just plot the cost function J you're optimizing and that should decrease monotonically, like so. Alright. Because as you train, hopefully your trading error, your cost function J should decrease. So with early stopping, what you do is you plot this and you'd also plot your dev set, which error. And again, this can be a classification error on your development set or something like the cost function, like the logistic loss or the log loss. Evaluating our dev set and what you find is that your dev set error will usually go down for a while and then it will increase from there. So what early stopping does is you would say, well, it looks like your neural network was doing best around that iteration. So we're just gonna stop training your neural network halfway and you know, take whatever value achieved this dev set error. So why does this work? Well, when you haven't run many iterations for your neural network yet, your parameters W will be close to zero. Because, you know, with random initialization, you probably initialize W to small random values. So before you train for a long time, W is still quite small. And as you iterate, as you train, W get bigger and bigger and bigger. Until here, maybe you have a much larger value of the parameters W for your neural network. So what early stopping does is by stopping halfway, you have only a, you know, mid size right W. And so similar to L2 regularization, by picking a neural network with smaller norm for your parameters W, hopefully your neural network is overfitting less. And the term early stopping refers to the fact that you're just stopping the training of your neural network early. I sometimes use early stopping when training a neural network, but it does have one downside. Let me explain. I think of the machine learning process as comprising several different steps. One is that you want an algorithm to optimize the cost function J. And we have various tools to do that, you know, such as gradient descent. And then we'll talk later about other algorithms like momentum and ALV and RMSprop and Adam and so on. But then after optimizing the cost function J, you also wanted to not overfit. And we have some tools to do that, such as your regularization, getting more data and so on. Now, in machine learning, we already have so many hyperparameters to search over, it's already very complicated to choose among the space of possible algorithms. And so I find machine learning easier to think about when you have one set of tools for optimizing the cost function j. And when you're focusing on optimizing the cost function j, all you care about is finding W and B so that j of WB is as small as possible. You just don't think about anything else other than reducing this. And then it's a completely separate task to not overfit, um, in other words, to reduce variance. And when you're doing that, you have a separate set of tools for doing it. And this principle is sometimes called, um, orthogonalization. And this is idea that you want to be able to think about one task at a time. Um, I'll say more about orthorganization in a later video. So if you don't fully get the concept yet, don't worry about it. But to me, the main downside of early stopping is that this couples these two tasks. So you no longer can work on these two problems independently. Because by stopping gradient descent early, you're sort of breaking whatever you're doing to optimize the cost function j. Because now you're not doing a great job reducing the cost function j, you're sort of not done that that well. And then you also simultaneously trying to not overfit. So instead of using different tools to solve the two problems, you're using one tool that kind of mixes the two. And this just makes the set of things you could try more complicated to think about. Rather than using early stopping, one alternative is just use L2 regularization. Then you can just train the neural network as long as possible. I find that this makes the search space of hyperparameters easier to decompose and easier to search over. But the downside of this though, is that you might have to try a lot of values of the regularization parameter lambda. And so this makes searching over many values of lambda more computationally expensive. And the real advantage of early stopping is that running the gradient descent process just once, you get to try out values of small w, mid size W and large w without needing to try a lot of values of the regularization L2 regularization hyperparameter lambda. If this concept doesn't completely make sense to you yet, don't worry about it. When we talk about ortho organization in greater detail in a later video, I think this will make a bit more sense. Despite its disadvantages, many people do use it. I personally prefer to just use L2 regularization and try different values of lambda. That's assuming you can afford the computation to do so. But early stopping does let you get a similar effect without needing to explicitly try lots of different values of lambda. So you've now seen how to use data augmentation as well as if you wish early stopping in order to reduce variance or to prevent overfitting in your neural network. Next, let's talk about some techniques for setting up your optimization problem to make your training go quickly.