L2 regularization shrinks weights toward zero, simplifying the network and keeping activations in their linear range so deep networks behave more like linear models. Larger lambda values reduce variance and overfitting. Plot the full regularized cost when debugging gradient descent.
Why does regularization help with overfitting? Why does it help with reducing variance problems? Let's go through a couple examples to gain some intuition about how it works. So recall that our high bias, high variance and. Right, just right. Pictures from earlier video had looked something like this. Now, let's say you're fitting a large and deep neural network. I know I haven't drawn this one too large or too deep, but let's say you think some neural network and is currently overfitting. So you have some cost function, right? J of WB equals some of the losses, like so. Right. And so what we did for regularization was add this extra term that penalizes the weight matrices from being too large. And we said that was a Frobenius norm. So why is it that shrinking the L2 norm or the Frobenius norm with the parameters might cause less overfitting? One piece of intuition is that if you crank the regularization lambda to be really, really big, they'll be really incentivized to set the weight matrices W to be reasonably close to zero. So one piece of intuition is maybe it'll set the weights to be so close to zero for a lot of hidden units that it's basically zeroing out a lot of the impact of these hidden units. And if that's the case, then, you know, this much simplified neural network becomes a much smaller neural network. In fact, it's almost like a logistic regression unit, you know, but stacked multiple layers deep. And so that will take you from this overfitting case much closer to the left, toward the high bias case. But hopefully there'll be an intermediate value of lambda that results in the result closer to this just right case in the middle. But the intuition is that by cranking up lambda to be really big, it'll set W close to zero, which in practice, this isn't actually what happens. We can think of it as 00ing out or at least reducing the impact of a lot of hidden units. So you end up with what might feel like a simpler network that gets closer and closer to. As if you were just using logistic regression. The intuition of completely zeroing out a bunch of hidden units isn't quite right. It turns out that what actually happens is it'll still use all the hidden units, but each of them will just have a much smaller effect. But you do end up with a simpler network and as if you have a smaller network that is therefore less prone to overfitting. So I'm not sure if this intuition helps, but when you implement regularization in the primary exercise, you actually see some of these variance reduction results yourself. Here's another attempt at additional intuition for why regularization helps prevent overfitting. And for this I'm gonna assume that we're using the TANH activation function, which looks like this, right? So this is a G of Z equals tanh of z. So if that's the case, notice that so long as Z is quite small, so if Z takes on only a smallish range of parameters, maybe around here, then you're just using the linear regime of the tanh function. It's only if Z is allowed to wander, oh, you know, to larger values or smaller values like so that the activation function starts to become less linear. So the intuition you might take away from this is that if lambda, the regularization parameter, is large, then you have that your parameters will be relatively small because they are penalized being large in the cost function. And so if the weights W are small, then because Z is equal to W, right? And then technically it's buzz B, but if W tends to be very small, then Z will also be relatively small. And in particular, if Z ends up taking relatively small values just in this little range, then G will be roughly linear. So it's as if every layer will be roughly linear, as if it's just linear regression. And we saw in course one that if every layer is linear, then your whole network is just a linear network. And so even a very deep network with a deep network with a linear activation function is at the end of the day, only able to compute a linear function. So it's not able to, you know, fit those very, very complicated decision, very non linear decision boundaries that allow it to, you know, really overfit. Right, the data set like we saw on the overfitting high variance case on the previous slide. Okay, so just to summarize, if the regularization parameter is very large, the parameter is W very small. So Z will be relatively small, kind of ignoring the effects of B for now. Um, but so Z is relatively, so Z would be relatively small, or really we should say it takes on a small range of values. And so the activation function, if it's tanh, say, will be relatively linear. And so your whole neural network will be computing something not too far from a big linear function, which is therefore a pretty simple function rather than a very complex, highly nonlinear function. And so it's also much less able to overfit. Okay, and again, when you implement regularization for yourself in the primary exercise, you'll be able to see some of these effects yourself. Before wrapping up our depth discussion on regularization, I just want to give you one implementational tip, which is that when implementing regularization, we took our definition of the cost function J and we actually modified it by adding this extra term that penalizes the weights being too large. And so if you implement gradient descent, one of the steps to debug gradient descent is to plot the cost function J as a function of the number of iterations of gradient descent. And you want to see that the cost function J, you know, decreases monotonically after every iteration of gradient descent. And if you're implementing regularization, then please remember that J now has this new definition. If you plot the old definition of J just this first term, then you might not see a decrease monotonically. So to debug gradient set, make sure that you're plotting this new definition of J that includes this second term as well. Otherwise you might not see J decrease monotonically on every single iteration. So that's it for L2 regularization, which is actually a regularization technique that I use the most in training deep learning models. In deep learning, there's another sometimes used regularization technique called dropout regularization. Let's take a look at that in the next video.
