Understanding Dropout (C2W1L07)

Dropout does this seemingly crazy thing of randomly knocking out units in your network. Why does it work so well as a regularizer? Let's gain some better intuition. In the previous video, I gave this intuition that Dropout randomly knocks out units in your network. So it's as if on every iteration, you're working with a smaller neural network. And so using a smaller neural network seems like it should have a regularizing effect. Here's the second intuition, which is, you know, let's look at it from the perspective of a single unit. Let's say this one. Now, for this unit to do its job, it has four inputs, and it needs to generate some meaningful output. Now, with dropout, the inputs can get randomly eliminated. You know, sometimes those two units would get eliminated, sometimes a different unit would get eliminated. So what this means is that this unit, which I'm circling in purple, it can't rely on any one feature because any one feature could go away at random, or any one of its own inputs could go away at random. So in particular, it'd be reluctant to put all of its bets on, say, just this input, right? The weights. We reluctant to put too much weight on any one input because it could go away. So this unit will be more motivated to spread out this weight and give you a little bit of weight to each of the four inputs to this unit. And by spreading out the weights, this will tend to have an effect of shrinking the squared norm of the weights. And so, similar to what we saw with L2 regularization, the effect of implementing dropout is that it shrinks the weights and does similar to L2 regularization, it helps to prevent overfitting. But it turns out that dropout can formally be shown to be an adaptive form of L2 regularization. But the L2 penalty on different weights are different depending on the size of the activations being multiplied to that weight. But to summarize, it is possible to show that dropout has a similar effect to L2 regularization. Only the L2 regularization applied to different weights can be a little bit different and even more adaptive to the scale of different inputs. One more detail for when you're implementing dropout. Here's a network where you have three input features. This is seven hidden units here. 7, 3, 2, 1. So one of the parameters we had to choose was the keep prop, right, which is a chance of keeping a unit in each layer. So it is also feasible to vary keep prop by layer. So for the first layer, your matrix W1 will be 3 by 7. Your second weight matrix will be 7 by 7, W3 will be 7 by 3, and so on. And so W2 is actually the biggest weight matrix, right? Because they're actually the largest set of parameters would be in W2, which is 7 by 7. So to prevent. To reduce overfitting of that matrix, maybe for this layer, I guess this is layer two, you might have a key prop that's relatively low, say 0.5. Whereas for different layers where you might worry less about overfitting, you could have a higher key prop. Maybe this is 0.7, maybe this is 0.7. Um, and then for layers where you don't worry about overfitting at all, you can have a keyprop of 1.0, right? So, you know, for clarity, these are numbers I'm drawing in the purple boxes. These could be different keyprops for different layers. Notice that the keyprop of 1.0 means that you're keeping every unit, and so you're really not using dropout for that layer. But for layers where you're more worried about overfitting, really the layers with a lot of parameters, you could set keyprop to be smaller to apply a more powerful form of dropout. It's kind of like cranking up the regularization parameter Lambda for L2 regularization, where you try to regularize some layers more than others. And technically you can also apply dropout to the input layer, where you can have some chance of, you know, just x ing out one or more of the input features, although in practice usually don't do that that often. Uh, and so a key problem of 1.0 is quite common. For the input layer, you might also use a very high value, maybe 0.9 is much less likely that, you know, you want to eliminate half of the input features. So usually keyprop, if you apply it at all, will be a number close to one, if you even apply dropout at all to the input layer. So just to summarize, if you're more worried about some layers overfitting than others, you can set a lower key prop for some layers than others. The downside is this gives you even more hyper parameters to search for using cross validation. One other alternative might be to have some layers where you apply dropout and some layers where you don't apply dropout, and then just have one hyperparameter, which is a key prop for the layers for which you do apply dropout. And before we wrap up, just a couple implementational tips. Many of the first successful implementations of dropouts were to computer vision. So in Computer vision. The input size is so big you're inputting all these pixels that you almost never have enough data. And so drawpot is very frequently used by encoding computer vision. And there are some computer vision researchers that pretty much always use it almost as a default. Um, but really the thing to remember is that dropout is a regularization technique. It helps prevent overfitting. And so unless my algorithm is overfitting, I wouldn't actually bother to use dropout. So it's used somewhat less often in other application areas. It's just for computer vision, you know, you usually just don't have enough data, so you're almost always overfitting. Which is why there tends to be some computer vision researchers that just square by dropout. But intuition doesn't always generalize, I think, to other disciplines. One big downside of dropout is that the cost function J is no longer well defined on every iteration. You're randomly, you know, killing off a bunch of nodes. And so if you're double checking the performance of gradient descent, it's actually harder to double check that. Right? You have a well defined cost function J that is going downhill on every iteration because the cost function J that you're optimizing is actually less. Less well defined or is certainly hard to calculate. So you lose this debugging tool to be able to plot a graph like this. So what I usually do is turn off dropout, or if you will set keyprop equals one and then run my code and make sure that it is monotonically decreasing J and then turn on dropout and hope that, you know, I didn't introduce bug into my code during dropout because you need other ways, I guess. But not plotting these figures to make sure that your code is working. The grain is in is working even with dropout. So with that, there are still a few more regularization techniques that are worth your knowing. Let's talk about a few more such techniques in the next video.