Why Does Batch Norm Work? (C2W3L06)

So why does batch not work? Here's one reason. You've seen how normalizing the Input features the X's to mean 0 and variance 1, how that can speed up learning. So rather than having some features that range from zero to one and some from one to a thousand, by normalizing all the features, input features X to take on a similar range of values that can speed up learning. So one intuition behind why Batchelor works is this is doing a similar thing, but but for the values in your hidden units and not just for your input layer. Now this is just a partial picture for what batch norm is doing. There are a couple further intuitions that will help you gain a deeper understanding of what batch norm is doing. Let's take a look at those in this video. A second reason why batch norm works is it makes weights later or deeper in the neural network, say the weights of layer 10 more robust changes to weights in earlier layers of the neural network, say in layer one. To explain what I mean, let's look at this motivating example. Let's say you're training a network, maybe a shallow network like logistic regression, or maybe a neural network, maybe, um, maybe a shallow network like logistic regression or maybe a deeper network on our famous cat detection task. But let's say that you've trained your data sets on all images of black cats. If you now try to apply this network to data with colored cats, where the positive examples are not just black cats like on the left, but to colored cats like on the right, then your classifier might not do very well. So in pictures, if your training set look like this, where you have positive examples here and negative examples here, but you were to try to generalize it to a data set where maybe the positive examples are here and the negative examples are here, then you might not expect a model trained on the data on the left to do very well on the data on the right, even though you know there might be the same function that actually works well. But you wouldn't expect your learning algorithm to discover that green decision boundary just looking at the data on the left. So this idea of your data distribution changing goes by the somewhat fancy name covariate shift. And the idea is that if you learn some X to Y mapping, if the distribution of X changes, then you might need to retrain your learning algorithm. And this is true even if the function, the ground truth function mapping from X to Y remains unchanged, which it is in this example, because the ground truth function is. Is this picture a cat or not? And the need to retrain your function becomes even more acute or becomes even worse if the ground truth function shifts as well. So how does this problem of covariate shift apply to a neural network? Consider a deep network like this, and let's look at the learning process from the perspective of this hidden layer, the third hidden layer. So this network has to learn the parameters W3 and B3. And from the perspective of the third hidden layer, it gets some set of values from the earlier layers and then has to do some stuff to hopefully make the output Y hat close to the ground true value. Yes. So let me cover up the nodes on the left for a second. So from the perspective of this third hidden layer, it gets some values, let's call them a 21, a 22, a 23 and a 24. But these values might as well be features x1, x2, x3, x4. And the job of the third hidden layer is to take these values and find a way to map them to y hat. So you can imagine doing gradient descent so that these parameters W3B3, as well as maybe W4, B4 and even W5B5, maybe trying to learn those parameters. So the network does a good job mapping from the values I drew in black on the left to the output values Y hat. But now let's uncover the left of the network again. The network is also adapting parameters W2B2 and W1B1. And so as these parameters change, these values A2 will also change. So from the perspective of the third hidden layer, these hidden unit values are changing all the time. And so it's suffering from the problem of covariate shift that we talked about on the previous slide. So what batch norm does is it reduces the amount that the distribution of these hidden unit values shifts around. And if we were to plot the distribution of these hidden unit values, maybe this is technically we normalize the z. So this is actually Z21 and Z22, and it's gonna plot two values instead of four values. So we can visualize it in 2D. What batch norm is saying is that the values for Z21 and Z22 can change, and indeed they will change when the neural network updates the parameters in the earlier layers. But what batch norm ensures is that no matter how it changes, the mean and variance of Z21 and Z22 will remain the same. So, so even if the exact values of Z21 and Z22 change, their mean and variance will at least stay, say, mean 0 and variance 1 or not necessarily mean 0. And variance 1. But whatever value is governed by beta 2 and gamma 2, which if the neural network chooses, can force it to be mean 0 and variance 1, or really any other mean variance. But what this does is it limits the amount to which updating the parameters in the earlier layers can affect the distribution of values that the third layer now sees and therefore has to learn on. And so batch norm reduces the problem of the input values changing. It really causes these values to become more stable so that the later layers of the neural network has more firm ground to stand on. And even though the input distribution changes a bit, it changes less. And what this does is even as the earlier layers keep learning, the amount that this forces the later layers to adapt to its earlier layers changes is reduced. Or if you will, it weakens the coupling between what the earlier layers parameters has to do and what the later layers parameters have to do. And so it allows each layer of the network to learn by itself, you know, a little bit more independently of other layers. And this is the effect of speeding up learning in the whole network. So I hope this gives some better intuition. But the takeaway is that dash norm means that, especially from the perspective of one of the later layers of the neural network, the earlier layers don't get to shift around as much because they're constrained to have the same mean and variance. And so this makes the job of learning in the later layers easier. It turns out BatchNorm has a second effect. It has a slight regularization effect. So one non intuitive thing about batch norm is that each mini batch, I'll say mini batch xt, has the values zt has the values zl scaled by the mean and variance computed on just that one mini batch. Now, because the mean and variance computed on just that mini batch, as opposed to computed on the entire data set, that mean and variance has a little bit of noise in it because it's computed just on your mini batch of say, 64 or 128 or maybe 256 or larger training examples. So because the mean and variance is a little bit noisy, because it's estimated with just a relatively small sample of data, the scaling process going from ZL to Z tilde L, that process is a little bit noisy as well, because it's computed using a slightly noisy mean and variance. So similar to dropout, it adds some noise to each hidden layer's activations. The way dropout adds noise is it takes a hidden unit and it multiplies it by 0 with some probability and multiplies it by 1 with some probability. And so dropout adds multiplicative noise because it's multiplying by 0 or 1, whereas dash norm has multiple of noise because it's scaling by the standard deviation as well as additive noise because it's subtracting the mean where here the estimates of the mean and the standard deviation are noisy. And so similar to dropout, bash norm therefore has a slight regularization effect because by adding noise to the hidden units, it's forcing the downstream hidden units not to rely too much on any one hidden unit. And so similar to dropout, this adds noise to hidden layers and therefore has a very slight regularization effect because the noise added is quite small. This is not a huge regularization effect. And you might choose to use batch norm together with dropout, and you might use batch norm together with dropout if you want the more powerful regularization effect of dropout. And maybe one other slightly non intuitive effect is that if you use a bigger mini batch size, right? So if you use a mini batch size of say 512 instead of 64, by using a larger mini batch size, you're reducing this noise and therefore also reducing this regularization effect. So that's one strange property of trans dropout, which is that by using a bigger mini batch size, you reduce the regularization effect. Having said this, I wouldn't really use batch norm as a regularizer. That's really not the intent of batch norm. But sometimes it has this extra intended or unintended effect on your learning algorithm. But really don't turn to batch norm as a regularization. Use it as a way to normalize your hidden units activations and therefore speed up learning. And I think the regularization is an almost your unintended side effect. So I hope that gives you better intuition about what batch norm is doing. Before we wrap up the discussion on batch norm, there's one more detail I wanna make sure you know, which is that batch norm handles data one mini batch at a time. It computes mean and variances on meany batches. So at test time you're trying to make predictions, trying to evaluate the neural network. You might not have a meany batch of examples. You might be be processing one single example at a time. So at test time you need to do something slightly differently to make sure your predictions make sense. In the next and final video on batch norm, let's talk over the details of what you need to do in order to take your neural network train using batch norm to make predictions.