Batch normalization standardizes hidden layer activations within each mini-batch, then applies learned scale and shift parameters (gamma and beta). This technique widens the range of working hyperparameters and makes training deep networks easier.
Sergei Iofi and Christian Zegedy (as stated by the speaker).
Gamma and beta are learnable parameters updated by the learning algorithm.
In the rise of deep learning, one of the most important ideas has been an algorithm called batch normalization, created by two researchers, Sergei Iofi and Christian Zegedy. Batch normalization makes your hyperparameter search problem much easier, makes your neural network much more robust to the choice of hyperparameters. There's a much bigger range of hyperparameters that work well, and they'll also enable you to much more easily train even very deep networks. Let's see how batch normalization works. When training a model such as logistic regression, you might remember that normalizing the input features can speed up learning. So you can compute the means, subtract off the means from your training set, compute the variances, um, this sum of xi squared, this is a element Y squaring right, and then normalize your data set according to the variances. And we saw in an earlier video how this can turn to contour of your learning problem from something that might be very elongated. You know, something that is more round and easier for an algorithm like gradient descent to optimize. So this works in terms of normalizing the input feature values to a neural network. Auto logistic regression. Now, how about a deeper model? You have not just input features X, but in this layer you have activations A1, in this layer you have activations A2 and so on. So if you want to train the parameters, say W3, B3, then won't it be nice if you can normalize the mean and variance of A2 to make the training of W3B3 more efficient? In the case of logistic regression, we saw how normalizing x1, x2, x3 maybe helps you train W&B more efficiently. So here the question is, for any hidden layer, can we normalize the values of a, let's say A two in this example by really any hidden layer. So as to train W3B3 faster, right? Since A2 is the input to the next layer, that therefore affects the training of W3 and B3. So this is what batch norm does. Batch normalization, or batch norm for short, does. Although technically we'll actually normalize the values of not a 2 but z 2, there is some debate in the deep learning literature about whether you should normalize the value before the activation function so Z2, or whether you should normalize the value after applying the activation function A2, in practice, normalizing Z2 is done much more often. So that's the version I'll present and what I would recommend you use as a default choice. So here is how you would implement Batch norm. Given some intermediate values. In your neural net, let's say that you have some hidden unit values, Z1 up to ZM. And this is really from some hidden layer. So it'd be more accurate to write this as Z for some hidden layer I for I equals 1 through m. But to reduce writing, I'm going to omit this square bracket L, just to simplify the notation on this line. So given these values, what you do is compute the mean as follows. Again, all this is specific to some layer L, but I'm omitting the square bracket L, um, and then you compute the variance using the pretty much the formula you would expect. And then you would take each of the zi's and normalize it. So you get zi normalized by subtracting off the mean and dividing by the standard deviation. Um, for numerical stability, we usually add epsilon to denominator like that, just in case sigma squared turns out to be zero in some estimate. And so now we've taken these values z and normalize them to have mean zero and standard unit variance. So every component of z has mean 0 and variance 1. But we don't want the hidden units to always have mean 0 and variance 1. Maybe it makes sense for hidden units to have a different distribution. So what we'll do instead is compute. We call this Z tilde equals gamma zinorm plus beta. And here gamma and beta are learnable parameters of your model. So if you're using gradient descent or some other algorithm like gradient descent, with momentum RMS proper add, you would update the parameters gamma and beta, just as you would update the weights of your neural network. Now, notice that the effect of gamma and beta is that it allows you to set the mean of Z tilde to be whatever you want it to be. In fact, if gamma equals square root sigma squared plus epsilon. So if gamma were equal to this denominator term, and if beta were equal to mu, so this value up here, then the effect of gamma xe, norm plus beta is that it would exactly invert this equation. So if this is true, then actually Z tilde I is equal to zi. And so by an appropriate setting of the parameters gamma and beta, this normalization step, I.e. these four equations, is just computing essentially the identity function, but by choosing other values of gamma and beta, this allows you to make the hidden unit values have other means and variances as well. And so the way you fit this into your neural network is whereas previously you were using these values Z1, Z2, and so on, you would now use Z tilde I instead of zi for the later computations in your neural network. And if you want to put back in this square bracket L, you know, to explicitly denote which layer it is in, you can put it back there. So the intuition I hope you take away from this is that we saw how normalizing the input features X can help learning in a neural network. And what batch norm does is it applies that normalization process not just to the input layer, but to the values even deep in some hidden layer in the neural network. So you apply this type of normalization to normalize the mean and variance of some of your hidden unit's values, Z. But one difference between the training input and these hidden unit values is you might not want your hidden unit values to be forced to have mean 0 and variance 1. For example, if you have a sigmoid activation function, you don't want your values to always be clustered here. You might want them to have a larger variance or have a mean that's different than zero in order to better take advantage of the non linearity of the sigmoid function, rather than have all your values be in just this linear regime. So that's why with the parameters gamma and beta, you can now make sure that your zi values have the range of values that you want. But what it does really, it ensures that your hidden units have standardized mean and variance, where the mean and variance are controlled by two explicit parameters, gamma and beta, which the learning algorithm can set to whatever it wants. So what it really does is it normalizes the mean and variance of these hidden unit values. Really the zi's to have some fixed mean and variance, and that mean and variance could be 0 and 1, or it could be some other value and it's controlled by these parameters gamma and beta. So I hope that gives you a sense of the mechanics of how to implement batch norm, at least for a single layer in a neural network. In the next video, I want to show you how to fit batch norm into a neural network, even a deep neural network, and how to make it work for the many different layers of the neural network. And after that we'll give some more intuition about why batchstorm could help you train your neural networks. So in case why it works still seems a little bit mysterious. Stay with me. And I think in two videos from now we'll really make that clearer.
