Fitting Batch Norm Into Neural Networks (C2W3L05)

So you've seen the equations for how to implement batch norm for maybe a single hidden layer. Let's see how it fits into the training of a deep network. So let's say you have a neural network like this. You've seen me say before that you can view each hidden unit as computing two things. First it computes Z and then it applies the activation function to compute A. And so you can think of each of these circles as representing a two step computation. And similarly for the next layer, that's z21 and a21 and so on. So if you were not applying batch norm, you would have an input X feed into the first hidden layer and then first computer z1, and this is governed by the parameters w1 and b1. And then ordinarily you would feed z1 into the activation function to compute a1. But what we'll do in batch norm is take this value z1 and apply batch norm, sometimes abbreviated BN, to it. And that's gonna be governed by parameters beta1 and gamma1. And this will give you this new normalized value z1. And then you feed that to the activation function to get a1, which is g1 applied to z tilde1. Now you've done the computation for the first layer where this batch norm step really occurs, you know, in between the computation from Z and A. Next you take this value A1 and use it to compute Z2. And so this is now governed by W2B2. And similar to what you did for the first layer, you would take Z2 and apply it through batch norm. I'm gonna abbreviate that bn. Now this is governed by batch norm parameters specific to the next layer. So beta 2, gamma 2, and now this gives you z tilde 2, and you use that to compute a 2 by applying the activation function and so on. So once again, the batch norm step, you know, kind of happens between computing Z and computing A. And the intuition is that instead of using the unnormalized value Z, you're gonna use the normalized value z tilde that's in the first layer and second layer as well. Instead of using the unnormalized value z 2, you're gonna use the mean and variance normalized values z tilde 2. So the parameters of your network are going to be W1, B1. It turns out we'll get rid of the B parameters, but we'll see why in the next slide. But for now, um, imagine your parameters are the usual W1, B1 through WL, BL and we've added to this neural network additional parameters beta 1, gamma 1, beta 2, gamma 2 and so on. You know, for each layer in which you are applying batch norm, um, for clarity, note that these betas here, these have nothing to do with the hyperparameter beta that we had for momentum over the computing the various exponentially weighted averages. You know, the authors of the ADAM paper had used beta in their paper to denote that hyperparameter. The authors of the BatchNorm paper had used beta to denote this parameter. But these are two completely different betas. I decided to stick with beta in both cases, in case you read the original papers. But the beta one, beta two and so on that bash norm tries to learn is a different beta than the hyperparameter beta used in momentum and in the Adam and RMSprop algorithms. But so now that these are the new parameters of your algorithm, you would then use whatever optimization you want, such as gradient descent, in order to implement it. So, for example, you might compute D beta l for a given layer and then update the parameters. Beta gets updated as beta minus the learning rate times D beta L. And you can also use Adam or RSprop or Momentum in order to update the parameters beta and gamma, not just gradient descent. And even though in the previous video I'd explained what the batch norm operation does, computes mean and variances and subtracts and divides by them, if you're using a deep learning programming framework, usually you won't have to implement the batch norm step or the batch norm layer yourself. Um, in some of the programming frameworks it can be sort of one line of code. So for example, in the TensorFlow framework, you can implement batch normalization with, you know, this function. We'll talk more about programming frameworks later, but in practice, you might not end up needing to implement all these details yourself. But it's still worth knowing how it works so that you can get a better understanding of what your code is doing. But implementing batch norm is often, you know, something like one line of code in the deep learning frameworks. Now, so far we've talked about batch norm as if you were training on your entire training set at a time, as if you were using batch gradient descent. In practice, batch norm is usually applied with mini batches of your training set. So the way you actually apply batch norm is you take your first mini batch and compute Z1, same as we did on the previous slide, using the parameters W1, B1, and then you take this, just this mini batch and compute the mean and variance of the z ones on just this mini batch, and then batch norm would subtract by the mean and divide by the standard deviation and then rescale by beta 1 gamma 1 to give you Z1. And all this is on the first mini batch. Then you apply the activation function, you know, to get A1, and then you compute Z2 using W2, B2 and so on. So you do all this in order to perform one step of say, gradient descent on the first mini batch and then you go to the second mini batch x2 and you do something similar where you now compute z1 on the second mini batch and then use batch norm to compute z1 tilde. And so here in this batch norm step, you'd be normalizing Z tilde using just the data in your second mini batch. So this batch norm step here is looking at the examples in your second mini batch, computing the mean and variances of the Z1s on just that mini batch and then rescaling by beta and gamma to get z tilde and so on. And you do this with the third mini batch and keep training. Now there's one detail to the parameterization that I want to clean up, which is previously I said that the parameters was wl BL for each layer, as well as beta l and gamma l. Now notice that the way zl is computed is as follows. Zl equals wl times A of L minus 1 plus B of L. But what batch norm does is it's going to look at the mini batch and normalize zl to first have mean zero and standard variance and then a rescale by beta and gamma. But what that means is that whatever is the value of BL is actually gonna just get subtracted out. Because during that batch norm normalization step, you're going to compute the means of the zls and then subtract out the mean. And so adding any constant to all of the examples in a meme bash, it doesn't change anything because any constant you add will get cancelled out by the mean subtraction. So step so if you're using batch norm, you can actually eliminate that parameter, or if you want, think of it as setting it permanently to zero. So then the parameterization becomes zl is just wl times al minus one. And then you compute zl normalized. And when you compute z tilde equals gamma zl plus beta, you end up using this parameter beta l in order to decide what's the mean of z tilde l, which is what gets passed to the next layer. So just to recap because batch norm zeroes out the mean of these ZL values in the layer, there's no point having this parameter bl, and so you might as well get rid of it and instead it's sort of replaced by beta L, which is a parameter that controls that ends up affecting the shift or the bias terms. Finally, remember that the dimension of zl, because if you're doing this on one example is going to be nl by 1, and so bl had dimension nl by 1, if nl is the number of hidden units in layer l. And so the dimension of beta l and gamma L is also going to be nl by one, because that's the number of hidden units you have. You have nl hidden units. And so beta l and gamma L are used to scale the mean and variance of each of the hidden units to whatever the network wants to set them to. So let's put it all together and describe how you can implement gradient descent using bash norm. Assuming you're using mini bash gradient descent, you iterate For t equals 1 to the number of mini batches you would implement forward prop on mini batch XT and and doing forward prop in each hidden layer, use batch norm to replace zl with z l. And so this ensures that within that mini batch the value Z end up with some normalized mean and variance and the values. And the version with the normalized mean and variance is this Z tilde l. And then you use backprop to compute DW DB for all the values of L, d, beta D gamma. Although technically, since we've gotten rid of B, this actually now goes away. And then finally you'd update the parameters. So you know, W gets updated as W minus the learning rate times this, as usual, beta gets updated as Beta minus the learning rate times db. And similarly for gamma and um, if you've computed the gradients as follows, you could use gradient descent. That's what I've written down here. But this also works with gradient descent with momentum or RMS PROP or adam, where instead of taking this gradient descent update, you could use the updates given by these other algorithms, as we discussed in the previous week's videos. So these other optimization algorithms as well can be used to update the parameters beta and gamma, that batch norm added to your algorithm. So I hope that gives you a sense of how you could implement batch norm from scratch if you wanted to. If you're using one of the deep learning programming frameworks, which we'll talk more about later, hopefully you can just call someone else's implementation in the primary framework, which will make using Batch Norm much easier. Now, in case BatchNorm still seems a little bit mysterious, if you're still not quite sure why it speeds up training so dramatically, let's go to the next video and talk more about why BatchNorm really works and what it's really doing.