Batch normalization uses running averages of training means and variances at test time instead of computing statistics from single examples. Deep learning frameworks handle this automatically with learned gamma and beta parameters to normalize each test example.
Batch norm processes your data one mini batch at a time. But at test time you might need to process the examples one at a time. Let's see how you can adapt your network to do that. Recall that during training, here are the equations you use to implement batch norm. Within a single mini bash, you sum over that mini bash of the zi values to compute the mean. Um, so here you're just summing over the examples. In one mini bash, I'm using M to denote the number of examples in the mini batch, not not in in the whole training set. Then you compute the variance, and then you compute z norm by scaling by the mean and standard deviation with epsilon added for numerical stability. And then Z tilde is taking z norm and rescaling by gamma and beta. So notice that mu and sigma squared, which you need for this scaling calculation, are computed on the entire mini batch. But at test time you might not have a mini batch of 64, 128 or 256 examples to process at the same time. So you need some different way of coming up with mu and sigma squared. And if you have just one example, taking the mean and variance of that one example doesn't make sense. So what's actually done in order to apply your neural network at test time is to come up with some separate estimate of of mu and sigma squared. And in typical implementations of batch norm, what you do is estimate this using a exponentially weighted average where the average is across the mini batches. So to be very concrete, here's what I mean. Let's pick some layer l and let's say you're going through mini batches x1, x2 together with the corresponding values of y and so on. So when training on X1 for that layer l, you get some mu l, and in fact I'm going to write this as mu for the first mini batch and that layer. And then when you train on the second mini batch for that layer and that mini batch, you end up with some second value of mutation. And then for the third mini batch in this hidden layer, you end up with some third value for mu. So just as we saw how to use an exponentially weighted average to compute a mean of theta 1, theta 2, theta 3. When you are trying to compute a exponentially weighted average of the current temperature, you would do that to keep track of some. So what's the latest average value of this mean vector you've seen? So that exponentially weighted average becomes your estimate for what the mean of the z's is for that hidden layer. And similarly, you use an exponentially weighted average to keep track of these values of sigma squared that you see on the first mini batch in that layer, sigma squared that you see on the second mini batch, and so on. So you keep a running average of the mu and the sigma squared that you're seeing for each layer as you train the neural network across different mini batches. Then finally, at test time, what you do is in place of this equation, you would just compute z norm using whatever value z you have and using your exponentially weighted average of the mu and sigma squared, whatever was the latest value, you have to do the scaling here. And then you would compute z tilde on your one test example using that z norm that we just computed on the left and using the beta and gamma parameters that you you had learned during your neural network training process. So to take away from this is that during creating time, mu and sigma squared are computed on an entire mini batch of, you know, say 64, 128 or some number of examples. But at test time, you might need to process a single example at a time. So the way to do that is to estimate mu and sigma squared from your training set. And there are many ways to do that. Um, you could, in theory, run your whole training set through your final network to get mu and sigma squared. But in practice, what people usually do is implement an exponentially weighted average, where you just keep track of the mu and sigma squared values you're seeing during training and use an exponentially weighted average, also sometimes called a running average, to just get a rough estimate of mu and sigma squared. And then you use those values of mu and sigma squared at test time to do the scaling you need of the hidden unit values z. In practice, this process is pretty robust to the exact way you use to estimate mu and sigma squared. So I wouldn't worry too much about exactly how you do this. And if you're using a deep learning framework, they'll usually have some default way to estimate mu and sigma squared. That should work reasonably well as well. But in practice, any reasonable way to estimate the mean and variance of your hidden unit values, z, should work fine at test sum. So that's it for bash norm. And using it, I think you'll be able to train much deeper networks and get your learning algorithm to run much more quickly. Before we wrap up for this week, I want to share with you some thoughts on deep learning frameworks as well. Let's start to talk about that in the next video.
