Andrew Ng explains how to normalize neural network inputs by subtracting the mean and dividing by standard deviation. Normalized features create more spherical cost contours, which speeds up gradient descent and makes training more stable. Always use the training set's mean and standard deviation to normalize test data.
When training a neural network, one of the techniques that will speed up your training is if you normalize your inputs. Let's see what that means. Let's see if a training set with two input features. So the input features X are two dimensional. And here's a scatter plot of your training set. Normalizing your inputs corresponds to two steps. The first is to subtract out or to zero out the mean. So you set mute equals 1 over m sum over I of xi. So this is a vector. And then x gets set as x minus mu for every training example. So this means you just move the training set until it has zero mean. And then the second step is to normalize the variances. So notice here that the feature x1 has a much larger variance than the feature x1 x2 here. So what we do is set sigma equals 1 over m sum of xi2, because this is element y squaring. And so now sigma squared is a vector with the variances of each of the features. And notice we've already subtracted out the mean. So xi squared, element Y squared is just the variances. And you take each example and divided by you know this vector sigma squared. And so in pictures you end up with this, where now the variance of x1 and x2 are both equal to 1. Oh, and one tip. If you use this to scale your training data, then use the same mu and sigma squared to normalize your test set, right? In particular, you don't want to normalize the training set and the test set differently. Whatever this value is and whatever this value is, use them in, you know, these two formulas so that you scale your test set in exactly the same way, rather than estimating mu and sigma squared separately on your training set and test set. Because you want your data, both training and test examples, to go through the same transformation defined by the same mu and sigma squared calculated on your training data. So why do we do this? Why do we want to normalize the input features? Recall that the cost function is defined as written on the top, right? It turns out that if you use unnormalized input features, it's more likely that your cost function will look like this, like a very squished out bow, a very elongated cost function where, you know, the minimum you're trying to find is maybe over there. But if your features are on very different scales, say the feature X1 ranges from 1 to 1000 and the feature X2 ranges from the 0 to 1, then it turns out that the ratio or the range of values for the parameters W1 and W2 will end up taking on very different values. And so maybe these axes should be W1 and W2. But intuition, I'll plot W and B. Your cost function can be a very elongated bow like that. So if you plot the contours of this function, you can have a very elongated function like that. Whereas if you normalize the features, then your cost function will on average look more symmetric. And if you are running gradient descent on the cost function like the one on the left, then you might have to use a very small learning rate, because if you're here, you know, then gradient descent might need a lot of steps to oscillate back and forth right before it finally finds its way to the minimum. Whereas if you have a more spherical contours, then wherever you start, gradient descent can be pretty much go straight to the minimum. You can take much larger steps for gradient descent need rather than needing to oscillate around like the picture on the left. Of course, in practice W is a high dimensional vector. Um, and so trying to plot this in 2D doesn't convey all the intuitions correctly, but the rough intuition that your cost function will be, you know, more round and easier to optimize when your features are all on similar scales. Not all, not from 1 to 1,000, 0 to 1, but mostly from, you know, minus 1 to 1 or with about similar variance as each other. That just makes your cost function J easier and faster to optimize. Um, in practice, if one feature, say x1 ranges from 0 to 1 and x2 ranges from minus 1 to 1 and x3 ranges from 1 to 2, you know, these are fairly similar ranges. So this will work just fine is when they're on dramatically different ranges, like one is from one to 1,000 and another from zero to one. That that really hurts your optimization algorithm. But by just setting all of them to a zero mean and say variance one, like we did in the last slide, that just guarantees that all your features are a similar scale and will usually help your learning algorithm run faster. So if your input features came from very different scales, maybe some features are from zero to one, some from one to a thousand, then it's important to normalize your features. Um, if your features came in on similar scales and this step is less important, although performing this type of normalization pretty much never does any harm. So I'll often, you know, do it anyway if I'm not sure sure whether or not it'll help with speeding up training for your algorithm. So that's it for normalizing your input features. Next, let's keep talking about ways to speed up the training of your neural network.
