Proper weight initialization prevents vanishing and exploding gradients in deep networks. He initialization uses variance 2/n for ReLU, Xavier initialization uses 1/n for tanh. Scale weights as W = np.random.randn(shape) sqrt(constant / n), where n is the number of inputs.
In the last video, you saw how very deep neural networks can have the problems of vanishing and exploding gradients. It turns out that a partial solution to this doesn't solve it entirely, but helps a lot is better or more careful choice of the random initialization for your neural network. To understand this, let's start with the example of initializing the weights for a single neuron. And then we'll go on to generalize this to a deep network.
Let's go through this with an example with just a single neuron, and then we'll talk about a deep net later. So for a single neuron, you might input four features, X1 through X4, and then you have some A equals G of Z and then it outputs some y. And later on for a deeper net, you know, these inputs will be right some layer A of L. But for now, let's just call this X for now. So Z is going to be equal to w1x1 plus w2x2 plus dot dot dot plus I guess wn xn and. And let's set b equals zero. So, you know, let's just ignore B for now. So in order to make Z not blow up and not become too small, you notice that the larger n is, the smaller you want WI to be, right? Because Z is a sum of WI xi. And so if you're adding up a lot of these terms, you want each of these terms to be smaller. One reasonable thing to do would be to set the variance of wi to be equal to 1 over n, where n is the number of input features that's going into a neuron. So in practice, what you can do is set the weight matrix W for a certain layer to be np.random.randn, you know, and then whatever the shape of the matrix is, it fills this out here and then times square root of one over the number of features that are fed into each neuron in layer L. So it's gonna be N of L minus 1, because that's the number of units that are feeding in to each of the units in layer L. It turns out that if you're using a relu activation function, then rather than 1 over n, it turns out that setting the variance to 2 over n works a little bit better. So you often see that in initialization, especially if you're using a relu activation function. So. So if gl of Z is relu of Z. Oh, and depending on how familiar you are with random variables, it turns out that sampling a Gaussian random variable and then multiplying it by square root of this that sets the variance to to be equal to this thing to be 2 over N. Okay. And the reason I went from n to this n superscript L minus 1 was in this example with logistic regression, which is at n input features. But in more general case, layer L would have n l minus 1 inputs each of the units in that layer. So if the input features or input activations are roughly mean 0 and standard variance and variance 1, then this will cause Z to also be taken on a similar scale. And this doesn't solve, but it definitely helps reduce the vanishing and exploding gradients problem because it's trying to set each of the weight matrices W, you know, so that it's not too much bigger than one and not too much less than one so it doesn't explode or vanish too quickly. I'll just mention some other variants. The version we just described is assuming a RELU activation function and is by a paper by he et al. A few other variants. If you are using a TANH activation function, then there's a paper that shows that instead of using the constant two, it's better to use the constant one and so one over this instead of two. And so you multiply by the square root of this. So this square root term would replace this term and, and you use this if you're using a TANH activation function. This is called a Xavier initialization. And another version worked out by Yoshi Benjo and his colleagues you might see in some pavers, but is to use this formula, which has some other theoretical justification. Um, but I would say if you're using a RELU activation function, which is really the most common activation function, I would use this formula. Um, if you're using tanh, you could try this version instead. And some authors will also use this. But in practice, I think all of these formulas just give you a starting point. It gives you a default value to use for the variance of the initialization of your weight matrices if you wish the variance here. This variance parameter could be another thing that you could tune with your hyperparameters. So you could have another parameter that multiplies into this formula and tune that multiplier as part of your hyperparameter search. Sometimes the tuning that hyperparameter has a modest size effect. It's not one of the first hyperparameters I would usually try to tune, but I've also seen some problems where tuning this, you know, helps helps a reasonable amount. But this is usually lower down for me in terms of how important it is relative to the other hyperparameters you could tune. So I hope that gives you some intuition about the problem of vanishing or exploding gradients, as well as how choosing a reasonable scaling for how you initialize the weights. Hopefully that makes your weights you're not explode too quickly and not decay to zero too quickly. So you can train a reasonably deep network without the weights or the gradients exploding or vanishing too much. When you train deep networks, this is is another trick that will help you make your neural networks train much more quickly.
