Initializing neural network weights to zero breaks training because hidden units compute identical functions and stay identical. Use random Gaussian initialization instead, like W = np.random.randn(...) 0.01, while biases can stay zero. Large initial weights saturate sigmoid and tanh, slowing learning.
Questions this video answers
n0 = 2 (inputs); n1 = 2 (hidden units).
W1 is a 2 × 2 matrix.
Random Gaussian (e.g., np.random.randn(...) multiplied by a small constant such as 0.01).
Yes — biases (B) can be initialized to zeros; the symmetry problem concerns W, not B.
Chapters
When you train your neural network, it's important to initialize the weights randomly. For logistic regression, it was okay to initialize the weights to zero, but for a neural network of initialize the weights, the parameters to all zero and then apply gradient descent, it won't work. Let's see why. So you have here two input features. So n0 is equal to 2 and two hidden units. So n1 is equal to 2. And so the matrix associated with the hidden layer or W1 is going to be 2 by 2. Let's say that you initialize it to all zeros, so 0, 0, 002 by 2 matrix, and let's say B1 is also equal to 0, 0. Turns out initializing the bias terms B to 0 is actually okay, but initializing W to all zeros is a problem. So the problem with this form of initialization is that for any example you give it, you have that a 11 and a 12 will be equal, right? So this activation and this activation will be the same because both of these hidden units are computing exactly the same function. And then when you compute back propagation, it turns out that DZ11 and DZ12 will also be the same kind of by symmetry, right? Both of these hidden units will initialize the same way. Technically, for what I'm saying, I'm assuming that the outgoing weights are also identical, so that W2 is equal to. But if you initialize the neural network this way, then this hidden unit and this hidden unit are completely identical. So they're completely. Sometimes you say they're completely symmetric, which just means that they're computing exactly the same function. And by kind of a proof, by induction, it turns out that after every single iteration of training, your two hidden units are still computing exactly the same function. This is possible to show that DW will be a matrix that looks like this, where every row takes on the same value. So when you perform a weight update, so you perform. When you perform a weight update, W1 gets updated as W1 minus alpha times DW. You find that W1 after every iteration will have the first row equal to the second row. So. So it's possible to construct a proof by induction that if you initialize all the weights, all the values of w to 0, then because both hidden units start off computing the same function, and both hidden units have the same influence on the output unit, then after one iteration, that same statement is still true, the two hidden units are still symmetric. And therefore, by induction, after two iterations, three iterations, and so on, no matter how long you train the neural network, both hidden units are still computing exactly the same function. And so in this case, there's really no point to having more than one hidden unit, because they're all computing the same thing. And of course, for larger neural networks, let's say you have three features and maybe a very large number of hidden units. A similar argument works to show that with a neural network like this, because I won't draw on all the edges, if you initialize the weights to zero, then all of your hidden units are symmetric. And no matter how long you run gradient descent, they'll all continue to compute exactly the same function. So that's not helpful because you want the different hidden units to compute different functions. The solution to this is to initialize your parameters randomly. So here's what you do. You can set W1 equals NP random RandN. This generates a Gaussian random variable 2, 2. And then usually you multiply this by a very small number, such as 0.01. So you initialize it to very small random values, and then B, it turns out that B does not have this symmetry problem, what's called the symmetry breaking problem. So it's okay to initialize B to just zeros, because so long as W is initialized randomly, you start off with the different hidden units computing different things, and so you no longer have this symmetry breaking problem. And then similarly for W2, you can initialize that randomly, and B2, you can initialize that to zero. So you might be wondering, you know, where did this constant come from and why is it 0.01? Why not with the number 100 or 1000? Turns out that we usually prefer to initialize the weights to very, very small random values. Because if you're using a, say, tanh or sigmoid activation function, or if you have a sigmoid even just at the output layer, if the weights are too large, then when you compute the activation values, remember that z1 is equal to w1x plus b, and then a1 is the activation function applied to z1. So if w is very big, Z will be very big, or at least some values of Z will be either very large or very small. And so in that case, you're more likely to end up at these flat parts of the the tanh function or the sigmoid function, where the slope or the gradient is very SL small, meaning that gradient descent would be very slow, and so learning would be very slow. So just to recap, if W is too large, you're more likely to end up even at the very start of training with very large values of z, which causes your TANH or sigma activation function to be saturated, um, thus slowing down learning. If you don't have any sigmoid or TANH activation functions throughout your neural network, this is less of an issue. But if you're doing binary classification and your output unit is a sigmoid function, then you just don't want the initial parameters to be too large. So that's why multiplying by 0.01 would be something reasonable to try, or any other small number. And same for W2, right? This can be a random dot randn. I guess this would be one by two in this example, times 0.01. Oh, missing a s there. So finally, it turns out that sometimes there can be better constants than 0.01 when you're training a neural network with just one hidden layer. This is a relatively shallow neural network without too many hidden layers. Setting it to 0.01 will probably work okay, but when you're training a very, very deep neural network, then you might want to pick a different constant than 0.01. And in next week's material, we'll talk a little bit about how and when you might want to choose a different constant than 0.01. But either way, it will usually end up being a relatively small number. So that's it for this week's videos. You now know how to set up a neural network with a hidden layer. Initialize the parameters, make predictions using forward prop, as well as compute derivatives and implement gradient descent using backprop. So with that, you should be able to do the quizzes as well as this week's programming exercises. Best of luck with that. I hope you have fun with the programming exercise and look forward to seeing you in the week. Four Materials.
