In very deep neural networks, gradients shrink or explode exponentially as they backpropagate through layers, depending on whether weight matrices are smaller or larger than 1. A 152-layer model shows why this matters, and careful weight initialization can help, but the problem persists in networks this deep.
One of the problems with training neural.
Networks, especially very deep neural networks, is that of vanishing and exploding gradients. What that means is that when you're training a very deep network, your derivatives or your slopes can sometimes get either very, very big or very, very small, maybe even exponentially small. And this makes training difficult. In this video, you see what this problem of exploding or VA and vanishing gradients really means, as well as how you can use careful choices of the random weight initialization to significantly reduce this problem. Let's say you're training a very deep neural network like this. To save space on this slide, I've drawn it as if you have only two hidden units per layer, but it could be more as well. But this neural network will have parameters W1, W2, W3, and so on up to wl. For the sake of simplicity, let's say we're using an activation function G, so a linear activation function. And let's ignore B. Let's set B of L equals 0. So in that case, you can show that the output Y will be wl times wl minus 1 times wl minus 2, down to w3, w2 w1 times x. But if you want to just check my math, w1 times x is going to be z1, right? Because b is equal to zero. So z1 is equal to, I guess, w1 times x and then plus b, which is zero. But then a1 is equal to g of z1. But because we use a linear activation function, this is just equal to z1. So this first term w1x is equal to a1. And then by similar reasoning, you can figure out that W2 times W1 times X is equal to A2, because that's going to be G of Z2 is going to be G of W2 times A1, right? Which you plug that in here. So this thing is going to be equal to a two. And then, you know, this thing is going to be a three, and so on until the product of all these matrices gives you, um, oh, y hat, not Y. Now, let's say that each of your weight matrices wl, um, is equal to, let's say it's just a little bit larger than one times the identity. So it's 1.5, 1.500, right. Uh, technically the last one has different dimensions. So maybe this is just the rest of these weight matrices. Then Y hat will be, you know, ignoring this last one for a different dimension, It'll be this 1.500, 1.5 mat to the power of L minus 1 times X. Because if we assume that each one of these matrices, you know, is equal to this thing, it's really 1.5 times the identity matrix, then you end up with this calculation. And so Y hat will be essentially 1.5 to the power of L to the power of l l minus 1 times x. And if L is large, for a very deep neural network, Y hat will be very large. In fact, this grows exponentially. It grows like one point to the number of layers. And so if you have a very deep neural network, the value of Y will explode. Now, conversely, if we replace this with 0.5, so something less than 1, then this becomes 0.5 to the power of L, right? This matrix becomes 0.5 to the O minus 1 times X, again ignoring WL. But so if each of your matrices are less than 1, then if, let's say x1, x2 were 1:1, then the activations will be 1/2, 1/2, 1:4, 1:4, 1:8, 1:8, and so on until this becomes right 1 over 2 to the L. So the activation values will decrease exponentially as a function of the depth, as a function of the number of layers l of the network. So there'd be a very deep network. The activations end up decreasing exponentially. So the intuition I hope you can take away is, is that if the weights W, if they're all, you know, just a little bit bigger than one or just a little bit bigger than the identity matrix, then with a very deep network, deactivations can explode. And if W is, you know, just a little bit less identity, right? So if this was maybe 0.9, 0.9, right. Then if a very deep network, deactivations will decrease exponentially. And even though I went through this argument in terms of activations increasing or decreasing exponentially as a function of L, a similar argument can be used to show that the derivatives or the gradients you compute with gradient descent will also increase exponentially or decrease exponentially as a function of the number of layers. With some of the modern neural networks, you actually have L equals 150. Microsoft recently got great results of a, I think 152 layer neural network. Um, but with such a deep neural networks, if your activations, your gradients increase or decrease exponentially as a function of L, then these values could get really big or really small. And this makes training difficult. Especially if your gradients are exponentially small in L, then, you know, gradient descent will take tiny little steps, and it'll take a long time for gradient descent to learn anything.
To summarize, you've seen how deep networks suffer from the problems of vanishing or exploding gradients. In fact, for a long time, this problem was a huge barrier to training deep neural networks. It turns out there's a partial solution that doesn't completely solve this problem, but that helps a lot, which is careful choice of how you initialize the weights. To see that, let's go on to the next video.
