Skip connections in ResNets let you add layers without hurting training because the network can learn the identity function easily. Same convolutions preserve dimensions so you can add residual blocks directly, and when dimensions change, a learned projection aligns them for elementwise addition.
Chapters
So why do resnets work so well? Let's go through one example that illustrates why Resnets work so well, at least in the sense of how you can make them deeper and deeper without really hurting your ability to at least get them to do well on the training set. And hopefully, as you've understood from the third course in this sequence, doing well on the training set is usually a prerequisite to doing well on your holdout or on your death or on your test sets. So. So being able to at least train a resnet to do well on a training set is a good first step toward that. Let's look at an example. What we saw in the last video was that if you make a network deeper, it can hurt your ability to train the network to do well on the training set. And that's why sometimes you don't want a network that is too deep. But this is not true, or at least it's much less true when you're training a resnet. So let's go through an example. Let's say you have X feeding in to some big neural network and this outputs some activation al. Let's say for this example that you're gonna modify the neural network to make it a little bit deeper so you the same big nn. And this outputs alpha. And we're going to add a couple extra layers to this network. So let's add one layer there and another layer there and this will output Al +2 only. Let's make this a RESNET block, a residual block with that extra shortcut. And for the sake of argument, let's say throughout this network we're using, um, the RELU activation function. So all the activations are going to be, you know, greater than or equal to zero with the positive exception of the input X. Right? Because the RELU activation outputs numbers that are either zero or positive. Now let's look at what Al +2 will be. To copy the expression from the previous video, Al +2 will be ReLU applied to Z L +2 and then +AL, where this addition of al comes from the short circuit from the skip connection that we just added. And if we expand this out, this is equal to G of WL plus 2 times A of L plus 1 plus BL plus 2. So that's ZL plus 2 is equal to that plus AL. Now notice something. If you're using L2 regularization or weight decay, that will tend to shrink the value of wl plus two. Um, if you're applying weight decay to B, that will also shrink this, although I guess in practice sometimes you do and sometimes you don't apply weight decay to B. But W is really the key term to pay attention to here. And if WL plus 2 is equal to 0, and let's say for the sake of argument that B is also equal to zero, then these terms go away because they're equal to zero. And then G of Al, this is just equal to Al, right? Because we assumed we're using the RELU activation function, and so all the activations are non negative, and so G of Al is the RELU applied to a non negative quantity. So you just get back al. So what this shows is that the identity function is easy for residual block to learn, and it's Easy to get Al +2 equal to Al because of this skip connection. And what that means is that adding these two layers to the neural network, it doesn't really hurt your neural network's ability to do as well as this simpler network without these two extra layers, because it's quite easy for it to learn the identity function to just copy Al to Al +2 using despite the addition of these two layers. And this is why adding two extra layers, adding this residual block to the somewhere in the middle or to the end of this big neural network, it doesn't hurt performance. But of course our goal is to not just not hurt performance, it's to help performance. And so you can imagine that if all of these hidden units, if they actually learn something useful, then maybe you can do even better than learning the identity function. And what goes wrong in very deep plane nets, in very deep networks without these residual, these script connections, is that when you make the network deeper and deeper, it's actually very difficult for it to choose parameters that learn even the identity function, which is why a lot of layers end up making your result worse rather than making your result better. And I think the main reason the residual network works is that it's so easy for these extra layers to learn the residual, to learn the identity function, that you're kind of guaranteed that it doesn't hurt performance. And then, you know, a lot of time you maybe get lucky and it even helps performance, or at least it's easier to go from a decent baseline of not hurting performance, and then gradient descent can only improve the solution from there. So one more detail in the residual network that's worth discussing, which is through this addition here, we're assuming that ZL plus 2 and AL have the same dimension. And so what you see in Resnet is a lot of use of same convolutions, so that the dimension of this is equal to the dimension, I guess, of this layer or of the output layer. So you can actually do this short circuit connection because the same convolution preserves dimensions. And so it makes it easier for you to, you know, carry out, to have this short circuit and then carry out this addition of two equal dimension vectors, um, in case the input and output have different dimensions. So for example, if this is 128 dimensional and Z, or therefore Al is 256 dimensional, as an example, what you would do is add an extra matrix, we call that WS over here. And WS in this example would be a 2, 5, 6 by 128 dimensional matrix. So then WS times AL becomes 2, 56 dimensional and this addition is now between 2, 2 5, 6 dimensional vectors. And there are a few things you could do with ws. It could be a matrix of parameters to be learned. It could be a fixed matrix that just implements zero padding. So it takes Al and then zero pads it to be 256 dimensional. And, and either of those versions, I guess, could work. So finally, let's take a look at resnets on images. So these are images I got from the paper by her et al. Um, this is an example of a plane network. And in, in which you input an image and then have a number of conv layers until eventually you have a softmax output at the end. To turn this into a resnet, you add those extra skip connections. And I just mentioned a few details. There are a lot of three by three convolutions here. And all these, most of these are three by three same convolutions. Um, and that's why you're adding your equal dimension feature vectors. So rather than a fully connected layer, these are actually convolutional layers, but because they're same convolutions, the dimension is preserved. And so the ZL Plus 2 Plus AL addition makes sense. And similar to what you've seen in a lot of networks before, you have a bunch of convolutional layers and then there are occasionally pooling layers as well, or pooling, or pooling like layers. And whenever one of those things happen, then you need to make an adjustment to the dimension which we saw on the previous slide, you can do of the matrix ws. And then as is common in these networks, you have confconf conf and then at the end you know, have a fully connected layer that then makes a prediction using a softmax. So that's it for Resnets. Next, there's a very interesting idea behind using neural networks with one by one filters, one by one convolutions. So what good is a one by one convolution? Let's take a look at the next video.
