Lecture segment (Andrew Ng) introducing ResNets (residual networks). Explains residual blocks and skip/shortcut connections that add an earlier activation to a later layer before the ReLU, enabling training of very deep networks (e.g., >100 layers) by mitigating vanishing/exploding gradients. Describes construction (stacking residual blocks), contrasts plain networks vs ResNets on training error, and notes an associated programming exercise.
Very, very deep neural networks are difficult to train because of vanishing and exploding gradients, types of problems. In this video, you learn about skip connections, which allows you to take the activation from one layer and suddenly feed it to another layer even much deeper in a neural network. And using that, you would have built resnet, which enables you to train very, very deep networks, sometimes even networks of over a hundred layers. Let's take a look. Resnets are built out of something called a residual block. Let's first describe what that is. Here are two layers of a neural network where you start off with some activation in layer L. Then you go to Al +1, and then deactivation two layers later is Al +2. So to go through the steps in this computation, you have Al, and then the first thing you do is you apply this, um, linear operator to it, which is governed by this equation to go from Al to compute zl plus one by multiplying by the weight matrix and adding that bias vector. After that you apply the RELU nonlinearity to get, um, Al +1. And that's governed by this equation where Al +1 is G of ZL +1. Then in the next layer, you apply this linear step again. So it's governed by that equation, right? So this is quite similar to this equation we saw on the left. And then finally, you apply another RELU operation which is now governed by that equation where G here would be the relu nonlinearity. And this gives you, you know, Al +2. So in other words, for information from Al to flow to Al +2, it needs to go through all of these steps, which I'm going to call the main path of this set of layers in a residual net. We're going to make a change to this. We're going to take alpha and just fast forward it, copy it much further into the neural network to here and just add Al before applying to nonlinearity the relu nonlinearity. And I'm gonna call this the shortcut. So rather than needing to follow the main path, the information from Al can now follow a shortcut to go much deeper into the neural network. And what that means is that this loss equation goes away and we instead have that the output Al +2 is the ReLU nonlinearity G applied to ZL +2 as before, but now +Al. So the addition of this Al here, it makes this a residual block. And in pictures, you can also modify this picture on top of, by drawing this extra shortcut to go here. And we're gonna draw it as going into this second layer Here, because the shortcut is actually added before the RELU nonlinearity. So each of these nodes here, right, that applies a linear function and a relu. So Al is being injected after the linear part, but before the RELU part. And sometimes instead of the terms shortcut, you also hear the term skip connection. And that refers to Al just skipping over a layer or kind of skipping over almost two layers in order to pass this information deeper into the neural network. So what the inventors of resnet, so that would be Kai Minghe, Xiang Yujiang, Xiao Qingren and Jian sun, what they found was that using residual blocks allows you to train much deeper neural networks. And the way you build a resnet is by taking many of these residual blocks, blocks like these, and stacking them together to form a deep network. So let's look at this network. This is not a residual network. Let's call this a plane network. This is the terminology of the RESNET paper. To turn this into resnet, what you do is you add all those skip connections, although those short circuit connections like so, so every two layers ends up with that additional change that we saw on the previous slide to turn each of these into a residual block. So this picture shows five residual blocks stacked together. And this is a residual network. And it turns out that if you use, you know, a standard optimization algorithm, such as gradient descents or one of the fancy optimization algorithms to train a plane network. So without all the extra residual, without all the extra shortcuts or skip connections I just drew in empirically, you find that as you increase the number of layers, the training error will tend to decrease after a while, but then they'll tend to go back up. Um, and in theory, as you make a neural network deeper, you know, it should only do better and better on the training set, right? So theory, in theory, having a deeper network should only help. But in practice, on reality, having a plane network, so not a resnet, but having a plane network that's very deep means that your optimization algorithm just has a much harder time training. And so in reality, your training error gets worse if you pick a network that's too deep. But what happens with resnets is that even as the number of layers gets deeper, you can have the performance of the training error kind of keep on going down. You know, even if we train a network with over 100 layers, um, and then now some people experimenting with networks that are over 1000 layers, although I don't see that used much in practice yet. But by taking these activations, be it X or these intermediate activations and allowing it to go much deeper in the neural network. This really helps with the vanishing and exploding gradient problems, and allows you to train much deeper neural networks without really appreciable loss in performance. And maybe at some point this will plateau, this will flatten out, and it doesn't help that much the deeper and deeper networks. But resonance has certainly been effective at helping train very deep networks. So you've now gotten an overview of how Resnets work, and in fact, in this week's programming exercise, you get to implement these ideas and see it work for yourself. But next I want to share with you better intuition, or even more intuition about why Resnets work so well. Let's go on to the next video.
