C4W1L11 Why Convolutions

For this final video for this week, let's talk a bit about why convolutions are so useful when you include them in your neural networks. And then finally, let's briefly talk about how to put this all together and how you could train a convolutional neural network when you have a label training set. I think there are two main advantages of convolutional layers over just using fully connected layers. And the advantages are parameter sharing and sparsity of connections. Let me illustrate an example. Let's say you have a 32 by 32 by 3 dimensional image, and this actually comes from the example from the previous video. But let's say you use a 5x5 filter with 6 filters. And so this gives you a 28 by 28 by 6 dimensional output. So 32 by 32 by 3 is 3072, and 20 by 20 by 6, if you multiply all those numbers is 4704. And so if you were to create a neural network with 3,072 units in one layer, and, you know, 4,704 units in the next layer, and if you were to connect every one of these neurons, then the weight matrix, the number of parameters in the weight matrix would be 3,072 times 4,704, which is about 14 million. So that's just a lot of parameters to train. And you know, today you can train neural networks with even more parameters than 14 million. But considering that this is just a pretty small image, this is a lot of parameters to train. And of course, if this were to be a thousand by thousand image, then this weight matrix would just become infeasibly large. But if you look at the number of parameters in this convolutional layer, each filter is 5 by 5. So each filter has 25 parameters plus a bias parameter. Means you have 26 parameters per filter and you have 6 filters. So the total number of parameters is that which is equal to 156 parameters. And so the number of parameters in this conf layer remains quite small. And the reason that a confnet has relatively small parameters is really two reasons. One is parameter sharing and, and parameter sharing is motivated by the observation that a feature detector, such as vertical edge detector, that's useful in one part of the image is probably useful in another part of the image. And what that means is that if you've figured out, say a three by three filter for detecting vertical edges, you can then apply the same 3 by 3 filter over here and in the next position over and in the next position over and so on. And so each of these, you know, feature detectors, each of these outputs can use the same parameters in lots of different positions in your input image in order to detect, say, a vertical edge or some other feature. And I think this is true for low level features like edges, as well as for higher level features like maybe detecting the eye that indicates a face or a cat or something is there. But being able to share in this case the same nine parameters, that to compute all 16 of these outputs is one of the ways the number of parameters is reduced. And it also just seems intuitive that a feature detector, like a vertical edge detector, computed for the upper left hand corner of the image, the same feature seems like it will probably be useful, has a good chance of being useful for the lower right hand corner of the image. So maybe you don't need to learn separate feature detectors for the upper left and the lower right hand corners of the image. And maybe you do have a data set where, you know, the upper left hand corner and the lower right hand corner have different distributions. So they maybe look a little bit different, but they might be similar enough, they're sharing feature detectors in all across the image works just fine. The second way that confnets get away with having relatively few parameters is by having sparse connections. So here's what I mean. If you look at this zero, this is computed via 3 by 3 convolution. And so it depends only on this 3 by 3 input grid of cells. So it's as if this output unit on the right is connected only to 9 out of these 6 by 636 input features. And in particular the rest of these pixel values, you know, all of these pixel values, all of these pixel values do not have any input on this, do not have any effect on that output. So that's what I mean by sparsity of connections. As another example, this output depends only on these nine input features. And so it's as if only those nine input features are connected to this output and the other pixels just don't affect this output at all. And so through these two mechanisms, a neural network has a lot fewer parameters, which allows it to be trained with smaller training sets, and is less prone to be overfitting. And sometimes you also hear about convolutional neural networks being very good at capturing translation invariance. And that's the observation that a picture of a cat shifted a couple pixels to the right is still pretty clearly a cat. And the convolutional structure helps the neural network encode the fact that an image shifted a few pixels should result in pretty similar features and should probably be assigned the same output label. And the fact that you're applying the same filter, you know, it's all the positions of the image, both in the earlier layers and in the later layers, that helps a neural network automatically learn to be, you know, more robust to capture, to better capture this desirable property of translation invariance. So these are maybe a couple of reasons why convolutions or convolutional neural networks work so well in computer vision. Finally, let's put it all together and see how you can train one of these networks. Let's say you want to build a cat detector and you have a labeled training set as follows. Where now X is, is a image, and the y's can be binary labels or one of k classes. And let's say you've chosen a convolutional neural network structure, maybe starting the image and then having, you know, convolutional and pooling layers and then some fully connected layers followed by a softmax output that then outputs Y hat. The conf layers and the fully connected layers will have various parameters W as well as biases B. And so any setting of the parameters therefore lets you define a cost function similar to what we had seen in the previous courses, where with randomly initialized parameters W and B, you can compute the cost J as the sum of losses of your neural networks predictions on your entire training set, maybe divide it by M. So to train this neural network, all you need to do is then use gradient descent or some other algorithm like gradient descent with Momentum or RMSprop or Adam or something else in order to optimize all the parameters of the neural network to try to reduce the cost function J. And you find that if you do this, you can build a very effective CAT detector or some other detector. So, congratulations on finishing this week's videos. You've now seen all the basic building blocks of a convolutional neural network and how to put them together into an effective image recognition system. In this week's programming exercises, I think all of these things will become more concrete and you get a chance to practice implementing these things yourself and seeing it work for yourself. Next week, we'll continue to go deeper into convolutional neural networks. I mentioned earlier that there's just a lot of hyperparameters in convolutional neural networks. So what I want to do next week is show you a few concrete examples of some of the most effective convolutional neural networks, so you can start to recognize the patterns of what types of network architectures are effective. And one thing that people often do is just take the architecture that someone else has found and published in a research paper and just use that for your application. And so, by seeing some more concrete examples next week, you also learn how to do that better. And beyond that, next week we'll also just give better intuitions about what makes ConvNet work well. And then in the rest of the course, we'll also see a variety of other computer vision applications, such as object detection and neural style transfer, how to create new forms of artwork using these types of boundaries. So that's it for this week. Best of luck with the homeworks, and I look forward to seeing you next week.