The video provides a comprehensive walkthrough of convolutional neural networks (CNNs) and a detailed examination of the AlexNet architecture. It begins by recapping the basic CNN pipeline: an input image passes through successive convolutional layers, each followed by a ReLU activation and pooling, with depth increasing to capture progressively complex features—from simple edges to textures and object parts. The speaker explains kernels as learnable filters, illustrating vertical and horizontal edge detectors, and shows how ReLU zeroes out negative responses, highlighting the role of max‑pooling (including overlapping pooling) in spatial down‑sampling. The discussion then shifts to AlexNet, describing its 5 convolutional layers (kernel sizes, number of filters per layer) and 3 fully‑connected layers (4096 neurons in the first two, 1000‑class output). Key architectural choices such as overlapping max‑pooling, local response normalization (LRN), and the exclusive use of ReLU are covered, with comparative results showing ReLU achieving a 25 % error rate six times faster than tanh. The speaker contrasts LRN with modern batch normalization, explains why dropout is applied only to fully‑connected layers, and reports AlexNet’s top‑1 (≈37 %) and top‑5 (≈18 %) error rates. The latter part of the session addresses practical training concerns: detecting overfitting via separate training, validation, and test splits; monitoring loss curves; and briefly touches on optimizers, epochs, and other techniques (e.g., different pooling strategies) that can improve accuracy.
Chapters
So basically, the last day also we gave a brief description of conditional neural networks. So today is a recapitulation with some visualization. So we know that the— basically the structure. So basically, this is the structure. So basically, this is the structure of architecture of convolutional neural networks. So there is an input of the image, input array, and there are a convolutional layer followed by an activation function such as ReLU and pooling. And there are consecutive such layers, and each is of increasing depth. Generally, each is of increasing depth so that the first layer generally extracts the features, like the very basic features like edges. Then gradually it tries to detect more complex patterns. And finally, ultimately, after the convolutional layers, the output of the convolutional layer is flattened into an array, and then it is fed to the fully connected layer. So basically, if I talk about the kernel, So basically, if I talk about the kernel, so the kernels is basically like a filter. So the more number of kernels are there, each kernel tries to detect a certain kind of pattern, right? Okay, so here if you see this image, so if you see this image image basically. So like a kernel, for example, to detect the vertical lines, so how shall the kernel look like? So it will be a— for a 2D, for a 2D image, for, for here is a grayscale image, so you can represent it in a 2D array. So basically for here you can see the array representation for vertical lines will be— for the kernel will be like -1, -2, -1, 0, 0, 0, 1, 2, 1 in the column-wise direction. So if you see here, if you see here, you can see that there is some sort of vertical line. So here, when it sums up, when it sums up, you are getting a positive value. And if you see further Okay, so if you see, when we are gradually moving it, we are getting positive values, but where there are no vertical lines, you will generally see there is negative values coming, right? So what happens is that when we pass it through the ReLU function where there are positive values, so basically you can detect that where there are positive values, there is certain a vertical line present there. So, and so that will be given by that value. But where you are getting negative values with the ReLU function, that is converting into zeros. So zero means there is no intensity, right? So in those places, you are— the filtered image will look like this. Like here, after passing through the ReLU, you will see zero. So there is no intensity. And in these places, generally there are some positive values. So there are certain, you can say, some vertical lines. So in this way, for example, for a horizontal curve, for representing a horizontal kernel, you can just transpose this matrix. Like, in the middle there will be zeros, in the second row there will be zeros, and above and in the first row and the third row there will be some values. So the pattern will be of that sort. So while filtering, it will happen like this. Now, as I said, the first layer is generally not that complex, so it detects very minute edges, like very— like just edges, vertical lines, sharp, sharpened lines, outlines. And the next layer, we try to detect the textures using those edges with more complex kernels. We detect the texture followed by patterns, and gradually we get the image. So the main thing is we— why we do so is that we have to just detect what the object is. Like, what is the object? We don't care about the very much of internal details. Like, if it is a cat, it should be a cat, but we are not very much focused on the very much details, like exactly how the cat looks. So the basic structure is enough to detect that, yes, this is a cat. And so the next time, just going to discuss the AlexNet paper. So AlexNet paper, yeah. So this basically is the paper which was published in 2006. 2010, I guess. So for, you know, for in a competition called ImageNet Large Visual Image Recognition competition, where this, uh, it was done on the ImageNet dataset which has 1.2 million, uh, 1.2 million training images, and there were 50,000 validation images and 1 lakh 50,000 testing images. So this, uh, in this architecture there are 5 convolutional layers. And there are 3 fully connected layers. So the image sizes were of 228 × 228 into 3, into 3 order, of the order. And the first connected layer has 96 kernels. The first layer has 96 kernels, each of 11 × 11 × 3 sizes. And the second, or the second, third, and the fourth layers had 256 kernels. Sorry, it had 384 kernels. Yes, it has 384 kernels.
And the—
okay, I think I'm wrong somewhere. Sorry, the second layer had 256 kernels, and the third and the fourth layer has 384, and again the fifth layer has again 256 kernels. And then followed by the 3 fully connected layers, and the each of the layer— the first 2 layers had 4, like 4,096 neurons in each of the layers, or you can say nodes. And the third was that the output has 10— you have to classify in 10, you know, 1,000 images. There were 1,000 categories, so the output has 10,000 nodes— sorry, 1,000 nodes. Sorry, 1,000 nodes. And basically, in the first two layers, you can see there is, uh, for each of the layers, we were doing max pooling. Actually, it is overlap max pooling because the window, the window size was of 3x3 order. And the stride was of 2x2, or rather I could say that stride was 2. So it was overlapping, if you can visualize. So why they did overlap max pooling is that it decreased the error rate by around 0.3 or 0.4%. So in the first two, they were doing max pooling, overlap max pooling, and in the first two layers also, they did local response normalization and local response normalization. And also they, in all the layers, they applied a ReLU activation function. So it was, if you see the graph, okay, so if you see the graph, like they compared the results of like two activation functions, ReLU and tanh(x) activation function. And it was seen that by applying ReLU, they were able to achieve 25% error rate. They were able to reach it 6 times faster than the tanh activation function. So ReLU was a better option because you can see that we reached way more faster to the 25% error rate. And while max pooling, as I said, it decreased the error by 0.3 and 0.4%. And yes, they did local response normalization. So right now, generally in the architectures, we do batch normalization. So these are the formulas you can see. So why we do local response normalization, or right now in the present architectures, we do batch normalization in the convolutional layers, because by doing so, we are actually trying to minimize the values into 0 to 1. So whenever the values are in the 0 to 1 range, so it's better to— easier to calculate. You will see that the learning rate will be way more faster, as well as the regularization. It also does certain side of regularization. Regularization. So that's the main thing behind it. And apart from that, we don't apply dropout in the convolutional layers because we only apply dropout in the fully connected layers. So it is so because we mainly— the main motive of dropout is to prevent overfitting. So in the fully connected layers, we don't want that the spatial data— that there is any spatial relationship. So we do dropout to minimize that overfitting, to prevent that there is some connection between the spatial data. But whereas in convolutional layers, it is all about spatial relationship, like from edges, from correlating edges, we form pattern. So that's why in convolutional layers, we don't use dropout. So that's the main point behind it. And by using this Allisonet architecture, Top 1% error was around 18%, and we were about to reach— okay, sorry, top 1% error was around 37%, and the top 5% error was about 18%. And these are the other architectures. You can see, like, this was the first architecture in 2010, and the first was like 8 layers, as I just discussed, and all the other layers are really huge, 200, and you can see the errors are gradually decreasing. So this is all about presentation, various architectures. That's all for now.
Anything?
One question here. So optimizers, so we did not talk anything about optimizers while talking about CNN.
Oh. Okay, so optimizers, like we are optimizing like in one sense, like we are already optimizing, right? Like we are using, by optimizing means we are using dropout, so it's kind of optimizing.
So exactly what you want to optimize? So optimize in the sense when it moves from one layer to other layer.
Yes.
So one was, one is to configure the epoch so that when it runs more around the time, say, to learn more and improve efficiency. Is there any other way to optimize while moving from one layer to each layer, improving the accuracy?
I'll say yeah, we are with, uh, for a certain epoch, right? Yeah, like, yeah. So basically what we are doing is like we are doing the normalizations and normalization, and we are doing the relu, so, and we are doing like overlap max pooling, as I said, that With overlap max pooling, we are able to optimize it by 0.3 and 0.4% error rate. So these are the ways. And is there any other way? He knows more than me.
Yeah. Generally, if you are talking about increasing the accuracy, then there are different architectures, different ways you can try. So one way he said is overlap pooling. So generally, max pooling and average pooling are used, but overlap pooling and other Different ways, at some level it's just intuition, like getting the accuracy, trying out different things, experimenting and researching, and this is one of the results. And various people have tried different techniques and their accuracy have shown some point increases or maybe increased by +1 or +5.
How did that last— you were saying this paper, right, AlexNet paper?
Yeah.
So there, was there any scenario of overfitting? And how— yeah, so my question is basically, how do we know that this was not an overfitted model when it was trained?
So like, from the testing, like, there are 3 kinds of tests, right? Like, sorry, there are 3 datasets. One is the training set, the other is the validation dataset, and other is the testing dataset. So in the testing dataset, you can see whether it is overfitting or not. Like, you can see the errors. Even in batches we can print, right? Yeah, yeah, sure.
You can see and see whether—
yeah, like there is a graph, like, so for each batch you can print out the validation accuracy and validation loss. So you can see the training loss is decreasing and the validation loss is decreasing. If you see the training loss is decreasing but the validation loss is increasing, then your model is overfitting the training dataset.
Yeah, that's the point where you should just stop. And also, like, in some architectures fits Like also the dataset, like for certain architectures, it may be that certain dataset does not suit. It also can happen.
