MIT 6.S094: Convolutional Neural Networks for End-to-End Learning of the Driving Task

All right, welcome back. Everyone sound okay? All right, so today we will. We talked a little bit about neural networks, started to talk about neural networks yesterday. Today we'll continue to talk about neural networks that work with images, convolutional neural networks. See how those types of networks can help us drive a car if we have time. We'll cover a simple illustrative case study of detecting traffic lights. The problem of detecting green, yellow, red. If we can't teach our neural networks to do that, we're in trouble. But it's a good, clear illustrative case study of a three class classification problem. Okay, next, there's Deep Tesla here, looped over and over in a very short gif. This is actually running live on a website right now. We'll show it towards the end of the lecture. This once again, just like deep traffic is a neural network that learns to steer a vehicle based on the video of the forward roadway. And once again doing all of that in the browser using JavaScript. So you'll be able to train your own very network to drive using real world data. I'll explain how. We will also have a tutorial and code briefly describe today at the end of lecture, if there's time, how to do the same thing in TensorFlow. So if you want to build a network that's bigger, deeper and you want to utilize GPUs to train that network, you want to not do it in your browser, you want to do it offline using TensorFlow and having a powerful GPU on your computer. And we'll explain how to do that. Computer vision. So we talked about vanilla machine learning where there's no where the size yesterday, where the size of the input is small for the most part. The number of neurons in the case of neural networks is on the order of 10, 100, 1000. When you think of images, images are a collection of pixels. One of the most iconic images from computer vision, in the bottom left there is Lena. I encourage you to Google it and figure out the story behind that image. It was quite shocking when I found out recently. So once again, computer vision is these days dominated by data driven approaches by machine learning, where all of the same methods that are used on other types of data are used on images, where the input is just a collection of pixels and pixels are numbers from 0 to 255 discrete values. So we can think exactly what we've talked about previously. We could think of images in the same exact way. It's just numbers. And so we can do the same kind of thing. We can do supervised learning where you have an input image and output label. The input image here is a picture of a woman. The label might be woman. Unsupervised learning. Same thing. We'll look at that briefly as well, is clustering images into categories. Again, semi supervised and reinforcement learning. In fact, the Atari games we talked about yesterday do some pre processing on the images. They're doing computer vision. They're using convolutional neural networks, as we'll discuss today. And the pipeline for supervised learning is again the same. There's raw data in the form of images. There's labels on those images. We perform a machine learning algorithm, performs feature extraction. It trains given the inputs and outputs on the images and the labels of those images, constructs the model, and then tests that model. And we get a metric, an accuracy. Accuracy is the term that's used to often describe how well a model performs. It's a percentage. I apologize for the constant presence of cats throughout this course. I assure you this course is about driving, not cats, but images are numbers. So for us, we take it for granted. We're really good at looking at and converting visual perception as human beings, converting visual perception into semantics. We see this image and we know it's a cat. But a computer only sees numbers. RGB values for a colored image. There's three values for every single pixel from 0 to 255. And so given that image, we can think of two problems. One is regression, and the other is classification. Regression is, when given an image, we want to produce a real valued output back. So if we have an image of the forward roadway, we want to produce a value for the steering wheel angle. And if you have an algorithm that's really smart, it can take any image of the forward roadway and produce the perfectly correct steering angle that drives the car safely across the United States. We'll talk about how to do that and where that fails. Classification is when the input again is an image and the output is a class label, a discrete class label. Underneath it, though, often is still a regression problem. And what's produced is a probability that this particular image belongs to a particular category. And we use a threshold to chop off the outputs associated with low probabilities and take the labels associated with the high probabilities and convert it into a discrete classification. I mentioned this yesterday, but it bears saying again, computer vision is hard. We once again take it for granted. As human beings, we're really good at dealing with all these problems. There's viewpoint variation. The object looks totally different in terms of the numbers behind the images in terms of the pixels when viewed from a different angle viewpoint variation objects, when you're standing far away from them or up close, are totally different size. We're good at detecting that they're different size. It's still the same object as human beings, but that's still a really hard problem because those sizes can vary drastically. We talked about occlusions and deformations with cats. Well understood problem. There's background clutter. You have to separate the object of interest from the background. And given the three dimensional structure of our world, there's a lot of stuff often going on in the background, the clutter. There are intraclass variation that's often greater than interclass variation, meaning objects of the same type often have more variation than the objects that you're trying to separate them from. There's the hard one for driving illumination. Light is the way we perceive things. The reflection of light off the surface and the source of that light changes the way that object appears. And we have to be robust all of that. So the image classification pipeline, this is the same as I mentioned, there's categories, it's a classification problem. So there's categories of cat, dog, mug, hat, and you have a bunch of examples, image examples of each of those categories. And so the input is just those images paired with the category. And you train to map, to estimate a function that maps from the images to the categories. For all of that you need data, a lot of it. There is, unfortunately, there's a growing number of data sets, but they're still relatively small. We get excited. There are millions of images, but they're not billions or trillions of images. And these are the data sets that you will see if you read academic literature, most often mnist, the one that's been beaten to death and then we'll use as well in this course is a data set of handwritten digits. Where the categories are 0 to 9. ImageNet, one of the largest image data sets, fully labeled image data sets in the world, has images with a hierarchy of categories from wordnet and what you see there is a labeling of what images associated with which words are present in the data set. CIFAR 10 and CIFAR 100 are tiny images that are used to prove in a very efficient and quick way offhand, that your algorithm that you're trying to publish on or trying to impress the world with works well. It's small, it's a small data set. CIFAR 10 means there's 10 categories and places is a data set of natural scenes woods, nature, city, so on. So let's look at CIFAR. 10 is a data set of 10 categories. Airplane, automobile, bird, cat, so on. They're shown there with sample images as the rows. And so let's build a classifier that's able to take images from one of these 10 categories and tell us what is shown in the image. So how do we do that? Once again, all the algorithm sees is numbers. So we have to try to have, at the very core, we have to have an operator for comparing two images. If you're given an image, and I want to say if it's a cat or a dog, I want to compare it to images of cats and compare it to images of dogs and see which one matches better. So there has to be a comparative operator. Okay? So one way to do that is take the absolute difference between the two images pixel by pixel, take the difference between each individual pixel shown on the bottom of the slide for a 4 by 4 image, and then we sum that pixel wide, pixel wise, absolute difference, into a single number. So if the image is totally different, pixel wise, that'll be a high number. If it's the same image, the number will be zero. Oh, it's the absolute value too, of the difference. That's called L1 distance doesn't matter. When we speak of distance, we usually mean L2 distance. And so if we try to. So we can build a classifier that just uses this operator to compare to every single image in the data set and say, I'm going to pick the. I'm going to pick the category that's the closest. Using this comparative operator, I'm going to find I have a picture of a cat and I'm going to look through the data set and find the image that's the closest to this picture. And say that is the category that this picture belongs to. So if we just flip the coin and randomly pick which category an image belongs to, we get that accuracy would be on average 10%. It's random. The accuracy we achieve with our brilliant image difference algorithm that just goes to the data set and finds the closest one is 38%, which is pretty good. It's way above 10%. So you can think about this operation of looking through the data set and finding the closest image as what's called K nearest neighbors, where K in that case is one, meaning you find the one closest neighbor to this image that you're asking a question about and accept the label from that image. You could do the same thing. Increasing K, increasing K to 2 means you take the two nearest neighbors, you find the two closest in terms of pixel wise image difference images to this particular query image and find which category do those belong to. What's shown up top on the left is the data, the data set we're working with red, green, blue. What's shown in the middle is the one nearest neighbor classifier, meaning this is how you segment the entire space of different things that you can compare. And if a point falls into any of these regions, it will be immediately associated with the nearest neighbor algorithm to belong to that image to that region. With five nearest neighbors, there's immediately an issue. The issue is that there's white regions, there's tie breakers, where your five closest neighbors are from various categories. So it's unclear what you belong to. So if we. This is a good example of parameter tuning. You have one parameter K, and you have to your task as a machine, as a teacher of machine learning, you have to teach this algorithm how to do your learning for you is to figure out that parameter that's called parameter tuning or hyperparameter tuning as it's called in neural networks. And so on the bottom right of the slide is on the X axis is K as we increase it from 0 to 100 and on the Y axis is classification accuracy. It turns out that the best K for this Data set is 7, 7 nearest neighbors. With that we get a performance of 30% human level performance. And I should say that the way that number is we get that number, as we do with a lot of the machine learning pipeline process, is you separate the data into the parts of the dataset you use for training and sep and another part that you use for testing, you're not allowed to touch the testing part, that's cheating. You construct your model of the world on the training data set and you use what's called cross validation where you take a small part of the training data showing fold five there in yellow to leave that part out from the training and then use it as part of the hyperparameter tuning. As you train, figure out with that yellow part fold 5 how well you're doing and then you choose a different fold and see how well you're doing and keep playing with parameters, never touching the test part. And when you're ready, you run the algorithm on the test data to see how well you really do, how well it really generalizes. Yes. Question is there any way to like determine an intuition of what a good K may be? Or do you just have to run through all the examples? So the question was, is There a good way to. Is there any good intuition behind what a good K is? There's general rules for different data sets, but usually you just have to run through it. Grid search, brute force. Yes. Question. Good question. Yes. Yes. The question was, Is each pixel 1 number or 3 numbers? For majority of computer vision throughout its history use grayscale images. So it's one number, but RGB is three numbers and there are sometimes a depth value too. So it's four numbers. So it's. If you have a stereo vision camera that gives you the depth information of the pixels, that's a fourth. And then if you stack two images together, it could be six. In general, everything we work with will be three numbers. For a pixel it was. Yes. So the question was for the absolute value, just one number. Exactly right. So in that case those grayscale images. So it's not RGB images. So that's. You know, this algorithm is pretty good. If we use the best we hyper. We optimize the hyper parameters of this algorithm. Choose K of 7. Seems to work well for this particular CIFAR 10 data set. Okay, we get 30% accuracy. It's impressive. Higher than 10%. Human beings perform at about 94, slightly above 94% accuracy for CIFAR 10. So given an image, it's a tiny image, I should clarify, it's like a, it's a little icon. Given that image, human beings are able to determine accurately one of the 10 categories with 94% accuracy. And the currently state of the art convolutional neural networks is 95, it's 95.4% accuracy. And believe it or not, it's a heated battle. But the most important, the critical fact here is it's recently surpassed humans and certainly surpassed the K nearest neighbors algorithm. So how does this work? Let's briefly look back. It all still boils down to this little guy. The neuron that sums the weights of its inputs, adds a bias, produces an output based on an activation, a smooth activation function. Yes. Question. The question was you take a picture of a cat, so you know it's a cat, but that's not encoded anywhere. Like you have to write that down somewhere. So you have to write as a caption, this is my cat. And then the unfortunate thing, given the Internet and how witty it is, you can't trust the captions and images because maybe you're just being clever. And it's not a cat at all, it's a dog dressed as a cat. Yes. Question, are CNNs in general better than caterers? Haters sorry, CNNs do better than what are. Do convolutional neural networks in general do better than any neighbors? Yeah. So the question was do convolutional neural networks generally do better than nearest neighbors? There's very few problems on which neural networks don't do better. Yes, they almost always do better, except when you have almost no data. So you need data. And convolutional neural networks isn't some special magical thing. It's just neural networks with some, with some cheating upfront that I'll explain some tricks to try to reduce the size and make it capable to deal with images. So again, yeah, the input is in this case that we looked at classifying an image of a number as opposed to doing some fancy convolutional tricks. We just take the, the entire 28 by 28 pixel image, that's 784 pixels as the input. That's 784 neurons in the input, 15 neurons on the hidden layer and 10 neurons in the output. Now everything we'll talk about has the same exact structure, nothing fancy. There is a forward pass through the network where you take an input image and produce an output classification. And there's a backward path to the network through backpropagation where you adjust the weights when your prediction doesn't match the ground truth output. And learning just boils down to optimization. It's just optimizing a smooth function differentiable function that's defined as the loss function. That's usually as simple as a squared error between the true output and the one you actually got. So what's the difference? What are convolutional neural networks? Convolutional neural networks take, take inputs that have some spatial consistency, have some meaning to the spatial, have some spatial meaning in them. Like images. There's other, there's other, other things you can think of the dimension of time and you can input audio signal into a convolutional neural network. And so the input is usually for every single layer that's a convolutional layer. The Input is a 3D volume and the output is a 3D volume. I'm simplifying because you can call it 4D too, but it's 3D. There's height, width and depth. So that's an image. The height and the width is the width and the height of the image. And then the depth for grayscale image is 1. For an RGB image is 3. For a 10 frame video of grayscale images, the depth is 10. It's just a volume, a three dimensional matrix of numbers. And the only thing that a convolutional layer does is take a 3D volume as input, produce a 3D volume as output, and has some smooth function operating on the inputs, on the sum of the inputs. That may or may not be a parameter that you tune, that you try to optimize. That's it. It's Lego pieces that you stack together in the same way as we talked about before. So what are the types of layers that a convolutional neural network have? There's inputs. So, for example, a color image of 32 by 32 will be a volume of 32 by 32 by 3. The convolutional layer takes advantage of the spatial relationships of the input neurons. And a convolutional layer, it's the same exact neuron as for a fully connected network, the regular network we talked about before, but it just has a narrower receptive field. It's more focused. The, the inputs to a neuron on the convolutional layer come from a specific region from the previous layer. And the parameters on each filter, you could think of this as a filter because you slide it across the entire image and those parameters are shared. So as opposed to taking the. For if you think about two layers, as opposed to connecting every single pixel in the first layer to every single neuron in the following layer, you only connect the neurons in the input layer that are close to each other to the output layer. And then you enforce the weights to be tied together spatially. And what that results in is a filter. Every single layer on the output you could think of as a filter that gets excited, for example, for an edge. And when it sees this particular kind of edge in the image, it'll get excited. It'll get excited in the top left of the image. Top right, top left, bottom left, bottom right. The assumption there is that a powerful feature for detecting a cat is just as important no matter where in the image it is. And this allows you to cut away a huge number of connections between neurons. But it still boils down on the right as a neuron that sums up collection of inputs and applies weights to them. The tunable, the spatial arrangement of the output volume relative to the input volume is controlled by three things. The number of filters. So for every single quote unquote filter, you'll get an extra layer on the output. So if the input. Let's talk about the very first layer. The input is 32 by 32 by 3. It's an RGB image of 32 by 32. If the number of filters is 10, then the resulting depth, the resulting number of stacked channels in the output will be 10. Stride is given is the step size of the filter that you slide along the image. Oftentimes that's just one or three and that directly reduces the size, the spatial size, the width and the height of the output image. And then there is a convenient thing that's often done is padding the image on the outside with zeros so that the input and the output have the same height and width. So this is a visualization of convolution. And I encourage you to kind of maybe offline think about what's happening. It's similar to the way human vision works, crudely. So if there's any experts in the audience. So the input here on the left is a collection of numbers 0, 1, 2 and a filter. There is two filters shown as W0 and W1. Those filters shown in red are the different weights applied on those filters. And each of the filters have a certain depth, just like the input, a depth of three. So there's three of them in each column. And so let's see. Yeah. And so you slide that filter along the image, keeping the weights the same. The this is the sharing of the weights. And so your first filter, you pick the weights. This is an optimization problem. You pick the weights in such a way that it fires, gets excited at useful features and doesn't fire for not useful features. And then there's the second filter that fires for useful features and not and produces a signal on the output depending on a positive number, meaning there's a strong feature in that region and a negative number if there isn't. But the filter is the same. This allows for drastic reduction in the parameters. And so you can deal with inputs that are 1000 by 1000 pixel image, for example, or video. There's a really powerful concept there, the spatial. The spatial sharing of weights. That means there's a spatial invariance to the features you're detecting. It allows you to learn from arbitrary images so you don't have to be concerned about pre processing the images in some clever way. You just give the raw image. There's another operation, pooling. It's a way to reduce the size of the layers by for example, in this case max pooling for taking a collection of outputs and choose x1 and summarizing those collection of pixels such that the output of the pooling operation is much smaller than the input. Because the justification there is that you don't need a high resolution localization of exactly where, which pixel is important in the image. According to, you know, you don't need to know exactly which pixel is associated with the cat ear or, you know, a cat face, as long as you kind of know it's around that part. And that reduces a lot of complexity in the, in the operations. His question, The question was, when is too much pooling? When do you stop pulling? So pooling is a very crude operation that doesn't have any. One thing you need to know is it doesn't have any problem parameters that are learnable. So you can't learn anything clever about the, about pooling. You're just picking, in this case, max pool. So you're picking the largest number. So you're reducing the resolution. You're losing a lot of information. There's an argument that you're not, you know, losing that much information as long as you're not pooling the entire image into a single value. But you're gaining training efficiency, you're gaining the memory size, the reducing the size of the network. So it's definitely a thing that people debate and it's a parameter that you play with to see what works for you. Okay, so how does this thing look like as a whole? A convolution on your neural network, the input is an image. There's usually a convolutional layer, There is a pooling operation, another convolutional layer, another pooling operation, and so on. At the very end, if the task is classification, you have a stack of convolutional layers and pooling layers. There are several fully connected layers. So you go from those, the spatial convolutional operations, to fully connecting every single neuron in a layer to the following layer. And you do this so that by the end you have a collection of neurons. Each one is associated with a particular class. So in what we looked at yesterday, as the input is an image of a number 0 through 9, the output here would be 10 neurons. So you boil down that image with a, with a collection of convolutional. Convolutional layers with one or two or three fully connected layers at the end that all lead to 10 neurons. And each of those neurons job is to get, get fired up when it sees a particular number and for the other ones to produce a low probability. And so this kind of process is how you have the 95 percentile accuracy on the CIFAR 10 problem. This here is ImageNet data set that I mentioned. It's how you take this image of a leopard of a container ship and produce a probability that that is a container ship or a leopard. Also shown there are the outputs of the other nearest, nearest neurons in terms of their confidence. Now, you can use the same exact operation by chopping off the fully connected layer at the end. And as opposed to mapping from image to a prediction of what's contained in the image, you map from the image to another image and you can train that image to be one that gets excited spatially, meaning it gives you a high, close to one value for areas of the image that contain the object of interest and then a low number for areas of the image that are unlikely to contain that image. And so from this you can go on the left an original image of a woman and a horse, to a segmented image of knowing where the woman is and where the horse is and where the background is. The same process can be done for detecting the object. So you can segment the scene into a bunch of interesting objects, candidates for interesting objects, and then go through those candidates one by one and perform the same kind of classification as in the previous step, where it's just an input is an image, and the output.