MIT 6.S094: Introduction to Deep Learning and Self-Driving Cars

All right. Hello everybody. Hopefully you can hear me well. Yes, yes. Great. So welcome to course 6S094, Deep Learning for Self Driving Cars. We will introduce to you the methods of deep learning of deep neural networks using the guiding case study of building self driving cars. My name is Lex Friedman. You get to listen to me for majority of these lectures. And I am part of an amazing team with some brilliant TAs. Would you say brilliant Dan Brown. You guys want to stand up? You okay? They're in front row. Spencer William Angel. Spencer Dodd. And all the way in the back, the smartest and the tallest person I know, Benedict Jennick. So what you see there on the left of the slide is a visualization of one of the two projects that one of the two simulations games that we'll get to go through. We use it as a way to teach you about deep reinforcement learning, but also as a way to excite you by challenging you to compete against others. If you wish to win a special prize yet to be announced, super secret prize. So you can reach me and the TAs@deepcarsmit.edu if you have any questions about the tutorials, about the lecture, about anything at all. The website Cars MIT Edu has the lecture content code tutorials again like today. The lecture slides for today are already up in PDF form. The slides themselves, if you want to see them, just email me. But they're over a gigabyte in size because they're very heavy in videos. So I'm just posting the PDFs and there will be lecture videos available a few days after the lecture is given. So speaking of which, there is a camera in the back. This is being videotaped and recorded, but for the most part the camera is just on the speaker. So you shouldn't have to worry. If that kind of thing worries you, then you could sit on the periphery of the classroom. Or maybe I suggest sunglasses and a mustache. Fake mustache. That would be a good idea. There is a competition for the game that you see on the left. I'll describe exactly what's involved. In order to get credit for the course, you have to design a neural network that drives the car just above the speed limit, 65 miles an hour. But if you want to win, you need to go a little faster than that. So who this class is for, you may be new to programming, new to machine learning, new to robotics, or you're an expert in those fields, but want to go back to the basics. So what you will learn is an overview of deep reinforcement learning. A convolutional neural networks, recurrent neural networks and how these methods can help improve your each of the components of autonomous driving perception, visual perception, localization, mapping, control, planning and the detection of driver state. Okay, two projects, code name Deep Traffic is the first one there is in this particular formulation of it there is seven lanes. It's a top view. It looks like a game, but I assure you it's very serious. It is the agent in red. The car in red is being controlled by neural network and we'll explain how you can control and design the various aspects of the various parameters of this neural network. And it learns in the browser. So this we're using convnet JS which, which is a library that is programmed by Andre Karpathy in JavaScript. So amazingly we live in a world where you can train in a matter of minutes a neural network in your browser and we'll talk about how to do that. The reason we did this is so that there is very few requirements to get you up and started with neural networks. So in order to complete this project for the course, you don't need any requirements except to have a Chrome browser and to win the competition you don't need anything except a Chrome browser. The second project, code name Deep Tesla or Tesla is using data from a Tesla vehicle of the forward roadway and using end to end learning, taking the image and putting it into convolutional neural network that directly maps a regressor that maps to a steering angle. So all it takes is a single image and it predicts a steering angle for the car and we have data for the car itself and you get to build a neural network that tries to do better, tries to steer better or at least as good as the car. Okay, let's get started with the question, with the thing that we understand so poorly at this time because it's so shrouded in mystery. But it fascinates many of us and that's the question of what is intelligence? This is from a 1996 March 1996 Time magazine. And the question can machines think? Is answered below with they already do. So what, if anything, is special about the human mind? It's a good question for 1996, a good question for 2016. 17 now the future. And there's two ways to ask that question. One is the special purpose version. Can an artificial intelligence system achieve a well defined, specifically formally defined finite set of goals? And this little diagram from a book that got me into artificial intelligence as a bright eyed high school student. The Artificial intelligence, the modern approach. This is A beautifully simple diagram of a system. It exists in an environment, it has a set of sensors that do the perception, it takes those sensors in, does something magical. There's a question mark there. And with a set of effectors, acts in the world, manipulates objects in that world. And so special purpose we can, under this formulation, as long as the environment is formally defined, well defined, as long as a set of goals are well defined, as long as a set of actions, sensors and the ways that the perception carries itself out is well defined, we have good algorithms of which we'll talk about, that can optimize for those goals. The question is, if we inch along this path, will we get closer to the general formulation, to the general purpose version of what artificial intelligence is? Can it achieve poorly defined, unconstrained set of goals with an unconstrained, poorly defined set of actions and unconstrained, poorly defined utility functions? Rewards. This is what human life is about. This is what we do pretty well most days. Exists in an undefined, full of uncertainty world. So, ok, we can separate tasks into three different categories. Formal tasks. This is the easiest. It doesn't seem so. It didn't seem so at the birth of artificial intelligence, but that's in fact true. If you think about it, the easiest is the formal tasks. Playing board games, theorem proving, all the kind of mathematical logic problems that can be formalized, formally defined. Then there's the expert tasks. So this is where a lot of the exciting breakthroughs have been happening, where machine learning methods, data driven methods, can help aid or improve on the performance of our human experts. This means medical diagnosis, hardware design, scheduling. And then there is the thing that we take for granted, the trivial thing, the thing that we do so easily every day when we wake up in the morning. The mundane tasks of everyday speech, of written language, of visual perception, of walking, which we'll talk about in today's lecture, is a fascinatingly difficult task and object manipulation. So the question is that we're asking here, before we talk about deep learning, before we talk about the specific methods, we really want to dig in and try to see what is it about driving? How difficult is driving? Is it more like chess, which you see on the left there, where we can formally define a set of lanes, a set of actions, and formulate it as this. There's five set of actions. You can change a lane, you can avoid obstacles, you can formally define an obstacle, you can formally define the rules of the road. Or is there something about natural language, something similar to everyday conversation about driving, that requires a Much higher degree of reasoning, of communication, of learning, of existing in this under actuated space. Is it a lot more than just left lane, right lane, speed up, slow down. So let's look at it as a chess game. Here's the chess pieces. What are the sensors we get to work with on the self on an autonomous vehicle? And we'll get a lot more in depth on this, especially with the guest speakers who've built many of these. There's radar, there's the range sensors, radar, lidar that give you information about the obstacles in the environment, that help localize the obstacles in the environment. There's the visible light, camera and stereo vision that gives you texture information that helps you figure out not just where the obstacles are, but what they are. Helps to classify those, helps to understand their subtle movements. Then there is the information about the vehicle itself, about the trajectory and the movement of the vehicle that comes from the GPS and IMU sensors. And there is the state of the rich state of the vehicle itself. What is it doing? What are all the individual systems doing? That comes from the CAN network. And there is one of the less studied but fascinating to us on the research side is audio. The sounds of the road that provide the rich context of a wet road. The sound of a road that when it stopped raining but it's still wet, the sound that it makes, the screeching tire and honking. These are all fascinating signals as well. And the focus of the research in our group. The thing that's really much under investigated is the internal facing sensors. The driver sensing the state of the driver. Where are they looking? Are they sleepy? The emotional state, are they in the seat at all? And the same with audio that comes from the visual information and the audio information. More than that, here's the tasks. If you were to break into modules. The task of what it means to build a self driving vehicle. First you want to know where you are. Where am I? Localization and mapping. You want to map the external environment, figure out where all the different obstacles are, all the entities are, and use that estimate of the environment to then figure out where I am, where the robot is. Then there's scene understanding. It's understanding not just the positional aspects of the external environment and the dynamics of it, but also what those entities are. Is it a car? Is it a pedestrian? Is it a bird? There's movement planning. Once you have kind of figured out to the best of your abilities, your position and the position of other entities in this world, there's figuring out a trajectory through that world. And finally, once you've figured out how to move about safely and effectively through that world, is figuring out what the human that's on board is doing. Because as I will talk about, the path to a self driving vehicle, and that is hence our focus on Tesla may go through semi autonomous vehicles where there is a. Where the vehicle must not only drive itself, but effectively hand over control from the car to the human and back. Okay, quick history. Well, there's a lot of fun stuff from the 80s and 90s, but. The big breakthroughs came in the second DARPA Grand Challenge with Stanford Stanley when they won the competition, one of five cars that finished. This was an incredible accomplishment. In a desert race, a fully autonomous vehicle was able to complete the race in record time. The DARPA urban challenge in 2007, where the race was the task was no longer a race to the desert, but through a urban environment. And CMU's boss with GM won that race. And a lot of that work led directly into the acceptance and large major industry players taking on the challenge of building these vehicles. Google now Waymo self driving car. Tesla with its Autopilot system and now autopilot 2 system, uber with its testing in Pittsburgh. Then there's many other companies, including one of the speakers for this course of mutatomy that are driving the wonderful streets of Boston. Okay, so let's take a step back. We have. If we think about the accomplishments in the DARPA challenge and if we look at the accomplishments of the Google self driving car, which essentially boils the world down into a chess game, it uses incredibly accurate sensors to build a three dimensional map of the world, localize itself effectively in that world and move about that world in a very well defined way. Now what if driving, the open question is if driving is more like a conversation, like a natural language conversation, how hard is it to pass the Turing test? The Turing test as the popular current formulation is can a computer be mistaken for a human being? In more than 30% of the time when a human is talking behind a veil, having a conversation with either a computer or human, they mistake the other side of that conversation for being a human when it's in fact a computer. And the way you would in a natural language build a system that has successfully passes the Turing test is the natural language processing part to enable it to communicate successfully. So generate language and interpret language, then you represent knowledge, the state of the conversation transferred over time. And the last piece, and this is the hard piece, is the automated reasoning is reasoning. Can we teach machine learning methods to reason that's, that is something that will propagate through our discussion because as I will talk about the various methods, the various deep learning methods, neural networks are good at learning from data, but they're not yet. There's no good mechanism for reasoning. Now, reasoning could be just something that we tell ourselves we do to feel special, to feel like we're better than machines. Reasoning may be simply something as simple as learning from data. We just need a larger network. Or there could be a totally different mechanism required and we'll talk about the possibilities there. Yes. No, it's very difficult to find these kind of situations in the United States. So the question was, for this video, is it in the United States or not? I believe it's in Tokyo. So India, a few European countries are much, are much more towards the direction of natural language versus chess. In the United States, generally speaking, we follow rules more concretely. The quality of roads is better, the marking on the roads is better. So there's less requirements there. These cars are driving on the left side. I see. I just. Okay, yeah, you're right. It is because. Yep. Yeah, so. But it's certainly not the United States. I'm pretty spent quite a bit of Googling trying to find the United States, and it's, it's difficult. So let's talk about the recent breakthroughs in machine learning. And what is at the core of those breakthroughs is neural networks that have been around for a long time. And I will talk about what has changed, what are the cool new things and what hasn't changed, and what are its possibilities. But first, a neuron, crudely, is a computational building block of the brain. I know there's a few folks here, neuroscience folks, this is hardly a model. It is mostly an inspiration. And so the, the human neuron has inspired the artificial neuron, the computational building block of a neural network. Of an artificial neural network. Now, to give you some context, these neurons for both artificial and human brains are interconnected. In the human brain, there's about, I believe, 10,000 outgoing connections from every neuron on average, and they're interconnected to each other. The largest current, as far as I'm aware. Artificial neural network has 10 billion of those connections synapses. Our human brain, to the best estimate that I'm aware of, has 10,000 times that. So 100 to 1,000 trillion synapses. Now, what is a artificial neuron? This building block of a neural network, it takes a set of inputs. It puts a weight on each of those inputs, sums them together, applies a bias value on each, that sits on each neuron, and using an activation function that takes as input that sum plus the bias and squishes it together to produce a zero to one signal. And this allows us, a single neuron could take a few inputs and produce as an output a classification, for example, a 0, 1. And as we'll talk about simply, it can serve as a linear classifier. So it can draw a line. It can learn to draw a line between, like what's seen here, between the blue dots and the yellow dots. And that's exactly what we'll do in the IPYTHON notebook that I'll talk about. But the basic algorithm is you initialize the weights on the inputs and you compute the output. You perform this previous operation I talked about, sum up and compute the output. And if the output does not match the ground truth, the expected output, the output that it should produce, the weights are punished accordingly. And we'll talk through a little bit of the math of that. And this process is repeated until the perceptron does not make any more mistakes. Now, here's the amazing thing about neural networks. There's several, and I'll talk about them. One, on the mathematical side is the universality of neural networks with just a single layer. If we stack them together, a single hidden layer, the inputs on the left, the outputs on the right, and in the middle there's a single hidden layer. It can closely approximate any function, any function. So this is an incredible property that with a single layer, any function you could think of, that you know, you can think of driving as a function. It takes as input the, the world outside, as output, the control of the vehicle. There exists a neural network out there that can drive perfectly. It's a fascinating mathematical fact. So we can think of this then, these functions as a special purpose function, special purpose intelligence. You can take, say as input the number of bedrooms, the square feet, the type of neighborhood, those are the three inputs it passes that value through to the hidden layer. And then one more step, it produces the final price estimate for the house or for the residence. And we can teach a network to do this pretty well in a supervised way. This is supervised learning. You provide a lot of examples where you know the number of bedrooms, the square feet, the type of neighborhood, and then you also know the final price of the house or the residence. And then you can, as I'll talk about, through a process of back propagation, teach these networks to make this prediction pretty well. Now, some of the exciting breakthroughs recently have been in the general purpose intelligence. This is from Andre Karpathy, who is now at OpenAI. I would like to take a moment here to try to explain how amazing this is. This is a game of Pong. If you're not familiar with Pong, there's two paddles and you're trying to bounce the ball back and in such a way that prevents the other guy from bouncing the ball back at you. The artificial intelligence agent is on the right in green. And up top is the score, 8 to 1. Now, this takes about three days to train on a regular computer. This network. What is this network doing? It's called the policy network. The input is the raw pixels. They're slightly processed. And also you take the difference between two frames, but it's basically the raw pixel information that's the input. There's a few hidden layers and the output is a single probability of moving up. That, that's it, that's, that's the whole. That's, that's the whole system. And what it's doing is it learns not. You don't know at any one moment, you don't know what the right thing to do is, is it to move up, is it to move down. You only know what the right thing to do is by the fact that eventually you win or lose the game. So this is the amazing thing here is there's no supervised learning about. There's no like universal fact about any one state being good or bad and any one action being good or bad in any state. But if you punish or reward every single action you took, every single action you took for entire game based on the result. So no matter what you did, if you won the game, the end justifies the means. If you won the game, every action you took and every action state pair gets rewarded. If you lost the game, it gets punished. And this process, with only 200,000 games, where the system just simulates the games, it can learn to beat the computer. This system knows nothing about Pong, nothing about games. This is general intelligence, except for the fact that, that it's just a game of Pong. And I will talk about how this can be extended further and why this is so promising and why this is also. We should proceed with caution. So again, there's a set of actions you take up, down, up, down, based on the output of the network. There's a threshold. Given the probability of moving up, you move up or down based on the output of the network and you have a set of states. And every single state action pair is rewarded. If there's a win and is Punished if there's a loss. When you go home, think about how amazing that is. And if you don't understand why that's amazing, spend some time on it. It's incredible. Sure, sure thing. The question was, what is supervised learning? What is unsupervised learning? What's the difference? So supervised learning is. When people talk about machine learning, they mean supervised learning. Most of the time, supervised learning and is learning from data. It's learning from example, when you have a set of inputs and a set of outputs that you know are correct, what are called ground truth. So you need those examples, a large amount of them, to train any of the machine learning algorithms to learn, to then generalize that to future examples. This is actually, there's a third one called reinforcement learning, where the ground truth is sparse, the information about when something is good or not. The ground truth only happens every once in a while at the end of the game, not every single frame. And unsupervised learning is when you have no information about the outputs that are correct or incorrect. And it is the excitement of the deep learning community is unsupervised learning. But it has achieved no major breakthroughs at this point. This is the. I'll talk about what the future of deep learning is, and a lot of the people that are working in the field are excited by it. But right now, any interesting accomplishment has to do with supervised learning. The green one, right? Yeah. And the brown one is just a heuristic solution. Like, look at the velocity. So basically, the reinforcement learning here is learning from somebody who has certain rules. And how can that be guaranteed that you would generalize to somebody else, for example? So the question was, the green paddle learns to play this game successfully against this specific one brown paddle, with us operating under specific kinds of rules, how do we know it can generalize to other games, other things? And it can't. But the mechanism by which it learns generalizes. So as long as you let it play, As long as you let it play in whatever world you want it to succeed in long enough, it will use the same approach to learn to succeed in that world. The problem is this works for worlds you can simulate. Well, unfortunately, one of the big challenges of neural networks is that they're not currently efficient learners. We need a lot of data to learn anything. Human beings need one example oftentimes, and they learn very efficiently from that one example. So. And again, I'll, I'll, I'll, I'll talk about that as well. It's a good question. So the drawbacks of neural networks. So if you think about the way a human being would approach this game, this game of Pong, they would only need a simple set of instructions. You're in control of a paddle and you can move it up and down. And your task is to bounce the ball past the other player controlled by AI. Now the human being would immediately, they may not win the game, but they would immediately understand the game and will be able to successfully play it well enough to pretty quickly learn to beat the game. But they need to have a concept of control, what it means to control a paddle. They need to have a concept of a paddle, they need to have a concept of moving up and down and a ball and bouncing. They have to know, they have to have at least a loose concept of real world physics that they can then project that real world physics onto the two dimensional world. All of these concepts are concepts that you come to the table with. That's knowledge. And the kind of way you transfer that knowledge from your previous experience, from childhood to now. When you come to this game, that is something is called reasoning. Whatever reasoning means. And the question is whether through this same kind of process you can see the entire world as a game of Pong and reasoning is simply ability to simulate that game in your mind and learn very efficiently, much more efficiently than 200,000 iterations. The other challenge of deep neural networks and machine learning broadly is you need big data, inefficient learners. As I said, that data also needs to be supervised data. You need to have ground truth, which is very costly for annotation. A human being looking at a particular image, for example, and labeling that as something, as a cat or a dog, whatever objects is in the image, that's very costly. And for particularly for neural networks, there's a lot of parameters to tune. There's a lot of hyper parameters. You need to figure out the network structure first. How does this network look? How many layers, how many hidden nodes, What type of activation functions on each node. There's a lot of hyper parameters there. And then once you build your network, there is parameters for how you teach that network. There's learning rate, loss function, mini batch size, number of training, iterations, gradient update, smoothing and selecting even the optimizer with which you solve the various differential equations involved. It's a topic of many research papers. Certainly it's rich enough for research papers, but it's also really challenging. It means that you can't just plop a network down and it will solve the problem generally. And defining a good loss function or in the case of Pong or games, a good reward function is difficult. So here's a game. This is a recent result from OpenAI. I'm teaching a network to play the game of Coast Runners. And the goal of Coast Runners is to go. You're in a boat. The task is to go around a track and successfully complete a race against other people you're racing against. Now this network is an optimal one. And what it's figured out that actually in the game it gets a lot of points for collecting certain objects along the path. So what you see is it's figured out to go in a circle and collect those, those green turbo things. And what is figured out is you don't need to complete the game to earn the reward. Now that more sort of. And despite being on fire and hitting the wall and going through this whole process, it's actually achieved at least a local optima given the reward function of maximizing the number of points. And so it's figured out a way to earn a higher reward while ignoring the implied bigger picture goal of finishing the race, which us as humans understand much better. This raises for self driving cars ethical questions besides other questions. You can watch this for hours and it will do that for hours. And that's the point is is. Hard to teach. It's hard to encode the formally define a utility function under which an intelligent system needs to operate. And that's made obvious even in a simple game. And so what is. Yep question. So the question was what's an example of a local optimum that an autonomous car so similar to the coaster what would be the example in the real world for an autonomous vehicle? And it's a touchy subject but it would certainly have to be involved the choices we make under near crashes and crashes the choices a car makes when to avoid. For example, if there's a crash imminent and there's no way you can stop to prevent the crash, do you keep the driver safe or do you keep the other people safe? And there has to be some even if you, if you, even if you don't choose to acknowledge it, even if it's only in the data and the learning that you do, there's an implied reward function there and we need to be aware of that reward function is because it may find something until you actually see it, we won't know it. Once we see it, we'll realize that oh that was a bad design. And that's the scary thing. It's hard to know ahead of time what that is. So the recent Breakthroughs from deep learning came several factors. First is the compute. Moore's Law CPUs are getting faster, 100 times faster every decade. Then there's GPUs, also the ability to train neural networks and GPUs. And now ASICs has created a lot of capabilities in terms of energy efficiency and being able to train larger networks more efficiently. There is larger. Well, first of all, in the 21st century there's digitized data, there's larger data sets of digital data and now there is that data is becoming more organized. Not just vaguely available data out there on the Internet, it's actual organized Data sets like ImageNet, certainly for natural language. There's large data sets, there is the algorithm innovations, backprop, back propagation, convolutional neural networks, LSTMs, all these different architectures for dealing with specific types of domains and tasks. There's the huge one is infrastructure is on the software and the hardware side. There's git ability to share an open source way software. There is pieces of software that make robotics and make machine learning easier. ROS TensorFlow. There's Amazon mechanical Turk which allows for efficient cheap annotation of large scale data sets as AWS in the cloud, hosting machine learning, hosting the data and the compute. And then there's a financial backing of large companies, Google, Facebook, Amazon. But really nothing has changed. There really has not been any significant breakthroughs. We're using these convolutional neural networks have been around since the 90s, neural networks have been around since the 60s. There's been a few improvements, but the hope is that's in terms of methodology. The compute has really been the workhorse. The ability to do the hundredfold improvement every decade holds promise. And the question is whether that reasoning thing I talked about is all you need is a larger network. That is the open question. So some terms for deep learning. First of all, deep learning is.