Sacha Arnoud, Director of Engineering, Waymo - MIT Self-Driving Cars

Today we have the Director of Engineering, Head of Perception at Waymo, a company that's recently driven over 4 million miles autonomously and in so doing inspired the world in what artificial intelligence and good engineering can do. So please give a warm welcome to Sasha Arnoux.

Thanks a lot, Lex, for the introduction. Well, it's a pretty packed house. Thanks a lot. I'm really excited. Thanks a lot for giving me the opportunity to be able to come and share my passion with self driving cars and be able to share with you all the great work we've been doing at Waymo over the last 10 years and give you more details on the recent milestones we've reached. So, as you see, we'll cover a lot of different topics, some more technical, some more about context. But over the content I have three main objectives that I'd like to convey today. So keep that in mind as we go through the presentation. My first one is to give you some background around the self driving space and what's happening there, and what it takes to build self driving cars, but also give you some behind the scene views and tidbits on the history of machine learning, deep learning and how it all came together within the big Alphabet family, from Google to Waymo. Another piece. Obviously, another objective I have is to give you some technical meat around around the techniques that are working today on our self driving cars. So I think during the class you hear a lot, you've heard a lot about different deep learning techniques, models, architectures, algorithms, and I'll try to put that in a current hall so that you can see how those pieces fit together to build the system we have today. And last but not least, I think as Lex mentioned, it takes a lot more actually than algorithms to build sophisticated systems such as our self driving cars. And fundamentally it takes a full industrial project to make that happen. And I'll try to give you some color, which hopefully is a little different from what you've heard. During the week, I'll try to give you some color on what it takes to actually plan out such an industrial project in real life and essentially productionize machine learning. So we hear a lot of talk, we hear a lot about self driving cars. It's a very hot topic and for very good reasons. I can tell you for sure that 2017 has been a great year for Waymo. Actually only a year ago in January 2017, Waymo became its own company. So that was a major milestone and a testimony to the robustness of the solution so that we could move to a productization phase. So what you see on the picture here is our latest generation self driving vehicle. So it is based on the Chrysler Pacifica. You can already see a bunch of sensors. I'll come back to that and give you more insights on what they do and how they operate. But that's the latest and greatest. So self driving indeed draws a lot of attention and for very good reason. I personally believe, and I think you will agree with me, that self driving really has the potential to deeply change the way we look about mobility and the way we move people and things around. So only to cover a few aspects here, obviously that I don't want to go into too many details, but safety is one of the main motivations. 94% of US crashes today involve human errors. A lot of those errors are around distraction and things that could be avoided. So safety is a big piece of it. Accessibility and access to mobility is also a big motivation of ours. So obviously the self driving technology has the potential to make it very available and cheaper for more people to be able to move around. And last but not least is efficiency, collective efficiency. So not only we spend a lot of time in our cars in long commute hours, I personally spend a lot of time in long commute hours. And that time we spend in traffic probably could be better spent doing something else than having to drive the car in complicated situations beyond traffic. Obviously self driving technology has the potential to deeply change the way we think about traffic, parking spots, urban environments, city design. So that's why it's a very exciting topic. So that's why we made it. Our mission at Wavell is fundamentally to make it safe and easy to move people and things around. So that's a nice mission and we've been on it for a very long time. So actually the whole adventure started close to 10 years ago in 2009 and at the time that started under the umbrella of a Google project that you may have heard of called Chauffeur. And back in those days, so remember, we are before the deep learning days, at least in the industry. And so really back in those days, the first objective of the project was to try and assemble first prototype vehicle take off the shelf sensors, assemble them together and try to go and decide if self driving is even a possibility. It's one thing to have some prototype somewhere, but is that even a thing that is worth pursuing, which is a very common way for Google to tackle problems. So the genesis for that work was to come up with a pretty aggressive objective. So the first milestone for the team was to essentially assemble ten 100 mile loops in Northern California around Mountain View and try and figure out for a total of 1000 miles and try and see if they could build a first system that would be able to go and drive those loops autonomously. So they were not afraid. So the team was not afraid. So those loops went through some very aggressive patterns. So you see that some of those loops go through the Santa Cruz Mountains, which is an area in California that as you see, I'll show you a video that has very small roads and two way traffic and cliffs with negative obstacles and complicated patterns like that. Some of those paths were going on highways and one of the busiest highways. Some of those routes were going around Lake Tahoe, which is in the Sierras in California, where you can encounter different kinds of weather and again different kinds of roads conditions. Those routes were going around bridges and the Bay Area has quite a few bridges to go through. Some of them were even going through dense urban areas. So you can see San Francisco being driven, you can see Monterey, some of the Monterey centers being driven. And as you'll see on the video, those truly bring dense urban area challenges. So since I promised it, so here you're going to see some pictures of the driving. It's kind of working. So here with better quality. So here you see the roads I was talking about on the Santa Cruz Mountains, driving in the night, animals crossing the street, freeway driving, going through. Peter, that's the mantra area that is fairly dense. There's the aquarium there, a pretty popular one. That's the famous Lombard street in San Francisco that you may have heard of. San Francisco always brings its unique set of challenges between fog and slopes and in that case even sharp turns. So that was all the way back in 2010. So those 10 loops were successfully completed 100% autonomously back in 2010. So that's more than eight years ago. So on the heels of that success, the team decided, and Google decided that self driving was worth pursuing and moved forward with the development of the technology and testing. So we've been at it for all those years and have been working very hard on it. Historically, Waymo and I think all the other companies out there have been relying on what we call safety drivers to still sit behind the wheels, even if the car is driving autonomously. You still have a safety driver who is able to take over at any time and make sure that we have very safe operations. And we've been accumulating miles and knowledge and developing the system, many iterations of the system across all those along all those years. We reached a major milestone, as Lex mentioned back in November, where for the first time we reached a level of confidence and maturity in a system that we felt confident and proved to ourselves that it was safe to remove the safety driver. As you can imagine, that's a major milestone because it takes a very high level of confidence to not have that backup solution of a safety driver to take over where something to arise. So here I'm going to show you a small video, a quick capture of that event. So the video is from one of the first times we did that. Since then we've been continuously operating driverless cars, self driving cars in the Phoenix area in Arizona to expand our testing. So here you can see the video is playing. So you can see our Chrysler Pacifica. So here we have members of the team were acting as passengers getting on a back seat. You can notice that there is no driver on the driver's seat. So here we are running car hailing kind of service. So the passengers simply press a button, the application knows where they want to go and the car goes no one on the driver's seat. So we started with a fairly constrained geographical area in Chandler, close to Phoenix, Arizona. And we are hard working to expand the testing and the scope of our operating area since then. So that goes well beyond single car, single day. Not only we do that continuously, but we also have a growing fleet of self driving cars that we are deploying there all the way and looking for a product launch pretty quickly. So I talked about 2010 and we are in 2018 and we're getting there, but it took quite a bit of time. So I think one of the key ideas that I'd like to convey here today and that I will go back to during the presentation is how much work and how much work it takes to really take a demo or something that's working in the lab to something that you feel safe to put on the roads and get all the way to that depth of understanding, that depth of perfection in your technology that you operate safely. So one way to say that is that when you're 90% done, you still have 90% to go. So 90% of the technology takes only 10% of the time. In other words, you need to 10x, you need to 10x the capabilities of your technology, you need to 10x your team size and find ways for more engineers and more researchers to collaborate together. You need to 10x the capabilities of your sensors, you need to 10x fundamentally the overall quality of the system. Right. And your testing practices as we'll See in a lot of the aspects of the program, and that's what we've been, that's what we've been working on. So beyond the context of self driving cars, I want to spend a little bit of time to give you kind of an insider view of the rise of deep learning. So remember I mentioned that back in 2009, 2010, deep learning was not really available yet in full capacity in the industry. And so over those years, actually it took a lot of breakthroughs to be able to reach that stage. And one of them was the algorithm breakthrough that deep learning gave us. And I'll give you a little bit of a backstage view on what happened at Google during those years. So, as you know, Google has committed itself to machine learning and deep learning very early on. You may have heard of the Google Brain, what we call internally the Google Brain team, which is a team fundamentally hard at work to lead the bleeding edge of research, which is known, but also leading the development of the tools and infrastructure of the whole machine learning ecosystem at Google and Waymo, to essentially allow many teams to develop machine learning at scale all the way to successful products. So they've been working and pushing the deep learning technology, has been pushing the field in many directions, from computer vision to speech understanding to nlp, and all those directions are things that you can see in Google products today. So whether you're talking Google Assistant or Google Photos, speech recognition or even Google Maps, you can see the impact of deep learning in all those areas. And actually, many years ago, I myself was part of the Street View team and I was leading an internal program, an internal project that we called Street Smart. And the goal we had at Street Smart was to use deep learning and machine learning techniques to go and analyze Street View imagery. And as you know, that's a fairly big and varied corpus, so that we could extract elements that are core to our mapping strategy and that way build better Google Maps. So for instance, on that picture, so that's a panorama or a piece of a panorama from Street View imagery. And you can see that there are a lot of pieces in there that if you could find and properly localize, would drastically help you build better maps. So street numbers obviously that are really useful to map addresses, street names that when combined even on similar techniques from aerial views, will help you properly draw all the routes and give a name to them. And those two combines actually allow you to do very high quality address lookups, which is a common query on Google Maps, general text, and more specifically text on business facades that allow you to not only maybe localize business listings that you may have gotten by other means to actual physical locations, but also build some of those local listings directly from scratch. And more traffic oriented patterns, whether it's traffic lights, traffic signs that can be used then for ETA navigation, ETA predictions and stuff like that. So that was our mission. One of the, as I mentioned, one of the hard pieces to do is to map addresses at care. And so you can imagine that we had the breakthrough when we first were able to properly find those street numbers out of the street view imagery and out of the facade. Solving that problem actually requires a lot of pieces. Not only you need to find what, where the street number is on the facade, which is, if you think about it, a fairly hard semantic problem. What's the difference between a street number versus another kind of number versus other text? But then obviously read it because there's no point having pixels if you cannot understand the number that's on the facade all the way to properly geo localizing it so that you can then put it on Google Maps. And so the first deep learning application that succeeded in production, and that's all the way back to 2012 that we had, the first system in production, was really the first breakthrough that we had across Alphabet on our ability to properly understand real scene situations. So here I'm going to show you a video that kind of sums it up. So look, every one of those seg is actually a view starting from the car, going to the physical number of all those house numbers that we've been able to detect and transcribe. So here that's in Sao Paulo and where you can see that when all that data is put together, it gives you a very consistent view of the addressing scheme. So that's another example. Similar things, obviously we have more, that's in Paris where we have even more imagery. So more views of those physical numbers that if you're able to triangulate, you're able to localize them very accurately and have very accurate maps. So the last example I'm going to show is in Cape Town in South Africa, where again, the impact of that deep learning work has been huge in terms of quality. So many countries today actually have up to more than 95% of addresses mapped that way. So doing similar things. So obviously you can see a lot of parallelism between that work on 3D imagery and doing the same on the real scene on the car. But obviously doing that on the car is even harder. It's even harder because you need to do that real time and very quickly with low latency. And you also need to do that in an embedded system. So the cars have to be entirely autonomous. You cannot rely on a connection to a Google data centers and first you don't have the time in terms of latency to bring data back and forth. But also you cannot rely on the connection for the safe operation of your system. So you need to do the processing within the car. So that's a paper that you can read that dates all the way back to 2014 where for the first time, by using slack different techniques we were able to put deep learning at work inside that constrained real time environment and start to have impact and in that case around pedestrian detection. So as I said, there are a lot of analogies. You can see that to properly drive that scene like Street View you need to see the traffic light. You need to understand if the light is red or green. And that's what essentially will allow you to do that processing. Obviously driving is even more challenging beyond the real time. I know if you saw the cyclist going through. So we have weird stuff happening on the scene that you need to detect and properly understand, interpret and predict. And at the same time here I explicitly took a night driving example to show you that what you can choose when you take pictures of Street View and do it in daytime and in perfect conditions, driving requires you to take the conditions that they are and you have to deal with it. So there has been, from the very early beginning there's been a lot of cross penalization between the rear scene work. So here I took a few papers that we did in Street View that obviously if you read them, you'll see directly apply to some of the stuff we do on the cars. But obviously that collaboration between Google Research and Waymo historically went well beyond studio only and across all the research groups. And that still is a very strong collaboration going on that enables us to stay on the bleeding edge of what we can do. So now that we looked a little bit at how things happened, I want to spend more time and go into more of the details of what's going on in the cars today and how deep learning is actually impacting our current system. So I think during the, if I looked at the courses properly, I think during the week you went through the major pieces that you need to master to make a self driving car. I'm sure you heard about mapping localization. So putting the car within those maps and understanding where you are with pretty good accuracy perception, scene understanding, which is a higher level semantic understanding of what's going on in the scene starting to predict what the agents are going to do around you so that you can do better motion planning. Obviously there's a whole robotics aspect. At the end of the day, the car in many ways acts like a robot, whether it's around the sensor data or even the control interfaces to the car. And for everyone who has dealt with hardware and robotics, you will agree with me that it's not a perfect world and you need to deal with those errors. Other pieces that you may have talked about is around simulation and essentially validation of whatever system you put together. So obviously machine learning and deep learning have been having a deep impact in a growing set of those areas. But for the next minutes here I'm going to focus more on the perception piece, which is a core element of what the self driving car needs be to do. So what is perception? So fundamentally, perception is a system in the car that needs to build an understanding of the world around it. And it does that using two major inputs. The first one is prior on the scene. So for instance, to give you an example, it would be a little silly to have to recompute the actual location of the road, the actual interconnectivity of the intersection, of every intersection once you get on the scene, because those things you can pre compute. You can pre compute in advance and save your onboard computing for other tasks that are more critical. So really. So that's often referred to as the mapping exercise, but really it's about reducing the computation you're going to have to do on the car once it drives. The other big input obviously is what sensors are going to give you once you get on the spot. So sensor data is the signal that's going to tell you what is not like what you mapped. And the things is the traffic light red or green? Where are the pedestrians? Where are the cars? What are they doing? So as we saw on the initial picture, we have quite a set of sensors on our self driving cars. So they go from vision systems. Radar and LIDAR are the three big families of sensors we have. One point to note here is that they are designed to be complementary. So they are designed to be complementary first in their localization on the car. So we don't put them in the same spot because obviously blind spots is a major issue and you want to have good coverage of the field of view. The other piece is that they are complementary in their capabilities. So for instance, to give you an example, cameras are going to be very good to give you a dense representation. It is very dense set of information. It contains A lot of semantic information, you can really see a large number of details. But for instance, they are not really good to give you depth, or it's much harder, expensive to get depth information out of camera systems. So systems like LiDAR, for instance, will give you very good when you hit objects, will give you a very good depth estimation. But obviously they're going to lack a lot of the semantic information that you will find on camera systems. So all those sensors are designed to be complementary in terms of their capabilities. It goes without saying that the better your sensors are, the better your perception system is going to be. So that's why at Waymo, we took the path of designing our own sensors in house and enhancing what's available off the shelf today. Because it seemed important for us to go all the way to be able to build a self driving system that we could believe in. And so that's what perception does. Takes those two inputs and build a representation of the scene. So at the end of the day, you have to realize that in nature that work of perception is really what differentiates deeply differentiates what you need to do in a self driving system as opposed to a lower level driving assistance system. In many cases, if you do speed cruise or if you do a lot of lower level driver assistance, a lot of the strategies can be around not bumping into things. If you see things moving around, you group them, you segment them appropriately in blocks of moving things and you don't hit them, you're good enough. In most cases. When you don't have a driver on the driver's seat, obviously the challenge totally changes scale. So to give you an example, for instance, if you're on the lane and you see a bicyclist going more slowly on the lane right of you, and there's a car next to you, you need to understand that there's a chance that that car is going to want to avoid that bicyclist is going to swerve. And you need to anticipate that behavior so that you can properly decide whether you want to slow down, give space for the car, or speed up and have the car go behind you. Those are the kinds of behaviors that go well beyond not bumping into things and that require a much deeper understanding of the world that's going on around you. So let me put it in picture and we come back to that example in a court case. So here is a typical scene that we encountered at least. So here obviously you have a police car pulled over, probably pulled over someone there. You have a cyclist on the road moving forward, and we need to drive through that situation. So the first thing you can do, you have to do, obviously, is the basics. So out of your sensor data, understand that a set of point clouds and pixels belong to the cyclist. Find that you have two cars on the scene, the police car and the car parked in front of it. Understand the policeman as a pedestrian. So basic level of understanding, obviously, you need a little more than that. You need to go deeper in your semantics. Obviously you need, if you understand that the flashing lights are on, you understand that the police car is becoming an EV and is performing something on the scene. If you understand that this car is parked, obviously that's a valuable piece of information that's going to tell you whether you can pass it or not. Something you may have not noticed is that there are also cones. So there are cones here on the scene that would prevent you, for instance, to go and drive that pathway if you wanted to. Next level of getting closer to behavior prediction, obviously, if you also understand that actually the police car has an open door, then all of a sudden you can start to expect a behavior where someone is going to get out of that car. And the way you swerve, even if you were to decide to swerve, or the way someone getting out of that car would impact the trajectory of the cyclist, is something you need to understand in order to properly and safely drive. Only then, only when you have that depth of understanding, you can start to come up with realistic behavior predictions and trajectory predictions for all those agents on the scene so that you can come up with a proper strategy for your planning control. So how is deep learning playing into that whole space? And how is deep learning impacting used to solve many of those problems? So remember when I said, when you're 90% done, you still have 90% to go. So I think that starts to beat us. I also talked about how robotics and having sensors in real life is not a perfect world. So actually it is a big piece of the puzzle. So I wish sensors would give us perfect data all the time and would give us a perfect picture that we can reliably use to do deep learning. But unfortunately, that's not how it works. So here, for instance, you see an example where you have a pickup truck, so the imagery doesn't show it, but you have smoke coming out of the exhaust, and you have exhaust that's triggering lidar laser points. Not very relevant for any behavior prediction or for your driving behavior. So those points, obviously, and it's safe to go and drive through them so those are very safe to ignore in terms of scene understanding. So filtering a whole bunch of data coming off your sensors is a very important task because that reduces the computation you're going to have to do. But also key to operate safely, a more subtle one but important one, are around reflections. So we are driving a scene. There's a car here on the camera picture. The car is reflected in the bus. And if you just do naive detection, you're going to especially that if the bus moves along with you and everything moves, which is very typical, and everything moves, then you're going to have, all of a sudden you're going to have two cars on the scene. And if you take that car too seriously all the way to impacting your behavior, obviously you're going to make mistakes. So here I showed you an example of reflections on the visual range. But obviously that affects all sensors in slightly different matters. But you could have the same effect, for instance, with LIDAR data, where for instance, you drive a freeway and you have a road sign on top of the freeway that will reflect in the back window of the car in front of you. And then showing a reflected sign on the road, you better understand that the thing you see on the road is actually your reflection and not try to swerve around and trying to avoid that thing on the 65 mph trajectory. So that's a big complicated challenge. But assume we are able to get to property sensor data that we can start to process with our machine learning. So by the way, a lot of the signal processing PCs actually already use machine learning and deep learning because as you can see, for instance, in the reflection space, you need to, at the end of the day, you can do some tricks to understand the difference in the signal. But at the end of the day, at some point for some of them, you're going to have to understand to have a higher level of understanding of the scene and realize it's not possible possible that the car is hiding behind the bus, given my field of view, for instance. But assuming you have good sensor data, filtered out sensor data, the very next thing you're going to want to do is typically apply some kind of convolution layers on top of that imagery. So if you're not familiar with convolution layers, so that's a very popular way to do computer vision because it, it relies on connecting neurons with kernels that are going to learn layer after layer features of the imagery. So those kernels typically work locally on the sub region of the image and they're going to understand lines, they're going to understand contours, and as you build up, layers are going to understand higher and higher level of feature representations that ultimately will tell you what's happening on the image. So that's a very common technique and much more efficient, obviously, than fully connected layers, for instance. That wouldn't work. But unfortunately, a lot of the state of the art is actually in 2D convolutions. So again, they've been developed by.