This podcast clip captures a conversation between Lex Fridman and George Hotz about autonomous driving. Hotz proposes a three-part taxonomy—Static (route-only/map/localization), Dynamic (real-time detection and behavior prediction), and Counterfactual (influence on other agents)—arguing static can be largely solved with HD maps and LiDAR-based localization, dynamic requires real-time models of other agents, and the counterfactual is the hardest open problem.
Questions this video answers
September 5, 2019.
"up to 100,000 miles" (quoted claim).
"10 centimeters without LiDAR" vs "1 cm with LiDAR" (verbal claims).
"100% with mapping and localization" (verbal claim).
Chapters
Foreign. So the way you leapfrog, right, is you come up with an idea or you take a direction, perhaps secretly that the other people aren't taking. And so cruise, Waymo, even Aurora, I don't know.
Aurora Zoox is the same stack as well. They're all the same code base even. And they're all the same DARPA Urban challenge code base.
So the question is, do you think there's a room for brilliance and innovation there that will change everything? Like say, okay, so I'll give you examples. It could be if revolution and mapping, for example, that allow you to map things, do HD maps of the whole world, all weather conditions somehow really well, or revolutionist simulation to where the. All the. What you said before becomes incorrect, that kind of thing. Any room for breakthrough innovation?
What I said before about. Oh, they actually get the whole thing. Well, I'll say this about. We divide driving into three problems, and I actually haven't solved the third yet, but I have an idea how to do it. So there's the static. The static driving problem is assuming you are the only car on the road. And this problem can be solved 100% with mapping and localization. This is why farms work the way they do. If all you have to deal with is the static problem and you can statically schedule your machines. Right. It's the same as statically scheduling processes. You can statically schedule your tractors to never hit each other on their paths because you know the speed they go at. So that's the static driving problem. Maps only helps you with the static driving problem.
Yeah, the question about static driving.
Yeah.
You just made it sound like it's really easy.
Stack driving is really easy.
How easy? How? Well, because the whole drifting out of lane, when, when Tesla drifts out of lane, it's failing on the fundamental static driving problem.
Tesla is drifting out of lane. The static driving problem is not easy for the world. The static driving problem is easy for one route.
One route and one weather condition with one state of lane markings. And like no deterioration, no cracks in the road.
I'm assuming you have a perfect localizer. So that solves for the weather condition and the, the lane marking condition.
But that's the problem is how could you. How do you have a perfect.
You can build perfect. Localizers are not that hard to build.
Okay, come on now. With, with. With lidar.
With lidar. Yeah.
Or with lidar.
Okay. Lidar. Yeah, but you use lidar, right? Like use lidar, build a perfect localizer. Building a perfect localizer without lidar, it's gonna be. It's gonna be hard. You can get 10 centimeters without LiDAR. You can get 1cm width LiDAR.
I'm not even concerned about the 1 or 10 centimeters. I'm concerned if every once in a while you're just way off.
Yeah. So this is why you have to carefully make sure you're always tracking your position. You want to use LiDAR camera fusion, but you can get the reliability of that system up to 100,000 miles. And then you write some fallback condition where it's not that bad if you're way off. Right. I think that you can get it to the point it's like azzle D that you're. You're never in a case where you're way off and you don't know it.
Yeah. Okay, so this is brilliant. So that's the static.
Static.
We can. Especially with LIDAR and good HD maps. You can solve that problem easy.
The static static problem. So easy.
Very typical for you to say something is easy. I got it. It's not as challenging as ones. Okay.
Well, it's okay. Maybe it's obvious how to solve it. The third one's the hardest. Where do we get. And a lot of people don't even think about the third one and even see it as different from the second one. So the second one is dynamic. The second one is like, say there's a. An obvious example is like a car stopped at a red light. Right. You can't have that car in your map.
Yeah.
Because you don't know whether that car is going to be there or not. So you have to detect that car in real time. And then you have to, you know, do the appropriate action. Right. Also, that car is not a fixed object. That car may move and you have to predict what that car will do. Right. So this is the dynamic problem. Yeah. So you have to deal with this. This involves again, like, you're going to need models of other people's behavior. Do you.
Are you including in that. I don't want to step on. On the third one. But are you including in that your influence and people?
That's the third one.
Okay.
That's the Baron. We call it the counterfactual.
Yeah. I just talked to Judea Pro, who's obsessed with counterfactuals.
Oh, yeah, Yeah. I read his books. So the static and the dynamic.
Yeah.
Our approach right now for lateral will scale completely to the static and dynamic. The counterfactual. The only way I have to do it yet. The thing that I want to do once we have all these cars is I want to do reinforcement learning on the world. I'm always going to turn the exploiter up to max. I'm not going to have them explore. But the only real way to get at the counterfactual is to do reinforcement learning because the other agents are humans.
