EP 486: Autonomous Driving: How new NVIDIA tech will make it a reality

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life. This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and. And everyday life.

Could this year be the year that autonomous vehicles become the norm?

Right.

You know, I know we've been hearing about how AI is going to, you know, improve autonomous driving, and it's going to bring us, you know, fully autonomous vehicles. We've been hearing this for many years, but I think today's guest is going to help really open our eyes and our ears to maybe some of these new technological breakthroughs that have actually changed the research going into this and might make this a possibility. And, you know, we are here at Nvidia's GTC conference, a lot of new announcements when it comes to autonomous vehicles. And we're going to be going over those and, you know, talking about how this might impact all of us, right? The roads we drive, the safety of the future of autonomous vehicles. And I'm very excited for today's conversation. I hope you are, too. So, hey, what's going on, y'?

All?

My name is Jordan Wilson, and welcome to Everyday AI. This is your daily live stream, podcast and free daily newsletter, helping us all keep up with the ever changing world of AI and how we can actually learn from the experts, you know, helping build it all so we can grow our companies and our careers. That's what you're trying to do. You're definitely in the right place. Make sure, if you haven't already, please go to your everydayai.com Sign up for the free daily newsletter. We're going to be recapping all of the important insights from from today's interview there, as well as everything else that you need to stay ahead. All right, so enough chit chat. I'm excited for today's guest and talk about everything new that Nvidia is working on in the autonomous vehicle space. It is a lot, so please help me. Welcome to the Everyday AI Show, Marco Pavone, the associate professor, an associate professor at Stanford University and the lead autonomous vehicle research at Nvidia. Marco, thank you so much for joining the Everyday AI Show.

Thank you for having me.

All right, I'm excited for this conversation. This is, I think, one of those hot topics that people love talking about. But before we dive in, can you just tell everyone a little bit about your background and what you do here at Nvidia?

Yeah, so I'm aeroboticist. So I'm a faculty at Stanford and I also lead research at Nvidia. My work is in the field of autonomous robotics. So how do we make robotic systems capable of making decisions on their own, especially in high stake applications like for example, self driving vehicles or out of space vehicles. And prior to joining Stanford, I was a roboticist at another jet propulsion lab. So still working on self driving vehicles but on Mars.

Yeah, yeah. It's not every day you get to talk to someone that's, you know, helped the autonomous, you know, which, which project was that on Mars that it was.

This was one of the Mars landing missions where NASA sent a rover to Mars and the mission was a high success.

Love that. So yeah, we're bringing some research from Mars to your streets here. So there was a lot that's been announced so far at Nvidia GTC when it comes to autonomous vehicles. But one of the things I wanted to talk about is Halos. So can you tell our audience what that is and how it's ultimately going to impact everyone else on the roads?

Yeah, sure. So Halos is a full stack system that comprises hardware, software, tools and safety principles to combine all of these elements into a safe driving stack. And it's exciting because it's basically unifying all the investments that Nvidia has been made in the past two years on the topic of automotive safety into a unified program. And I believe this program is going to boost safety both with respect to the, you know, own ad program, internal at Nvidia, but also helping partners in making their program, the programs, even safer.

So, you know, can you just bring us up to speed? Right, because you know, even people, you know, who've listened to this show, we've talked about autonomous vehicles and AI I, you know, in years past. But bring us up to today what, you know, what has been successful because there's obviously, you know, fully self driving cars on the road, right. In certain states where it's allowed. But you know, where are we at, you know, in the, you know, fully self driving autonomous vehicles? What's working, what's not?

Well, autonomous vehicles are becoming a reality and I'm sure that you have heard this sentence many times in the last few years. But yeah, like, you know, if you come to San Francisco, you will see Robo taxis providing rights to customers without any CPT driver on board. And very advanced driver assistance systems like for example the Tesla Autopilot are becoming available on a massive scale. So My point is that the self driving technology is now graduating into becoming a consumer technology. That's why I feel confident by saying that the autonomous vehicles are becoming a reality. We still have challenges, so we have, you know, solved. The problem yet is I like to draw a parallelism with respect to aviation. It took us like a hundred years to get to an industry that is as safe as it is today and as efficient as it is today. So this is a marathon in is not a sprint. What are the key technological challenges? Well, in the context of full self driving vehicles like robo taxi systems, their deployments are still relatively limited in few cities in the world. And making scaling up those deployments still represent significant technological challenges as requires scaling the algorithms allow the algorithm generalize to new situations in a much more effective way. So we need some innovation there throughout the development cycle, from simulation to training to algorithm design and so on. Same thing for semi automated systems. The semi automated systems are already available worldwide, but of course we want to increase the availability at. That's a game that requires technology.

And can you maybe just you know, help our audience better understand Nvidia's footprint right now in the, you know, kind not even just the autonomous vehicle industry, but just the auto industry in general. Because, you know, it's something I've found out through the years by getting to talk to, you know, really smart people such as yourself. But, you know, people don't know, you know, you probably have, you know, or might have Nvidia, you know, hardware in your car, right, Like Tesla, you know, is probably using, right, Nvidia's data, you know, their data centers. But can you just bring us up to speed? What is Nvidia's footprint right now in the auto industry?

Yeah, absolutely. So Nvidia first of all is both a product company and an ecosystem company. So Nvidia has a substantial investment in developing its own autonomous vehicle solution which we call Nvidia Pride. And it is doing so in collaboration with partners such as, for example, Mercedes. And it's also helping, since it's also an ecosystem company, it's also helping other AV companies to develop their own AV programs in many different ways. There is not like a unique recipe. It could be by providing the Nvidia automotive graded chip. Many of the autonomous vehicle companies out there are using indeed Nvidia hardware. It may be by providing the data centers to train the AI. It could be by providing simulation technologies and so on and so forth. Every company is a bit different, but this is also what from a researcher like myself makes it exciting because I have an opportunity to really scale up my contributions even beyond the confines of Nvidia to really the entire ecosystem.

So, you know, speaking of, of different brands or different companies, also some exciting news. Nvidia and GM talk about this partnership little bit and you know, when we, when we might see, you know, that partnership absolutely out there on the roads.

Right.

I'm not going to hold you to it.

Right.

But like, what, what's coming in this partnership?

Well, in terms of timing, as a little bit of sensitivity, of course comment on that, I would say though, that is a super exciting partnership. I was also impact involved in the discussion, so I'm really happy to see that coming to fruition. One of the interesting aspects of it is that there's a partnership regarding automotive, but also manufacturing and potentially email. Robotics is a very broad partnership that also plays well with Nvidia's broader ambitions. Of course, as we said before, Nvidia has a very strong autonomous vehicle program. But Nvidia is also scaling up this program to what we refer to as a physical AI program, whereby cars are just one sensation, one embodiment of a broader concept that is that of physical AI, in addition, for example, to humanoids or autonomous mobile robots and so on. So this collaboration with General Motors can also involve this kind of broader vision.

So like simply like, does this just mean in the future, you know, are we going to see, you know, different versions of GM's vehicles have autonomous capabilities? Or is the long term goal with this partnership to have maybe most or all of GM's vehicles? Like, what's that going to look like? Is it just going to be kind of like certain, certain vehicles in the future are going to, you know, kind of benefit from this partnership? Or is it just kind of all vehicles, like, longer down the line?

As I said, this is. Okay. No, no, no. Okay. All right, all right.

We'll, we'll, we'll, we'll just have to follow up on that when the, when the news does come out. But you know, so I'm curious, like.

You know, very fresh out of press.

Yeah, it's, it's extremely fresh. Extremely fresh. So, yeah, we'll, we'll follow up with that once, once that news is officially released. But you know, one thing that I do want to talk about is, you know, and we even mentioned this, right, like there's been a lot of excitement right, in this space for 3, 5, 10 years. What specifically do you think has changed over the last couple of years that that leads you and your team to believe that, you know, now is this time, you know, that, you know, we are kind of hitting that, that moment when this might become much more common. Autonomous vehicles on the road.

So for first of all, even more than 10 years. Right?

That's true, yeah.

Doing my PhD between 2006 and 2010 at MIT, that was the time where the autonomous driving technology was starting with its baby steps. I would say it's almost been 20 years.

Okay, there we go.

So the way we build autonomous system today is very different from what we were used to do 20 years ago. Of course there have been of course a lot of lessons learned, but most importantly, the technology has changed. And there is not like this single technology that has really changed the game completely, but a convergence of technologies from all the way from hardware, so basically having dedicated chips and dedicated sensors all the way to as you can imagine, AI becoming pervasive in the design of autonomous systems. And in that context, I think one of the most exciting opportunities is a little bit debated in the community is the opportunity of leveraging so called Internet pre trained models. You might be familiar with ChatGPT for example, with the idea that with this type of models we can, we have an opportunity to bring Internet scale knowledge to the task of driving. Think about how you learned about driving. It took you a few hours probably to learn how to drive a car simply because you brought a lifetime of experiences sure to the task of driving. Well, that's the part of this about behind using this kind of interrelated models to bring multiple lifetime of experiences of generalist knowledge to the task of driving. So that's another AI in general and potentially internal pre trained models in particular provide the key opportunities to improve the technology. And another, I would say big technology that has made amazing process in the past two, three years is the simulation technology. And Nvidia actually had a number of announcements related to simulation. Simulation is, has always been a holy grail in robotics. And now finally we are simulators that we can use throughout the development life cycle from the training of the vehicle, the training of the AI, all the way to the testing of the AI. So the key, the challenge with simulation historically has been the so called simulation to realism gap. And this gap is becoming increasingly closer along a number of dimensions in terms of visual realism, in terms of behavioral realism, how faithfully we replicate the behaviors of humans in the road, and so on and so forth. So long story short, I wouldn't say it's a single technology that is really pushing this industry forward, is really a convergence of technologies from the Chip all the way to the algorithm or the simulation that all of this, you know, now are finally coming together.

Yeah. And, and I do want to talk about that a little bit more. So kind of this, this concept of using, you know, Nvidia's new generative AI technology, Cosmos. Correct. So like walk us through that. And you know, I know this might be difficult, you know, to imagine on the podcast, so in the newsletter we'll link to some of these videos, you know, and how Cosmos helps. But walk these, these simulations and how specifically.

Right.

Maybe, you know, since we've hit this generative AI wave, how does that help with, you know, Nvidia's ability to use more diverse simulations? And how does that make ultimately the autonomous vehicle sector safer?

Yeah, that's a great question. So typically simulation is restricted to the scenarios that are authored by a human. So a human saying, okay, I want to test a vehicle with a particular intersection. So I'm going to draw a map with respect to which a vehicle has to drive. It's fine. Of course, it doesn't scale to millions of cases. Right. Or there are new technologies referred to as neural reconstruction technologies that allow you to reconstruct in 3D scenarios out of drives that are recorded. This is all fine, but for autonomous vehicles, it's really a game of the last 5% or 1%. It's all about thinking about very complicated corner cases. And that's where generative simulation comes in. So this new technology, and Cosmos is one of the prominent examples, allow you to simulate, allowing you to generate a simulation out of actual prompts or images. So this allows you to create a completely new simulation scenarios to, for example, stress test your vehicle. So it really allows you to automatically and in a way that is highly scalable, generate a plethora of corner cases that can allow you to better, more robust systems and also test those systems. Now, it's still a technology in a development, so there are still challenges. For example, physics realism is a challenge. To what extent is generated Simulations, for example, obey the law of physics. But there, there's been quite a bit of progress in improving the physics realism. So I'm very hopeful that this technology will be yet another tool that there's no time engineering I can leverage in order to build a more capable and safer autonomous vehicles. I think not just for Nvidia, I believe that the Cosmos and broadly generated simulation is going to have a significant impact in the whole industry, but more broadly in the whole robotics industry.

Yeah, And I think that's really important to bring up because you know, especially if you've been to a city like San Francisco or, you know, I've seen, you know, Waymos and in Austin, Texas. Right. So these, these vehicles and this technology has, you know, it's been out on the roads, right. It's out there in the wild. Which, you know, allows Nvidia and other players in the space to gather that actual real life data. Right. So, you know, I'm wondering, you know, if the simulation side is improving and you're able to, you know, simulate more scenarios with the Cosmos platform. You know, what are still some of those bigger hurdles, right, Aside from, you know, just more time and, you know, more data from the real world, from the cars. Right. What are some of those other big hurdles that the space is still looking to maybe overcome?

Well, data is a big one. So simulation is going to help, but you still need to have data, real data to ground your, your system. So the, the hurdle is to make technologies, AI technologies, they can adapt with an increasingly lower amount of data to new areas. And it is a problem that maybe it's a bit difficult to appreciate, but it's crucial. Transportation is a very location specific phenomenon. Like, I don't know if you have Italian followers, I'd like to give an example that actually this is also a tip for you. If you go to Italy, I love it.

Please, please, I have to, I have to soon.

If you go to Italy, as someone blinks the light at you, typically that is a kind sign. It means you can cross in front of me, I'm eating at you. If you go to the south of it, and it's typically blinking is an aggressive sign. It means that don't you dare cross in front of me because I'm not going to stop. And if you cross me, we're going to crash.

Right?

So it's the same country, few hundreds kilometers apart, with two completely different behaviors. So this is what I'm saying. You still need to have a some location specific data to train your AI. So then the game is how do. But that is expensive to acquire. So how through better simulation and better algorithms, we can decrease the reliance on real data. There always will be some need for real data. The question is how we can reduce it so that we can really quickly expand to new domains. Technically we refer to those as operational design domains. To really make this technology financially viable, we know it's technologically feasible. Waymo is developing robot access in San Francisco. But to make it financially viable, we have to be able to scale it up. Scale it up means to be Able to kick a data flywheel that is not too onerous in terms of how much data we need. Simulation is one tool, but better algorithm design is another tool, driving down the cost of some key sensors. Like, for example, as you might have heard, there is quite a bit of discussion in the community in terms of to what extent you want to have a sensor system that is very much camera centric or maybe also relying on ladders and so on and so forth. So these are additional discussion. Again, it's not just a single technology, it's a combination of technologies. But I would say the capability of scaling operational design domains more seamlessly is the major challenge which will be solved through a combination of a number of technologies, from redundancy, the sensor level to simulation, to algorithms that adapt more quickly to new scenarios with the best.

That's, that's a fascinating example, right, Because I never thought about that, that, you know, it's not just a one size fits all approach for autonomous vehicles. Yeah, yeah, you're like, definitely think about that. So, like, like as an example, right? Like even I'm thinking in the United States, people have different driving styles, right? From state to state, city, you know, big city, urban areas. Right. So, you know, I'm curious, you know, what are some maybe successes that you've found in addressing those? Or how do you even go about knowing those things? Right? Aside from like, you know, you, you know, because, you know, you, you have lived there, you know, but for everyone else and for all these other challenges that maybe the industry hasn't thought about, I mean, how do you, you know, start to tackle these issues? Is it just, you know, maybe, oh, there's, there was an accident and we don't know why because, you know, all of our data was right and, you know, is that kind of how you discover these things?

Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies.

In the world do.

Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on Genesis. Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Genai. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on to use Gen AI. So whether you're looking for chat, GPT training for thousands, or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies.

In the world do.

Go to your everydayai.com partner to get in contact with our team. Or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on Geni.

Yeah, so that's one of the reason why the AV industry has been increasingly shifting from a paradigm whereby most of the possible cases were hypothesized by humans and then coded into the brain of the autonomous vehicles, to a data driven paradigm where we let the autonomous vehicle learn from the experience, basically from demonstrations, because that is more scalable as a technology. So all these new behaviors essentially are to a large extent learned from data that is acquired through other test vehicles or even maybe dashcam videos. The good thing is that videos abound on the Internet and that's yet another opportunity to more seamlessly scale up the AI to new operational design domains. So bottom line is that these days many of those behavioral nuances are learned through data, which again means that we need to have technologies that allow us to make as much use, as much efficient use of this data as possible. The good thing is that data abounds. So we do need data acquired from a test fleet for sure. But one thing that abounds on the Internet is videos, especially driving videos. And that is again a new modality that as an autonomy engineers we have at our disposal to allow the scaling of this technology in an Even more efficient way.

And I'm curious because on the data point, I may be wrong in this assumption, but I'm guessing very early on a lot of the data that you might get, you know, from those driving videos, let's say if there is a bunch from five years ago, I'm guessing that the other cars for the most part were not autonomous.

Right.

They were driven by humans. So. So, you know, I'm curious, like what are you all at Nvidia and again, maybe just the broader industry doing to account for that. Right? Because what if in five years it's 10% autonomous vehicles? And you know, how can you say, oh, in the training data this was an autonomous vehicle versus this is a human. And you know, what challenges does that bring in the future? Will it just be autonomous vehicles be able to more communicate with each other so they know, okay, you're an autonomous vehicle, you have the HALO system, so this is what you're going to do?

That's a great question. And there are a lot of sub questions. So first of all, as you alluded to, now that we're moving toward a more data driven paradox, then the brain of the autonomous vehicles becomes very much dependent on the data that you use to train the brain. So it becomes imperative and is one of the hallmarks of Kalos to develop AI creation workflows that allow you to remove unsafe behaviors or biases from your training set. And interestingly, this is another domain where we use internal pre trained models now not as drivers, but but as judges that allow us to judge whether a given demonstration is a demonstration of safe driving or not. And of course there are humans in the loop to align the judgment of these AI models to we as humans, we think is acceptable driving. But we train this basically AI models who serve as judges at scale to remove all those biases that you were mentioning. Now moving forward, AV is a very weird technology because we are solving the hardest problem first. Going back again to aviation, you know, I have this kind of dual aerospace and airspace background. It's like as if the brothers as the first problem they wanted to solve was a supersonic flight. That's the state of the AV industry. We are solving the hardest problem first because we're solving the problem where there are only few automated vehicles. Everybody else is a human. Like you have human driven vehicles. There is no dedicated infrastructure, but it's just basically what it is now in the future there will be a higher penetration of autonomous vehicle. So your question is how that will change. The technology definitely provides an opportunity to make the technology even safer. But as with everything, you have an opportunity, but you also have a challenge. What is the challenge? Well, there are multiples. So let's assume that, for example, autonomous vehicles could communicate with each other. In principle, that is great because it allows some level of coordination which has clearly an immediate impact on safety, but it exposes your decision making capabilities to external interference. So, for example, cybersecurity becomes much more of a threat than it is now, where the system is basically very much confined within the vehicle. You might introduce latencies. You know, sometimes when you, when you have a call on your phone, you don't care if your communication drops for a little bit. That could be fatal in the case of an automated vehicle. So I'm not to say that these are impossible challenges. I'm just saying that it's not as simple as people might think. Think that vehicle to vehicle, vehicle to infrastructure might simplify the problem, let alone the challenge of who is going to place that infrastructure, how you're going to standardize that infrastructure. So yes, when there will be high reputation autonomous vehicles, there will be opportunities to make this technology even safer. But the exact mechanics about how we will do it is still subject of the discussion.

So, Marco, we've covered a lot in this conversation. So, you know, as we wrap up, what do you think is maybe the one most important thing for our viewers and listeners to know about? You know, specifically even new advancements that were announced here at GTC and how that is seemingly going to quickly change the future of autonomous driving on our roads?

I think there were two unanswered broadly that are going to have a significant impact in the field of vehicle autonomy. First, all the announcements related to simulation and second, announcements related to foundation models, Internet retraining models that would be used in the context of physical AI. One of the big announcements, actually this announcement was made at CES and then has been refined at GDC is that of Cosmos, which is by now a sort of umbrella term where we cover both video generation models, particularly useful for simulation and reasoning models, particularly useful for autonomous driving in the real world. So these are definitely technologies that are worthwhile to keep in mind if you are an AUTONOMY researchers. Many of these technologies are available open source on the aging phase and at Nvidia. That's one of the reasons why I'm excited about being at Nvidia. We publish a lot, we share a lot of our knowledge. And again, this is because Nvidia is both a product company and an ecosystem company. So we want to make sure that as we grow, the entire ecosystem really grows.

I love it. I think I just became so much more informed on everything, Autonomous vehicles, what Nvidia is working on. And I really hope that our audience did as well. So, Marco, thank you so much, much for your time and coming on the Everyday AI show to share with us. We appreciate it.

Likewise. And if you want to become an Autonomy engineer, you're welcome to take some of the courses.

There we go. Hey, at least I'm. I'm part part of the way there, right? I went from zero to one, I think so. Hey, that was a lot of fantastic information. Nvidia is working on a ton of advancements, so if you missed anything in there, if you want to know more, we're going to be recapping today's conversation in the newsletter. So thank you for joining us. If you haven't already, go sign up for that newsletter.

Read it today.

It's going to be a great one at your everydayai. Com. So thank you so much for tuning in. Hope to see you back tomorrow and every day for more Everyday AI. Thanks, y'. All.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.