EP 233: Robots Among Us - How NVIDIA is building the future of robotics

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

It seems like in the past couple of weeks, robotics have been everywhere, and that can't even be truer than this week, as we've seen at GTC here at the GTC Conference with Nvidia. I'm extremely excited today to talk about the robotics industry and what's been going on, and some recent announcements from Nvidia with the man himself, but more in that in a second. As a reminder, if you're joining us live, thank you very much. Please make sure to check out your show notes or if you're on the podcast, because you can still, it is not too late to register for free for the Nvidia GTC Conference. So you can do that. You can go watch sessions, win a gpu. You know, there's DLI credits. You can go out and learn a lot of things. So make sure to check that out in the show notes and in the newsletter as well, all right? And you can always get that information on our website too, at your everydayai.com. all right, enough of that. Let's get into why you showed up today, why you're listening to the podcast. It's to hear about the future of robotics. And there is so much going on, especially it seems like in the past couple of weeks, in the past couple of days here at Nvidia gtc. So with that, please help me welcome our guest for today, Amit Goel, the Director of Robotics at Nvidia. Amit, welcome to the Everyday AI Show.

Thanks, Jordan. Thanks. My pleasure to be here today.

Yeah. Can you tell us just a little bit about what you do at your role at Nvidia?

Sure. So I head up our ecosystem team for robotics and the product management for our embedded products that go into the robots. Been at Nvidia for about 13 years, and so my team is working with all the partners that you see that are building the next generation of AI robots.

And I'm sure over the course of 13 years working at Nvidia, there's been a fair share of excitement and new advancements. But how do you feel about the state of robotics today?

At least from an outsider, from an.

Everyday person like myself, it seems like the sector is just exploding. Is that how you feel as well? Or does it always just feel like, you know, like. Like this?

No, I think, you know, with the coming of AI, robotics has been making steady progress. But over the last 612 months, we've seen sort of a shift that has happened, an inflection point, if you may, where people are starting to realize that there is a lot that can be done, and a lot of that can be attributed to the large language models and generative AI. The fact that you can now have a lot of data and use that to make new tasks for robots is just a turning point. The robots have been there for a while. AI has been helping, but they have been typically still a single function robot, right? Amazon talks a lot about their robots in the warehouses that are picking and packaging things for you, but that robot, if you ask it to make a coffee for you, can't do it. So that is the change that has happened, the change where now with the large models, you can really train a robot on multiple modalities. Understand from what you and I are doing, right, you don't need a PhD to be programming a robot. That's what is going to really unlock the capabilities of robots for everyone.

If you're watching now, what Amit just said is amazingly accurate in putting everything into perspective. And why now? But I want to dig in even deeper because what, you know, people may or may not realize robotics is not new, right? Robotics have been used across many industries and verticals for decades. Why, though, that inflection point now, you know, you said over the last couple of months, is it just because of large language models, or are there other factors that are kind of bringing the robotics industry to this inflection point?

I would say it's a confluence of several things that has made this possible. Right at the heart of it is, of course, this ability of large language models to consume huge amount of data and learn from it without somebody actually having to annotate everything. You may have seen the work from Nvidia research called Eureka, where we taught a hand to spin a pen. In the past, even just telling what what the robot needs to do was so hard. Right now you can type. I want you to move this pen with your fingers and all the complex math underneath it is taken care of by a large model. So that's one thing. The second thing that has happened is in order to develop these robots and test these robots, you can't really do them in physical world. These robots are expensive. Physical world is, you have to be careful, safe around it. This simulation has come a long way. We started working in this platform called Omniverse, which is designed for simulating things. We announced the Isaac Lab. This is where you can safely and very quickly test these robots. That's the second thing that came out. And the third thing was in general, the computing power to run these big rope models, to run all the AI capabilities, you need to have the computing power in the robot. You cannot be running its brain somewhere else in the cloud because it needs to just make a decision if it needs to stop moving its hand or not. So I think the culmination of those three things, the large language models, a great simulation environment to test these robots before you put them in the real world. And the third, all the horsepower that you need to do the processing, it all came together. And so that's what has kicked off this big bang of new generation of robots.

Yeah, and it's definitely that even if we look out here on the GTC floor, you see robotics everywhere. Maybe that you wouldn't expect if you were at the GTC conference 10 years ago. You maybe might not expect this, but I do want to zoom out a little bit and even talk specifically about Nvidia's role in all this, because you're not technically making the robots. So tell, tell us a little bit about how Nvidia works. It seems like there's so many moving parts, both literally and, and physically. Right. Or, you know, metaphorically, I guess. But how does Nvidia's role play into the, the grander scheme?

That's a great question. I mean, when people look at all our demos and things that we show for our technology, they often get confused. Like look at Media's making a robot, we're making a humanoid, we're making a mobile robot. That's not what we do. Our ambition is to be the foundry, be that platform where the robotics developers, the people who are making the hardware, the people who are making the end application, can come to this foundry with their use case, with their data, with their robot hardware, and take back the intelligence for those robots. So that's Nvidia's role being that underlying platform on which people can bring their stuff, build the final thing, take it back and deploy it in the real world.

What does that actually mean when we talk about Isaac Lab? I guess inside or underneath the Nvidia Omniverse? What does that actually mean you know, for, you know, the world of manufacturing, production, etc. How does that change what companies and enterprises can do if they bring their simulation into Isaac Lab?

That's great question. For example, if you see the announcements that we had today at gtc, this week at gtc, I'll highlight a few. We had an announcement with Yaskawa industrial robots that you will see in a lot of manufacturing, a lot of logistics places, typically the way those robots were programmed was you get the robot, somebody goes there, spends several weeks, months programming them, and then you have one application done. Now with Isaac Lab and the work that we're doing with Yaskawa, they can create a model, a digital twin of their robot. In Omniverse, they can use the foundation models, these models that already know how to pick things, but they don't really know how to pick things for that particular robot. So that's the adaptation that they need to do specific to their hardware. These are things that we call fine tuning. They can create a digital trend. They can fine tune them and take out of it a working skill, a working task. And that's what we demonstrated here with Yaskawa. They worked with us, brought their robot in all the secret sauce of their robot, trained it in the lab, right in the virtual lab, and then deployed it to the real world. Similar things that we did with Boston Dynamics. They have probably the best team to make some of the amazing robots. Their mechanical design, the dynamics, the software that runs on the robot is fantastic. But every new skill to be added to the robots took a long time. And together with them, we announced an end to end platform. You can take the Boston Dynamics robot, they worked with us to create a simulation digital twin of their robot. And now you can make it run in the sand. You can create a snow environment, you can create forest environment, learn all those things. And when it's learned and take this brain back, put it on an Nvidia Jetson, right. To put it on the robot and you have your skill. So that's essentially what it looks like. Bring your hardware, bring your use case, create the environment, test it, take it back to the real world.

Yeah. I think the only downside of that is we may get fewer of those viral videos that we always see of the Boston Dynamic robot dog running across all different surfaces. But I'm really interested because even it seems public perception around robotics, it seems, is also changing at the same time. What does it mean as we even transition from general robotics to now what people are saying, humanoids, is it more of, you know, that that single use, you know, Amazon warehouse, that's still general robotics. But, you know, when a humanoid type robot can do anything, is that, you know, when it's a humanoid, can you explain, you know, what that even means as we kind of look forward into a future of these humanoids?

Yeah, it's if, if you look around you, you don't see Many robots, right? And even the ones that you see, probably the Roomba in your house, the challenge has been human robot interaction. As you and I cannot even explain what's going on in the Roomba's mind. It just keeps going around the same place, sometimes over and over again. Right. So in order to robots and people to coexist, there are two important things they need to operate in shared spaces. The world was created for us, for people. Most of the world has been created for us. The factories are created for, still created for a lot of robots. The new warehouses are created for robots, but majority of the world is created for humans. So they need to exist in a form factor that they can live around people. Second thing that is important is they should be able to engage with people. You should be able to understand what the robot is doing, and the robot should be able to understand what you want it to do. That's essentially why humanoid form factor is so interesting. We have a lot of data on humans, you and I doing our things. There is an ample amount of data for that. And for generative AI, the one thing that you need is a lot of data. There is a lot of data for humans that helps with the humanoid form factor. Again, these are designed with the dexterity and the capabilities so they can exist around people in the shared spaces. Last piece is with the large models, with this generative AI, you can really engage with them. We've seen some of the recent work where you can tell the robot what to do. You can even ask how it did it. Right. So you can have more trust and comfort around these machines.

Yeah. You talked about the need for a space for humans and robots to kind of coexist. I think maybe for people like you and I, maybe that's exciting. Right. But for some people might make them feel uncomfortable. I mean, should people feel afraid of coexisting with humanoids? And what does that look like? Or maybe what can you say to those people who do feel a little uneasy maybe sharing that space?

I think, you know, there is just, if you look at this data right now, there are just so many jobs where we don't have people to work on. And all of these robots will need human supervisors. Right. So instead of actually doing some of those work, anybody who's worried, they should not be, because they would be actually telling what these robots should do. Right. So we'll still have human supervisors determining the task, determining what the robots are expected to do. So there is just a lot of need for those skills that are. Robots have been great at dull, dangerous and dirty that we'll be able to offload to them and still keep monitoring them and control them and tell them what is needed.

Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies.

In the world do.

Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website or will help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen AI.

You know, aside from those, you know, dangerous and dirty, which I love, I just love the alliteration there. It makes it so easy to see where robots can go in the future. But I also assume that there's just more, you know, repetitive tasks that a lot of right now, you know, humans are doing. So and maybe that's the reason why a lot of people, you know, are scared of AI or are not fully on board with, with robotics. You know, I guess in that situation, whether you're talking about manufacturing, production plants, etc. Especially those, you know, kind of like low hanging, you know, you might think of them as low hanging fruit where these humanoids can go in and place and be placed there. For those humans in those industries, where should they even be focusing their time and attention right now? Because you can sit around and say, oh well, you know, what happens if a humanoid is out here doing this in two years or you can do something about it. So where would you focus or where would you recommend people, you know, maybe in those industries start focusing their time and attention on.

I think the first thing is to identify these tasks, right? That's the number one thing to do is to identify what are the of your operations that you could automate and start preparing for it. Right? You know, bringing automation is not just bringing a robot and putting it out there. You need to prepare the. You need to prepare your operations, your IT systems to work with those robots and you need to prepare your workforce. So I think just first thing is, you know, knowing what these robots are going to be capable of, right. Not assuming that they'll do everything, you know, all your job, not going to happen anytime soon, but they will be really good at, you know, picking things up from a cart, loading it up somewhere else. Right. We're seeing what's happening in warehousing, picking things, putting in the packages, shipping them out. So there are a lot of these things which are going to happen in the near term. Right. So identifying those tasks and then working with this ecosystem to help them define those tasks and preparing the environment for those things. So I think those are some of the areas where they should focus on and not be worried about it at all. Yeah.

The Nvidia CEO, your CEO Jensen, yesterday said in a session that the ChatGPT moment for robotics is near. It's soon. What's your thoughts on that and what do you think that moment will actually be for the robotics industry or humanoids industry? Like, what is that ChatGPT moment, you think?

Yeah. What was different about ChatGPT versus prior AI was that how who could use it? Right. I think that the capabilities were of course amazing, but how accessible was it was the important piece as well. So I think for me, when I look at about think about ChatGPT moment of robotics, I think about somebody who does not have a robotic background. If they feel comfortable that I can use a robot, I can program a robot, I can tell what I wanted to do, I think that would be the big moment where you've said, okay, we're about there, right. Instead of that handful of specialist robotics, PhDs and master's degree people who can really work with these robots. When most people would feel comfortable using these robots, operating these robots, telling them what to do, I think that would be, in my mind, the moment.

What do you think still has to be done or still has to be accomplished in order to get there? Because at least for me, when I'm was listening to the keynote earlier this week at gdp and again, as a reminder, if you're listening live, you can sign up and re watch that keynote. Great, great. Two hours. You won't regret it. But it seems like all of the pieces to, to get to what you just described are either there or were recently announced. So what are still the obstacles to get to that moment?

I think we've seen like from the AV industry, for example, right? The physical World is a lot more complex. There are just, there are a lot of variations and there are real implications of things not working out right. So you know, while we have the key technology pieces, getting it to that mass deployment will require, especially because these are physical things that have to exist around and be doing stuff. I think there is a lot to be discovered on actually what happens when you put them in real world, that aspect. The second is the data. In order to realize this large language models based and really generative AI based robots, we need a lot of data. A lot of work is happening right now on how can we bring that data. Again we are seeing people creating a big farm of teleoperated robots so they can just keep them doing things and collect that data. There is work happening on learning from imitation. How are we doing those things and translating that? I think there's a lot of work that needs to happen on the data side. A lot of work that has to happen on real physical implementation. The gaps that we will see when we put it on the physical system and then, yeah, I think those are the two big things that we still need to solve. But I think as you said, all the key building blocks are there to get us to that promised land.

Yeah. And I think in his keynote, Incense Jensen has talked about even this transition from Hopper to Blackwell and what that means being able to. I think the example that he gave is training, you know, 1.8 trillion parameter GPT4 model in, you know, I think a fourth of the time or something like that. What does that mean though for other industries such as robotics? Right. Like when we look over the course of the coming months or quarters or year when this, you know, these, these GPU chips and the Jetson, you know, what does that mean for, for your industry and what you maybe were not capable to do before just because of computer.

What does that open up?

What do these announcements open up?

It's just a question of iterations. The way to get to perfection for these robots is to iterate quickly. And if you had to wait for six to 12 months to train a policy, if you can do that in six weeks, that just means you can iterate faster. And that's exactly what we are starting to see now with some of these early partners who are adopting these technologies, the pace has picked up. Right? Two years has come down to two months and over time it'll come down to two weeks. So you can iterate faster, faster, faster. And then that's the big impact this technology has. And second thing, it'll do is also it'll bring down the cost in order to really deploy these, build these capabilities. That's scale. You need to do it in a way that is also, you know, cost effective. Right. So if it's taking you two years, there is no way it can be cost effective. Right. But if it's happening in two months and two days, it's going to be cost effective. So, so the scale will come and it'll allow people to do iterate fast. It'll allow people to scale their, you know, learning pieces and, and third piece is it'll help to bring the cost down to something that is more commercially viable in the long term.

Yeah. Even with that speed of development and deployment. Let's say you and I are having this exact same conversation next year at gtc. What do you hope we're talking about next year? Maybe just for Nvidia or maybe just for robotics and humanoids in general. Where do you hope we are at this point next year?

Well, I think we saw a lot of humanoid robots today that were standing and doing some stuff, but I think next year when we are here, we might be talking about why are they doing that, why are they not doing something? So I don't know. I think, you know, I don't want to be forecasting things here, but the pace at which innovation is happening is amazing and I'm super excited about what, what it's going to, you know, unlock for robotics and looking forward to that.

Yeah. And I, and I do want to get back to, you know, something that we talked about earlier. You know, we kind of talked about why now and why this inflection point. And one of those factors was large language models. It's, it's generative AI. You know, I think a lot of people have saw, you know, I've seen some demos recently, you know, the figure and, and the chat GPT kind of collaboration. Is that what you kind of see happening, you know, in the near future? Just, you know, everyday people having the ability, you know, like you said, you don't need necessarily a master's degree in deep learning or robotics. Is that kind of what we should be looking at as, you know, maybe being the near future? If the technology is kind of already there by, you know, being able to have a robot or a humanoid help with everyday tasks just by using natural language?

I think that's absolutely correct, Jordan. The way generative AI is actually helping this next generation robots is in multiple ways. One is of course, how you are engaging with the robot. How are you telling the tasks to do. How are you building that human robot interaction? But in the background, there is a lot of work that's happening on. Well, what does it mean for a robot when you say, hey, go pick up, find that can and put it in the trash? What does it mean in the robot language world behind the herd? I think there is a lot of research that is happening. There's a lot of core engineering that has to happen to translate that into a robot specific language. This is where we announced the Groot model, the foundation model, where, which can take text, it can take videos, demonstration. Hey, I want you to stack this thing this way. There's a lot of that work that needs to happen in the background for translating that into a robot specific language. Then also, the way generative AI is helping is creating all this human world is big and complex. How are you going to create all these environments in simulation? So generative AI is going to help there as well with the generative capabilities to create environment, create settings where you can just say, hey, I'm trying to deploy this in a kitchen with this much height, shelves, all these utensils should be there, and boom, it creates an environment for you to go play with. I think there's just a lot of things that are happening on the core technology side as well to bring that future forward.

Yeah, you know, speaking of that future forward, you know, you're in a very unique position as the director of robotics here at Nvidia and even Nvidia's place in the grand scheme of things. Right. So. So you get a little bit of taste of, you know, what your hardware partners are doing, you know, other partners you're working with in cloud and the AI side. But what's the thing that excites you? Right? Like, once we wrap up the craziness of the GTC conference, what are you going to be excited to be working on and what are you going to be paying attention to in the coming weeks and months?

I think for me, in the coming weeks and months, the exciting part is the traditional robotics, like the industrial robot arm, the collaborative robots, the AMRs, that are already there today, they have a lot of potential, right? The new things with humanoids and, you know, other embodiments will come. But we also announced two key things, right? Isaac Manipulator and Isaac Perceptor, which applies to the robots of today, the robots that are operating today. So in the coming months, that is what I'm most excited about, is how we will make these existing robots smarter. How can we make them easy for people to use. How can we help these company go from hundreds and thousands to tens of thousands of robots with AI and with the tools that we are providing? So I think that's what I'm most excited about, making those tangible things happen in the near term with the existing robots.

And so we've talked about so many things on today's show, from new announcements that you had here at Nvidia GTC to partnerships that you just unveiled to the future of robotics and humanoids. But maybe as we wrap this up, what is the one big takeaway that you want people to walk away from this conversation on? Specifically on where Nvidia is focusing and doing their work right now? What's that one big takeaway message you have for people?

I think the one takeaway would be that we are building the foundry and the foundation for the world's roboticists to come and build, realize their dreams, right? We're building all. It takes a long, long time. A lot of key technologies, as Jensen said, the soul of Nvidia, right? Graphics and AI and simulation, right? All physics. These three pieces are core building blocks of robotics and we've been investing on it for so many years. So now I think we have that solid foundation on which the new generation of robots can be built.

It's such an exciting time, not just for the industry, but how it's ultimately going to play out in the real world, in the business sector and everything else. So, Amit, thank you so much for joining the Everyday AI show. We really appreciate your time.

Thanks for having me. I really enjoyed this conversation.

All right. And hey, as a reminder, there is so much more. Make sure to not only check out the show notes of the live stream or the podcast, but also the newsletter. So go to your everydayai.com Sign up for the newsletter and we'll have information in there as well where you can still register for the Nvidia GTC conference. Don't worry, it's not too late. If you slept through the first two and a half days, you can go back. Some of the sessions that we were talking about here, you can go back, watch those replays, sign up. You can do it for free as well as enter into our giveaway to win a GPU as well as DLI credits. You gotta go check that out. And you gotta also continue to join us. We have a lot more here from the GTC conference. Thank you for tuning in. We hope to see you back later today and tomorrow and every day for more Everyday AI. Thanks y'.

All.

And that's a wrap for today's edition of Everyday. AI, thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.