Short YouTube commentary by Dr. Zsolnai (Two Minute Papers) summarizing a robotics research collaboration between the Universities of Texas (Austin), Pennsylvania (UPenn) and NVIDIA that uses large language models as text-based instruction plus NVIDIA’s prior "Eureka" work to wrap tasks into video-game-like simulations. The team trains robots with heavy domain randomization (changing visuals, physics, etc.) so policies transfer to real robots; demos include a robot balancing on a ball (including a deflating ball) and a full uncut 5-minute real-world demonstration. The project is open-source and the video ends with a cloud-sponsor call-to-action (Microsoft Azure AI).
Chapters
Fellow scholars, I have something truly special for you today. The improvements in humanoid robotics these days is absolutely astounding. Tesla's Optimus Learn to sort battery cells or take a walk around the office. You hear a lot about that. However, we will look at something else instead. Have you heard that research papers in this area are also improving at an incredible pace? Very few people know that, and what this can do is absolutely incredible. For instance, hold on to your papers, fellow scholars, and just look at this scientist casually taking its robot dog for a walk while it is balancing on a ball. Holy mother of papers. How could this possibly work? Well, we know exactly how it works because we have a research paper to look at and I will tell you about it. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This is a collaboration between the Universities of Texas, Austin, Pennsylvania and Nvidia. And it is meant to harness the power of large language models. ChatGPT. Like AI assistants, if you will. But wait, how does that even work? Well, first we give it some instructions, that is text. And then what happens? Eureka. Ok, but what is Eureka? It is a previous Nvidia paper that can take a task and create a video game around it, if you will, where these little robots can train safely and efficiently. And don't forget, this is a video game world where we can do anything we wish, including domain randomization. This means that we can change the colors of the environment, the levels, the laws of physics, and ask the robot to learn in a way that it can be successful in all of these environments. Then, when moving out into the real world, the hope is that this kind of video game knowledge will be useful there too. So is it? Well, we have seen how well it works in reality, so the answer is yes. Fantastic. But it gets better. We just talked about domain randomization. What does this give us? Well, it gives us superpowers. For instance, the robot will be prepared for everything. It can not only balance on the ball, but we can even kick the ball under it. And will it be able to stay on top of it? Oh yeah. Excellent. But it gets even better. Now let's try putting it on a deflating ball. And this is a huge challenge. Does it stay on the ball? Look, it does. That is incredible. We are now seeing it at work in real time. But it gets even more impressive if I show you the start of the process. Look at the movement required to succeed here. And now look at this. A completely different kind of movement is required as the ball deflates. And it works just fine on a continuously changing surface. That is one of the most impressive demos I have ever seen. But wait, is this robot really this reliable or are these cherry picked experiments where otherwise it might not work at all well? Luckily the paper also has a full uncut 5 minute demonstration and it is working remarkably well out there in the real world. Amazing. And talking about the real world, the robot is given a real body and it has to make sure that it makes movements that do not break this body. Yes, you are seeing correctly, it is trying to minimize the torque exerted on its hip to ensure its own longevity. Welcome to the real world little robot. What a time to be alive. And they gave all this knowledge for us free of charge and made the project open source too. Now I would like to ask your scholarly opinion on Tesla Optimus. I know that these developments interest a lot of you fellow scanners. However it it is not a research paper so it is harder to know the limitations of the technique that is shown off in these videos. Would you like to see a more speculative video about these when they appear? If so, these would come with a disclaimer that it is not a peer reviewed paper and it is harder for us to find the limitations of the method. But a little more discussion on it could still be very useful. Let me know in the comments below. And if you wish to run your own experiments in the cloud, make sure to check out Microsoft Azure AI. Azure AI is a powerful cloud platform that offers you the best tools for your AI projects with responsible AI built in. And here comes the best part. You can even try it out for free through the link in the video description.
