Dr. Zsolnai explains how robots trained in simulation using reinforcement learning and curiosity can transfer those skills to the real world. The robot learns to navigate, stand up, open doors, and handle packages, solving the problem of scarce real-world training data that limits robot development compared to the abundant text data available for language models.
Here you see an incredible new robot that learned to explore, stand up and even handle packages and more. Now wait a second. What you see here should be impossible. Why is that? Why is this impossible? Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Well, to understand what is going on here, let's look into the popular large language models for first, you see, when training a large language model to understand English and be a good assistant, we can give it tons and tons of data from the Internet to learn on. For instance, the early GPT2 read Amazon reviews and learned to write new ones. This was possible because tons and tons of training data was available. However, this should be impossible for a real robot in the real world. Why is that? Well, have a look at this paper where scientists at Nvidia tried to teach a software agent to run and it kind of learned to run. Ok, but after quite a bit of training. At first it looked like this. Now, letting a robot loose in a lab while flailing around would not work too well because it would injure itself and its environment before any meaningful learning could happen. So in software there is plenty of training data to learn from, but for robotics there is no data. None. Or even if there is, there is just not enough to learn anything. So how can we solve this? Well, the solution is my favorite, which is learning inside a simulation, letting the robot play around in a video game. Here they are doing reinforcement learning, which means truly playing a video game. Here you do something and if you do well, you get a score, a reward. If you don't do so well, you get nothing or a negative score. Now we also need a little magic sauce here to make sure that the robot is curious and wants to explore and understand the world around it. Normally that is not that difficult. For instance, DeepMind's agents can play video games as they are yearning for a high score. So they will go around and explore. However, there is a huge problem. A TV problem. Yes, you heard it right. The AI has a TV problem. Agents like these can encounter a TV in a virtual world and then this happens. Yes, they get addicted to it and they never want to leave. Now that is real human like behavior if I've ever seen one. This happens because the agent is getting new information all the time and it's much more interesting than these otherwise boring levels. So how did they make this little robot curious? Well, by engineering the rewards of the game in a way that for instance, if the angle or the velocity of the door changes it, it gets a reward. Thus, look, it starts experimenting with it now new task. If we wish to get it to move boxes around to where they belong, we can craft a reward involving the velocity of the box and the distance from the container. Now hold on to your papers, fellow scholars, because this is the moment of truth. And let's jump from the video game world into the real world and, and see how this little robot fares. And whoa, look at that. It can navigate around in the real world competently and it can even stand up. It now opens those doors and can handle packages well. As you see, it handles them at least as gently as some of the delivery services out there. Another human like traitor. Perfect. Now, jokes aside, by further tailoring these reward functions, I would imagine that it could be much gentler with those packages and even doors. Just one more paper down the line and also bravo. What has been achieved here is extremely hard. Why? Well, because we cannot just make a simpler video game for this robot to learn on. It has to be good enough to even work in the real world, after all. And if the knowledge from here does not translate properly to here, things can break really quickly. This was an amazing feat. And just imagine that now we have better and better tools to create virtual worlds. A lot of our episodes are on new papers that improve these potential simulation environments. And every single one of those could be one step closer for us to have these little AIs train in these virtual worlds and then come out into the real world, help us out and do all this safely. For instance, these little robots could help us with last mile delivery. The idea behind this paper could also help us create better self driving cars that can safely train in an environment where we can make crazy hard levels for it. And with a powerful computer, have it trained for years and years in computer simulation time, not in real time, and then come out as a safe and competent AI agent. What a time to be alive. And this is not science fiction, not at all. In an earlier paper we saw Nvidia's AI train for 33 years in simulation time to play in Minecraft. And boy, did it learn a great deal in there. Now this new work, of course, does not come without limitations. Hand engineering reward functions is a bit of a limitation here, which means that we have to write these rewards ourselves. So if we need it to perform a new task, it needs a new reward function for that, limiting the generality of the agent. An earlier paper we talked about here could potentially remedy that as it is about teaching these AI agents to find out what a good score would be. Experiment tracking, model evaluation and production, monitoring for your deep learning projects and LLM apps. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
