DeepMind trained robots with 20 degrees of freedom in simulation to play soccer, then transferred the skills to real hardware. After self-play training, the robots walked 2-3 times faster than the manufacturer's baseline, got up 50% quicker, and kicked 34% faster. Simulation compressed years of training into hours.
Questions this video answers
April 29, 2024
20 degrees of freedom
Seven skills (listed in transcript)
Walks and turns 2–3× faster; takes close to half as much time to get up; kicks a ball 34% faster.
A proximity-based penalty: if you get too close to the other robot, you get a penalty.
Chapters
Fellow scholars, I think you're going to love this. This is a follow up to an amazing project where AI agents learn to play football. Soccer. Initially they start out like this. Well, learning is a strong word to use here, but now let's jump five years in time and look, they are now competent players. But wait a second, was this really five years of training? Yes, it was. However, not in our time, but in their time. If you have a powerful computer, you can simulate these little AIs much, much faster than real time. And now we are going to look at a similar really exciting Sim 2 reel project, which means that first the agents learn to play in a simulation in in a computer game, if you will. And then they come out into the real world as real robots to shoot a real ball. We talked a bit about football robots before, but these new results are truly something else. However, I am a little worried. Now I hear you asking, why are you worried? Well, look at this. In this earlier software project, there was no referee and therefore there was no penalty for basically completely destroying each other. And look at that. Absolute pandemonium ensued. So now I am a little worried that these little robot scholars are also going to beat up each other really bad. So do they. I will tell you in a moment why this is not the case. Dear fellow scholars, this is two minute papers with Dr. Caro Zsolnai. So first they learn in a video game world. And I already see that this is going to be quite tough. Look, 20 degrees of freedom. That means 20 controllable joints. That gives us at least 20 things that can go wrong. This is not going to be glorious, not in the slightest. That means that we have to start out from the humble beginnings. It means that controlling the arms and the limbs of this robot will not be easy. So much so that they first need to learn to stand, walk, and of course get up after falling. Then basic training in football. Wow, this is not looking great. I wonder if enough learning can happen here so we get to see anything interesting? Hmm, not sure. Then soccer training against an opponent. This is not too impressive at this point. And it will probably get even worse upon uploading this AI into a real robot. Especially that these poor little robots barely see anything. And when they start running, oh my goodness, it gets even worse. So now we see that this is a super difficult task. So let's see if it can be done at all. First, a penalty kick is simulated in a video game world. And now hold on to your papers, fellow scholars. And yes, that was a good kick. Finally, learning is happening. But this is when things are going well. So what if things are not going well at all? Well, after a bit more learning, look how robust it became against perturbations. What does that mean? Well, this is a fancy way of saying that there was lots of fun to be had in the lab that day and it can recover from all this. Wait. Not only that, but it can even get up and score. Bravo. But it gets worse. I know that for a fact. Why do I know that for a fact? Because I had the huge honor of recording this piece of footage myself in person and at the Google DeepMind lab. Look at that. Ouch. I did not expect that at all. And it really shows how these robots can fail in the most spectacular manner. Wow. And a big thank you for Google DeepMind for the trip. So, are they destroying each other? Not quite. Hmm. Why not? Because even though we don't have a referee in person, they were told that the rule is that if you get too close to the other robot, you get a penalty. Phew. So we can expect peaceful matches after this insanity from a previous work. And then something incredible happened. After playing more and more against themselves, they learned new things. And now they have seven absolutely incredible new skills. One, they were also advised to avoid high knee torques during the video game phase. Essentially, they were taught to take it easy or otherwise that knee is not going to make it. I love that. As soon as they are locked into a somewhat human like body, they now have knee pain of sorts. Welcome to the real world, little AI. And over time, yes, they learned to walk and run a bit softer. So good. Two, they can now kick even a moving ball. Three, they learn to block the other robot's shot with their bodies. That is amazing. Four, they can now turn much better and look. Five, they can also do something that I did not expect at all and that is getting up from the back. So cool. Six, they can also anticipate what is about to happen and position themselves accordingly. And seven, their ball control skills are also a thing of beauty. Bravo. And now here comes the absolute crown jewel result. The manufacturer of this robot has little handcrafted scripts programs that control the robot for these movements. This is what it looks like. These are carefully crafted by engineers who know these models really well. And now let's compare it to their learned behavior in this project. Oh my goodness, this is so much better. And they learned all this by themselves. The AI agent now walks and turns two to three times faster. Takes only close to half as much time to get up and get this fellow scholars. It even kicks a ball 34% faster than then this manufacturer baseline. Holy mother of papers. This is an absolutely incredible paper and I feel honored to witness it together with you. Wow. And note that this training could not have been done entirely in the real world, but first only in a video game world because it would have taken too long and the robots could also have hurt themselves in the process. But after learning in a video game, we get this absolute miracle. And remember, in the game they can put in years and years of training in just a few hours. What a time to be alive. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.
