NCsoft trained an AI to generate realistic character animations from motion-capture data in minutes, making it responsive enough for real-time games. The system learns from unorganized footage and works for stylized or non-human characters like quadrupeds.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a bunch of completely unorganized human motion data, give it to an AI, grab a controller, and get an amazing video game out of it, which we can play almost immediately. Why almost immediately? I'll tell you in a moment. So how does this process work? There are similar previous techniques that took a big soup of motion capture data and outbid each other on what they could learn from it. And they did it really well. For instance, One of these AIs was able to not only learn from these movements, but even improve them and even better adapt them to different kinds of terrains. This other work used a small training set of general movements to reinvent a popular high jump technique, the Fosbury Flop. By itself, this allows the athlete to jump backward over the bar, the thus lowering their center of gravity. And it could also do it on Mars. So cool. In the meantime, Ubisoft also learned to not only simulate, but even predict the future motion of video game characters, thus speeding up this process. You can see here how well its predictions line up with the real reference footage. And it could also stand its ground when previous methods failed. Ouch. So, are we done here? Is there nothing else to do? Well, of course there is. Here. In goes this soup of unorganized motion data, which is given to train a deep neural network. And then this happens. Yes, the AI learned how to weave together these motions so, so well that we can even grab a controller and start playing immediately. Or almost immediately. And with that, here comes the problem. Do you see it? There is still a bit of a delay between the button press and the motion. A simple way of alleviating that would be to speed up the underlying animations. Well, that's not going to be it, because. Because this way the motions lose their realism. Not good. So, can we do something about this? Well, hold on to your papers and have a look at how this new method addresses it. We press the button and. Yes, now that is what I call quick and fluid motion. Yes, this new AI promises to learn time critical responses to our commands. What does that mean? Well, see the white bar here? By this amount of time, the motions have to finish. And the blue is where we are currently. So the time critical part means that it promises that the blue bar will never exceed the white bar. This is wonderful. Just ask the gamer. Has it happened to you that you already pressed the button, but the action didn't execute in time? They will say that it happens all the time, but with this we can perform a series of slow U turns and then progressively decrease the amount of time that we give to the character and see how much more agile it becomes. Absolutely amazing. The motions really changed, and I find all of them quite realistic. Maybe this could be connected to a game mechanic where the super quick time critical actions deplete the stamina of our character quicker, so we have to use them sparingly. But that's not all it can do. Not even close. You can even chain many of these crazy actions together. And as you see, our character does these amazing motions that look like they came straight out of the Witcher series. Loving it. What you see here is a teacher network that learns to efficiently pull off these moves. And then we fire up a student neural network that seeks to become as proficient as the teacher, but with a smaller and more compact neural network. This is what we call policy distillation. So I hear you asking, is the student as good as its teacher? Let's have one more look at the teacher and the student. They are very close. Actually. Wait a second. Did you see it? The student is actually even more responsive than his teacher was. This example showcases it more clearly. It can even complete the slalom course. And we might even be able to make a parkour game with it. And here comes the best part. The required training data is not a few days, not even a few hours, only a few minutes. And the training time is in the order of just a few hours. And. And this we only have to do once. And then we are free to use the trained neural network for as long as we please. Now, one more really cool tidbit in this work is that this training data mustn't be realistic. It can be a soup of highly stylized motions, and the new technique can still weave them together really well. Is it possible that. Yes. My goodness. It. It is possible. The input motions don't even have to come from humans. They can come from quadrupeds too. This took only five minutes of motion capture data, and the horse AI became able to transition between these movement types. This is yet another amazing tool in democratizing character animation. Absolutely amazing. What a time to be alive. Weights and Biases provides tools to track your experiments in your deep learning projects. What you see here is their tables feature. And the best part about it is that it is not only able to handle pretty much any kind of data you can throw at it, but it also presents your experiments to you in a way that is easy to understand. It is used by many prestigious labs, including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals and academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
