Researchers extract human poses from YouTube videos, then use simulation and reinforcement learning to teach AI characters to perform the same acrobatics. The system handles obstacles, works on different body types, and can complete partial motions from a single image, though complex dances like Gangnam Style still trip it up.
Dear fellow scholars, this is two minute papers with. If we have an animation movie or a computer game, where like in any other digital medium, we wish to have high quality lifelike animations for our characters, we likely have to use motion capture. Motion capture means that we put an actor in a studio and we ask this person to perform cartwheels and other motion types that we wish to transfer to our virtual characters. This works really well. But recording and cleaning all this data is a very expensive and laborious process. As we are entering the age of AI, of course, I wonder if there is a better way to do this. Just think about it. We have no shortage of videos here on YouTube about people performing cartwheels and other moves. And we have a bunch of learning algorithms that know what pose they are taking during the video. Surely we can make something happen here, right? Well, yes and no. A few methods already exist to perform this, but all of them have deal breaking drawbacks. For instance, this previous work predicts the body poses for each frame, but each of them have small individual inaccuracies that produce this annoying flickering effect. Researchers like to refer to this as the lack of temporal coherence. But this new technique is able to remedy this great result. This new work also boasts a long list of other incredible improvements. For instance, the resulting motions are also simulated in a virtual environment. And it is shown that they are quite robust. So much so that we can throw a bunch of boxes against the AI and it can still adjust to it. Kind of. These motions can be retargeted to different body shapes. You can see, as it is demonstrated here quite aptly with this neat little nod to Boston Dynamics, it can also adapt to challenging new environments. Or get this, it can even work from a single photo instead of a video by completing the motion seen within. What kind of wizardry is that? How could it possibly perform that? First we take an input photo or video and perform pose estimation on it. But this is still a per frame computation. And you remember that this doesn't give us temporal consistency. This motion reconstruction step ensures that we have smooth transitions between the poses. And now comes the best part. We start simulating a virtual environment where a digital character tries to move its body parts to perform these actions. If we do this, we can not only reproduce these motions, but also continue them. This is where the wizardry lies. If you read the paper, which you should absolutely do, you will see that it uses OpenAI's amazing proximal policy optimization algorithm to find the best motions. Absolutely amazing. So this can perform and complete a variety of motions, adapt to more challenging landscapes, and do all this in a temporally smooth manner. However, the Gangnam Style dance still proves to be too hard. The technology is not there yet. We also thank In Silicone Medicine for supporting this video. They work on AI based drug discovery and aging research. They have some unbelievable papers on these topics. Make sure to check them out and this paper as well in the video description. Thanks for watching and for your generous support and I'll see you next time.
