This YouTube explainer (Two Minute Papers by Dr. Zsolnai) reviews a collaboration between NVIDIA, the University of Toronto, and the Vector Institute on improved human pose estimation. The video explains common failures (temporal flicker and foot sliding) and describes a pipeline that runs pose estimates, retargets them to a game character, and corrects motion using a physics simulation to produce physically-correct, temporally-consistent motion that can be retargeted to textured virtual characters — enabling easier capture of human movement for VR/Metaverse applications. The clip includes dynamic demonstrations (sprinting, tennis serve) and ends with a sponsor message from Weights & Biases.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see how Nvidia's crazy new AI is able to understand basically any human movement. So how is this even possible? And even more importantly, what is pose estimation? Well, simple. A video of people goes in and the posture they are taking comes out. Now you see here that previous techniques can already do this quite well. What's more, if we allow an AI to read the WI fi signals bouncing around in the room, it can perform pose estimation even through walls. Kind of. Now you may be wondering, what is pose estimation good for? By the end of this video, you will see that this can help us move around in virtual worlds. The Metaverse, if you will. But let's not rush there yet, because still there are several key challenges here. One, we need to be super accurate to even put a dent into this problem. Why? Well, because if we have a video and we are off by just a tiny bit from frame to frame, this kind of flickering may happen. That's a challenge. 2. Foot sliding. Yes, you heard it right. Previous methods suffer from this phenomenon. You can see it in action here, and also here too. So why does this happen? It happens because the technique has no knowledge of the physics of real human movements. So scientists at Nvidia, the University of Toronto and the Vector Institute fired up a collaboration. And when I first heard about their concept, I thought, you're doing what? But check this out. First, they perform a regular pose estimation. Of course, this is no good, as it has the dreaded temporal inconsistency. Or in other words, flickering. And in other cases, often foot sliding too. Now hold on to your papers, because here comes the magic. They transfer the motion to a video game character and embed that character in a physics simulation. In this virtual world, the motion can be corrected to make sure they are physically correct. Now remember, foot sliding happens because of the lack of knowledge in physics. So perhaps this idea is not that crazy after all. Let's have a look. Now, this will be quite a challenge. Explosive sprinting motions and. And. Whoa. This is amazing. This, dear fellow scholars, is superb pose estimation and tracking. And how about this? A good tennis serve includes lots of dynamic motion. And just look at how beautifully it reconstructs this move. Apparently, physics works. Now, the output needn't be Stickman. We can retarget these to proper textured virtual characters built from a triangle mesh. And that is just one step away from us being able to appear in the Metaverse. No head mounted displays are required, no expensive studio and no motion capture equipment is required. So what is required? Actually nothing, just the raw input video of us. That is insanity. And all this produces physically correct motion. So that crazy idea about taking people and transforming them into video game characters is not so crazy after all. So now we are one step closer to be able to work and even have some coffee together in a virtual world. What a time to be alive. This video has been supported by weights and biases. Being a machine learning researcher means doing tons of experiments and of course creating tons of data. But I am not looking for data, I am looking for insights. And weights and biases helps with exactly that. They have tools for experiment tracking, dataset and model versioning and even hyperparameter optimization. No wonder this is the experiment tracking tool choice of OpenAI, Toyota Research, Samsung and many more prestigious labs. Make sure to use the link wnb me paperintro or just click the link in the video description and try this 10 minute example of weights and biases today to experience the wonderful feeling of training a neural network and being in control of your experiments. After you try it, you won't want to go back. Thanks for watching and for your generous support and I'll see you next time.
