A short YouTube explainer (Two Minute Papers) by Dr. Zsolnai describing a recent AI/graphics paper that maps human motion to virtual characters with very different morphologies using only three-point input (VR headset + two controllers). The workflow: use motion-capture data to initialize correspondences, train a reinforcement-learning agent on the noisy animation, then discard mocap and run online from headset/controllers only; add auxiliary signals during training (then throw them away) and a head-orientation correction to avoid unnatural poses. Claimed practical efficiencies: training on a single consumer GPU for ~24 hours (one-time), using ~4 hours of human footage, and real-time inference thereafter. Limitations: some manual joint-mapping is required per new character.
Questions this video answers
Three-point input: VR headset and two controllers (head + two hands).
Approximately four hours of footage.
Training took 24 hours on one consumer graphics card (single machine).
Chapters
I am very happy today because we have an incredible paper here. Get this. It helps us transform into virtual characters and transfer our movements to them. Wow. But it gets better. We can even control virtual characters with different morphology than our bodies. We can be a tiny little mouse or a dinosaur. And wait, it gets even better. All this from a three point input, not full body. This is not motion capture where an entire body suit is used and we know the movement of every body part. No sir, what we know here is so much more challenging. Just the movement of the VR headset and two controllers, nothing else. That is really tough. And to me it sounds absolutely impossible. So, is it impossible? Well, almost. Let's see if this paper can pull it off. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. First, for a training step. It uses motion capture data to learn which human joints match the virtual character's joints for training. Unfortunately, this does not work. Great. But wait, don't despair. Not all is lost. Why is that? Well, we can now create a reinforcement learning agent which can lean on this jerky animation and then learn to control it properly here. Wow, this looks so much better. Loving it. And note that at this step we can throw away all the motion capture data and we only need the headset and the two controllers. So far, so good. So, are we done? Well, we are not done yet. Not even close. We also need a little trickery for dealing with the head orientation or otherwise we will look like this. And it gets even worse. Due to the heavy head of the mouse character, even this can happen. Ouch. But after this additional step, look much better. Additionally, to add more realism, we can even decide how we wish to control the tail of the dinosaur as well. Loving it. So, are we done now? Well, we are still not done yet. The paper also proposes enriching the training data with additional information for the AI to learn and then throwing this data away. But come on, if we don't do that, what's the worst that can happen? Well, this. This can happen. Ouch. But putting this all together. Hold on to your papers, fellow scholars. And look, it finally works. Now we have an absolutely miraculous algorithm that can perform these movements across different morphologies with different character types. It does it in an energy efficient way so that we get relatively elegant smooth movements and not those jerky ones. Excellent. But wait a second. Are you thinking what I am thinking? Remember, it has information from the headset and the controllers. Does that mean that. Yes, that's exactly what it means. It has no lower body information whatsoever. And yet it can still reconstruct all of these movements really well. Wow, that is crazy. And it gets better. Larger and smaller characters will be moved around differently. You just walk around in your own pace. But for them, depending on their size, more or maybe fewer tippy taps are required. And advanced movements are also possible. Staggering, balancing, you name it. Well, for smaller, more agile characters that is. And we can even try our dancing moves in this case. They are perhaps a little inept, but in a very cute way. And not just movement, but acting can be done. So good. Loving it. And yes, it gets even better, if that is at all possible. I have found three more mind blowing things about this paper. One, it has been trained on just one machine with one consumer graphics card and not even a high end. One, the training took 24 hours and this step only has to be done once. And then we can use it in real time for as long as we wish. Two, so how much training data is required for this? Get this, only four hours. It looks at four hours of footage from a bunch of humans and then it generalizes the movement to new people. And three, what you are about to see now is the reason two minute papers exist. Look, this paper has been seen by less than a hundred people. Such an amazing work and I am worried that if we don't talk about it here, almost no one will talk about it. So if you fellow scholars would help me spread the word of this amazing AI computer graphics paper, that would be amazing. Thank you. Now don't forget, this technique is still not completely automatic. I noted earlier that some of the joints of the human are mapped to the new character. So you can't just drop in a completely new character without some additional work. However, in return, vastly different morphologies work. I would estimate that not a great deal of work has to be done per character, but it still requires a bit of work and expertise. And I bet that just one more paper down the line and we will be able to do this automatically. Just imagine a world where we can all become virtual characters, where we can play together and the way we move is translated by an algorithm to feel like we are really any creature we want. Even if they have a completely different morphology. Big head, small head, floppy ears or a tail, anything you can imagine. And I am absolutely blown away by the fact that it may not even need to track our lower body for that. Only the three point input, head and hands. And don't forget the differences between these models are huge. There are vast variations in weight. 25x difference between the lightest and heaviest character tail joints may or may not exist, and even if they do, there is a different amount of joints. Same with the ear joints, and the amount of torque they can produce is also subject to a great deal of variation. And having a modern algorithm that can translate all of our motions to these characters is absolutely incredible. Wow. What a time to be alive. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances for just 1.99per hour. Yes, 1.99and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances. Workstations or servers. Make sure to go to lambdalabs. Com papers to sign up for one of their amazing GPU instances today.
