Dr. Károly Zsolnai-Fehér explains how six sensors on your body can track your full-body movements and transfer them onto a virtual character in real time. No cameras needed, works in the dark, and runs at 90 frames per second. Two players can face off in virtual table tennis with only minor delay and jitter.
Questions this video answers
Six inertial measurement units.
90 frames per second.
December 31, 2021.
Dr. Károly Zsolnai-Fehér.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see how easily we can transfer our motion onto a virtual character and even play virtual table tennis as if we were a character in a computer game. This research work proposes to not use the industry standard motion capture sensors to do this. Instead they promise a neural network that can perform full body motion capture from six inertial measurement units. These are essentially gyroscopes that report accelerations and orientations. And this is the key. This presents us with two huge advantages. Hold on to your papers for advantage number one, which no cameras are needed. Yes, that's right. Why is that great news? Well, a camera is a vision based system and if people are far away from the camera, it might have some trouble making out what they are doing. Of course because it can barely see them. But not with the inertial measurement units and this neural network, they can also be further away, maybe even a room or two away and no problem. So good. And if it doesn't even need to see us, we can hide behind different objects. And look, it can still reconstruct where we are loving it. Or we can get two people to play table tennis. We can only see the back of this player and the occlusion situation is getting even worse as they turn and jump around a great deal. Now you made me curious, lets look at the reconstruction together. Wow, that is an amazing reconstruction. In the end, if the players agree that it was a good match, they can hug it out. Or wait a second, maybe not so much. In any case, the system still works. And advantage number two. Ah, of course, since it thinks in terms of orientation, it still doesn't need to see you. So, so. Oh yes, it also works in the dark. Note that so do some infrared camera based motion capture systems, but here no cameras are needed. And let's see that reconstruction. There is a little jitter in the movement, but otherwise very cool. And if you have been holding onto your paper so far, now squeeze that paper because I am going to tell you how quickly this runs. And that is 90 frames per second, easily in real time. Now of course not even this technique is perfect. I am sure you noticed that there is a bit of a delay in the movements and often some jitter too. And as always, do not think of this paper as the final destination. Think of this as an amazing leap forward and always, always apply the first law of papers. What is that? Well, just imagine how much this could improve. Just two more papers down the line. I am sure there will be no delays and no jitter. So from now on we are one step closer to be able to play virtual table tennis or even work out with our friends in a virtual world. What a time to be alive. This video has been supported by Weights and Biases. Check out their recent offering Fully Connected, a place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
