Perfect Virtual Hands - But At A Cost! 👐

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. The promise of virtual reality is indeed truly incredible. If one day it comes to fruition, doctors could be trained to perform surgery in a virtual environment. We could train better pilots with better flight simulators, expose astronauts to virtual zero gravity simulations, you name it. This previous work uses a learning based algorithm to to teach a head mounted camera to tell the orientation of our hands at all times. Ok, so what can we do with this? A great deal. For instance, we can type on a virtual keyboard or implement all kinds of virtual user interfaces that we can interact with. We can also organize imaginary boxes. And of course we can't leave out the two minute paper's favorite going into a physics simulation and playing it with our own hands. But of course not everything is perfect here. However, look, hand hand interactions don't work so well. So folks who prefer virtual reality applications that include washing our hands should look elsewhere. And in this series we often say one more paper down the line and it will be significantly better. So now here's the moment of truth. Let's see that one more paper down the line. Lets go in guns blazing and give it examples with challenging hand hand interactions, deformations, lots of self contact and self occlusion. Take a look at this footage. This seems like a nightmare for any hand reconstruction algorithm. Who the heck can solve this? And look, interestingly they also recorded the hand model with gloves as well. How curious. And now hold on to your papers because these are not gloves. No, no, no. What you see here is the reconstruction of the hand model by a new algorithm. Look, it can deal with all of these rapid hand motions. And what's more, it also works on this challenging hand massage scene. Look at all those beautiful details. It not only fits like a glove here too, but I see creases, folds and deformations too. This reconstruction is truly out of this world. To be able to do this, the algorithm has to output triangle meshes that typically contain over 100,000 faces. Please remember this as we will talk about it later. And now let's see how it does all this magic. Because there's plenty of magic under the hood. Let's look at five ingredients that are paramount to getting an output of this quality. Ingredient number one is the physics term. Without it, we can't even dream of tracking self occlusion and contact properly. Two, since there are plenty of deformations going on in the input footage, the deformation term accounts for that. It makes a huge difference in the reconstruction of the thumb here. And if you think wow, that is horrific. Then you'll need to hold on to your papers for the next 1, which is 3, the geometric consistency term. This one is not for the faint of the heart. You have been warned. Are you ready? Let's go. Yikes. A piece of advice. If you decide to implement this technique, make sure to include the geometric consistency term so no one has to see this footage ever again. Thank you. With the worst already behind us, let's proceed to ingredient number four, the photo consistency term. This ensures that fingernail tips don't end up sliding into the finger. And five, the collision term fixes problems like this to make sure that the fingers don't penetrate each other. And this is an excellent paper. So, in the evaluation section, these terms are also tested in isolation. And the authors tell us exactly how much each of these ingredients contribute to the solution. Now, these five ingredients are not cheap in terms of computation time. And remember, we also mentioned that many of these meshes have several hundred thousand faces. This means that this technique takes a very long time to compute all this. It is not real time. Not even close. For instance, reconstructing the mesh for the hand massage scene takes more than 10 minutes per frame. This means hours or even days of computation to accomplish this. Now, the question naturally, is that a problem? No, not in the slightest. This is a zero to one paper, which means that it takes a problem that was previously impossible and now it makes it possible. That is absolutely amazing. And as always, research is a process and this is an important stepping stone in this process. I bet that two more good papers down the line and we will be getting these gloves interactively. I am so happy about this solution, as it could finally give us new ways to interact with each other in virtual spaces, add more realism to digital characters, help us better understand human human interactions, and it may also enable new applications in physical rehabilitation. And these reconstructions indeed fit these tasks like a glove. What a time to be alive. Perceptilabs is a visual API for TensorFlow, carefully designed to make machine learning as intuitive as possible. This gives you a faster way to build out models with more transparency into how your model is architected, how it performs and how. And how to debug it. Look, it lets you toggle between the visual modeler and the code editor. It even generates visualizations for all the model variables and gives you recommendations both during modeling and training. And does all this automatically. I only wish I had a tool like this when I was working on my neural networks during my PhD years. Visit perceptilabs.com papers to easily install the free local version of their system today. Our thanks to Perceptilabs for their support and for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time.