Dr. Zsolnai explains a learning-based method for tracking hands in VR with minimal latency and jitter. The system handles occlusion and poor lighting, enabling virtual UIs and physics interactions, though hand-to-hand contact still trips it up.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. The promise of virtual reality is indeed truly incredible. If one day it comes to fruition, doctors could be trained to perform surgery in a virtual environment. We could train better pilots with better flight simulators, expose astronauts to virtual zero gravity simulations, you name it. An important part of doing many of these is simulating walking in a virtual environment. You see, we can be located in a small room, put on a VR headset and enter a wonderful expansive virtual world. However, as we start walking, we immediately experience a big problem. What is that problem? Well, we bump into things. As a remedy, we could make our virtual world smaller, but that would defeat the purpose. This earlier technique addresses this walking problem spectacularly by redirection. So what is this redirection thing exactly? Redirection is a simple concept that changes our movement in the virtual world. So it deviates from our real path in the room in a way that both lets us explore the virtual world and not bump into walls and objects in reality in the meantime. Here you can see how the blue and orange lines deviate, which means that the algorithm is at work. With this, we can wander about in a huge and majestic virtual landscape and or a cramped bar, even when being confined to a small physical room. Loving the idea, but there is more to interacting with virtual worlds than walking. For instance, look at this tech demo that requires more precise hand movements. How do we perform these? Well, the key is controllers. Clearly they work, but can we get rid of them? Can we just opt for a more natural solution and and use our hands instead? Well, hold on to your papers because this new work uses a learning based algorithm to teach a head mounted camera to tell the orientation of our hands at all times. Of course, the quality of the execution matters a great deal. So we have to ensure at least three things. One is that the hand tracking happens with minimal latency, which means that we see our actions immediately with minimal delay. Two, we need low jitter, which means that the key points of the reconstructed hand should not change too much from frame to frame. This happens a great deal with previous methods. And what about the new one? Oh yes, much smoother. Checkmark. Note that the new method also remembers the history of the hand movement and therefore can deal with difficult occlusion situations. For instance, look at the pinky here. A previous technique would not know what's going on with it, but this new one knows exactly what is going on because it has information on what the hand was doing a moment ago. And three, this needs to work in all kinds of lighting conditions. Let's see if it can reconstruct a range of mythical creatures in poor lighting conditions. Yes, these ducks are reconstructed just as well as the mighty Poke monster and these scissors too. Bravo. So what can we do with this? A great deal. For instance, we can type on a virtual keyboard or implement all kinds of virtual user interfaces that we can interact with. We can also organize imaginary boxes. And of course we can't leave out the two minute paper's favorite going into a physics simulation and playing with it. But of course not everything is perfect here. Look, hand to hand interactions don't work so well, so folks who prefer virtual reality applications that include washing our hands should look elsewhere. But of course, one step at a time. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus, they are the only cloud service with 48GB RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances with workstations or servers. Make sure to go to lambdalabs.com papers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
