Researchers created an AI that reconstructs realistic finger movements for virtual hands using only wrist and object sensors, no finger tracking needed. It handles different objects and scales in near-real-time, though thin handles and pose errors still trip it up.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see how we can use our hands, but not our fingers to mingle with objects in virtual worlds. The promise of virtual reality VR is indeed truly incredible. If one day it comes to fruition, doctors could be trained to perform surgery in a virtual environment, expose astronauts to virtual zero gravity simulations, work together with telepresence applications, you name it. The dream is getting closer and closer, but something is still missing. For instance, this previous work uses a learning based algorithm to teach a head mounted camera to tell the orientation of our hands at all times. One more paper down the line. This technique appeared that can deal with examples with challenging hand to hand interactions, deformations, lots of self contact and self occlusion. This was absolutely amazing. Because these are not gloves. No, no. This is the reconstruction of the hand by the algorithm. Absolutely amazing. However, it is slow and mingling with other objects is still quite limited. So what is missing? What is left to be done here? Let's have a look at today's paper and find out together. This is its output. Yes, mingling. That looks very natural. But what is so interesting here? The interesting part is that it has realistic finger movements. Well, that means that it just reads the data from the sensors on the fingers. Right? Now hold on to your papers and we'll find out once we look at the input. Oh my. Is this really true? No sensors on the fingers anywhere. What kind of black magic is this? And with that, we can now make the most important observation in the paper. And that is that it reads information from only the wrist and the objects in the hand. Look, the sensors are on the gloves, but none are on the fingers. Once again, the sensors have no idea what we are doing with our fingers. It only reads the movement of our wrist and the object. And all the finger movement is synthesized by it automatically. Whoa. And with this, we can not only have a virtual version of our hand, but but we can also manipulate virtual objects with very few sensor readings. The rest is up to the AI to synthesize. This means that we can have a drink with a friend online, use a virtual hammer to, depending on our mood, fix or destroy virtual objects. This is very challenging because the finger movements have to follow the geometry of the object. Look here the same hand is holding different objects and the AI knows how to synthesize the appropriate finger movements for both of them. This is especially apparent when we change the scale of the object. You see, the small one requires small and precise finger movements to turn around. These are motions that need to be completely resynthesized for bigger objects. So cool. And now comes the key. So does this only work on objects that it has been trained on? No, not at all. For instance, the method has not seen this kind of teapot before and still it knows how to use its handle and to hold it from the bottom too, even if both of these parts look different. Be careful though. Who knows, maybe virtual teapots can get hot too. What's more, it handles the independent movement of the left and right hands. Now, how fast is all this? Can we have coffee together in virtual reality? Yes, absolutely. All this runs in close to real time. There is a tiny bit of delay though, but a result like this is already amazing. And this is typically the kind of thing that can be fixed one more paper down the line. However, not even this technique is perfect. It might still miss small features on an object. For instance, a very thin handle might confuse it. Or if it has an inaccurate reading of the hand pose and distances. This might happen, but for now, having a virtual coffee together. Yes, please sign me up. This episode has been supported by lambda gpu cloud it if you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus, they are the only cloud service with 48GB RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
