AI Learns Facial Animation in VR

Dear fellow scholars, this is two minute papers with. One of the main promises of virtual reality VR, in short, is enhancing the quality of our remote interactions. With VR, we could talk with our colleagues and beloved ones through telepresence applications that create a virtual avatar of us, much like the ones you see here. Normally, this requires putting sensors all over our faces to be able to reconstruct the gestures we make. A previous work used a depth camera that was hanging off of the VR headset, thus having a better look at the entirety of our face. While a later work used a mouth camera to solve this problem. This new paper attempts to capture all of our gestures by using a headset without these additional complexities. By using no more than three infrared cameras, no extra devices hanging off of the headset, nothing. All of them are built into the headpiece. This means two key challenges. One is the fact that the sensor below sees the face in an uncomfortable oblique angle. Below you see exactly the data that is being captured by the three sensors. And two, the output of this process should be a virtual avatar. But it is unclear what the correspondence between all this data and the animated character should be. So the idea sounds great. The only problem is that this is near impossible. So how did the researchers end up doing this? Well, what they did is they built a prototype headset with six additional sensors. Now, wearing this headset would perhaps not be too much more convenient than the previous works we've looked at a moment ago. But don't judge this work just yet because this additional information is required to create the output avatar. And then the smaller three sensor headset can be trained by dropping these additional views. In short, the augmented, more complex camera is used as a crutch to train the smaller headset. Amazing idea. I love it. Our more experienced fellow scholars also know that there is a little style transfer magic being done here. And finally, all of these partial views are then stitched together into the final avatar. You can also see here that it smokes. The competition uses only three sensors and does all this in real time. Wow. If you want to show your friends how you are about to sneeze in the highest possible quality video footage, look no further. Now, I'm a research scientist by day and I also run my own projects where I cannot choose my own hosting provider. And every time I have problems with it, I tell my wife that I wish we could use Linode. Linode is the world's largest independent cloud hosting and computing provider and they just introduced a GPU server pilot program. These GPU instances are tailor made for AI, scientific computing and computer graphics projects. Yes, exactly the kind of works you see here in this series. If you feel inspired by these works and you wish to run your own experiments, or deploy your already existing works through a simple and reliable hosting service, make sure to join over 800,000 other happy customers and choose Linode. Note that this is a pilot program with limited availability. To reserve your GPU instance at a discounted rate, make sure to visit linode.com papers or click the link in the description and use the promo code PAPERS20 to get $20 free on your account. You also get super fast storage and proper support. If you have any questions, give it a try today. Our thanks to Linode for supporting the series and helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.