Dr. Zsolnai covers Microsoft research that uses synthetic virtual humans to train facial-landmark detectors. The new method detects roughly 700 landmarks per face, runs at 150 FPS on CPU alone, and handles occlusions better than before. It works from color video only, matching the quality of Apple's ARKit.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see Microsoft's AI looking at a lot of people who don't exist. And then we will see that these virtual people can teach it something about real people. Now, through the power of computer graphics algorithms, we are able to create virtual worlds. And of course, within these virtual worlds, virtual humans too. So here is a wacky idea. If we have all this virtual data, why not use these instead of real photos to train a new AI to do useful things with them? Hmm. Wait a second. Maybe this idea is not so wacky after all. Especially because we can generate as many of these virtual humans as we wish. And all this data is perfectly annotated. The location and shape of the eyebrows is known even when they are occluded. And we know the depth and geometry of every single hair strand of the beard. If done well, there will be no issues about the identity of the subjects or the distribution of the data. Also, we are not limited by our wardrobe or the environments we have access to. In this virtual world, we can do anything we wish. So good. But of course, here is the ultimate question that decides the fate of this project. And that question is, does this work? And what is all this good for? And the crazy thing is that Microsoft's previous AI technique could now identify facial landmarks of real people, but it has never seen a real person before. How cool is that? But this is a previous work and now a new paper has emerged. And in this one, scientists at Microsoft said, how about more than 10 times more landmarks? Yes, this new paper promises no less than 700. When I saw this, I thought, are you kidding? Are we going 10x just one more paper down the line? Well, I will believe it when I see it. Let's see a different previous technique from just two years ago. You see that we have temporal consistency issues. In other words, there is plenty of flickering going on here. And there is one more problem. These facial expressions are giving it a hard time. Can we really expect any improvement over these two years? Well, hold on to your papers and let's have a look at the new method and see for ourselves. Look at that. It not only tracks a ton more landmarks, but the consistency of the results has improved a ton as well. So it both solves a harder problem and it also does it better than the previous technique. Wow. And all this just one more paper down the line. My goodness, I love it. I feel like this new method is the first that could even track Jim Carrey himself. And we are not done yet. Not even close. It gets even better. I was wondering if it still works in the presence of occlusions. For instance, whenever the face is covered by hair or clothing or a flower. And let's see, it still works amazingly well. What about the colors? That is the other really cool thing. It can for instance, tell us how confident it is in these predictions. Red means that the AI has to do more guesswork, often because of these occlusions. My other favorite thing is that this is still trained with synthetic data. In fact, it is key to its success. This is one of those success stories where training an AI in a simulated world can be brought into the real world and it still works spectacularly. There is a lot of factors at play here. So let's send a huge thank you to computer graphics researchers as well for making this happen. These virtual characters could not be rendered and animated in real time without decades of incredible graphics research works. Thank you. And now comes the ultimate how much do we have to wait for these results? This is incredibly important. Why? Well, here is a previous technique that was amazing at tracking our hand movements. Do you see these gloves? Yes. Those are not gloves. And this is how a previous method understands our hand motions. Which is to say it can reconstruct them nearly perfectly. Stunning work. However, these are typically used in virtual worlds and we had to wait for nearly an hour for such a reconstruction to happen. Do we have the same situation here? You know, 10 times better results in facial landmark detection. So what is the price that we have to pay for this one hour of waiting? Again? Well, not at all. If you have been holding on to your paper so far. Now squeeze that paper. Because it is not only real time, it is more than twice as fast as real time. It can churn out 150 frames per second and it doesn't even require your graphics card. It runs on your processor. That is incredible. Here is one more comparison against the competitors. For instance, Apple's AR kit runs on their own iPhones. They can make use of the additional depth information that is a goldmine of information. But this new technique doesn't. It just takes color data. That is so much harder. But in return it will run on any phone. Can these results compete with Apple's solution with less data? Lets have a look. My goodness, I love it. The results seem at the very least comparably good. That is once again amazing progress in just one paper. Also what I am really excited about is that variants of this technique may also be able to improve the fidelity of these deepfakes out there. For instance, here is an example of me becoming a bunch of characters from from Game of Thrones. This previous work was incredible because it could even track where I was looking. Imagine a new generation of these tools that is able to track even more facial landmarks and democratize creating movies, games and all kinds of virtual worlds. Yes, with some of these techniques we can even become a painting or a virtual character as well. And even the movement of our nostrils would be transferred. What a time to be alive. So does this get your mind going? What would you use this for? Let me know in the comments below. This episode has been supported by Runway Professional and Magical AI Video Editing for Everyone. I often hear you fellow scholars asking, okay, these AI techniques look great, but when do I get to use them? And the answer is right now. Runway is an amazing video editor that can do many of the things that you see here in this series. For instance, it can automatically replace the background behind a person, it can do inpainting for videos amazingly well. It can keep track of objects as they move, and it also has bead detection, automatic subtitles, noise removal, you name it. No wonder it is used by editors, post production teams and creators at companies like cbs, Google Vox and many others. Make sure to go to runwayml.com papers. Sign up and try it for free today. Thanks for watching and for your generous support and I'll see you next time.
