Károly Zsolnai-Fehér covers a new AI technique that extracts pose and facial data from videos to create realistic impersonations with control over expressions, blinking, and gaze. The method can reenact deceased actors and allows separate editing of individual facial features.
Dear fellow scholars, this is two minute papers with earlier we talked about an amazing technique where the inputs were a source video of ourselves and a target actor and the output was a reenactment. In other words, a video of this target actor with our facial gestures. This requires only a few minutes of video from the target, which is usually already available on the Internet. Essentially, we can impersonate other people, at least for one video. A key part of this new technique is that it extracts additional data such as pose and eye positions both from the source and target videos and uses this data for the reconstruction as opposed to the original face to face technique from two years ago which was already mind blowing. You see here that this results in a new learning based method that supports the reenactment of eyebrows and blinking, changing the background plus head and gaze positioning as well. So far this would still be similar to a non learning based technique we've seen a few episodes ago and now hold on to your papers. Because this algorithm enables us to not only impersonate but also control the characters in the output video. The results are truly mesmerizing. I almost fell out of the chair when I first seen them. And what's more, we can create really rich reenactments by editing the expressions pose and blinking separately by hand. What also needs to be emphasized here is that we see and talk to other human beings all the time, so we have a remarkably keen eye for these kinds of gestures. If something is off just by a few millimeters or is not animated in a way that is close to perfect, the illusion immediately falls apart. And the magical thing about these techniques is that every single iteration we get something that is way beyond the capabilities of the previous methods and they come in quick succession. There are plenty of more comparisons in the paper as well, so make sure to have a look. It also contains a great idea that opens up the possibility of creating quantitative evaluations against ground truth footage. Turns out that we can have such a thing as ground truth footage. I wonder when we will see the first movie with this kind of reenactment of an actor who passed away. Do you have some other cool applications in mind? Let me know in the comments section and if you enjoyed this episode. Make sure to pick up some cool perks on our Patreon page where you can manage your paper addiction by getting early access to these episodes and more. We also support cryptocurrencies. The addresses are available in the video description. Thanks for watching and for your generous support and I'll see you next time.
