A new face-swapping method uses four neural networks to transfer gestures, pose, and appearance from one face to another in video. It handles occlusions like microphones and generalizes to unseen subjects better than prior work like Face2Face.
Dear fellow scholars, this is two Minute papers with. Recently, we have experienced an abundance of papers on facial reenactment in machine learning research. We talked about a technique by the name Face2Face back in 2016, approximately 300 videos ago, it was able to take a video of us and transfer our gestures to a target subject. This was kind of possible at the time with specialized depth cameras, but until Face to Face appeared and took the world by storm. As it was able to perform what you see here with a regular consumer camera, however, it only transferred gestures. So of course, scientists were quite excited about the possibility of transferring more than just that. But that would require solving so many more problems. For instance, if we wish to turn the head of the target subject, we may need to visualize regions that we haven't seen in these videos, which also requires an intuitive understanding of hair, the human face, and more. This is quite challenging. So can this be really done? Well, have a look at this amazing new paper. You see here, the left image, this is the source person. The video on the right is the target video. And our task is to transfer not just the gestures, but the pose, gestures and appearance of the face on the left to the video on the right. And this new method works like magic. Look, it not only works like magic, but pulls it off on a surprisingly large variety of cases, many of which I haven't expected at all. Now, hold on to your papers, because this technique was not trained on these subjects, which means that this is the first time it is seeing these people. It has been trained on plenty of people, but not these people. Now, before we look at this example, you are probably. Well, the occlusions from the microphone will surely throw the algorithm off, right? Well, let's have a look. Nope, no issues at all. Absolutely amazing. Love it. So how does this wizardry work exactly? Well, it requires careful coordination and between no less than four neural networks where each of which specializes for a different task. The first two is a reenactment generator that produces a first estimation of the reenacted face, and a segmentation generator network that creates this colorful image that shows which region in the image corresponds to which facial landmark. These two are then handed over to the third network, the inpainting generator, which fills the rest of the image. And. And since we have overlapping information, in comes the fourth blending generator to the rescue to combine all this information into our final image. The paper contains a detailed description of each of these networks, so make sure to have a look. And if you do, you will also find that there are plenty of comparisons against previous works. Of course, Face to Face is one of them, which was already amazing. And you can see how far we've come in only three years. Now, when we try to evaluate such a research work, we are curious as to how these individual puzzle pieces, in this case the generator networks, contribute to the final results. Are all of them really needed? What if we remove some of them? Well, this is a good paper, so we can find the Answer in Table 2, where all of these components are tested in isolation. The downward and upward arrows show which measure is subject to minimization and maximization. And if we look at this column, it is quite clear that all of them indeed improve the situation and contribute to the final results. And remember all this from just one image of the source person. Insanity. Thanks for watching and for your generous support, and I'll see you next time.
