A new AI method transfers your gestures from a video onto any image, painting, or sculpture in real time. It works from a single reference image and runs on consumer GPUs. The technique struggles with large pose changes and shoulder movement.
This AI paper feels like an absolute revolution. Where? Wait, what is going on here? What happened to these paintings? And also fantastic news. You can try this technique right now. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Yes. With this new work, you can be the person whose gestures are transferred to these characters. So, so how does this insanity work? Well, simple. You just add one image, not even a collection of images of the target face, and a video of yourself, and then bam. That's it. You can become that character. Super cool. And this is super challenging. You see, we humans know that this is a human eye moving, but the target can truly be a photo or a painting or a sculpture, or anything you can think of. And these gestures have to be transferred onto the target. Each human face has unique morphology, unique proportions. And to think that an AI based technique is able to understand all this and create near perfect solutions. I am completely stunned. Wow. And even in the quantitative tests, it is so much better than many of the previous works out there. Goodness. And these are not some ancient techniques? No, no. For instance, any portrait and the face adapter papers are both from this year. Yes. Not even a year later and we get this kind of quality. And believe it or not, you have seen nothing yet. Yes, the results are of higher quality than previous methods. That is already amazing. But look. Oh my. It also promises controllability. Really? Controllability, while retaining this kind of quality. That cannot possibly be true. Or can it? Well, let's have a look together. We can ask this character to open its eyes and. And wow. I don't think I have seen a single frame in this animation that doesn't belong there. And there is even a slight bit of eyebrow movement as well. Very subtle, but very cool. The case with the mouth opening I found to be a little more unnatural. But I feel that this is something that is going to become near perfect. Just one more paper down the line. And now hold on to your papers, fellow scholars, because we are not done until we have tried this. Oh, animals work too. How cool is that? It understands which part of our face is which, and it is able to not only map it to animals, but results in the paper. And that is exactly the case here. So many kinds of faces, different proportions, art styles. It seems like this technique can do absolutely everything. But you don't have to believe it. You can try it right now, assuming we don't crash the website, which we usually do when so many of you curious fellow scholars click these links and the link is available below. And get this, if you would run it locally on a consumer graphics card. It runs faster than real time. At this point I am out of words. Now wait, one more thing and just imagine what we could do with a technique like this where the driving sequence is us but with a significantly lower resolution video and the target photo is also us. So we could reenact ourselves. Of course that doesn't make any sense. Or wait a minute. With that we could do facetime video and teleconferencing with our loved ones where we would appear as ourselves. However, then we wouldn't need to transmit high resolution videos of ourselves, just a coarse video on a flaky hotel connection and one reference image of ourselves and it will still look crystal clear. Loving it. What a time to be alive. Now not even this technique is perfect. In the presence of large pose variations and significant shoulder movements, it typically does not perform so well. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. If you're looking for an Nvidia GPU cloud service for large scale AI projects, check out One Click Clusters by Lamda, featuring up to 512 Nvidia H100 100 GPUs with InfiniBand Networking. And hold on to your papers because with One Click Clusters you can access all these GPUs on demand with reservations starting at just two weeks. Try it out@LambdaLabs.com paper or click the link in the description.
