NVIDIA's new AI can generate new camera angles of a person's head from a single photo or webcam video in milliseconds, without needing to scan or calibrate anything first. It works on glasses, babies, cats, and stylized images, though it still struggles with temporal flicker and occasional artifacts like misplaced facial hair.
Questions this video answers
In a few tens of milliseconds (interactive-rate).
A single photo or commodity webcam video; no person-specific calibration required.
Chapters
With Apple releasing their Vision Pro headset, you are seeing these virtual Persona views all around us. Essentially what it does is that it scans your face and creates a video game character out of it. That is hard. But you know what's even harder? Creating virtual Personas, but without all of those cameras being attached to our faces. That sounds absolutely impossible. So is it possible? It is, but not easily. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Well, this new AI paper from scientists at Nvidia promises exactly that. The only problem is that it seems completely impossible. Consider this input image. And now these three techniques. Oh my. To say that they have trouble reconstructing it and synthesizing new views would be an understatement. This indeed looks absolutely impossible. Yes, until you look at what this paper can do. Hold on to your papers, fellow scholars, and look. Wow. It was able to look at just this one photo and synthesize this person from different angles that it has never seen before. And it does all this from so little information. And if that was not good enough, in a moment, this will get way, way better. Now let's take it out for a spin. Let's give it video input, but just the commodity camera input from the front, like a webcam. And here, here it is synthesizing the new angles and running in real time on a commodity graphics card. Just imagine having a photo or a camera on you and suddenly appearing in your favorite video game. And it even supports more difficult cases where we put on or remove headphones. If you look closely, you get a few weird frames in the meantime. And there is a bit of flickering too. We will analyze that a little more with the help of cats. Yes, cats. Glasses also work. Love it. And oh my goodness, look. Even the reflections on the glass lenses are modeled in a believable manner. Incredible. Now let's put this new AI to the real test. Doing well for one or two people, that's great. But now let's subject it to a greater variety. And ooo, it seems to work extremely well, even on babies, you name it. And note that this doesn't require any person specific calibration. It could see you for the very first time and it can pull this off. Thus you can even put a doll in there and it will work. Absolute insanity. And it gets better. So could it possibly work on stylized images? Well, it had never seen a stylized image, so not a chance. Right? Well, let's see together. Look at that. Goodness. And the most important of all, of course, does it work on cats? I would say Apart from some minor temporal coherence issues on the fur, there is a little flickering effect effect that you see as the camera turns around. Apart from that, this is excellent. Wow. And there are previous techniques that took in the order of minutes to give us some of these results that you saw here. So the new one is much better. Thus it is much slower too, right? Nope. No sir. All this runs in a few tens of milliseconds. So essentially, it is interactive. So good. So what else can we use this for? Well, of course, to appear in video games, but also video conferencing with our loved ones. But wait, I hear you asking Karoj, why do I need this avatar? I am already there, right in front of the camera. Why not just use the camera feed itself? Why would we need all this magic? Well, consider that now you don't even need to do that. What you need to send over is just one image of yourself. Just a tiny bit of data. Directions that describe how the head moves around, where the eyes look, and so on. And from this tiny little data, it can create your avatar that is getting closer and closer to the real deal. I would not be surprised if the difference in some cases would be that a hundred times less data that would be a complete game changer. For virtual work meetings or whenever we would like to talk to our beloved ones on a flaky Internet connection. Those could become way, way better. Now, not even this technique is perfect. For instance, the beard here seems to have stuck to the wrong surface. And a word about temporal coherence. You see that these are not perfect yet. Not even close. However, let's invoke the first law of Papers. The first Law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. You see, earlier, when style transfer appeared, this was a technique that could fuse two images together. And it used one for content and one for style. We could also do this for video. However, these flickering effects were so strong that they made the technique almost completely unusable. And then you know what's coming. Oh yes. Just one more paper down the line. And we got this. And just imagine what this technique could look like. Just one or two more papers down the line. What a time to be alive. Now, I am about to jump on a plane to fly to San Francisco for the first time ever. And I will speak at the Fully Connected Conference. And after that, I can't wait to meet and talk to you fellow scholars there. There are some more tickets left if you wish to come see you there. I'll be back in a few more days and the papers will be back on the regular schedule as always. And if you wish to run your own experiments in the cloud, make sure to check out Microsoft Azure AI. Azure AI is a powerful cloud platform that offers you the best tools for your AI projects with responsible AI built in. And here comes the best part. You can even try it out for free through the link in the video description.
