Dr. Zsolnai reviews NVIDIA's fix for a glitch in face-generating AI where beards and textures flicker when animating. The new equivariant filter design produces smooth, photorealistic videos of faces, paintings, and animals with barely any extra computing cost.
July 24, 2021.
December 2019.
"Texture sticking."
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we will see how a small change to an already existing learning based technique can result in a huge difference in its results. This is Stylegan 2, a technique that appeared in December of 2019. It is a neural network based learning algorithm that is capable of synthesizing these eye poppingly detailed images of human beings that don't even exist. This is all synthetic. It also supports a cool feature where we can give it a photo. Then it embeds this image into a latent space. And in this space we can easily apply modifications to it. Okay, but what is this latent space thing? A latent space is a made up place where we are trying to organize data in a way that that similar things are close to each other. In our earlier work we were looking to generate hundreds of variants of a material model to populate this scene. In this latent space, we can concoct all of these really cool digital material models. A link to this work is available in the video description. Stargan uses walks in a similar latent space to create these human faces and animate them. So let's see that when we take a walk in the internal latent space of this technique, we can generate animations. Let's see how Stargan 2 does this. It is a true miracle that a computer can generate images like this. However, wait a second. Look closely. Did you notice it? Something is not right here. Don't despair. If not, it is hard to pin down what the exact problem is. But it is easy to see that there is some sort of flickering going on. So what is the issue? Well, the issue is that there are landmarks, for instance the beard, which don't really or just barely move. And essentially the face is being generated as under it. With these constraints, the authors refer to this problem as texture sticking. The AI suffers from a sticky beard. If you will imagine saying that 20 years ago to someone, you would end up in a madhouse. Now, this new paper from scientists at Nvidia promises a tiny but important architectural change. And we will see if this issue, which seems like quite a limitation, can be solved with it or not. And now hold on to your papers and let's see the new method. Holy mother of papers. Do you see what I see here? The sticky beards are a thing of the past and facial landmarks are allowed to fly about freely. And not only that, but the results are much smoother and more consistent to the point where that it can not only generate photorealistic images of virtual humans come on that is so 2020. This generates photorealistic videos of virtual humans. So I wonder, did the new technique also inherit the generality of Stargan 2? Let's see, we know that it works on real humans and now paintings and art pieces. Yes, excellent. And of course, cats and other animals as well. The small change that creates these beautiful results is what we call an equivariant filter design. Essentially, this ensures that final details move together in the inner thinking of the neural network. This is an excellent lesson on how a small and carefully designed architectural change can have a huge effect on the results. If we look under the hood, we see that the inner representation of the new method is completely different from its predecessor. You see, the features are indeed allowed to fly about. And the new method even seems to have invented a coordinate system of sorts to be able to move these things around. What an incredible idea. These learning algorithms are getting better and better with every published paper now. Good. It is only marginally more expensive to train and run than Stylegan 2. And the less good news is that training these huge neural networks still requires a great deal of computation. The silver lining is that if it has been trained once, it can be run inexpensively for as long as we wish. So images of virtual humans might soon become a thing of the past. And because from now on we can generate photorealistic videos of them. Absolutely amazing. What a time to be alive. This video has been supported by weights and biases. Check out their recent offering Fully Connected, a place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
