Dr. Zsolnai demos NVIDIA's new image-editing AI that makes semantic changes like opening eyes, adding smiles, or changing hairstyles across photos, drawings, and statues. The model learns concepts from just 16 labeled examples and generalizes well, though it struggles with out-of-distribution cases.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to do some incredible experiments with Nvidia's next level image editor, AI. Now there are already many AI based techniques out there that are extremely good at creating new human or even animal faces. You see here how beautifully this alias Freegan technique can morph one result into another. It truly is a sight to behold. These are really good at editing images. You see, this was possible just a year ago, but this new one, this can do more semantic edits to our images. Let's dive in and see all the amazing things that it can do. If we take a photo of our friends we haven't seen in a while, what happens? Well, of course the eyes are closed or are barely open. And from now on, this is not a problem. Done. And if we wish to add a smile or take it away, boom. That is also possible. Also the universal classic, looking too much into the camera. Not a problem. Done. Now the AI can do even more kinds of edits. Hairstyle, eyebrows, wrinkles, you name it. However, that's not even the best part. You have seen nothing yet. Are you ready for the best part? Hold on to your papers and check this out. Yes, it even works on drawings, paintings and oh my, even statues as well. Absolutely amazing. How cool is that? I love it. Researchers refer to these as out of domain examples. The best part is that this is a proper learning based method. This means that by learning on human portraits it has obtained general knowledge. So now it doesn't just understand these as clumps of pixels, it now understands concepts. Thus it can reuse its knowledge even when facing a completely different kind of image. Just like the ones you see here. This looks like science fiction. And here it is right in front of our eyes. Wow. But it doesn't stop there. It can also perform semantic image editing. What is that? Well look, we can upload an image, look at the labels of these images and edit the labels themselves. Well, ok, but what is all this good for? Well, the AI understands how these labels correspond to the real photo. So check this out. We do the easy part, edit the labels and the AI does the hard part, which is changing the photo appropriately. Look. Yes, this is just incredible. The best part is that we are now even seeing a hint of creativity with some of these solutions. And if you are one of those folks who feel like the wheels and rims are never quite big enough, well, Nvidia has got you covered, that's for sure. And here comes the kicker. It learned to create these Labels automatically by itself. So how many label to image pairs did it have to look at to perform all this? Well, what do you think? Millions? Or maybe hundreds of thousands? Please leave a comment. I'd love to hear what you think. And the answer is 16. What? 16 million? Nope. 16. Two to the power of four. And that's it? Well, that is one of the most jaw dropping facts about this paper. This AI can learn general concepts from very few examples. We don't need to label the entirety of the Internet to have this technique be able to do its magic. That is absolutely incredible. Relearning from just 16 examples. Wow. The supplementary materials also showcase loads of results, so make sure to have a look at that. If you do, you'll find that it can even take an image of a bird and adjust their beak sizes even to extreme proportions. Very amusing. Or we can even ask them to look up. And I have to say, if no one would tell me that these are synthetic images, I might not be able to tell. Now note that some of these capabilities were present in previous techniques, but this new method does all of these with really high quality. And all this in one elegant package. What a time to be alive. Now, of course, not even this technique is perfect. There are still cases that are so far outside of the training set of the AI that the result becomes completely unusable. And this is the perfect place to invoke the first law of Papers. What is that? Well, the first law of papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. And don't forget a couple papers before this. We were lucky to do this and now see how far we've come. This is incredible progress in just one year. So what do you think? What would you use this for? I'd love to hear your thoughts, so please let me know in the comments below. What you see here is a report of this exact paper we have talked about with which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system, you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through WNB.com papers. Or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
