Dr. Zsolnai breaks down a research paper on controllable image synthesis using semantic masks to guide AI generation. You can swap scene elements like sky and mountains, mix appearances between images, and morph styles while keeping reflections and materials coherent. The technique could give artists real control over what AI generates.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Not so long ago we talked about a neural image generator that was able to dream up beautiful natural scenes. It had a killer feature where it would take as an input not only the image itself, but the label layout of this image as well. That is a goldmine of information. And including this indeed opens up a killer application. Look, we can even change the scene around by modifying the labels on this layout. For instance, by adding some mountains, make a grassy field and add the lake. Making a scene from scratch from a simple starting point was also possible with this technique. This is already a powerful learning based tool for artists to use as is. But can we go further? For instance, would it be possible to choose exactly what to fill these regions with? And this is what today's paper excels at. And it turns out it can do much, much more. Let's dive in. One, we can provide it this layout, which they refer to as a semantic mask. And it can synthesize clothes, pants and hair in many, many different ways. Heavenly. If you have a closer look, you see that fortunately it it doesn't seem to change any other parts of the image. Nothing too crazy here. But please remember this and now would be a good time to hold on to your papers because two it can change the sky or the material properties of the floor and. Wait, are you seeing what I am seeing? We cannot just change the sky because we have a lake there reflecting it. Therefore the lake has to change too. Does it? Yes, it does. It indeed changes other parts of the image when it is necessary, which is a hallmark of a learning algorithm that truly understands what it is synthesizing. You can see this effect especially clearly at the end of the looped footage when the sky is the brightest. Loving it. So what about the floor? This is one of my favorites. It doesn't just change the color of the floor itself, but it performs proper material modeling. Look, the reflections also become glossier over time. A proper light transport simulation for this scenario would take a very, very long time. We are likely talking from minutes to hours. And this thing has never been taught about light transport and learned about these materials by itself. Make no mistake, these may be low resolution pixelated images, but this still feels like science fiction. Two more papers down the line and we will see HD videos of this. I am sure. The third application is something that the authors refer to as appearance mixture, where we can essentially select parts of the image to our liking and fuse these selected aspects together into a new image. This could more or less be done with traditional handcrafted methods too, but for it can also do style morphing where we start from image A and change it until it looks like image B and back. Now normally this can be done very easily with a handcrafted method called image interpolation. However, to make this morphing really work, the tricky part is that all of the intermediate images have to be meaningful, and as you can see, this learning method does a fine job at that. Any of these intermediate images can stand on their own. I'll try to stop the morphing process at different points so you can have a look and decide for yourself. Let me know in the comments if you agree. I am delighted to see that these image synthesis algorithms are improving at a stunning pace and I think these tools will rapidly become viable to aid the work of artists in the industry. This episode has been supported by Weights and Biases. In this post they show you how to visualize your scikit learn models with with just a few lines of code. Look at all these beautiful visualizations. So good you can even try an example in an interactive notebook through the link in the video description. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
