Dr. Zsolnai walks through three new AI tools for editing photos: changing a car's shape from sporty to boxy, adjusting a painting's brushstrokes and style, and tweaking faces with precise controls for expression, age, and lipstick. Each uses neural networks that let you slide between options instead of typing commands.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Here you see people that don't exist. How can that be? Well, they don't exist because these images were created with a neural network based learning method by the name Stargan2, which can not only create eye poppingly detailed looking images, but it can also fuse these people together or generate cars, churches, and of course cats. An even cooler thing is that many of these techniques allow us to exert artistic control over these images. So how does that happen? How do we control a neural network? It happens through exploring latent spaces. And what is that? A latent space is a made up place where we are trying to organize data in a way that similar things are close to each other. What you see Here is a 2D latent space for generating different fonts. It is hard to explain why these fonts are similar, but most of us would agree that they indeed share some common properties. The cool thing here is that we can explore this latent space with our cursor and generate all kinds of new fonts. You can try this work in your browser. The link is available in the video description. And luckily we can build a latent space not only for fonts, but for nearly anything. I am a light transport researcher by trade, so in this earlier paper we were interested in generating hundreds of variants of a material model to populate this scene. In this latent space, we can concoct all of these really cool digital material models. A link to this work is also available in the video description. So let's recap. One of the cool things we can do with latent spaces is generate new images that are somewhat similar. But there is a problem as we go into nearly any direction, not just one thing, but many things about the image change. For instance, as we explore the space of fonts here, not just the width of the font changes, everything changes. Or if we explore materials here, not just the shininess or the colors of the material change, everything changes. This is great to explore if we can do it in real time. If I change this parameter, not just the car shape changes, the foreground changes, the background changes. Again, everything changes. So these are nice and intuitive controls, but not interpretable controls. Can we get that somehow? The answer is yes. Not everything must change. This previous technique is based on Stylegan 2 and is called Style Flow and it can take an input photo of a test subject and edit a number of meaningful age, expression, lighting, pose, you name it. For instance, it could also grow Elon Musk, a majestic beard, and that's not all, because Elon Musk is not the only person who got a beard. Look, this is me here after I got locked up for dropping my papers and I spent so long in here that I grew a beard. Or I mean this neural network gave me one. And since the punishment for dropping your papers is not short, in fact it is quite long, this happened. Ouch. I hereby promise to never drop my papers ever again. You will also have to hold on to yours too, so stay alert. So apparently interpretable controls already exist and I wonder how far can we push this concept. Beard or no beard is great, but what about cars? What about paintings? Well, this new technique found a way to navigate these latent spaces and introduces three amazing new examples of interpretable controls that I haven't seen anywhere else yet. 1. It can change the car geometry. We can change the sportiness of the car and even ask the design to be more or less boxy. Note that there is some additional damage here, but we can counteract that by changing the foreground to our taste. For instance, add some grass in there. 2. It can repaint paintings. We can change the roughness of the brushstrokes, simplify the style. Or even rotate the model. This way we can create or adjust without having to even touch a paintbrush. Three Facial expressions First, when I started reading this paper I was a little suspicious. I have seen these controls before so I looked at it like this. But as I saw how well it did, I went more like this. And this paper can do way more. It can add lipstick, change the shape of the mouth or the eyes and do all this with very little collateral damage to the remainder of the image. Loving can also find and blur the background similarly to those amazing portrait mode photos that newer smartphones can do. And of course it can also do the usual adjusting the age, hairstyle or growing a beard. So with that, there we go. Now with the power of neural network based learning methods, we can create new car designs, can repaint paintings without ever touching a paintbrush, and give someone a shave. It truly feels like we are living in a science fiction world. What a time to be alive. This episode has been supported by Weights and biases. In this post they show you how to use their tool to visualize confusion matrices and find out where your neural network made mistakes and what exactly those mistakes were. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
