Dr. Zsolnai explains a two-step method that lets you edit natural scenes by specifying attributes like season or weather. A neural network generates a matching image while preserving the original layout, then transfers its style back to keep your content intact.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. In the last few years, the pace of progress in machine learning research has been staggering. Neural network based learning algorithms are now able to look at an image and describe what's seen in this image. Or even better, the other way around, generating images from a written description. You see here a set of results from Biggan, a state of the art image generation technique, and marvel at the fact that all of these images are indeed synthetic. The GAN part of this technique abbreviates the term generative adversarial network. This means a pair of neural networks that battle each other over time to master a task. For instance, to generate realistic looking images when given a theme. After that, Stylegan and even its second version appeared, which, among many other crazy good features, opened up the possibility to lock in several aspects of these images. For instance, age, pose, some facial features, and more. And then we could mix them with other images to our liking, while retaining these locked in aspects. I am loving the fact that these newer research works are moving in the direction of more artistic control. And the paper we'll discuss today also takes a step in this direction. With this new work, we can ask to translate our image into different seasons, weather conditions, time of day and more. Let's have a look. Here we have our input and imagine that we'd like to add more clouds and translate it into a different time of the day. And there we go. Wow. Or we can take this snowy landscape image and translate it into a blooming flowery field. This truly seems like black magic, so I can't wait to look under the hood and see what is going on. The input is our source image and a set of attributes where we can describe our artistic vision. For instance, here, let's ask the AI to add some more vegetation to this scene. That will do. Step number one. This artistic description is routed to a scene generation network which hallucinates an image that fits our description. Well, that's great. As you see here, it kind of resembles the input image, but still it is substantially different. So why is that? If you look here, you see that it also takes the layout of our image as an input. Or in other words, the colors and the silhouettes describe what part of the image contains a lake, vegetation, clouds and more. It creates the hallucination according to that. So we have more clouds. That's great. But the road here has been left out. So now we are stuck with an image that only kind of resembles what we want. What do we do now? Now, step number two. Let's not use this hallucinated image directly, but apply its artistic style to our source image. Brilliant. Now we have our content but with more vegetation. However, remember that we have the layout of the input image that is a goldmine of information. So, are you thinking what I am thinking? Yes. Including this indeed opens up a killer application. We can even change the scene around by modifying the labels on this layout. For instance, by adding some mountains, make it aggressive field and add a lake. Making a scene from scratch from a simple starting point is also possible. Just add some mountains, trees, a lake and you are good to go. And then you can use the other part of the algorithm to transform it into a different season, time of day, or even make it foggier. What a time to be alive. Now, as with every research work, there is still room for improvements. For for instance, I find that it is hard to define what it means to have a cloudier image. For instance, the hallucination here works according to the specification. It indeed has more clouds than this. But for instance, here I am unsure if we have more clouds in the output. You see that perhaps it is even less than in the input. It seems that not all of them made it to the final image. Also, do fewer and denser clouds qualify as cloudier? Nonetheless, I think this is going to be an awesome tool as is and I can only imagine how cool it will become two more papers down the line. This episode has been supported by Weights and Biases in this post. They show you how to easily iterate on models by visualizing and comparing experiments in real time. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
