Finally, AI-Based Painting is Here!

Dear fellow scholars, this is two Minute papers with zsolnai. A few years ago, the Generative Adversarial Network architecture appeared that contains two neural networks that try to outcompete each other. It has been used extensively for image generation and has become a research subfield of its own. For instance, they can generate faces of people that don't exist and much, much more. This is great. We should be grateful to live in a time when breakthroughs like this happen in AI research. However, we should also note that artists usually have a vision of the work that they would like to create. And instead of just getting a deluge of new images, most of them would prefer to have some sort of artistic control over the results. This work offers something that they call semantic paintbrushes. This means that we can paint not in terms of colors, but in terms of concepts. Now, this may sound a little nebulous, so if you look here, you see that as a result, we can grow trees, change buildings, and do all kinds of shenanigans without requiring us to be able to draw the results by hand. Look at those marvelous results. It works by compressing down these images into a latent space. This is a representation that is quite sparse and captures the essence of these images. One of the key ideas is that this can then be reconstructed by a generator neural network to get a similar image back. However, the twist is that while we are in the latent domain, we can apply these intuitive edits to this image. So when the generator step takes place, it will carry through our changes. If you look at the paper, you will see that just using one generator network doesn't yield these great results. Therefore, this generator needs to be specific to the image we are currently editing. The included user study shows that the new method is preferred over the previous techniques. Now, like all of these methods, this is not without limitations. Here you see that despite trying to remove the chairs from the scene, amusingly, we get them right back. That's a bunch of chairs, free of charge. In fact, I'm not even sure how many chairs we got here. If you figure that out, make sure to leave a comment about it. But all in all, that's not what we asked for. And solving this remains a challenge for the entire family of these algorithms. And good news, in fact, when talking about a paper, probably the best kind of news is that you can try it online through a web demo right now. Make sure to try it and post your results here if you find anything interesting. The authors themselves may also learn something new from us about interesting new failure cases. It has happened before in this series. This episode has been supported by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. It is like a shared logbook for your team and with this you can compare your own experiment results, put them next to what your colleagues did and you can discuss your successes and failures much easier. It takes less than five minutes to set up and is being used by OpenAI, Toyota Research, Stanford and Berkeley. It was also used in this OpenAI project that you see here, which we covered earlier in the series. They reported that experiment tracking was crucial in this project and that this tool saved them quite a bit of time and money. If only I had access to such a tool during our last research project where I had to compare the performance of neural networks for months and months. Well, it turns out I will be able to get access to these tools because get this, it's free and will always be free for academics and open source projects. Make sure to Visit them through wandb.com papers wandb.com papers or just click the link in the video description and sign up for a free demo today. Our thanks to Weights and Biases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.