Dr. Zsolnai explains StyleGAN2-ADA, an AI that generates realistic dog photos using 10 times fewer training images than the previous version. The model needed over 3,300 experiments to get right, and it's already in Photoshop for neural filters and photo restoration.
Questions this video answers
December 2019.
About six months later.
>3,300 experiments.
255 experiments.
Roughly ten times fewer images; can train from less than 10,000 images.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to explore a paper that improves on the incredible Stargan 2. What is that? Stargan 2 is a neural network based learning algorithm that is not only capable of creating these eye poppingly detailed images of human beings that don't even exist, but it also improved on its previous version in a number of different ways. For instance, with the original Stagean method, we could exert some artistic control over these images. However, look, you see how this part of the teeth and eyes are pinned to a particular location and the algorithm just refuses to let it go, sometimes to the detriment of its surroundings. The improved Stargan 2 method addressed this problem. You can see the results here. Teeth and eyes are now allowed to float around freely and perhaps this is the only place on the Internet where we can say that and be happy about. Could also mix two images together and it could do it not only for human faces, but for cars, buildings, horses and more. And get this, this paper was published in December 2019 and since then it has been used in a number of absolutely incredible applications and follow up works. Let's look at three of them. One, the first question I usually hear when I talk about an amazing paper like this is ok, great, but when do I get to use this? And the answer is right now, because it is implemented in Photoshop in a feature that is called neural filters. Two, artistic control over these images has improved so much that that now we can pin down a few intuitive parameters and change them with minimal changes to other parts of the image. For instance, it could grow Elon Musk a majestic beard. And Elon Musk was not the only person who got an algorithmic beard. I hope you know what's coming. Yes, I got one too. Let me know in the comments whose beard you liked better. Three, a nice follow up paper that could take a photo of Abraham Lincoln and and other historic figures and restored their images as if we were time travelers and took these photos with a more modern camera. The best part here was that it leveraged the superb morphing capabilities of Stargan 2 and took an image of their siblings, a person who has somewhat similar proportions to the target subject, and morphed them into a modern image of this historic figure. This was brilliant because restoring images is hard, but with stag and two, morphing is now easy. So the authors decided to trade a difficult problem for an easier one. And the results speak for themselves. Of course, we cannot know for sure if this is what these historic figures really looked like. But for now, it makes one heck of a thought experiment. And now let's marvel at these beautiful results with the new method that goes by the name stargan2ada. While we look through these results, all of which were generated with the new method, here are three things that it does better. One, it often works just as well as Stylegan two, but requires ten times fewer images for training. This means that now it can create these beautiful images. And this can be done by training a set of neural networks from less than 10,000 images at a time. Whoa. That is not much at all. Two, it creates better quality results. The baseline here is the original Stylegan 2. The numbers are subject to minimization and are a measure of the quality of these images. As you see from the bolded numbers, the new method not only beats the baseline method substantially, but but it does it across the board. That is a rare sight indeed. And three, we can train this method faster. It generates these images faster and in the meantime also consumes less memory, which is usually in short supply on our graphics cards. Now, we noted that the new version of the method is called stylegan2ada. What is ADA? This part means Adaptive Discriminator Augmentation. What does that mean exactly? This means that the new method endeavors to squeeze as much information out of these training datasets as it can. Data augmentation is not new. It has been done for many years now. And essentially this means that we rotate, colorize, or even corrupt these images during the training process. The key here is that with this, we are artificially increasing the number of training samples the neural network sees. The difference here is that they used a greater set of augmentations. And the adaptive part means that these augmentations are tailored more to the dataset at hand. And now comes the best part. Hold on to your papers and let's look at the timeline here. Stargan 2 appeared in December 2019, and Stargan 2 ADA. This method came out just half a year later. Such immense progress in just six months of time. The pace of progress in machine learning research is absolutely stunning these days. Imagine what we will be able to do with these techniques just a couple more years down the line. What a time to be alive. But this paper also teaches a very important lesson to us that I would like to show you. Have a look at this table. That shows the energy expenditure for this project for transparency, but it also tells us the number of experiments that were required to finish such an amazing paper. And that is more than 3,300 experiments. 255 of which were wasted due to technical problems. In the foreword of my PhD thesis I wrote the research is the study of failure. More precisely, research is the study of obtaining new knowledge through failure. A bad researcher fails 100% of the time, while a good one only fails 99% of the time. Hence, what you see written here and in most papers is only 1% of the work that has been done. I would like to thank Felicia, my wife, for providing motivation, shielding me from distractions and bringing sunshine to my life to endure through many of these failures. This paper is a great testament to show how difficult the life of a researcher is. How many people give up their dreams after being rejected once or maybe two times. 10 times. Most people give up after 10 tries and just imagine having a thousand failed experiments. That and still not even being close to publishing a paper yet. And with a little more effort this amazing work came out of it. Failing doesn't mean losing. Not in the slightest. Huge congratulations to the authors for their endurance and for this amazing work. And I think this also shows that these weights and biases experiment tracking tools are invaluable because it is next to impossible to remember what went wrong with each of them and and what should be fixed. What you see here is a report of this exact paper we have talked about which was made by weights and biases. I put a link to it in the description, make sure to have a look. I think it helps you understand this paper better. If you work with learning algorithms on a regular basis. Make sure to check out weights and biases. Their system is designed to help you organize your experiments and it is so good it could shave off weeks or even months of work from your projects and and is completely free for all individuals, academics and open source projects. This really is as good as it gets and it is hardly a surprise that they are now used by over 200 companies and research institutions. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
