A Two Minute Papers YouTube episode (presented by Zsolnai, uploaded by LARRUZ) explains what makes a good image-generator AI. It defines GANs and overfitting (memorization vs. generalization), then presents three evaluation techniques—visual nearest-training-image checks, human raters, and the inception score—gives numerical inception-score examples (real ≈233; older synthetic ≈50; recent best ≈166), and notes limitations of current evaluation methods.
Questions this video answers
January 23, 2019.
LARRUZ (presenter identified as Zsolnai).
Approximately 233.
Around 50.
Close to 166.
Chapters
Dear fellow scholars, this is two minute papers with zsolnai. In this series, we frequently talk about generative adversarial networks or GANs. In short, this means a pair of neural networks that battle each other over time to master a task. For instance, to generate realistic looking images from a written description. Here you see Nvidia's amazing work that was able to dream up high resolution images of imaginary celebrities. In the next episode, we will talk some more about their newest work that does something like this and is even better at it. Believe it or not. I hope you have subscribed to the channel to make sure not to miss out on that one. And for now, while we marvel at these outstanding results, I will quickly tell you about overfitting and what it has to do with images of celebrities. When we train a neural network, we wish to make sure that it understands the concepts we are trying to teach it. Typically, we feed it a database of labeled images where the labels mean that this depicts a dog and this one is not a dog, but a cat. After the training step took place in the ideal case, it will be able to build an understanding of these images so that when we show them new previously unseen images, it would be able to correctly guess which animals they depict. However, in many cases, we start training the neural network in and during the training. It gives us wonderful results and it gets the animals right every single time. But whenever it sees new previously unseen images, it can't tell a dog from a cat at all. This peculiar case is what we call overfitting. And this is the bane of machine learning research. Overfitting is like the kind of student we all encountered at school who is always very good at memorizing the textbook, who but can't solve even the simplest new problems on the exam. This is not learning, this is memorization. Overfitting means that a neural network does not learn the concept of dogs or cats. It just tries to memorize this database of images and is able to regurgitate it for us. But its knowledge cannot generalize for new images. That's not good. I want intelligence, not a copying machine. So at this point it is probably clearer what images of celebrities have to do with overfitting. So how do we know that this algorithm doesn't just memorize the celebrity image dataset it was given and can really generate new imaginary people. Is it the good kind of student or the lousy student? Technique number one, let's not just dream up images of new celebrities, but also visualize images from the training data that that are similar to this image. If they are too similar, we have an overfitting problem. Let's have a look. Now, it is easy to see that this is proper intelligence and not a copying machine, because it was clearly able to learn the facial features of these people and combine them in novel ways. This is what scientists at Nvidia did in their paper and are to be commended for that Technique number two, well, just take a bunch of humans and let them decide whether these images differ from the training set and if they are realistic. This kinda works. But of course costs quite a bit of money labor and we end up with something subjective. We better not compare the quality of research papers based on that. If we can avoid it, and get this, we can actually avoid it by using something called the inception score. Instead of using humans. This score uses a neural network to have a look at these images and measure the quality and the diversity of the results. As long as the image produces similar neuron activations within this neural network, two images will be deemed to be similar. Finally, this score is an objective way of measuring progress within this field and it is of course subject to maximization. So now you of course wish to know what the state of the art is today. For reference, a set of real images has an inception score of 233. And the best works that produced synthetic images just a few years ago had a score of around 50. To the best of my knowledge, as of the publishing of this video, the highest inception score for an AI is close to 166. So we've come a long, long way. You can see some of these images here. Truly exciting. What a time to be alive. The disadvantages of the method is that one, because the diversity of the outputs is also to be measured, it requires many thousands of images. This is likely more of an issue with the problem definition itself and not this method. And also since this means that the computers and and not real people have to do the work, we can give this one a pass. Disadvantage number two, I will include this paper in the video description for you, which basically describes that there are cases where it is possible to get the network to think an image is of higher quality than another one, even if it clearly isn't. Now you see that we have pretty ingenious techniques to measure the quality of image generator AI programs and and of course this area of research is also subject to improvement and I'll be here to tell you about it. Thanks for watching and for your generous support and I'll see you next time.
