A short Two Minute Papers video explaining recent image-to-image translation methods that use generative adversarial networks (GANs) to convert sketches, maps, and labels into photorealistic images. The speaker highlights a newer technique that produces multiple diverse, high-quality outputs (trading off diversity and quality favorably), shows comparisons to ground-truth photos, and notes that source code is available (check the license).
Chapters
Dear fellow scholars, this is 2 Minute Papers with recently, a new breed of AI techniques surfaced that were capable of this new thing called image translation. And by image translation I mean that they can translate a drawn map to a satellite image, take a set of color labels and make a photorealistic facade, or take a sketch and create a photo out of it. This is done through a generative adversarial network. This is an architecture where we have a pair of neural networks. One that learns to generate new images and the other learns to tell a fake image from a real one. As they compete against each other, they get better and better without any human interaction. In these earlier applications, unfortunately, the output is typically one image. And since there are many possible shoes that could satisfy our initial sketch, it is highly unlikely that the one we are offered is exactly what we envisioned. This improved version enhanced this algorithm to be able to produce not one, but an entire set of outputs. And as you can see here, we have a night image and a set of potential daytime translations on the right that are quite diverse. I really like how it has an intuitive understanding of the illumination differences of the building during night and daytime. It really seems to know how to add lighting to the building. It also models the atmospheric scattering during daytime and creates multiple kinds of pretty convincing clouds or puts hills in the background. The results are both realistic and the additional selling point is that this technique offers an entire selection of outputs. What I found to be really cool about the next comparisons is that the ground truth images are also attached for reference. If we can take a photograph of a city at nighttime, we have access to the same view during the daytime too. Or we can take a photograph of a shoe and draw the outline of it by hand. As you can see here, there are not only lots of high quality outputs, but in some cases the ground truth image is really well approximated by the algorithm. This means that we give it a crude drawing and it could translate this drawing into a photorealistic image. I think that is mind blowing. The validation section of the paper reveals that this technique provides a great trade off between diversity and quality. There are previous methods that perform well if we need one high quality solution or many not so great ones. But overall, this one provides a great package for artists working in the industry. And this will be a godsend for any kind of content creation scenario. The source code of this project is also available and make sure to read the license before starting your experiments. Thanks for watching and for your generous support and I'll see you next time.
