A Two Minute Papers video (Nov 5, 2017) presenting an NVIDIA research paper on advanced image-translation using generative adversarial networks. The narrator explains image translation and GANs, notes this work uses six adversarial networks and a shared latent space to translate between domains, and highlights applications: converting real-world video to video-game style and vice versa, breed/attribute editing (e.g., changing hair color, adding sunglasses). Training remains non-trivial and convergence is unclear; source code is released under a Creative Commons NonCommercial-NoDerivatives license.
Chapters
Dear fellow scholars, this is two minute. Papers with researchers at Nvidia have been hitting it out of the park lately and this work is no different. This technique performs image translation, which means that the input is an image of our choice and the output is the same image with somewhat different semantics. For instance, an image of a city can be translated to a map of this city, or a daytime photo or video can be converted to appear as if it were shot during the night. And throughout the video you'll see so much more of these exciting applications. A typical way of accomplishing this is done by using generative adversarial networks. This is a pair of neural networks where the generator network creates new synthetic images trying to fool a discriminator network which learns to tell a fake synthesized image from a real one. These two neural networks learn together where one tries to come up with better solutions to fool the discriminator, where the discriminator seeks to get better at telling forgeries from the real photographs. In the end, this rivalry makes both of them get better and better, and the final result is an excellent technique to create convincing image translations. In this work, not two but six of these networks are being used, so make sure to have a look at the paper for details. There was an earlier work that was able to perform image translation by leaning on a novel cycle consistency constraint. This means that we assume that the source image can be translated to the target image and then this target image can be translated back to look exactly like the source. This kinda means that these translations are not arbitrary and are mathematically meaningful operations. Here the new technique builds on a novel assumption that there exists a latent space in which the input and the output images can both coexist. This latent space is basically an intuitive and concise representation of some more complicated data. For instance, earlier we experimented a bit with fonts and had seen that even though the theory of font design is not easy, we can create a two dimensional latent space that encodes simple properties like curvature that can describe many, many fonts in an intuitive manner. Remarkably, with this new work, converting dogs and cats into different breeds is also a possibility. Interestingly, it can also perform real to synthetic image translation and vice versa. So that means that it can create video game footage from our real world videos, and even more remarkably, convert video game footage to real world video. This is insanity. One of the craziest ideas I've seen in a while. Bravo. And now hold on to your papers. Because it can also perform attribute based image translation. This means that, for instance, we can grab an image of a human face and transform the model's hair to blonde. Add sunglasses or smiles to it at will. A limitation of this technique is that training is still non trivial as it still relies on generative adversarial networks and it is not yet clear whether there is a point to which the training converges or not. The source code of this project is also available. Make sure to take a good look at the license before doing anything because it is under the Creative Commons, non commercial and no derivatives license. Thanks for watching and for your generous support and I'll see you next time. Ra.
