This is a short Two Minute Papers YouTube episode (host: Károly Zsolnai-Fehér) summarizing a recent Nvidia paper that performs few-shot image-to-image translation. The host explains that the Nvidia system uses a generative adversarial network plus a class encoder to create low-dimensional class embeddings, enabling translation into previously unseen target classes from only a few example images (demo on pet images, e.g., converting a golden retriever image into other dog breeds). The video also notes limitations (large differences or unusual head poses can challenge the model) and includes a Lambda Labs sponsorship and a web demo invitation.
Chapters
This episode has been supported by Lambda Labs. Dear fellow scholars, this is two minute papers with. Let's talk about a great recent development in image translation. Image translation means that some image goes in and it is translated into an analogous image of a different class. A good example of this would be when we have a standing tiger as an input and we ask the algorithm to translate this image into the same tiger lying down. This leads to many amazing applications. For instance, we can specify a daytime image and get the same scene during nighttime. We can go from maps to satellite images, from video games to reality, and more. However, much like many learning algorithms today, most of these techniques have a key limitation. They need a lot of training data. Or in other words, these neural networks require seeing a ton of images in in all of these classes before they can learn to meaningfully translate between them. This is clearly inferior to how humans think, right? If I would show you a horse, you could easily imagine, and some of you could even draw what it would look like if it were a zebra instead. As I'm sure you have noticed by reading arguments on many Internet forums, humans are pretty good at generalization. So how could we possibly develop a learning technique that that can look at very few images and obtain knowledge from them that generalizes? Well, have a look at this crazy new paper from scientists at Nvidia that accomplishes exactly that. In this example, they show an input image of a golden retriever, and then we specify the target classes by showing them a bunch of different animal breeds. And look, in goes your golden and out comes a pug or any other dog breed you can think of. And now hold on to your papers, because this AI doesn't have access to these target images and it had only seen them the very first time as we just gave it to them. It can do this translation with previously unseen object classes. How is this insanity even possible? This work contains a generative adversarial network, which assumes that the training set we give it contains images of different animals. And what it does during training is practicing the translation process between these animals. It also contains a class encoder that creates a low dimensional latent space for each of these classes, which means that it tries to compress these images down to a few features that contain the essence of these individual dog breeds. Apparently, it can learn the essence of these classes really well because it was able to convert our image into a pug, and without ever seeing a pug other than this one target image. As you can see here, it comes out way ahead of previous techniques. But of course, if we give it a target image that is dramatically different than anything the AI has seen before, it may falter. Luckily, you can even try it yourself through this web demo which works on pets. So make sure to read the instructions carefully and let the experiments begin. In fact, due to popular request let me kick this off with Lisa, my favorite chihuahua. I got many tempting alternatives but worry not. In reality she will stay as is. I was also curious about trying a non traditional head position and as you see with the results this was a much more challenging case for the AI. The paper also discusses this limitation in more detail. You know the saying two more papers down the line and I am sure this will also be remedied. I am hoping that you will also try your own pets and as a fellow scholar, you will flood the comment section here with your findings. Strictly for science, of course. If you're doing deep learning, make sure to look into Lambda GPU systems. Lambda offers workstations, servers, laptops, and the GPU cloud. For deep learning. You can save up to 90% over AWS, GCP and Azure GPU instances. Every Lambda GPU system is Pre installed with TensorFlow, PyTorch and Keras. Just plug it in and start training. Lambda customers include Apple, Microsoft and stanford. Go to lambdalabs.compapers or click the link in the video description to learn more. Big thanks to Lamda for supporting two minute Papers and helping us make better videos. Thanks for watching and for your generous support and I'll see you next time.
