Dr. Zsolnai shows AI that transforms animals between breeds and species while keeping their pose intact. The system works on unseen animals, morphs between two style images at once, and sometimes fails when parts are cropped or poses don't match.
2020-08-30
Quadro RTX 6000, RTX 8000, and V100 instances
lambdalabs.compapers
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. The research field of image translation with the aid of learning algorithms has been on fire lately. For instance, this earlier technique would look at a large number of animal faces and could interpolate between them. Or in other words, blend one kind of dog into another breed. But that's not all, because it could even transform dogs into cats, or create these glorious plump cats and cheetahs. The results were absolutely stunning. However, it would only work on the domains it was trained on. In other words, it could only translate to and from species that it took the time to learn about. This new method offers something really amazing. It can handle multiple domains or multiple breeds, if you will, even ones that it hadn't seen previously. That sounds flat out impossible. So let's have a look at some results. This dog will be used as content, therefore the output should have a similar pose, but its breed has to be changed to this one. But there is one little problem, and that problem is that the AI has never seen this breed before. This will be very challenging because we only see the head of the dog used for style. Should the body of the dog also get curly hair? You only know if you know this particular dog breed, or if you're smart and can infer missing information by looking at other kinds of dogs. Let's see the result. Incredible. The remnants of the leash also remain there in the output results. It also did a nearly impeccable job with this bird, where again, the style image is from a previously unseen breed. Now, this is of course a remarkably difficult problem domain translating into different kinds of animals that you know nothing about. Apart from a tiny cropped image, this would be quite a challenge even for a human. However, this one is not the first technique to attempt to solve it. So let's see how it stacks up against a previous method. This one is from just a year ago, and you will see in a moment how much this field has progressed since then. For instance, in this output we get two dogs, which seems to be a mix of the content and the style dog. And while the new method still seems to have some structural issues, the dog type and the pose is indeed correct. The rest of the results also appear to be significantly better. But what do you think? Did you notice something weird? Let me know in the comments below. And now let's transition into image interpolation. This will be a touch more involved than previous interpolation efforts. You see, in this previous paper we had a source and the target image, and the AI was asked to generate intermediate images between them. Simple enough. In this case however, we have not two but three images as an input there will be a content image. This will provide the high level features such as pose and its style is going to transition from this to to this. The goal is that the content image remains intact while transforming one breed or species into another. This particular example is one of my favorites. Such a beautiful transition and surely not all but many of the intermediate images could stand on their own. Again, the style images are from unseen species. Not all cases do this well with the intermediate images. However, here we start with one eye because the content and the style image have one eye visible while the target style of the owl has two. How do we solve that? Of course with nuclear fission look very amusing. Loving this example, especially how impossible it seems because the owl is looking into the camera with both eyes while we see its backside below the head. If it looked to the side like the input content image, this might be a possibility, but with this contorted body posture I am not so sure. So I'll give it a pass on this one. So there you go, transforming one known animal into a different one that the AI has never seen before. And it is already doing a more than formidable job at that. What a time to be alive. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances. And hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure plus they are the only cloud service with 48gb of RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances. Workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
