StarGAN2 generates photorealistic faces and transforms cats into dogs by learning separate latent spaces for each animal type. It can edit human faces by adding beards or aging them while keeping pose intact, though it sometimes struggles with details like ear shapes.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we have a selection of learning based techniques that can generate images of photorealistic human faces for people that don't exist. These techniques have come a long way over the last few years. So much so that we can now even edit these images to our liking by, for instance, putting a smile on their faces, making them older or younger, adding or removing a beard, and more. However, most of these techniques are still lacking in two one is diversity of outputs and two, generalization to multiple domains. Typically, the ones that work on multiple domains don't perform too well on most of them. This new technique is called Stargan2 and addresses both of these issues. Let's start with the humans. In the footage. Here you see a lot of interpolation between test subjects, which means that we start out from a source person and generate images that morph them into the target subjects. Not in any way, but in a way that all of the intermediate images are believable. In these results, many attributes from the input subject, such as pose, nose type, mouth shape and position are also reflected on the output. I like how the motion of the images on the left reflects the state of the interpolation. As this slowly takes place, we can witness how the reference person grows out a beard. But we are not nearly done yet. We noted that another great advantage of this technique is that it works for multiple domains. And this means of course, none other than us. Looking at cats morphing into dogs and other animals in these cases, I see that the algorithm picks up the gaze direction, so this generalizes to even animals. That's great. What is even more great is that the face shape of the tiger appears to have been translated to the photo of this cat. And if we have a bigger cat as an input, the output will also give us this lovely and a little plump creature. And look here. The cat in the input is occluded in this target image, but that is not translated to the output image. The AI knows that this is not part of the cat, but an occlusion. Imagine what it would take to prepare a handcrafted algorithm to distinguish these features. My goodness. And now onto dogs. What is really cool is that in this case, bendy ears have their own meaning. And and we get several versions of the same dog breed, with or without them. And it can handle a variety of other animals too. I could look at these all day. And now, to understand why this works so well, we first have to understand what a latent space is. Here you see an Example of a latent space that was created to be able to browse through fonts and even generate new ones. This method essentially tries to look at a bunch of already existing fonts and tries to boil them down into the essence of what makes them different. It is a simpler, often incomplete, but more manageable representation for a given domain. This domain can be almost anything. For instance, you see another technique that does something similar with material models. Now the key difference in this new work compared to previous techniques is that it creates not one latent space, but but several of these latent spaces for different domains. As a result, it can not only generate images in all of these domains, but can also translate different features. For instance ears, eyes, noses from a cat to a dog or a cheetah in a way that makes sense and the results look like absolute witchcraft. Now, since the look on this cheetah's face indicates that it has had enough of this video, just one more example before we go. As a possible failure case, have a look at the ears of this cat. It seems to be in a peculiar midway land between a pointy and a bent ear, but it doesn't quite look like any of them. What do you think? Maybe some of you cat people can weigh in on this? Let me know in the comments. What you see here is an instrumentation of this exact paper we have talked about, which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
