Károly Zsolnai-Fehér walks through a neural network that generates full human bodies from a single low-quality image. It can transform crude drawings into photorealistic people, shift someone into new poses and unseen angles, and transfer appearance from one photo onto another.
Károly Zsolnai-Fehér
patreon.com/TwominutePapers
Dear fellow scholars, this is two minute papers with. In this series we often discuss that neural networks are extraordinarily useful for classification tasks. This means that if we give them an image, they can tell us what's on it, which is great for self driving cars, image search and a variety of other applications. However, fewer people know that they can also be used for image generation. We've seen many great examples of this where Nvidia's AI was able to dream up high resolution images of imaginary celebrities. This was done using a generative adversarial network, an architecture where two neural networks battle each other. However, these methods don't work too well if we have too much variation in our data sets. For instance, they are great for faces, but not for synthesizing the entire human body. This particular technique uses a different architecture and as a result can synthesize an entire human body and is also able to synthesize both shape and appearance. You will see in a moment that because of that it can do magical things. For instance, in this example, all we have is one low quality image of a test subject as an input and we can give it a photo of a different person. What happens now is that the algorithm runs pose estimation on this input and transforms our test subject into that pose. The crazy thing about this is that it even creates views for new angles we didn't even have access to. In this other experiment we have one image on the left. What we can do here is that we specify not a person, but draw a pose directly, indicating that we wish to see our test subject in this pose. And and the algorithm is also able to create an appropriate new image. And again, it works for angles that require information that we don't have access to. These new angles show that the technique understands the concept of shorts or trousers, although it seems to forget to put on socks sometimes. Truth be told, I don't blame it. What is even cooler is that it seems to behave very similarly for a variety of different inputs. This is non trivial as this property doesn't just emerge out of thin air and will be a great selling point for this new method. It also supports a feature where we need to give a crude drawing to the algorithm and it will transform it into a photorealistic image. However, it is clear that there are many, many ways to fill this drawing with information. So how do we tell the algorithm what appearance we are looking for? Well, worry not, because this technique can also perform appearance transfer. This means that we can exert artistic control over the output by providing a photo of a different object and it will transfer the style of this photo to our input. No artistic skills needed, but good taste is as much of a necessity as ever. Yet another AI that will empower both experts and novice users alike. And while we are enjoying these amazing results, or even better, if you have already built up an addiction for the papers, you can keep it in check by supporting us on Patreon and in return getting access to these videos earlier. You can find us through patreon.com TwominutePapers There is a link to it in the video description and as always, to the paper as well. Thanks for watching and for your generous support and I'll see you next time.
