This AI Creates Human Faces From Your Sketches!

Dear fellow scholars, this is two minute papers with Dr. Zsolnai in 2017. So more than 300 episodes ago we talked about an algorithm that took a 3D model of a complex object and would give us an easy to follow step by step breakdown on how to draw it. Automated drawing tutorials, if you will. This was a handcrafted algorithm that used graph theory to break these 3D objects into smaller, easier to manage pieces. And since then, learning algorithms have improved so much that we started looking more and more to the opposite direction. And that opposite direction would be giving a crude drawing to the machine and getting a photorealistic image. Now that sounds like science fiction until we realize that scientists at Nvidia already had an amazing algorithm for this around one and a half years ago. In that work, the input was a labeling which we can draw ourselves and the output is a hopefully photorealistic landscape image that adheres to these labels. I love how first only the silhouette of the rock is drawn, so we have this hollow thing on the right that is not very realistic and then it is now filled in with the bucket tool and there you go and next thing you know you have an amazing looking landscape image. It was capable of much, much more. But what it couldn't do is synthesize human faces this way. And believe it or not, this is what today's technique is able to do. Look, in goes our crude sketch as a guide image and out comes a nearly photorealistic human face that matches it. Interestingly, before we draw the hair itself, it gives us something as a starting point. But if we choose to wear, we can also change the hair shape and the outputs will follow our drawing really well. But it goes much further than this as it boasts a few additional appealing features. For instance, it not only refines the output as we change our drawing, but since one crude input can be mapped to many, many possible people, these output images can also be further art directed with these sliders. According to the included user study, Journeyman users mainly appreciated the variety they can achieve with this algorith. If you look here, you can get a taste of that. While professionals were more excited about the controllability aspect of this method that was showcased with the footage with the sliders. Another really cool thing that it can do is called face copy paste, where we don't even need to draw anything and just take a few aspects of human faces that we would like to combine and there you go. Absolutely amazing. This work is not without failure cases. However, you have probably noticed, but the AI is not explicitly instructed to match the eye colors where some asymmetry may arise in the output. I am sure this will be improved. Just one more paper down the line and I am really curious where digital artists will take these techniques in the near future. The objective is always to get out of the way and help the artist spend more time bringing their artistic vision to life and spend less time on the execution. This is exactly what these techniques can help with. What a time to be alive. What you see here is an instrumentation for a previous paper that we covered in this series which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.