Researchers built an AI that turns photos into caricatures by combining two neural networks: one transfers artistic style while keeping the face recognizable, the other warps facial geometry using 60 facial landmarks. You can dial up or down how exaggerated the distortion gets.
Dear fellow scholars, this is two minute Papers with Style transfer is an interesting problem in machine learning research where we have two input images, one for content and one for style and the output is our content image reimagined with this new style. The cool part is that the content can be a photo straight from our camera and the style can be a painting which leads to super fun and really good looking results. This subfield is only a few years old and has seen a number of amazing papers. Style transfer for HD images, videos and some of these forgeries were even able to make professional art curators think that they were painted by a real artist. So here's a crazy idea. How about using style transfer to create caricatures? Well, this sounds quite challenging. Just think about it. A caricature is an elusive art where certain human features are exaggerated and generally the human face needs to be simplified and boiled down into its essence. It is a very human thing to do. So how could possibly an AI be endowed with such a deep understanding of, for instance, a human face? That sounds almost impossible. Our suspicion is further reinforced as we look at how previous style transfer algorithms tried to deal with this problem not too well, but no wonder. It would be unfair to expect great results as this is not what they were designed for. But now look at these truly incredible results that were made with this new work. The main difference between the older works and this One is that one it uses generative adversarial networks GANs. In short, this is an architecture where two neural networks learn together. One learns to generate better forgeries and the other learns to find out whether an image has been forged. However, this would still not create the results that you see here. An additional improvement is that we have not one, but two of these GANs. One deals with style, but it is trained in a way to keep the essence of the image and the other deals with changing and warping the geometry of the image to achieve an artistic effect. This leans on the input of a landmark detector that gives it around 60 points that show the location of the most important parts of a human face. The output of this geometry again is a distorted version of this point set which can then be used to warp the style image to obtain the final output. This is a great idea because the amount of distortion applied to the points can be controlled so we can tell the AI how crazy of a result we are looking for. Great. The authors also experimented applying this to video. In my opinion, the results are incredible. For a first crack at this problem, we are probably just one paper away from an AI automatically creating absolutely mind blowing caricature videos. Make sure to have a look at the paper as it has a ton more results. And of course every element of the system is explained in great detail. There is and if you enjoyed this episode and you would like to access all future videos in early access or get your name immortalized in the video description as a key supporter, please consider Supporting us on Patreon.com TwominutePapers the link is available in the video description. We were able to significantly improve our video editing rig and this was possible because of your generous support. I am so grateful that thank you so much. And this is why every episode ends with the usual quote. Thanks for watching and for your generous support and I'll see you next time.
