This AI Makes The Mona Lisa Speak…And More!

This episode has been supported by lamda. Dear fellow scholars, this is two minute papers with. In an earlier episode, we covered a paper by the name Everybody Dance. Now, in this stunning work, we could take a video of a professional dancer, then record a video of our own, let's be diplomatic, less beautiful moves, and then transfer the dancer's performance onto our own body in the video. We call this process motion transfer. Now look at this new also learning based technique that does something similar where in goes a description of a pose, just one image of a target person, and on the other side, out comes a proper animation of this character according to our prescribed motions. Now, before you think that it means that we would need to draw and animate stick figures to use this, I. I will stress that this is not the case. There are many techniques that perform pose estimation where we just insert a photo or even a video and it creates all these stick figures for us that represent the pose that people are taking in these videos. This means that we can even have a video of someone dancing and just one image of the target person and the rest is history. Insanity. That is already amazing and very convenient. But this paper works with a video to video problem formulation, which is a concept that is more general than just generating movement. Way more. For instance, we can also specify the input video of us, then add one or at least a few images of the target subject, and we can make them speak and behave using our gestures. This is already absolutely amazing. However, the more creative minds out there are already thinking that if we are thinking about images, it can be a painting as well, right? Yes, indeed. We can make the Mona Lisa speak with it as well. It can also take a labeled image. This is what you see here, where the colored and animated patches show the object boundaries for different object classes. Then we take an input photo of a street scene and we get photorealistic footage with all the cars, buildings and vegetation. Now, make no mistake, some of these applications were possible before, many of which we showcased in previous videos, some of which you can see here. What is new and interesting here is that we have just one architecture that can handle many of these tasks. Beyond that, this architecture requires much less data than previous techniques, as it often needs just one or at most a few images of the target subject to do all this magic. The paper is ample in comparison to these other methods. For instance, the FID measures the quality and the diversity of the generated output images and is subject to minimization. And you see that it is miles beyond these previous works. Some limitations also apply if the inputs stray too far away from topics that the neural networks were trained on. We shouldn't expect results of this quality, and we are also dependent on proper inputs for the poses and segmentation maps for it to work well. The pace of progress in machine learning research is absolutely incredible, and we are getting very close to producing tools that can be actively used to empower artists working in the industry. What a time to be alive. If you are a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lamda GPU Cloud I've talked about Lambda's GPU workstations in other videos and I'm happy to tell you that they are offering GPU cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser. And finally, hold on to your papers because the Lambda GPU Cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs.compapers and sign up for one of their amazing GPU instances today. Our thanks to Lambda for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.