Károly Zsolnai-Fehér explains how AI transfers a professional dancer's movements onto someone else in video. The technique uses pose estimation to map the dancer's skeleton, then a generator network fills in the details while adversarial training keeps the result realistic and temporally smooth.
Dear fellow scholars, this is two minute papers with do you remember style transfer? Style transfer is a mostly AI based technique where we take a photograph, put a painting next to it, and it applies the style of the painting to our photo. That was amazing. Also, do you remember pose estimation? This is a problem where we have a photograph or a video of someone and the output is a skeleton that that shows the current posture of this person. So how about something that combines the power of pose estimation with the expressiveness of style transfer? For instance, this way we could take a video of a professional dancer, then record a video of our own, let's say moderately beautiful moves, and then transfer the dancer's performance onto our own body in the video. Let's call it motion transfer. Have a look at these results. How cool is that? As you see, these output videos are quite smooth. And this is not by accident. It doesn't just come out like that. With this technique, temporal coherence is taken into consideration. This means that the algorithm knows what it has done a moment ago and will not do something wildly different, making these dance motions smooth and believable. This method uses a generative adversarial network where we have a neural network for pose estimation or in other words, generating the skeleton from an image and a generator network to create new footage. When given a test subject and a new skeleton posture, these two neural networks battle each other and teach each other to distinguish and create more and more authentic footage over time. Some artifacts are still there, but note that this is among the first papers on this problem and and it is already doing incredibly well. This is fresh and experimental, just the way I like it. Two follow up papers down the line and will be worried that we can barely tell the difference from authentic footage. Make sure to have a look at the paper where you will see how the Pix2Pix algorithm was also used for image generation. And there is a nice evaluation section as well. And now let the age of AI based dance videos begin. If you enjoyed this episode, please consider supporting us on Patreon where you can pick up really cool perks like early access to these videos, voting on the order of future episodes and more. We are available at patreon.com twominutepapers or just click the link in the video description. Thanks for watching and for your generous support and I'll see you next time.
