A short video (two-minute-papers style) by Károly Zsolnai-Fehér explaining a research paper on creating controllable characters from real-world videos. The core advance is not just pose estimation but making characters controllable via a controller: the system can take an input pose plus a controller command to produce the next pose (pose-to-pose network) and then render that pose into an image (pose-to-frame architecture). The method includes training that encourages continuous character masks for smooth motion; demos show controller-driven novel actions and changed backgrounds. The presenter notes visible artifacts but frames the paper as an early, promising step toward turning real videos into interactive game characters.
Questions this video answers
We Can All Be Video Game Characters With This AI
Károly Zsolnai-Fehér
June 22, 2019
Chapters
Dear fellow scholars, this is two minute papers with the title of this paper is very descriptive. It says controllable characters extracted from real world videos. This sounds a little like science fiction, so let's pick this apart. If we forget about the controllable part, we get something that you've seen in this series many pose estimation. Pose estimation means that we have a human character in an image or a video, have a computer program look at it and tell us the current position this character is taking. This is useful for medical applications such as detecting issues with motor functionality, fall detection, or we can also use it for motion capture for our video games and blockbuster movies. So just performing the pose estimation part is a great invention, but relatively old news. So what's really new here? Why is this work interesting? How does it go beyond pose estimation? Well, as a hint, the title contains an additional word, controllable. So look at this. Woohoo. As you see, this technique is not only able to identify where a character is, but we can grab a controller and move it around. This means making this character perform novel actions and showing it from novel views. That's really remarkable because this requires a proper understanding of the video we are watching. And this means that we can not only watch these real world videos. As you see this small piece of footage used for the learning, but by performing these actions with the controller, we can make a video game out of it. Especially given that here the background has also been changed to achieve this. This work contains two key elements. Element number one is the pose to pose network that takes an input posture and the button we pushed on the controller and creates the next step of the animation. And then element number two, the pose to frame architecture, blends this new animation step into an already existing image. The neural network that performs this is trained in a way where it is encouraged to create these character masks in a way that is continuous and doesn't contain jarring jumps between the individual frames leading to smooth and believable movements. Now, clearly anyone who takes a cursory look sees that the animations are not perfect and still contain artifacts. But just imagine that this paper is among the first introductory works on this problem. Just imagine what will have two more papers down the line. I can't wait. Thanks for watching and for your generous support and I'll see you next time.
