AI Learns Human Pose Estimation From Videos | Two Minute Papers #237

Dear fellow scholars, this is two Minute Papers with zsolnai. This project is a collaboration between INREA and Facebook AI Research and is about pose estimation. Pose estimation means that we take an input photo or in the cooler case, video of people and the output should be a description of their postures. This is kind of like motion capture for those amazing movie and computer game animations, but without the studio and the markers. This work goes even further and tries to offer a full 3D reconstruction of the geometry of the bodies, and it is in fact doing way more than that, as you will see in a minute. Neural networks are usually great at these tasks, provided that we have a large number of training samples to train them. So the first step is gathering a large amount of annotated data. This means an input photograph of someone which is paired up with the correct description of their posture. This is what we call one training sample. This new proposed data set contains 50,000 of these training samples and using that we can proceed to step number two, training the neural network to perform pose estimation. But there is more to this particular work. Normally this pose estimation takes place with a 2D skeleton, which means that most techniques output a stick figure, but not in this case because the dataset contains segmentations and Dense correspondences between 2D images and 3D models. Therefore, the network is also able to output fully 3D models. There are plenty of interesting details shown in the paper. For instance, since the annotated ground truth footage in the training set is created by humans, there is plenty of missing data that is filled in by using a separate neural network that is specialized for this task. Make sure to have a look at the paper for more cool details like this. This all sounds good in theory, but but a practical application has to be robust against occlusions and rapid changes in posture. And the good thing is that the authors published plenty of examples with these that you can see here. Also, it has to be able to deal with smaller and bigger scales when people are closer or further away from the camera. This is also a challenge. The algorithm does a really good job at this and remember, no markers or studio setup is required and everything that you see here is performed interactively. The dataset will appear soon and it will be possible to reuse it for future research works. So I expect plenty of more collaboration and follow up works for this problem. We are living amazing times indeed. Thanks for watching and for your generous support and I'll see you next time.