A convolutional neural network predicts 2D joints from images, then an optimization model uses body constraints to reconstruct 3D pose and shape. The method handles occlusion and lighting challenges, with applications in games, sports analysis, and robotics.
Dear fellow scholars, this is two minute papers with Pose estimation is an interesting area of research where we typically have a few images or video footage of humans and we try to automatically extract the pose this person was taking. In short, the input is mostly a 2D image and the output is typically a skeleton of the person. Applications of pose estimation include automatic creation of assets for computer games and digital media, analyzing and coaching the techniques of athletes, or helping computers understand what they see for the betterment of robotics and machine learning techniques. And this is just a taste. The list was by no means exhaustive, beyond the obvious challenge of trying to reconstruct 3D information from a simple 2D image. The problem is fraught with difficulties as one has to be able to overcome the ambiguity of lighting occlusions and clothing covering the body. A tough problem, no question about that. An ideal technique would do this automatically without any user intervention, which sounds like wishful thinking. Or does it? In this paper, a previously proposed convolutional neural network is used to predict the position of the individual joints. And curiously, it turns out that we can create a faithful representation of the 3D human body from that by means of optimization. We have had a previous episode on mathematical optimization, you know the drill. The link is available in the video description box. What is remarkable here is that not only the pose, but the body type is also inferred. Therefore, the output of the process is not just the skeleton, but full 3D geometry. It is coarse geometry, so don't expect a ton of details, but it's 3D geometry more than what most other competing techniques can offer to ease the computational burden of this problem. In this optimization formulation, healthy constraints are assumed that apply to the human body, such as avoiding unnatural knee and elbow bends and self intersections. If we use these constraints, the space in which we have to look for possible solutions shrinks considerably. The results show that this algorithm outperforms several other state of the art techniques by a significant margin. It is an auspicious opportunity to preserve and recreate a lot of historic events in digital form, maybe even use them in computer games. And I'm sure that artists will make great use of such techniques. Really well done. The paper is extremely well written. The mathematics and the optimization formulations are beautiful. It was such a joy to read. Regarding the future, I am pretty sure we are soon going to see some pose and skeleton transfer applications via machine learning. The input would be a real world video with a person doing something and we could essentially edit the video and bend these characters to our will. There are some exploratory works in this area already. The Disney guys, for instance, are doing quite well. There will be lots of fun to be had indeed. Also make sure to check out the YouTube channel of Welch Labs, who has a great introductory series for neural networks, which is, in my opinion, second to none. He also has a new series called Learning to see where he codes up a machine learning technique for a computer vision application. It is about counting the number of fingers on an image. Really cool, right? The quality of these videos is through the roof. The link for both of these series are available in the description box. Make sure to check them out. Thanks for watching and for your generous support and I'll see you next time.
Sam.
