Károly Zsolnai-Fehér explains how a neural network reconstructs a full 3D face from a single photo. The network learns from paired 2D images and 3D geometry, then outputs a voxel model. It handles different poses and expressions well, though it struggles with unseen expressions and flickers on video.
Dear fellow scholars, this is two minute papers with now that facial recognition is becoming more and more of a hot topic, let's talk a bit about 3D face reconstruction. This is a problem where we have a 2D input photograph or a video of a person, and the goal is to create a piece of 3D geometry from it. To accomplish this, previous works often required a combination of proper alignment of the face, the multiple photographs, and dense correspondences, which is a fancy name for additional data that identifies the same regions across these photographs. But this new formulation is the holy grail of all possible versions of this problem because it requires Nothing else but one 2D photograph. The weapon of choice for this work was a convolutional neural network, and the dataset the algorithm was trained on couldn't be simpler. It was given a large database of 2D input image and 3D output geometry pairs. This means that the neural network can look at a lot of these pairs and learn how these input photographs are mapped to 3D geometry. And as you can see, the results are absolutely insane, especially given the fact that it works for arbitrary phase positions and many different expressions, and even with occlusions. However, this is not your classical convolutional neural network because as we mentioned, the input is 2D and the output is 3D. So the question immediately arises what kind of data structure should be used for the output. The authors went for a 3D voxel array, which is essentially a cube in which we build up the face from small identical Lego pieces. This representation is similar to the terrain in the game Minecraft, only the resolution of these blocks is finer. The process of guessing how these voxel arrays should look based on the input photograph is referred to in the research community as volumetric regression. This is what this work is about. And now comes the best part. An online demo is also available where we can either try some prepared images or we can also upload our own. So while I run my own experiments, don't leave me out of the good stuff and make sure you post your results in the comment section. The source code is also available for you fellow tinkerers out there, the limitations of this technique includes the inability of detecting expressions that are very far away from the ones that seen in the training set. And as you can see in the videos, temporal coherence could also use some help. This means that if we have video input, the reconstruction has some tiny differences in each frame. Maybe a recurrent neural network, like some variant of long short term memory could address this in the near future. However, those are trickier and more resource intensive to train properly. Very excited to see how these solutions evolve. And of course, Two Minute Papers is going to be here for you to talk about some amazing upcoming works. Thanks for watching and for your generous support and I'll see you next time.
