A neural network reconstructs full 3D face models from a single photo, generating both geometry and texture. The technique splits texture into low and high-frequency maps, using convolutional networks to fill in missing fine details. A user study validated the results against existing methods.
Dear fellow scholars, this is Two Minute Papers with How cool would it be to be able to place a character representing us in a digital film or a computer game? Of course, it would clearly be an extremely laborious task to digitize the 3D geometry and the albedo map of our face. This albedo map means a texture, a colored pattern that describes how our skin reflects and absorbs light. Capturing such a representation is clearly a very lengthy and expensive process. So get this completely crazy idea. This technique creates this full digital representation of any face from no more than one simple photograph. We can even get historical figures in our digital universe. All we need is one photograph of them. And now feast your eyes on these incredible results after taking a photograph. This technique creates two of these albedo maps. One is a complete low frequency map which records the entirety of the face, but only contains the rough details. The other albedo map contains finer details, but in return is incomplete. Do you remember the texture synthesis methods that we discussed earlier in the series? The input was a tiny patch of image with a repetitive structure, and after learning the statistical properties of these structures, it was possible to continue continue them indefinitely. The key insight is that we can also do something akin to that here as well. We take this incomplete albedo map and try to synthesize the missing details. Pretty amazing idea indeed. The authors of the paper invoke a classical learning algorithm, a convolutional neural network to accomplish that. The deeper the neural network we use, the more high frequency details appear on the outputs, or in other words, the crisper the image we get. In the paper you will find a detailed description of their crowdsource user study that was used to validate this technique, including the user interface and the questions being asked. There are also some comparisons against PatchMatch, one of the landmark techniques for texture synthesis that we have also talked about in an earlier episode. It's pretty amazing to see this two minute paper's knowledge base grow and get more and more intertwined. I hope you are enjoying the process as much as I do. Also, due to popular request, the Two Minute Papers T shirts are now available. This time we are using a different service for printing these shirts. Please give us some feedback on how you liked it. I've put my email address in the video description. If you attach a photo of yourself wearing some cool 2 Minute Papers merch, we'll be even more delighted. Just open twominutepapers.com and you'll immediately have access to it. This link will bring you to the service that ships to the U.S. the link for shipping outside the U.S. is available in the video description. Thanks for watching and for your generous support, and I'll see you next time.
