Károly Zsolnai-Fehér explains how researchers extract 3D shape and material properties from video footage. The challenge: figuring out lighting, geometry, and materials all at once, then refining iteratively until the simulation matches the original footage photorealistically.
Dear fellow scholars, this is two minute papers with Imagine the we put an object on a robot arm and the input is a recorded video of it and the output would be the digital geometry and the material model for this object. This geometry and material model we can plug into a photorealistic light simulation program to have a digital copy of our real world object. First, we have to be wary of the fact that normally solving such a problem sounds completely hopeless. We have three variables that we have to take into consideration. The lighting in the room, the geometry of the object, and the material properties of the object. If any two of the three variables is known, the problem is relatively simple and there are already existing works to address these combinations. For instance, if we create a studio lighting setup and know the geometry of the object as well, it is not that difficult to capture the material properties. Also, if the material properties and lighting is known, there are methods to extract the geometry of the objects. However, this is a way more difficult formulation of the problem because out of the three variables not two, not one, but zero are known. We don't have control over the lighting. The material properties can be arbitrary and the geometry can also be anything. Several very sensible assumptions are being made, such as that our camera has to be stationary and the rotation directions of the object should be known, and some more of these are discussed in the paper in detail. All of them are quite sensible and they don't feel limiting. The algorithm works the following in the first step we estimate the lighting and leaning on this estimation we build a rough initial surface model and in the second step using this surface model we see a bit clearer and therefore we can refine our initial guess for the lighting and the material model. However, now that we know the lighting a bit better, we can again get back to the surface reconstruction and improve our solution there. This entire process happens iteratively, which means that first we obtain a very rough initial guess for the surface and and we constantly refine this piece of surface to get closer and closer to the final solution. And now feast your eyes on these incredible results and marvel at the fact that we know next to nothing about the input and the geometry and material properties almost magically appear on the screen. This is an amazing piece of work and lots and lots of details and results are discussed in the paper, which is quite well written and was a joy to read. Make sure to have a look at it. The link is available in the video description. Also, thanks for all the kind comments. I've experienced a recent influx of emails from people all around the world expressing how they are enjoying the series and many of them telling their personal stories and their relation to science and how these new inventions are being talked about over dinner with the family and relatives. It has been such a delight to read these messages. Thank you and also thanks for watching and for your generous support. See you next time.
