A neural network reconstructs physical material properties from a single photo instead of two, using local and global components to capture texture and lighting. It works better than previous methods but struggles with highly scattering materials like fabric and skin.
Dear fellow scholars, this is Two Minute Papers with. With this technique we can take a photograph of a desired material and use a neural network to create a digital material model that matches it. That we can use in computer games and animation movies. We can import real world materials in our virtual worlds, if you will. Typically, to do this, an earlier work required two photographs, one with flash and and one without. To get enough information about the reflectance properties of the material. Then a follow up AI paper was able to do this from only one image. It doesn't even need to turn the camera around the material to see how it handles reflections, but can learn all of these material properties from only one image. Isn't that miraculous? We talked about this work in more detail in two minute papers episode 88. That was about two years ago. I put a link to it in the video description. Let's look at some results with this new technique. Here you see the photos of the input materials and on the right, the reconstructed material. Please note that this reconstruction means that the neural network predicts the physical properties of the material which are then passed to a light simulation program. So on the left you see reality and on the right the prediction plus simulation results. Under a moving point light. It works like magic. Love it. As you see in the comparisons here, it produces results that are closer to the ground truth than previous techniques. This method is designed in a way that enables us to create a larger training set for more accurate results. As you know, with learning algorithms we are always looking for more and more training data. Also, it uses two neural networks instead of one, where one of them looks at local nearby features in the input and the other one runs in parallel and ensures that the material that is created is also globally correct. Note that there are some highly scattering materials that this method doesn't support. For instance fabrics or human skin. But since producing these materials in a digital world takes quite a bit of time and expertise, this will be a godsend for the video games and animation movies of the future. Thanks for watching and for your generous support and I'll see you next time.
