Dr. Károly Zsolnai-Fehér explains how neural networks can compress material measurements into tiny models that render cloth, rocks, and shells with fine detail. A four-layer network learns from reflectance data and stores what used to take massive files in something cheap to evaluate.
Dr. Károly Zsolnai-Fehér
July 21, 2021
Four layers
Weights & Biases
Over 200 companies and research institutions
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This image is the result of a light simulation program created by research scientists. It looks absolutely beautiful. But the light simulation algorithm is only part of the recipe here. To create something like this, we also need a good artist who can produce high quality geometry lighting and of course, good lifelike material models. For instance, without the materials part, we would see something like this. Not very exciting, right? Previously we introduced a technique that learns our preferences and helps filling these scenes with materials. This work can also generate variants of the same materials as well. In a later technique, we could even take a sample image, completely destroy it in Photoshop and our neural networks would find a photorealistic material that matches these crazy ideas. Links to both of these works are available in the video description. And to improve these digital materials, this new paper introduces something that the authors call a multi resolution neural material representation. What is that? Well, it is something that is able to put amazingly complex material models in our light transport programs. And not only that, but oh my, look at that. We can even zoom in so far that we see the snagged threads. That is the magic of the multi resolution part of the technique. The neural part means that the technique looks at lots of measured material, reflectance data. This is what describes a real world material and compresses this description down into a representation that is manageable. Ok, why? Well look, here is a reference material. You see, these are absolutely beautiful, no doubt, but are often prohibitively expensive to store directly. This new method introduces these neural materials to approximate the real world materials, but in a way that is super cheap to compute and store. So our first question, how do these neural materials compare to these real reference materials? What do you think? How much worse a quality do we have to expect to be able to use these in our rendering systems? Well, you tell me, because you're already looking at the new technique right now. I quickly switched from the reference to the result with the new method already. How cool is that? Look, this was the expensive reference material and this is the fast neural material counterpart for it. So how hard is this to pull off? Well, let's look at some results side by side. Here is the reference and here are two techniques from one and two years ago that try to approximate it. And you see that if we zoom in real close, these fine details are gone. Do we have to live with that? Or maybe can the new method do better? Hold on to your papers and let's see. Wow. While it is not 100% perfect, there is absolutely no contest compared to the previous methods. It outperforms them handily in every single case of these complex materials I came across. And when I say complex materials, I really mean it. Look at how beautifully it captures not only the texture of this piece of embroidery, but when we move the light source around. Oh wow. Look at the area here around the vertical black stripe and how its specular reflections change with the lighting. And note that none of these are real images. All of them come from a computer program. This is truly something else. Loving it. So if it really works so well, where is the catch? Does it only work on cloth like materials? No, no, not in the slightest. It also works really well on rocks, insulation, foam, even turtle shells and a variety of other materials. The paper contains a ton more examples than we can showcase here, so make sure to have a look in the video description. I guess this means that it requires a huge and expensive neural network to pull off, right? Well, let's have a look. Whoa. Now that's something it does not require a deep and heavy duty neural network. Just four layers are enough. And this, by today's standard, is a lightweight network that can take these expensive reference materials and compress them down in a matter of milliseconds. And they almost look the same materials in our computer simulations straight from reality. Yes please. So from now on we will get cheaper and better material models for animation, movies, computer games and visualization applications. Sign me up right now. What a time to be alive. This episode has been supported by Weights and Biases. In this post they show you how to use their tool to analyze chest x rays and more. If you work with learning algorithms on a regular basis, make sure to check out weights and biases. Their system is designed to help you organize your experiments and it is so good it could shave off weeks or even months of work from your projects and is completely free for all individuals, academics and and open source projects. This really is as good as it gets and it is hardly a surprise that they are now used by over 200 companies and research institutions. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
