Differentiable Material Synthesis Is Amazing! ☀️

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. I am a light transport researcher by trade and I am very happy today because we have an absolutely amazing light transport paper we are going to enjoy today. As many of you know, we write these programs that you can run on your computer to simulate millions and millions of light rays and calculate how they get absorbed or scattered off of our objects in a virtual scene. Initially, we start out with a really noisy image, and as we add more rays, the image gets clearer and clearer over time. We can also simulate sophisticated material models in these programs. A modern way of doing that is through using these material nodes. With these, we can conjure up a ton of different material models and change their physical properties to our liking. As you see, they are very expressive indeed. However, the more nodes we use, the less clear it becomes how they interact with each other. And as you see, every time we change something, we have to wait until a new image is rendered. That is very time consuming. And more importantly, we have to have some material modeling expertise to use this. This concept is very powerful. For instance, I think if you watch the Perceptilabs sponsorships part at the end of this video, you will be very surprised to see that they also use node groups. But with theirs, you don't build material models. You can build machine learning models. What would also be really cool if we could just give the machine a photo and it would figure out how to set up these nodes so it looks exactly like the material in the photo. So is that possible or is that science fiction? Well, have a look at our paper called Photorealistic Material Editing. With this technique, we can easily create these beautiful material models in a matter of seconds. Even if we don't know a thing about light transport simulations, it does something that is similar to what many call differentiable rendering. Here is the workflow. We give it a bunch of images like these which were created on this particular test scene, and it guesses what parameters to use to get these material models. Now, of course, this doesn't make any sense whatsoever because we have produced these images ourselves, so we know exactly what parameters to use to produce this. In other words, this thing seems useless. And now comes the magic part, because we don't use these images. No, no. We load them into Photoshop and edit them to our liking and just pretend that these images were created with the light simulation program. This means that we can create a lot of quickly and really poorly executed edits. For instance, the stitched specular highlight in the first example isn't Very well done. And neither is the background of the gold target image in the middle. However, the key observation is that we have built a mathematical framework which makes this pretending really work. Look, in the next step, our method proceeds to find a photorealistic material description that when rendered, resembles this target image and works well even in the presence of these poorly executed edits. So these materials are completely made up in Photoshop and it turns out we can create photorealistic materials through these node graphs that look almost exactly the same. Quite remarkable. And the whole process executes in 20 seconds. If you are one of the more curious fellow scholars out there, this paper and its source code are available in the video description. Now, this differentiable thing has a lot of steam. For instance, there are more works on differentiable rendering. In this other work we can take a photo of a scene and the learning based method turns the knobs until it finds a digital object that matches its geometry and material properties. This was a stunning piece of work for from Venta Jakob and his group, of course, who else? They are some of the best in the business. And we don't even need to be in the area of light transport simulations to enjoy the benefits of differentiable formulations. For instance, this is differentiable physics. So what is that? Imagine that we have this billiard game where we would like to hit the white ball with just the right amount of force and from the right direction, such that the blue ball ends up close to the black spot. Well, this example shows that this is unlikely to happen by chance and we have to engage in a fair amount of trial and error to make this happen. What this differentiable programming system does for us is that we can specify an end state, which is the blue ball on the black dot, and it is able to compute the required forces and angles to make this happen. Very close. So after you look here, maybe you can now guess what's next for this differentiable technique. It starts out with a piece of simulated ink with a checkerboard pattern and it exerts just the appropriate forces so that it forms exactly the yin yang symbol shortly after. And now that we understand what differentiable techniques are capable of, we are ready to proceed to today's paper. This is a proper, fully differentiable material capture technique for real photographs. All this needs is one flash photograph of a real world material. We have those around us in abundance. And similarly to our previous method, it sets up the material nodes for it. That is a good thing because I don't know about you, but I do not want to touch this mess at all. Luckily, we don't have to look. The left is the target photo and the right is the initial guess of the algorithm. That is not bad, but also not very close. And now hold on to your papers and just look at how it proceeds to refine this material until it closely matches the target. And with that, we have a digital representation of these materials. We can now easily build a library of these materials and assign them to objects in our scene. And then we run the light simulation program. And here we go. Beautiful. At this point, if we feel adventurous, we can adjust small things in the material graphs to create a digital material that is more in line with our artistic vision. That is great because it is much easier to adjust an already existing material model than creating one from scratch. So what are the key differences between our work from last year and this new paper? Our work made a rough initial guess and optimized the parameters afterwards. It was also chock full of neural networks. It also created materials from a sample, but that sample was not a photograph, but a photoshopped image. That is really cool. However, this new method takes an almost arbitrary photo. Many of these we can take ourselves or even get them from the Internet. Therefore, this new method is more general. It also supports 131 different material node types, which is insanity huge. Congratulations to the authors. If I would be an artist, I would want to work with this right about now. What a time to be alive. So there you go. This was quite a ride and I hope you enjoyed it just half as much as I did. And if you enjoyed it at least as much as I did, and you feel a little stranded at home and are thinking that this light transport thing is pretty cool and you would like to learn more about it. I held a master level course on this topic at the Technical University of Vienna. Since I was always teaching it to a handful of motivated students. I thought that the teachings shouldn't only be available for the privileged few who can afford a college education, but. But the teachings should be available for everyone. Free education for everyone. That's what I want. So the course is available free of charge for everyone. No strings attached. Make sure to click the link in the video description to get started. We write a full light simulation program from scratch there and learn about physics, the world around us, and more. Perceptilabs is a Visual API for TensorFlow, carefully designed to make machine learning as intuitive as possible. This gives you a faster way to build out models with more transparency into how your model is architected, how it performs, and how to debug it. Look, it lets you toggle between the visual modeler and the code editor. It even generates visualizations for all the model variables and gives you recommendations both during modeling and training, and does all this automatically. I only wish I had a tool like this when I was working on my neural networks during my PhD years. Visit perceptilabs.com papers to easily install the free local version of their system today. Our thanks to Perceptilabs for their support and for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time.