Károly Zsolnai-Fehér shows how AI learns to synthesize material models and render photorealistic images in 4 milliseconds instead of 40-60 seconds. A differentiable rendering technique optimizes materials, textures, and geometry to match target images by working backwards from the final result.
Questions this video answers
Károly Zsolnai-Fehér
About 4 milliseconds per image
Typically 40 to 60 seconds per change
More than a hundred different materials
Sponsored by Weights & Biases; said to be used by OpenAI, Toyota Research, Stanford, and Berkeley
Chapters
Dear fellow scholars, this is two minute papers with zsolnai. This beautiful scene is from our paper by the name Gaussian Material Synthesis. And it contains more than a hundred different materials, each of which has been learned and synthesized by an AI. None of these daisies and dandelions are alike. Each of them have a different material model. Normally, to obtain results like this, an artist has to engage in a direct interaction with an interface that you see here. This contains a ton of parameters, and to be able to use it properly, the artist needs to have years of experience in photorealistic rendering and material modeling. But unfortunately, the problem gets even worse since a proper light simulation program needs to create an image with the new material parameters. This initially results in a noisy image that typically takes 40 to 60 seconds to clear up. We have to wait out this 40 to 60 seconds for every single parameter change that we make. This would take several hours in practical cases. The goal of this project was to speed up workflows like this by teaching an AI the concept of material models such as metals, minerals, and translucent materials with our technique. First, we show the user a gallery of random materials, who assigns a score to each of them, saying that, I like this one, I didn't like that one, and get an AI to learn our preferences and recommend new materials for us. We also created a neural renderer that replaces the light simulation program and creates a near perfect image of the output in about 4 milliseconds. That's not real time. That's 10 times faster than real time. That is very fast and accurate. However, our neural renderer is limited to this scene that you see here. So the question is, is it possible to create something that is a little more general? Well, let's have a look at this new work that performs something similar that they call differentiable rendering. The problem formulation is the we specify a target image that is either rendered by a computer program or even better, a photo. The input is a pitiful approximation of it. And now hold on to your papers because it progressively changes the input materials, textures, and even the geometry to match this photo. My goodness, even the geometry. This thing is doing three people's jobs. When given a target photo. And you haven't seen the best part here, because there is an animation that shows how the input evolves over time. As we run this technique, as we start out, it almost immediately matches the material properties and the base shape. And after that, it refines the geometry to make sure that the more intricate details are also matched properly. As always, some limitations apply. For instance, area light sources are fine, but it doesn't support point Light sources may show problems in the presence of discontinuities and mirror like materials. I cannot wait to see where this ends up a couple papers down the line and I really hope this thing takes off. In my opinion, this is one of the most refreshing and exciting ideas in photorealistic rendering research as of late. More differentiable rendering papers please. I would like to stress that there are also other works on differentiable rendering. This is not the first one. However, if you have a closer look at the paper in the description, you will see that it does better than previous techniques in this series. I try to make you feel how I feel when I read these papers and I hope I have managed this time, but you be the judge. Please let me know in the comments and if this got you excited to learn more about light transport, I am holding a Master level course on it at the Technical University of Vienna. This course used to take place behind closed doors, but I feel that the teachings shouldn't only be available for the 20 to 30 people who can afford a university education, but they should be available for everyone. So we recorded the entirety of the course and it is now available for everyone free of charge. If you are interested, have a look at the video description to watch them. This episode has been supported by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. It can save you a ton of time and money in these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. In this tutorial they show you how to visualize your machine learning models and how to choose the best one with the help of their tools. Make sure to Visit them through wendb.com papers, wandb. Com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
