A recorded technical talk (June 17, 2020) by Dr. Zsolnai presenting a new paper on material editing. The work frames material synthesis as finding shader parameters that reproduce an edited target image; it uses a neural renderer to approximate the forward render and compares two inverse approaches — optimization-based fitting and learned inversion networks. Each method has weaknesses (optimizers are slow and can get stuck; inversion nets fail for out-of-domain edits), so the paper proposes a hybrid: use inversion networks to initialize an optimizer, producing fast (≈20 s) and robust material recovery that handles specular edits, inpainting, stitching and blended edits. The video notes limitations (canonical scene / carved sphere samples) and points to code, pretrained models, and the paper teaser appearing on the 2020 Computer Graphics Forum cover.
Questions this video answers
June 17, 2020.
Approximately 20 seconds.
They advised the work (acknowledged as advisors).
The 2020 Computer Graphics Forum cover page.
Source code and pretrained networks on the authors' website, released under a permissive license.
Chapters
Dear fellow scholars, this is not 2 minute papers with Dr. Zsolnai due to popular demand, this is a surprise video with the talk of our new paper that we just published. This was the third and last paper in my PhD thesis and hence this is going to be a one off video that is longer and a tiny bit more technical. I am keenly aware of it, but I hope you'll enjoy it. Let me know in the comments when you have finished the video. And worry not all the upcoming videos are going to be in the usual two minute papers format. The paper and the source code are all available in the video description. And now let's dive in. In a previous paper, our goal was to populate this scene with over 100 materials with a learning based technique and create a beautiful planet with rich vegetation. The results looked like this. One of the key elements to accomplish this was to use a neural renderer or or in other words, a decoder network that you see here, which took a material's shader description as an input and predicted its appearance, thereby replacing the renderer we used in the project. It had its own limitations. For instance, it was limited to this scene with a fixed lighting setup and only the material properties were subject to change. But in return it mimicked the global illumination render render rapidly and faithfully. And in this new work, our goal was to take a different vantage point and help artists with general image processing knowledge to perform material synthesis. Now this sounds a little nebulous, so let me explain. One of the key ideas is to achieve this with a system that is meant to take images from its own renderer, like the ones you see here. But of course we produce these ourselves, so obviously we know how to do it, so this is not very useful yet. However, the twist is that we only start out with an image of this source material and then load it into a raster image editing program like Photoshop and edit it to our liking and just pretend that this is achievable with our render. As you see, many of these target images in the middle are results of poorly executed edits. For instance, the stitched specular highlight in the first example isn't very well done, and neither is the background of the gold target image in the middle. However, in the next step, our method proceeds to find a photorealistic material description that when rendered, resembles this target image and works well even in the presence of these poorly executed edits. The whole process executes in 20 seconds. To produce a mathematical formulation for this problem, we we started with this. We have an input image T and edit it to our liking to get the target image T with a tilde. Now we are looking for a shader parameter set X that when rendered with the phi operator, approximates the edited image. The constraint below stipulates that we would remain within the physical boundaries for each parameter. For instance, albedos between 0 and 1, proper indices of refraction, and so on. So how do we deal with phi? We use the previously mentioned neural renderer to implement it. Otherwise this optimization process would take 25 hours. Later we made an equivalent unconstrained reformulation of this problem to be able to accommodate a greater set of optimizers. This all sounds great on paper and works reasonably well for materials that can be exactly matched with this shader or like this one. This optimizer based solution can achieve it reasonably well, but unfortunately for more challenging cases, as you see the target image on the lower right, the optimizer's output leaves much to be desired. Note again that the result on the upper right is achievable with the shader, while the lower right is a challenging imaginary material that we are trying to achieve. The fact that this is quite difficult is not a surprise because we have a nonlinear and non convex optimization problem which is also high dimensional. So this optimization solution is also quite slow, but it can start inching towards the target image. As an alternative solution, we also developed something that we call an inversion network. This addresses the adjoint problem of neural rendering, or in other words, we show it the edited input image and out comes the shader that would produce this image. We have trained nine different neural network architectures for this problem, which sounds great. So how well did it work? Well, we found out that none of them are really satisfactory for more difficult edits. Because all of the target images are far, far outside of the training domain. We just cannot prepare the networks to be able to handle the rich variety of edits that come from the artists. However, some of them are, one could say, almost usable. For instance, number one and five are not complete failures. And note that these solutions are provided instantly. So we have two techniques. None of them are perfect for our a fast and approximate solution with the inversion networks and a slower optimizer that can slowly inch towards the target image. Our key insight here is that we can produce a hybrid method that fuses the two solutions together. The workflow goes as we take an image of the initial source material and edit it to our liking to get this target image. Then we create a coarse prediction with a selection of inversion networks to initialize the optimizer with the prediction of one of these neural networks, preferably a good one. So the optimizer can start out from a reasonable initial guess. So how well does this hybrid method work? I'll show you in a moment. Here we start out with an achievable target image and then try two challenging image editing operations. This image can be reproduced perfectly as long as the inversion process works reliably. Unfortunately, as you see here, this is not the case. In the first row, using the optimizer and the inversion networks separately, we get results that fail to capture the specular highlight properly. In the second row, we have deleted the specular highlight on the target image on the right and replaced it with a completely different one. I like to call this the FrankenBRDF and it would be amazing if we could do this, but unfortunately both the optimizer and the inversion networks flounder. Another thing that would be really nice to do is deleting the specular highlight and filling the image via image inpainting. This kind of works with the optimizer, but you'll see in a moment that it's not nearly as good as it could be. And now, if you look carefully, you see that our hybrid method outperforms both of these techniques. In each of the three cases in the paper, we report results on a dozen more cases as well. We make an even stronger claim in the paper where we say that these results are close to the global optimum. You see the results of this hybrid method if you look at the intersection of NELDERMID and nn, they are highlighted with the red ellipses. The records in the table show the RMS errors and are subject to minimization. With this, you see that this goes neck and neck with the global optimizer, which is highlighted with green. In summary, our technique runs in approximately 20 seconds. Works for specular highlight editing, image blending, stitching, inpainting and more. The proposed method is robust, works even in the presence of poorly edited images, and can be easily deployed in already existing rendering systems and allows for rapid material prototyping for artists working in the industry. It is also independent of the underlying principle shader, so you can also add your own and expect it to work well as long as the neural renderer works reliably. A key limitation of the work is that it only takes images in this canonical scene with a carved sphere material sample. But we conjecture that it can be extended to be more general and propose a way to do it. In the paper, make sure to have a closer look if you are interested, the teaser image of this paper is Showcased in the 2020 Computer Graphics Forum cover page. The whole thing is also quite simple to implement and we provide the source code pre trained networks on our website and all of them are under a permissive license. Thank you so much for watching this and a big thanks to PETA Vonka and Michael Wimmer for advising this work.
