Presentation of a system that lets non-expert users create photorealistic materials from target images by combining optimization and a neural-network inversion encoder. The hybrid method initializes an optimizer with the network prediction to produce high-quality materials quickly (≈30s for optimization, milliseconds for encoder), enabling rapid artistic iteration and sequence prediction with a 1–2s per image budget.
Creating photorealistic materials for light transport algorithms requires carefully fine tuning a set of material properties to achieve a desired artistic effect. This is a lengthy process that involves a trained artist with specialized knowledge. In this work, we propose a system that only requires basic image processing knowledge and enables users without photorealistic rendering experience to create high quality materials. This is highly desirable as human thinking is inherently visual and not based on physically based material parameters. In our proposed workflow, all the user needs to do is apply a few intuitive transforms to a source image and in the next step our technique produces the closest photorealistic material that approximates this target image. One of our key observations is that even though this process target image is often not physically achievable, in many cases a photorealistic material model can be found that closely matches this image. Our method generates results in less than 30 seconds and works in the presence of poorly edited target images like the discoloration of the pedestal or the background of the gold material here. This technique is especially useful early in the material design process where the artist seeks to rapidly iterate over a variety of possible artistic effects. We also propose an extension to predict image sequences with a tide budget of 1 to 2 seconds per image. To achieve this, we propose a simple optimization formulation that is able to produce accurate solutions but takes relatively long due to the lack of a useful initial guess. Our other main observation is that an approximate solution can also be achieved without an optimization step by implementing a simple encoder neural network. The main advantage of this method is that it produces a solution within a few milliseconds with the drawback that the provided solution is only approximate. We refer to this as the inversion technique. Both of these solutions suffer from drawbacks. The optimization approach provides results that resemble the target image but is impracticable due to the fact that it requires too many function evaluations and and get stuck in local minima, whereas the inversion technique rapidly produces a solution but is more approximate in nature. We show that the best aspects of these two solutions can be fused together into a hybrid method that initializes our optimizer with the prediction of the neural network. This hybrid method opens up the possibility of creating novel materials by stitching together the best aspects of two or more materials, deleting unwanted features through image inpainting, contrast enhancement, or even fusing together tool materials. These synthesized materials can also be easily inserted into already existing scenes by the user. In this scene, we made a material mixture to achieve a richer nebula effect inside the glass. We also show in the paper that this hybrid method not only gives a head start to the optimizer by endowing it with a useful initial guess, but provides strictly higher quality outputs than any of the two previous solutions on all of our test cases. Furthermore, if at most a handful of materials are sought, the total modeling times reveal that our technique compares favorably to previous work on mass scale material synthesis. We believe this method will offer an appealing entry point for novices into the world of photorealistic material modeling. Thank you for your attention.
