This transcript describes a SIGGRAPH presentation of a learning-based system for rapid, mass-scale photorealistic material synthesis. The pipeline learns user preferences from a modest number of high-scoring samples, generates many recommended materials from the learned distributions, and uses a convolutional neural network to predict rendered images in real time (under 3 ms per image). High-dimensional shader parameters are embedded into an intuitive 2D latent space for interactive fine-tuning; Gaussian process regression is applied to produce preference maps that, combined with similarity maps, guide users toward desirable variants. The system supports extensions such as procedural albedos and displacements and is demonstrated on scene population and material refinement examples.
Chapters
Creating high quality photorealistic materials for light transport simulations typically includes direct hands on interaction with a principled shader. This means that the user has to tweak a large number of material properties by hand and has to wait for a new image of it to be rendered after each interaction. This requires a fair bit of expertise and the best setups are often obtained through a lengthy trial and error process. To enhance this workflow, we present a learning based system for rapid mass scale material synthesis. First, the user is presented with a gallery of materials and the assigned scores are shown in the upper left. Here we learn the concept of glassy and transparent materials. By learning on only a few tens of high scoring samples, our system is able to recommend many new materials from the learned distributions. The learning step typically takes a few seconds where the recommendations take negligible time and can be done on a mass scale. Then these recommendations can be used to populate a scene with materials. Typically, each recommendation takes 40 to 60 seconds to render with global illumination, which is clearly unacceptable for real world workflows even for mid sized galleries. In the next step we propose a convolutional neural network that is able to predict images of these materials that are close to the ones generated via global illumination and takes less than 3 milliseconds per image. Sometimes a recommended material is close to the one envisioned by the user but requires a bit of fine tuning. To this end, we embed our high dimensional shader descriptors into an intuitive 2D latent space where exploration and adjustments can take place without any domain expertise. However, this isn't very useful without additional information because the user does not know which regions offer useful material models that are in line with their scores. One of our key observations is that this latent space technique can be combined with Gaussian process regression to provide an intuitive color coding of the expected preferences to help highlighting the regions that may be of interest. Furthermore, our convolutional neural network can also provide real time predictions of these images. These predictions are close to indistinguishable from the real rendered images and are generated in real time. Beyond the preference map. This neural network also opens up the possibility of visualizing the expected similarity of these new materials to the one we seek to fine tune. By combining the preference and similarity maps, we obtain a color coding that guides the user in this latent space towards materials that are both similar and have a high expected score. To accentuate the utility of our real time variant generation technique, we show a practical case where one of the grape materials is almost done, but requires a slight reduction in vividity. This adjustment doesn't require any domain expertise or direct interaction with the material modeling system and can be done in real time. In this example, we learn the concept of translucent materials from only a handful of high scoring samples and generate a large amount of recommendations from the learned distribution. These recommendations can then be used to populate a scene with relevant materials. Here we show the preference and similarity maps of the learned translucent material space and explore possible variants of an input material. These recommendations can be used for mass scale material synthesis and the amount of variation can be tweaked to suit the user's artistic vision. After assigning the appropriate materials, displacements and other advanced effects can be easily added to these materials. We have also experimented with an extended, more expressive version of our shader that also includes procedural textured albedos and displacements. The following scenes were populated using the material learning and recommendation and latent space embedding steps. We have proposed a system for mass scale material synthesis that is able to rapidly recommend a broad range of new material models after learning the user preferences from a modest number of samples. Beyond this pipeline, we also explored powerful combinations of the three use learning algorithms, thereby opening up the possibility of real time photorealistic material visualization, exploration and fine tuning in a 2D latent space. We believe this feature set offers a useful solution for rapid mass scale material synthesis for novice and expert users alike and hope to see more exploratory works combining the advantages of of multiple state of the art learning algorithms in the future.
