Separable Subsurface Scattering - Unofficial talk by Károly Zsolnai

Friendly greetings to everyone. My name is Karol Zsolnai and I promise to you that the pronunciation of my name is going to be the most complicated thing in this unofficial talk. This piece of work is a collaboration between Activision Blizzard, the University of Zaragoza and the Technical University of Vienna. The quest here was to render images with really high quality subsurface scattering in real time on commodity hardware. Normally, to render photorealistic images, we populate a scene with objects, add a camera and a light source, and start tracing rays from the camera to determine the incoming gradients. Now, even though there is a large volume of research going on on how to do this efficiently, this is still a really time consuming process. What's more, this figure here shows light transport only between surfaces, meaning that we suppose that rays of light propagate only in vacuum. If we extend our program to support participating media, we can render volumetric effects like smoke, haze and many others, and also translucent materials such as skin, plant leaves, marble, wax and so on. However, this extension bumps up the dimensionality of the integral we need to solve, making the process even more time consuming. But the reward for this is immense. Here on the left, you can see how our skin would look like without subsurface scattering. It is indeed a very important factor in the visual appearance of many translucent materials. It is not a surprise that the motion picture and the game industries are yearning for a real time solution for this. There are fortunately simplified models to render subsurface light transport in optically thick materials. What we do here is take an infinite half space of a chosen translucent material light, an infinitesimally thin pencil beam from above. In normal incidence, this beam will penetrate the surface of the material and will start to attenuate as it becomes more and more submerged into the medium. During this process, these photons undergo many scattering events and eventually exit somewhere away from the origin. Counting up these photons exiting at different distances, we can build a histogram that we call diffusion profile and we'll denote as rd. This is an actual simulated diffusion profile. What it looks like if we look from above. Another important bit of preliminary knowledge is that we can directly use these diffusion profiles by convolving them with an input irradiance map to add subsurface scattering to it as a post processing step. This is how the result looks like after the convolution is applied. Now this is remarkable as we don't have to run a fully ray traced simulation with participating media. However, these signals are stored as images. So normally this means that we compute a 2D convolution between them. Unfortunately, this is very costly, but there are techniques to reduce this problem to several much cheaper 1D convolutions. One example is Dion's excellent technique. He takes into consideration that in a homogeneous and isotropic medium, the diffusion profiles are radially symmetric. Therefore, it is possible to take a 1D slice of this profile as shown below here, and trying to fit it with a sum of Gaussians which are individually also radially symmetric. This means that we can use a cheaper set of 1D convolutions instead of using the 2D profile directly. This is an example input signal and the results with Dion's technique with different number of Gaussians could compared to the true diffusion kernel. It is important to point out that given the mathematical properties of Gaussians, this technique requires one horizontal and one vertical convolution per Gaussian. These are 1D convolutions. This also means that if we'd like to obtain high quality subsurface scattering, we need at least four Gaussians and therefore eight convolutions. This is not suitable for most real time applications. However, it is a really smart idea and hasn't been really improved since 2007. And honestly, when we started this project, we didn't think anyone could realistically come up with something that's better than this. Our quest was nonetheless to obtain high fidelity results with a separable kernel using only two convolutions, which is marked with green up there. Visualizing the SVD of the diffusion profile, it is clear that the signal is non separable. It is not possible to write this 2D function as a mere product of 1D functions. However, this semilock plot tells us that the higher rank singular values decay rapidly, meaning that most of the information here is not random. It has a lot of structure. Therefore, a rank 1 approximation sounds like a good starting point. The plan was to treat the diffusion profile here on the right as a matrix for which we compute the svd. Here you can see the one singular value that we're taking and the corresponding left and right singular vectors that we here denote by a. We then compute one horizontal and one vertical convolution using these singular vectors to reconstruct the diffusion kernel and obtain the output. This is the Input and the Rank 1 SVD reconstruction. This would be the ground truth. And now we can see that the separable SVD approximation is indeed looking very grim. There is a world of a difference between the two, so wow, this is surprising, especially considering the fact that the Eckhart Young theorem teaches us that the SVD is the best reconstruction in terms of the Frobenius norm, which corresponds here to the RMS error. This is the absolute best reconstruction we can obtain with respect to the RMS error. Very disappointing. Here is Deon's algorithm, the one with the 1D slice and fitting with one Gaussian. This means the same amount of convolutions and hence the same execution time as the rank 1 SVD and the ground truth. This is how the SVD looks like on a real world scene compared to using the true kernel. Looking at it again in our disappointment, we notice that the SVD yields an overall darker image. Therefore the reconstruction is not energy conserving. A new idea came up. Maybe we would try this before putting the project on ice and calling it a day. What if we would solve a minimization problem where the reconstructed kernel would still be as close as possible to the diffusion profile, but would also be energy conservant? This should definitely be a viable option and the results are dreadful. Horrendous. I just. I just don't know what to say. Look at the nose. Somehow, as if the input irradiance signal showed up as a really nasty ringing artifact. And we had the same around the ear. We visualized the actual kernel on a disk of light to see what went wrong here. And yes, we see that it is indeed dreadful. Nothing like the simulated diffusion kernel. But we hadn't the slightest idea why this happened. Visualizing the kernel itself in a simple 1D plot and staring at it for a while, it looks like a great separable approximation with respect to the RMS error. Most of the energy of the signal is close to the origin, and the optimizer tries to reconstruct these details as closely as possible. Please note that the kernel plots are deceiving. These signals indeed have the same amount of energy, but the tail of the fit is extending really far away and this makes up for the seemingly less energy of the signal. This very delicate thing took a week of my life and I kind of want it back. So what if we would minimize not only the RMS error by itself, which forces the optimizer to concentrate the energy to the origin of the signal where the energy spike is. But we would add a guide function which behaves like an envelope that tells the optimizer to possibly reconstruct regions not close to the origin, but focus a bit more on far in scattering. This is the fit we originally had, and this is a very simple distance weighted guide function I had in mind. Imagine that we now have a more general model for which we used k0, a constant envelope. To obtain these horrendous results, I will now tell the Optimizer to use K1, which means that we would give higher weight to the regions further away from the origin. This is what we obtained. Very intuitive. We have a signal with the same amount of energy as if we pushed it from the top to neglect the origin and add this energy to the tail of the signal to focus on the reconstruction of far ring scattering. And now we can even go K2, which is essentially squishing the signal a bit more to emphasize farrange scattering at the price of neglecting sharp close range details. Back to the original fit. Let's weight it by distance a bit by plugging K1 in the optimizer. Almost there. OK, let's go K2 a bit more emphasis on farrain scattering. Now this looks remarkably close to the ground truth. This is the journey behind the guided optimization technique that is separable, requires only two convolutions and is one of the techniques we propose for applications with strict real time time constraints. We also propose another technique which we have mathematically derived and for which I admit not having an intuitive story. So, before I dare show you the next slide, take a big breath and let's go. So sorry. This is how it looks like and what is remarkable about it that this follows a completely different paradigm. What we are aiming for here is not to make our kernel close to the original diffusion kernel, but trying to make the result of the convolution the same. It is almost like minimizing the L2 distance of the resulting convolved images, not the kernels themselves. Fitting for images, not kernels. This is impossible to solve for a general case. So in order to accomplish this one needs to confine the solution to a class of input irradiance signals, input images, where it would work well. In our derivation, we plugged in 1D signals as inputs. This means that the technique should behave analytically on signals like this. And the most remarkable thing, that this is a mere rank 1 separable approximation that is really analytic for this class of signals. This means that it mimics the effect of the true kernel perfectly. Let's take a look at this on a practical case. Of course not all signals are 1D signals. This is our result with the analytic pre integrated kernel and the ground truth very close to being indistinguishable. Furthermore, it is really simple to implement and it has a closed form solution that does not require any kind of optimization procedure. One more interesting detail for the more curious minds this technique is analytic for a greater class than only 1D signals, a class that we call additively separable signals. So what about artistic editing? The pre integrated technique is great, but it does not offer any kind of artistic control over the output. The guided approximation requires optimization, but in return it also offers some degree of artistic freedom on how the desired output should look like. We also have this technique which is simply using two separable Gaussians of different variance values, one each to provide perfect artistic freedom in adjusting the magnitudes of close and far in scattering. Note that these two Gaussians are not the same as in Deon's approach with the two Gaussians as we don't use the radially symmetric 1D signal directly. A real world example this is the input irradiance, heavily exaggerated foreign scattering, heavily exaggerated close range scattering, and a more conservative really good looking mixture of the two. Wrapping it up the SVD is great for applications that can afford higher rank reconstructions. The kernel pre integration is a simple technique that is analytic for additively separable signals. Guided optimization, a more general version of the pre integration that can be conveniently tuned with one parameter, manual approximation. Many degrees of freedom for artists, while it has a quite reasonable accuracy that is comparable to 4 Gaussians with previous techniques, different techniques with different levels of scientific rigor and different target audiences ranging from scientists to artists working in the industry. Now, even though we used examples with skin to demonstrate our techniques, it is important to point out that they work for a variety of translucent media such as plants, marble, fruits in the still life and milk. The most important take home message from this project, and at least for me, is that it's entirely possible to do academic research together with companies and create results that can make it to multi million dollar computer games, but also having proven results that are useful for the scientific community. Thank you.