Ray Tracing: How NVIDIA Solved the Impossible!

Dear fellow scholars, this is two minute papers with Dr. Zsolnai, or not quite to be more exact, I have had the honor to hold a talk here at gtc. And today we are going to marvel at a seemingly impossible problem and four miracle papers from scientists at Nvidia. Why these four? Could these papers solve the impossible? Well, we shall see together in a moment. If we wish to create a truly gorgeous photorealistic scene in computer graphics, we usually reach out to a light transport simulation algorithm. And then this happens. Oh yes, concept number one, noise. This is not photorealistic at all. Not yet anyway. Why is that? Well, during this process, we have to shoot millions and millions of light rays into the scene to estimate how much light is bouncing around. And before we have simulated enough rays, the inaccuracies in our estimations show up as noise in these images. This clears up over time, but it may take from minutes to days for this to happen, even for a smaller scene. For instance, this one took us three full weeks to finish. Three weeks? Weeks? Yes. Really? I am not kidding. Ouch. Solving this problem in real time seems absolutely impossible. Which has been the consensus in the light transport research community for a long while. So much so that at the most prestigious computer graphics conference, Siggraph, there was even a course by the name Ray Tracing is the future and will ever be. This was a bit of a wordplay, yes, but I hope you now have a feel of how impossible this problem feels. And when I was starting out as a first year PhD student, I was wondering whether real time light transport will be a possibility within my lifetime. It was such an outrageous idea, I usually avoided even bringing up the question in conversation. And boy, if only I knew what we are going to be talking about today. Wow. So we are still at the point where these images take from hours to weeks to finish. And now I have good news and bad news. Let's go with the good news first. If you overhear some light transport researchers talking, this is why you hear the phrase important sampling a great deal. This means to choose where to shoot these rays in the scene. For instance, you see one of those smart algorithms here called Metropolis Light Transport. This is one of my favorites. It typically allocates these rays much smarter than previous techniques, especially on difficult scenes. But let's go even smarter. This is my other favorite, Vence Jakob's manifold exploration paper at work here. This algorithm is absolutely incredible. And the way it develops an image over time is one of the most beautiful sights in all light transport research. So if we understand correctly, the more Complex these algorithms are, the smarter they can get. However, at the same time, due to their complexity, they cannot be implemented so well that on the graphics card. That is a big bummer. So what do we do? Do we use a simpler algorithm and take advantage of the ever improving graphics cards in our machines or write something smarter and miss out on all of that? So now I can't believe I am saying this, but let's see how Nvidia solved the impossible through four amazing papers. And that is how they created real time algorithms for light transport. Voxel cone tracing. Oh my. This is an iconic paper that was one of the first signs of something bigger to come. Now hold on to your papers and look at this. Oh my goodness. That is a beautiful real time light simulation program. And it gets better because this one paper that is from 2011, a more than 10 year old paper, and it could do all this. Wow. How is that even possible? We just discussed that we'd be lucky to have this in our lifetimes. And it seems that it was already here 10 years ago. So what is going on here? Is light transport suddenly solved? Well, not quite. This solves not the full light transport simulation problem, but it makes it a little simpler. How? Well, it takes two big shortcuts. Shortcut number one, it subdivides the space into voxels, small little boxes, and it runs the light simulation program on this reduced representation. Shortcut number two, it only computes two bounces for each light ray for the illumination. That is pretty good, but not nearly as great as a full solution with potentially infinitely many bounces. It also uses tons of memory, so plenty of things to improve here. But my goodness, if this was not a quantum leap in light transport simulation, I really don't know what is. This really shows that scientists at Nvidia are not afraid of completely rethinking existing systems to make them better. And boy, isn't this a marvelous example of that. And remember all this in 2011, 2011, more than 10 years ago. Absolutely mind blowing. And one more thing. This is the culmination of software and hardware working together, designing them for each other. This would not have been possible without it. But once again, this is not the full light transport. So can we be a little more ambitious and hope for a real time solution for the full light transport problem? Well, let's have a look together and find out. And here is where paper number two comes to the rescue. In this newer work of theirs, they presented an amusement park scene that contains a total of over 20 million triangles. And it truly is a sight to behold. Let's see. And. Oh my goodness. This does not take from minutes to days to compute. Each of these images were produced in a matter of milliseconds. Wow. And it gets better. It can also render this scene with 3.4 million light sources. And this method can render not just an image, but an animation of it interactively. What's more, the more detailed comparisons in the paper reveal that this method is ten to a hundred times faster than previous techniques. And it also maps really well on onto our graphics cards. Ok, but what is behind all this wizardry? How is this even possible? Well, the magic behind all this is a smarter allocation of these ray samples that we have to shoot into the scene. For instance, this technique does not forget what we did just a moment ago when we move the camera a little to advance to the next image. Thus, lots of information that is otherwise thrown away can now be reused as we advance the animation. Now, note that there are so many papers out there on how to allocate these rays properly. This field is so mature. It truly is a challenge to create something that is just a few percentage points better than previous techniques. It is very hard to make even the tiniest difference. And to be able to create something that is 10 to 100 times better in this environment, that is insanity. And this proper ray allocation has one more advantage. What is that? Well, have a look at this. Imagine that you are a good painter and you are given this image. Now your task is to finish it. Do you know what this depicts? Hmm, maybe. But knowing all the details of this image is out of question. Now look, we don't have to live with these noisy images. We have denoising algorithms tailored for light simulations. This one does some serious legwork with this noisy input. But even this one cannot possibly know exactly what is going on because there is so much information missing from the noisy input. And now, if you have been holding onto your paper so far, squeeze that paper because look, this technique can produce this image in the same amount of time. Now we're talking. Now let's give this to the denoising algorithm. And yes, we get a much sharper, more detailed output. Actually, let's compare it to the clean reference image. Yes, yes, yes. This is much closer. This really blows my mind. We are now one step closer to proper interactive light transport. Now, note that I use the word interactively twice here. I did not say real time. And that is not by mistake. These techniques are absolutely fantastic. One of the bigger leaps in light transport research, but they still cost A little more than what production systems can shoulder. They are not quite real time yet and I hope you know what's coming. Oh yes, paper number three. Check this out. This is their more recent result on the Paris Opera House scene, which is quite detailed. There is a ton going on here and you are all experienced fellow scholars now. So when you see them flicking between the raw noisy and the denoised results, you know exactly what is going on. And hold on to your papers because all this takes about 12 milliseconds per frame. That is over 80 frames per second. Yes, yes, yes. My goodness. That is finally in the real time domain. And then some. What a time to be alive. Ok, so where's the catch? Our keen eyes see that this is a static scene. It probably can deal with dynamic movements and rapid changes in lighting. Can it? Well, let's have a look. Wow. I cannot believe my eyes. Dynamic movement check mark. And here. This is as much change in the lighting as. As we would ever want. And it can do this too. And we are still not done yet at this point. Real time is fantastic. I cannot overstate how huge of an achievement that is. However, we need a little more to make sure that the technique works on a wide variety of practical cases. For instance, look here. Oh yes, that is a ton of noise. And it's not only noise, it is the worst kind of noise. And that is high frequency noise. The bane of our existence. What does that mean? It means these bright fireflies, if you show that to a light transport researcher, they will scream and run away. Why is that? It is because these are light paths that are difficult to get to and hence take. Take a ton more ray samples to clean up. And you know what is coming, of course. Here is paper number four. Let's see what it can do for us. Am I seeing correctly? That is so much better. This seems nearly hopeless to clean up in a reasonable amount of time. And this, this might be ready to go as is with a good noise filter. How cool is that? Now, talking about difficult light paths, let's have a look at this beautiful caustic pattern here. Do you see it? Well, of course you don't. This region is so under sampled we not only can't see it, it is hard to even imagine what should be there. So let's see if this new method can accelerate progress in this region. That is not true. That just cannot be true. When I first saw this paper, I could not believe this and I had to recheck the results over and over again. This is at the very least a hundred times more developed caustic pattern once again with a good noise filter. Probably ready to go as is. I absolutely love this one. Now note that there are still shortcomings. None of these techniques are perfect. Artifacts can still appear here and there and around specular and glossy reflections. Things are still not as clear as the reference simulation. However, we now have real time light transport and not only that, but the direction we are heading to is truly incredible and amazing. New papers are popping up while what feels like every single month. And don't forget to apply the first law of papers which says that research is a process. Do not look at where we are, look at where we will be two more papers down the line. Also, Nvidia is amazing at democratizing these tools and putting them into the hands of everyone. Their tech transfer track record is excellent. For instance, their marbles demo is already out there and not many know that they already have a denoising technique that is online and ready to use for all of us. This one is a professional grade tool right there and many of the papers that you have heard about today may see the same fate. So real time light transport for all of us. Sign me up right now if you're looking for inexpensive cloud GPUs for AI. Lambda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers because with Lambda GPU Cloud you can get on demand a 100 instances for $1.10 per hour versus $4.10 per hour with AWS. That's 73% savings. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Thanks for watching and for your generous support and I'll see you next time.