NVIDIA's Ray Tracing AI - This is The Next Level! 🤯
Dr. Zsolnai explains how NVIDIA uses neural networks with classical rendering techniques to speed up ray tracing. The method starts from an informed image instead of black, cutting noise and convergence time. It works across many scenes but still fails on some, like glass containers with complex shapes.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a light simulation algorithm that doesn't work too well and infuse it with a powerful neural network. And what happens? Bam. This happens. Wow. So what was all this about? Well, when we fire up a classical light simulation program, first we start out from. Yes, absolutely nothing. And look, over time as we simulate the path of more and more light rays, we get to know something about the scene. And slowly over time, the image cleans up. But there is a problem. What is the problem? Well look, the image indeed cleans up over time. But no one said that this process would be quick. In fact, this can take from minutes in the case of this scene to get this, even days for this scene. In our earlier paper we rendered this beautiful but otherwise sinfully difficult scene and it took approximately three weeks to finish. And it also took several computers running at the same time. However, these days neural network based learning methods are already capable of doing light transport. So why not use them? Well, yes they are, but they are not perfect and we don't want an imperfect result. So scientists at Nvidia said that this might be the perfect opportunity to use control variants. What are those and why is this the perfect opportunity? Control variates are a way to inject our knowledge of a scene into the light transport algorithm. And here's the Any knowledge is useful as long as it gives us a head start, even if this knowledge is imperfect. Here is an illustration of that. What we do is that we start out using that knowledge and make up for the differences over time. Ok, so how much of a head start can we get with this? Normally we start out from a black image and then a very noisy image. Now hold on to your papers and let's see what this can do to help us. Wow, look at that. This is incredible. What you are looking at is not the final rendering, it is this is what the algorithm knows. And instead of the blackness, we can start out from this. Goodness. It turns out that this might not have been an illustration, but the actual knowledge the AI has of the scene. So cool. Let's look at another example in this bedroom. This will be our starting point. My goodness, is. Is that really possible? The bed and the floor are almost completely done. The curtain and the refractive objects are noisy. But do not worry about those for a second. This is just a starting point. And it is still way better than starting out from a black image to be able to visualize what the algorithm has learned with a little more Fineness. We can also pick a point in space and learn what the world looks like from this point and how much light scatters around it. And we can even visualize what happens when we move this point around. Man, this technique has a ton of knowledge about these scenes. So once again, just to make sure, we start out not from a black image, but. But from a learned image. And now we don't have to compute all the light transport in the scene. We just need to correct the differences. This is so much faster. But actually, is it? Now let's look at how this helps us in practice. And that means, of course, equal time comparisons against previous light transport simulation techniques. And when it comes to comparisons, you know what I want? Yes. I want Eric Veitch's legendary scene. See, this is an insane scene where all the light is coming from not here, but the neighboring room through a door that is just slightly ajar. And the path tracer, the reference light transport simulation technique, behaves as expected. Yes, we get a very noisy image because very few of the simulated rays make it to the next room. Thus most of our computation is going to waste. This is why we get this noisy image. And let's have a look what the new method can do in the same amount of time. Wow. Are you seeing what I am seeing? This is completely out of this world. My goodness. But this was a pathological scene designed to challenge our light transport simulation algorithms. What about a more typical outdoor scene? Add tons of incoming light from every direction. And my favorite, water caustics. Do we get any advantages here? Hmm, the path tracer is quite noisy. This will take quite a bit of time to clean up. Whereas the new technique. Oh my. That is a clean image. How close is it to the fully converged reference image? Well, you tell me, because you're already looking at that. Yes, now we are flicking between the new technique and the reference image and I can barely tell the difference. There are some, for instance, here, but that seems to be about it. Can you tell the difference? Let me know in the comments below. Now, let's try a bathroom. Lots of shiny surfaces and specular light transport. And the results are. Look at that. There is no contest here. A huge improvement across the whole scene. And believe it or not, you still haven't seen the best results yet. Don't believe it. Now, if you have been holding on to your papers, squeeze that paper and look at the art room example here. This is an almost unusable image with classical light transport. And. Are you ready? Well, look at this. What in the World. This is where I fell off the chair when I was reading this paper. Absolute madness. Look, while the previous technique is barely making a dent into the problem, the new method is already just a few fireflies away from the reference. I can't believe my eyes. And this result is not just an anomaly. We can try a kitchen scene and draw similar conclusions. Let's see, now we're talking. I am out of words now. Despite all these amazing results. Of course not even this technique is perfect. This torus was put inside a glass container and is a nightmare scenario for any kind of light transport simulation. The new method successfully harnesses our knowledge about the scene and accelerates the process a great deal. But once again we get fireflies. These are going to be difficult to get rid of and will still take a fair bit of time. But my goodness, if this is supposed to be a failure case, then yes, sign me up right now. Once again, the twist here is not just to use control variates, the initial knowledge thing, because in and of itself it is not new. I, like many others, have been experimenting with this method back in 2013 almost 10 years ago, and back then it was nowhere near as good as this one. So what is the twist then? The twist is to use control variates and infuse them with a modern neural network. Infusing previous techniques with powerful learning based methods is a fantastic area of research these days. For instance, here you see an earlier result, an ancient light transport technique called radiosity. This is what it was capable of back in the day. And here is the neural network infused version. Way better. I think this area of research is showing a ton of promise and I'm so excited to see more in this direction. So what do you think? What would you use this for? I'd love to hear your thoughts. Please let me know in the comments below. And when watching all these beautiful results, if you feel that this light transport thing is pretty cool and you would like to learn more about it. I held a master level course on this topic at the Technical University of Vienna. Since I was always teaching it to a handful of motivated students, I thought that the teachings shouldn't only be available for the privileged few who can afford a college education, but the teachings should be available for everyone. Free education for everyone. That's what I want. So the course is available free of charge for everyone. No strings attached. So make sure to click the link in the video description to get started. We write a full light simulation program from scratch there and learn about physics, the world around us and more if you watch it, you will see the world differently. This video has been supported by weights and biases. Look at this. They have a great community forum that aims to make you the best machine learning engineer you can be. You see, I always get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. In this forum you can share your projects, ask for advice, look for collaborators and more. Make sure to visit wnb me paperforum and say hi or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
