NVIDIA’s New AI: Ray Tracing From 10,000 Images!

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today. I am very happy. Do you know why? Of course, because we are going to talk about an incredible AI light transport simulation paper. You see, these simulations normally take place by creating a little virtual scene. And we would like to see what this virtual scene would look like in reality. We can do this by using a technique called ray tracing which simulates the path of millions and millions of light rays to create a beautiful image or even video. This technology is also used for simulating the path of sound waves in some modern speakers and even in Apple's new Vision Pro headset. So, millions of rays, Right? Sounds great. However, look, this is anything but a beautiful scene. Why is that? Well, before we simulate enough light rays, which may take from minutes to hours, we will get a noisy image. This image will get less and less noisy over time. However, we don't have hours of time per image in a video game or any kind of virtual world. We prefer real time results. So what do we do? Well, we start noise filtering. These are techniques where we can show a noisy image and magically they clear up this noise by imagining what this image could depict. Some of them are incredibly good. And many of these AI based techniques are so fast that they even work for video. However, not so fast. Unfortunately, if we add volumetric effects like haze and smoke, things get a lot more difficult. Look, this is the noisy input with smoke. And a previous technique does this. Whoa, that is not great. Well, do not worry, because this technique doesn't yet understand smoke, just hard surfaces. So let's see what this previous technique with additional training on volumes can do. Ouch. Well, smoke looks a bit more like smoke, but unfortunately the whole scene got destroyed. So is that it? We can perform incredible ray tracing on these beautiful scenes, but only without haze, smoke and similar volumetric effects. Well, don't despair quite yet. Hold on to your papers, fellow scholars, and have a look at the new technique. Wow, this is way better than the previous technique. Actually it is not perfect, but so much closer to the true simulation result. This is the one that would have taken up to several hours to compute. And this is what we now get almost immediately from the new AI. My goodness. And once again, this is compared to the previous method that was trained on volumes two. And this was not by chance. When comparing other cases, the new method also comes out ahead. So if the new one is so good, this previous one must be from super long ago, right? Some ancient paper. Well, is it some ancient paper? Nope, not at all. This is The Neural Temporal Adaptive Sampling and denoising paper from 2020. Such incredible improvement in just one paper. Wow. And hold on to your papers, fellow scholars, because here comes the best part. All this runs now. Not in hours and not even in minutes. What you see here runs in real time. It can really take this incredibly noisy sequence of images and my goodness, it reconstructs these beautiful videos from it that are closer to reality than ever before. I absolutely love this. It almost feels like we are living in a science fiction movie. And can that really be? Yes, it gets even better. The technique decomposes these images into surface and volume components so it can separate the opaque objects in the scene from the smoke itself. Is that good? Well, to say that is good news would be an understatement. Because look, it can even reimagine the scene as if the lighting was different. Or we can even play with the smoke plumes themselves. So from now on, we will likely be able to enjoy these beautiful games with simulated smoke. Bravo. What a time to be alive. Also, training this neural Network took only 24 hours on a single Nvidia graphics card. That is insanity. And that just needs to happen once and then everyone in the world can enjoy it for as long as they wish. And it didn't even take looking at millions and millions of noisy and clean input image pairs to train this network. It only took a bit more than 10,000 images. That is not a lot proper learning from very little data. I am out of words. And when watching all these beautiful results. If you feel that this ray tracing thing is pretty cool and you would like to learn more about it. I held a master level course on this topic at the Technical University of Vienna. Since I was always teaching it to a handful of motivated students, I thought that it's not right to have the teachings only be available for the privileged few who can afford a college education. No, the teachings should be available for everyone. Free education for everyone. That's what I want. So the the course is available free of charge for everyone. No strings attached. So make sure to click the link in the video description to get started. We write a full light simulation program from scratch there and learn about physics, the world around us and more. If you watch it, you will see the world differently.

Weights and Biases provides tools to track your experiments in your deep learning projects. What you see here is their artifacts feature which speeds up the most common machine learning steps like uploading raw data, splitting it into training, validation and test sets. And of course the best part, starting to train a neural network. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.

Thanks for watching and for your generous support and I'll see you next time.