NVIDIA's DLSS 3.5 uses AI to generate 7 of 8 pixels in games, combining super-resolution with frame synthesis. The new ray reconstruction model, trained on 5 times more data, replaces separate denoising and upscaling steps to better preserve details in ray-traced images. The tech ships to all RTX GPUs, though quality varies by game.
Questions this video answers
September 19, 2023.
DLSS 3 generates 7 out of every 8 pixels (>85%).
About five times more training data than previous models.
Generated images in about 3 milliseconds versus 40–60 seconds (approximately 20,000× faster).
All RTX graphics cards (including older models).
Chapters
You hear it everywhere. DLSS has this DLSS that. But what is it and why is it doing the impossible? This AI based technology is being talked about in the media a great deal, and it is supposedly a way of dramatically speeding up computer games and virtual worlds. So does this really work? Well, when Jensen Huang, Nvidia's CEO, says that every single pixel on your screen for these games will not be rendered, but will be generated soon, presumably by an AI, that really got my attention. That sounds insane. And if hearing this, you think that this cannot possibly be true, I don't blame you for a second. We can't just create all this footage. There has to be a proper system that can compute reflections, diffuse interactions and all that, right? Well, not quite. In 2018, I had a little project where I wanted to generate photorealistic images of material models quickly. Now, generating an image through ray tracing took up to 40 to 60 seconds, and that was a bit too long for my taste. So I tried to write a neural renderer that could take just the description of a material model and it would immediately, in real time, synthesize what it would look like. And it was able to do all this not in 60 seconds, but in 3 milliseconds? Yes, it was 20,000 times faster. However, it was limited to this very scene. So how did this research field improve in the last five years? Do we have anything more usable now? Oh boy, if only I could tell you. Well, meet DLSS Deep Learning super sampling. This is a system where hardware and software works together to create an incredible experience where games and virtual worlds really run faster than what should be possible. So how? Well, as of DLSS version 3, get this by generating 7 out of every 8 pixel that appears on the screen. Yes, more than 85% of the pixels are just generated, not computed by for instance, ray tracing or traditional techniques. Yes, that sounds flat out impossible, but it is possible. The first step is running super resolution in real time. This means that a coarse image goes in and an AI finds out. If we pretended that this is a fine image, instead, what are the details that are missing? And it synthesizes those details. Second, optical flow. We can look at two adjacent frames of the video game and try to estimate what has happened between them and where things are going. With this, entire intermediate frames can be synthesized, creating the illusion as if the game was running smoother than our hardware can run it. Combining the two together with a pool of very few pixels that are computed, both image quality and smoothness can be improved at the same time. What a time to be alive. But it does not stop there. It gets better. They have also announced DLSS 3.5. So hopefully all this can be improved even further through something that they call ray reconstruction. Oh my goodness. This is going to be so good. But how? Well normally when we perform ray tracing, we simulate the path of millions and millions of rays and bounce them around in the scene. And if we can't afford to wait for up to hours at a time for an image, we will have to settle for a really noisy image like this. Denoising techniques exist that try to guess what the missing information could be. But they are not perfect. Not even close. Some important details between objects can be missed entirely. And still reflections can be a lot less clear than in a full simulation. And unfortunately, we have a problem. It gets even worse. This image will undergo an upscaling step where these errors get magnified even more. Denoising and upscaling are two separate steps, typically done by two separate models. So why not do both of them with the same AI model? And now ray reconstruction enters the scene. It learned on a ton more information than the previous models. Five times more training data was given to it. And look at this step. This one is right on the money. This is tailored to retain high frequency information, fine details to make sure that before the upscaling step, we have the highest quality information available. So is it better than what denoisers did before? Oh my goodness. Look at that. We finally get that shadowy region back that was filled in by incorrect information with the Denoisers before. Yummy. So good. And that high frequency information retaining capability can also be witnessed here. Loving it. Now note that ideally here we talk about peer reviewed research papers where we can see all the weak points and failure cases. That is my home turf. This is not a research paper, so it is harder for me to find the flaws. So bear in mind that it may have weaknesses that were not shown here. And in the meantime, this technology is being handed out to millions and millions of people all around the world. And it is incredible. So this will be available only for the shiniest newest graphics card owners, right? Well, hold on to your papers, fellow scholars, because here comes the best part. No, this comes to all RTX graphics cards, even the older ones that you can pick up for a couple hundred bucks. This can breathe new life into aging hardware, which is highly appreciated. Note that the particular game or app you're running has to be ready for DLSS 3.5 for all this magic to happen. Now, I am a light transport researcher by trade. I dream with rays of light and ray tracing if you will. So I am incredibly happy to see this technology make it into the hands of real users in the real world in such an incredible way. And make no mistake, this is not perfect by any means. There are games where it does not appear to work very well and some people prefer to not use it at all anywhere. However, note that this is still nascent technology and it can already synthesize more than 85% of these frames, which is something that I thought would only be possible in a science fiction movie. Maybe not even there. I am absolutely stunned by these results, even with their weaknesses. And don't forget to invoke the First Law of Papers. The first Law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub, and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through WNB ME2MP or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for helping us make better videos for you.
