Károly Zsolnai-Fehér breaks down Nvidia's new technique that combines ray tracing with Gaussian splatting for particle-based 3D scenes. The method produces sharper reflections, refractions, and shadows while running at 10-78 FPS and using less memory than standard Gaussian splatting.
Today we are going to talk about Nvidia's crazy new work on fusing two ideas together that I think is completely impossible to pull off. One idea is taking ray tracing, the light simulation technology you hear a great deal about these days. This is what makes many video games and virtual worlds absolutely beautiful. This simulates the path of millions of millions of rays of light to give you realistic reflections, refractions off of glass, sophisticated material models, realistic human skin, milk marble. This takes a scene with a given 3D geometry and breathes life into it. This can do pretty much anything. However, it is slow and needs a lot of computation from seconds to minutes to hours. Ouch. The other idea is a technique that get this represents the scene with not 3D geometry, only with particles. This is called Gaussian splatting. It is incredibly fast, easily fast. Faster than real time. However, specular reflections are not great and many of the light transport effects that you just saw don't look so good here. It can also eat up a bit of memory, which is scarce and expensive on graphics hardware these days. However, this runs in real time. So one technique that can do everything math is slow and one that is limited, but super fast. One that works with 3D geometry, the other works with points. And scientists at Nvidia say we are going to fuse the two together into one unified algorithm. I'm going to be honest with you, I am a bit skeptical here. At least I was when reading this paper. Seriously, they are thinking about doing ray tracing, but not on 3D geometry, but on particles, points if you will. The initial comparisons against Gaussian splatting, the point based technique, show some promise. The new technique has more detail and a heck of a lot less of these really blurry patches. So far so good, but it gets better. So let's push it to the limits through four beautiful experiments. Experiment one. Okay, this is not bad. Look, I am warming up to the idea of ray tracing points, but only slightly. These are synthetic objects that are working well. So far so good. Now let's look at some real scenes. And here is where things get weird. Experiment 2. A full scene from real life with not synthetic data, but real stuff. And oh my, look, the specular reflections are absolutely amazing. As a ray tracing researcher by trade, it really makes my heart sing. And the spinning also makes my head dizzy. So let's proceed. I am seeing incredible detail everywhere. A variety of material models in the kitchen and goodness, just look at that. Glossy reflections off of the table. Crisp, high resolution, truly incredible. Not perfect. But I am slowly Getting convinced here, but I still have some conditions. Experiment three, Large scale scenes. That would be one of my conditions. And wow, this might be really good news in terms of memory consumption because all this can be rendered at the same time. Hmm. Experiment four. Oh yeah. Advanced light transport effects. Reflections. Check mark. Shadows. Check mark. And now my favorite, refractions. You can insert a glass object into a scene and even play with the material properties, bending not only light rays, but the laws of physics too. Really cool. Now the most important part. I wonder how fast this algorithm is. I mean, they combined a fast and a slow technique together. So which property did it inherit? Is it fast or is it slow? That is going to be what makes or breaks this work. And now hold on to your papers, fellow scholars, because it is significantly faster than the slow ray tracers. Get this. Depending on the scene, it can produce from 10 to 78 frames per second. So in many cases, real time techniques from just a couple years ago could do one frame per second at most and nothing even close to this quality. And now we get this. Wow, I am stunned. What a time to be alive. And it doesn't end there. It eats about half as much memory as Gaussian splatting. Also, they give the full description of the technique away for free. So you may not even need to buy new hardware to enjoy this. I think this will be a complete game changer. Just imagine that in just one paper we went from minutes per frame down to real time performance. And this is the point where I drop my papers. Now, not even. This is perfect. The blurry patches are not completely gone. But that might happen one or two more papers down the line. And yes, this is where we invoke the first law of papers. It says that research is a process. Do not look at where we are, look at where we will be two more papers down the line. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. Fellow scholars, if you are like me, it takes you a while to truly understand the paper, especially when it's not right in your area. Macro.com is a new AI document reader that gives you superpowers. Click on any term or figure and it explains it to you without leaving your page. You can even talk to the paper or ask questions too. Try it for free@macro.com papers and get $5 with the code papers.
