This is a short YouTube commentary by Dr. Zsolnai (LARRUZ) that explains recent computer-graphics work combining fast Gaussian-splat scene representations with ray-tracing-style secondary rays to achieve real-time, high-quality reflections, refractions and translucency. The video contrasts rasterization (fast but limited) with traditional ray tracing (high quality but slow), describes the 3D Gaussian Unscented Transform ("3D GUT"), notes open-source code availability, and highlights applications in gaming and self-driving-car training.
Chapters
There are two ways of creating and rendering a virtual world. This is what we call rasterization. This is what most computer games have used for many years now. It is super fast and most of the time it looks good enough. But only if you feel that you can live a happy life without super high quality reflections and refractions. But not me, because for that you need something more. My favorite, ray tracing. Look at these beauties. This you cannot get through rasterization, only through ray tracing. But there is a problem. It simulates the path of millions and millions of light rays, which takes super long to compute, like orders of magnitude longer, sometimes from minutes to even weeks. Ouch. This one is a beauty. Reflections, refractions, volumetric caustics. Oh my. But it took us three weeks to render. So rasterization fast, limited ray tracing, a complete solution, but slow. So I guess choose one, right? Well, scientists at Nvidia had an absolutely insane idea. They said why not do both? Well, my opinion was before reading this paper, don't do both. Because it is impossible to pull off. These two ideas are like oil and water. They just don't mix. You most likely end up with the disadvantages of both. That is a limited but super expensive solution, but no thank you. So did they pull off the impossible? We'll get to that in a moment. And then splat happens. I mean, a paper called Gaussian Splats came out and took the world by storm. What is that? It's a technique that thinks in terms of little Gaussians representing a scene like this as a collection of little bumps, if you will. And note that these scenes can come from real life. You just walk around with your camera and then you can play in a digital version of it. And the cool part is that it gets better. It is incredibly fast, easily faster than real time. So far so good. However, here is the problem. Mirror like specular reflections are not great. And many of the beautiful ray tracing effects that you just saw earlier don't look so good there. It can also eat up quite a bit of memory, which is scarce and expensive on graphics hardware these days. But the limitations don't stop there. It doesn't support any fancy camera models, for instance, fisheye cameras. Don't even think about it. Also no camera tricks like rolling shutter. So there is tons of potential in Gaussian splats. But if I think about the fact that we get no refractions, none of the good stuff that we fellow scholars love so much, this all ranges from very difficult to impossible. But don't Despair. Because here comes the impossible but at the very least insane idea. Let's do both rasterization and ray tracing at the same time. This research work does it by taking Gaussian splatting and adding to it something they call secondary rays. Oh, this means that there are finally rays of light in the system and they are allowed to bounce around. So, are you thinking what I am thinking? Oh yes. Hold on to your papers, fellow scholars. And look at that. A proper virtual world running in real time and high quality reflections. I am absolutely loving this one. And refractive objects like glass also. Wow. Now let's whip out that fisheye camera. And yes, finally. This all used to be impossible and now it is not only here, but it runs in real time. And here comes my favorite favorite part. The name of this work. You'll love this. They call it the 3D Gaussian unscented transform 3D G U T. Yup, but that's not how we will call it. No sir, we shall call it 3D gut. Much better. And here is a nice little surprise. Do you know what super important AI area likes? Weird camera models and rolling shutter training. Self driving cars. Oh yes, and this is going to be super useful for that too. But is it really better than previous techniques? Let's see together. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. So 3D gat looks amazing. But we are fellow scholars here. We want to see the comparison against previous techniques. Well, it turns out there is one earlier method that can extend Gaussian splats to fisheye cameras. But oof. I see lots and lots of artifacts and distortions here. So do you have the guts to have a look at the new technique? Let's see. Oh my. That is a huge difference. The previous one had trouble, especially with objects that are closer to the camera. But this new one. Loving it. But the story doesn't end here. There is so much more below the surface. We are going to look at this too in a moment. And here comes the best part. They didn't just close it down to sell it to us. Nope. All this is available right now for all of us for free. The source code is out there. I put a link in the video description. So I hope you fellow scholars have the guts to play with it. Sorry, I promised myself not to do it again and I just did. But in fact, some of you already do play with it. This fellow scholar says he trained it to only 30% and it already looks amazing even through through the compression artifacts from social media okay, but I hear you asking, wait a minute. What is this beauty? Well, I said that there is more below the surface. I said that because this other work is about light transport that happens within these objects. That is subsurface scattering, the art of rendering beautiful translucent objects. Now you can do that in rasterization. Here is a work called Separable subsurface scattering. And it gives you super fast human skin, marble, milk, you name it. I love working on things like this so much. And it is so simple. When we published it, we even implemented a version of it that fits into four kilobytes. That is crazy. It fits into a file that is smaller than half a second of MP3 music. Much smaller. It is available on my website. The link is in the description and to the best of my knowledge, you can also use it in Unreal Engine as well. So this is subsurface scattering but for rasterization, not for Gaussian splats. What about that? Well, this amazing new work offers you exactly that and it gets even better. It supports relighting too. So you have your object, you add an image and it imagines what that object would look like in these different environments. Super fun. And if you don't like what you got, you can even edit these materials so you can go from skin to glass to wet. I mean, just think about that. Gaussian splats are not old at all. It's about two years old. And now through the power of open research and human ingenuity and a hint of AI reflections, refractions and translucency is already possible for Gaussian splats. Yes, now you can get reasonably high quality translucent objects in these virtual worlds too. So next level computer games and virtual worlds are coming. And self driving cars can also learn better. I am kind of stunned that almost nobody is talking about these papers. But this is why two minute papers exist. And don't forget, we are all fellow scholars here. So we love to look at these papers for more information. And it goes without saying that everything is available in the video description. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. Fellow scholars, check this out. This is Deep Infra, an AI inference cloud where you can get access to the best OpenAI AI models like Deep Seq, Lemma and pay up to 10 times less than elsewhere. Or use my favorite, the amazing Flux text to image AI, generate speech from text and more. It is super easy to use. Try it out today@deepinfra.com papers or click the link in the description and you get $20 of credit for free.
