Unreal Engine 5 - Ray Tracing Supercharged!

An absolutely huge revolution is coming in gaming and virtual worlds and it is a big deal. I just read something impossible here. Unreal Engine 5 claims that it has real time path tracing. Now hold on for a minute. When they say path tracing, they mean a light simulation program where we shoot rays of light into the scene, bounce it around objects and depending on their geometry and materials, computer compute what this scene would look like in reality. Pretty wild. Now it is a true miracle that we can do that at all. But it has a cost, a steep cost. These simulations can take from minutes to hours. And this earlier experiment of ours took three weeks and the help of a whole community to compute all this for just one image. Although it was totally worth it. I mean, look at that reflected volumetric caustics love it. So here it says real time path tracing. That is insane to start with. This is something that takes forever, not something that you can just get in real time. Not even close. How is that even possible? And is that even possible at all? Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Now many techniques claim that they are doing some sort of ray tracing in real time. There are hundreds of papers, but most of them are really limited in some sense. One of the better ones is called voxel cone tracing, one of the blockbuster papers that can do something like that. It truly changed how I look at research to this day. But that one also has its limitations. It cannot compute too many light bounces. Light can leak through objects in unexpected ways and the resolution is not that great at places and so on. Nvidia's RTX ray tracing video cards are getting closer and closer, which is fantastic and it truly makes a difference in some new games. But depending on how you set it up, glitches and unexpected results may still come out. So regular offline path tracing is very slow but very accurate. Look at this progress bar below. It is still going for just this one frame. It is slowly getting better. But we still have plenty of noise in the scene. And most real time solutions are very quick but also inaccurate. But perhaps not this one. Hold on to your papers, fellow scholars. And look, same scene but with real time pass tracing. And at 60 frames per second, absolutely amazing. But wait, we have three important questions. Question number one. So what do we get with all this ray tracing thing? Should we be excited? The answer is, in my opinion, yes, fellow scholars, this will get us a ton of value. For instance, it is not that sensitive to the number of lights in the scene. So you can go ham and just put a ton of lights in there. Beautiful. Also better computations for lights, indirect illumination and shadows. More sophisticated material models and generally higher quality images. For instance, you see that with the previous technique we have these thin beams that hit the ground. But look around those. There should be more light scattering around. And I see none of that. And when using ray tracing. Let's have a look. Bravo. Look at how creamy and beautiful that is. And the whole image is also brighter, but in an intelligent way, so to say. And the shadows of the character are truly a sight to behold. The longer you look, the more beautiful it is. Wow. Chef's kiss. I want this everywhere. And here it is. And when we move lights around in a video game. Look at that. That is the incredible interplay of light and geometry that is on a different level than we've seen before. Like I said, this kind of shadow computation is also just amazing. And this is a path traced forest in this scene. I am wondering why they uploaded this in 1080p resolution in the age of 4k and 5k monitors. I would have loved to see this in all its glory. And check out this scene that has all of these puzzle pieces put together. I am a light transport researcher by trade, so for me, when I was working on my PhD, these scenes took from minutes to even weeks to compute for just one image. And now to see all of these papers come alive in a system unreal engine that is free for most people. I am truly out of words. And just think about being an artist and building a virtual world where you see all of these beautiful light interactions in real time as you build the level. What a joy. My goodness. Now we have two more questions. Question 2. How close is it to the offline version of path tracing that takes much longer? The answer is quite surprising. In many cases, quite close. It is not exactly the same, but look at those tasty glossy reflections. This one is very close. And this one, well, not so much. But the rest, including the super soft shadows on the floor, seem quite good to me. So now question number three. So how is all this black magic possible? Well, instead of simulating billions and billions of light rays, it only simulates millions. Hmm. But how can we just do millions instead of billions which is a thousand times less and still get this close? That sounds insane to me. In fact, that feels flat out impossible. Well, first, it does not use ray tracing everywhere. It only uses it where it is really needed. Typically indirect lighting where light has bounced around at least a a couple times. In a diffuse environment. Specular glossy reflections are really good. When Ray traced, so those are better too. And it builds upon a technique called Restor. They say it is a spatiotemporal sampling technique. But I hear you asking, what does that even mean? Well, sampling means that we shoot a ray of light through a pixel and bounce it around in the scene. And when we do that, we hope to also get to know the more information about its neighbors. So that is spatial information, but it also makes use of temporal information, that is the time domain. What happened to this pixel just a moment ago? There are also denoising algorithms tailored to raytracers and rister plays really nicely with them. And I'll note that everything that you saw here today is child's play compared to what ray tracing techniques are truly capable of and will be capable of in real time in the future. Why? Well, because now only the top shelf hardware can do this. So many games are still made not with ray tracing in mind, but as a nice to have addition that some people use. But later when all hardware will be able to do this, then they will be able to embrace more sophisticated incredible material models, higher resolution geometry. Wow. It will be insane. Just imagine that we might get these incredible scenes in our games virtual worlds. Just two more papers down the line. Go papers. Now, because of all this, if you got excited for ray tracing and I mean how can you not. I would like to offer you a full master level course on how to program such a system from scratch. Full light transport from nothing. And only today for a low, low price of nothing. Yes, I give this away to all of you for free as a gift, no strings attached. You can't even pay for it even if you wanted. Check it out in the video description. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. To run your own experiments on an Nvidia gpu. Check out Lamda. I use it myself regularly for these videos. Hmm, look at that. It's. You can generate high quality images in less than a second per image. I did a ton more of them and paid less than a dollar for all this. Crazy. Seriously. Try it out now@lambdalabs.com papers or click the link in the description.