Adobe & NVIDIA’s New Tech Shouldn’t Be Real Time. But It Is.

Look at this. Absolutely gorgeous. It contains lots of beautiful glinty particles. And believe it or not, this one is rendered on my laptop. And in real time. Yup. This research is free and open for everyone, so you can try it too. I'll tell you how. Now, if you ever looked at fresh snow under a street lamp or metallic car paint in the bright sun, you see this amazing Galinthi explosion. And if you turn your head, or in this case, the camera around, oh, my. It's an incredible sight. But it's also incredibly difficult to simulate in a computer program. Why? Because these surfaces have millions of microscopic reflective flakes. If you try to simulate them all, your computer crashes. If not, you get a boring, bland object in your games and movies. So how do we compute all this without using gigabytes of memory or destroying our frame rate? Now, hold on to your papers, fellow scholars, because this incredible, incredible new technique can do that. And it promises more than 280 frames per second. Wow, that's crazy. On a consumer Nvidia graphics card. But you don't even need that. It runs in real time on my much less powerful laptop, too. Okay, but how? Well, imagine trying to host the world's biggest party. Usually you need a guest list to know where everyone is standing. Well, this paper. Throw away that list, brother, you won't need it. Okay, but how do we host a party without a guest list? Well, instead of remembering where every guest, every glitter particle is, they use a bouncer, a really muscular guy. And this guy doesn't need a list. He uses some mathematical rule to decide exactly where a guest should be standing. The moment you look at that spot, he generates the party guests on the fly instantly. We'll go into some more details, but I cannot resist showing you these results. In the meantime, it can even nail that sun on the ocean look. Now, it does not look nearly perfect because this is not a Whitewater simulation. So it does not have foam and bubbles. But that's not the point here. The point is that you can rotate the camera around, it looks great, and it remains temporally stable. What does that mean? Normally, I would say that this means the technique remembers what it did a moment ago, but it does not. And that is the key. It is so fast for every frame, it can very easily and quickly recalculate the result. And it is so accurate, it will always look exactly the same, so no crazy jumps. And this is super useful because you don't need to use a lot of memory to have these millions and millions of glinty little mirrors. On your objects. So when the camera moves, the sparkles don't flicker like a broken strobe light. No, they shimmer beautifully. Okay, now wait a second. This is not the first technique that is able to render glints. So how does it do against previous techniques? Well, here is how it does against one of the industry standard sampling techniques called ggx. This is an equal time comparison. So both methods were given the same amount of time. This allows us to check how fast the noise clears up. But this new technique makes all this really easy because I can stop this at any frame I want. And look, the new one always seems better than ggx. Crazy. Why though? Well, because GGX searches for the sparkles blindly, so the image stays noisy and takes a long time to clear up. But the new technique knows exactly where they are, so it cleans up the image much quicker. Ok, now we said the muscular bouncer guy has a secret. What is the secret? How does he control the crowd? So there is a big party going on without being overwhelmed with the guest list? Well, first he divides the dance floor into a grid. If you look from far away, he groups the guests into big blocks and just tells you don't worry about it, there's a party over there. But as you walk closer, he breaks those blocks down into smaller VIP sections, revealing the individual dancers. He manages the crowd density dynamically, so you never see the empty spaces, but you also never get overwhelmed by the crowd. Amazing. So here the dance floor is the surface of the object and the dancers are the little flakes that create the sparkles. So you get to simulate as much detail as needed with the minimum amount of compute necessary. Absolutely spectacular work. Now, this car example, it looks alright. I'll be honest here. Not the best parameter setup for my liking. However, it has its moments. But this reveals an incredible property of the new technique. It can be UV free. What does that mean? Usually to put a texture on a 3D object, like this dragon, you have to unwrap its skin onto a flat 2D image. Just like flattening a piece of gift wrapping paper. This process is called UV mapping. And for complex shapes. Ooh, it is a nightmare. The paper tears, stretches or has ugly seams. No thanks. But here you don't even need that. Why? Because our bouncer doesn't need to look at a 2D map or a guest list. No, instead he operates in the 3D world, not on a flat piece of paper. This means you can have a complex car chassis or a twisting dragon. Doesn't matter. And look, the sparkles just magically appear in the right place instantly. No ugly seams and no stretching. Absolutely incredible work. But if you look closely, it teaches us a great deal more than it seems at first. You see, the system discards the massive guest list and uses a bouncer, a simple math rule, to generate them on the fly. But that is incredible life advice. This is exactly what I taught my students at the university. Stop hoarding information. Don't memorize the encyclopedia. That's useless. Learn the principles, learn the rules. This way you can derive the answer in any situation quickly. Also, don't forget this method works in 3D space, refusing to flatten the object or onto a 2D map to avoid tearing and stretching. Great life advice. Maintain your dimensionality. Do not flatten your 3D personality into a 2D label just to make it easier for others to process you. No, stay 3D. Fellow scholars, I also live my life this way. This video series is an absolute madhouse. But it's my madhouse and I love it. For some people it is too much. For you fellow scholars, it is just right. Hopefully. And I am super lucky to have all of you on our journey. The papers couldn't exist without you. Thank you so much and we can learn so much from these papers. It's incredible. Now, all this wizardry comes from the heavy hitter scientists at Adobe Research Nvidia and Aalto University. And we get all this for free. I can't believe it. How amazing is that? What a time to be alive. Now, before I show you how you can try it right now, I'll tell you that not even this technique is perfect. First, the method is not strictly energy conserving, meaning the simulation can artificially gain or lose light energy near domain boundaries. Now, for video games and movies, usually not a problem, the difference is so little. But for super scientific experiments, however, look elsewhere. Second, some parameter pairs are not independent, so certain combinations can lead to counterintuitive visual results. Third, if you want the UV free wrapping paper property, then it is a bit slower. So trying to be accurate and not overstate things here. Okay, and you fellow scholars can try it right now. There is a link to the paper in the video description and another one for the demo. Just click and run it in your browser. Then if you click and go left to right, you get less or more glinty particles. Look at this. So pretty. And if you go down or up, you can change the roughness of the surface to rougher. They also give us the full source code. It can be implemented in about 337 lines of code. Kind of insane. And thus you can also play with the parameters. Press a button, recompile, and the world suddenly works a bit differently. Here you see me running the full deep seq AI model through lambda GPU cloud 671 billion parameters running super fast and super reliably. This is insane. I love it and I use it on a regular basis. Lamda provides you with powerful Nvidia GPUs to run your own chatbots and experimen. Seriously? Try it out now at Lambda AI Papers or click the link in the description.