This content was created by LARRUZ. This content was uploaded by LARRUZ. This is a YouTube video titled "NVIDIA’s New AI: Impossible Weather Graphics!" from the Two Minute Papers series (presented by Dr. Zsolnai) published May 26, 2025. The content is a single-speaker explanatory presentation/demo describing recent AI research for weather synthesis and desynthesis (adding/removing fog, rain, snow), inverse rendering (relighting, viewpoint changes, material edits), and a self-supervised bootstrapping approach that generates paired data for training. The presenter highlights qualitative results across many scenes, discusses practical applications (e.g., training self-driving car simulators), notes technical challenges (specular reflections, synthesizing occluded content), and briefly mentions running large models on Lambda GPU cloud as a sponsor mention.
Chapters
What you are seeing here is basically impossible to do. Yet it just happened. So how and what is going on here? Well, this is a brand new AI technique where in goes your video and it changes the weather effects on it. No 3D modeling required. No physics simulation required. Also no camera calibration required. Crazy. And then you shall hear about the legend of the apple that went into another dimension and how to solve this problem and more. And I said impossible because look, this is what happens when you try previous techniques to do that. In goes this video. Then little AI add some fog to it. Oh my. I can't see anything. And it gets worse. It is not even realistic. This NEV2V technique was published less than a year ago. Not some ancient technique. Ok, now a different work. Ouch. They clearly don't understand how to do this properly. It seems absolutely impossible until this new work. Look at that. Wow. This is crazy good. And not only at fog. Rain also not a problem. Look at that beauty. It makes any place look like London. Kidding, kidding. Snow synthesis is also incredibly amazing. Look at how cozy this is. Loving it. Okay, okay, but wait a minute. You can take any technique, choose that one scene it does really well on and advertise only that. What we fellow scholars want to know is whether it is actually a practical technique. Does it work on a variety of scenes? And you bet your papers it does. Look at that. A huge variety of scenes. And all of them are absolute beauties. Now and just think about training self driving cars. They could experience countless weather scenarios safely in a simulation thanks to this. So this was weather synthesis. Now get this. What about weather desynthesis? Yes. What about removing weather effects? Now that is an entirely different beast. I think that is truly impossible. Why? Partly because it is kind of like asking an AI to undo milk from coffee. How is that even possible? You see, if you wish to remove the fog, none of the previous techniques are really capable of doing that. No wonder. I mean adding rain or snow to already existing footage. Okay, you need to change some pixels around. Tough, but possible. But when you remove the fog, new things appear. Oh my. You would have to synthesize and fill in new information. What could be in the background? Well, you have to put something there that makes sense. And for that you need to not just draw a few pixels. No, no. For that you need to understand the world. So can the new technique do it? Now hold on to your papers fellow scholars. And oh my goodness, it did it. And it not only did it, but when you try rain removal, previous techniques are either not really doing anything or the only one that does in return also changes whole scene. We did not have colorful cars in the input footage at all. And the new one, once again, incredibly amazing. Same with snow. But now check this out, because it gets even better. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Yes. No one said that fog should be turned off like a light switch. Sometimes you want a little of it? Hmm, not a problem. You get a little slider that you can play with. In the case of fog, you can change the fog density. And for snow. Oh, I love this one. In the case of snow, you can play with the amount of coverage that you wish to see there. And a bonus paddle coverage. Now let's stop for a second there. What paddles? Don't forget paddles make the roads more specular, which means that they are reflective. But to become reflective they have to reflect something. And once again, this is something that cannot just come from thin air. This is something that the AI needs to synthesize. You see, I am a light transport researcher by trade and I tried to resist the urge to mention specular reflections. And of course I failed again. So this is how this amazing weather transition video was made in the intro. And here is the craziest thing about this work. It kind of teaches itself. Originally it performed only weather removal. They used it to create pairs of the same scenes. One with rain and one without. Then they used this data to train a weather synthesis AI model. So in a way, it kind of teaches itself. I think self supervised bootstrapping would be a good term to call this behavior. Absolutely mind blowing. Now just imagine combining the weather synthesis with this amazing technique. Yup, this is the legend of the apple that disappeared into another dimension. Okay, what is going on here? The problem here is that we have an input image. And what would we like to extract from it? Everything, Absolutely everything. Geometry information, depth information, material properties, everything. And then use that to re render this image in a different way. Perhaps with different lighting, little alternative realities. So cool. However, not so fast. When a previous technique looks at this apple and we are asking it to reconstruct the scene and rotate it. Oof, I am seeing bad news already. Look, it underestimated the shadow in the scene. And when we rotate it. Oh goodness. That's like a sandwich in a shady shop. From the outside it looks amazing. When you look into it, you get a black hole. Nothing. Now let's see the new technique. Oh my goodness, it pulled it off. Proper inverse rendering right there. And once again, not just on one scene, but on more lifelike environments too. But the point is that if this is a proper AI inverse renderer, then it can do so much more. For instance, you can not only relight the scenes or look at them from a new viewpoint. No, no, get this. You can also edit the materials and the results are still photorealistic. Like editing reality itself. Crazy. And you can do this too. Wait a second. I hear you asking what happened here? Well, not all of the objects in this scene are real. This was inserted by this new technique. Same here. You can maybe kind of tell if you look closely. But one more paper down the line from here? Not a chance. Another impossible problem solved with the new research paper. And while we look at these beautiful results. This is Two Minute Papers. Elsewhere you get McDonalds. Here you get the papers. Proper research papers from a scientist. Hope you are enjoying it as much as I am enjoying it. Here you see me running the full deep seq AI model through lambda GPU cloud. 671 billion parameters. Running super fast and super reliably. This is insane. I love it and I use it on a regular basis. Lambda provides you with powerful Nvidia GPUs to run your own chatbots and experiments. Seriously. Try it out now at Lambda aipapers. Or click the link in the description.
