Dr. Zsolnai shows how DALL-E 2 generates realistic light effects: denoising simulator outputs, creating caustics including heart-shaped ones, producing translucency with backlighting, and mimicking digital artist Reynante Martinez's style.
November 10, 2022.
Dr. Zsolnai (Two Minute Papers / LARRUZ).
Code '2Minute' for 10% off the first month on Runway.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Finally, this is my happy episode. Well, of course I am happy in every episode, but this, this is going to be my happy, happy episode, if you will. Why is that? Well, buckle up because today we are going to use OpenAI's DALL E2A text to image AI and we will see what it is made of. Can it create beautiful light transport effects or not? We will see through four beautiful experiments. For instance, this may sound like science fiction, but today we will also see if it can recreate this scene from a true master of digital 3D art. So what is this light thing I keep talking about? A light transport simulation means a computer program that is able to compute the path of light rays to create beautiful images like this and this and this. And our key problem is that initially we only get noisy images and it can take a long time for the simulator to eliminate this noise. So can Dall E2 help with that? Well, how? For instance, it can perform a variant generation where in goes one image and the AI synthesizes other similar images. This is really cool as it means that the AI has a good understanding of what it sees and can create different variations of it and. Wait a minute. Are you thinking what I am thinking? Oh yes. Experiment number one, denoising lets give this noisy input image from a light transport simulator, give it to the variant generator and see if it is able to recreate the essence of the image, but without Denoise. Let's see. Well, that is super interesting. It did not Denoise the image, but it did something else. It tried to understand what the noise is in in this context and found it to be some sort of gold powder. How cool is that? Based on the insights gained here, let's try again with a little less noise. Oh yes. This difficult scene would normally take even up to days to compute correctly. Do you see these light streaks here? We would need to clean those up. So variant generation is. It's your turn again. And look at that. Wow. We get a noise free image that captured the essence of our input. I cannot believe it. So good. Interestingly, it did not ignore the light streaks, but it thought that this is the texture of the object and synthesized the new ones accordingly. This actually means that Dall E2 does what it is supposed to be doing. Faithfully reproducing the scene and putting a different spin on it. So cool. And I think this concept could be supercharged by generating such a noisy input quickly, then denoising it with one of those handcrafted techniques for these images. These are typically not perfect, but they may be just good enough to kickstart the variant generator. I would love to see some more detailed experiments in this direction. Now, what else can this do? Well, experiment number two, my favorite caustics. Oh yes. These are beautiful patterns of reflected light that we see a lot of in real life. And they produce some of the most beautiful images any light transport simulation can offer. Yes, that's right. With such a simulation we can compute this too. How cool is that? So now let's ask Dall E2 to create some of these for us. And the results are truly sublime. So, regular caustics. Check mark. And what about those fun heart shaped caustics? When we put a ring in the middle of an open book. My goodness. The AI understands that and it really works. Loving it. However, if you look at those beautiful volumetric caustics when running variant generation on that, it only kind of works. There are some rays of hope here, but otherwise I feel that the AI thinks that this is some sort of laser experiment instead. And also, don't forget about Daniel Baskin's amazing results. Who created these drinks. But wait, we are light transport researchers here, so we don't look at the drink. What do we look at? Yes, of course, the caustics. Beautiful. And if we are looking at beautiful things. Time for experiment number three. Subsurface scattering. What is that? Oh boy. Subsurface scattering is the beautiful effect of light penetrating our skin, milk and other materials and bouncing inside before coming out again. The lack of this effect is why the skin looks a little plasticky in older video games. However, light transport simulation researchers took care of that too. This is from our earlier paper with the Activision Blizzard game development company. This is the same phenomenon. A simulation without subsurface scattering. And this one is with simulating this effect and in real time. Beautiful. You can find the link to this paper in the video description. So can an AI pull this off today? That's impossible, right? Well, it seems so. If I plainly asked for subsurface scattering from Dall E2, I did not get any of that. However, when prompting a text to image AI, we have to know not only what we wish to see, but but how to get it out of the algorithm. So if we ask for translucent objects with strong backlighting. Bingo. Dall E2 can do this too. So good. Loving it. And now hold on to your papers because now is the time for our final experiment. Experiment Reproducing the work of a true master. If the previous experiment was nearly impossible. I really don't know what this is. Here is a beautiful little virtual world from Reynante Martinez. And it really speaks for itself. Now let's put it into the variant generator and see what Dall E2 is made of. Wow, look at that. These are incredibly good. Not as good as the master himself, but I think the first Law of Papers should be invoked here. Wait, what is that? The first Law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line and two more papers down the line. I have to say I can imagine that we will get comparable images. I also love how it thinks that fingerprints are part of the liquid. It is a bit of a limitation, but. But a really beautiful one. What a time to be alive. And we haven't even talked about indirect illumination, dispersion and many other amazing light transport effects. I really hope we will see some more experiments, perhaps from you fellow scholars in this direction too. By the way, I have a master level light transport simulation course for all of you. Free of charge, no strings attached. And we write a beautiful little simulator that can create this image and more. The link is in the video description. This episode has been supported by Runway Professional and Magical AI Video Editing for Everyone. I often hear you fellow scholars asking, okay, these AI techniques look great, but when do I get to use them? And the answer is right now. Runway is an amazing video editor that can do many of the things that you see here in this series. For instance, it can automatically replace the background behind a person. It can do inpainting for videos amazingly well and can do even text to image, image to image, you name it. No wonder it is used by editors, post production teams and creators at companies like cbs, Google Vox and many other. Make sure to go to runwayml.com papers, sign up and try it for free today. And here comes the best part. Use the code 2Minute at checkout and get 10% off your first month. Thanks for watching and for your generous support and I'll see you next time.
