Dr. Zsolnai explains inverse rendering, a technique that reconstructs 3D scenes, geometry, and materials from images. Researchers at UC Irvine and Nvidia can infer object shape from shadows alone, and the source code is available. Runtimes range from minutes to two hours.
In computer games and animation movies, we have a 3D scene with objects and materials and we wish to see an image how it would look in reality. That is called rendering. One of my favorite ways to do rendering is through ray tracing, which is an incredible technique that simulates how light interacts with the scene and gives you these beautiful results. That was rendering scene goes in, an image comes out. But now imagine the opposite. What is this crazy talk? Well, hear me out. Imagine that you have images and you want to get what the scene was behind the image. Get a digital 3D model of the scene. An image goes in and a scene comes out. And imagine how cool that would be when you want to create a video game in, you just make an image and out comes the scene for the game. However, not so fast, dear fellow scholars. This is Two Minute Papers with Dr. Zsolnai. And here the question what is the geometry there? What materials are they made of? Lighting, everything that needs an expert and many, many work hours. Here you see Andrew Price assembling such a scene in Blender, a 3D editor program. Imagine that you have a photo of the result. Then you have to do all this sculpting, assigning materials, lighting and more. And then when you are done rendering, but at this point you just think that you are done because your results are obviously not the same as the target image. So you have to play with the geometry, lighting and materials and render the image again. This takes forever, hours, days, even weeks, depending on the complexity of the scene. And it is super challenging. So we look at a photo and build a scene by hand. I imagine that in a science fiction world we could perhaps have an amazing algorithm that could do this automatically. Lets call this inverse rendering. How would that even work? Well, few of you fellow scholars know there are previous works that do that. They are absolutely amazing. You see how these beautiful pieces of geometry grow out of nothing. You just specify what you want with a 2D image and it creates a 3D model. And we are talking detailed 3D models here. Super incredible. But this is just 3D modeling. What about materials? And now have a look at this research paper from the University of California, Irvine and Nvidia. And it can do some amazing things. First, if we put a few light sources on a painting, it can reconstruct the painting itself. It is similarly good with reconstructing the material of this object too if it can take a look at a set of images of it. But here come my favorites. Here is a tree. Now hold on to your papers, fellow scholars, because the task is super challenging. What you get is not even a view of the plant. No, no. You get a look at just the shadow of the plant. Now figure out what the plant looks like. Previous techniques couldn't do it. They were just trying and trying and. And not getting anywhere. But the new method, first it tries to sculpt the object in different possible ways to be able to match its shadow. And as it does that, you can see its current gas for the geometry of the tree itself. So can it do. Can. Wow. Absolutely amazing. And all this from just seeing a shadow. The processes that you see here are sped up and this one took only 16 minutes. I think for a human being, doing this sounds almost impossible in any amount of time, let alone in a few minutes. Incredible. Now we have two more really cool tests for it. And then a huge surprise. Here is one more case where we reconstruct an octagon. And by merely looking at its shadow again, this one checks out. And now we have a world map relief. In a large room with a window. It gets to look at some images of it. But ultimately what it needs to reconstruct is the bumps in the geometry that create this effect. And I love seeing how it morphs this solution into being so cool. And since the quickest scene goes from 12 minutes up to about 2 hours, that can already be useful for some tasks. But just imagine what we will be capable of two more papers down the line. My goodness. So we don't need to live in a science fiction world for a technique like this. It is right here. I am getting goosebumps. And it is clearly not as good as Andrew. No one is. But it is an amazing step forward in creating virtual worlds, maybe even video games from just an image or a drawing. Don't forget, scientists at Google DeepMind are already working on the video game part. And good news, the source code is also available. So we get all this knowledge for free. I don't know if I told you before, but I love research papers. And if you wish to run your own experiments in the cloud, make sure to check out Microsoft Azure AI. Azure AI is a powerful cloud platform that offers you the best tools for your AI projects with responsible AI built in. And here comes the best part. You can even try it out for free through the link in the video description.
