NVIDIA’s New AI Grows Objects Out Of Nothing! 🤖

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see how Nvidia's new AI transfers real objects into a virtual world. So what is going on here? Simple. In goes just one image or a set of images of an object. And the result is that an AI really transfers this real object to into a virtual world almost immediately. Now that sounds like science fiction. How is that even possible? Well, with this earlier work, it was possible to take a target geometry from somewhere and obtain a digital version of it by growing it out of nothing. This work reconstructed its geometry really well. But geometry only. This other work tried to reconstruct not just the geometry, but everything. For instance, the material models too. Now, incredible as this work is, it is still baby steps in this area. As you see, both the geometry and the materials are still quite coarse. So is that it then? Is the transferring real objects into virtual worlds dream dead? It seems so. Why? Because we either have to throw out the materials to get a really high quality result, or if we wish to get everything, we have to be OK with a coarse result. But I wonder, can this be improved somehow? Well, let's find out together. And here it is. Nvidia's new work tries to take the best of both worlds. What does that mean? Well, they promise to reconstruct absolutely everything. Geometry, materials and even the lighting setup. And all of this with high fidelity. Well, that sounds absolutely amazing, but I will believe it when I see it. Let's see together. Well, that's not quite what we are looking for, is it? This isn't great, but this is just the start. Now hold on to your papers and marvel at how the AI improves this result over time. Oh yes, this is getting better. And my goodness, after just as little as two minutes, we already have a usable model. That is so cool. I love it. We go on a quick bathroom break and the AI does all the hard work for us. Absolutely amazing. And it gets even better. Well, if we are okay with not a quick bathroom break, but with taking a nap, we get this just an hour later. And if that is at all possible, it gets even better than that. How is it possible? Well, imagine that we have a bunch of photos of a historical artifact and you know what's coming? Of course. Creating a virtual version of it and dropping it into a physics simulation engine where we can even edit this material or embed it into a class simulation. How cool is that? And I can't believe it. It still doesn't stop there. We can even change the lighting around it and see what it would look like in all its glory. That is absolutely beautiful. Loving it. And if we have a hot dog somewhere and we already created a virtual version of it, but now what do we do with it? Of course we engage in the favorite pastime of the computer graphics researcher, that is throwing jelly boxes at it. And with this new technique, you can do that too. And even better, we can take an already existing solid object and reimagine it as if it were made of jelly. No problem at all. And you know what? It is final boss time. Let's not just reconstruct an object. Why not throw a an entire scene at the AI, See if it buckles. Can it deal with that? Let's see. And I cannot believe what I am seeing here. It resembles the original reference scene so well, even when animated, that it is almost impossible to find any differences. Have you found any? I have to say I doubt that because I have swapped the labels. Oh yes, this is not the reconstruction. This is the real reconstruction. This will be an absolutely incredible tool in democratizing, creating virtual worlds and giving it into the hands of everyone. Bravo Nvidia. So what do you think? Does this get your mind going? What else could this be useful for? What do we expect to happen? A couple more papers down the line, Please let me know in the comments below. I'd love to hear your thoughts. What you see here is a report of this exact paper we have talked about, which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system, you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.