NVIDIA's new AI generates textured 3D models of cars, animals, and chairs from text. A single technique now handles multiple tasks that previously needed separate algorithms, and it supports geometry-vs-texture awareness and latent-space interpolation.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see how this new AI can create entire virtual worlds out of thin air. And to do all that, we don't even need to be a professional artist. This can do so much. But how much exactly? Well, let's give it a hard time and see what it can do through four really cool examples. And then it gets even better. I'll tell you in a moment why. One, as you see here, it can generate a great variety of different object types. Cars, animals, chairs, you name it. And would you look at that. I am very happy because I am already seeing two really nice features here. Do you see them too? One is that these are textured objects. Most previous techniques just give us the geometry, but with this one, we get nicely textured objects. That is super nice. But look, these are also regular 3D meshes, which means a data structure that can be used by most 3D modeling systems right away. As it is, that is not true for all previous techniques. For instance, some of them even give us point clouds instead. And textures aside, if you compare the quality of the results to these previous methods, there is a night and day difference. This is so much better. So cool. Now too. The fact that it can generate a great variety of objects is all well and good, but what if we kind of like this object and find it to be almost exactly what we like, but we would wish to change it a little? Can we do that? Well, that's tough. For that, we would need a latent space interpolation technique that is akin to this earlier fun generation method. Or like this material generator from our earlier paper, where we can walk around on this 2D plane and fine tune a material that we already like. Doing the same with these objects would be fantastic. But let's not kid ourselves. That is significantly harder in this context. So can it do it? And the answer. Look at that. Yes, it can. And what is even more impressive is that nearly all of the intermediate results make sense. That is the hardest part. And if it stopped there, that would already be amazing. But it doesn't stop there. There is so much more. 3. Get this. It even understands the difference between geometry and and texture. This means that whenever we modify the geometry, the lamps still have the appropriate texture and colors. And it also understands that as we create an SUV from a smaller car, the doors and windows now have different placements. This is an excellent showcase of an AI technique that actually understands what it is generating. And now here comes my favorite. They also promise text Guided shape generation. Hmm, does this ring a bell? Oh, yes. We can prompt this technique like. We can prompt a text to image AI like Dall E too. That would be amazing. But have a look. This is a previous technique trying to give different animals tiger stripes or. Actually, I am not even sure what this is trying to do. Or make these dogs a bit fluffier. Those are noble goals. But unfortunately, this previous technique could not pull this off. Not even close. And now hold on to your papers and let's have a look at the new technique's results. Oh my goodness. There is no contest here. Tiger stripes, fluffy dogs, you name it. With this new method, we can create a tiger or a panda out of any other animal. Or ask for a brick house, or burn it down by just writing these words and pressing enter. This is incredible. We just write the text, the AI does all the hard work, and we take credit for it. How cool is that? And wait a second. We noted that this was the best of what a previous method could do. How long ago is this paper from? And this is where I fell off the chair when reading this new paper. This previous method is not even a year old. It was published approximately 11 months ago. And this kind of progress in just a year? I am stunned. And wait. I promise to tell you how it gets even better. Well, it does all these amazing things. And not only that, but normally with previous handcrafted techniques, if we wanted to do these four things, we would have needed four different algorithms for that years ago. If we wanted to conquer Chess and Starcraft and Dota 2, these are three different applications and thus need three different handcrafted algorithms to solve. However, this is not the case here, because this new technique can do all of them at the same time. This is not four tasks and four techniques. This is just one technique that is super important. Why? Well, there are many definitions for artificial intelligence, but most of them contain some sort of generality. A good AI technique should be able to learn from a large data set and obtain general knowledge from it. And this is an excellent testament of that. And just imagine what this technique will be able to do just a couple more papers down the line. What a time to be alive. This episode has been supported by Runway Professional and Magical AI Video Editing for Everyone. I often hear you fellow scholars asking, okay, these AI techniques look great, but when do I get to use them? And the answer is right now. Runway is an amazing video editor that can do many of the things that you see here in this series. For instance, it can automatically replace the background behind a person. It can do inpainting for videos amazingly well and can do even text to image, image to image, you name it. No wonder it is used by editors, post production teams and creators at companies like cbs, Google, Vox and many other. Make sure to go to runwayml.com papers, sign up and try it for free today. And here comes the best part. Use the code 2Minute at checkout and get 10% off your first month. Thanks for watching and for your generous support and I'll see you next time.
