NVIDIA Omniverse: Virtual Worlds Come Alive!

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to talk about the incredible things Nvidia showcased at their recent keynote at Siggraph, the world's premier computer graphics conference. Now, I am a computer graphics researcher by trade. I work in light transport simulations, in other words, ray tracing. So many of the things that you will see here today makes me very, very happy. For instance, this is a showcase of what their Omniverse system can do. Now this tool is for building virtual worlds. It can do a lot, from storing volumetric effects to simulating them, to animating robot hands, materials, animation, high quality geometry, and hair simulations too. The incredible scenes that you see here were built by USD Composer. This is an Omniverse app to build large scale scenes. And now hold on to your papers, fellow scholars, because some of these virtual worlds are becoming incredibly convincing. Now what does that even mean? Well, I talked about simulations that are almost indistinguishable from reality in a little paper not so long ago. Yes, what you see here is a computer simulation, not reality. The link to the paper is available in the video description. So today we can scientists at Nvidia perform something similar. How good are their virtual worlds? So let's see what they have here. One Open USD. USD stands for Universal Scene Descriptor. And this supports creating virtual worlds in a way such that visual effects people, designers, engineers and roboticists can work together easily. For instance, this is a virtual car that is almost the same as the real car it represents and it is assembled from the smallest individual parts. Then artists can see what the car would look like in reality, create new variants of it, create a gorgeous backdrop where they would showcase this car, try different environments, and it is even possible to take it for a ride. And what really blows my mind is not only that all this is possible, but that all this is possible today in real time. Everything runs as we are looking at it. Wow. Number two. Now if this system can create really detailed geometry, that is great. However, not everyone is a CAD designer or understand materials as a physicist does, but everyone can speak English. So they have this new interface where all we need to know is English. And thus we can now get a 3D floor plan, generate lighting through the free and open source blender, and through Adobe Firefly integration, we can get floor textures by just typing. I feel like we are going towards a future where English is going to be the interface of almost everything. You won't need to be an expert at everything. You will have an expert AI assistant at your fingertips. With a ton of knowledge and infinite patience ready to work for you. How amazing is that? And you can simulate not only how things look, but how things work too. Incredible. Number three, they also showed us something that they call Workbench. This enables fine tuning already existing AI models by giving them additional knowledge. For instance, with a text to image model. If we are asking for a toy Jensen in outer space, we get this. Wait a minute. That's not the Jensen we are looking for, is it? Now of course there are. This is because this model has never heard of Jensen before. So we can teach it about him using these eight images. Then ask again and bam. There we go. Into outer space we go. Of course, they have also showcased shiny new hardware. Well, mostly shiny. And finally, my favorite. If I hadn't seen anything else, just this one thing you are seeing here, that would have been enough for me to create a video about this keynote. And this is a video honoring some of the incredible works in computer graphics. Back from the olden times. With the incredible animated human hand back from 1972. From the legendary Ed Catmull, co founder of Pixar, the first ray tracing program from Turner with it from 1979. Imagine simulating refractions on a computer more than 40 years ago. Wow. And it goes up to what we are capable of today. I am so happy to be able to share all of this with you. And just imagine what we will be capable of just two more papers down the line. My goodness, what a time to be alive. Weights and Biases provides tools to track your experiments in your deep learning projects. And they now have a one hour course with the legendary scientist Andrew Ng on how to perform things that we are talking about in these videos. For instance, they teach you how to train diffusion models, how to run and evaluate large language models and even fine tune them for your tasks and more. You can even run these code snippets as you are watching the series. It is fantastic. Make sure to visit them through WNB Me papercourse or just click the link in the video description and you can take this course for free. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.