NVIDIA researchers build photorealistic 3D worlds from Minecraft-like blocks, then use them to train self-driving cars and robots. The process starts with simple geometry, adds physics simulations and hair-strand detail, then upsamples everything to look real. They also released NVLM, a vision-language model they say rivals ChatGPT.
Today we will see how we can train a real self driving car by playing a computer game for free. A Minecraft style game, Kind of. But how does that even make sense? The real world is detailed and Minecraft worlds. Well, not so much. So how could an AI possibly learn anything from this? We'll find out how and then we are going to see how Nvidia also gave us a free AI that could be a challenger to even ChatGPT itself. Wow. And then at the end we get a virtual haircut so you can look gorgeous online and then find new ways to ruin it. But what is going on here and why? I'll tell you now, this is an absolute blockbuster research work and it goes from Minecraft style worlds to this somehow. But how? And why? Well, first of course, scientists at Nvidia are really smart and any excuse to play video games at work is great. Great in my book. Good job. So the goal here is to create a virtual video game version of the real world to train AIs and cute little robots in there. First we can start by synthesizing a collection of blocks like in Minecraft. But there is one problem. Obviously the world is so much more detailed than that. No matter, because this technique can generate new worlds which are really blocky at first, and then subdivide these blocks into smaller ones and even smaller ones. And now we have a much more detailed world. Or we can take a bunch of points called a point cloud, and from that create this thing called a triangle mesh. But that's still not too detailed. And now hold on to your papers, fellow scholars, because then they gave it to an AI based technique and this came out. Wow. Now this game is good enough so it can be used to train self driving cars and teach them to handle unexpectedly expected scenarios safely in a simulation before moving them into the real world. But not so fast. This virtual world learning trick is also one of the key reasons their humanoid robots started to work. Now, whatever you ask for, remember to always say please to your toaster so it might remember and spare you in the future. But there is a problem. A huge problem. It just doesn't fit. You can model a huge city with their new technique, but that doesn't mean that it can fit into the memory of just one computer. So does it? Ouch. Only with this quality, which is not great. It's like squeezing an elephant into a mini Cooper. But if you have a huge virtual world that does not fit on one graphics card, you can distribute it among several cards. So does it work? Oh my. Just look at that realism with eight graphics cards. And I thought I only needed to sell one kidney to buy these. Time to look for the guy with a big trench coat that can house a lot of them. But it does not end there. This work can also do souping fine details onto it. No sir, they have done it for drumroll physics simulations. Oh baby. The two minute papers special. Look at how much more detailed it suddenly got. Loving it. And they just give this all away to all of us for free. Even the source code is absolutely amazing. But we are not done yet. Not even close. Now one more amazing thing that we just got for free. Oh my goodness. Nvidia also released nvlm, an AI for free that rivals and in some cases even beats the Great ChatGPT. This is unbelievable. And it also has the ability to see. So you can ask it to explain memes or to describe an image in great detail. So I plug this into an AI on Lambda GPU cloud to recreate it. And I gotta say that is not Jensen at all. But their AI got the rest of the composition down real good. There are other completely free models that do all this too. So we are absolutely spoiled here. This can even call in sick for you while you play video games at home. What a time to be alive. Now time for a haircut. What is going on here? Well, this technique looks at your hair and promises to reconstruct it strand by strand so you can get a digital copy of it. But wait, this is not the first technique to do this. However, look, previous methods did not do too well in this. And the new one. Yeah, much better. And here is my favorite part. Now let's use it to put us into a virtual video game world. And we can simulate it in different lighting conditions. And the best of them all bouncy physics simulations. Loving it. Or if you want to ruin it, there are already works on simulating liquid hair interactions and they are absolutely gorgeous. I mean, look at that. Yes, bad hair days can still be beautiful. And one more thing, look. This takes a few hours on a single graphics card. So fellow scholars, you know what that means? Yep. Back to the guy with the trench coat. Alright. And if you feel inspired and you wish to run your own experiments on Nvidia GPUs, check out Lambda. Lambda is a cloud service for large scale AI projects and it is my absolute favorite. Seriously, try it out now@lambdalabs.com papers or click the link in the description. If you use Lambda, you can run your own experiments on super powerful hardware. And you also support two minute papers.
