A Two Minute Papers-style YouTube video by Károly Zsolnai-Fehér (LARRUZ) presenting a new NVIDIA research paper that produces a versatile virtual character capable of many motions (walking, sitting, cartwheels, dancing) and adapting to new terrains. The video also explains a 3D denoising approach that yields textured 3D models (text→3D/materials) and showcases a browser-based world-builder that generates playable 3D worlds from text or images ("text-to-everything"). The presenter demos features, cautions that research is experimental, and links to try the tools and to Lambda Labs for compute.
Chapters
Today we will see a virtual character who thinks he is king learning to do crazy things like this. And this. Just promise to not ask it to perform a cartwheel down the stairs because ouch. And we are also going to see what the heck this is. And then this crazy world builder AI that you can try right now. And this problem is horrible. Hard. Like really hard. Previous techniques are quite limited. For instance, this one is trained with reinforcement learning and can do locomotion really well. And now you want running or jumping? Anything else? Nope. When they try something new, they look like these scholars who have completely lost their papers. This is a hard problem. But this new Nvidia paper is something else. This guy is the king of computer animation. Well, so it thinks. Let's see if it really is true. It can walk naturally. That's good. And we can also ask it to sit on his throne. And after he is done with his kingly duties, he can also sit down and watch two minute papers with the bros. So far, so good. Now you wish to see a cartwheel and there you go. But that's not the only thing it can do. For instance, it was trained on a flat surface. And when asked here to reach the target location as quickly as it can after that, what did it learn later? Hmm. Look. It can not only perform new kinds of motions, but it can also deal with new terrains. Well, a bit like a drunkard, but can still hold balance. That is fantastic. And here is where it gets weird. This new technique gives us one AI that can perform a wide range of tasks. How wide? Well, it also has dancing skills. Hmm, maybe it is a little better on gravel. Well, it's okay, but not great. Who's gonna tell the little AI he thinks he is king, so. Not me. Why? See what it can do. I don't want to be karate chop down like that. Especially that it can hold its balance even on gravel. You don't want to mess with this guy. And wait, it gets even better. If you are thinking about Text to Image get out of here. Text to motion is where it's at. Just write what you want and you shall get it. Once again, please do not ask it to perform a cartwheel down the stairs. Promise. And now, what is going on here? What is this insanity? Believe it or not, this is a denoising process. Text to image AIs start out from a bunch of noise and over time they remove the noise and we get an image. And what you see here is noise. But in 3D. If we remove that noise over time, what do we get a 3D model? But it doesn't only do shape, it does material synthesis too. They have their own materials, so you can even put them in a virtual environment. And lighting will also have an effect on them. Beautiful specular highlights right there. So good. So we had text to motion, text to 3D and now hold on to your papers fellow scholars for Text two everything. Yep, that's right. I don't even know why we bother with anything else here. You can start out with an input image of a place and as you move around it generates the world on the fly. A 3D world that you can play in. This world can be a real place, it can be a painting, or it can be a Minecraft style world too. And now tell me that we are not living in a simulation. And great news. You can try it right now and build your world in your browser through the link in the description. And you can also choose from a lot of different gorgeous styles. Please just don't build a world full of stairs. And this is a magic button. Believe it or not, if you click this you get sued by Nintendo. And with tools like this, I am really excited for the future. I. I mean how cool is this? You can just write a piece of text and create a world. Create characters within that world, move that character and create your own little games. None of these techniques are perfect currently. This is research after all. But just imagine what we will be able to do a couple more papers down the line. That is the first law of papers. What a time to be alive. And. And let me know in the comments below what you fellow scholars would use this for. And if you feel inspired and you wish to run your own experiments on Nvidia GPUs, check out Lamda Lambda is a cloud service for large scale AI projects and it is my absolute favorite. Seriously. Try it out now@lambdalabs.com papers or click the link in the description. If you if you use Lambda, you can run your own experiments on super powerful hardware and you also support two minute papers.
