NVIDIA’s New AI: Impossible Video Game Animations!

This amazing fresh new AI work from Nvidia just came out and it is unbelievable and can do the craziest things. My goodness, this is amazing. Okay, what is going on here? Well, we have seen earlier AI based works about text to motion. Writing a short prompt and then it makes a virtual character move accordingly. But I bet you haven't seen this. What is this? This is called genmo and it is not text to motion. No, no, this is everything to motion. For instance, one, you can start out from a recorded video of yourself and it learns and transfers those moves to a virtual character. That is crazy because it has to look at 2D pixels and convert them into moving around with joints and limbs in 3D. So we climb the stairs. Nice. Now what? Two. Now we want to do a lunge on one leg, but we are a bit lazy. Do you know what? Just add the text prompt and ask the AI to do it. This is incredible. But it gets better. Now three, add some music as an input. That of course I can't show you because you know the copyright situation on YouTube. So is this good? Well, this is how I walk to work every day. Because doing this is my dream job. And now phase four. Now that's what I call a more challenging case. And wow. I am very impressed by how well it is able to look at a video, no other data and gives you the right solution. I am loving this already. Now let's start with something simple but funny. Little AI, climb up the invisible stairs. Now realize that you are standing on invisible stairs. And then see how much weight it can support. Excellent construction. And now let's make it harder. See these silhouettes? These are keyframes. It has to take these positions at the exact time on the timeline we said also different kinds of inputs. Now I really want to see this first phase walk and look around. Nothing crazy there. And now. Oh my. Nailed it. Next one also fine. And then phase three with the video. This is fantastic. And now comes the interesting part. You see, it's tough to be able to take all of these inputs. But the real challenge is weaving them together at these breakpoints. And if you look carefully now, you will truly understand why this paper is absolute magic. Look, we change the initial footage, then we transition into the lunges and oh my goodness. It takes the style of the previous motion into account and does the lunges accordingly. Absolute magic. I can imagine just firing up a lambda instance and getting all of these results in just a few seconds. Wow. And now this is where the magic happens. If you created your scene and you are not happy with the timings. You can just edit it. Look super intuitive. Also super challenging. Not for you, but for the AI. But you see, with this, you can't just cut off part of the animation as it gets shortened. No, it has to redo it from scratch and make it transition into the next motion seamlessly. So is it seamless? Let's see. Hmm. I think it is as smooth a transition as it gets. Bravo. So let's pop the question. Can it actually deal with real dancing movements of people who do this for a living? Let's start with the easier case. It can dance. Cha cha cha. I can recognize it being cha cha cha. So that's good. But now hold on to your papers, fellow scholars, and look at this. Real dancers from real clips that cannot possibly work. Or so I thought at first. These are the best results I've seen in this area. I cannot stop looking at them. And don't forget, this is not pose estimation, not just marking up the video with a doll in 2D. These are actual 3D joint and limb movements that are produced by the AI. Stunning. But it gets funnier. How? Dear fellow scholars, this is Two Minute Papers with Dr. Caro Zsolnai. Check this one out. It is now acting like a monkey. Hmm. This person looks like they are typing something on a giant keyboard. Oh no. He is posting some nonsense about AI on Twitter. Hmm. Actually that's right. I think this one is the most realistic of them all. It is acting like a monkey, isn't he? Bravo. And if you are not a dancer, don't worry. Just sit in your chair with a coffee and write as many prompts as you want, even in short segments. And it doesn't seem to break a sweat. An absolutely fantastic AI contribution to computer games and virtual worlds. It looks like the real deal and we still got it earlier than GTA 6. What a time to be alive. Now limitations. It only handles full body motion, no facial gestures, no hand articulation. I think that is quite clear. It is not yet an end to end solution as it also relies on an off the shelf slam method that is simultaneous localization and mapping. This helps extracting useful information out of the videos like camera position and direction. And hand it off to this AI technique. In short, it cannot do every processing needed alone. But it is close. Now. This also has a heavy diffusion backbone. It needs five denoising steps. So I think it might not be real time yet, but nearly there. Once again, super fresh new paper and almost nobody is talking about this. That's crazy town and that's why so many of you come and watch Two Minute Papers and I thank you for the honor for coming with us all this journey for such a long time. Now as this is paper and episode number 962. Yes, almost a thousand papers and you fellow scholars just keep showing up. Huge honor. Thank you so much. Now I did not see that the source code would be published for this work, but knowing Nvidia, this is the kind of work they usually give us the source code for. So fingers crossed and then just imagine what we will be capable of just two more papers down the line. That is the first law of Papers. Here you see me running the full deep seq AI model through lambda GPU cloud 671 billion parameters running super fast and super reliably. This is insane. I love it and I use it on a regular basis. Lamda provides you with powerful Nvidia GPUs to run your own chatbots and experiments. Seriously. Try it out now at Lambda AI Papers or click the link in the description.