A Two Minute Papers video (Two Minute Papers with Dr. Zsolnai) reviews a recent collaboration led by NVIDIA (with four universities) that uses diffusion-based models plus a closed-loop corrective controller to generate believable character animations from text prompts. The host demos several behaviors (squats, zombie walk, couch potato, walking/jogging/waltzing to a point, punches, kicks), explains that diffusion models start from noise and suffer residual artifacts (notably foot pops), and describes how the controller corrections dramatically raise success rates (previously ~2% for getting up from the couch to ~98%, with some tasks reaching 100%). The paper, source code, and instructions are publicly available; the video also mentions Lambda Labs as a way to run experiments on Nvidia GPUs. Remaining limitations include some wobbliness in motions and imperfect punch force.
Questions this video answers
January 19, 2025
NVIDIA and four other universities
From ~2% (previous techniques) to ~98% (new technique)
Yes — the research paper and source code are available free of charge
Chapters
This new AI based technique enables us to write a piece of text and animate a virtual character to do exactly as we say. So first do a squat. Yup, that does it for the exercise today. We are fellow scholars after all. Now walk like a zombie. That looks good. Now end of shift. Engage couch potato mode. Perfect. Now I only need a bunch of research papers to read and I am a happy man. Now there is one problem here. What is the problem? What you just saw is nearly impossible to pull off. Why is that? Well, let's use a previous technique and I want to see a bloodthirsty warrior mercilessly destroying its enemies. Well, this is not that. And this technique was published less than a year ago. So you fellow scholars see that this is a tough, tough problem if you want to use a diffusion based AI model. And this paper is an incredible collaboration between between Nvidia and four other universities to crack the code to creating believable animations with diffusion based AI techniques. As many of you know, these start out from noise, but this noise in this case does not create an image, but character animation instead. As you know, these diffusion based methods can be used for way more than just images. For instance, they can even take a piece of noise and create geometry out of it. So, so why not animation? Well, here is the reason. Oh yes, we can start out from noise and slowly eliminate it, but still at the end some noise remains and we have some consistency issues. So a diffusion model by itself is not sufficient here. But this new technique has a brilliant idea to fix it. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. What is the brilliant idea? Well, look at the blue markers here. This is what the noisy diffusion model says the character should be doing. Actually it's pretty good, but oh my, do you see what I see here? The left and right foot, at some point just make a pop. Yup, artifacting issues. So here is where this new technique comes into play. Oh yes, it fixes these problems, finally creating a believable movement sequence. Absolutely loving this. And this is why it is called closed. Because a new controller system works together with this diffusion model to correct it. Ok, so what can it do? Well, it can create a sequence where for instance, we walk from A to B. Okay, but we can already do that. So is that it? Why is that useful? Now hold on to your papers fellow scholars, because there are many ways to take this journey. And we can add a simple piece of text that tells the character how to do that. Look at that. What a beauty. Hopping, walking, jogging, or it can even waltz to the desired point. So cool. So we can tell it to sit down on the sofa and relax. I said relax. Oh, now we're talking. And those merciless attacks can now not only be punches, these finally work now. But with the power of text prompting, it can also push with two hands or even do a high kick. So is it perfect? Well, let's have a look together. Now, I gotta say, these punches are not something that strike fear in my heart. They wouldn't do a lot of damage. Maybe if a fight breaks out in a virtual retirement home, this could be appropriate. So is that perfect? Not even close. I mean, look at how wobbly these moves still are. No wonder, because this is a research paper that takes something that was previously impossible and makes it possible. So, yes, this is early. You will likely not see this in the games coming next week. But don't forget one thing. AI and computer graphics research progresses incredibly quickly. Here you see what AI video generation was capable of just a year ago and today. Wow. So don't just look at what we can do now. Try to think two more papers down the line and imagine what we will be capable of. That is the first law of papers. Now this is a good paper. So it describes how each individual proposed technique changes the outcome for the better. You can see these little pieces in action. But it also documents how the success rates increase with these correction steps. I mean, previous techniques could not get out of the couch at all, or only with a 2% success rate. And now the new one. Oh my. Look, it can get up from the couch 98% of the time. That's better than what most humans can do, isn't it? So from now on, no more couch potato AIs are allowed. And there are cases where we are up to a hundred percent. Incredible. And all this in just one paper. What a time to be alive. And yes, here we love taking a deeper look at papers and measurements. Welcome to Two Minute Papers. And they did not keep the details to themselves. The entire research paper and source code for the project is available for all of us free of charge. Thank you. So in future video games, characters may finally be able to perform new moves that they were never programmed to do. So what would you fellow scholars use this for? Let me know in the comments below. To run your own experiments on an Nvidia gpu. Check out Lamda. I use it myself regularly for these videos. Hmm, look at that. You can generate high quality images in less than a second per image. I did a ton more of them and paid. Paid less than a dollar for all this crazy. Seriously. Try it out now@lambdalabs.com papers or click the link in the description.
