Solved: The Bug That Haunted AI Video For Years
Dr. Károly Zsolnai-Fehér explains why AI video models generate stiff motion despite photorealistic frames: training data contains conflicting motion cues from cartoonish videos. A new approach uses optical flow to filter bad motion examples and compresses billions of internal signals down to hundreds of dimensions. Users preferred the cleaned model 74.1% of the time.
Chapters
Today, generating eye-poppingly high-quality videos just by writing a text prompt is possible. You can also get exceptional controllability as well. Also, you can generate 3 movies that look completely different, but land on the same ending. Almost anything you can think of becomes achievable, effortless, and inexpensive. Now, how they are kinda taking over the internet, is another story. But pretty much all of these systems have a huge problem. What is the problem? Is it issues with photorealism? No! In photorealism, these AIs are second to none. I am a light transport researcher by trade, I like to write programs that create photorealistic images, and I feel that many of their results are nearly impeccable. I spent more than a decade trying to learn this craft, and these AI systems are picking it up at an incredible speed. That is absolutely crazy. But, not so fast. What about motion? Well, now we got a problem. Motion breaks the spell. The frame looks right, but the movement feels wrong. And at this point, most AI researchers just say, no problem, just give it more training data, More compute. And we are done. Let's actually test that. This is the base amount of compute for OpenAI Sora from 2 years ago. Base amount of compute. Yuck! This is not great. And if you look closer, I think you shouldn't, but if you do, you notice that this is what nightmares are made of. Now, if we add 4 times more compute, We get this. Perfect? Not even close. But the trend is shouting at us. Now, with 32 times more compute, we get this. Now we're talking! The results start to sing. So, case closed, right? If the motion is not good, and if you don't have more compute, because who does these days, well then, let's just add more training data. Let it look and learn some more. Except that this is completely wrong. This is what this paper is about. When we see an AI generate motion, they developed a technique that is able to ask, okay little AI, where did you learn that? I love that! Let me give you an example. A foam cube floating on water. And it gives us waves crashing over a pier, surfing, and splashing ocean waves. This is so cool! So this is where the knowledge came from. But wait, they say that if there are positive examples for your learning, I wonder what negative samples look like? Oh, this makes sense! These are really the worst for learning! Why? Because cartoons, for instance, teach completely conflicting information about physics. In cartoons, characters pause mid-air before falling, maybe even holding a tiny little umbrella. Bodies bounce like rubber and snap back into their original shape a moment later. Fun for us, not so fun for an AI model trying to learn real physics. Wait a second. I have an idea. What if we don't just put in more training data? What if we give it less? Cut out those bad influences. Can it do better? Well, let's try it out together. Yes, with the base model, we get a coin which is spinning around the wrong axis. And now? Hold on to your papers, Fellow Scholars, because here comes the magic. After cutting out these bad influences, plus fine-tuning the AI with the good ones, look at that. That is a beautiful spinning coin. And I gotta say, I was a bit less impressed by the ball example. Yes, the new one is better, but we have seen plenty of systems pull off this kind of movement. In any case, we are Fellow Scholars here, we don't hand out medals for a couple cherry-picked examples. No! We are more rigorous than that. We look at the research paper. Does the paper deliver? Oh yes! Yes, it does! I look at the user study and see that it lands the punch. They asked people to judge whether the new or previous method was better. They did it across 50 videos and 17 participants. That is 850 little tests. And drumroll, it has a 74.1% win rate over the original. That is stunning! Okay, so how on earth did they do that? Can we catch an AI in the act of remembering? Is that even possible? And what does that mean for us? Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Now that's a late cold open. Alright. Now, they did two things to ensure that this concept works properly. One, you need to be able to separate how things move from how things look. To do that, they introduce a motion masking step through a technique we call optical flow. An old idea, but works great for tracking the path of points over a video. Good call. But here is the genius part. They don't apply this mask to the video itself. Nope. Instead, they apply that mask to the internal learning signals of the AI. This helps them discover where decisions are coming from. Genius idea, yes. But unfortunately, too, there is a huge problem with this. What is the problem? Well, modern AI models have over 1 billion parameters. Storing and comparing the full learning signals for thousands of videos just takes too much computer memory and time. That's crazy town. Not feasible. Instead, they found a way to, get this, compress down these more than a billion numbers into— excuse me? Am I seeing correctly? That's right. 512. Down from more than a billion. And the results are almost the same. Wow! That is insane! The technique they use is called the Johnson-Lindenstrauss projection, and it was used in Google's TurboQuant compression algorithm as well. That is the one to ease memory constraints of large language models on your GPU. What does it do? What it does is it shrinks high-dimensional data into a tiny space, but in a way that it preserves the relative distance between these numbers. Picture a wooden chair. The chair lives in 3D. The shadow lives in 2D. Now, the shadow needs much less data. And if the scene is set up right, the distance between the 4 chair legs remains exactly the same. And that means that the projection allows us to retain important properties of the data, but cut away a lot of fat. And all this is put together to achieve one thing: to be able to find what videos influenced the to make fast decisions. And then, to cut away all the junk knowledge. And that is also super important for our thinking. You see, there are topics where I hoped that the more I read, the smarter I would get. Read more, grow wiser. Not true. There are many areas where the more I read, the more I found that I just got stupider. It took me years and years to find out that there are topics where you can read and learn all you want, if the quality of information is low, it does not educate, instead, it deforms your thinking. So, what is the solution? Well, you need to be able to separate the real from the fantasy. You don't need more, you need less, and you need better. Like you saw in the paper, truth is the best teacher. And you don't need a lot of it. This technique just showed that a tiny clean signal beats a mountain of junk. Slow down, don't take everything in. Try to verify what you actually hear, and try to take in less. To me, this is the main message of the paper. Brilliant work, brilliant lesson. Love it. And they promise that we'll get the code for free. What a time to be alive! Here you see me running the full DeepSeq AI model through Lambda GPU cloud. 671 billion parameters, running super fast and super reliably. This is insane. I love it. And I use it on a regular basis. Lambda provides you with powerful NVIDIA GPUs to run your own chatbots and experiments. Seriously, try it out now at lambda.ai/papers or click the link in the description.
