Meta's VideoJam generates videos with realistic physics: water splashing, bubbles floating, a roller-skating raccoon. It uses a technique called inner guidance to keep motion consistent across frames, and early results rival OpenAI's Sora and DeepMind's VO2.
Now check this out, fellow scholars. My goodness. These stunning results were all made by a new text to video AI called VideoJam. Now the first question is of course, can it compete with OpenAI's Sora? That is a groundbreaking system that was absolutely amazing at remembering details. You look at something, it gets occluded and bam. It has to come back the same. However, it still has its flaws. There are consistency issues. Ouch. Also, sometimes prompt comprehension is not the best here. Pull ups are supposed to happen, but it doesn't. So our question is, can this new system out Sora. Sora. Well, let's see together. Oof. With Sora, this footage is not usable unless you are making a horror movie. But with VideoJam, oooo. There is no contest. And when trying the Juggler, we get very similar results. And the new one is so good I have to slow it down and look at it frame by frame. Yes, we are now at the point where we have to be pixel peeping to find out that this is not real. And it has a much better understanding of motion and physics. When we pour water into glass, just wow. Look at how beautiful it is. How it even models how bubbles are formed. But that is nothing compared to what you are going to see now. You see, I spent many years of my life running lighting up computer simulations for things like water drops and crown splashes. They are incredibly complex. Here is an earlier paper that can do it. But the amount of expertise required for that is just ridiculous. Well, not anymore. Look, it looked at many videos from real life and it now understands that too. Something that took me years to understand. And now you just enter a prompt and the AI just gets it and aha. Did you catch it? We found a little weakness. There is a little pop every few frames. And when it comes to understanding the world and physics, look, I mean, just try to write a computer program for blowing out these candles. All the chemical reactions, turbulent wind flows, almost impossible. But with VideoJam. There you go. I can't believe what I am seeing here. Look at how lifelike that is. Wow. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Now, it is not only smart when it comes to physics, but it is also creative. Now let's put roller skates on a raccoon. If I ask a text to image AI, I often get four roller skates. Sometimes three. Okay, but when I start thinking about it, I ask, how would this actually work in reality? Well, let's ask VideoJam. And now hold on to your papers, fellow Scholars, because this is the part where I fell off the chair when reading this paper. It says, nope, wait. We are using just two roller skates and reserve the front two hands to push itself, then to balance itself and for breaking as well. Kind of. See, here is the difference. This is not just making images, this is making videos. And in the videos, it actually has to work. It has to dream up things that really work. Okay, now that I calmed down somewhat, let's compare it to DIT the technique it is based on. And in research papers, the result is almost always, always the. The new technique works better in some places and worse in others. But not here. I mean, come on. It outperforms its predecessor by a great deal on every single example I have seen. That is stunning. So how is all this magic possible? Well, let's pop the hood and. Hmm. Interestingly, the idea is surprisingly simple. Step number one, training. We give it a few video frames and then we ask now little a pred predict what is about to happen. Now stage two, video creation. We let the AI generate the next frames, but now we give it a little helper that is its own motion predictions. It looks at what it thinks should move and where they should move and uses that to guide itself toward smoother, more natural motion. They call it inner guidance. And there is two things you should know about it. One, it works. Two, and here comes the key. If you take away one thing from this video, it should be this one. This can be applied to any other video model. Yes, this is like a magic ingredient in a soup, and it can enhance any other soup that you might have around. Yummy. And we also have a comparison to DeepMind's VO2 on the same prompt. And yes, we see that they are comparable, but I don't think these two works are competing because VO2 might get further improved by this idea. That is absolutely amazing. Now, limitations. The results are not super high resolution. I think we can all agree on that. I also haven't found a way to run it myself. However, the paper is available, so I think you might see it introduced into other systems very soon. And with that, finally everyone will be able to become a film director. You won't need tons of money and expensive equipment anymore. All you will need is a text prompt and a vivid imagination. But the AI can help with even that because it has to create things that really work, kind of. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. Would you like to run your own copy of Deep Seq in the cloud cheaply without using the official app? Yes. Then try Lambda GPU Cloud. They have so many powerful Nvidia GPUs with tons of memory. I use them regularly too. Thanks. Seriously, Try it out now@lambdalabs.com papers or click the link in the description.
