DeepMind tested video-generating AIs on physical prediction tasks like rotating teapots and objects in water. Most models failed badly, producing physically impossible continuations. Even the best scored under 30%, and surprisingly, they handle fluids better than solid objects.
The multiframe version of Video Poet (though its accuracy was still reported as below 30%).
OpenAI Sora.
Today, we are going to see AI techniques fail in ways that are ridiculous. It's going to be an absolute disaster. Why? Well, now we know that AI techniques can generate videos from a piece of text. I always think, okay, but do they really understand what they are looking at? Do they really know the physics of these objects? Oh, I like this question a lot because it sounds very philosophical and sounds pretty much impossible to answer. But wait a minute. According to this amazing new paper, there is an actual answer to this question. Incredible. We can ask questions to test whether these AIs actually understand what they are seeing. But not this way. If you visualize what is inside of a neural network, you don't get a lot of useful information, only a bunch of numbers. Remember, this is not human intelligence. This is artificial intelligence. So how do we ask them? Well, scientists at Google DeepMind say, let's show them the start of the video. And if the AI understands it, it will be able to tell us what will happen in the next five seconds. We humans know what is about to happen, but do they? Let's have a look through four experiments where each gets more challenging than the last one. First, let's start simple. A rotating teapot. Pika 1.0 says yes. Stand aside, everyone. I got this. It will not rotate, but grow a pedestal. Oh my. That is a complete disaster. And this is just the simplest question. Now then, Lumiere. No, no, no. This is a rotating teapot, and I shall do exactly that. Wait, where were the handles? Exactly. Object permanence is not my strong point, but OpenAI Sora and Runways Gen 3 kind of gets it. Not perfect, but not bad. Now two, let's paint something. There is a bit of a rotation, and it is clear as day what is about to happen. Yes. Now, let's start this with OpenAI Sora. Because it was right last time and. Oh my goodness, that is not even close. Then Pika 1.0 says yes. I know what's going on here. Zooming in. And then something happens. Also wrong. And if we ask Lumiere. Come on, man, are you even trying? Now if we ask Video Poet. Now this is quite reasonable. Not perfect, but better. Now three, light versus heavy. A classic. We expect that the heavy kettlebell object will leave a larger imprint on the pillow than a light scrap of paper. Easy, right? Well, let's see together. Video Poet says, choose me. I know, I know. The evil pillow is the paper. And as a revenge, we then stab it. Yes, that is what will happen. Pika 1.0 says, Nothing to see here. Just zooming, Just keep zooming. And then OpenAI. Sora says, well I do not know what it says. Wow, all of them are absolute disasters. So what do we do now? Do we make it easier for them? Ha. Nope. We make it even harder. Dear fellow scholars, this is two minute papers with Dr. Zsolnai four. We start with a match on fire, which we put into water. What happens then? Runway Gen 3 says, of course it floats. Lumiere says, nope, you're dead wrong. Fires clearly exist underwater. And Video Poet says that's also wrong. Of course an explosion happens. And the best of them all in this case is Sora, which kind of gets it. Wait a minute. Then it gets lit on fire by the water again. Oh my. And this was just the start. Scientists tested these AIs on a heck of a lot more solid dynamics, fluid dynamics, optics, thermodynamics, magnetism, you name it. So what is the result? Man, I am so curious. Wow, look at that. Sora came in dead last, while the multiframe version of Video Poet aced it. However, look, this is still below 30%, so it only aced it compared to its competition. But overall the result is that they mostly have no idea about physics. Another interesting thing is that they understood fluid mechanics better than solid dynamics, which is really interesting because if you study the basics of both, I think fluids are way harder. Not even close. But not for the AI. This is not human intelligence. This is a different kind of intelligence. And these results are really surprising to me. These techniques can generate lots and lots of photorealistic footage, but apparently visual realism and physical understanding do not go hand in hand. In other words, the best, most photorealistic looking systems don't necessarily understand the world around us too well. Now there is another study that shows pictures to these GPT like AI assistants and asks questions about them. Think of this like a visual IQ test where we ask about temperature, air pressure, or smash a watermelon and ask about that. And the results are each of them know different areas a bit better, but all of them are surprisingly poor. But why? I mean, everyone says that many of these are PhD level AI systems. So what happened? What went wrong here? Well, two. One, physical understanding differs significantly from the tasks that these systems are trained for. So just teach them more about that, right? Nope. Two, Shockingly, as we teach these algorithms more, they don't start scoring better on tests like these. So yes, AIs can do amazing things today, but they are not human intelligence. They are a fundamentally different kind of intelligence that still has a long way to go. I hope you had as much fun with this as I had. I love this one. And if you wish to see more, subscribe and hit the bell icon. And what do you fellow scholars think? Let me know in the comments below. Fellow scholars, if you're like me, it takes you a while to truly understand the paper, especially when it's not right in your area. Macro.com is a new AI document reader that gives you superpowers. Click on any term or figure and it explains it to you without leaving your page. You can even even talk to the paper or ask questions too. Try it for free@macro.com papers and get $5 with the code papers.
