Google’s New AI Just Made A Movie!

Google's new text to video AI has amazing capabilities that I have not expected. It has learned on 10 billion video tokens, little pieces. So what did it learn? For instance, it can create a minute long movie about Rookie the raccoon. And the script for this movie was also written by an AI. That is excellent. But I expected that. However, what I did not expect was this. And this. And this. And there is so much more. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Yes, it can do the usual suspect's text to video. You provide a text snippet and out comes all this. However, even in that, it can give us something new. For instance, it can create longer videos than most previous systems. Some of these results can become longer than 15 seconds. So we don't have to splice a movie together from these small 4 second cuts, like with many previous techniques. Now, temporal coherence is not perfect in these cases, so the scene can change unexpectedly over time. But it is so much better than anything else I've seen so far. Great work. But it can do so much more. For instance. Wait, films need sounds too. Can an AI help us with that? Now hold on to your papers, fellow scholars, because this AI has also learned on 58 billion audio tokens. So what did it learn? Well, let's listen together and take a careful look at how good it is in making sure that the sounds line up with the motions in the video. These were real good. Wow, what an incredible new tool. We can also perform controllable video editing. For instance, if we are asking for a raccoon dancing in Times Square, we get this. Okay, but we are experienced fellow scholars over here. We are. We ask. Ok, but what kind of dancing are we talking about? And oh yes, we can ask it to do the robot, and then the dance will indeed resemble the robot dance. Or how about this one? Nice and fluid hip movements, very sharp. Now it can also do interactive video editing. This is amazing. For first, after writing the prompt about this rusty steampunk robot, we don't just have to accept this one. No sir. We can generate several candidates, and if any of those caught our eye, we can even extend it. This is due to the technique's autoregressive nature. It sees the world as a series of tokens, a series of numbers and characters, if you will. And thus, we can easily ask it to extend the video with, with or without a prompt, thus creating a system that is art directable. So good. And if we can't write a prompt because we have something really Specific in mind, just add an image and then look. It will make it come alive so that Mona Lisa can start yawning. Or that wanderer with the Klaus Kinski like hair can marvel at this beautiful windy place. And if we are happy with the idea of the movie, but we would like to try different stylizations of it, we can try them immediately, creating completely different kinds of moods for the same or very similar content. And what you see here are just the trivial cases of stylization. And now let's bring in the big guns. If we like these teddy bears and have the feeling that the floor looks beautifully glossy and almost like ice. And then bam. There we go. Ice skating in a winter wonderland. Loving it. Or if you feel like this wombat looks like someone on the beach with a beach ball in hand. There you go. I can only imagine what miracles you smart and creative fellow scholars will be able to create with this in the future. And you even get to control the camera motions in these videos as well. And you haven't even heard about the two coolest things about this. 1. If you feel that this tool can do absolutely everything, you are completely right. In fact, everything is optional here. Just chuck in whatever you want. Even a small piece of an image that you would like to make a video of, but at the same time extend it via outpainting. That is also possible. Mind blowing. This is the full package. And it gets better. Two, it can do all of these in a zero shot manner. It can synthesize things it hasn't seen before. That bends the mind a little. So a word on that. Essentially it means that we can ask it to create something that it does not have in its training set. However, it can lean on its already existing knowledge about the world. If you are asking for a vehicle, that vehicle should probably have a steering wheel or some mechanism for controlling it, and so on. It is a bit like an actor who can improvise a new scene when given a description. But wait, we are still not done. So how quick is it? It is not in the order of seconds per frame, but frames per second. So we get one second of video every four to five seconds. That is absolutely amazing. I consider that to be blazing fast and marvelous as this paper is. Of course it is still not perfect. There are trade offs to be taken here. For instance, the resolution of these videos is not the greatest. Something that can be remedied by current super resolution tools. And of course, just imagine what we will be capable of just two more papers down the line. What a time to be alive. And just as I finish with all this what do I see? Oh my an even better video generator paper from Google. Before you ask, of course we are going to talk about this one too soon. If you find it interesting subscribe and hit the bell icon to not miss out on it. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.