Google’s Video AI: Outrageously Good! 🤖

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. I cannot believe that this paper is here. This is unbelievable. So what is going on here? Yes, that's right. We know that these modern AI programs can paint images for us anything we wish. But today we are going to find out whether they can also do it with video. You see an example here and here. Are these also made by an AI? Well, I'll tell you in a moment. So, video. That sounds impossible. That is so much harder. You see, videos require a much greater understanding of the world around us, so much more computation. And my favorite, temporal coherence. What is that? This means that a video is not just a set of images, but a series of images that have to relate to each other. If the AI does not do a good job at this, we get this flickering. So as all of this is so hard, we will be able to do this maybe in 5 to 10 years or maybe never. Well, scientists at Google say not so fast. Now hold on to your papers and have a look at this. Oh my goodness. Is it really here? I am utterly shocked, but the answer is yes. Yes it is. So now let's have a look at three of my favorite examples and then I'll tell you how much time this took. By the way, it is an almost unfathomably short time. Now one, the concept is the same. One simple text prompt goes in, for instance, a happy elephant wearing a birthday hat walking under the sea. And this comes out. Wow, look at that. That is exactly what we were asking for in the prompt. Plus, as I am a light transport researcher by trade, I am also looking at the waves and the sky through the sea, which is absolutely beautiful, but it doesn't stop there. I also see every light transport researcher's dream there. Water caustics. Look at these gorgeous patterns. Now, not even this technique is perfect. You see that temporal coherence is still subject to improvement. The video still flickers a tiny bit and the task is also changing over time. However, this is incredible progress in so little time. Absolutely amazing too, in good two minute papers fashion. Now let's ask for a bit of physics. A bunch of autumn leaves falling on a calm lake, forming the text, image and video. I love it. You see, in computer graphics, creating a simulation like this would take quite a bit of 3D modeling knowledge to. And then we also have to fire up a fluid simulation. This does not seem to do a great deal of two way coupling, which means that the water has an effect on the leaves. You see it advecting this leaf here, but the leaves do not seem to have a huge effect on the water itself. This is possible with specialized computer graphics algorithms like this one. And I bet it will also be possible with IMAGEN video version 2. Now I am super happy to see the reflections of the leaves appearing on the water. Good job little AI. And to think that this is just the first iteration of Imagen Video. Wow. By the way, if you wish to see how detailed a real physics simulation can be, make sure to check out my Nature Physics comment paper in the video description. Spoiler alert. The surprising answer is that they can be almost as detailed as real life. I was also very happy with this splash and with this turquoise liquid's movement in the glass too. Great simulations on version 1. I am so happy now. 3. Give me a teddy bear doing the dishes. Woah, is this real? Yes it is. It really feels like we are living inside a science fiction movie now. It's not perfect. You can see that it is a little confused by the interaction of these objects. But if someone told me just a few weeks ago that an AI would be able to do this, I wouldn't have believed a word of it. And it not only has a really good understanding of reality, but it can also combine two previous concepts, a teddy bear and washing the dishes, into something new. My goodness, I love it. Now while we look at some more beautiful results, we noted that this is incredible progress in so little time. But how little exactly? Well, if you have been holding onto your paper so far, now squeeze that paper because OpenAI's Dall E2 text to image AI appeared in April 2022. Google's Imagen also text to image appears one month later, May 2022. That is incredible. And get this only five months later, by October 2022 we get this. An amazing text to video AI. I am out of words. Of course it is not perfect. The hair of pets is typically still a problem and the complexity of this ship battle is still a little too much for it to shoulder. So version one is not going to make a new Pirates of the Caribbean movie, But maybe version 3? Two more papers down the line, who knows. Ah yes, about that. The resolution of these videos is not too bad at all. It is in 720p. The literature likes to call it high definition. These are not 4k like many of the shows you can watch on your TV today. But this quality for a first crack at the problem is simply stunning. And don't forget that first it synthesizes a low resolution video, then upscales it for through super resolution, something Google is already really good at. So I would not be surprised for version 2 to easily go to full HD and maybe even beyond. As you see, the pace of progress in AI research is nothing short of amazing. And if like me, you are yearning for some more results, you can check out the paper's website in the video description where as of the making of this video, you get a random selection of results, refresh it a couple times and see if you get something new. And if I could somehow get access to this technique, you bet that I'd be generating a ton more of these update. I cannot make any promises, but good news, we are already working on it. A video of a scholar reading exploding papers absolutely needs to happen. Make sure to subscribe and hit the bell icon to not miss it in case it happens. You really don't want to miss that. So from now on, if you're wondering what a wooden figurine surfing in outer space looks like, you need to look no further. What a time to be alive. So what do you think? Does this get your mind going? What would you use this for? Let me know in the comments below. This episode has been supported by Runway Please Professional and Magical AI Video Editing for Everyone. I often hear you fellow scholars asking, okay, these AI techniques look great, but when do I get to use them? And the answer is right now. Runway is an amazing video editor that can do many of the things that you see here in this series. For instance, it can automatically replace the background behind a person. It can do inpainting for videos amazingly well and can do even text to image, image to image. You name it. No wonder it is used by editors, post production teams and creators at companies like cbs, Google, Vox, and many other. Make sure to go to runwayml.com papers, sign up and try it for free today. And here comes the best part. Use the code 2Minute at checkout and get 10% off your first month. Thanks for watching and for your generous support and I'll see you next time.