Dr. Zsolt Zsolnai reviews Google's new AI video editor that can transform videos with text prompts. It turns a papaya into a cake, adds a shark swimming next to a sea turtle, and morphs an orangutan into a dancing bear. The method works by adding noise to frames and using diffusion to reshape them toward your text description, though fine details still get lost.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see that AI video generation is amazing. But it is old news. Yes, really? So this paper just appeared a few months ago. Why would this be old news? Well, what you see here is an AI that can perform video generation. In goes a piece of text and out comes a video. However, this is not video editing. Why? In this case, we don't really have fine granular control over what happens in this video. However, get this. This new paper promises an incredible leap forward where we can actually edit these videos. How? Well, look, it can take a collection of photos of our favorite teddy bear and make a video about it and imagine what it would look like if it were walking. So cool. And it gets even better. We can even provide it a video if we are looking for even more artistic control. And start cutting a papaya. And then ask the AI to transform the papaya into a cake. And it did it just like that. Wow. We can also make this video a little more dramatic by pretending there is a flood going on. And of course, as a computer graphics person who loves to write simulations like this, I have to say, when I look at the interaction between the car and the water, I absolutely love what I am seeing here. It does not seem nearly perfect. However, I only see flaws when. When I am looking for them specifically with a critical eye. Loving it. And wait. It gets even better. So far we have seen video to video, multiple images to video, and now it promises something even more amazing. What is that? It is just one image to video. What are you trying to tell me? That I provide just this one image here and I get a full video of someone pouring coffee? That can't be right. But hold on to your papers, fellow scholars, just in case. And now, let's see.
Wow.
It can do this too. I can't believe it. And now come the results that made me fall out of the chair when reading this paper. I wasn't holding on to my papers enough. So I tell you in advance and. And I hope you will not make my mistake. I loved how it made an animation of this black bean plant growing. But it turns out we can give it even more granular artistic instructions. Look here we can not only ask the AI to add a buffalo bathing in the river, but we can make it gradually zoom out and reveal it. An AI cinematographer. How cool is that? Now, of course, not even this technique is perfect. There are some issues with the horns and more, but this is still incredible progress. Just one more paper down the Line. I am stunned. And it gets even better. For instance, with this, we can now make really fun artistic choices. Get this. We can ask the AI to pretend as if our toy was a real animal. And it just does it. How is this even possible? Oh yes. Now I know what you're waiting for. Let's pop the hood and look inside. Aha. Yes. First our video goes in. And then downscaling happens. This means that the number of pixels and thus the amount of detail in the image is now less. Don't forget this part. This will become important in a moment. So one more time. After this point, we get less detail. And then something really surprising happens. We make it even worse. We add noise. Why is that? Why would we make this less detailed video even worse? How does that make sense? Well, this added noise helps us by using enabling us to use previously existing diffusion based approaches. With that, we can use this noisy video as a starting point and slowly reorganize this noise to resemble this text prompt a little more. What does this mean exactly? Well, in this case, a completely new video for us. Really? Cool. Now let's use our newfound knowledge of the algorithm and let's apply it to the sea turtle. This is a real video. And now let's ask the AI to add a shark. Which it indeed does. Excellent. You see, the framing of the video remains the same, but a shark has been added. Loving it. However, are you seeing what I am seeing? Oh yes. We now see the result of this downscaling step. Look. Some detail in the water waves is now gone. This effect is also particularly visible in the case of this orangutan. Also, not every result is perfect, of course. It truly is a miracle of AI research that we can transform this eating monkey into a dancing bear. But then something interesting happens. Whoa. The bear either leans forward and the AI did not think about filling the background, or alternatively, the bear disappears into a black hole. In which case, this is perhaps authentic footage. Who knows? And these are the early days of AI video editing. What a time to be alive. And don't forget, this is not just AI video generation. This is so much better than that. This is true video editing. And as always, with a work like this, as a true fellow scholar, please always apply the First Law of Papers. The first Law of Papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. So what do you think? What would you use this for? Let me know in the comments below.
This video has been supported by weights and biases. Check out their recent offering Fully Connected, a place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders, and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to WeightsAndBiases for their long standing support and for helping us make.
Better videos for you.
Thank you.
Thanks for watching and for your generous support and I'll see you next time.
