Stable Video AI Just Got Supercharged! - For Free!

I cannot believe this is happening. We just talked about this paper and this paper and they both have already been surpassed. And as we go, it will just keep getting better and better and better. And amazing news. Luckily you can also try this one yourself for free. So what is going on here? Dear fellow scholars, the this is two minute papers with Dr. Zsolnai. Well, in about 2021 we had the first usable text to image AIs appear. You write a text prompt and out comes an image. Then this concept just got better and better. These AIs are unbelievably good. So what's next? Well, of course, not just a stationary image, but multiple images one after another. Yes, a video, if you will. In November 2023, with stable video diffusion, we now have completely free and a fully open source technique that will generate videos from your text prompts. And alternatively, it can also take an image from you and make it come to life. Absolutely amazing. But this is all old news. This is all in the past. And now, just a month later, look at this. My goodness. Yes. Now we have customizable camera motion for these AI generated videos. It can perform the usual bread and butter camera motions like pan up, pan down, zoom out, and it can even control the speed at which these movements take place. That is nice, but I have to say I kind of expected that. But it gets better. What I did not expect is that we can also specify camera motions ourselves and can even draw all kinds of curves that would make the scene really pop. Now, in this case, the results are clearly not perfect. But we are experienced fellow scholars here and we already know that if we see results like this, and if we look just two more papers down the line, it will get so much better. But this is not the first technique to perform something like this. So let's compare it to a previous work. Ah, yes, as expected. Unfortunately we have lots of artifacts. The subsequent images have to follow each other naturally. But it does not give us the impression that this is a video from out there in the wild. It is missing the the most important part, Temporal coherence, the lifeblood of videos. Now let's see the new technique. And. Whoa. This is so much better. Temporal coherence for days. Still not perfect. But this is something that could help a ton of you creative fellow scholars out there. But you know what's coming. Yes, it gets even better. But how? Wait, I hear you saying Caroj. The main issue with stable video diffusion was that sometimes there was very little motion in the footage. In particular, look here. The camera moves, but the subject does not move. And this new technique is better in a sense that it allows you to customize the motion of the camera, but it does not allow you to move the subjects themselves. Except that video composer, the previous work does attempt to do that too. So how did it do? Well, it did extremely well, but not in the way you think. For this skier, it has many artifacts, but as an unintentional side effect, look, this AI has also invented an excellent cloning machine. Not exactly what we asked for, but come on. Also, this falling feather is also trying a little too hard to adhere to this curve. And the motion is unfortunately not believable. So that's it, right? No video control for the subject. Well, don't despair. And now hold on to your papers, fellow scholars. And look, the new technique. Look at the beautiful movement of the feather. Wow. That is a lot more believable. And so is the skier. Just one skier, if I may add, but a good one at that. And all this just one more paper down the line. I can hardly believe it. And we can use this for so much. The new technique can make that wind chime sway in the wind. And we can even specify the wind for it. The paper planes can fly as we wish them to fly. And we can even let those cats and zebras out for a walk. This is like an AI puppeteer who can simultaneously control the stage and the puppets at the same time. And you know what's coming. Yes, it gets better again. We can even combine the two together and make the camera and the subject move at the same time. I absolutely love it. The results are clearly not perfect, but if you get something that you don't like, don't forget, these run for nearly free. And the whole thing just takes a few seconds. That means that you can have as many samples as you wish. Just look at the different variants on how the skateboarding teddy bear behaves, and choose the one that is the most in line with your artistic vision. So good. And I bet that just two more papers down the line and we will be able to create short movies with it just by writing text prompts and drawing the motions for the actors in the movie. And it will make all of that happen. And here comes the best part. You can try it too, for free. As of the making of this video, I tried this demo and it worked flawlessly. Now, don't forget, there are a lot of you fellow scholars, and we have crashed a great many websites before, too. So once again, during the scholarly stampede, please be patient. And if you enjoyed this video and would like to see more content like this. Make sure to subscribe and hit the bell icon to not miss out. We have some absolutely amazing papers coming coming up soon. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.