Dr. Károly Zsolnai-Fehér explains an AI that turns a single photo into a looping video by learning motion patterns from 2,500 videos. It can animate flowers and sleeping cats, slow down motion, or magnify it, but struggles with image distortion and missing backgrounds.
Dr. Károly Zsolnai-Fehér
October 9, 2023
$1.99 per hour
This is an incredible AI technique where in goes a photo and then out comes a video of it. But wait, not any kind of video. Even better, a seamlessly looping video. I love it. What I found really surprising is that it works quite well on a variety of cases. And not only on simple periodic movements for flowers, but for even a. A sleeping cat and more. Wow. But hold on to your flowers. I mean papers, fellow scholars, because look. Boing. And now a bigger one. And when I tug here, a smaller one. These boings work on other scenes too. And all this little world model from one simple image. And this AI. Incredible. So not just animating the image, but we can even play with it. So how is this even possible? This is an AI model that has looked at videos and motion trajectories that happen in these videos. And based on that, it tries to create something that they call a neural stochastic motion texture. These motion textures describe how pixels should move when they are tugged at how the Boeing should happen. But wait a second. If it has this kind of representation of the motion, maybe it can also manipulate already existing videos too, right? Oh yes. Look, we can take an existing video and even magnify the amount of motion that is happening there. And I am very impressed. I just kept looking and looking and couldn't find a single issue with these videos. I. It would be very difficult to tell that they aren't real. And it gets better. It can even do a slow motion version of already existing videos too. So good. Previous techniques were not nearly as good as this one in creating these animations. Look, this image is trying to perform the impossible which is showcasing motion in just one still image. So what is happening here? Well, we take a small slice of this photo and try to visualize how this slice is moving over time. A previous technique did this stochastic I2V is from just two years ago. And the new technique. OK, this looks better. But we are experienced fellow scholars here. We wish to see how it relates to the real motion. So. Oh my goodness, look at that. No wonder these videos are so realistic. The synthesized motions are in some cases nearly indistinguishable from reality. Now not even this technique is perfect. Look, if I pull this away, I now have distorted the original image a bit too much. And what is behind here? We do not know, because information is missing. Now, image inpainting techniques already exist, so that can be remedied very easily. However, the next one cannot, or at least not easily. Look. And the background also does not work here. Now I think this will surprise you. Being a computer graphics researcher, I have to note that there was an excellent graphics paper called Video Textures which was a handcrafted technique that could do something like this. Yes, you are seeing correctly more than 20 years ago. Goodness. Then the input was a video and it tried to find connection points where it could be connected back to the start of the video and playing it again, thus creating a looping video from a non looping one. It was a little more elaborate than that, but this was the key idea and it is a powerful idea. I wrote my bachelor thesis and a paper 15 years ago based on this technique which was a procedural brushstroke generation technique that could learn your brushstrokes and apply it to already existing works. It was a ton of fun and it was made for a feature length movie. The paper and video of it is available in the Video Description and now an animation from a single photo. Creamy slow motion movement even. Love it and just imagine what we will be capable of just one more paper down the line. My goodness, what a time to be alive. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances for just 1.99per hour. Yes, 199 and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. This was two minute papers with Dr. Karoj Zsolnai. Subscribe if you wish to see more.
