This AI Makes Beautiful Videos From Your Images! 🌊

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. In June 2020, OpenAI published an incredible AI based technique by the name ImageGPT. The problem here was simple to understand, but nearly impossible to actually do. So here goes. We give it an incomplete image and we ask the AI to fill in the missing pixels. That is of course, an immensely difficult task because these images may depict any part of the world around us. It would have to know a great deal about our world to be able to continue the images. So how well did it do? Let's have a look. This is undoubtedly a cat, but look, see that white part that is just starting. The interesting part has been sneakily cut out of the image. What could that be? A piece of paper? Something else? Now let's leave the dirty work to the machine and ask it to finish it. Oh yeah, that makes sense now. Even better, let's have a look at this water droplet example too. We humans know that since we see the remnants of ripples over there too, there must be a splash. But the question is, does the AI know that? Oh yes, yes it does. Amazing. And a true image for reference. But wait a second. If ImageGPT could understand that this is a splash and finish the image like this, then here is an absolutely insane idea. If a machine can understand that this is a splash, could it maybe not only finish the photo but but make a video out of it? Yes, that is indeed an absolutely insane idea. We like those around here. So what do you think? Is this a reasonable question or is this still science fiction? Well, let's have a look at what this new learning based method does. When looking at such an image, it would do something very similar to what we would do. Look at the image, estimate the direction of the motion, recognize that these ripples should probably travel outwards. And based on the fact that we've seen many splashes in our lives, if we had the artistic skill, we could surely fill in something similar. So can the machine do it too? And now hold on to your papers, because this technique does exactly that. Whoa. Please meet Eulerian motion synthesis. And it not only works amazingly well, but look at the output video. It even loops perfectly. Yum, yum, yum. I love it. And it works mostly on fluid and smoke. I like that. I like that a lot. Because fluids and smoke have difficult but predictable motion. That is an excellent combination for us, especially given that you see plenty of those simulations on this channel. So if you are a long time fellow scholar, you already have a keen eye for them. Here are a few example images paired with the synthesized motion fields. These define the trajectory of each pixel or in other words, regions that the AI thinks should be animated and how it thinks should be animated. Now it gets better. I have found three things that I did not expect to work but was pleasantly surprised that they did. 1. Reflections kind of work. 2. Fire kind of works. And now if you have been holding onto your paper so far, now squeeze that paper because here comes the best one.

3.

My beard works too. Yes, you heard it right. Now first things first, this is not any kind of beard. This is an algorithmic beard that was made by an AI and now it is animated as if it were a piece of fluid using a different AI. Of course this is not supposed to be a correct result, just a happy accident. But in any case, this sounds like something straight out of a science fiction movie. I also like how this has a nice Obi Wan Kenobi quality to it. Loving it. Thank you very much to my friend Oliver Wong and the authors for being so kind and generating these results only for us. That is a huge honor. Thank you. This previous work is from 2019 and creates high quality motion, but has a limited understanding of the scene itself. And of course, let's see how the new method fares in these cases. Oh yeah, this is a huge leap forward. And what I like even better here is that new research techniques often provide different trade offs than previous methods, but are rarely strictly better than them. In other words, competing techniques usually do some things better and some other things worse than their predecessors. But not this look. This is so much better across the board. That is such a rare sight. Amazing. Now of course not even this technique is perfect. For example, this part of the image should have been animated but remains stationary. Also, even though it did well with reflections, refraction is a tougher knot to crack. Finally, thin geometry also still remains a challenge, but this was one paper that made the impossible possible and just think about what we will be able to do two more papers down the line. My goodness, what a time to be alive.

This episode has been supported by weights and biases. In this post they show you how.

To use their tool to get a.

Neural network to generate captions for your.

Images using an attention model.

During my PhD studies I trained a ton of neural networks which were used in our experiments. However, over time there was just too much data in our repositories and what I am looking for is not data but insight. And that's exactly how weights and biases helps you by organizing your experiments. It is used by more than 200 companies and research institutions including OpenAI, Toyota Research, GitHub and more. And get this Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get.

A free demo today.

Our thanks to WeightsAndBiases for their long standing support and for helping us make.

Better videos for you. Thanks for watching and for your generous support and I'll see you next time.