Dr. Zsolnai presents an AI that generates and edits videos in multiple ways: creating new variations, mixing dynamics from two videos, stretching or compressing time, drawing to control motion, and replacing regions like skies or people. Results are inconsistent but sometimes photorealistic, and it handles all these tasks faster and at higher resolution than previous methods.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. This new learning based method is capable of generating new videos, creating lava from water, enhancing your dance videos, adding new players to a football game, and more. This one will feel like it can do absolutely anything. For instance, have a look at these videos. Believe it or not, some of these were generated by an AI from this new paper. Let's try to find out together which ones are real and which are generated. Now before we start, many of the results that you will see here will be flawed. But I promise that there will be some gems in there too. This one doesn't make too much sense. One down. 2. The bus appears to be changing length from time to time. 3. The bus pops out of existence. 4. Wait, this actually looks pretty good. Until we find out that someone is about to get rear ended, this leaves us with two. Are they real or are they fakes? Now please stop the video and let me know in the comments what you think. So this one is real footage. Ok, what about this guy? Well, get this. This one is actually one of the generated ones. Just this one was real. And this is what this new technique is capable of. We give it a sample video and it reimagines variations of it. Of course the results are hit or miss. For instance, here people and cars just pop in and out of existence. But if we try it on the billiard balls. Wow. Now this is something. Look at how well it preserved the specular highlights. And also the shadows move with the balls in the synthetic variations too. Once again, the results are generally hit or miss. But if we look through a few results, we often find some gems out there. But this is nothing compared to what is to come. Let's call this first feature Video Synthesis. And we have four more really cool applications which with somewhat flawed but sometimes amazing results. Now check this out. Here comes number two, video analogies. This makes it possible for us to mix two videos that depict things that follow a similar logic. For instance, here, only four of these 16 videos are real. The rest are all generated. Now here comes feature number three. Time retargeting. For instance, we can lengthen or shorten videos. Well that sounds simple enough. No need for an AI for that. But here's the key. This does not mean just adding new frames to the end of the video. Look, they are different. Yes, the entirety of the video is getting redesigned. Here. We can use this to lengthen videos without really adding new content to it. Absolutely amazing. We can also use it to shorten a video. Now of course, once Again, this doesn't mean that it just chops off the end of the video. No, no, it means something much more challenging. Look, it makes sure that all of your killer moves make it into the video, but in a shorter amount of time. It makes the video tighter, if you will. Amazing. Feature number four, Sketch to video. Here we can take an input video, add a crude drawing, and the video will follow this drawing. The result is of course not perfect, but in return, this can handle many number to number transitions. And now feature number five. Video inpainting on steroids. Previous techniques can help us delete part of an image or even a video and generate data in these holes that make sense given their surroundings. But this, this one does something way better. Look, we can cut out different parts of the video and mark the missing region with a color. And then what happens? Oh yes, the blue regions will contain a player from the blue team. The white region, a player from the white team. Or if we wish to get someone out of the way, we can do that too. Just mark them green. Here the white regions will be inpainted with clouds and the blues with birds. Loving it. But wait a second. We noted that there are already good techniques out there for image inpainting. There are also good techniques out there for video time retargeting. So what is so interesting here? Well, two things. One, Here, all of these things are being done with just one technique. Normally we would need a collection of different methods to perform all of these, but here, just one. AI two. Now hold on to your papers and look at this. Whoa. Previous techniques take forever to generate these results. 144pmeans an extremely crude pixelated image and, and even then they take from hours to, my goodness, even days. So what about this new one? It can generate much higher resolution images and in a matter of minutes. That is an incredible leap in just one paper. Now, clearly most of these results aren't perfect. I would argue that they are not even close to perfect. But some of them already show a lot of promise. And as always, dear fellow scholars, do not forget to apply the first law of papers which says, do not look at where we are, look at where we will be. Two more papers down the line and two more papers down the line. I am sure that not two but five or maybe even six out of these six videos will look significantly more real.
This video has been supported by weights and biases. They have an amazing podcast by the name Gradient Descent where they interview machine learning experts who discuss how they use learning based algorithms to solve real world problems they've discussed biology, teaching robots, machine learning in outer space, and a whole lot more. Perfect for a fellow scholar with an open mind. Make sure to visit them through wnb, me, GD or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping.
Us make better videos for you. Thanks for watching and for your generous support. Thanks for watching and for your generous support and I'll see you next time.
