New AI Research Work Fixes Your Choppy Videos! 🎬

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take a bad choppy video and make a beautiful smooth and creamy footage out of it. With today's camera and graphics technology, we can create videos with 60 frames per second. Those are really smooth. I also make each of these videos using 60 frames per second. However, it almost always happens that I encounter paper videos that have only from 24 to 30 frames per second. In this case, I put them in my video editor that has a 60fps timeline where half or even more of these frames will not provide any new information. That is neither smooth nor creamy. And it gets worse. Look, as we try to slow down the videos for some nice slow motion action, this ratio becomes even worse, creating an extremely choppy output video because we have huge gaps between these frames. So does this mean that there is nothing we can do and have to put up with this choppy footage? No, not at all. Look at this technique from 2019 that we covered in an earlier video. The results truly speak for themselves. In goes a choppy video and out comes a smooth and creamy result. So good. But wait, it is not 2019, it is 2021. And we always say that two more papers down the line and it will be improved significantly. From this example, it seems that we are done here. We don't need any new papers. Is that so? Well, let's see what we have. Only one more paper down the line. Now look, it promises that it can deal with 10:1 or even 20:1 ratios. Which means that for every single image in the video it creates 10 or 20 new ones. And supposedly we shouldn't notice that. Well, those are big words, so I will believe it when I see it. Lets have a look together. Holy mother of papers. This can really pull this off. And it seems nearly perfect. Wow. It also knocked it out of the park with this one. And all this improvement in just one more paper down the line. The pace of progress in machine learning research is absolutely amazing. But we are experienced fellow scholars over here, so we will immediately. Is this really better than the previous 2019 paper? Let's compare them. Can we have side by side comparisons? Of course we can. You know how much I love fluid simulations. Well, these are not simulations, but a real piece of fluid. And in this one there is no contest. The new one understands the flow so much better. While the previous method sometimes even seems to propagate the waves backwards in time. A big checkmark for the new one. In this case, the previous method assumes linear motion when it shouldn't thereby introducing a ton of artifacts. The null one isn't perfect either, but it performs significantly better. Do not worry for a second. We will talk about linear motion some more in a moment. So how does all this wizardry happen? One of the key contributions of the paper is that it can find out when to use the easy way and the hard way. What are those? The easy way is using already existing information in the video and computing in between states for a movement. That is all well and good if we have simple linear motion in our video. But look, the easy way fails here. Why is that? It fails because we have a difficult situation where reflections off of this object rapidly change and it reflects something. We have to know what that something is. So look, this is not even close to the true image. Which means that here we can't just reuse the information in the video. This requires introducing new information. Yes, that is the hard way. And this excels when new information has to be synthesized. Let's see how well it does. My goodness, look at that. It matches the true reference image almost perfectly. And also look, the face of the human did not require synthesizing a great deal of new information. It did not change over time. So we can easily refer to the previous frame for it. Hence the easy way did better here. Did you notice? That is fantastic because the two are complementary. Both techniques work well, but they work well elsewhere. They need each other. So yes, you guessed right. To tie it all together, there is an attention based averaging step that helps us decide when to use the easy and the hard ways. Now this is a good paper. So it tells us how these individual techniques contribute to the final image. Using only the easy way can give us about 26 decibels. That would not be the previous methods in this area. However, look, by adding the hard way, we get a premium quality result that is already super competitive. And if we add the step that helps us decide when to use the easy and hard ways, we get an extra decibel. I will happily take it. Thank you very much. And if we put it all together. Oh yes, we get a technique that really outpaces the competition. Excellent. So in the near future, perhaps we will be able to record a choppy video of a family festivity and have a chance at making this choppy video enjoyable. Or maybe even create slow motion videos with a regular camera. No slow motion camera is required. What a time to be alive. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure. Plus they are the only cloud service with 48gb RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.