Dr. Zsolnai covers a new video stabilization method that produces full-frame results with minimal artifacts by improving motion estimation and using neighboring frames. It beats recent competitors on benchmarks and won 60-90% preference in a 46-person user study, though it takes 9.5 seconds per frame to process.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to talk about video stabilization. A typical application of this is when we record family memories and other cool events. And sometimes the footage gets so shaky that we barely know what is going on. In these cases, video stabilization techniques can come to the rescue. Which means that in goes a shaky video and out comes a smooth video. Well, that is easier said than done. Despite many years of progress, there is a great selection of previous methods that can do that. However, they suffer from one of two issues. Issue number one is cropping. This means that we get usable results, but we have to pay a great price for it, which is cropping away a great deal of the video content. Issue number two is when we get the entirety of the video, no cropping. However, the price to be paid for this is that we get lots of issues that we call visual artifacts. Unfortunately, today when we stabilize, we have to choose our poison. It's either cropping or artifacts. Which one would you choose? That is difficult to decide of course, because none of these two trade offs are great. So our question today, can we do better? Well, the law of papers says that of course, just one or two more papers down the line and this will be way better. So let's see. This is what this new method is capable of. Hold on to your papers and notice that this will indeed be a full size video. So we already know that probably there will be artifacts. But wait a second, no artifacts. Whoa. How can that be? What does this new method do that previous techniques didn't? These magical results are a combination of several. One, the new method can estimate the motion of these objects better. Two, it removes blurred images from the videos, and three, collects data from neighboring video frames more effectively. This leads to a greater understanding of the video it is looking at now. Of course, not even this technique is perfect. Rapid camera motion may lead to warping, and if you look carefully, you may find some artifacts, usually around the sides of the screen. So far we have looked at previous methods and the new method, it seems better. That's great. But how do we measure which one is better? Do we just look? An even harder question would be if the new method is indeed better. Ok, but by how much better is it? Let's try to answer all of these questions. We can evaluate these techniques against each other in three different ways. One, we can look at the footage ourselves. We have already done that and we had to tightly hold on to our papers. It has done quite well in this test. Test number two is a Quantitative test. In other words, we can mathematically define how much distortion there is in an output video, how smooth it is and more, and compare the output videos based on these metrics. In many cases, these previous techniques are quite close to each other. And now let's unveil the new method. Whoa. It scored best or second best on six out of eight tests. This is truly remarkable, especially given that some of these competitors are from less than a year ago. That is nimble progress in machine learning research. Loving it. And the third way to test which technique is better and by how much is by conducting a user study. The authors have done that too. In this, 46 humans were called in, were shown the shaky input video, the result of a previous method and the new method, and were asked three which video preserves the most content, which has fewer imperfections, and which is more stable? And the results were stunning. Despite looking at many different competing techniques, the participants found the new method to be better and at the very least, 60% of the time on all three questions, in some cases even 90% of the time or higher, praise the papers. Now there is only one question left. If it is so much better than previous techniques, how much longer does it take to run? With one exception, these previous methods take from half a second to about 7 1/2 seconds per frame. And this new one asks for 9 1/2 seconds per frame. And in return, it creates these absolutely amazing results. So from this glorious day on, fewer or maybe no important memories will be lost due to camera shaking. What a time to be alive. Perceptilabs is a visual API for TensorFlow, carefully designed to make machine learning as intuitive as possible. This gives you a faster way to build out models with more transparency into how your model is architected, how it performs, and how to debug it. Look, it lets you toggle between the visual modeler and the code editor. It even generates visualizations for all the model variables and gives you recommendations both during modeling and training. And does all this automatically. I only wish I had a tool like this when I was working on my neural networks during my PhD years. Visit perceptilabs.com papers to easily install the free local version of their system today. Our thanks to Perceptilabs for their support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
