Dr. Károly Zsolnai-Fehér reviews a new technique that animates still photos. It runs in about 1 second on a smartphone, versus 20 seconds on heavy GPU hardware for older methods, and uses handcrafted rules instead of deep learning.
July 13, 2021
About 20 seconds per image and required a large GPU.
About 1 second and can run on a smartphone.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Look at this video of a moving escalator. Nothing too crazy going on, only the escalator is moving. And I am wondering, would it be possible to not record a video for this, just an image, and have one of these amazing learning based algorithms animated? Well, that is easier said than done. Look, this is what was possible with the research work from two years ago. But the results are, well, what you see here. So how about a method from one year ago? This is the result, a great deal of improvement. But the water is not animated in this region and is generally all around the place. And we still have a lot of artifacts around the fence. And now hold on to your papers and let's see this new method. And whoa, look at that. What an improvement. Apparently we can now give this one a still image. And for the things that should move, it makes them move. It is still not perfect by any means, but this is so much progress in just two years. And there is more. Get this. For the things that shouldn't move, it even imagines how they should move. It works on this building really well. But it also imagines how my thigh would move around, or my beard, which is not mine, by the way, but was made by a different AI. Or the windows. Thank you very much for the authors of the paper for generating these results only for us. And this can lead to really cool artistic effects. For instance, this moving brick wall or animating the stairs here. Loving it. So how does this work exactly? Does it know what regions to animate? No, it doesn't and it shouldn't. We can specify that ourselves by using a brush to highlight the region that we wish to see animated. And we also have a little more artistic control over the results by prescribing a direction in which things should go. And it appears to work on a really wide variety of images, which is only one of its most appealing features. Here are some of my favorite results. I particularly love the one with the upper rotation here. Very impressive. Now let's compare it to the previous method from just one year ago and let's see what the numbers say. Well, they say that the previous method performs better on fluid elements than the new one. My experience is that it indeed works better on specialized cases like this fire texture, but on many water images they perform roughly equivalently. Both are doing quite well. So is the new one really better? Well, here comes the interesting part. When presented with a diverse set of images, look, there is no contest here. The previous one creates no results. Incorrect results or if it does something, the new technique almost always comes out way better. Not only that, but let's see what the execution time looks like for the new method. How much do we have to wait for these results? The one from last year took 20 seconds per image and required a big honking graphics card, while the new one only needs your smartphone and runs in what, just one second? Loving it. So what images would you try with this one? Let me know in the comments below. Well, in fact, you don't need to just think about what you would try because you can try this yourself. The link is available in the video description. Make sure to let me know in the comments below if you had some success with it. And here comes the even more interesting part. The previous method was using a learning based algorithm, while this one is a bona fide, almost completely handcrafted technique, partly because training neural networks requires a great deal of training data and there are very few if any training examples for moving buildings and these other surreal phenomena. Ingenious. Huge congratulations to the authors for pulling this off. Now. Of course, not even this technique is perfect. There are still cases where it does not create appealing results. However, since it only takes a second to compute, we can easily retry with a different pixel mask or direction and see if it does better. And just imagine what we will be able to do two more papers down the line. What a time to be alive. This video has been supported by Weights and Biases. They have an amazing podcast by the name Gradient Descent where they interview machine learning experts who discuss how they use learning based algorithms to solve real world problems. They've discussed biology, teaching robots, machine learning in outer space, and a whole lot more. Perfect for a fellow scholar with an open mind. Make sure to visit them through WNB me, GD or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
