5:08
Károly Zsolnai-Fehér explains how researchers are applying the Ken Burns effect, that slow camera pan across still photos, to 3D using depth estimation and image inpainting. The technique lets you move through a flat image as if it has depth.
Károly Zsolnai-Fehér explains how researchers are applying the Ken Burns effect, that slow camera pan across still photos, to 3D using depth estimation and image inpainting. The technique lets you move through a flat image as if it has depth.
Dear fellow scholars, this is two minute papers with have you heard of the Ken Burns effect? If you have been watching this channel, you have probably seen examples where a still image is shown and a zooming and panning effect is added to it. It looks something like this. Familiar, right? The fact that there is some motion is indeed pleasing for the eye. But something is missing. Since we are doing this with 2D images, all the depth information is lost. So we are missing out on the motion parallax effect that we would see in real life when moving around the camera. So in short, this is only 2D. Can this be done in 3D? Well, to find out, have a look at this. Wow. I love it. Much better, right? Well, if we would try to perform something like this without this paper, we'd be met with bad news. And that bad news is that we have to buy an RGBD camera. This kind of camera endows the 2D image with depth information, which is specialized hardware that is likely not available in our phones as of the making of this video. Now, since depth estimation from These simple monocular 2D images without depth data is a research field of its own, the first step sounds simple enough. Take one of those neural networks, then ask it to try to guess the depth of each pixel. Does this work? Well, let's have a look as we move our imaginary camera around. Uh oh, this is not looking good. Do you see what the problems are here? Problem number one is the presence of geometric distortions. You see it if you look here. Problem number two is referred to as semantic distortion in the paper. Or in other words, we now have missing data. But this poor tiny human's hand is also. Ouch. Let's look at something else instead. If we start zooming in into images, which is a hallmark of the Ken Burns effect, it gets even worse. We get artifacts. So how does this new paper address these issues? After creating the first coarse depth map, an additional step is taken to alleviate the semantic distortion issue. And then this depth information is upsampled to make sure that we have enough fine details to perform the 3D Ken Burns effect. Let's do that. Unfortunately, we are still nowhere near done yet. Previously occluded parts of the background suddenly become visible and we have no information about those. So how can we address that? Do you remember image inpainting? I hope so, but if not, no matter. I'll quickly explain what it is. Both learning based and traditional handcrafted algorithms exist to try to fill in dismissing information in images with sensible data by looking at its surroundings. This is also not as trivial as it might seem. First, for instance, just filling in sensible data is not enough because this time around we are synthesizing videos. It has to be temporally coherent, which means that there must not be too much of a change from one frame to another or else we'll get a flickering effect. As a result, we finally have these results that are not only absolutely beautiful but but the user study in the paper shows that they stack up against handcrafted results made by real artists. How cool is that? It also opens up really cool workflows that would normally be very difficult if not impossible to perform. For instance, here you see that we can freeze this lightning bolt in time, zoom around and marvel at the entire landscape. Love it. Of course, limitations still apply. If we have really thin objects such as this flagpole, it might be missing entirely from the depth map. Or there are also cases where the image inpainter cannot fill in useful information. I cannot wait to see how this work evolves a couple papers down the line. One more interesting tidbit. If you have a look at the paper, make sure to open it in Adobe Reader. You will likely be very surprised to see that many of these things that you think are still images are actually animations. Papers are not only getting more mind blowing by the day, but also more informative and beautiful as well. What a time to be alive. This video has been supported by you on Patreon. If you wish to support the series and also pick up cool perks in return, like early access to these episodes or getting your name immortalized in the video description, make sure to Visit us through patreon.com TwominutePapers thanks for watching and for your generous support and I'll see you next time.
