A short Two Minute Papers video explaining a 2017 algorithm that performs video inpainting — filling holes across space and time — by enforcing spatiotemporal consistency via an optimization procedure. The clip describes use cases (object removal, restoring old footage, filling missing frames), gives an example (Charlie Chaplin restoration), and notes that source code is available in the video description.
Dear fellow scholars, this is two minute papers with Today we are going to talk about an algorithm that is capable of filling holes in space and time. Classical image inpainting, or in other words, filling holes in images is something that we've explored in earlier episodes. There are several incredible techniques to take care of that, but with this piece of work it is possible to generalize such a technique for video and and fill holes for not only one image, but a series of images, like removing an umbrella to create an unoccluded view to the beach, or removing a waving person from a video of us jogging. These results are truly incredible, and even though this method was published long, long ago, it still enjoys a great deal of reverence among computer graphics practitioners. Not only that, but this algorithm also serves the basis of the awesome Content Aware Fill feature in Adobe Photoshop C. In this problem formulation, we have to make sure that our solution has spatiotemporal consistency. What does this mean exactly? This means that holes can exist through space and time, so multiple frames of a video may be missing, or there may be regions that we wish to cut out not only for one image, but for the entirety of the video. The field in regions have to be consistent with their surroundings if they are looked at as an image, but there also has to be a consistency across the time domain, otherwise we would see a disturbing flickering effect in the results. It is a really challenging problem indeed, because there are more constraints that we have to adhere to. However, a key observation is that it is also easier because in return we have access to more information that comes from the previous and next frames in the video. For instance, here you can see an example of retouching old footage by removing a huge pesky artifact. And clearly we know the fact that Charlie Chaplin is supposed to be in the middle of the image only because we have this information from the previous and next frames. All this is achieved by an optimization algorithm that takes into consideration that consistency has to be enforced through the spatial and the time domain at the same time. It can also be used to fill in completely missing frames of a video. Or it also helps where we have parts of an image missing after being removed by an image stabilizer algorithm. Video editors do this all the time, so such a restoration technique is super useful. The source code of this technique is available. I've put a link to it in the video description. Thanks for watching and for your generous support, and I'll see you next time.
