Learning to Fill Holes in Images | Two Minute Papers #130

Dear fellow scholars, this is two minute papers with. This paper is from 2007, from 10 years ago, and I'm sure you'll be surprised by how well it is holding up to today's standards. For me, it was one of the works that foreshadowed the incredible power of data driven learning algorithms. So let's grab an image and cut a sizable part out of it and try to algorithmically fill it with data that makes sense. Removing a drunk photobombing friend from your wedding picture or a building blocking a beautiful view to the sea are excellent and honestly painfully real examples of this. This problem we like to call image completion or image inpainting. But mathematically this may sound like crazy talk. Who really knows what information should be there in these holes, let alone a computer. The first question is why would we have to synthesize all these missing details from scratch? What? Why not start looking around in an enormous database of photographs and look for something similar? For instance, let's unleash a learning algorithm on 1 million images. And if we do so, we could find that there may be photographs in the database that are from the same place. But then what about the illumination? The lighting may be different. Well, this is an enormous database. So then we pick a photo that was taken at a similar time of the day and and use that information. And as we can see in the results, the technique works like magic. Awesome. It doesn't require user made annotations or any sort of manual labor. These results were way, way ahead of the competition. And sometimes the algorithm proposes a set of solutions that we can choose from. The main challenge of this solution is finding similar images within the database. And fortunately, even a trivial technique that we call nearest neighbor search can rapidly eliminate 90 of the dissimilar images. The paper also discusses some of the failure cases which arise mostly from the lack of high level semantic information. For instance, when we have to finish people. Which is clearly not what this technique is meant to do, unless it's a statue of a famous person with many photographs taken in the database. Good that we are in 2017 and we know that plenty of research groups are already working on this and I wouldn't be surprised to see a generative adversary network based technique to pop up for this in the very near future. Thanks for watching and for your generous support and I'll see you next time.