NVIDIA's neural network can remove objects from photos by learning to fill holes with realistic content and style, outperforming older patch-copying methods. The technique handles irregular shapes and works on asymmetrical scenes where traditional inpainting fails.
Károly Zsolnai-Fehér
June 9, 2018
patreon.com/twominutepapers
Dear fellow scholars, this is two minute papers with Ever had an experience when you shot an almost perfect photograph of for instance an amazing landscape, but unfortunately it was littered with unwanted objects. If only we had an algorithm that could perform image inpainting. In other words, delete a small part of an image and have it automatically filled in. So let's have a look at Nvidia's AI based solution. On the left you see the white regions that are given to the algorithm to correct and and on the right you see the corrected images. So it works amazingly well. But the question is why? This is an established research field. So what new can an AI based approach bring to the table? Well, traditional non learning approaches either try to fill these holes in with other pixels from the same image that have similar neighborhoods, copy, paste something similar if you will, or they try to record the distribution of pixel colors and and try to fill in something using that knowledge. And here comes the important part. None of these traditional approaches have an intuitive understanding of the contents of the image and that is the main value proposition of the neural network based learning techniques. This work also borrows from earlier artistic style transfer methods to make sure that not only the content, but the style of the inpainted regions also match the original image. It is also remarkable that this new method works with images that are devoid of symmetries and can also deal with cases where where we cut out really crazy irregularly shaped holes. Of course, like every good piece of research work, it has to be compared to previous algorithms. As you can see here, the quality of different techniques is measured against the reference output and it is quite clear that this method produces more convincing results than its competitors. For reference, Patchmatch is a landmark paper from almost 10 years ago that still represents the state of the art for non learning based techniques. The paper contains a ton more of these comparisons, so make sure to have a look. Without doubt this is going to be an invaluable tool for artists in the future. In fact, in this very series we use Photoshop's built in image inpainting tool on a daily basis. So this will make our lives much easier. Loving it. Also, did you know that you can get early access to each of these videos if you are addicted to the series? Have a look at Our Patreon page, patreon.com twominutepapers or just click the link in the video description. There are also other really cool perks like getting your name as a key supporter in the video description, or deciding the order of the next few episodes. We also support cryptocurrencies. The addresses are in the video description. And with this, you also help us make better videos in the future. Thanks for watching and for your generous support, and I'll see you next time.
