Researchers developed an AI that removes obstructions like fences and reflections from images by analyzing multiple views of the same scene. It works on photos and video, with a fast mode running in seconds or a slower 15-minute version for better results.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Approximately two years ago we covered a work where a learning based algorithm was able to read the WI FI signals in a room to not only locate a person in a building, but even estimate their pose. An additional property of this method was that as you see here, it does not look at images but radio signals which also traverse in the dark and therefore this pose estimation also works well in poor lighting conditions. Today's paper offers a learning based method for a different, more practical data completion problem and it mainly works on image sequences. You see, we can give this one a short image sequence with obstructions, for instance the fence here, and it is able to find and remove this obstruction. And not only that, but it can also show us what is exactly behind the fence. How is that even possible? Well, note that we mentioned that the input is not an image, but an image sequence, a short video if you will. This contains the scene from different viewpoints and is one of the typical cases where if we would give all this data to a human, this human would take a long long time, but would be able to reconstruct what is exactly behind the fence because this data was visible from other viewpoints. But of course, clearly this approach would be prohibitively slow and expensive. The cool thing here is that this learning based method is capable of doing this automatically, but it does not stop there. I was really surprised to find out that it even works for video outputs as well. So if you did not have a clear sight of that tiger in the zoo, do not despair, just use this method and. And there you go. When looking at the results of techniques like this, I always try to look only at the output and try to guess where the fence was, obstructing it with many simpler image inpainting techniques. This is easy to tell if you look for it, but here I can't see a trace. Can you let me know in the comments? Admittedly, the resolution of this video is not very high, but the results look very reassuring. It can also perform reflection removal and some of the input images are highly contaminated by these reflected objects. Let's have a look at some results. You can see here how the technique decomposes the input into two images, one with the reflection and one without. The results are clearly not perfect, but they are easily good enough to make my brain focus on the real background without being distracted by the reflections. This was not the case with the input at all. Bravo. This use case can also be extended for videos and I wonder how much temporal coherence I can expect in the output. In other words, if the technique solves the adjacent frames too differently, flickering is introduced in the video, and this effect is the bane of many techniques that are otherwise really good on still images. Let's have a look. There is a tiny bit of flickering, but the results are surprisingly consistent. It also does quite well when compared to previous methods, especially when we are able to provide multiple images as an input. Now note that it says hours without online optimization. What could that mean? This online optimization step is a computationally expensive way to further improve separation in the outputs, and with that the authors propose a quicker and a slower version of the technique. This one without the online optimization step runs in just a few seconds, and if we add this step we will have to wait approximately 15 minutes. I had to read the table several times because researchers typically bring the best version of their technique to the comparisons, and it is not the case here. Even the quicker version smokes the competition. Loving it. Note that if you have a look at the paper in the video description, there are of course more detailed comparisons against other methods as well. If these AR glasses that we hear so much about come to fruition in the next few years, having an algorithm for real time glare reflection and obstruction removal would be beyond amazing. We truly live in a science fiction world. What a time to be alive. This episode has been supported by Snap, Inc. What you Here is SnapML, a framework that helps you bring your own machine learning models to Snapchat's AR lenses. You can build augmented reality experiences for Snapchat's hundreds of millions of users and help them see the world through a different lens. You can also apply to SNAP's AR Creator residency program with a proposal of how you would use Lens Studio for a creative project. If selected, you could receive a grant between 1 to $5,000 and work with Snap's technical and creative teams to bring your ideas to life. It doesn't get any better than that. Make sure to go to the link in the video description and apply for the residency program and try snapml today. Our thanks to Snap Inc. For helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
