A Two Minute Papers episode (hosted by Dr. Zsolnai, uploaded Oct 13, 2020) reviewing recent advances in AI-based video inpainting. The video contrasts older image-based inpainters (e.g., PatchMatch, earlier Nvidia work) with a new learning-based method that achieves stronger temporal coherence and can also spatially expand frames. Qualitative demos (person/bear removal, shadow masking, frame expansion) are shown, then quantitative evaluation on the DAVIS dataset (150 sequences) where the new method notably outperforms prior work. The episode closes with a sponsorship plug for Lambda GPU Cloud.
Questions this video answers
October 13, 2020.
Dr. Zsolnai (Two Minute Papers); content created/uploaded by LARRUZ.
Densely Annotated Video Segmentation (DAVIS) — 150 video sequences.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Have you ever had a moment where you took the perfect photo? But upon closer inspection, there was this one annoying thing that ruined the whole picture. Well, why not just take a learning algorithm to erase those cracks in the facade of a building or a photobombing ship, or to even reimagine ourselves with different eye colors. Or we can try one of the many research works that are capable of something that we call image inpainting. What you see here is the legendary patch match algorithm at work, which believe it or not, is a handcrafted technique from more than 10 years ago. Later, scientists at Nvidia published a more modern inpainter that uses a learning based algorithm to do this more reliably and for a greater variety of images. These all work really well. But a common denominator for these techniques is that they all work on inpainting still images. Would this be a possibility for video? Like removing a moving object or a person from a video? Is this possible? Or is it science fiction? Let's see if these learning based techniques can really do more. And now hold on to your papers because this new work can really perform perfect proper inpainting for video. Let's give it a try by highlighting this human and pro tip also highlight the shadowy region for inpainting to make sure that not only the human, but its silhouette also disappears from the footage. And look. Wow. Let's look at some other examples. Now that's really something because video is much more difficult due to the requirement of temporal coherence, which means that it's not nearly enough. If the images are inpainting really well individually, they also have to look good. If we weave them together into a video, you will hear and see more about this in a moment. Not only that, but if we highlight a person, this person not only needs to be inpainted, but we also have to track the boundaries of this person throughout the footage and then inpaint a moving region. We get some help with that, which I will also talk about in a moment. Now, as you see here, these all work extremely well and believe it or not, you have seen nothing yet because so far. Another common denominator in these examples was that we highlighted regions inside the video. But that's not all. If you have been holding onto your paper so far, now squeeze that paper because we can also go outside and expand our video spatially with even more content. This one is very short, so I will keep looping It. Are you ready? Let's go. Wow. My goodness. The information from inside of the video frames is reused to infer what should be around the video frame. And all this in a temporally coherent manner. Now of course this is not the first technique to perform this, so let's see how it compares to the competition by erasing this bear from the video footage. The remnants of the bear are visible with a wide selection of previously published techniques from the last few years. This is true even for these four methods from last year. And let's see how this new method did on the same case. Yup, very good. Not perfect. We still see some flickering. This is the temporal coherence example of or the lack thereof that I have promised earlier. But now let's look at this example with the BMXrider. We see similar performance with the previous techniques. And now let's have a look at the new one. Now that's what I'm talking about. Not a trace left from this person. The only clue that we get in reconstructing what went down here is the camera movement. It truly feels like we are living in a science fiction world. What a time to be alive. Now these were the qualitative results and now let's have a look at the quantitative results. In other words, we saw the videos, now let's see what the numbers say. We could talk all day about the peak signal to noise ratios or structural similarity or other ways to measure how good these techniques are, but you will see in a moment that it is completely unnecessary. Why is that? Well, you see here that the second best results are underscored and highlighted with blue. As you see, there is plenty of competition as the blues are all over the place. But there is no competition at all for the first place because this new method smokes the competition in every category. This was measured on a dataset by the name Densely Annotated Video Segmentation Davis. In short, this contains 150 video sequences and it is annotated, which means that many of the objects are highlighted throughout this video. So for the cases in this dataset, we don't have to deal with the tracking ourselves. I am truly out of ideas as to what I should wish for. Two more papers down the line. Maybe not only removing the tennis player, but putting myself in there as a proxy. We can already grab a controller and play as if we were real characters in real broadcast footage. So who really knows? Anything is possible. Let me know in the comments what you have in mind for potential applications and what you would be excited to see two more papers down the line. This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lambda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold on to your papers because Lambda GPU Cloud can cost less than half of AWS and Azure. Plus they are the only cloud service with 48 gigabyte RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for for one of their amazing GPU instances today. Our thanks to Lamda for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
