Dr. Zsolnai explains a neural network that removes backgrounds from video using just two photos of the same scene, one with and one without the subject. A generator predicts the alpha matte while a discriminator network refines it, producing clean composites even with moving backgrounds, though large motion still causes problems.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When shooting feature length movies or just trying to hold meetings from home through Zoom or Skype, we can make our appearance a little more professional by hiding the mess we have in the background by changing it to something more pleasing. Of course, this can only happen if we have an algorithm at hand that can detect what the foreground and the background is, which typically is easiest when we have a green screen behind us that is easy to filter for even the simpler algorithms out there. However, of course not everyone has a green screen at home, and even for people who do may need to hold meetings out there in the wilderness. Unfortunately, this would mean that the problem statement is the exact opposite of what we've said. Or in other words, the background is almost anything else but a green screen. So is it possible to apply some of these newer neural network based learning algorithms to tackle this problem? Well, this technique promises to make this problem much, much easier to solve. All we need to do is take two photographs, one with and one without the test subject, and it will automatically predict an alpha matte that isolates the test subject from the background. If you have a closer look, you'll see the first part of why this problem is difficult. This matte is not binary. So the final compositing process is given not as only foreground or only background for every pixel in the image, but there are parts typically around the silhouettes and hair that need to be blended together. This blending information is contained in the gray parts of the image and are especially difficult to predict. Let's have a look at some results. You see the captured background here and the input video below, and you see that it is truly a sight to behold. It seems that this person is really just casually hanging out in front of a place that is definitely not a whiteboard. It even works in cases where the background or the camera itself is slightly in motion. Very cool. It really is much, much better than these previous techniques where you see that temporal coherence is typically a problem. This is the flickering that you see here which arises from the vastly different predictions for the alpha matte between neighboring frames in the video. Opposed to previous methods, this new technique shows very little of that. Excellent. Now we noted that a little movement in the background is permissible, but it really means just a little. If things get too crazy back there, the outputs are also going to break down. This wizardry all works through a generative adversarial network in which one neural network generates the output results. This by itself didn't work all that well because the images used to train this neural network can differ greatly from the backgrounds that we record out there in the wild. In this work, the authors bridged the gap by introducing a detector network that tries to find faults in the output and tell the generator if it has failed to fool it. As the two neural networks duke it out, they work and improve together, yielding these incredible results. Note that there are plenty of more contributions in the paper, so please make sure to have a look for more details. What a time to be alive. What you see here is an instrumentation of this exact paper we have talked about, which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
