A new AI method segments people and their visual effects from videos, then removes or duplicates them cleanly. It handles shadows, reflections, smoke, and ripples attached to subjects, and works with moving cameras. Some artifacts remain, but the results are striking.
Dr. Zsolnai (Two Minute Papers / uploaded by LARRUZ)
August 25, 2021
Over 200 companies and research institutions
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to see magical things that open up when we are able to automatically find the foreground and the background of a video. Let's see why that matters. This new technique leans on a previous method to find the boy and the dog. Let's call this level one segmentation. So far so good, but this is not the state of the art yet. Now comes level two. It also found the shadow of the boy and the shadow of the dog. Now we're talking. But it doesn't stop there, it gets even better. Level 3. This is where things get out of hand. Look, the dog is occluding the boy's shadow and it is able to deal with that too. So if we can identify all of the effects that are attached to the boy and the dog, what can we do with all this information? Well, for instance, we can even remove them from the video. Nothing to see here. Now a common problem is that still the silhouette of the subject still remains in the final footage. So let's take a close look together. I don't see anything at all. Wow. Do you let me know in the comments below just to showcase how good this removal is. Here is a good technique from just one year ago. Do you see it? This requires the shadows to be found manually, so we have to work with that. And still in the outputs you can see the silhouette we mentioned. And how much better is the new method? Well, it finds the shadows automatically. That is already mind blowing. And the outputs are. Yes, much cleaner, not perfect. There is still some silhouette action, but if I were not actively looking for it, I might not have noticed. Can also remove people from this trampoline scene. And not only the bodies, but it also removes their effect on the trampolines as well. Wow. And as this method can perform all this reliably, it opens up the possibility for new magical effects. For instance, we can duplicate this test subject and even fade it in and and out. Note that it has found its shadows as well. Excellent. So it can deal with finding not only the shape of the boy and the dog, and it knows that it's not enough to just find their silhouettes, but it also has to find additional effects they have on the footage. For instance, their shadows. That is wonderful. And what is even more wonderful is, is that this was only one of the simpler things it could do. Shadows are not the only potential correlated effects. Look, a previous method was able to find the swan here, but it's not enough to remove it because it Also has additional effects on the scene. What are those? Well, look, it has reflections and it creates ripples too. This is so much more difficult than. Than just finding shadows. And now let's see the new method. And whoa. It knows about the reflections and ripples, finds both of them and gives us this beautifully clean result. Nothing to see here. And also, look at this elephant. Removing just the silhouette of the elephant is not enough. It also has to find all the dust around it. And it gets worse. The dust is changing rapidly over time. And believe it or not, wow. It can find the dust tool and remove the elephant. Again, nothing to see here. And if you think that this dust was the new algorithm at its best, then have a look at this drifting car. Previous method. Yes, that is the car. But you know what I want? I want the smoke gone too. So that's probably impossible, right? Well, let's have a look. Wow. I can't believe it. It grabbed and removed the car and the smoke together. And once again, nothing to see here. So what are those more magical things that this opens up? Watch carefully. It can make the colors pop here. And remember, it can find the reflections of the flamingo. So it keeps not only the flamingo, but the reflection of the flamingo in color as well. Absolutely amazing. And if we can find the background of the video, we can even change the background. This works even in the presence of a moving camera, which is a challenging problem. Now, of course, not even this technique is perfect. Look here, the reflections are copied off of the previous scene and it shows on the new one. So what do you think? What would you use this technique for? Let me know in the comments. Or if you wish to discuss similar topics with other fellow scholars in a warm and welcoming environment, make sure to join our Discord channel. Also, I would like to send a big thank you to the mods and everyone who helps running this community. The link to the server is available in the video description. You are invited. What you see here is a report of this exact paper we have talked about, which was made by weights and biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better if you work with learning algorithms on a regular basis. Make sure to check out weights and biases. Their system is designed to help you organize your experiments. And it is so good, it could shave off weeks or even months of work from your projects and is completely free for all individuals, academics and open source projects. This really is as good as it gets. And it is hardly a surprise that they are now used by over 200 companies and research institutions. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
