Dr. Károly Zsolnai-Fehér explains a technique that removes people from video by detecting their pose and related motion like shadows, reflections, and photobombs, then uses a neural renderer to fill in the gaps. The method warps time to align events and runs in an interactive app for real-time previews.
Questions this video answers
November 28, 2020
PIX2PIX
More than 200 companies and research institutions
Weights & Biases (W&B)
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Let's ask these humans to jump into the pool at the same time. On three, two, one, go. Then this happens. And if we slow things down like we would slow down an action sequence in a movie. Look, this is one, and then two, and then three. Not even close to being in the same time. We see that this footage is completely unsalvageable. And please note the smile at the end of the video. I'll tell you in a moment why. Imagine that after a little training, the little chaps were able to jump into the pool at the same time. Congratulations. But wait a second. Look. The smile is the same as in the previous footage. This is not a different video. This is the same video as before. But. But it has been retimed to seem as if the jumps happened at the same time. Absolutely amazing. And how about this footage? They move in tandem. Totally in tandem. Right. Except that this was the original footage, which is a complete mess. And now with this technique, it could be retimed as if this happened. Incredible work. So what is this with Orgerdri? This is a new learning based technique that can pull off nice tricks like this. And it can deal with cases that would otherwise be extremely challenging to retime by hand. To find out why, let's have a look at this footage. Mhm, got it. And now let's try to pretend that this meeting never happened. Now first, to be able to do this, we have to be able to recognize the test subjects of the video. This method performs pose estimation. These are the skeletons that you see here. Nothing new here. This can be done with off the shelf components. However, that is not nearly enough to remove or retime them. We need to do more. Why is that? Well, look. Mirror reflections and shadows. If we would only be able to remove the person, these secondary effects would give the trick away in a second. To address this issue, one of the key contributions of this new AI based technique is that it is able to find these mirror reflections, shadows and even more, which can be defined as motions that correlate with the test subject, or in other words, things that move together with the humans. And if we have all of these puzzle pieces, we can use a neural renderer to remove people or even adjust the timing of these videos. And now hold on to your papers, because it is not at all limited to shadows and mirror reflections. Remember this example here. The algorithm recognizes that these people cause deformations in the trampolines and is able to line them up together. Note that we need some intense and variable time warping to make this happen. And as an additional bonus, the photobombing human has been removed from the footage. The neural renderer used in this work is from the PIX2PIX paper from only three years ago and now, with some ingenious improvements, it can be elevated to a whole new level. Huge congratulations to the authors for this incredible achievement. And all this can be done in an interactive app that is able to quickly preview the results for us. What a time to be alive. What you see here is a report of this exact paper we have talked about which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. During my PhD studies I trained a ton of neural networks which were used in our experiments. However, over time there was just too much data in our repositories and what I am looking for is not data but insight. And that's exactly how Weights and Biases helps you by organizing your experiments. It is used by more than 200 companies and research institutions including OpenAI, Toyota Research, GitHub and more. And get this Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnba.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
