Dr. Károly Zsolnai-Fehér shows how neural networks can replace skies in video, inserting moving spaceships, castles, planets, and storms. The technique uses sky detection, optical flow to track motion, and lighting adjustment to match the scene.
Questions this video answers
Dr. Károly Zsolnai-Fehér
November 24, 2020
A sky matting network; a motion estimator (optical flow); and a recoloring module.
Inserting a moving spaceship, placing a castle, adding an extra planet, and synthesizing a thunderstorm.
Weights & Biases
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Through the power of neural network based learning algorithms today it is possible to perform video to video translation. This means that for instance, in goes a daytime video and out comes a nighttime version of the same footage. This previous method had some remarkable properties. For instance, look, you can even see the reflections of the night lights appearing on the car hood. This is reassuring news indeed, because the spark of some sort of intelligence was already there in these algorithms. And believe it or not, this is old, old news because we have been able to do these for about three years now. Of course this was not perfect. The results were missing a lot of details. The frames of the video were quite a bit apart, artifacts were abundant and more. But this paper was a huge leap at the time. And today I wonder how far have we come in three years? What can we do now that we were not able to do three years ago? Let's have a look together. This input video is much more detailed, a lot more continuous. So I wonder what we could do with this. Well, hold on to your papers and woohoo. Look at that. One, we have changed the sky and put a spaceship in there. That is already amazing. But look. Two, the spaceship is not stationary, but moves in harmony with the other objects in the video. And three, since the sky has changed, the lighting situation has also changed. So the colors of the remainder of the image also have to change. Let's see the before and after on that. Yes, excellent. And we can do so much more with this. For instance, put a castle in the sky. Or you know what, let's think big. Make it an extra planet, please. Wow. Thank you. And wait a minute, if it is able to recolor the image after changing its surroundings, how well does it do it? Can we change the background to a dynamic one? What about a thunderstorm? Oh yeah, that's as dynamic as it gets. And the new method handles this case beautifully. And before we look under the hood to see how all this wizardry is done, let's list our expectations. One, we expect that it has to know what pixels to change to load a different sky model. Two, it should know how the image is changing and rotating over time. And three, some recoloring has to take place. Now let's have a look and see if we find the parts we are expecting. And yes, it has a sky matting network. This finds the parts of the image where the sky is. There is the motion estimator that computes the optical flow of the image. This tracks the movement of the sky over time and there is the recoloring module as well. So there we go. This can do not only sky replacement, but detailed weather and lighting synthesis is also possible for these videos. What a time to be alive. Now if you remember, first we looked at the results, listed our expectations and then we looked at the architecture of this neural network. This is a technique that I try to teach to my students in my Light transport simulation course at the Technical University of Vienna and I think if you check it out you will be surprised by how effective it is. Now note that I was always teaching it to a handful of motivated students and I believe that the teachings shouldn't only be available for the privileged few who can afford a education, but the teachings should be available for everyone. Free education for everyone. That's what I want. So the course is available free of charge for everyone. No strings attached. So make sure to click the link in the video description to get started. We write a full light simulation program from scratch there and learn about physics, the world around us and more. And what is perhaps even more important is that I try to teach you a powerful way of understanding difficult mathematical concepts. Make sure to have a look. What you see here is a report of this exact paper we have talked about which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
