A neural network learns to remove shadows from portrait photos by training on images with artificially added shadows. The method handles most shadows well but struggles with detailed hair and sometimes produces unnatural results.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When we look at the COVID page of a magazine, we often see lots of well made but also idealized photos of people. Idealized here means that the photographer made them in a studio where they can add or remove light sources and move them around to bring out the best from their models. But most photos are not made in the studio. They are made out there in the wild with where the lighting is what it is and we can't control it too much. So with that today, our question is what if we could change the lighting after the photo has been made? This work proposes a cool technique to perform exactly that by enabling us to edit the shadows on a portrait photo that we would normally think of deleting. Many of these have to do with the presence of shadows, and you can see here that we can really edit these after the photo has been taken. However, before we start taking a closer look at the editing process, we have to note that there are different kinds of shadows. One, there are shadows cast on us by external objects. Let's call them foreign shadows. And there is self shadowing, which comes from the model's own facial features. Let's call those facial shadows. So why divide them into two classes? Simple. Because we typically seek to remove foreign shadows and edit facial shadows. The removal part can be done with a learning algorithm, provided that we can teach it with a lot of training data. Let's think about ways to synthesize such a large data set. Let's start with the foreign shadows. We need image pairs of test subjects with and without shadows to have a neural network learn about their relations, since removing shadows is difficult without further interfering with the image. And the authors opted to do it the other way around. In other words, they take a clean photo of the subject that's the one without the shadows, and then add shadows to it algorithmically. Very cool. And the results are not bad at all. And get this, they even accounted for subsurface scattering, which is the scattering of light under our skin. That makes a great deal of a difference. This is a reference from a paper we wrote with scientists at the University of Zaragoza and the Activision Blizzard company to add this beautiful effect to their games. Here is a shadow edge without subsurface scattering. Quite dark. And with subsurface scattering, you see this beautiful glowing effect. Subsurface scattering indeed makes a great deal of difference around hard shadow edges. So huge thumbs up for the authors for including an approximation of that. However, the synthesized photos are still a little suspect. We can still Tell that they are synthesized. And that is kind of the point. Our question is, can a neural network still learn the difference between a clean and a shadowy photo? Despite all this? And as you see, the problem is not easy. Previous methods did not do too well on these examples when you compare them to the reference solution. And let's see this new method. Wow. I can hardly believe my eyes. Nearly perfect. And it did learn all this on not real but synthetic images. And believe it or not, this was only the simpler part. Now comes the hard part. Let's look at how well it performs at editing the facial shadows. We can pretend to edit both the size and the intensity of these light sources. The goal is to have a little more control over the shadows in these photos. But whatever we do with them, the outputs still have to remain realistic. Here are the before and after results. The facial shadows have been weakened and depending on our artistic choices, we can also soften the image a great deal. Absolutely amazing. As a result, we now have a two step algorithm that first removes foreign shadows then and is able to soften the remainder of the facial shadows, creating much more usable portrait photos of our friends. And all this after the photo has been made. What a time to be alive. Now of course, even though this technique convincingly beats previous works, it is still not perfect. The algorithm may fail to remove some highly detailed shadows. You can see how the shadow of the hair remains in the output. In this other output, the hair shadows are handled a little better. There is some dampening, but the symmetric nature of the facial shadows here put the output results in an interesting no man's land where the opacity of the shadow has been decreased. But the result looks unnatural. I can't wait to see how this method will be improved. Two more papers down the line I will be here to report on it to you. So make sure to subscribe and hit the bell icon to not miss out on that. This episode has been supported by Weights and Biases. In this post they show you how to use their system to train distributed models with keras. You get all the required source code to make this happen, so make sure to click the link in the video. Description Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
