Neural Portrait Relighting is Here!

Dear fellow scholars, this is two minute papers with in computer graphics, when we are talking about portrait relighting, we mean a technique that is able to look at an image and change the lighting and maybe even the materials or geometry after this image has been taken. This is a very challenging endeavor. So can neural networks put a dent into this problem and give us something new and better? You bet. The examples that you see here are done with this new work that uses a learning based technique and is able to change the lighting for human portraits and only requires one input image. You see, normally using methods in computer graphics to relight these images would require trying to find out what the geometry of the face, materials and lighting is from the image, and then we can change the lighting or other parameters, run a light simulation program and hope that the estimations are good enough to make it realistic. However, if we wish to use neural networks to learn the concept of portrait free lighting, of course we need quite a bit of training data since this is not trivially available. The paper contains a new DataSet with over 25,000 portrait images that are relit in five different ways. It also proposes a neural network structure that can learn this relighting operation efficiently. It is shaped a bit like an hourglass and contains an encoder and decoder parts. The encoder part takes an image as an input and estimates what lighting could have been used to produce it, while the decoder part is where we can play around with changing the lighting and it will generate the appropriate image that this kind of lighting would produce. What you see here are skip connections that are useful to save insights from different abstraction levels and transfer them from the encoder to the decoder network. So what does this mean exactly? Intuitively, it is a bit like using the lighting estimator network to teach the image generator what it has learned. So do we really lose a lot if we skip the skip connections? Well, quite a bit. Have a look here. The image on the left shows the result using all skip connections, while as we traverse to the right we see the results omitting them. These connections indeed make a profound difference. Let's be thankful for the authors of the paper as putting together such a dataset and trying to get an understanding as to what network architectures it would require to get great results like this takes quite a bit of work. I'd like to make a note about modeling subsurface light transport. This is a piece of footage from our earlier paper that we wrote as a collaboration with the Activision Blizzard company and you can see here that including this indeed makes a profound difference in the looks of a human face. I cannot wait to see some follow up papers that take more advanced effects like this into consideration for relighting as well. If you wish to find out more about this work, make sure to click the link in the video description. This episode has been supported by Weights and Biases. Here you see a write up of theirs where they explain how to visualize gradients running through your models and illustrate it through the example of predicting protein structure. They also have a live example that you can try Weights and Biases provides tools to track your experiments in your deep learning projects. It can save you a ton of time and money in these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.