Google’s New AI Puts Video Calls On Steroids! 💪

Dear fellow scholars, this is two minute papers with Dr. Zsolnai. With the increased popularity of online meetings, telepresence applications are on the rise where we can talk to each other from afar. Today, let's see how these powerful new neural network based learning methods can be applied to them. It turns out they can help us do everyone's favorite, which is showing up to a meeting and changing our background to pretend we are somewhere else. Now that is a deceptively difficult problem. Here the background has been changed. That is the easier problem. But look, the lighting of the new environment hasn't been applied to the subject. And now hold on to your papers and check this out. This is the result of the new technique after it recreates the image as if she was really there. I particularly like the fact that the results include high quality specular highlights too. Or in other words, the environment reflecting off of our skin. However, of course this is not the first method attempting this. So let's see how it performs compared to the competition. These techniques are from one and two years ago and they don't perform so well. Not only did they lose a lot of detail all across the image, but the specular highlights are gone. As a result, the image feels more like a video game character than a real person. Luckily, the authors also have access to the reference information to make our job comparing the results easier. Roughly speaking, the more the outputs look like this, the more the better. So now hold on to your papers and let's see how the new method performed. Oh yes, now we're talking. Now of course, not even this is perfect. Clearly the specularity of clothing was determined incorrectly and the matting around the thinner parts of the hair could be better, which is notoriously difficult to get right. But this is a huge step forward in just one one paper and we are not nearly done. There are two more things that are found to be remarkable about this work. One is that the whole method was trained on still images, yet it still works on video too. And we don't have any apparent temporal coherence issues. Or in other words, no flickering arises from the fact that it processes the video not as a video, but a series of separate images. Very cool. Two, if we are in a meeting with someone and we really like their background, we can simply borrow it. Look, this technique can take their image, get the background out, estimate its lighting, and give the whole package to us too. I think this will be a game changer. People may start to become more selective with these backgrounds. They not just because of how the background looks, but because how it makes them look. Remember, lighting off of a well chosen background makes a great deal of a difference in our appearance in the real world and now with this method in virtual worlds too. And this will likely happen not decades from now, but in the near future. So this new method is clearly capable of some serious magic. But how? What is going on under the hood? To achieve this, this method performs two important steps to accomplish this. Step number one is matting. This means separating the foreground from the background and then if done well, we can now easily cut out the background and also have the subject on a separate layer and proceed to step number two which is relighting. In this step, the goal is to estimate the illumination of the new scene and recolor the subject as if she were really there. This new technique performs both, but most of the contributions lie in this step. To be able to accomplish this, we have to be able to estimate the material properties of the subject. The technique has to one where the diffuse parts are, these are the parts that don't change too much as the lighting changes, and 2 where the specular parts are. In other words, shiny regions that reflect back the environment more clearly. Putting it all together, we get really high quality relighting for ourselves. And given that this was developed by Google, I expect that this will supercharge our meetings quite soon. And just imagine what we will have two more papers down the line. My goodness, what a time to be alive. This video has been supported by Weights and Biases. Check out their recent offering Fully Connected, a place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.