Is Videoconferencing With Smart Glasses Possible? 👓

Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to have a look at the state of egocentric video conferencing. Now this doesn't mean that only we get to speak during a meeting. It means that we are wearing a camera which looks like this. And the goal is to use a learning algorithm to synthesize this frontal view of us. Now note that what you see here is the recorded reference footage. This is reality. And this would need to be somehow synthesized by the algorithm. If we could pull that off, we could add a low cost egocentric camera to smart glasses and it could pretend to see us from the front, which would be amazing for hands free video conferencing. That would be insanity. But wait a second. How is this even possible for us to even have a fighting chance? There are four major problems to overcome here. One, this camera lens is very close to us, which. Which means that it doesn't see the entirety of the face. That sounds extremely challenging. And if that wasn't bad enough too, we also have tons of distortion in the images. Or in other words, things don't look like they look in reality. We would have to account for that too. Three, it would also have to take into account our current expression, gaze, blinking and more. Oh boy. And finally four, the output needs to be photorealistic or even better, video realistic. Remember, we don't just need one image, but a continuously moving video output. So the problem is once again, input egocentric view output synthesized front of view. This is the reference footage, reality, if you will. And now let's see how this learning based algorithm is able to reconstruct it. Um, hello. Is this a mistake? They look identical, as if they were just copied here. No, you will see in a moment that it's not a mistake. This means that the AI is giving us a nearly perfect reconstruction of the remainder of the human face. That is absolutely amazing. Now, it is still not perfect. There are some differences. So how do we get a good feel of where the inaccuracies are? The answer is a difference image. Look. Regions with warmer colors indicate where the reconstruction is inaccurate compared to the real reference footage. For instance, with an earlier method by the name Pix2Pix, the hair and the beard are doing fine. While we have quite a bit of reconstruction error on the remainder of the face. So did the new method do better than this? Let's have a look together. Oh yeah, it does much better across the entirety of the face. It still has some trouble with the cable and the glasses, but otherwise this is a clean, clean image. Bravo. Now, we talked about the challenge of reconstructing expressions correctly. To be able to read the other person is of utmost importance during a video conference. So how good is it at gestures? Well, let's put it through an intense stress test. This is as intense as it gets without having access to Jim Carrey as a test subject, I suppose. And I bet there was a lot of fun to be had in the lab on this day. And the results are outstanding. Especially if we compare it again to the Pix2 Pix technique from 2017. I love this idea because if we can overcome the huge shortcomings of the egocentric camera, in return, we get an excellent view of subtle facial expressions and can deal with the tiniest eye movements, even twitches, tongue movements and more. And it really shows in the results. Now, please note that this technique needs to be trained on each of these test subjects. About four minutes of video footage is fine, and this calibration process only needs to be done once. So once again, the technique knows these people and had seen them before. But in return, it can do even more. If all of this is synthesized, we have a lot of control over the data and and the AI understands what much of this data means. So with all that extra knowledge, what else can we do with this footage, for instance, we can not just reconstruct, but create arbitrary head movement. We can guess what the real head movement is because we have a view of the background, we can simply remove it. Or from the movement of the background, we can infer what kind of head movement is taking place. And what's even better, we can not only get control over the head movement and change it, but even remove the movement from the footage altogether. And we can also remove the glasses and pretend to have dressed properly for an occasion. How cool is that? Now, make no mistake, this paper contains a ton of comparisons against a variety of other works as well. Here are some, but make sure to check them all out in the video description. Now, of course, even this new method isn't perfect. It does not work all that well in low light situations. But of course, let's leave something to improve for the next paper down the line. And hopefully in the near future, we will be able to seamlessly get in contact with our loved ones through smart glasses and egocentric cameras. What a time to be alive. What you see here is a report of this exact paper we have talked about with was made by weights and biases. I put a link to it in the description. Make sure to have a look I think it helps you understand this paper better if you work with learning algorithms on a regular basis. Make sure to check out weights and biases. Their system is designed to help you organize your experiments and it is so good it could shave off weeks or even months of work from your projects and is completely free for all individuals, academics and and open source projects. This really is as good as it gets and it is hardly a surprise that they are now used by over 200 companies and research institutions. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.