Video commentary (Two Minute Papers with Dr. Zsolnai, published 2021-03-19) reviewing an NVIDIA paper that reconstructs video by transmitting only a single reference image plus low-dimensional head-pose and expression data, discarding the original video frames. The presenter demonstrates that the method can reconstruct visually convincing video at far lower bitrates than H.264 and recent baselines (authors claim ~10x efficiency), and showcases additional capabilities: head rotation/frontalization, framing correction, and motion transfer between people. Noted limitations include artifacts around occluders, some neck jumpiness and warping. The video links to a Weights & Biases report and includes a sponsorship mention.
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This paper is really something else. Scientists at Nvidia came up with an absolutely insane idea for video conferencing. Their idea is not to do what everyone else is doing, which is transmitting our video to the person on the other end. No, no, of course not. That would be too easy. What they do in this work is take only the first image from this video and they throw away the entire video afterwards. But before that, it stores a tiny bit of information from it, which is how our head is moving over time and how our expressions change. That is an absolutely outrageous idea. And of course we like those around here. So does this work? Well, let's have a look. This is the input video. Note that this is not transmitted. Only the first image and some additional information and the rest of this video is discarded. And hold on to your papers, because this is the output of the algorithm compared to the input video. No, this is not some kind of misunderstanding. Nobody has copy pasted the results there. This is a near perfect reconstruction of the input, except that the amount of information we need to transmit through the network is significantly less than with previous compression techniques. How much less? Well, you know what's coming, so let's try it out. Here is the output of the new technique. And here is the comparison against H264, a powerful and commonly used video compression standard. Well, to our disappointment, the two seem close. The new technique appears better, especially around the glasses, but the rest is similar. And if you have been holding onto your paper so far, now squeeze that paper, because this is not a reasonable comparison. And that is because the previous method was allowed to transmit 6 to 12 times more information. Look, as we further decrease the data allowance of the previous method, it can still transmit more than twice as much information. And at this point there is no contest. This bitrate would be unusable for any kind of video conferencing. While the new method uses less than half as much information and still transmits a sharp and perfectly fine video. Overall, the authors report that their new method is ten times more efficient. That is unreal. This is an excellent video reconstruction technique, that much is clear. And if it only did that, it would be a great paper. But this is not a great paper. This is an absolutely amazing paper. So it does even more. Much, much more. For instance, it can also rotate our head and make a frontal video can also fix potential framing issues by translating our head and transferring all of our gestures to a new model. And it is also evaluated well, so all of these features are tested in isolation. So look at these two previous methods trying to frontalize the input video. One would think that it's not even possible to perform properly given how much these techniques are struggling with the task. Until we look at the new method. My goodness. There is some jumpiness in the neck movement in the output video here and some warping issues here, but otherwise very impressive results. Now, if you have been holding onto your papers so far, squeeze that paper. Because these previous methods are not some ancient papers that were published a long time ago. Not at all. Both of them were published within the same year and as the new paper. How amazing is that? Wow. I really liked this page from the paper which showcases both the images and the mathematical measurements against previous methods side by side. There are many ways to measure how close two videos are to each other. The up and down arrows tell us whether the given quality metric is subject to minimization or maximization. For instance, pixel wise errors are typically minimized, so lesser is better. But we are to maximize the peak signal to noise ratio. And the cool thing is that none of this matters too much as soon as we insert the new technique which really outpaces all of these. And we are still not done yet. So we said that the technique takes the first image, reads the evolution of expressions and the head pose from the input video and then it discards the entirety of the video save for the first image. The cool thing about this was that we could pretend to rotate the head pose information and the result is that the head appears rotated in the output image. That was great. But what if we take the source image from someone and take this data, the driving key point sequence from from someone else? Well, what we get is motion transfer. Look, we only need one image of the target person and we can transfer all of our gestures to them in a way that is significantly better than most previous methods. Now, of course not even this technique is perfect. He still struggles a great deal in the presence of occluder objects. But still, just the fact that this is possible feels like something straight out of a science fiction movie. What you see here is a report of this exact paper we have talked about, which was made by weights and biases. I put a link to it in the description, make sure to have a look. I think it helps you understand this paper better. During my PhD studies I trained a ton of neural networks which were used in our experiments. However, over time there was just too much data in our repositories and what I am looking for is not data but insight. And that's exactly how Weights and Biases helps you by organizing your experiments. It is used by more than 200 companies and research institutions including OpenAI, Toyota Research, GitHub and more. And get this Weights and Biases is free for all individuals, academics and and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
