A short YouTube explainer by Károly Zsolnai-Fehér summarizing a paper on neural foveated reconstruction: a neural network reconstructs near-perfect images from highly sparse foveated input (under 10% of pixels) for head-mounted displays by using gaze information over time. The video highlights visual examples (reconstruction vs. sparse input), compares against multi-resolution methods and PatchMatch inpainting, notes training on ~350,000 short natural video sequences, real-time performance, and that the paper (first author Anton Kaplan) includes rigorous mathematical evaluation and a user study. The video ends with personal remarks and a Linode sponsorship/CTA.
Chapters
Dear fellow scholars, this is two minute papers with zsolnai. As humans, when looking at the world, our eyes and brain does not process the entirety of the image we have in front of us, but plays an interesting trick on us. We can only see fine details in a tiny, tiny foveated region that we are gazing at, while our peripheral or indirect vision only sees a sparse blurry version of the image and the rest of the information is filled in by our brain. This is very efficient because our vision system only has to process a tiny fraction of the visual data that is in front of us and it still enables us to interact with the world around us. So what if we would take a learning algorithm that does something similar for digital videos? Imagine that we would need to render a sparse video with only every 10th pixel filled with information and some kind of neural network based technique would be able to reconstruct the full image similarly to what our brain does. Yes, that sounds great, but that is very little information to reconstruct an image from. So is it possible? Well, hold on to your papers because this new work can reconstruct a near perfect image by looking at less than 10% of the input pixels. So we have this as an input and we get this. Wow. What is happening here is called a neural reconstruction of foveated rendering data or you are welcome to refer to it as foveated reconstruction. In short, during your conversations over dinner, the scrambled text part here is quite interesting. One might think that, well, it could be better, however, given the fact that if you look at the appropriate place in the sparse image, the I not only cannot read the text, I am not even sure if I see anything that indicates that there is a text there at all. So far the example assumed that we are looking at a particular point in the middle of the screen. And the ultimate question how does this deal with a real life case where the user is looking around? Well, let's see. This is the input and the reconstruction witchcraft. Let's have a look at some more results. Note that this method is developed for head mounted displays where we have information on where the user is looking over time and this can make all the difference in terms of optimization. You see a comparison here against a method labeled as multi resolution. This is from a paper by the name foveated 3D graphics and you can see that the difference in the quality of the reconstruction is truly remarkable. Additionally, it has been trained on 350,000 short natural video sequences and the whole thing runs in real time. Also note that we often discuss image inpainting methods in this series. For instance, what you see here is the legendary patchmatch algorithm that is one of these and it is able to fill in missing parts of an image. However, in image inpainting, most of the image is intact with smaller regions that are missing. This is even more difficult than image inpainting because the vast majority of the image is completely missing. The fact that we can now do this with learning based methods is absolutely incredible. The first author of the paper is Anton Kaplan, who's a brilliant and very rigorous mathematician. So of course the results are evaluated in detail both in terms of mathematics and with the user study. Make sure to have a look at the paper for more on that. We got to know each other with Anton during the days when all we did was light transport simulations all day every day and were always speculating about potential projects. And to my great sadness, somehow, unfortunately, we never managed to work together for a full project again. Congratulations Anton. Beautiful work. What a time to be alive. This episode has been supported by Linode. Linode is the world's largest independent cloud computing provider. They offer affordable GPU instances featuring the Quadro RTX 6000 which is tailor made for AI, scientific computing and computer graphics projects. Exactly the kind of works you see here in this series. If you feel inspired by these works and you wish to run your experiments or deploy your already existing works through a simple and reliable hosting service, make sure to join over 800,000 other happy customers and choose Linode. To spin up your own GPU instance and receive a $20 free credit, visit linode.com papers or click the link in the video description and use the promo code Papers20 during sign up. Give it a try today. Our thanks to Linode for supporting the series and helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
