Neural networks can now upscale low-resolution game graphics in real-time while keeping motion smooth across frames. A new method beats Unreal Engine's TAU and other existing techniques, running efficiently enough for live gameplay.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Let's talk about video super resolution. The problem statement is simple. In goes a course video, the technique analyzes, it guesses what's missing and out comes a detailed video. You know, the CSI thing and hands. However, of course reliably solving this problem is anything but simple. This previous method is called Tacogan and and it is able to give us results that are often very close to reality. It is truly amazing how much this technique understands the world around us from just this training set of low and high resolution videos. However, as amazing as super resolution is, it is not the most reliable way of delivering high quality images in many real time applications. For instance video games. Note that in these applications we typically have more data at our disposal, but in return the requirements are also higher. We need high quality images at least 60 times per second. Temporal coherence is a necessity. Or in other words, no jarring, jumps and flickering is permitted. And hence one of the techniques often used in these cases is called supersampling. At the risk of simplifying the term supersampling means that we split every pixel into multiple pixels to compute a more detailed image and then display that to the user. Does this work? Yes it does. It works wonderfully, but it requires a lot more memory and computation. Therefore it is generally quite expensive. So our question today, is it possible to use these amazing learning based algorithms to do it a little smarter? Let's have a look at some results from a recent paper that uses a neural network to perform super sampling at a more reasonable computational cost. Now in goes the low resolution input and my goodness, like magic, out comes this wonderful much more detailed result. And here is the reference which is the true higher resolution image. Of course, the closer the neural super sampling is to this, the better. And as you see, this is indeed really close and much better than the pixelated inputs. Let's do one more. Wow, this is so close. I feel we are just a couple papers away from a result being indistinguishable from the real reference image. Now we noted that this previous method has access to more information than the previously showcased super resolution method. It looks at not just one frame, but a few previous frames as well, can use an estimation of the motion of each pixel over time and also gets depth information. These can be typically produced inexpensively with any major game engine. So how much of this data is required to train this neural network? Hold on to your papers. Because 80 videos were used and the training took approximately one and a half days on one Titan V which is an expensive but commercially available graphics card. And no matter, because this step only has to be done once and and depending on how high the resolution of the output should be. With the faster version of the technique, the upsampling step takes from 8 to 18 milliseconds, so this runs easily in real time. Now of course, this is not the only modern super sampling method. This topic is subject to a great deal of research, so let's see how it compares to others. Here you see the results with taau, the temporal upsampling technique used in Unreal Engine, an industry standard game engine. And look, this neural supersampler is significantly better at anti aliasing or in other words smoothing these jagged edges. And not only that, but it also resolves many more of the intricate details of the image. Temporal coherence has also improved a great deal as you see the video output is much smoother for the new method. This paper also contains a ton more comparisons against recent methods, so make sure to have a look. Like many of us, I would love to see a comparison against Nvidia's DLSS solution, but I haven't been able to find a published paper on the later versions of this method. I remain very excited about seeing that too. And for now the future of video game visuals and other real time graphics applications is looking as excited as as it's ever been. What a time to be alive. This episode has been supported by Weights and Biases. In this post they show you how to use their system to explore what exact code changes were made between two machine learning experiments. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs as such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
