Károly Zsolnai-Fehér explains Google's super resolution method for upscaling images without extra data. The learning-based approach groups image patches by features and maps them from low to high resolution, outperforming neural networks while training on just 10,000 images in about an hour.
Questions this video answers
Approximately 10,000 images.
About one hour of training time.
Károly Zsolnai-Fehér.
February 1, 2017.
Chapters
Dear fellow scholars, this is two minute papers with what is super resolution? Super resolution is a process where our input is a coarse low resolution image and the output is the same image, but now with more details and in high resolution. We'll also refer to this process as image upscaling. And in this piece of work we are interested in performing single image super resolution, which means that no additional data is presented to the algorithm that could help the process. Despite the incredible results seen in practically any of the crime solving television shows out there, our intuition would perhaps say that this problem, for the first sight sounds impossible. How could one mathematically fill in the details when these details are completely unknown? Well, that's only kind of true. Let's not confuse super resolution with image inpainting where we essentially cut an entire part out of an image and try to replace it, leaning on our knowledge of the surroundings of the missing part. That's a different problem. Here the entirety of the image is known and the details require some enhancing. And this particular method is not based on neural networks, but is still a learning based technique. The cool thing here is that we can use a training data set that is for all intents and purposes arbitrarily large. We can just grab a high resolution image, convert it to a lower resolution and we immediately have our hands on a training example for the learning algorithm. These would be the before and after images, if you will. And here during learning, the image is subdivided into small image patches and buckets are created to aggregate the information between patches that share similar features. These features include brightness, textures and the orientation of the edges. The technique looks at how the small and large resolution images relate to each other when viewed through the lens of these features. Two remarkably interesting things arose from this experiment. One, it outperforms existing neural network based techniques and two, it only uses 10,000 images and one hour of training time, which is in the world of deep neural networks, is so little it's completely unheard of. Insanity. Really, really well done. Some tricks are involved to keep the memory consumption low. The paper discusses how it is done and there are also plenty of other details within. Make sure a look. As always, it is linked in the video description. It can either be run directly on the low resolution image or alternatively we can first run a cheap and naive decade old upscaling algorithm and run this technique on the upscaled output to improve it. Note that super resolution is a remarkably competitive field of research. There are hundreds and hundreds of papers appearing on this every year and almost every single one of them seems to be miles ahead of the previous ones, where in reality the truth is that most of these methods have different weaknesses and strengths and so far I haven't seen any technique that would be viable for universal use. To make sure that a large number of cases is covered, the authors posted a sizable supplementary document with comparisons. This gives so much more credence to the results. I am hoping to see a more widespread adoption of this in future papers in this area. For now, when viewing websites I feel that we are close to the point where we could choose to transmit only the lower resolution images through the network and perform super resolution on them locally on our phones and computers. This will lead to significant savings on network bandwidth. We are living amazing times indeed. If you are enjoying the series, make sure to subscribe to the channel or you can also pick up really cool perks on our Patreon page through this icon with the letter P. Thanks for watching and for your generous support and I'll see you next time.
