A new AI method upsamples images in real time using progressive learning and generative adversarial networks. It does 2× upsampling in under a second and 8× in about 4.5 seconds while matching state-of-the-art quality. The authors released the source code.
2018-09-06
Less than 1 second
About 4.5 seconds
Yes (speaker notes that source code is available)
Dear fellow scholars, this is two minute papers with when looking for illustrations for a presentation, most of the time I quickly find an appropriate photo on the Internet. However, many of these photos are really low resolution. This often creates a weird situation where I have to think okay, do I use the splotchier lower resolution image that gets the point across or take a high resolution crisp image that is less educational? In case you are wondering, I encounter this problem for almost every single video I make for this channel. As you can surely tell, I am waiting for the day when super resolution becomes mainstream. Super resolution means that we have a low resolution image that lacks details and we feed it to a computer program which hallucinates all the details onto it, creating a crisp, high resolution image. This way I could take my highly relevant but blurry image, improve it and use it in my videos. As adding details to images clearly requires a deep understanding of what is shown in these images, our seasoned fellow scholars immediately know that learning based algorithms will be ideal for this task. While we are looking at some amazing results with this new technique, let's talk about the two key differences that this method introduces. 1. It takes a fully progressive approach, which means that we don't immediately produce the highest resolution output we are looking for, but slowly leapfrog our way through intermediate steps, each of which is only slightly higher resolution than the input. This means that the final output is produced over several steps where each problem is only a tiny bit harder than the previous one. This is often referred to as curriculum learning and it not only increases the quality of the solution, but is also easier to train as solving each intermediate step is only a little harder than the previous one. It is a bit like how students learn in school. First, the students are shown some easy introductory tasks to get a grasp of a problem and slowly work their way towards mastering a field by solving problems that gradually increase in difficulty. 2. Now we can start playing with the thought of using a generative adversarial network. We talk a lot about this architecture in this series. At this time I will only note that training these is fraught with difficulties and so every bit of help we can get is more than welcome. So the role of curriculum learning is to help easing this process. Note that this research field is well explored and has a remarkable number of papers, so I was expecting a lot of comparisons against competing techniques and when looking at the paper and the supplementary materials, boy did I get it. Make sure to have a look at the paper. It contains a very exhaustive validation section which reveals that if we measure the error of the solution it in terms of human perception, it is only slightly lower quality than the best technique. However, this one is five times quicker, offering a really nice balance between quality and performance. So what about the actual numbers for the execution time? For instance, upsampling an image to increase its resolution to twice its original size takes less than a second, and we can go up to even eight times the original resolution, which is which also only takes four and a half seconds. The quality and the execution times indicate that we are again one step closer to mainstream super resolution. What a time to be alive. The source code of this project is also available. Thanks for watching and for your generous support, and I'll see you next time.
