Károly Zsolnai-Fehér explains a CNN that handles image downscaling, decolorization, and HDR tone mapping at once. In user studies, the learned method beat Reinhard's tone mapper roughly 75% of the time, outperforming classical algorithms across all three tasks.
February 20, 2019.
Károly Zsolnai-Fehér.
About 75% of users preferred the new method.
Dear fellow scholars, this is two minute papers with Whenever we build a website, a video game, or do any sort of photography and image manipulation, we often encounter the problems of image downscaling, decolorization and HDR tone mapping. This work offers us one technique that can do all three of these really well. But first, before we proceed, why are we talking about downscaling? We are in the age of AI, where a computer program can beat the best players in chess and go. So why talk about such a trivial challenge? Well, have a look here. Imagine that we have this high fidelity input image and due to file size constraints, we have to produce a smaller version of it. If we do it naively, this is what it looks like. Not great, right? To do a better job at this, our goal would be that the size of the image would be reduced, but while still retaining the intricate details of this image. Here are two classical downsampling techniques. Better, but the texture of the skin is almost completely lost. Have a look at this. This is what this learning based technique came up with. Really good, right? It can also perform decolorization. Again, a problem that sounds trivial for the unassuming scholar, but when taking a closer look, we notice that there are many different ways of doing this, and somehow we seek a decolorized image that still relates to the original as faithfully as possible. Here you see the previous methods that are not bad at all, but this new technique is great at retaining the contrast between the flower and its green leaves. At this point, it is clear that deciding which output is the best is highly subjective. We'll get back to that in a moment. It is also capable of doing HDR tone mapping. This is something that we do when we capture an image with a device that supports a wide dynamic range, in other words, a wide range of colors, and we wish to display it on our monitor, which has a more limited dynamic range. And again, clearly there are many ways to do that. Welcome to the wondrous world of tone mapping. Note that there are hundreds upon hundreds of algorithms to perform these operations in computer graphics research. And also note that these are very complex algorithms that took decades for smart researchers to come up with. So the seasoned fellow scholar shall immediately ask, why talk about this work at all? What's so interesting about it? The goal here is to create a little more general learning based method that can do a great job at not one, but all three of these problems. But how great exactly? And how do we decide how good these images are? To answer both of these questions at the same time, if you've been watching this series for a while, then you are indeed right. The authors created a user study which shows that for all three of these tasks, according to the users, the new method smokes the competition. It is not only more general, but also better than most of the published techniques. For instance, Reinhard's amazing tone mapper has been an industry standard for decades now. And look, almost 75% of the people prefer this new method over that. What required super smart researchers before can now be done with a learning algorithm. Unreal. What a time to be alive. A key idea for this algorithm is that this convolutional neural network that you see on the left is able to produce all three of these operations at the same time. And to perform that, it is instructed by another neural network to do this in a way that preserves the visual integrity of the input images. Make sure to have a look at the paper for more details on how this perceptual loss function is defined. And if you wish to help us tell these amazing stories to even more people, please consider supporting us on Patreon. Your unwavering support on Patreon is the reason why this show can exist, and you you can also pick up cool perks there, like watching these videos in early access, deciding the order of the next few episodes, or even getting your name showcased in the video description. As a key supporter, you can find us at patreon.com TwominutePapers or as always, just click the link in the video description. Thanks for watching and for your generous support, and I'll see you next time.
