An untrained convolutional neural network with random weights can restore images better than classical methods. It removes JPEG artifacts, fills in missing parts, upscales, and denoises without ever being trained on data.
Dear fellow scholars, this is two minute papers with this work is about performing useful image restoration tasks with a convolutional neural network. With an additional twist, its main use cases are as 1. In the case of JPEG artifact removal, the input is this image with many blocky artifacts that materialized during compression and the output is a restored version of this image. 2. Image inpainting where some regions of the input image are missing and are to be filled with useful and hopefully plausible information. 3. Super resolution where the input image is intact but is very coarse and has low resolution and the output should be a more detailed higher resolution version of the same image. This is the classic enhanced scenario from the CSI TV series. It is typically hard to do because there is a stupendously large number of possible high resolution image solutions that we could come up with as an output. 4. Image Denoising is also a possibility. The standard way of doing these is that we train such a network on a large database of images so that they can learn the concept of many object classes such as humans, animals and more, and also the typical features and motifs that are used to construct such images. These networks have some sort of understanding of these images and hence can perform these operations better than most handcrafted algorithms. So let's have a look at some comparisons. Do you see these bold lettered labels that classify these algorithms as trained or untrained? The bicubic interpolation is a classic untrained algorithm that almost naively tries to guess the pixel colors by averaging its neighbors. This is clearly untrained because it does not take a database of images to learn on. Understandably the fact that these results are lackluster is to show that non learning based algorithms are not great at this. The SR restnet is a state of the art learning based technique for super resolution that was trained on a large database of input images. It is clearly doing way better than bicubic interpolation. And look, we have this deep prior algorithm that performs comparably well but is labeled to be untrained. So what is going on here? And here comes the twist. This convolutional neural network is actually untrained. This means that the neuron weights are randomly initialized which generally leads to completely useless results on most problems. So no aspect of this network works through the data it has learned on all the required information is contained within the structure of the network itself. We all know that the structure of these neural networks matter a great deal, but in this case it is shown that it is at least as important as the training data itself. A very interesting and esoteric idea indeed. Please make sure to have a look at the paper for details as there are many details to be understood. To get a more complete view of this conclusion in the comparisons beyond the images, researchers often publish this PSNR number that you see for each image. This is the peak signal to noise ratio, which means how close the output image is to the ground truth, and this number is of course always subject to maximization. Remarkably, this untrained network performs well on both images with natural patterns and man made objects. Reconstruction from a pair of flash and no flash photography images is also a possibility, and the algorithm does not contain the light leaks produced by a highly competitive handcrafted algorithm. A joint bilateral filter. Quite remarkable indeed. The supplementary materials and the project website contain a ton of comparisons against competing techniques, so make sure to have a look at that if you would like to know more. The source code of this project is available under the permissive Apache 2.0 license. If you have enjoyed this episode and you feel that eight of these videos a month is worth a dollar, please consider supporting us on Patreon. $1 is almost nothing, but it keeps the papers coming. Recently we have also added the possibility of one time payments through PayPal and cryptocurrencies. I was stunned to see how generous our crypto loving fellow scholars are since most of these crypto donations are anonymous and it is not possible to say thank you to everyone individually. I would like to say a huge thanks to everyone who supports the series and this applies to everyone regardless of contribution. Just watching the series and spreading the word is already a great deal of help for us. Thanks for watching and for your generous support and I'll see you next time.
Sam.
