A neural network uses spatial and temporal convolutions to restore and colorize old film footage. The method achieves 3-4 dB improvement over prior approaches and produces more stable, artifact-free results across frames using just six color reference images.
Dear fellow scholars, this is two minute papers with in this series we often discuss a class of techniques by the name Image inpainting. Image inpainting methods are capable of filling in missing details from a mostly intact image. You see the legendary patchmatch algorithm at work here, which is more than 10 years old and it is a good old computer graphics method with no machine learning insight. And after so much time, 10 years is an eternity in research years. It still punches way above its weight. However, with the ascendancy of neural network based learning methods, I am often wondering whether it would be possible to take a more difficult problem, for instance, in painting not just images but movies as well. For instance, let's take an old, old black and white movie that suffers from missing data, flickering blurriness, and interestingly, even the contrast of the footage has changed as it faded over time. Well, hold on to your papers because this learning based approach fixes all of these and even more. Step number one is restoration, which takes care of all of these artifacts and contrast issues. You can not only see how much better the restored version is, but it is also reported what the technique did exactly. However, it does more. What more could we possibly ask for? Well, colorization. What it does is that it looks at only six colorized reference images that we have to provide and uses this as art direction and propagate it to the remainder of the frames. And it does an absolutely amazing work at that. It even tells us which reference image it is looking at when colorizing some of these frames. So if something does not come out favorably, we know which image to recolor. The architecture of the neural network that is used for all this also has to follow the requirements appropriately. For instance, beyond the standard spatial convolution layers, it also makes ample use of temporal convolution layers, which helps smearing out the colorization information from one reference image to multiple frames. However, in research a technique is rarely the very first at doing something, and sure enough, this is not the first technique that does this kind of restoration and colorization. So how does it compare to previously published methods? Well, quite favorably with previous methods. In some cases the colorization just appears and disappears over time, while it is much more stable here. Also fewer artifacts make it to the final footage, and since cleaning these up is one of the main objectives of these methods, that's also great news. If we look at some quantitative results, or in other words numbers that describe the difference, you can see here that we get a 3-4 DB cleaner image, which is outstanding Note that the decibel scale is not linear but a logarithmic scale. Therefore, if you read 28 instead of 24 it does not mean that it's just approximately 15% better. It is a much, much more pronounced difference than that. I think these results are approaching a state where they are becoming close to good enough so that we can revive some of these old masterpiece movies and give them a much deserved facelift. What a time to be alive. This episode has been supported by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. It can save you a ton of time and money in these projects and is being used by OpenAI and Toyota Research, Stanford and Berkeley. They also wrote a guide on the fundamentals of neural networks where they explain in simple terms how to train a neural network properly, what are the most common errors you can make and how to fix them. It is really great. You got to have a look, so make sure to Visit them through wendb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
