Dr. Zsolnai explains why colorizing photos is harder than it looks. Translucent grapes and complex lighting trip up most methods. A new approach detects objects first, colorizes them individually, then blends everything together for better results than prior techniques like Deoldify.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to talk about image colorization. This is a problem where we take an old, old black and white photo, run it through one of these amazing new learning based algorithms and out comes an image that is properly colored. The obvious application of this is of course image restoration. What you see here is some amazing results with this new method. So, so clearly this can be done very well and you will find out in a moment how challenging this problem is. But there are also less obvious applications. For instance, image compression too. Just imagine that we wouldn't have to transmit colored images over the Internet. Black and white photos would be fine and if there is an AI based algorithm in every computer, they would be able to restore the colors perfectly. This would save a lot of bandwidth and energy and it would be a glorious thing to do. So how well do the current learning based algorithms do at image colorization? Wow, that is a hard, hard question and let's find out together why. First let's have a look at the results of this new method that you saw so far against previously existing techniques. Here is the black and white input image and here's the ground truth output that we have concealed from the algorithms. Only we see the result. This will remain our secret. And the algorithms only have access to the black and white input. Let's start. Yup, these are all learning based methods. So they all appear to know what a strawberry is and they color it accordingly. So far, so good. However, this problem is much, much harder than just colorizing strawberries. We have grapes too.
That does not sound like much, but.
There are many problems with grapes. For instance, there are many kinds of grapes and they are also translucent and therefore their appearance depends way more on the lighting of the scene. That's a problem because the algorithm not only has to know what objects are present in this image, but what the lighting around them looks like and how their material properties should interact with this kind of lighting. Goodness, that is a tremendously difficult problem. Previous methods did a reasonably good job at colorizing the image, but the grapes remain a challenge. And now hold on to your papers and let's see the new method. Wow, just look at that. The translucency of the grapes is captured beautifully and the colors are very close to the ground truth. But that was just one example. What if we generate a lot of results and show them to a few people? What do the users say? Let's compare the Deoldify technique with this new method. Deoldify is an interesting specimen as it combines three previously published papers really well. But it is not a published paper. Fortunately its source code is available and it is easy to compare results against it. So let's see how it fares against this new technique. Apparently they trade blows, but more often than not the new method seems to come out ahead. Now note that we have a secret, and that secret is that we can see the reference images which are hidden from the algorithms. This helps us a great deal because we can mathematically compare the results of the learning algorithms to the reference. So what do the numbers say? Yup, now you see why I said that it is very hard to tell which algorithm is the best. And because they all trade blows and depending on how we measure the difference between two images, we get a different winner. All this is true until we have a look at the results with this new paper. If you have been holding on to your papers, now squeeze that paper. Because this new method smokes the competition on every dataset, regardless of what we are measuring. So what is this black magic? What is behind this wizardry? This method uses an off the shelf object detection module that takes the interesting elements out of the image which are then colorized one by one. Of course, this way we haven't colorized the entire image, so parts of it would remain black and white, which is clearly not what we are looking for. As a remedy, let's color the entire image independently from the previous process. Now we have an OK quality result where the colors have reached the entirety of the image. But now what do we do? We also colorize the objects independently. So some things are colorized twice and they are different. This doesn't make any sense whatsoever until we introduce a novel fusion module to the process that stitches these overlapping results into one coherent output. And the results are absolutely amazing. So how much do we have to wait to get all this? If we take a smallish image, the colorization can take place five times a second and just two more papers down the line. This will easily run in real time, I am sure. You know what? With the pace of progress in machine learning research today, maybe even just one more paper down the line. So what about the limitations? You remember that this contains an object detector and if it goes haywire, all bets are off and we might have to revert to the only OK full image colorization method. Also note that all of these comparisons showcased image colorization, while the Oldify is also capable of colorizing videos as well. But of course, one step at a time. What a time to be alive.
What you see here is a report of this exact paper we have talked about with was made by Weights and Biases.
I put a link to it in the description. Make sure to have a look. I think it helps you understand this paper better.
Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you have an open source, academic or personal project you you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or click the link in the video description to start tracking your experiments in five minutes. Our thanks to WeightsAndBiases for their long standing support and for helping us make.
Better videos for you. Thanks for watching and for your generous support and I'll see you next time.
