Dr. Zsolnai explains a neural network technique called time travel rephotography that restores, colorizes, and sharpens old photographs. The method transforms historic portraits of Abraham Lincoln and Thomas Edison into photorealistic modern images, outperforming earlier AI approaches in user studies.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai.
Today we are going to travel in time with the ascendancy of neural network based learning techniques. This previous method enables us to take an old, old black and white movie that suffers from a lot of problems like missing data, flickering and more, give it to a neural network and have it restore it for us. And here you can not only see how much better this restored version is, but it took it one step further. It also performed colorization essentially. Here we could produce six colorized reference images and the neural network uses them as art direction and propagates all this information to the remainder of the frames. So this work did restoration and colorization at the same time. This was absolutely amazing. And now comes something even better. Today we have a new piece of work that performs not only restoration and colorization, but super resolution as well. What this means is that we can take an antique photo which suffers from a lot of issues. Look, these old films exaggerate wrinkles a great deal. They even darken the lips and do funny things with red colors. For instance, subsurface scattering is also missing. This is light penetrating our skin and bouncing inside before coming out again. And the lack of this effect is why the skin looks a little plasticky here. Luckily we can simulate all these phenomena on our computers. I am a light transport researcher by trade, and this is from our earlier paper with the Activision Blizzard game development company. This is the same phenomenon, a simulation without subsurface scattering. And this one is with simulating this effect. Beautiful. You can find a link to this paper in the video description. So, with all these problems with the antique photos, our question what did Lincoln really look like? Well, let's try an earlier framework for restoration, colorization and super resolution. And. And. Well, unfortunately most of our issues still remain. Lots of exaggerated wrinkles, plasticky look, lots of detail missing. Can we do better? Well, hold on to your papers and observe the output with the new technique. Wow. The restoration indeed took place properly, brought the wrinkles down a much more realistic level. Skin looks like skin because of subsurface scattering. And the super resolution part is responsible for a lot of new detail everywhere, but especially around the lips. Outstanding. It truly feels like this photo has been rephotographed with a modern camera. And with that, please meet time travel rephotography. And the curious thing is, is that all this sounds flat out impossible. Why is that? Since we don't have old and new image pairs of Lincoln and many other historic figures, the question naturally arises in the mind of the Curious fellow, how do we train a neural network to perform this? And the answer is that we need to use their siblings. Now this doesn't mean that Lincoln had a long lost sibling that we don't know about. What this means is that as the output image is fed through our neural network, we can generate a photorealistic image of someone and this someone kind of resembles the target subject and has all the details filled in. Then in the next step, we can start morphing the sibling until it starts resembling the test subject. With this previously existing Stagnant two technique, we morphing is now easy to do, but restoration is hard. So essentially with this we can skip the difficult restoration part and just do the easier morphing instead, trading a difficult problem for an easier one. Absolutely brilliant idea. And if you have been holding onto your paper so far, now squeeze that paper because it can do even more age progression. Look, if we only have a few target photos of Thomas Edison throughout his life, these will be our yardsticks. And the algorithm is able to generate his aging process between these yardstick images. And the best part is that these images have different lighting pose. And none of this is an issue for the technique, it just doesn't care. And it still works beautifully. Wow. So we saw earlier that there are other methods that attempt to do this too. At least the colorization part. Yes, we have colorization and other techniques in abundance. So how does this compare to those? It appears to outpace all of them really convincingly. The numbers from the user study and the algorithmically generated scores also favor the NO technique. This is a huge leap forward. Do you have some other applications in mind for this new technique? Let me know in the comments what you would do with this or how you would like to see it improved. Now of course, not even this technique is perfect. Blurry and noisy regions can still appear here and there. And note that Stylegan 2, the basis for this algorithm, came out just a little more than a year ago. And it is amazing that we are witnessing such incredible progress in so little time. My goodness. And just imagine what the next paper down the line will bring. What a time to be alive.
What you see here is a report for a previous paper that we covered.
In this series, which was made by Weights and Biases.
Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get.
A free demo today.
Our thanks to WeightsAndBiases for their long standing support and for helping us make.
Better videos for you. Thanks for watching and for your generous support and I'll see you next time.
