Dr. Zsolnai demonstrates a new AI face restoration method that sharpens old photos with more detail than previous techniques. He applies it to historic images including Einstein, shows how a slider controls how much the face changes, and notes it runs in real-time at 14 frames per second.
Questions this video answers
March 9, 2023.
DFDNet (2020), GFPGAN (2021), GPEN (2021).
Approximately 14 times every second (about 14 frames per second).
"100 instances for $1.10 per hour versus $4.10 per hour with AWS" (stated as 73% savings).
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to take these old low quality photos and ask an AI what it thinks they should look like. And yes, we are going to see a restored version of this photo of Einstein and this one too. And the results are absolutely amazing. In fact, I could hardly believe some of the results, so I had to check them over and and over again. So what is the problem here? Well, we have one input photo which is a blurry mess, and when we try to restore it with a previous method, the output is also a blurry Mess. This was DFDNet, a research paper from 2020. Surely something better must be out there by now. And yes, there is. This is GFPGAN, a technique from 2021. This is significantly better, but the hair is full of artifacts and the photo itself is still not that sharp. Now have a look at this. This is GPEN, also a paper from 2021. And this is much better than its predecessors. But if you think this image has detail, now have a look at the new technique. Wow, this is so much better. The hair has tons of detail, the glasses finally look right. And now that we are comparing the two side by side, we see an interesting effect. The previous method created an idealized image where a lot of wrinkles and other high frequency effects are now gone. And believe it or not, this aspect of the new AI is controllable. I'll tell you about it in a moment. Now, this new technique is so good we can even use it on old photos and get this, even movies due to copyright issues. I can only show you still images from here, but I made sure to put the full videos in the video description so you fellow scholars can have a look. I will note that I have found some temporal coherence issues, in other words, some jumpy behavior, but it was very mild. I think many people wouldn't even notice that an AI restoration algorithm was at work here. And I know what you're waiting for. Now it's Einstein time. I will say in advance that we are going to talk about what this new AI thinks Einstein looked like. We cannot be sure. Now hold on to your papers and let's see. Whoa. This is incredible. Image restoration. And the technique delivers amazingly well over and over again. Now, once again, we cannot say that this is exactly what Einstein looked like, but I feel that with every paper we get closer and closer. And here I will try my best to show you the strengths and weaknesses of this algorithm so you can decide for yourself. Now, I promised that we are going to talk about identity preservation. So let's have a look together at this. We can use this parameter before restoration. This preserves the identity of the subject at the cost of some additional quality. Or if we are looking for a super high quality output and are okay with the results not looking like exactly the same person, we can also do this. Yes, this gives us a super high quality image, but the magnitude of the changes can be such that we get a different person. Now, my first thought was that, well, what is the point of this? Why would anyone use this? But then it struck me. Are you thinking what I am thinking? Look. Got it. We can use this on synthetic characters created by an AI. These are sometimes highly imperfect images. And in this case, we can supposedly get a super high quality version of this human. Let's see. Wow. Now that's what I call a huge difference. Restoration happened. The facial features are a little different, but this is a synthetic character. In this case, we might decide that we don't mind this trade off so much. Now for the case where we pull the slider to the right to have slightly lower quality, but preserve the identity of the image better. This is obviously super useful. However, how do we know if these results are truly good? We cannot compare this with high fidelity photos of Einstein. The whole point of the technique is that such high quality photos. Well, one way is to take an image of someone we know, delete parts of it, have it, restore it, and compare to the original. Now, there will be some failure cases from previous techniques that look a tiny bit frightening. So if you are not ready for that, stop the video now. And otherwise, let's see. Whoa. To say that some of these previous methods were not so good at this would be an understatement. Now let's see the new technique. Wow. This is so far beyond the previous techniques, it is hard to believe that such progress can be made in just one paper. But if we take the time to look through the results, it delivers over and over again, and at this point, there is no doubt in my mind that we are going to see people who are not with us anymore in all their glory on our screens. That is incredible. So how long do we have to wait for such an image? Well, not long at all. This new technique can pull this off not in minutes and not even in seconds. It does this 14 times every second. Wow. But you know what? I have good news. You fellow scholars can try it right now online. Let the experiments begin. So what do you think? Whose image would you like to restore? Isaac, asimov anyone? Let me know in the comments below. If you're looking for inexpensive cloud GPUs for AI, Lamda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance. And hold on to your papers. Because with Lambda GPU Cloud you can get on demand a 100 instances for $1.10 per hour versus $4.10 per hour with AWS. That's 73% savings. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs. Com papers to sign up for one of their amazing GPU instances today. Thanks for watching and for your generous support, and I'll see you next time.
