Karoly Zsolnai-Feher explains Google's RealFill, a technique that fills missing parts of photos with actual historical content instead of AI-generated plausible content. Given just one to three other photos of the same scene, it reconstructs missing regions with near-photographic accuracy, though it still struggles with text.
One to three photos.
This idea sounds like science fiction, but by the end of the video you will see that this makes perfect sense. So what is it? Well, consider image inpainting. It is amazing. What can it do? Well, when we cut out a part of an image and then bam. It fills in the void with plausible information. This is image inpainting. And now it works on video too. But it gets even crazier. Image outpainting also works. Whoa, what is that? Well, we can essentially extend the image in any direction and once again fill it in with plausible data. Now please note the choice of words here in both cases. I said it fills it in with plausible data. Data that could be there, but synthetic nonetheless. Now here is an insane idea from Google's researchers. What if we would take these and fill them in not with information that could have been there, but with information that was actually there. Filling in with reality, that is of course impossible, right? Well, look. Oh yes, it seems like it is impossible. If we try to complete this image with previous techniques, for instance, stable diffusion, we get something that is plausible. You know, the hat continues the post, its also continue. That is good. But still it is likely not the real thing. So can I get the real thing? Well, let's think together. What if we are trying to outpaint a historical building? Wait a minute. That is the key. If we try to fill in information for something that we have other photos for, it might be possible. Let's give it a try. This is the incomplete input photo. And here are our other photos. Now note that this is still quite hard. We can't just copy it. The angles are different, the lighting is different, lens distortion is really different. But in the age of AI, let's see if it can be done. And. Oh wow, look at that. Perfection. And it can do it for a variety of scenes, over and over again. It appears to work pretty much everywhere. Well, it does not work everywhere. I'll tell you about it in a moment. But all this is absolutely amazing. But still. Wait, how do we know how the real photos are if there is nothing to compare to? Well, of course, by making sure that there is something to compare to. Let's take a real photo cut off the top. And now we know exactly what should be there. Stable diffusion does not know. Paint by example. A paper from almost exactly a year ago does not know at all. But the new technique called real fill. This one knows. Look. That is incredible. Almost pixel perfect reconstruction. My goodness, what a time to be alive. Now note that this is not a copying machine. It has Access to information about the room. But it has to understand which part is missing and what that part would look like from this angle. So it feels in reality after all. And it does it over and over again with breathtaking accuracy. Now, I noted that it is still not perfect. I mean, all of these look nearly perfect. So where are the issues? Ah, of course, text. It's always the text every time. We finally left behind the age of AI systems generating mangled and incorrect hands mostly. But text is still a challenge. I am fairly sure that this is something that will be possible. Just one more paper down the line. And can you imagine what will be possible just two more papers down the line? My goodness. We can already do a pretty good reconstruction from just one image. Not even a set of images, just one image. And this is supposed to be a failure case. If this is a failure case, bravo. Sign me up right now. So adding a little more information to the AI by reusing already existing images, that was the crazy idea. Which in hindsight makes perfect sense. What a brilliant paper. Loving it. And one more thing. I have a little daughter and when she was a baby we we could not really afford a good smartphone to take better images of her. However, there are a lot of pictures and I was thinking that over my lifetime there surely will be an AI that will be able to upscale those not great images to a higher resolution. And it should not just fill in things that could be, but with things that are really there. And finally we are here. I can't believe it. And all it needs is one to three photos. And as a family we have thousands of photos of ourselves to learn from. So good. We are sponsored by weights and biases, experiment tracking, model evaluation and production monitoring for your deep learning products and LLM apps. It is the best. Everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
