Dr. Zsolnai demos Photoshop's new generative fill tool, which removes objects and fills gaps based on text prompts. He tests replacing a bottle with a flower pot, extending images seamlessly, and handling reflections. The feature lags behind research but works well enough that millions will use it.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to have a look at how Photoshop just got transformed by the generative AI revolution and more. Previously if we had an image with an object that we wanted to cut out, but in the meantime also fill in the gap with sensible information, we could use the Content Aware fill feature and then this happens. It has done a passable job and a little fiddling with the parameters probably yields an OK solution. So can we do even better? Well, through the power of the papers, yes. Now we can Experiment number one. Let's try the new generative fill feature where we can add a text prompt and the void shall be filled with exactly that. So that is a wooden table. So little AI, please fill in a wooden table. Wait a minute, that is not exactly what I was thinking about, but perhaps it did exactly what it was asked. That is a wooden table. Can't argue with that. Alright, experiment two. This one was a sharp object in focus, but it can deal with blurred objects in the background. So let's change this bottle to a flower pot. Does it know how this needs to be done? Will it be out of focus? Hmm, that is an excellent solution. Great job. I really like that. We can also choose from several different variants. Now experiment number three. I am a light transport researcher by trade, so I would love to see some caustics that is light refracted through the glass here or on the table and ouch. Ok, this one is a little rough around the edges at the moment. Although if we are looking for reflections it can do that reasonably well. Now experiment number four. When trying this on not just some small thing, but on higher quality images created with some of the best generative AIs out there, the quality differences are apparent. This will not always be able to keep up. Experiment five. This is going to be very interesting and relevant. It can do not only generative inpainting, but outpainting as well. Oh yes, if we say that we wish to extend an image outwards, it can do that really well. In this case I don't see any seams or in other words, I don't see traces of the algorithm and anywhere. If I was shown just the end result, I would not be able to tell that some wizardry was going on. Others also seem to get really good results with this already. However, don't take these results and flaws for granted. These models are improving at an incredibly rapid pace. For instance, a new paper describes a technique that is capable of something even more incredible and that is magically changing the posture by proportions and the pose of already existing images. This is incredible. And do you know what is even more incredible? You are not going to believe this. This paper has only been out for a few days and an unofficial implementation of it already exists. Which means that with a little technical expertise you can already try it yourself. The link is available in the video description. And get this. There are even two new techniques for generating images without even writing a text prompt. How is that even possible? Well, have a look at this here. We just drop in an image and it generates beautiful variants for it. But we are not done yet. Not even close. If we are looking for a bit more control, but we still don't want to enter a text prompt, which basically sounds almost impossible. Well, we have another new tool. Dear fellow scholars, have a look at this prompt. Free diffusion paper. Yes, this generates images and we don't even need to enter text. All we need to do is add an image and a little scribble and a new image shall be born that follows both the scribble and the style of the input image. And get this, everything that you saw here just happened not within the last few years, but within the last few days. Wow, that is the amazing time we live in today. What a time to be alive. The generative Photoshop extension is a bit behind the state of the art, that much is clear. But it is super easy to use and will now be handed out to millions and millions of people. You see, research scientists are writing these amazing papers that go beyond what is possible today. And these are the next wave of techniques that you might see in industry standard tools like Photoshop. So as you see, the papers are real. As real as it gets. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud computer. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with Lambda GPU Cloud you can now get on demand H100 instances for just $2.40per hour. They are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today. Thanks for watching and for your generous support and I'll see you next time.
