A new AI method lets you move objects inside photos while keeping reflections and shadows realistic. It uses diffusion models with independent region control to prevent the edited areas from bleeding into each other, though rotations and extreme close-ups still cause problems.
If we have a photo that is almost perfect, but we wish to delete something, image inpainting techniques already exist to help us with that. Like this. However, what if we don't want to remove this dog, only place it somewhere else? Of course, that is still trouble. Look, we could inpaint the part that we deleted. But the new part of course does not look convincing at all. So is that impossible? Nope. This new AI paper promises that it can do exactly that. I would very much like to see that. Let's have a look. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. I am especially interested in this because previous techniques could perform it. But. Ouch. Not great. When we wish to move this puppy, they don't appear to understand the relationship between this good boy and its reflection. So, little doggy, time to move. And now the new technique. Goodness, this is fantastic. If I was given these images, I would be hard pressed to say whether they were changed. Even the reflection was move to its new place. Great job. Maybe if you look very closely at the splash, you might find out about the trick. This is a wonderful collaboration between several institutes that you see here. And I wonder what the key idea is here. The key is that we have diffusion based text to image models. These can generate images for us. That is great, but not all of them give us fine grained control. Some of the new papers however, can do that. We can highlight regions with these blobs and say, I would like to see a cat here, a rock and a cloud there. Loving it. So our problem is simple. Just move the blob, right? Well, let's see. Nope, not quite. Fellow scholars, do you see the problem here? The problem is that not only the blob changes. No, no. The whole image changes. That is too intrusive. So this amazing new paper found something really interesting. Let's use a previous technique and ask for a rabbit and a cat. There we go. We got the rabbit and we got the. Well, that is many things, but a cat it is not. Perhaps a hybrid of a rabbit and a cat. And oh yes, therein lies the problem. The information of the rabbit blob leaked into the cat blob and the cat has been rabidified. So what does the new technique do? And hold on to your papers, fellow scholars. Let's ask for one rabbit and one cat. And there we go, less leakage. Finally, the blobs are now independent. Loving it. And that is one of the key ideas that helps us move these objects in an already existing photo. Luckily there are Lots of examples shown in the paper. And I really like how it understands that some of the surroundings have to change when moving the object. For instance, the shadow has to go too, but everything else has to remain the same. It is a really tricky problem and this is a fantastic leap forward. Is it perfect? No, clearly not perfect. But when showing the results to a bunch of humans, the new technique has. Oh my. A significantly higher win rate against previous methods. And in some cases you can move not just the cat, but the rock too. Two objects. I can imagine a future paper where every single object is recognized in the images. Note, we can already do that quite reliably. But then also moving would be great. You know what else would be great? Rotations. Let's see if it is any good at it. Well, unfortunately it is not too good at that. So what about resizing? Hmm, also not the best. Small adjustments are kind of ok, but if we go bigger, these artifacts appear. And finally, the most hilarious, when two objects are moved to be close to each other. What happens? What do you think? Oh my, the dog just absorbed that piglet. What a pity. So not perfect. The paper also contains a ton more comparisons against previous techniques. It is super fun, so make sure to check it out in the description. It works on a variety of images. That is a good sign that something would work well in practice. And just imagine what we will be capable of two more papers down the line. I would like to make a prediction for that. As you move the puppy, you will see real time updates. Goodness, that would be fantastic. What a time to be alive. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. If you're looking for an Nvidia GPU cloud service for large scale AI projects, check out One Click Clusters by Lambda, featuring up to 512 Nvidia H100 GPUs with InfiniBand Networking. And hold on to your papers because with One Click Clusters you can access all these GPUs on demand with reservations starting at just two weeks. Try it out@LambdaLabs.com paper or click the link in the description.
