Drag Your GAN: This Will BLOW YOUR MIND!

This will blow your mind. There's a new model with the help of which you can easily manipulate images just by selecting a few points on the images. The results are amazing and they look so natural. It's a powerful new tool that lets you precisely control the pose, shape, expression and layout of objects in your images. And we are going to be looking at this in this video. So all you need is like different points on the images and define the direction in which you want the image to move. And the model will take the and the model will take care of the rest. It's. It's going to revolutionize how we edit and manipulate images. Now it's based on the work called Drag your GAN interactive point based manipulation on the generative image manifold. It's very different than the current stable diffusion based models. So most of the image generation models that you see today are based on technique called stable diffusion. Again, is a predecessor to stable diffusion. It's a completely different algorithm which was pretty famous two or three years ago, but have been kind of overshadowed by stable diffusion. The results that they are showing is absolutely amazing. Like from the demo, all you need to do is simply click or select a few points on the image and then define the direction of deformation or motion and the model takes care of the rest. So let's look at it how it actually works. Now if you look at this image here, they simply defined two points. One is going up and the second has the down direction. So the model actually is contact survey and it means that the user is trying to open the mouth of the line. Right. And in the second case there's another two points where it's simply moving the head in a specific direction. The great thing about this model is since it's context aware, so it not only changes, let's say the face, but the rest of the body as well. The model consists of two components. A feature based motion supervision that controls the motion and then a point tracking approach that uses GAN features. And according to them, with this approach anyone can deform an image with precise control or where pixels goes thus manipulating pose, shape, expression and layout of diverse categories such as animal cars, the humans landscape, etc. We will be looking at a few examples now before that, just regarding the code release. So it's not available yet, but it seems like they're going to be releasing the code sometime in June. Another question you guys might have is whether it can be integrated as a part of automatic 11.11, which is kind of the defective platform for running stable diffusion based model so it might be actually hard to do it because automatic 11 elements supports stable diffusion based models. But this is a completely different approach. But you never know about the open source community, so there will be ways to integrate it. Or even their own demo seems to have a pretty nice UI which can be used for manipulating images. All right, so let's look at a few examples on their website. So for example, this is a line and they're simply selecting a couple of points. So I think they want to move the face, right? And you see like how actually the whole body moves. And it just is. It's a very natural movement compared to what we have seen from other approaches so far. Here's another one. Again, the face is moving and then with the face movement, even the rest of the body also moves. Now from the demo it seems like you can select a specific area where you want the movements to happen. So let's say in this case the legs are not moving at all, only the face is moving. And along with the face, the hair and everything. Now this is a crazy one because if you see here, they're selecting just one of the legs of the horse and actually the rest of the body moves with it like it's running. And this is pretty awesome because this is being done on a single point. Again, like the head is going down and the rest of the body adjusts with it. So it seems like it's aware of the physics involved as well. Here's another example. In this case it's just a leg, I think, going back. And in one of the other example they were playing around with the trunk as well. So it's not like perfect yet. But this is just the first iteration, right? So you see the trunk of the elephant is moving. I actually am interested in this one. So basically you can take a picture and easily manipulate it, right? So in this case it's a human face, right? And you see it's actually a neutral face to a smiling face, right. And the great thing is that it looks very natural. So imagine the possibilities here how you can animate different images using this very simple approach. So with this you won't have to train, let's say something like a Lora or a Dream Booth model, but rather from a single image, you can create images in different postures, potentially different facial expressions, while preserving the features of the shape. So that's, that's pretty amazing to be able to do that. Now I think here's an example of a localized changes. So they selected just the eyes, right? And then I think this is going to just close the eye. Now you see there is a very small movement, right, Based on the number of steps. And then one only one of the eyes is closed, the other is open. This is pretty neat. Now with this you can even do landscape changes. So look at this. And it's selected. And now the reflection of the trees, this actually changes, right? So initially it was I think up to this point, and then it was dragged down. So if you don't like certain things in your images, you can easily manipulate them. It changed the horizon, which is crazy. I think this one, the mountains will go up. This is awesome. You know, here's another one one of microscopic creature, right? So they are able to control the movements. But I think it will depend on also the training data, what kind of training data was used and probably which objects were in the training data so that the manipulation looks more natural in some of these cases. Now, for all the car enthusiasts out there, you see, they can actually remove this, the ceiling of the car, right? It's pretty awesome. Now think about the possibilities that this can have in even like media, in how you manipulate images. Now one of the demos that which I really like is this, that you can actually rotate the objects. So for this, in this case, if you look at it, they are rotating the car and the structure of the car remains pretty consistent. So it's there are like small deformations, but not crazy. That will look really unnatural. And here's another one I think they completely rotated. Yeah. And this is small car, so they will simply rotate the whole car. So you can think about the potential applications that it can have. Okay, so it was a quick update. I just want to make this quick video covering the latest what is happening. The code is going to be hopefully available sometime in June and then we'll be able to play around with it. When you look at the speed of the current advancements, the only thing a person can say is what a time to be alive. Thanks for watching. See you in the next one.