DeepFakes: Bring Your Images to Life with Generative AI

Can you spot which video is real and which one is fake? To know the answer, watch till the end. In this video, I'm going to demonstrate a research project with the help of which you can take an image and a video and do motion transfer between them. The outcome is going to look something like this. The best part is it's open source and free to use, so you can run it on your own machine. The demos I'm about to show you are based on this paper. Clade Spline Motion model for image animation. I will put a link to the paper in the description before showing you the demos. Let me walk you through the process. So in this case you have a driver video and you want to animate your picture based on the content of this video. So in the first step we simply detect key points on the subject. So for example, in this case there are some key points that are detected on the face. Then for each consecutive frames, those key points are being tracked and that will give you the movement of the face. Next, for your input image, you first detect those key points. Then you create replicas of your input image. So those are going to be fake frames. And then you simply apply the transformation from the keyframes from the original video into these fake frame of the input image. So that will basically transform your input image based on each corresponding frame from the original video. And you combine them together and that's how you get the. And using this approach, you can actually create deep fakes. So it's really important for the audience to be aware of the risks and dangers involved with defects. The material I'm about to show you is simply for educational purposes. Here is the GitHub page of the project. All the models and codes are available at this page. So if you want to run them locally, you can do that. They have provided multiple interfaces. We're going to be trying the replicate demo. So if you simply click on this, this will take you to this replicate.com website. Let me walk you through what is actually happening on this page. Their model affects an image that you want to animate. So for example, in this case they have an example of image of Donald Trump, right? If not that, you can click on this drop file or click to select we will do this later on. Or you can actually even use your own image. So simply click on this link that will open your webcam and take your own picture, right? So that's the first part that you need. That's the input source image. The second part, a video based on which you want to animate your Image. They call it the driving video. You can also provide your own video if you want to. Next section is data sets. So they actually have four different models. This walks model is based on celebrity who is simply talking in front of the camera. But then they have a Tai Chi model where they have trained a model to basically learn Tai Chi movements. The sec one is the third one is TED where they have another model which will convert you an image such that you are giving a talk. Right. And then some animated gifs as well. But we're going to be focusing on the first model in this video. So here are some example animations. So the first one is the driving video. That's the one we saw. And the first row is different images and they are animated based on the style of the video that was provided. Please. A second set of images shows what to expect from the TED Talk model. So in that case you have one driving video and a bunch of static images and they are animated. You can see that these animated images are able to replicate driving video pretty accurately. Let's test this on our own images. So for that go to this demo tab. Now you have two options. Either you can provide an image by dragging here or clicking here. Or you can open your webcam and take your own picture. I'm going to do with the first option I'll select this image. If you have seen my previous video, this is the avatar that I use. In order to animate this image. All we have to do is come and click on submit. Once you click on submit, it will start the prediction process. Sometime it will give you a message saying that it's loading the model and could take up to three minutes. If that's the case, just wait for it. But basically what it's doing is it's creating those frames or just duplicating these frames and getting prediction on those. I think it did a pretty good job. And let's try one more. So I got some of these images from free website called Pixabay. You can actually also explore images from that. This is going to be take a pretty challenging one. So let's get the predictions on this one. One thing you need to be careful about is the training data or like the videos didn't contain any hand movements and stuff like that. So if you want to try this on your own video, make sure that it's just the face and not like hand movements and things like that. Specifically for this walks model, here are some example videos that we generated. Now if you look at some of them, they look pretty Realistic, especially for example, this person. One thing to note is in most of the cases, these AI generators are not able to accurately represent teeth. So whenever you want to see whether a video is AI generated like a deep fake or not, mostly look at the teeth. Right. So for Elon Musk, it does pretty good job, but for the other ones, if you focus on teeth, it doesn't do a good job. Another aspect to consider whenever you're generating these deep fakes is just make sure that the pose that you're going to use in in your image is very similar to the one that is in the driver video. Right. But I'm actually kind of impressed. Like, even with facial hairs, it's still able to follow the head movements from the driver video. If you have seen my DID video, you will notice that here the facial movements are a lot more natural compared to what we were able to achieve using that paid software. In this case, we are going to look at this example. So that's the driving video. We provided an image and here is the resultant animated video. Let's click on tweak. This will take you to the demo interface again. Now we can test it on our own images. So let me pick the Rachel again and come down. Just click Submit. So it's starting the model. It will take us some time. So let me pause the video here. I run it on multiple images. You can see that there are much more natural movements. It does seem to do a better job on female faces compared to the male face. I think it has to do with the female face structure that is in the driving video. I think overall, the facial muscle movement as well as the eye movements are following the driving video. So it seems to be working pretty great. In this last test, we're going to use our own video as I'm driving video. So click on upload. And here we select a very simple video. The person is performing simple movements, but it's going to be very challenging because this video contains facial hairs as well as eyeglasses. And our test subject is not going to have a facial hair as prominent, neither eyeglasses. So let's see how it behaves. So we will click Submit to process both our video as well as our input image. Here is the same test on some other videos. I feel like if you look at the outcomes, if you didn't know that this is the original video, right? And these are fake, I think for some of them, it's actually hard to figure out if they're even fake. So it does a pretty good job, especially given that the camera in the original video is moving so it makes it a lot more challenging. So with this approach you can potentially simply record your video and then use an avatar instead of your own face while you keep the audio since this model is able to capture the micro motions of the face so your avatar is going to have more natural movements and look apart from this model there are other models on the same web page which I cater towards full body movements so I would encourage everyone to explore those as well. As far as I'm aware there is no limit on how long the video can be. The only I think constraining factor is the time is going to take so the longer video are going to take much longer to process and you can also run these on your local machine if you want. If you watch the video till this point I hope you like the content so please consider liking the video and subscribing. Thanks for watching see you in the next one.