Jay Hack demos Shameo.tv, a tool that creates deepfake videos by syncing custom audio to celebrity faces using the Wav2Lip algorithm. Upload a video and record audio, get a realistic result in 30 to 60 seconds with the target's mouth movements matched to your voice.
Chapters
Hey folks, I'm Jay and I'm excited today to give a quick demo of Shamio.tv, which was my hackathon project with my friend Arshid for the Mira AI Incorporated Summer 2021 hackathon. So the idea is pretty simple. You probably know Cameo, which is a website you can go to to pay celebrities to make a video for you or your friends where they'll say something like happy birthday Jay or whatever. And we thought it'd be really cool to do that instead using artificial intelligence, deepfakes, and video processing. So you can upload a video of a celebrity, you can input some audio that you want them to say, and we'll use AI in order to essentially blend those two and lip sync the celebrity to say what you're saying. So hence the fake videos, questionable ethics portion of this. And you know, the vision here, looking at it in a broader sense, is really to democratize access to your face. Excuse me. So let's just hop right in. I'll show you how it works. Basically, the idea is we have a bunch of videos here that have already been pre-uploaded. So you have Stanley from The Office, This is Carole Baskin from Tiger King. You know, you've got Vince Carter, Taco Fall, Seth Curry, David Hasselhoff, and I'll just go with Tony Hawk here. So I get to record an audio piece that I'm gonna input and have him actually say. So, you know, this is just gonna be my best take at it.
Hey, what's up? This is Tony Hawk, the pro skater, and I'm an incredible skateboarder, but I just wanna say the best I've ever seen in my whole life are Jay and Arshid. They are so good at it, and congratulations to them for becoming the best skateboarders of all time.
And now that we've recorded this audio clip, we can go ahead and superimpose it on Tony Hawk's face. So here's the one part that takes a little while. It takes probably about 30 seconds to a minute to actually run the algorithm that will take the audio and blend it into the meat puppet, if you will, or the person's face. Unfortunately, this is the type of thing where if you're building it during a hackathon, you don't have time to really optimize it. But here we go.
Hey, what's up? This is Tony Hawk, the pro skater. And I'm an incredible skateboarder. But I just want to say the best I've ever seen in my whole life are Jay and Arshid. They are so good at it. And congratulations to them for becoming the best skateboarders of all time.
So as you can see, it looks pretty realistic. His mouth is definitely moving to create the shapes that would correspond to the sounds that I'm making. It's still in my voice instead of his, but you can do your best job or your best effort to replicate what their voice sounds like. So, you know, while we were building this, we had a lot of fun with our other teammates who were also at the hackathon. You know, we would record a video of them and shoot them a video of themselves, but with my voice or Arshad's voice just, you know, saying, "My name is Brandon and my hackathon project is so terrible," or something along those lines. It's a lot of fun. And, you know, once you create these, you can copy a link and share it with your friends or put it on Twitter what have you. So as you can see, it's pretty cool. So just a little bit more about this project, Shamio.tv. This is created using an algorithm called Wav2Lip. I'm not the person who— and neither is Arshad— the person who actually came up with the underlying algorithm here. We instead essentially borrowed, you know, the research from somebody else who did a fantastic job. And essentially what this does, it takes an audio clip as well as video where you extract the information from the faces of people and including, you know, various points on their face, like where their eyebrows are, where their mouth is, et cetera. And you combine these two using deep learning to output a new video that actually has the new mouth shape sort of superimposed. And then in terms of the tech stack that we used to create this, it was React, Next.js, FastAPI for the actual API layer, Firebase for most of the data management, and all of the AI portion was written in PyTorch. Cool. Thanks for checking it out. Hope you enjoyed. It might still be up by the time you see this, shamio.tv. So try and make your own. And send it to your friends. Thanks. Bye-bye.
