Károly Zsolnai-Fehér walks through deepfake tech that syncs lips and voices to video. The system redubs speeches in different languages, revoices meme GIFs, and animates characters. A user study showed people strongly preferred the newer method over older approaches.
Dear fellow scholars, hold on to your papers because we have an emergency situation. Everybody can make deepfakes now by recording.
A voice sample such as this one.
And the lips of the target subject will move as if they themselves were saying this. Not bad, huh? And now let's see what else this new method can do. First, let's watch this short clip of a speech. And make sure to pay attention to the fact that the louder voice is the English translator. And if you pay attention, you can hear the Chancellor's original voice in the background too.
Framework for our Future Cooperation Dear Emmanuel, only eight months ago you were awarded the Pulse.
So what is the problem here? Strictly speaking, there is no problem here. This is just the way the speech was recorded. However, what if we could recreate this video in a way that the Chancellor's lips would be synced not to her own voice, but to the voice of the English interpreter? This would give an impression as if the speech was given in English, and the video content would follow what we hear. Now, that sounds like something straight out of a science fiction movie, but perhaps even with today's advanced machine learning techniques. But let's see if it's possible. This is a state of the art technique from last year that attempts to.
Perform this development in a very different history. We signed this on the 56th anniversary of the Elysee Treaty of 1960.
Hmm. There are extraneous leap movements which are the remnants of the original video. So much so that she seems to be giving two speeches at the same time. Not too convincing. So is this not possible to pull off? Well, now, hold on to your papers and let's see how this new paper.
Does at the same problem in Germany.
But also for a starting point of a very different development and a very different history. We signed this on the 56th anniversary of the Elysee Treaty of 19.
Wow. Now that's significantly better. The remnants of the previous speech are still there, but the footage is much, much more convincing. What's even better is that the previous technique was published just one year ago by the same research group. Such a great leap in just one year. My goodness. So apparently this is possible. But I would like to see another example just to make sure.
Add what he'd like to on the first issue. It's not for me to comment on the visit of Madame Checkmark.
So far, this is an amazing leap, but believe it or not, this is just one of the easier applications of the new model. So let's see what else it can do. For instance, many of us are sitting at Home yearning for some learning materials. But the vast majority of these were recorded in only one language. What if we could redub famous lectures into many other languages?
Aufgrunde, Aufkommunsdes tiefen lernenserchnell punct zum BEI spielfer wenden selbst farender auto sie jetst om eine Filze.
Look at that.
Any lecture could be available in any language and look as if they were originally recorded in these foreign languages. As long as someone says the words. Which can also be kind of automated through speech synthesis these days. So it clearly works on real characters. But are you thinking what I am thinking? 3. What about lip syncing animated characters? Imagine if a line has to be changed in a Disney movie. Can we synthesize new video footage without calling in the animators for yet another all nighter? Let's give it a try.
I'll go around. Wait for my call.
Indeed we can. Loving it. Let's do one more for of course we have a lot of these meme gifs on the Internet. What about redubbing those with an arbitrary line of our choice? Yup, that is indeed also possible. Well done. And imagine that this is such a leap. Just one more work down the line from the 2019 paper. I can only imagine what results we will see when one more paper down the line. It not only does what it does better, but it can also be applied to a multitude of problems. What a time to be alive. When we look under the hood, we see that the two key components that enable this wizardry are here and here. So what does this mean exactly? It means that we jointly improve the quality of the lip syncing and the visual quality of the video. These two modules curate the results offered by the main generator neural network and reject solutions that don't have enough detail or don't match the speech that we hear and thereby they steer it towards much higher quality solutions. If we continue this training process for 29 hours for the lip sync discriminator, we get these incredible results. Now let's have a quick look at the user study and humans appear to almost never prefer the older method compared to this one. I tend to agree. If you consider these forgeries to be deepfakes, then there you go. Useful deepfakes that can potentially help people around the world stranded at home to study and improve themselves. Imagine what good this could do. Well done.
This episode has been supported by Lambda GPU Cloud. If you're looking for inexpensive cloud GPUs for AI, check out Lamda GPU Cloud. They've recently launched Quadro RTX 6000, RTX 8000 and V100 instances and hold onto your papers because Lambda GPU Cloud can cost less than half of AWS and Azure. Plus, they are the only cloud service with 48GB RTX 8000. Join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, where workstations or servers make sure to go to lambdalabs.com papers to sign up for one of their amazing GPU instances today. Our thanks to Lambda for their longstanding support and for helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time.
