This AI Makes "Audio Deepfakes"!

Your fellow scholars, this is too many papers with this man's name that is impossible to pronounce.

My name is Dr. Zsolnai. And indeed, it seems that pronouncing my name requires some advanced technology. So what was this? I promise to tell you in a moment, but. But to understand what happened here, first let's have a look at this deepfake technique we showcased a few videos ago. As you see, we are at a point where our mouth, head and eye movements are also realistically translated to a chosen target subject. And perhaps the most remarkable part of this work was that we don't even need a video of this target person, just one photograph. However, these deepfake techniques mainly help us in transferring video content. So what about voice synthesis? Is it also as advanced as this technique we are looking at? Well, let's have a look at an example and you can decide for yourself. This is a recent work that goes by the name Tecotron 2 and it performs AI based voice cloning. All this technique requires is a 5 second sound sample of us and is able to synthesize new sentences in our voice as if we uttered these words ourselves. Let's listen to a couple examples.

The Norsemen considered the rainbow as a bridge over which the gods passed from earth to their home in the sky. Take a look at these pages for Crooked Creek Drive. There are several listings for Gas Station. Here's the forecast for the next four days.

Wow, these are truly incredible. The timbre of the voice is very similar and it is able to synthesize sounds and consonants that have to be inferred because they were not heard in the original voice sample. And now let's jump to the next level and use a new technique that takes a sound sample and animates the video footage as if the target subject said it themselves. This technique is called neural voice puppetry and even though the voice is here are synthesized by this previous Techotron 2 method that you heard a moment ago. We shouldn't judge this technique by its audio quality, but how well the video follows these given sounds. Let's go.

The President of the United States is the Head of State and head of government of the United States indirectly elected to a four year term by the people through the Electoral College. The office holder leads the executive branch of the federal government and is the Commander in Chief of the United States Armed Forces. There are currently four living former presidents.

If you decide to stay until the end of this video, there will be another fun video sample waiting for you there. Now note that this is not the first technique to achieve results like this. So I can't wait to look under the hood and see what's new here. After processing the incoming audio, the gestures are applied to an intermediate 3D model which is specific to each person since each speaker has their own way of expressing themselves. You can see this intermediate 3D model here, but we are not done yet. We feed it through a neural renderer and what this does is apply this motion to the particular face model shown in the video. You can imagine the intermediate 3D model as a crude mask that models the gestures well, but does not look like the face of anyone. Where the neural renderer adapts the mask to our target subject, this includes adapting it to the current resolution, lighting, face position and more. All of which is specific to what is seen in the video. What is even cooler is that this neural rendering part runs in real time. So what do we get from all this? Well, one superior quality, but at the same time it also generalizes to multiple targets. Have a look here.

You know, I think we're in a moment of history where probably the most important thing we need to do is to bring the country together. And one of the skills that I.

Bring to bear and the list of great news is not over yet. You can try it yourself. The link is available in the video description. Make sure to leave a comment with your results. To sum up, by combining multiple existing techniques, it is important that everyone knows about the fact that we can both perform joint video and audio synthesis for a target subject. This episode has been supported by Weights and Biases. Here they show you how to use their tool to perform phase swapping and improve your model that performs it. Also, Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for their long standing support and for helping us make better videos for you.

Thanks for watching and for your generous support and I will see you next time.