Google's AI Clones Your Voice After Listening for 5 Seconds! 🤐

Dear fellow scholars, this is Two Minute Papers with Today we are going to listen to some amazing improvements in the area of AI based voice cloning. For instance, if someone wanted to clone my voice, there are hours and hours of my recordings on YouTube and elsewhere. They could do it with previously existing techniques. But the question today is, if we had even more advanced methods to do this, how big of a sound sample would we really need for this? Do we need a few hours? A few minutes? The answer is no, not at all. Hold on to your papers because this new technique only requires five seconds. Let's listen to a couple examples.

The Norsemen considered the rainbow as a bridge over which the gods passed from Earth to their home in the sky. Take a look at these pages for Crooked Creek Drive. There are several listings for Gas Station. Here's the forecast for the next four days.

These take the shape of a long round arch with its path high above and its two ends apparently beyond the horizon. Take a look at these pages for Crooked Creek Drive. There are several listings for Gas Station. Here's the forecast for the next four days.

Absolutely incredible. The timbre of the voice is very similar and it is able to synthesize sounds and consonants that have to be inferred because they were not heard in the original voice sample. This requires a certain kind of intelligence and quite a bit of that. So while we are at that, how does this new system work? Well, it requires three components. One, the speaker encoder is a neural network that was trained on thousands and thousands of speakers and is meant to squeeze all this learned data into a compressed representation. In other words, it tries to learn the essence of human speech from many, many speakers. To clarify, I will add that this system listens to thousands of people talking to learn the intricacies of human speech. But this training step needs to be done only once. And after that, it was allowed just five seconds of speech data from someone they haven't heard of previously. And later the synthesis takes place using this five seconds as an input. Two, we have a synthesizer that takes text as an input. This is what we would like our test subject to say, and it gives us a MEL spectrogram, which is a concise representation of someone's voice and intonation. The implementation of this module is based on DeepMind's Techotron 2 technique. And here you can see an example of this MEL spectrogram built for a male and two female speakers. On the left we have the spectrograms for the reference recordings, the voice samples, if you will. And on the Right. We specify a piece of text that we would like the learning algorithm to utter, and it produces these corresponding synthesized spectrograms. But eventually we would like to listen to something, and for that we need a waveform as an output. So the third element is thus a neural vocoder that does exactly that. And this component is implemented by DeepMind's WaveNet technique. This is the architecture that led to these amazing examples. So how do we measure exactly how amazing it is? When we have a solution, evaluating it is also anything but trivial. In principle, we are looking for a result that is both close to the recording that we have of the target person, but says something completely different. And all this in a natural manner. This naturalness and similarity can be measured, but we are not nearly done yet. But because the problem gets even more difficult, for instance, it matters how we fit the three puzzle pieces together and then what data we train it on, of course, also matters a great deal. Here you see that if we train on one data set and test the results against a different one, and then swap the two, and the results in naturalness and similarity will differ significantly. The paper contains a very detailed evaluation section that explains how to deal with these difficulties. The mean opinion score is measured in this section, which is a number that describes how well a sound sample would pass as genuine human speech. And we haven't even talked about the speaker verification part, so make sure to have a look at the paper. So indeed, we can clone each other's voice by using a sample of only 5 seconds. What a time to be alive. This episode has been supported by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. It can save you a ton of time and money in these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. They also wrote a guide on the fundamentals of neural networks where they explain in simple terms how to train a neural network properly, what are the most common errors you can make and how to fix them. It is really great. You got to have a look, so make sure to Visit them through wandb.com papers wandb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to WeightsAndBiases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.