Google's Translatotron translates speech directly to speech without converting to text first. Trained on a million voice samples, it can even synthesize the translation in the original speaker's voice. The system still has failure cases, but outperforms earlier approaches in listening tests.
June 20, 2019.
Approximately one million voice samples.
Dear fellow scholars, this is two minute papers with scientists at Google just released the Translatotron. This is an AI that is able to translate speech from one language into speech into another language. And here comes the first twist. Without using text as an intermediate representation, you give it the sound waves and you get the translated sound waves. And this neural network was trained approximately on a million voice samples. So let's see what Learning on this 1 million samples gives us. Listen. This is the input sentence in Spanish.
Y usted no sabra hamenos que usted pida.
And here it is translated to English, but using the voice of the same person.
And you will not know unless you ask. They have the hacereso. Stop doing that. Nuevos Empiliados de Cranfield University. New hires at Cranfield University.
This is incredible. However, there is another twist, perhaps an even bigger one. Believe it or not, this technique can not only translate, but also perform voice transfer. So it can say the same thing using someone else's voice. This means that the AI not only has to learn what to say, but how to say it. This is immensely difficult. It's also not easy to know what we need to listen to and when. So let me walk you through it. This is a sentence in Spanish.
Nadar con del fines.
This is the same sentence said by someone else, an actual person. And in English.
Swimming with dolphins.
And now the same thing, but synthesized by the algorithm using both of their voices.
Swimming with dolphins. Swimming with dolphins.
Let's listen to them side by side some more.
Swimming with dolphins. Swimming with dolphins.
This is so good. Lets have a look at some more examples.
Porlo tanto mirar al rededor del pais y lo quebes.
So look around the country and what do you see? So look around the country and what you see.
So look around the country and what he see.
Wow. The method performs the learning by trying to map these MEL spectrograms between multiple speakers. You can see example sentences here and their corresponding spectrograms, which are concise representations of someone's voice and intonation. And of course, it is difficult to mathematically formalize what makes a good translation and a good mimicking of someone's voice. So in these cases, we let people be the judge and have them listen to a few speech signals. Signals and asking them to guess which was the real person and which was the AI speaking. If you take a closer look at the paper, you will see that it smokes the competition. This is great progress on an immensely difficult task as we have to perform proper translation and voice transfer at the same time. It's quite a challenge. Of course, failure cases still exist. Listen, Dan.
C es la cosa. Dan. Yeah, that's the thing.
Dan.
Yeah, that's the thing. Dan. Yeah, that's the thing.
Dan.
Yeah, that's the thing. Dan. Yeah, that's the thing.
Just imagine that you are in a foreign country and all you need to do is use your phone to tell stories to people not only in their own language, but also using your own voice, even if you don't speak a word of their language. Beautiful. Even this video could perhaps be available in a variety of languages using my own voice within the next few years. Although I wonder how these algorithms would pronounce my name. So far, that proved to be quite a challenge for humans and AIs alike. And for now, all hail the mighty Translatron. In the meantime, I just got back from this year's NATO conference. It was an incredible honor to get an invitation to speak at such an event. And of course, I was happy to attend. As a service to the public, the goal of the talk was to inform key political and military decision makers about recent developments in AI so they can make better decisions for us. And I was so nervous during the talk. My goodness. If you wish to watch it, I put a link to it in the video description, and I may be able to upload a higher quality version of this video here in the future. Attending the conference introduced delays in our schedule. My apologies for that. And normally we would have to worry whether because of this, we'll have enough income to improve our recording equipment. However, with your support on Patreon, this is not at all the case. So I want to send you a big thank you for all your amazing support. This was really all possible thanks to you. If you wish to support us, just go to patreon.com twominutepapers or just click the link in the video description. Have fun with the video. Thanks for watching and for your generous support and I'll see you next time.
