Microsoft’s New AI Clones Your Voice In 3 Seconds!

Dear fellow scholars, this is two minute papers with Dr. Today I will show.

You a research paper that I can hardly believe exists.

It is about an amazing voice cloning.

Paper from Microsoft Research.

What does that mean? Well, voice cloning means that an AI listens to us speaking. We write a piece of text and it says it in our voice to see what that looks like. This is a previous work from Nvidia that was able to do that. Let's listen to Jamil recording these voice snippets.

Further details are expected later.

Okay, so how much of this do we need to train this earlier AI? Well, not the entire live recordings of the test subject, but much less only 30 minutes of these voice samples. The technique asks us to say these sentences and analyzes the timbre, prosody and the rhythm of our voice, which is quite a task. And what can it do afterwards? Well, it creates an AI Jamil, that can now say a scholarly message for you that I wrote.

This is a voice line generated by an AI. Wonder if you fellow scholars are going to notice. The first law of papers says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line.

Now hold on to your papers because.

Microsoft has a cracking new paper with an AI they call Volley that is able to do the same. But not in 30 minutes, not even in three minutes. It can do it in three seconds? Yes, that's right. All it needs is a three second snippet of our voice and it promises that it can clone it from that and it gets crazier, Much crazier. I will show you how in a moment. First, in goes a three second voice sample of us saying something. This does not need to match the text prompt.

Here he told his visitors as he lighted a pipe.

And this is going to be used for the learning. Now that it has learned this voice, here is what a previous technique could do in terms of cloning.

Instead of shoes, the old man wore boots with turnover tops and his blue coat had wide cuffs of gold braid.

And here is the new method.

Instead of shoes, the old man wore boots with turnover tops. And his blue coat had wide cuffs of gold braid.

Wow, the phrasing and timing are so much better. But actually, how good is it? How do we even know if this is really better? Well, easy. We can ask this person to read the prompt in their voice, hide it from the AI and compare side by side. Listen, AI goes first.

Instead of shoes, the old man wore boots with turnover tops and his blue coat had Wide cuffs of gold braid.

Real person. Second, instead of shoes, the old man.

Wore boots with turnover tops and his blue coat had wide cuffs of gold braid.

This is absolutely incredible. This truly feels like we are seeing history in the making. Here's another example. A speaker prompt for learning.

Milked cow and tames the fire.

And the new technique.

And lay me down in thy coal bed and leave my shining lot.

I love this. It even has a little personality in there. And the true human voice.

And lay me down in thy cold bed and leave my shining lot.

I have to say this is even better. But the AI was also out of this world compared to what we were able to do before then.

Lay me down in my cold bed and leave my shining lot.

Here is another.

I was in daily contact. We've made a couple of albums and.

Then we proceed to my favorite part of the paper.

Three advanced features.

So good. First, variety. This new technique can generate several variants of speech for the same prompt. So we can listen to them and choose the one that we like best. Listen. How? It puts the emphasis on different words.

Because we do not need it. Because we do not need it.

Loving it too. Get this. It can also listen to the 3 second example of our voice and preserve its emotions too. Here is an angry one.

Her face was against his breast. We have to reduce the number of plastic bags.

And here is a sleepy one.

Dig in. We have to reduce the number of plastic bags.

I really don't know what to say. This is absolutely incredible. And 3. It can also maintain not only the emotions but the ambiance and acoustic environment our sample has been recorded in. My favorite is this sample which sounds like an old crackly phone conversation like this.

Right?

What'd you like about One Flew over the Cuckoo's Nest?

So can it clone this kind of sample too? Let's listen together.

I think it's like you now. And more convenient too.

My goodness. I did not think that this was possible at all. Let alone from a tiny, tiny 3 second example. And just imagine the possibilities. This could bring back people who are not with us anymore and have them read us books and bedtime stories. How amazing is that? Maybe we will have the incredible Isaac Asimov reading his own robot box to us soon. What a time to be alive. And remember. Nvidia's previous technique needed 30 minutes for this. And now just one more paper down the line and 600 times less information is enough to create voice samples of this quality. 600 times less. And just imagine what we will be able to do. Two more papers down the line. My goodness. Now of course, this is a great research paper. So what does that mean? Of course that means that it has.

A thorough evaluation section.

So let's pop the hood and have a look. Whoa. I'll try to explain this. We first compare against previous techniques in terms of the word error rate metric. The two variants of the new technique come out the absolute best. However, correctness by itself is not enough. We also wish to know if the new samples are not only correct, but but also similar to the speaker in the input sample. So is it?

Wow.

The new technique comes out the absolute best on the two things at the same time. However. Wait, these two are some ancient techniques from long, long ago, right? Well, nope. Your TTS is from the same year as this paper and and so is Audiolm. This is so much progress in research in so little time. Now that truly feels like history in the making. So welcome to Two Minute Papers, land of the fellow scholars where we look at tables and make happy noises. So what do you think? Who else would you be interested in to read to you? Morgan Freeman all the things? Or maybe Caroi all the things? Let me know in the comments below. And if you wish to see more papers like this, please consider subscribing and even better, hitting the Bell icon if.

You'Re looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU Cloud compute. No commitments or negotiation required. Just sign up and launch an instance. And hold on to your papers because with Lambda GPU Cloud you can get on demand a 100 instances for $1.10 per hour versus $4.10 per hour with AWS. That's 73% savings. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.

Thanks for watching and for your generous support and I'll see you next time.