VoiceBox: Meta's NEW AI Clones Voices with only 2 Seconds of Audio!

In this video we are going to be looking at an amazing project from Meta AI called Voicebox. It's a text to speech model, but it can do a lot more than that. In text to speech mode, it will be able to generate natural sounding audio based on your input text. You can have a reference audio and then generate speech based on the text in that audio. So this is basically doing voice cloning. Now another great application is style transfer. So you can take somebody's audio and then reproduce another audio in another language based on that specific audio file. And you just need a two second audio clip. But the one application that I really like was audio based editing. So if you have a background noise in your audio, you can simply reconstruct the missing audio based on the text using this model without ever re recording your audio. Now in this video I want to show you a few demos which will show you how this model is different from the rest of text to speech models. If you are a content creator, a lot of time you have to rerecord your audio because you are interrupted by a doorbell, dog barking or somebody talking in the background. Now with this model you do not need to re record the speech anymore. As they are saying, it really works like a magic eraser to remove transient noise. So let me show you an example. Now let's assume here's the text that you are narrating. Now listen to the noisy speech pretty closely. This is the input speech. So for this part of the text, there's a dog barking in the background. Now here is how it's going to.

Sound like in zero weather in mid winter when the earth is frozen to a great depth below the surface and then driving over the unpaved country roads.

Okay, as you can listen that there was a dog barking and as a result you cannot really comprehend what is being said in this segment. This simply works as a magic eraser as I said. So the model input is basically all the speech except this part. So this part is going to be completely silent. So let's have a listen.

In zero weather in mid winter, when the earth is frozen, Unpaved country roads, they give forth a hard metallic.

So usually what you would do is you would re record the soul segment so that you do not have the background or Ponzi noise. However, using Voicebox, you do not need to do that anymore. Just by taking two seconds of your audio, it's able to reconstruct this part. So here is the output of the model.

In zero weather in mid winter, when the earth is frozen to A great depth below the surface. When in driving over the unpaved country roads, they give force a hard metallic ring.

As you can listen, it was able to reconstruct the missing audio flawlessly, which is pretty amazing. For each of the use cases I will be just showing you a single demo. However, they have provided a number of examples for each use case. So I would actually encourage everybody to go and have a listen to those. The second application which I'm personally really interested is is content editing. So a lot of time I misspeak different words and I have to re record it or edit it out. However, with this model now you do not need to do that at all. Now listen to the original speech. Here is the text of the original speech. For example, this person wants to rerecord this specific part occasion of common and constant recurrence. But instead of doing that, you can simply edit the text to rare and unpredictable circumstances and then use voice box to reproduce the speech. So let's first listen to the original speech.

Will find himself completely at a loss on occasions of common and constant recurrence. Speculative ability is one thing and a practical ability is another.

Now in the second case you simply need to edit the text and let's listen to the edited speech Based on the edited text.

Will find himself completely at a loss on rare and unpredictable circumstances. Speculative ability is one thing and a practical ability is another.

This was flawless and it's. It's amazing. The results are pretty great. This will make the life of editors so much easier. However, with a model like this, you can literally make anyone say anything you want. And you can embed this in their original speeches which is going to be even harder to detect. Now the next example is zero shot text to speech synthesis. Now this is very similar to what you would call voice cloning because you provide a reference audio and then the model is able to regenerate speech in the same voice from the text that you provide. So let's have a listen. Here is the input audio. It's just a two second reference audio. Gabes There is the first symptom. Now this model needs only two seconds to reproduce that speech. So here's the target text and here's the result of the model. Reproducing the speech in the same voice is the Swiss army knife of text to speech. Acing multiple languages, changing voice styles and dishing out custom. I would say the results are pretty awesome. I think it is able to reproduce the voice with a very good accuracy. However, you might be thinking that this is not Something new. There are probably much better models, for example bark, which can generate audios with very natural sounding audios. But where it shines is that you can use reference audio in one language and then reproduce audio in another language with the same voice. For example, one can generate English with a French prompt. So that's going to be a reference audio in French which would enable everybody to speak in any language in their own voice one day. Now they also say that voice box can also preserve the original temporal alignment between text and speech which can be used to converting dubbed speech to the original speakers voice. So this is pretty amazing because let's say if you record an audio, want to dub it in a different language, but you want to use the same person's voice, you can actually do that now. So the future iterations of this model can really have an impact on voice dubbing industry. Let's listen to this to the prompt or the reference audio is in French. So here's how it sounds like. Now if you want to generate an English audio based on this input text using the same voice, here is the output from voicebox.

We see by this example that admirable as is the progress accomplished in certain regions of physics, there still exist many over neglected regions which remain in painful darkness.

I personally feel like it does a pretty amazing job because for reference here's another text to speech system and this is the potential output from that remain in painful darkness. I actually prefer Voicebox output. Now these are some demos from Voicebox. Just keep in mind that you can reconstruct any person's voice just by using a two second audio. Now the question on everybody's mind is going to be can you use this and is it open source? Unfortunately the answer to that question is no at the moment. Because in this ethics statement they say we recognize that this technology brings the potential for misuse and unintended harm. In the paper they actually have presented a highly effective classifier that can distinguish between authentic speech. So that's natural speech versus audio generated with voice box. However, despite having highly effective classifier, they are not making voice box model or code publicly available at this time. Now just to summarize, it's the first iteration of their voice box and I think it does a pretty good job especially with these other features, for example content editing and then the ability to reconstruct somebody's voice in another language. These are exciting times for generative AI beyond LLMs. Hope you liked the video, thanks for watching and see you in the next one.