Voice Cloning with AI

Today, OpenAI released a blog post in which they're previewing their voice engine models, which are their voice cloning models. Voice cloning is nothing new. There are companies like eleven Labs who have really good models when it comes to cloning your own voices. And there are also open source projects such as the Cookie AI. However, when it comes to the quality of outputs, there's nothing close to what OpenAI has previewed today. So the blog post is called Navigating the Challenges and Opportunities of Synthetic Voices. Here is how it works. You need to provide your text input and then just a 15 second audio sample and that will generate natural sounding speech that closely resembles the original speaker. This works very similar to elevenlab and the Koki AI, because you also need a reference audio in that case as well. Now, OpenAI has not released the models and I think there are legitimate reasons not to do that. We are going to cover those at the end of the video, but the output quality is really great. This model seems to be an older model, so it was developed in late 2022. That means probably OpenAI has much better models. The same models are powering the ChatGPT voice. If you have used Chat GPT app on your phone, there is a voice to voice conversation that you can have and the same models are powering that voice chat. So OpenAI is not releasing the model yet. And the reason is that they want to take a cautious and informed approach to the broader release due to the potential for synthetic voice misuse. And I think it's, it's a real problem because if you look at the quality of output, it's, it's much better than anything out there. So let's look at a few examples. So here I'm going to play the reference audio. So this is the original 15 second audio that is fed into the model.

Force is a push or pull that can make an object move, stop or change direction. Imagine you're riding a bike down a hill. First the push you give off the ground is the force that gets you going.

Okay, so this is the audio of the person. Now the model is fed this text as well as the original audio. And here is the audio that it generated.

Some of the most amazing habitats on earth are found in the rainforest. A rainforest is a place with a lot of precipitation and it has many kinds of animals, trees and other plants. Tropical rainforests are usually not too far from the equator and are warm all year.

So I think like, it's extremely hard to say whether this is computer generated audio or not. For me it's really indistinguishable. So let's look at another example from the same person.

This story has been told and retold for thousands of years. What is the central message that it is teaching?

So when you listen to this, you don't really hear that robotic audio anymore. And even the punctuations when it's talking, I think at the correct places. So this is pretty amazing. They also showcased a few other use cases. So one of them is translating content. So let's say if I'm a content generator, I'm and I want to translate my content into another language, that can be done. They work with a company called Heygin, who has this video avatar generation model. So you basically provide your input audio and an image, and based on that input audio, it will create a video. Now you can use the same technology to provide a reference audio and then generate audios in another languages in the same way. So for example, here is a reference audio.

Friendship is a universal treasure. It brings joy, support and laughter into our lives. No matter where we are in the world, true friends stand by us through thick and thin, sharing our joys and easing our sorrows. Let's celebrate the bonds of friendship that connect us all across every language and culture.

Okay, so in the same voice, here is audio generated in Spanish.

La mistad esuntesoro universal a porta le gria apoyo y risas a nuestras vidas sin importar donde estemos en el.

Here's an example in. And if you want German. So you can actually get a sense of what type of quality it is, at least for me. It's really hard to distinguish whether this is a real audio or AI generated audio, which is pretty great, but also frightening. So they talk about a few other use cases, but the one that I personally found fascinating was this. So helping patient recover their voices. So here's an audio segment from a patient who lost their audio.

This is what my voice sounds like losing open and loose hex to speak smile clause bulls and them.

Okay, so here was the actual audio before the loss of audio.

When you have all of your ingredients together, you are going to put the chopped broccoli and chopped banana peppers inside the bowl. And now this is very adverse. You can use anything that you would like. If you want to use cucumbers, you can chop up cucumbers and put this in here.

So it seems like it's probably from a video recording. Probably not like for this purpose, but they took a video recording and now with that Audio, you can generate something like this. Hi everyone.

This is what my voice sounds like. Using OpenAI's new text to speech model called Voice Engine, I was able to use just 15 seconds of a video that I made for a class project to to be the reference audio source for the voice you hear right now. What do you think?

And here's an example. For example, they use the same for ordering food.

Can I please have a number one with large fries and with a strawberry shake?

And just imagine the quality of life improvement that something like this can bring is absolutely amazing. Now, although this technology is amazing, but this can have some real life consequences if used for nefarious purposes. So for example, here they describe that we hope this preview of Voice Engine both underscores its potential and also motivates the need to bolster societal reliance against the challenges brought by ever more convincing generative models. And a specific example that they give is that we specifically encourage steps like phasing out of voice based authentication as a security measure for accessing bank accounts and other sensitive information. If you're using your voice for any sort of authentication, it's definitely a good idea right now to stop doing that. Another aspect is educating the public and understanding the capabilities and limitation of AI technologies, including the possibility of deceptive AI content. I think we are already seeing a lot of deep fakes and they are going to get more and more believable over time. So it's really great to actually see what is possible with these AI technologies. So probably it's already a good idea not to trust anything that you see or hear online. Okay, so we don't know when OpenAI is going to ever release this as a product, but in terms of the capabilities, I think it's already pretty great and really hard to distinguish whether it's a real voice or an AI voice. Let me know what you think. Should OpenAI release this model to the public or not? I would be actually curious in the opinion of the community. Personally I'm kind of unsure because it has definitely great applications, but at the same time I think this has the most potential harmful implications as well, and which are immediate. Maybe it's going to be a good idea to add some sort of watermarks that can distinguish between AI generated and real voices. I hope you found this video useful. Thanks for watching and as always, see you in the next one.