Why Cartesia-AI's Voice Tech is a Game-Changer You Can't Ignore!

Today we're excited to release the first step in our mission to build real time multimodal intelligence for every device. Sonic. A blazing fast 135ms. Model, latency lifelike generative voice model and API.

Okay, this is probably one of the best text to speech system that I have seen that has amazing speed and is extremely versatile. So this is from a new company called Cartesia AI and they have been able to develop, I think one of the best text to speech system. So I'm going to show you how to use this in your own projects through their API. And it also has the ability to clone your voices. We are going to look at that in another video, but let me show you a few example audio segments and the speed of generation.

Ladies and gentlemen, gather around the wireless as we delve into a world of speakeasies, jazz and roaring excitement.

Okay, so it took about 0.15 seconds. So it's 1/10 of a second to actually generate that audio. Now let's listen to a barber shopman. This is pretty smooth and relaxing.

Hey there. Have a seat and relax. You're in for the best haircut and some great conversation.

Okay, let's look at or listen to a confident British man prepare to embark on a journey where intellect reigns supreme and victory is but a well executed maneuver away. Okay, so you can see there is there are a variety of different voices that you can choose from and the quality is pretty awesome. Like in some of the cases, you cannot say that these are AI generated voices.

I was only 5 years old when my father took me to my first rodeo. I've never forgotten that day.

With a firm stride and unwavering gaze, she faced the challenges ahead, knowing that within her lay the strength to conquer any obstacle that dared cross her path.

Alright, so here's the same example that we listened to in the beginning, but this time let me change the voice. So we're going to select Movieman and then we click on Speak.

Today we're excited to release the first step in our mission to build real time multimodal intelligence for every device. Sonic. A blazing fast 135ms. Model, latency lifelike generative voice model and API.

Okay, as you can see that actually the AI decides when to pause, right? When to slow down, when to speed up. And it's pretty amazing. It's just looking at this structure of the sentence and it's able to do that. So let me show you how to get started and we're going to use this in our own projects. So for that we need to get access to the API key. You will need to sign up for an account. So let me walk you through that process. So first you need to create an account. You can use either GitHub or your Google account. I'm going to use my Google account for this. Once you sign up, you can start using the API key in your own projects. I'll show you how to get that API key and you can also use their playground. Now you need to just click on go to Playground and here you're going to get access to all the different voices that are available. Alright, so once you create your account, the next step is to get the API key. Now before looking at getting the API key, I'll I would like to show you a couple of other things regarding these voices. So if you click on any one of them, you're going to see the voice name. That's the one that we chose. Then there is the voice id. This is the one that we will actually need to use with our API. And there is also a vector of embeddings that is going to be needed to be used when we are making an API call. Another thing that they have is you can actually control the speed, anger, curiosity, happiness, sadness and surprise levels. So this is pretty awesome because this platform gives you the ability to really customize the voice the way you want. Alright. Anyways, once you select a voice, just go to console, click on it, then click on new API key. This will generate a new API key for you and then you can just copy that API key. And we will need to use this in our own projects. They have really detailed documentations regarding how to use the API. So we're going to be using the Python API, but there is also support for JavaScript and TypeScript if you're interested in using that. And they have different levels of subscription. So currently I'm using the free plan. If you are trying to clone your voice, you will need to update it to one of the paid plans that I am planning on doing. So their paid plan, the first one is Pro, which is $5 per month and it gives you 100,000 characters, which is actually pretty generous I would say. And then there are some other options which really I think scales up the number of tokens that you can use depending on your need. Okay, so in a subsequent video I'll show you how to clone your voice and then use that in your own projects. But in this project our focus is going to be just to get started with the text to speech system. So I'll show you one example which is based on this code. And then I'll show you how to build a voice to voice system where you can talk to the LLM you using voice. And the response is going to be a voice output which is going to be generated by Cartesia. Okay, so first let me show you the standalone example. In this case we'll need to install these three packages. First is the NumPy, then sound device and then Cartesia. Alright, so I created a virtual environment which I'm called Verba. So we're going to just copy that command and we'll install this. This will install all the required packages for us. Alright, so let me walk you through the code. This is the example in which we're going to provide a bunch of text and the model is supposed to generate audio. So we are importing all the different packages that we will need. Then we will set up our API key. I'm setting this up as an environment variable and then loading it. So you will need access to the API key. Then we just check whether the API key is set correctly or not. Then we need to set the configurations. So the model ID is going to be upbeat Moon. That's the model that is available currently. What is going to be the output format. Then we create the client where we're going to be using the Cartesia text to speech system and provide our API key so that it can authenticate our account. And then you need to get a list of voices. So if you recall, there are a number of different voices that are available. So this will get list of all the voices that are available in the system. Now here we define a function. That function accepts a voice name, transcript and output path. So the voice name that we're going to be using is the Barbershop man voice. This is one of the voices that I actually showed you. So next it will take the device name as an input, gets the corresponding id. If you already know the id, then you can just provide the ID here. And the way you get the ID is just click on the name of the voices and then just copy this. So if you know the id, you don't really have to make that extra call to the API in order to get the voice id. And I'll show you an example of this in the second section when we're going to be creating that voice to voice chat assistant. So based on that, this will download the embeddings for that specific voice and you can now start using this model. After that we will use the client and we'll call the generate function on it, provide the transcript. So that's basically the text which are going, which is going to go as an input, then the voice that we selected. In this case, we are setting streaming to false, but if you want a streaming audio, you can actually set that and we provide the rest of the configurations that we have. Now I am actually measuring the time it takes to make the API call and get the output. And the reason is that when you make the API call, it actually takes a long time compared to the other playground. So I don't know what is going on here. The quality of output is pretty amazing, but it's definitely a lot slower when you're trying to do this through the API. And we're going to look at an example of why that is happening. And I'll also show you a way of you can potentially increase the speed of what you get here. Right. Then we get the output, and from the output we get the audio plus the corresponding sampling rate. We pass this on to the sound device to play it, and we also store it if you want, for your own purposes. Okay, so now we're going to run it. And here is a piece of text that I asked GPT4.0 to generate for us. Okay, now to get started, we're going to use the Python command and then we going to just call that script to run it. Now this does take quite a while because the EPA is pretty slow.

The Meaning of Life is a profound philosophical inquiry, often viewed as the pursuit of understanding our existence and place in the cosmos. It encompasses the quest for purpose, the search for truth, and the exploration of our intrinsic values. For some, it is about seeking wisdom, fostering connections with others, and striving for personal and collective growth. Ultimately, it is the journey of finding significance in our experiences and contributions within the vast tapestry of the universe.

Okay.

As you can see, it took about 18 seconds to get a response from the API, which is definitely a lot longer than what we saw in their playground. But the quality of output is pretty amazing. It's hard to say if it's actually AI generated, which is pretty awesome. Okay, in this next example, I'm going to show you voice to voice chat Assistant. This is a project that I'm currently working on. A link is going to be in the video's description. I'm not going to go into a lot of details. I plan to create a more detailed tutorial on how this works. And it also has a local component, so you can actually run everything locally with local models. You don't have to make any external API call. So it could be your Jarvis Assistant which is running completely locally. But I just want to show you an example of the integration of Cartesia in here. So for the transcription model, which is the speech to text, I'm going to be using Deepgram. You can also use Grok as well for LLM. In this case we're going to be using Grok. And for the text to speech model we're going to be using Cartesia. Now here are the different options that you can use so you can just replace each one of them. Again, link is going to be in the video description. You will have access to this code, but I'm going to be creating more detailed videos on this project. I'm actually pretty excited about this one and there are some amazing people who are contributing to the project as well. Alright, so let's run this. This is going to be Python run Voice Assistant Py hey, can you tell me about your.

I am an AI designed to assist and communicate with humans. I can answer questions, provide information and even have a conversation with you. I'm constantly learning and improving to better serve you.

All right, okay. As you can see, you can actually use it as a voice assistant. Now you probably noticed that the time it took to generate response were a lot quicker compared to when we were doing it for the first time. And the reason is that we are just storing everything in a JSON file. So for example, the ID that we have, the name that we are going to be using as well as the embedding model in here. So this will basically save you a few API calls and we probably expedite the process. Anyways, the code for this voice to Voice assistant is already on GitHub. I'll push the latest updates now. Since Cartesia has the ability to clone voices, I'm actually pretty excited about that. I'll be creating another video on it when I get to experiment more with the platform.

Hope you found this video useful. Subscribe to the channel for more amazing content and push that notification bell to never miss another video. Thanks for watching and as always, see you in the next one.