Creating Low Latency Voice Agents - Open Source 🗣️🗣️🗣️

Hey, what is your name and can you tell me about yourself?

Nice to meet you. My name is Verbie and I'm a friendly AI assistant here to help answer your questions and make your life easier. I'm constantly learning and improving so I can provide you with the most accurate and helpful information. I'm a chatty bot, but I'll keep my answers short and sweet.

Can you tell me a fun thing to do while I'm on a trip to Los Angeles over the weekend?

Lee has plenty of fun activities. Why not explore the Griffith Observatory at Sunset, take the tram ride, gaze at the city skyline and enjoy the stunning views of la. If you're visiting on a Friday or Saturday, stick around for the free public lectures and events. It's a unique and unforgettable LA experience.

A few days ago Google announced Gemini Live, which is their AI powered voice assistant. It's their answer to GPT4O with advanced voice mode, which was announced back in May 2024 but we are still waiting for it. To Google's credit, Gemini Live is available to advanced users and comes with some pretty amazing voices.

Here's one of the voices I have, but there are more to choose from.

Here's another voice I can use. You can pick me now and always make a change later in settings.

Or maybe you'd like to listen to a voice like this. A pretty great one if I do say so myself.

Great.

So this got me thinking whether we can replicate some of the functionality of these voice assistant using the existing tools that we have. So we're going to create a system that is going to have four different components. The first one is going to be user input and output. We'll get user input in the form of speech that is going to be passed on to a speech to text message model that will transcribe the user input. The text will be given to the LLM to generate a response in the form of text and then that text is going to be converted into speech using another model and we're going to play that speech back to the user. The system is going to have memory so it will remember previous conversations. Now keep in mind, unlike gemini live or gpt4o, this is not a multimodal system. We are putting together different models to generate voice to voice chat assistant and the goal is to generate a system that can have the least available latency using different tools available on the market. Now to put all these different models together we need a framework of and luckily I have been working on a project called Varby which is Supposed to be a voice assistant. I created this with the help of community a few months ago but never really got a chance to cover it on this channel. This is a modular voice assistant application for experimenting with state of the art transcription, response generation and text to speech models and it has support for a number of different API providers including OpenAI, Groq, ElevenLabs, Cartesia and Deepgram APIs. Support for other APIs is coming soon and you can also run Text to Speech, Speech to Text and local LLM models as a part of this system we already have 159 stars so check it out and make sure to give it a star if you like it. Idea is that you can select different models from different providers and see which one gives you the best possible latency response accuracy. For my experiments I found that the speech to text model hosted by CROC is the best model in terms of latency for speech to text conversion. Also for LLM infants the GROK hosted models are best in class and for text to speech we are going to be using Cartesia AI. Cartesia AI apart from Deep Chrome have over one of the best text to speech system and they have the lowest latency. Again you can replace all these models with local and open source models. However you will need a pretty powerful computer to run these and get responses in acceptable time frame. Project YOBE has support for local models and I'll show you how to set those up, but I'm not going to walk you through a whole tutorial in this specific video. So we'll set this up locally and then I'll show you a few options. But let's first talk about its features. It's a modular design so you can easily switch between different models for transcription, LLMs and text to speech. It has support for multiple API providers including OpenAI, Croc, DeepGrams, and we are planning on adding more. And it also has support for local models. It can record audio and playback as well. Now here's the project structure. If you want to change any model you will just need to change them in config py. Now here are the step by step instructions on how to get started. So first we'll need to clone the repo and then create a virtual environment, install all the required packages, provide different API keys that you will need and also update the config file. This will basically control which model you want to use for transcription, LLM and text to speech. And as I said you can run local models via Olama. There's support for fast Whisper so you can run that as well. And for local text to speech, we currently support mellow text to speech model but would love contributions from the community if you think that you can add extra features to this project. So please clone the repo, submit a pr, I would love to integrate those. So in order to clone it, just use git clone and then provide the repo. As you can see, I have already cloned the repo so I'm not going to do this step. Next I'm going to create a new conda environment using conda create n. The environment name is going to be verbie and we're going to be using Python 3.10 so it will ask me to install different packages. We're going to say yes, let's activate the virtual environment that we just created using conda activate verb and you can see that we are in this new virtual environment now. Next we need to install all the requirements, so we're going to use pip installation our requirements Txt. This will install different packages that we will need in order to run different models. Okay, next we need to set up which model to use for which purpose. So for that you need to go to the voice assistant package or folder, then click on config py. So here you will need to set up your transcription model, response model and the text to speech model. The possible values are OpenAI, GROQ, Deepcom and Fast Whisper API. There are detailed instructions in the repo on how to set up the FastWhisper API. If you're using the Whisper model locally for LLM, you can use these options. We're going to be adding more soon. And for the text to speech models, currently you can use OpenAI Deepgram 11 Labs mellow text to speech model that you will need to run locally. Instructions again are in the repo Cartesia AI so these are possible values. In my case I have found GROK for transcription plus the LLM inference to be the best options in terms of the lowest possible latency. And Cartesia also has pretty amazing response time for text to speech systems, so I'm going to set those up. But we will need to provide the API keys for these models in order for us to be able to use them. So for that you need to look at the examples ENV file. So provide the API keys for the models that you're using and then just rename this Examples env to env. Now I'm not going to show you my env because it has all my API keys. Now for Text to speech. The two best options are Deepgram as well as Cartesia. These are the two solutions that will give you the best possible latency. Elevenlabs is another option, but the latency is a bit higher compared to these other two options. So for example, here we can select different voices. So I'm going to select this Australian man and see how long it takes to generate speech with this text.

In the early dawn of human civilization, innovation was driven by necessity, from the invention of the wheel to the creation of written language.

So this is pretty fast, right? So that's why I wanted to use Cartesia AI for my text to speech model. Okay, so here are my settings again. Grok for transcription and for the LLM response and Cartesia for text to speech. And we can switch them around pretty easily. And for Grok, we're going to be using the llama3.8 billion model. Now, in order to run the system, we're going to run this file called Run Voice Assistant. In here. We're piecing all the different models together. You also want to provide your system message here. So here I'm saying you are a helpful assistant called Verbie. You're friendly and fun, help the user and I think, I guess like I'll modify this too and you will help the users with the request. Your answers are short and concise, right? So if you want to give your assistant any Persona, just change this system prompt. Now you can select different voices if you want. There are a number of different customizations that you can make. I'll probably create another subsequent video walking through the architecture and the code base. So let me know if that's something you are interested in. But in order to run our voice assistant, all you need to do is just type Run Python Voice Assistant Py. Hey, what is your name and can you tell me about yourself?

Nice to meet you. My name is Verbie and I'm a friendly AI assistant here to help answer your questions and make your life easier. I'm constantly learning and improving so I can provide you with the most accurate and helpful information. I'm a chatty bot, but I'll keep my answers short and sweet.

Can you tell me a fun thing to do while I'm on a trip to Los Angeles over the weekend.

Lea has plenty of fun activities. Why not explore the Griffith Observatory at sunset, take the tram ride, gaze at the city skyline and enjoy the stunning views of la. If you're visiting on a Friday or Saturday, stick around for the free public lectures and events. It's A unique and unforgettable LA experience.

Are there any beaches I can visit?

LA has some amazing beaches. You should totally check out Venice Beach, Santa Monica Pier and Malibu Beach. They're all within a 30 minute drive from each other. Venice beach has a lively boardwalk, street performers and great eats. Santa Monica Pier has an iconic Ferris wheel and amusement park. Malibu beach is a bit more laid back with stunning views and great surfing spots. Which one sounds like your vibe?

Okay, so I'm going to close this, but this was the real time responses that we're getting. There were no editing and this is the speed that you can expect, which is pretty amazing. Now right now we're just talking with the LLM itself, but you can basically replace the LLM with a rack system where it can look up information or you can give the LLM to do function calling if you want it to retrieve information from something like a web server or do web browsing for you. So the possibilities are limitless here and I'm going to be building on top of this project now just to show you how easy it is to replace this. So let's say we want to use the OpenAI version of the Whisper API. So I'm going to do that. Then let's say we also want to use an OpenAI model which is going to be GPT4O and for wise, I will still keep it to Cartesia. Let's run this again. Hey, I'm planning a trip to Paris. What are the different things that I should take during summer?

That sounds amazing. For a summer trip to Paris, make sure to 1. Light clothing, T shirts, shorts, dresses. 2. Comfortable walking shoes. 3. Sunglasses and a hat. 4. Light jacket or sweater for cooler evenings. 5. Sunscreen 6. Travel adapter for electronics. 7. Reusable water bottle. 8. A small backpack or tote for daily outings. 9. A travel guide or map. 10.

All right, so there was a little bit more latency compared to the previous approach. But let's try another voice system and in this case we're going to replace this with 11 labs. Can you tell me a fun fact about sun and the solar system?

Sure.

Did you know that the sun makes up about 99.86% of the total mass of our solar system? That's how massive it is compared to everything else orbiting around it.

Okay, that's great to know. What is the biggest planet in the solar system?

The biggest planet in our solar system is Jupiter. It's so large that over 1300 Earths could fit inside it.

Okay, so you can see this was a different voice from elevenlab, and you can experiment with different options in here. The whole idea behind this project is that it's modular. You can bring in different components based on your own needs, and in a subsequent video I'll show you how to run all of these things with local models. But for that, the latency is not going to be as good as you get through some of these APIs, especially if you're using Croc and Cartesia. So do check out the Worbie project. Would love contributions from the community to make it a better system. Also, if you like it, make sure to give it a star. We'll highly appreciate that. Anyways, thanks for watching and as always, see you in the next one.