Local and Open Source Speech to Speech Assistant

This is probably the most requested video on my project Verbi. Today I'm going to show you how to set up a completely local voice assistant. If you're not familiar with Verbi, it's my own project that lets you communicate with LLMs through voice. So you have user input as voice. Then there is a speech to text system, then there is an LLM followed by a text to speech and you get the speech output in this video. I'll show you how to replace all these three models with completely local models. I have covered the setup of Worbi in my previous videos. I'll put link to those. If you're new to Worby, I'll highly recommend to check those videos out. In terms of hardware, I'm running this on a MacBook Pro with M2 chip with 96 gigabytes of VRAM. Your speed of inference will vary depending on your own hardware, but you can run all these three models on a CPU machine as well, although the speed is going to be pretty slow. And if you like what you are seeing, make sure to give it a star on GitHub. Okay, let's get started. To set up Verbi locally with all local models, we'll need to set three different API endpoints. The initial setup is going to be very similar to what I have shown in the previous videos, so I'll highly recommend to check those out. We need three different API endpoints. The first one is Fast Whisper API. This is going to be used for converting our incoming speech to text. Then we need a local LLM which we are going to be serving through Ollama. And this is going to be used to generate responses for incoming input text. The last API endpoint is for text to speech and in this case we're using Mellow text to speech system that will convert text generated by the LLM into speech. Verbie is very modular in nature, so you can replace any of these different endpoints with the model of your choosing. We will start setting up our local LLM using Ollama. You need to have Ollama installed on your local machine. Now I'm going to be using the Llama 3.8 billion model, but you can use any model of your choice with Ollama. I have the Ollama client already up and running, so I'm going to just use this command Ollama run and then provide the name of the model that I want to run. Now if you are running this for the first time, this will download the model. I have already downloaded the model so we can simply type hi, then we get a response. This means that the API endpoint is running and the model or this client is able to communicate with the API endpoint. Next, we are going to set up our Fast Whisper API. This is going to be used for speech to text. It's based on this amazing project from one of our community members. I'll put a link to this project in the video description. Highly recommend to test it out. So we're going to follow these steps. First we need to clone the repo, then install all the required packages and then run the Fast API. Now you can create a different virtual environment for each of these API endpoints. But I am going to be using the same virtual environment which I created in my previous video and I'm calling this Verbie. So we will first activate that virtual environment for us. Next, I'm going to run the Git clone command. Now I have already cloned this, so we'll get this error message that the folder already exists. But if you are running this for the first time, it will simply clone this specific repo for you. Next we're going to CD into Fast Whisper API. This is the folder under which we can type LS and you can see that there is a requirements TXT file. So this is specific to our Fast Whisper API and in order to install all the required packages we're going to use pip install r requirements Txt and it will install everything that we'll need for this project. After that, the next step is to run the API itself. So we're going to use fastapi, then run and main py. So this will start a server for us which is listening at this specific port. We have two of our API endpoints running now. We need to set up the last one which is our text to speech system. I'm going to open another terminal here, so first I'm going to get out of the word be folder. Now, the instructions for mellow TTs depends on the operating system on which you are running this. So follow this link. Next, I am running this on Mac OS and instructions for installation on Mac OS as well as Linux are the same for Windows it's a little more involved in terms of steps. First we need to clone the repo, Then we're going to change directory to that repo, then run the installation command. So it will install Mello TTS as a package. So it's not going to be an independent repo anymore. Then we need to download our speech to text model, or actually text to speech model in this case. So here again, I'm going to be cloning the repo. I already have cloned it, so I'll get the same error. But then we're going to just change directory into this new folder. And if I type ls, you can see that here are all the files that are contained in there. After that I'm going to run the installation command. Now, depending on whether you're installing it for the first time, this can take some time because it has to download a whole bunch of files. And next we're going to run this command to download our models. So depending on your Internet speed, this can take a while because it's going to be downloading almost a half a gigabyte of model files. Okay, one small mistake that I made was I did not activate my virtual environment. I'm going to actually do this again. We want to install everything within Verbie, so we'll just rerun those commands that we had so far. I'm going to install everything and this is basically downloading the stuff. And that's why I said it can take some time. And I'm going to also download the model files again. Okay, now we need to actually change directly to the verbi folder. So I'm going to type CD and this will take me to the root folder. Then from here I'm going to CD to where we dev. This is the folder where I clone my verb directory. Now if you type ls, here are the different folders that you can expect. In your case, you probably are not going to see the front end. This is a code that I'm currently working on which will basically be used to create a UI with which hopefully is going to be coming up soon. This is with the help of some great community members. So we'll be is going to have a UI pretty soon. But in order to run this third API endpoint, we are going to type Python and then Voice Assistant. And then within that we have local TTS API PY file. So if you look at the Voice Assistant folder here we have local TTS API. This basically implements an API endpoint for Mello tts. Now here in this file you can see the configurations that I'm using, so it will look at your device. In my case it's going to be using mps. If you have CUDA available, you're going to get much better responses or it can even run on your CPU as well. We're using English as a language and then in here's a specific speaker id. So it's a voice that I selected, but there are a number of different voices available. I'll put a link to those. You can choose one of them. If you don't like this voice, we're going to just run this and if everything goes well, this is going to start our server. Now this command can take some time because it has to load the model and then it will start the API endpoint and start listening. So just be mindful of that. Okay, so here it's up and running. It's listening at incoming requests at Port 5150. So we are all set. We can now run Verbi. First we need to make sure that we change the configurations. So go to config PY file. Here we want to use the Fast Whisper API, our transcription model. Then for our response model we are going to replace it with Ollama. And for the text to speech model we are going to replace it with Melotts. So make sure you save the configurations. And now we are going to open another terminal and we'll run this Run Voice Assistant PY file. So now we open another terminal. First we need to activate the virtual environment which is going to be Verbi. In this case. Again, you want to make sure that you have all the three API endpoints up and running. So to run the main code we're going to use this file python run Voice Assistant py. I'm also running all the three API endpoints. We're going to see some of the traffic on the text to speech and speech to text system. Another thing, the speed is going to be dependent on your own hardware. So for the first couple of conversations I will show you the real time speed for the first response. It usually takes longer compared to the subsequent responses. So let's run this. Hey, what is your name? And can you tell me about yourself?

Hello, I'm Verby, a helpful assistant at your service. I'm here to assist you with any questions or tasks you may have. I'm friendly, fun and quick witted, so feel free to ask me anything. I'll do my best to provide concise and accurate answers. I don't have personal experiences or feelings like humans do, but I'm designed to be helpful and informative. Let's chat.

Okay. Can you tell me a joke?

Here's one. Why did the computer go to the doctor? Because it had a virus Hope that made you smile.

Ah, that's a good one. Can you tell me the distance between the sun and the moon?

The average distance from the sun to the moon is about 384.0 km 238900 miles. This distance varies slightly due to the elliptical shape of both orbits.

Okay, nice. That was informative. Can you give me a quick summary of what we have talked about so far?

We've had a nice chat. We started with introductions where I told you a bit about myself as Furby, the helpful assistant. Then I shared a joke to brighten your day. After that you asked about the distance between the sun and the moon and I provided an informative answer. That's the summary of our conversation so far.

You can see in the beginning it did did take some time for the first response, but I think it's really dependent on the amount of text that is generated by the LLM. Plus like the amount of time it will need for the text to speech system to generate a speech output. But the subsequent conversations were pretty short so you can see that it was able to generate responses pretty quickly. It's pretty good in terms of the speed as well and I think like on a similar hardware you would expect very similar speed. Now Verbie can also take actions on your behalf so it can use tools depending on the LLM that you you select. I have a video on that specific topic. I'll put a link to that in the video description. Also I will put a link to the first video which goes over the architecture of Verbi and how to set this up. If you are interested in learning more about Wormy or where this project is heading, make sure to subscribe to the channel. I will be making quite a few updates to the code base and my goal for this is to make it a project where you can experiment with different state of the art speech components in designing the best possible AI speech to speech system. So if that's something which interests you, make sure to subscribe to the channel. I hope you found this video useful. Thanks for watching and as always, see you in the next one.