Not Just Talk: A Voice Assistant That can take Actions

Hey, can you check my calendar and tell me what do I have in the next couple of days?

You have a team meeting scheduled for today at 9 o' clock in conference room A. Tomorrow you have a dentist appointment at 14:00 at 123 Health Street. And on the 26th, you're set for dinner with friends at 18:30 at the Italian Restaurant.

Oh, you're right. I'm meeting Dr. Brown for dinner. Can you pull up his contact details?

I found Dr. Brown's contact details. His email is dardabrownealth.com and his phone number is 555-123-4567.

Okay, good. Thanks for that. Also, can you check if I have any outstanding tasks that I need to do?

You have several tasks pending. The most urgent ones are finishing your project proposal by the 27th, which is marked as high priority and currently in progress, and buying groceries, which is due today with medium priority.

All right, thanks a lot for the help.

You're welcome. If you need any more assistance, feel free to ask.

We just saw a demo of Worgi, which is an open source wise assistant that I'm building with the help of Community. But unlike other wise assistants, it can actually take actions on behalf of the user. We can do that by leveraging the function calling abilities of modern LLMs. I covered the setup instructions of WORBI in the previous video. Link to that video is going to be in the description. If you are new to this, I'll highly recommend to watch that video. Now here is how WORBI works. We get the input from the user in the form of speech. Then we deploy a speech to text model that transcribes the speech and is sent to an LLM. Now the LLM gets to decide if it needs to use an external tool or API in the form of a function calling based on the user input. Then using that function call and the output from that step, the LLM will generate a response in the form of text that is sent to a text to speech system and the generated speech is played back to the user. Now, Webby is not a multimodal solution. It actually combines multiple different models for different operations. It supports multiple different APIs for proprietary models and has also support for local models. In a subsequent video I'll show you how to replace all these steps with local models. The code I'm about to show you is already available on GitHub, but it's not on the main branch. Instead it's on the agent branch. So if you want to replicate this, you will need to clone the agent branch using this command. If you would like to contribute to the project, make sure to submit a pull request. Verbie is able to take actions using function calling. So let's understand that real quick. When the user query comes into the LLM, the LLM will look at a list of available tools that it has, and based on that it will determine whether it can generate a response by its own or need to use a specific tool. So first it needs to pick a tool, then it needs to use that tool. The tool can be an external API call such as looking information on a calendar. It will get the response from the tool and then that response from the tool along with the initial prompt is used to generate a final response. The way we saw in the demo. First we need to clone the repo using GP get clone command. But since we want to clone a specific branch, that's why I'm using this dash branch command along with the name of the branch, so it will only clone that specific branch for us. Next we need to move to that folder that we just created or cloned. So we'll use the cdverby command to go to the verbby folder. After that. You need to create a new CONDA environment if you want to set up a new virtual environment. So you can use conda create dash n and you can provide the name of the virtual environment and then the version of Python that you want to use. I have already done this, so I'm not going to do this. Next we are going to activate the virtual environment using conda activate Verbie and you can see that we are in the new virtual environment. If you want to install all the required packages, just use the pip install r requirements txt file. This will install all the packages for you. Before we can run the project, we need to set up a few things. One of them is going to be API keys. So go to the example env file and rename it. You want to delete the example part, you're going to be left with env in here. You need to provide API keys for different API providers depending on which one you want to use. In my case, I'm using the Groq API key both for speech to text part as well as the LLM calling and we're using Cartesia for text to speech. But you can use any of these other models for the same purposes, or you can also use completely local solution. So all of these things can be done locally, which I'm going to show you in a subsequent video. The second Thing that you want to set up is config. So in here by default for transcription I'm using CROC for response model which is the LLM. I'm also again using grok. And for the text to speech model I am using Cartesia. Now there are specific versions that I'm using which I want to highlight. So for a GROK LLM I'm using a version that is specifically trained for tool usage or function calling. Also for transcription or speech to text we are using a distilled version of whisper large v3 that gives you a lot better latency compared to the normal Whisper Large model. Next let me show you which actions are supported. So for that we will check out this agentactions PY file. Here you can see there's a function for getting calendar events, looking at your recent emails, looking at the list of tasks, getting weather information, and so on and so forth. Now in general you will need to implement a specific API call to the service that is providing your calendar information or providing access to your email. So that needs to be implemented. But in my case I am creating dummy data for each one of them. So here is dummy data for our calendar and when I asked the model to look at my calendar it was able to retrieve information from here. Here are some emails, different tasks as well as dummy data for information related to weather, news and contact details. Here's Dr. Brown contact information that was able to pull but depending on your own personal assistant, you will need to implement these functions. We just need to make a small change to the main code of orbi and in the in here in the main branch you will see that it generates response directly using the LLM whenever a user query comes in. But in our case it's calling this Run conversation function which receives the message history or chat history plus the GROQ client that we have implemented. And inside the Run conversation function which is implemented in the AgentAction Py file, we provide a list of tools that the model has access to. The structure is very standard. This is based on the example provided by Grok. OpenAI of follows the same structure. So you basically need to provide information regarding the functions that the model has access to. Then we create a client that is going to use a specific model. So it can be an OpenAI model or Grok model. In this case we provide the messages that have been exchanged so far, list of tools and then the model can decide which function to use depending on the user input and we define the max number of tokens. If the model decides to use a function, it looks at the different functions that are available, picks the corresponding function, generates inputs to that function, make a call, generate the response from that function, and feeds that back into the model to generate final response to the user. I have created a number of different videos on how function calling works. I'll put a link to few of them in the video description. In our case we are using CROC both for the LLM as well as for the speech to text because it provides you the best possible inference speed. For speech text, I'm using a distilled whisper large V3 which is pretty amazing and it's much faster than the normal distal that is available on grok. And for the text to speech, again we're using Cartesia, but you can replace this with OpenAI or Deepgram. Now the beauty of this modeler approach is you can connect any tool that you want. So for example, if you want to connect a rag tool or rag function calling, you will just implement that in here. So in a subsequent video I'll actually show you how to use local GPT with this. That is a planned video that I'm going to be releasing soon. Okay, one last thing before we test it. The that is the system prompt. It's implemented in the main file which is run Voice assistant PY and the system prompt is you are verbie a comprehensive personal assistant with access to users calendar, emails, tasks, weather information, so on and so forth. Use the provided function to retrieve information and assist the user. Always provide thoughtful and detailed responses. In my case I had to hard code the current date and it's because of the way I have generated the dummy data. But you can use the Python date time package to retrieve the current date. If you look here, I have hard coded certain dates. So if I were to ask the model to give me my calendar for the next couple of dates, it first needs to figure out what the current date is. So I'm telling it the current date is August 24th. That way it will know that it needs to look at information for the next two days, which is going to be 25th and 26th. To run this I'm going to use Python. Run Voice Assistant PY file and let's see if it can use the information that is contained in here to generate responses. Here would be. I'm planning on a weekend party for my friends. Can you suggest a few things that we can do?

Sure, I can help with that. To give you the best suggestions, I need to know a few details. First, could you tell me the date for the party.

The weekend is on the 25th and 26th.

Great. I'll need to check the weather for those dates to suggest outdoor activities. Also, do you have any specific interests or preferences for the activities?

I want you to make some recommendations.

Based on the weather forecast. It looks like the 25th will have partly cloudy skies with a 20% chance of precipitation and a temperature of 22 degrees Celsius. This sounds like a great day for outdoor activities. However, the 26th is expected to have rain with an 80% chance of precipitation and a temperature of 18 degrees Celsius, which might be better suited for indoor activities.

Okay, this sounds awesome. Can you check my calendar if I have anything urgent or high priority on those dates?

You have a dentist appointment on the 25th at 14:00 and a dinner with friends on the 26th at 18:30. You might want to schedule your party around these events.

Okay, find the best suitable time and put it on my task list.

I've added plan and Schedule Weekend Party to your task list. With a high priority and due date on the 25th, you're all set to start planning your weekend party.

All right. Thanks Fobi.

You're welcome. If you need any more help or have other questions, feel free to ask. Enjoy your weekend party.

All right. As you can see, it's actually pretty smart. It's able to pull up information from different sources that I've provided, so if you connect different APIs it will be able to do those. You will have to implement those API calls yourself though. If you like the project, make sure to give it a star on GitHub and also would highly recommend to contribute to the project if you want. If you are building in this space, especially agents or rank, I do provide consulting and advising services to startups. If that is something you are interested in. Make sure to reach out. Details are going to be in the video description. I hope you found this video useful. Thanks for watching and as always, see you in the next one.