Creating JARVIS - Your Voice Assistant with Memory

Hey, how are you doing today?

I'm just a program, but I'm here and ready to help. How can I assist you today?

What is your name?

I'm called Assistant. How can I assist you?

Can you pick another name for yourself?

Sure. How about Aidan?

Okay, that sounds good. Can you tell me about gravity?

Sure. Gravity is a fundamental force that attracts two bodies towards each other. The strength of this force depends on the masses of the objects and the distance between them. Isaac Newton first described it with his law of Universal gravitation, and Albert Einstein later refined our understanding with his theory of General relativity, which explains gravity as the curvature of spacetime around massive objects.

Okay, that was great, but do you remember your name?

Yes. My name is Aidan. How can I help you? Next.

Great. Can you summarize the conversation that we had so far?

Of course. You greeted me and asked how I was doing. We then talked about my name and I suggested Aiden. Next. You asked about gravity and I provided a brief explanation. Finally, you confirmed my name and requested a summary of our conversation.

That's great. Thanks for all the help.

You're welcome. If you have any more questions, feel free to ask. Have a great day.

You too.

Thank you. Take care.

You just saw a demo of a voice assistant with memory that we are going to be creating today. Now, full disclaimer. This is not running locally. We are using external APIs, but in subsequent videos I'll show you how to do this with local and private models. Okay, so here's the architecture diagram. As I said, we're using external APIs, and the focus of this video is to use OpenAI's APIs. In a subsequent video, I'll show you how to replace most of these parts with grok, which will increase the speed of inference by a huge margin. And then we'll look at a completely local solution as well. So let me walk you through what exactly is happening. We capture audio through the microphone. Then we make an API call to the Whisper model, which is served on OpenAI server, and as a result of that, we'll get a transcription. So that is basically audio to text step. Then we're using GPT4O, which is the latest model that will take that text query as an input, generate a response in text, and then we use the OpenAI's voice engine to convert that text into audio, and we play that audio back to the user. And in between all these steps, we also keep track of the chat history so the modeler remembers the previous conversation. So in the rest of the video, we will walk through the code real quick. So I have tried my best to make it a modeler code. So there's a function for recording audio that that's basically listening to the microphone. And whenever the user starts talking, it will capture that audio and store it to this file called Test. We are going to look at each and each individual function in more detail. But I just want to walk you through the whole flow first. Then we use this transcribe audio function, which will read the file that was stored by the RecordAudio function along with the client. Now, in this case, our client is the OpenAI API. And within this client we actually select the whisper model. I'll show you that in a bit. And this will basically transcribe the audio into text. That text is then added to chat history. So you want to keep track of all the conversation that is happening. And since this is user input, so the role is going to be user and the content is going to be the actual question or query coming in from the user. Then we use this generate response function, which is using GPT4O to get an answer for the question that the user asks. And we get the text response in this response text. Now, since this is the response from the model, we also add this to chat history. And in this case the role is going to be assistant. Right? And then we take the final response and convert it into speech using this text to speech function. And the output is going to be written into output MP3 file that we play using Play underscore audio function. So hope this gives you a pretty good comprehension of what is happening. But let's look at each function in more detail and understand what exactly is going on. But before that, I want to show you how we are keeping track of chat history and what type of type of instructions are going into the model. So first we create this OpenAI client. I'm not using LangChain or anything. It's just the open AI Python package or Python client directly. And we Pass on our OpenAI API key. Then chat history is going to be a list of dictionaries. Now, in the first part, which is right before any conversation happens, we are setting up the system behavior using a system rule. And the system message is, you are a helpful assistant. Keep your answers short and concise. Okay, so this is going to be basically the main chat history that is going to be populated as the conversations goes on. And you will see examples of how this changes over time. So the first part is to record audio. Now let's look at this function. Okay, for this we are using a speech recognition package, which is a python package. So basically we initialize the recognizer and it's actively listen to your microphone. Now whenever it detects audio, it will start recording. And when you stop talking, that's when it's going to write that audio stream into the file. Okay, so the audio stream is stored in testwave file, then comes the transcription stage. So in transcript is transcribed audio. We pass on the client plus the audio file that we just generated. And basically we send that file to OpenAI API endpoint which is using the Whisper model to transcribe the audio. And the endpoint that we are using is the audio transcription endpoint. We will need to tell it which model to use, which is whisper v2 large, I believe, and then also provide the audio file and the result is a text transcription of your audio file. Okay, so we get our transcription and one part I think I missed was this. So the user input goes into our chat history list, right? With the role of user. Now the transcription that is generated will also be added or appended to the chat history list. So this is basically user input, model response, user input. Okay, so we have a text file that we want the model to actually generate a response for. Right? And the way we do it is we use this generate underscore response function. So you get the API client chat history, overall history, and in this case I'm using the chat completion endpoint. The model that we use is GPT4.0. So we will use this to generate the output. And the input is text, the output is text, and that text is going to be further fed into the audio generation model. Now, one thing to keep in mind, we are feeding the whole chat history. So that's why when I asked the model to summarize the previous conversation, it was able to look at the previous chat and based on that, it's able to summarize the whole conversation. Okay, so the model response is fed into the chat history as we said before, then time and then comes the step of text to speech. Now we take our client the response, only the response, not the complete chat history. And that is going to be stored in output MP3 and within this function. All we are doing is we are hitting another API endpoint. This time it's the audio speech endpoint. It's a text to speech model from OpenAI. So I think there are six different voices available. I selected one of them. So you pass on the model, the voice that you want to use and the actual text that you want to transcribe and this will create an audio file that is going to be stored in the file path that you have provided. Okay. Once the transcription is complete, we need to play the audio. And in order to do that we're going to be using the playaudio function which takes the path of the file and use pygame package to basically create an audio stream. Now a couple of things that I do here is I'm adding a timer to basically sleep for a little bit because sometimes I notice was that even before the audio generation part is complete, it would go and start listening to the user again. Right? So you want to make sure that you give it enough time that the whole audio is played before you start asking the user again. And once that is complete, you go again and start listening and in here and the user actually sees this recording started. So this is basically a cue for the user to start talking again. And we wrap this up inside an infinite while loop. So this thing is going to go on and on and on. Okay, so overall a very simple implementation. Now there are I think several opportunities in which you can improve this system. So as I said, I recently got access to Grok Whisper, so I'm going to be using that for the transcription part and hopefully that will give us a much better speed for the model generation part. We can also use Grok and because that gives you the best possible inference speed from an API. And I'm also thinking to replace the text to speech model with probably a model from elevenlab, because I think one of the voices is actually the voice of Jarvis. So that is going to be a fun thing to do. Now I do some cleanup as well. Once the audios are generated for this specific example, I didn't want it to do that because that adds some processing time. Okay. So my goal with this is not only to just create a voice to voice chat assessment, but this can open up so many different possibilities. For example, I have my own open source project called Local GPT that lets you chat with your documents. So you can integrate something like this, especially if it's running locally on your own hardware, to basically talk to your documents. So you have a query, the model will go retrieve the relevant information and you can have a chat conversation with your documents or you can add something like function calling that gives the model the ability to retrieve information not only from your documents, but even for them from the web or can also perform different operations for you. So that's another possibility that I'm trying to explore. So if you would like to contribute to this project or the open or the local GPT project. The code is going to be available on GitHub in a few days. I need to do some cleanup and also come up with an overall architectural diagram of how I perceive the system is going to work, so would love contributions from the community. Also, if you are interested, we have a very active discord community which has close to 4,000 members now. It has grown a lot over the last year or so, so if you're interested, make sure to check it out. There are some really helpful folks. If you have questions or if you want to contribute with your own expertise, those are welcome as well. And if you want to support my work, there are a few links in the video description. I hope you found this video useful. Thanks for watching and as always, see you in the next one.