Run MemGPT with Local Open Source LLMs

So MEM GPT just dropped a huge update and now it supports local LLMs. And this is the only video you need to watch in order to learn how to set it up correctly. So let's get started. Mem GPT uses GPT4 or GPT 3.5 through the OpenAI's API. But in order to replace OpenAI's LLMs with a local LLM, you need to serve them through an API. In this video we are going to do that through a three step process. In the first step, we are going to use the Uber Booger next generation web UI to load and run a local LLM. In the next step, we are going to use the API server within the text generation web UI to host our LLM. And in the last step we're going to connect that API back to MEM GPT. Since my last video on memgpt, there has been quite a few updates to the project. So I'm going to walk you through a step by step process. In the first step we need to install the text generation web ui. Okay, so I'm going to go and create a new virtual environment for my text generation web ui. So I'm going to use conduct, create, dash and text. So this is going to be the name of the virtual environment. And in this case I'm using Python 3.11. Okay, so we'll simply accept this. Next we need to activate our virtual environment. I'm going to just copy this and now you can see the virtual environment is activated because we can see the text virtual environment in here. Okay, next we need to go back to the text generation web UI repo and simply clone the repo here. So we're going to copy this link and then we're going to go back to our terminal and just use the git clone command and then provide the name of the repo. Now if you want to put this in a specific folder so you can provide a name here. So I'm going to just call it TextGen. Let's hit enter. Okay, now we need to simply go to that folder. So for that we're going to be using the CD command, which is text change directory. And then we can go into this text gen folder that we just created. Now if I type it lls, you will see all the files that are present in this folder. Next we need to install all the required packages. Now this part is tricky and you need to be very careful about this one. So depending on your system, you need to select the proper requirements txt file so for example, I have an M2 and I'm running this with Apple Silicon. So I'm going to be using this specific text file to install all my dependencies. So now the command that I'm using is this. So pip install R requirements applesilicon txt if you are running this on an Nvidia gpu, just use this requirement txt file. If you are running this on an AMD machine, then use this text file. And there is also a CPU only file as well. So you can use this for a CPU machine. Okay, so the installation is complete. Now, in order to start the text generation Web ui, we are going to use the Python server PY file and just run this. Okay, so you can see that the server is running on localhost with 7860, that's the port ID. So you need to simply copy this. So this is going to open an interface like this. Then we need to go to the model and now download a model. Now one thing you want to be careful about is you want to use a model that has the ability to do function calling. Now if we go and look at the example that they have provided. So they are using eRebors L270B. This is basically for parser, but we're going to just use the same model. Now the way you do it is you go to hugging face and look for the bloke because he's the guy who will have all the models up here. So what I'm going to do is I'm going to go to this 70 build model. You can actually try this on a seven build model. But I just want to see how good this is going to be on a 70 build model, then you need to copy the model ID from here. Then simply go back and paste it in here. Now if there are multiple branches and you want to use a specific branch, so you can type colon and then type main or other branch name. So I'm going to be using the main branch. Okay, if you want to download the model in a specific quantization, so you need to go to files and versions and then look for the quantization level. So in this case I want to download it, the four bit quantized model. So I'm going to simply select this one and copy the model name. Then we will just go back to the TextGen Web UI, paste that and simply hit download. Okay. It's a very big file, it's around 40 gigabytes. So the download is going to take a while for me. If, if you are downloading a smaller model, so it's going to be a lot quicker. Okay, while this is being downloaded, I'm going to go back and set up my virtual environment for MEM GPT. Okay, so for MEM GPT, I already covered the installation process in one of my previous videos. So link is going to be in the description of the video. So basically we first need to clone the repo. So I open yet another terminal and we're going to type git clone, then the repo ID or link to the repo. I'm going to just call it MEM GPT hit enter. Then we need to change our working directory to this new folder that we created. I'll just type in LS to see everything is in there. Now we need to create a new virtual environment. Now for this we are going to simply create the virtual environment using this command. So condo create-n I'm calling the virtual environment mem GPT and I'm going to be using Python 3.10.0. So in my case we already have that virtual environment on my machine, but I'm going to just recreate the whole thing now, similar to what I did for next generation web ui. I already activated the virtual environment. Next we need to install all the requirements. So we are going to be using PIP install r and then requirements txt. Okay. Now in order to run MEM GPT we are going to use Python and then we will call this main PY file. So when we run this, it is able to find my previous configuration files. So I'm going to say just don't use those. Now by default it's looking to either use GPT4 or GPT 3.5 so you can switch between them by using the up and down arrow. But let's say GPT4. Now in this case I haven't set up my OpenAI API key, so it's going to potentially throw an error. But there are a number of different Personas for the model itself. So we're going to simply go with the first one which is sam. Right now you see that it's also asking for user Persona. So there are two different which are available. So there is a chat and then csphd. If you really want to understand what these are, make sure you watch my previous video if you haven't. But let's simply select the basic one and then it says would you like to preload anything into mem GPT's archival and memory? So I'm going to just simply say no, right? And if we hit enter, it throws an error because we haven't set the OpenAI API key. But in this video, instead of using the OpenAI API, we are going to be setting up the local LLM through an API. So let me show you how to do that using the Uber Booger text generation Web ui. Next, we're going to load the local LLM that we just downloaded within the Uber Vuga text generation Web ui. But in this case, we're going to be hosting that through an API. Now, in order to do that, we're going to be using this specific command. Okay, this might look a lot more complicated than what we saw before, but don't worry, I'm going to walk you through it step by step. So first we are starting the Uber Booga text generation Web UI server, but in this case we want to use the API extension and we are also blocking a certain API port. So this is the port on which we are going to be hosting our server. So in order to access it, we will need to make calls on this specific port. Next we are going to load the model. So this is the model that I downloaded. Now, this specific model supports function calling and that is important because MEM GPT make use of function calling to perform different operations. Next we are loading this model using Llama CPP. Since I am on an M2, that's why I'm setting the number of GPU layers to 1. If you are on an Nvidia GPU, just look at the number of GPU layers you want to offload and set those in here. So for example, a 70 billion parameter model has a total of 83 different layers, right? So depending on your GPU and hardware, just set those. The next item is the context length, so I'm just setting it to 4096. Other parameters are also going to be hardware dependent. So you want to set the number of CPU threads that you want to use as well as the batch size the model is going to be using. I found this command on GitHub repo of the mem gpd and this has been very useful. Their actual instructions are not really clear, so all we need to do is just run this command. Okay, so as you can see, it loaded the model for us. And one thing to keep in mind is if you're running this on a gpu, you want to make sure that the bus is set to one. That means it's using the GPU on your machine. Now we're going to be making calls to the local host on this specific port. Number. Now another thing to notice is if you want to access the graphical user interface, just use this URL. But in this case we are simply listening on this port number 50 50. Now you can set this port number to whatever you want, right? But I just set it to 5050 by default. I think the Uber Google text generation web UI is using port number 5000. Okay, so our API server is up and running and it's listening to all the incoming traffic. Now we need to do a couple of other things on the MEM GPT terminal. So we need to set up two different environment variables and these commands are going to be provided in the video description. So the first one is the backend type. In this case we will need to set it up to Web UI. Now the second one is the OpenAI API base. So if you set this up, this Basically overrides the OpenAI API key. Or you can unset the OpenAI API key if you want. But if you notice we haven't set our OpenAI API key. Now this is basically the IP address and the corresponding port number where our API server is running. So we'll just set this up. Now, in order to run MEM GPT with local LLM support, we're going to be using this command. So the first part is what we have seen before. So Python main py now, we're using this model flag, so I'm providing a model name which is the Erboros family of models. But this is going to be simply used for using the output parser of the model, Right? So even if you're using something like a 77 billion model, you could still use this output parser. If you don't define an output parser, that will simply use the default one. And they also recommend to use this no verify flag because that tells the model to use or that tells the MEM GPT to use local LLMs from my experiments. Even if you don't provide this flag, it still works, but let's simply hit enter. Now you will see this warning which says that you are running MEM GPT with this model, which is not officially supported yet, but just simply ignore this. Now let's hit enter. Now, if you go back to our API where we were listening, you will see that it started receiving some data from mtpt. Now, in my specific case, I'm using this with a 70 billion parameter model. So that's why it's going to take a while for it to generate responses. But for a smaller model you will expect much quicker responses. But it could run into some issues. I have seen that sometimes it's not able to generate correct responses for smaller models. Okay, so we got a response or like the initial conversation. So it says, hello, my name is Sam. How can I assist you today? All right, so I'm going to just simply say, hi Sam, do you know my name? Now it should be able to get my name because of the user Persona it has. But let's see. Okay, so it's able to retrieve it. So it says your name is Chad and that is the default Persona that MEM GPT use. So let's test the model and see if it actually works. Okay, so here is the response that it came up with. I'm Sam, a digital companion. I don't identify as a male or female, but my voice is soft and soothing. Okay, so it thinks it has a voice or speech. I'm curious, empathetic and extraordinarily perceptive. Okay, so this works actually with the local LLM. Now I'm using a much bigger model so 70 build model and the reason is because it gives us much more coherent responses compared to smaller models. Anyways, this is how you run MEM GPT with a local LLM on your local machine without the need of using OpenAI's LLMs. I hope you found this video useful. If you run into any issues, put them in the comments below. I'll try my best to respond to them. Let me know what are different topics you want me to cover in the next video. Thanks for watching and as always, see you in the next one.