MemGPT: AI Agent 🤖 with Unlimited Memory 🧠 | Store and Load Converstations

If you have used any large language model based application, you are probably aware that one of the limiting factors of these systems is the limited context window or memory these models can hold. For example, gpt4 has a context window of 8,000 tokens. In comparison, llama2 has a context window of around 4,000 tokens. For reference, 100 tokens are almost equal to 75 words. This really limits the context that the large language model can use to generate answers. There are a number of open source projects which are looking to extend the context window of these LLMs. There is this new project called Memory GPT which is proposing a very innovative approach in order to extend the memory of these LLMs. In this video we are going to look at Memory GPT and understand how it works. Then I'll show you how to run this on your local machine and also how to actually customize different options within memory GPT. So let's get started. First we will look at a high level overview of this system and then I'll show you how to run this locally on your own machine. The paper is called Memory GPT Towards LLM as Operating Systems. If you are interested more in learning about it, link is going to be in the description of the video. Now in the abstract there are a few things which I would like to highlight. First, the current LLMs have a very limited context window which hinders their utility in tasks like extended conversation and document analysis. That means that in these applications we are actually limited by how much memory the model can hold and as a result we cannot work with say, extremely large documents, for example, hundreds of pages. Now in this paper they're proposing something called virtual context management, which is taking inspiration from memory management in traditional operating systems. So essentially you have different tiers of memory and they are controlled by this memory management system. So in simple words, here is how it works. So you have the LLM. So let's assume the LLM in this case is GPT4. Now it has a virtual context which comprises of two different components. One is the main context, that is the window or the context window of the LLM. So let's say for GPT4 this is going to be around 8,000 tokens. The second component is the external context and we're going to look at this in a little bit. Now the most important part is that the LLM itself is self aware. So it knows that it has a very limited context window. Now when it sees that it's reaching its limit of this main context. So since it's self aware, it will extract the most important information from the main context and put them in this permanent external context. Now, in order to achieve this, it uses different functions which can both read and write this external context or external memory. Now, what exactly is this external context? So think of this as an external vector structure where LLM is simply pushing the most important bits into that vector structure and whenever it needs, it reads this external context. Now if you look at this table, you will see that different models have different context window or maximum number of tokens. So for example, the llama2 family has 4,000 tokens. Claude2 has 100,000 tokens. Right? But using this memory GPT concept now you can theoretically expand the external context window to infinite memory. So that's the main idea in very simple terms. If you are interested in learning more about it, I'll put link to the paper in the description. Now let's look at how we can run this on our local machine. Here's the GitHub repo of the memory GPT project. So first and foremost we need to simply clone this repo. So I'm going to copy this link. Okay, so here I open another terminal and we are going to clone this repo. So we're going to use the git clone command, then provide the repo name in here and simply hit enter. Okay, so our repo is cloned. Now we simply need to go to this folder. Next we are going to create a new virtual environment using conda create-n then the name of the virtual environment. So I'm Simply calling it mem GPT and I want to use Python 3.10. So we already have this virtual environment created in my case, but I'm going to simply remove this existing virtual environment and create a new one. As you can see, the virtual environment is created. So in order to activate it, we're going to simply copy this, paste it in here. And now you can see that we are in the new virtual environment. Okay, next we need to go and install all the required packages. So we can do that using this PIP install r requirements txt I'm going to go back to my virtual environment and here we'll simply paste that command and run this. Okay, so the installation is complete in here. Now if you are on Windows, you will need to do this extra step where you will need to install PI read line package. Next we need to set up the OpenAI API key. So there are two different commands depending on whether you are on Windows or on Mac. So if you are on Mac or Linux, you will use the export command. If you are on Windows, you will need to use set, then Open API key and you provide your API key. Now, since I am on Mac, so I'm going to be using the export command. So all we need to do is just hit enter and that will set our OpenAI environment variable. And after this video, I'm going to simply revoke this API key. Okay, so before running Memory GPT, let's look at some of the options in here. Now, in order to run it, you simply need to type python main PY that will run the AI assistant and you can start chatting with it. Now, it also supports chat with documents as well. So we're going to look at that. That's an example application. However, something that I want to highlight is in here. So this chatbot or memory GPT supports Personas, both for the AI assistant as well as for the human. So what you can do is, for example, you can provide a Persona. So that's going to be basically a description of what the bot is supposed to look like or talk like. Right? So you simply provide that flag and then a reference to a text file where you set the different parameters for the Persona and then it will adopt that Persona. Similarly, for human, you can use this dash dash human flag and you will need to provide an example text file. And we're going to look at this like how you can actually change it. Another thing to keep in mind, by default it's using GPT4, but there is support for GPT 3.5. So if you want to use GPT 3.5 Turbo, you're going to use this model flag. Right, But I think this is not as accurate as GPT4, but GPT4 will cost you a lot more money. Okay, so now let's run the actual memory GPT model. Now, in order to run it, the first you need to simply invoke Python. And then we are going to call this main PY file and let's run this. Okay, so for the first time when you run it, you actually see this message. Boot up sequence complete, Persona activated, Testing messaging functionality. And I'll show you to how to change this Persona. Right, so we simply need to hit enter. Then the model is thinking, okay, so the first message is coming from the model itself and it says, hello, Chad, welcome. I'm curious about your interest, particularly in Formula one and sailing. Could you share what drives your passion towards these activities? So let's see where exactly this is coming from. Now, for that we need to go back to the folder that we cloned in the beginning. And in there there is this Memory GPT folder. And within that folder you will see two different folders. So one is the Personas and the other one is human. So in this case, if you go within the human, there is this example folder and within that you have two different files. One is the basic Txt. So here the name is Chad. Right? So this is basically a basic file that the LLM is using or Memory GPT is using in order to create the Persona. Or here's another example, and I think this is the one that is being used in the chat. Right? And here the first name is Chad. Right. The occupation is, let's say computer science, Ph.D. student at UC Berkeley. Interest are Formula One sailing. Right. So these are the things that are defined within this file, right? So you can actually modify these and create a different Persona for yourself. Similarly, for the AI assistant, the Personas are within this Personas folder. So there are a few examples you can go in here. And depending on the different applications that we're going to be running, it will adopt different Personas. So for example, if you are chatting with documents, so I think it adopts this Persona which says, I'm an AI assistant designed to help user with document analysis. Right. There is another Persona that they have added specifically for GPT 3.5. Right. So it says my name is Sam. Right? And you can basically define different parameters within a text file and just place that text file in here, right? So for example, if you want to create something like a Samantha. So you simply need to create a text file that has that specific Persona in here and then you can use it as a part of your prompt. Okay, so let's go back to the chat and let's see. We want to have a conversation with the model now. Now the first thing I want to do is to correct the model and say that my name is not Chad. Now what I did was I told it my name is not Chad, I'm Mohammed. Right? And now you can see the thinking process. So it says, that's interesting. There seems to be an error in the human memory. So that's basically the human Persona that it created. I need to correct this directly in the core memory. And it's essential to address each user correctly. Their names is an important part of their identity. It looks at the core memory and then it's going to replace the first name with the name that I provided here. They're simply exposing the internal thinking of the system. So now it says, now that I have corrected My memory. It's time to apologize to Muhammad for the mistake and rephrase the question. Right, here's the message that it came up with. Apologies for the mix of Muhammad, let's start anew and could you share what drives your passion towards Formula one in sailing? Right, so now it's basically waiting for my response. Now let's say if I continue this conversation and we are reaching the limit of the context window of GPT4, what is going to happen? So here is an example that is provided in the paper. Let's say the user is chatting with memorygpt or MEM GPT and it is almost towards the end of the context window. So let's assume it's like a very long conversation right now. In that case, the system generates a warning saying that the conversation history will soon reach its maximum length and be trimmed. Make sure to save any important information from the conversation to your memory before it is removed. Right, and that's when the memory GPT appends its working context and store key information regarding the conversation that it had so far into the external context or external memory. And whenever it needs any of this information, it will simply retrieve it from there. Okay, let me show you a couple of other neat things that you can do so you can actually continue this conversation with the model. Okay, so for example, I told it and always I'm actually interested in extreme sports like skydiving and paragliding. Okay, so again, it's simply updating its memory and with the interest now it's also adding skydiving and paragliding. Okay, now here's the fun part. There are a few other interactive commands that you can use. So for example, there is this slash save command which will save a checkpoint of the current agent or conversation state. So let's try to use this. Okay, so it saved the checkpoint of the conversation here to this JSON file and it also saved the persistent manager to this path. Right, now let's get out of this conversation and let's see if we can actually load this in a new conversation. So I'm going to type slash, slash, exit. This basically terminates the program. Okay, so this is actually the best part. So what I did was I saved checkpoint of my conversation with the agent. So what I'm going to do now is I will start that again. Now what you can actually do is you can go back and load the previous conversation and continue that conversation. Okay, so by default when I like I started a new conversation, so it says welcome Chad, and then it goes on to say hey, my name is Sam. And explanatory companion. Great. Both. But let's do this. So let's go type, slash, load. Then you can actually provide a checkpoint. But since I had a simply only one checkpoint, so it automatically looked at that and it said that no checkpoint specified. That's why loading the most recent checkpoint instead. And it loaded this checkpoint that we stored before. Now let's ask it, what is my name? Right, So I did not create a new Persona, but I'm using a previous checkpoint and let's see if it can remember my name. Okay, so it actually came up with the name Muhammad and it says there's no need to search for this information. It's already stored in the core memory. And Muhammad is probably. Muhammad probably wants to check if I remember him or not. Right. So this is basically the continuation of the previous conversation that I had with the agent. Now let's also see if it remembers any other information regarding me. So I'm asking it about my interest. Now again, it's simply thinking and it's able to actually correctly retrieve my interest. So it says your interest includes extreme sports like skydiving and paragliding. Right. And so on and so forth. Now the great thing about this is it's not only able to have like a pretty huge memory because of this external context it's working with, but you are able to also store and load your previous conversations. So essentially you can continue the same conversation for a very long time without worrying about the agent forgetting about you or your conversation, which I think makes it a very powerful system in itself. Now, apart from the chat functionality, they have a couple of other examples in here. So for example, you can chat with your documents. It can also compute embeddings and do rag. Right. Here's an example that talking to Llama Index API documents. Right? So you can look at these examples yourself or if there is interest, I will create subsequent videos on using memory GPT for chatting with the documents. And it seems like they're working on including support for open source large language model. This is an extremely useful project and I look forward to creating more content on this. If you found this video useful, consider liking the video and subscribe to the channel. Thanks for watching and as always, see you in the next one.