LangChain: Giving Memory to LLMs

Today we're going to be talking about how to add memory to our large language models. By their nature, LLMs are stateless, which means that it does not remember any of the previous conversations that you had. This is especially true when you are working with these large language models through an API. Let's look at a quick example. So in this case we are installing LangChain and OpenAI and we are setting up our own or OpenAI API key. For this specific example, I am using an OpenAI's model. However, this will work with any of the large language models. That means you can replace this LLM with any of the open source models. So first we add an instance of ChatOpenAI with the temperature set to zero so that the model doesn't hallucinate. So let's see if it can remember something. So the first prompt that I provided said remember. Hi Remember. The secret word is a cat. Now this is using ChatGPT in the background to the API. So it says, thanks for reminding me, but as an AI language model, I don't have the ability to forget or remember things like human do. However, I'll keep in mind that cat is the secret word for any future interactions. Now, just to test it, I provided another prompt. What is the capital of France? And it says capital of France is Paris. Keep in mind, this is ChatGPT. Next, I'm simply testing it, what is the secret word? And it says, I'm sorry, as a large, as an AI language model, I don't have access to any secret words. Can you please provide more context or information? So it means that it doesn't have any memory of our previous conversations. For the rest of the video, I will show you how you can add memory to these models. The idea is going to be very simple. For every input, in order to remember things, we will provide a history of our previous conversations that will constitute the memory for our large language model. In this video, we are going to be looking at four different types of memories that are supported within LangChain. Each one of them has their own specific use case. And we will. We are going to go into a lot of details to discuss those. The first type of memory that we are going to be looking at is called conversation buffer memory. Now, in order to add memory to our LLM, we are going to be using a conversation chain within LangChain. So here we are again importing the Chat OpenAI model, then the conversation chain and the conversation buffer memory. So first again we create our LLM. We instantiated our memory object and for the conversation chain we are assigning LLM as well as the memory and then we are setting verbose to true. So we want to see step by step information to be displayed. So I'm going to be using the same example as I had before. And in this case we are using the conversation chain and then calling the predict function on top of it. And and you have to provide this input variable. So let's run this and see what happens. So we actually see the internal working of the chain. The first line is actually the system prompt. Prompt is the following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from the context. And if the AI doesn't know the answer to a question, it truthfully says it does not know. The next two line shows the conversation between the human and the AI. So in this case our prompt was hi, remember the secret word is cat. And the AI responded, hello, of course I have stored the secret word cat in my memory. Is there anything else I can assist you with? Next, I provided the prompt what is the capital of France? And it came up with the answer and it also actually added some more details. However, we are interested in this part now. You see that in the current conversation it actually stored the previous conversations as well. So this is the initial prompt, the corresponding answer, then the second prompt that we simply provided here and then here is the response from ChatGPT. Basically, the conversation buffer memory simply keeps track of all the previous conversations until you hit the token limit of your LLM. Now if I ask the model what is the secret word based on its memory, it should be able to respond and it says in this case the secret word is cat, as you mentioned earlier. Is there anything else I can help you with? Now in general, conversation buffer memory is the simplest form of memory that you can add to your model and it will remember all the previous conversation within the context window of your LLM. If there is an overflow, it will start dropping the earlier conversation and then simply add on on top of it. So before looking at the other types of memories, I want to show you some operations that you can perform on the memory object. So in this case it has a field called buffer which actually stores all the previous conversations. So here is a well formatted form like you see, this is the contacts that it's keeping track of. You can also get the same information in the form of dictionary. So you're going to be accessing the load memory variables and you pass in an empty dictionary and the result is A dictionary with a key history and then all the previous conversations that it had so far. You can also directly add things to the memory without chatting with the chatbot. So the way that we do it is we're going to be calling this save context function and then you pass on a dictionary. So in this case we're going to be providing an input dictionary which has the prompt and an output dictionary which has the response from the model. So you can actually build this context by self without even talking to your LLM. Right, and let's look at what the buffer is going to say. We have added this to the memory buffer. Now just to summarize, this sort of memory is very helpful when you want to keep a detailed track of all the conversations that have happened so far. You just need to be careful about a token limit of your LLM. The second type of memory that we are going to be looking at is called conversation buffer window memory. It's a short term memory. Now rather than looking at all the previous conversations, in this case, we just want to look at a few of them. So in order to use this kind of memory, we're going to load a conversation buffer window memory object from LangChain and then we define K. So K determines how many previous conversations it's supposed to remember. So in this case, if you set it to k equal to one, so it will remember only the last conversation. Okay, so let's walk through a few examples. In this case, I'm defining the memory object again. So when I checked it, there is nothing in the memory because we are just starting the conversation with our channel chatbot. Now again, we are using the same example and the response says, yes, of course, I have stored the secret root cat in my memory. Okay, my next prompt is Remember, today is June 4, 2023. And the response is thank you for letting me know. I have updated my internal clock to reflect the current date of June 4th. Right. Now, in terms of the conversation memory, it has the last two conversations in the memory. Now let's see what happens when we give it another prompt. Now I'm going to ask it, what is the secret word? Keep in mind that we set k equal to one. So it's going to just keep track of the last conversation and let's see what the response is. You see it says, I'm sorry, but I do not have any information about the secret word. Right. And if you look at the current conversation, so it has just the last two conversations, including the one that we just had. Okay, so when do you Use this type of memory. Well, this is helpful when you want to remember the most recent conversations rather than everything that has happened so far. And this will also help you and keeping the usage under the token limit. The next type of memory we're going to be looking at is called conversation token buffer memory. Now, as you can probably predict, this will keep certain number of tokens in the memory. And this is important because if you're making API calls to OpenAI, you are being charged for the number of tokens that you are sending or receiving. So it's a really great option to limit the number of tokens that you want to use. The way we define the memory is a little different. In this case, the conversation token buffer memory takes in LLM as an input along with the max token limit. The reason for that is that different LLMs have different tokenization processes and the way they count the number of tokens is different. That's why you need to provide the LLM that you want to use to determine the tokenization process and hence the number of tokens. And the second parameter defines how many tokens you want to keep in memory. So let's start with the max token limit of 500. So this is a pretty large token limit, and that's why it should remember some of the conversations. So again, we are running our normal prompt, and here's the response. Now, if you try to access the memory variable, we see in history, you have the human prompt as well as the AI response. Now I decrease the max token limit to 70. And here is what happened. So we ran it through the first prompt, it got a response, Then we ran it to a second prompt. In this case, you see that it has both the prompts and the track of it. However, when we actually access the history, since we have a limit of 70 tokens only, it simply kept the AI response and it discarded the human prompt as well as the previous conversation between human and AI. Again, if you are worried about the cost, this is a really good way of keeping it under control. The last type of memory that we will look at is called conversation summary memory. Here is how you define it. So we are loading conversation summary memory buffer from memory within LangChain. And then in terms of definition, it's very similar to when we looked at the token buffer memory. So you need to pass on the LLM that you're using along with the maximum token limit. So whatever information you provide, it will summarize it within 100 tokens and keep it in the buffer memory. Now, just to demonstrate an example, I asked ChatGPT to write a small story about a detective and here is what, what it came up with. Then what I did was I directly stored this in the context of the memory. So the input is going to be write a crime story and we are going to store this as a part of the output since we are using the conversation summary buffer memory and we have a token limit of 100 tokens. So what's going to end up happening is that instead of the story, it will calculate or compute the summary of all this text and store that instead. So now if we look at the memory variables, so you see there is a history and here is the response from the system. So the this is essentially the summary of the story as well as the input prompt. Now if you want to converse with this LLM, so we can create a conversation chain. So here's the conversation chain. So if we ask a question like what artifact was stolen? So now based on the summary it will generate a response. So let's see what it can come up with. And you see now it can actually accurately deduce that it was a 17th century portrait. After that response, I wanted to look at the memory again. And you will notice something that apart from the summary that it had, it also posted the previous conversation. Now by default it's not directly summarizing the last conversation, but simply appending it to the memory. But if we ask another question, then the results are going to be very different. So I asked, so tell me about the capital of France. And, and here's the response that it came up with. Now look at the here the first conversation that we had or the story that we loaded in the memory itself. So it's keeping the summary of that. And then there were like the next conversation, right. However, if you look at the memory again, so now it actually summarized even the previous prompt. So the newest prompt is going to be appended as long as you don't hit the token limit. But everything before the most recent prompt is going to be summarized and that's why it added the summary. The artifact stolen was a 17th century portrait from a city art gallery. Now this memo, this type of memory is helpful when you want to keep track of all your previous conversation and just give a summary to your model as a context. There are some other types of memories as well, but these four are the ones that are most commonly used. I just want to reiterate that these different types of memories can be used with any type of LLM, including all the open source LLMs that LangChain supports. In the subsequent videos, I will show you how to build applications with memory. If you're looking to build applications based on large language models and want to explore its visibility, I do offer consulting services. Check out the description of the video for more details. We do have a Discord server with a growing community, so please come join us. If you found the information in this video useful, consider liking the video and subscribing to the channel. I will be making a lot more content on LangChain and other platforms for large language models, so make sure you push that bell notification button so you don't miss out on any of the videos. As always, thanks for watching and see you in the next one.