AI Dev 25 | Harrison Chase: Long Term Memory with LangGraph

Thank you for having me here. Excited to be here. This is a link to some of the workshop code that we'll be using if people want to follow along. If you don't, that's fine as well. I'll voiceover most things, so it's a bit of a choose your own adventure. My name's Harrison, I'm the co founder and CEO of LangChain. I'll do a little intro about myself while people pull up the materials if they want. So My background's in ML, MLOps. I've been working on LangChain for the past two and a half years. We build developer tools to help make it as easy as possible to build applications on top of LLMs. So we think LLMs are great, but we think they're even better when they're combined with your own data and your own APIs and things like that. And so we build a bunch of developer tools to help with that. One of the developer tools that we build is Langgraph. So Langgraph is an orchestration framework for building agent and multi agent systems. It's very low level, it's extremely customizable. Langgraph is extremely low level, is extremely customizable. So there's no hidden prompts, there's no hidden. Kind of like cognitive architectures. We don't enforce anything at all really. And so we've been. This is kind of like a follow up to LangChain. It's been out for about a year. We've seen pretty good adoption of it, including by other companies speaking at this event, like Replit, which was talking earlier. So they have their Replit agent which is built on top of Langgraph and Langsmith. And so this is where we are encouraging people to start if they are thinking about building agent or multi agent systems. One of the more advanced things that we're starting to see some people do with these systems is build in the concept of memory. So the idea of memory is if you interact with a human, they remember things about you, they remember things about previous interactions. And so it's not like having the same conversation over and over again. You don't have to constantly remind them of things. And because LLMs are so human, like in a lot of regards, I feel like sometimes we expect them to be the same way. But that doesn't happen. That doesn't just happen, that you have to build that into the system. This is still one of the more researching advanced topics out there, but we see a ton of interest in it. Actually today we released with Deep Learning, a course on Agentic memory with Langgraph. That's what I want to cover today. So I'll start by just walking through some slides and talking about the different types of memory that we see. And we've mapped them onto human memory, which may or may not be the right thing to do. Again, it's incredibly early on in this, in this whole space, but especially in the memory space. But I think it's a nice way to think about memory. So I'll talk about those and then I'll walk through an example in code and you can follow along if you want, of building an email assistant that has these different types of memory in it. So should be fun. Let's get started. So to help motivate some of this, we ran this survey a few months ago asking a bunch of developers where they thought agents could have the most impact. One of the top picked ones was basically personal assistance and productivity tasks. So that motivates an email assistant that we will kind of like walk through building today. And this is actually one thing I've been building for the past year and I have running on my emails. So this is a real, all of this is grounded in kind of like real use cases that I've built and used. And so the idea here is that the agent will be relatively simple. It will be connected to an email inbox, an email will come in. The agent, powered by an LLM will use a variety of tools such as a calendar tool to look at a calendar to schedule calendar events, and then an email writing tool to respond to the emails. And so this is kind of like the basic setup of the agent. And if you think about how you would interact with an agent like this, it's similar to how you might interact with an EA if, if you have one. And so they might ask you what types of emails you want to respond to versus ignore, versus you should get notified about. They might ask you your preferences of when you want to schedule meetings. They might ask you your preferences for your tone. And these are accumulated over time as they do things and ask questions and as you respond and give them feedback and change things. And so there's all these questions throughout this process that yes, you could bake in at the start through prompting, but one thing that we've noticed is that no one's really a great prompter yet. And even when I was writing the prompt for my email assistant, there are so many edge cases of different emails that come in that I could just not have predicted ahead of time. And so this idea of learning and adapting over Time through feedback, I think is really important towards bringing the email assistant from bringing something that's like 80% useful to like 95, 97, getting it up there. And so the idea of adding in memory to applications is to really help them to maintain context over time, to adapt to our preferences and learn from these past interactions. So what are some different types of memory? And so these are all grounded in different human types of memory. The first one is called semantic memory. This is typically referred to as just facts that you might remember. So the human example is things that you might learn in school, like specific dates, specific locations, things like that. Another type is episodic. These are experiences that you remember. So if you have a vivid experience of something that happened, you know, a week ago or a month ago or a year ago, that would be a type of episodic memory. So it's a specific concrete experience, not a fact, but a concrete experience. And then procedural are instructions. And so these are instincts or motor skills, like riding a bike. You know, that's a, that's a type of memory that we have. So how do we map these onto agents? So when we think about agents and when we think about semantic memory, this is pretty naturally represented as facts about a user. I think most people here may have heard of semantic search, so doing some sort of retrieval to select facts that maps very naturally onto that episodic can be represented as past agent actions. So if people have heard of few shot prompting, this is when you give an LLM examples of data points and how it should handle them. So inputs and expected outputs. And so this can be used to represent past agent actions and have this type of episodic memory. And then procedural are often just instructions. These are the system prompts that you give the agent. There's a supernatural corollary here. So these are kind of like the three types of memory that we see, what they look like in humans and how they are applied to agents. One other concept that I want to talk about is how we update these memories. So we see two different strategies for how folks do this. One is in the hot path. And so what I mean by this is if you're interacting with an Agentix system, it might update the memories before it responds to you. So in this kind of example, you see a user message, then you see a blue message which represents a tool call. So using a tool call to update memory and then you respond to the user and then same again. And so if you've played around with ChatGPT's memory feature, this is how it works under the hood. So it calls some memory functions to update it. It shows you in the UI that it remembered something and then it continues to respond. That's in the hot path. A separate one is updating things in the background. So this is when you have a conversation, you don't actually think about updating memory in the moment. And then sometime later there's some system that comes along and uses that conversation or those. I'm saying conversation because I think that's chat is the predominant ux. But by this I mean really like any interaction with an agent, regardless of the form factor. And so you take this conversation and you Update the memory 30 minutes later or something like that. And that's what we call in the background. And there's pros and cons to both of these. And so in the hot path you can show it to the user as it's happening. So it's kind of like user friendly and it applies right away, right? Like if it remembers something, that applies instantaneously. The downside of that is now your agent has to not only think about how to do its job, but also how to update its memory. And so you're giving it more things to think about, which can be a little bit tough for things to do. And then the other downside is that it can often or it can sometimes increase latency. And so if latency is super important, you might not want to do it that way. In contrast, in the background has the reverse effect. It doesn't introduce any latency and you can have a completely different system that's responsible for this. The con of this is that it might not be super transparent to the user and it might take a little bit for it to take effect. So these are some of the trade offs. We'll actually use both of these in some of the courses. So how are we gonna apply this to the email assistant? This is what we have, this is what we'll build. We'll first build an email assistant that will do a triage step. The triage step will basically be to filter out emails. So this is how I have my email assistant. There are emails that it should just ignore, there are emails that it should respond to, and then there are emails that it should notify me about. So it shouldn't respond to them, but it should notify me. So we'll use this to triage emails into three different categories. Then after that, if it's determined that it should respond to, we'll go on to the main agent where it has these tools and it will loop the agent we'll build. We'll just do a simple tool calling loop, but you can make this more complicated. So where will we add in memories? First we'll give that agent a separate memory tool. So we'll give it a memory tool for updating and for accessing semantic types of memories, so facts about the users that it's having a conversation with. Next, for the triage step, we'll give this episodic memory. So we'll give this few shot examples. We found that few shot example prompting is very helpful for classification tasks. And so this is a pretty natural, albeit simple, place to put this. And then third, we will have some of the system prompts of the agents, specifically some parts of the system prompt. So instructions for how to do meetings and how in the tone of the agent, as well as actually the triage instructions themselves. We'll have these be procedural memory system prompts that we update as well. So we'll be updating not the whole prompt, but the parts of the prompt that relate to these specific types of instruction. Cool. So this is overview of what we will be covering. There will be four kind of like notebooks that we go through. The first one will just set up the email assistant. And so if you're not familiar with Langgraph, the first one will be an introduction to that. We'll cover some of the basic topics. Then from there we'll spend one notebook on semantic memory, one notebook on episodic memory, and one notebook on procedural memory all layering on top of each other. So you get one agent with all of them at the end. Before I jump into that, any questions? We'll maybe take one or two if there are any. If there aren't, that's fine. We can jump into the fun stuff as well. Yeah.

So I think this makes total sense.

And I love it.

And it really works if you're talking.

About an individual user.

But if you were to expand this to some sort of enterprise setting, how do you deal with that memory management when business rules change all the time or memory can conflict or impact another.

Yeah, yeah, good question. So this, the question was about how do you deal with this as an enterprise setting where you maybe have multiple users? Things change over time and there might be conflicts between memory. Great question. So there's a few things to think about. One, when we talk about all of these memories, all of these memories will be namespaced to a certain collection. And so when we do this example, it will be namespaced to the user, the user that's using this email assistant, but you can namespace it to an organization, to a company, to the overall agent. So the agent itself could have its own memory. And so basically you can split this memory, whatever is most appropriate for the task at hand. So for example, if you have kind of like a SQL writing assistant, and there's a particular way of calculating ARR that is not specific to a user, that's specific to the whole organization. And so you might want to remember that. So the name spacing and how you think about memories, that's one thing that's important. Two, for a lot of these, it's not just additive, it's also kind of like updating memories. So if something changes, the idea would be that it would update that memory. And so actually this is like pretty easy to do for some of the procedural memory ones. It's a little bit harder to do for the few shot and semantics. So few shots are actually generally pretty additive. So you generally kind of accumulate these few shot examples. And then the semantic memory is tricky because when you have a new memory come in, you need to basically see what similar memories exist and then if they should be updated or if you should add a new one to that. And so in this. And so there actually is a bunch of nuance and trickiness there. And so in this course, we'll be using a package called langmem, which contains a bunch of utilities. And so it contains utilities for when it's updating a semantic memory, you don't just like extract a fact. You actually first find like five relevant facts, put them into context, and then ask it to either update those facts based on ID or add a new one. So there's some nuance there. And then to your third point, around temporal kind of like memories, things that might go out of date. That's a really good point, to be honest. We don't cover that in this yet. And we don't have too many utils around that in the memory system. I think the way you would probably want to do that is build some type of. You'd want to make the store aware of time in some capacity. So if you have something like this week, I cannot take meetings. It recognizes that that only applies for this week. We don't have anything like that at the moment. But that's a great kind of like advanced topic. Here. I'll take these two and then we'll go on.

So I'm curious, what is the persistence layer for this kind of memory? So, for example, semantic memory, should it go to like a Vector store as embeddings, proxy rules should go to a transaction store.

Yeah, really good question about the persistence layer. So for those of you not familiar with Langgraph, there are actually two different types of persistence layer that we have baked in. The first one that we introduced is what we call a check pointer. This is basically, you can think of this as a conversational thread now for long term memory, you want it to persist across threads. So we introduce a separate concept which we call a store. And this manages all three of those. The way it does that is it's actually quite simple. It's basically namespace key value store that supports semantic search. So for some of these, like the procedural ones and the episodic ones, actually episodic has some semanticness in as well. But for the system prompts, those are just stored as key value pairs and they're namespaced by whatever you want them to be. So if you want them to be namespaced by the user, they're namespaced by the user. If it's by the org, it's by the org. And then inside the agent, you look it up according to that, the semantic memory and episodic memory as part of the store, the store needs to support semantic search. And so then you can enable that and it will do semantic search over those examples. Last question before we go on.

So super quick. So I actually worked on something similar to this like a couple months ago and one of the problems that I saw is like, you know, if you have a high volume of emails, like token usage can pile up really fast. Do you have like, you know, any tips on how to control the amount of like, you know, tokens you're using there? I don't know, maybe you're like only looking at the subjects like, you know, do you have any tips to not go super over budget?

Yeah, no, that's a really good point. I think if you.