⚡️ OpenClaw's Memory Sucks and the fix is simple — Dhravya Shah, Supermemory

Okay, here we're here in a remote studio with Dhravya Shah of Supermemory. Welcome to Latentspace.

Thanks for inviting me.

Yeah, obviously you've been blowing up on the timeline for multiple years now. I just found out you launched Supermemory in 2023. It feels shorter than that, but also you've been doing this for a while.

Yeah, I've been in this space for way too long. Supermemory has been many different products. But now it's everything at once. I'll get back into that, but yeah, lots of things.

So what is this lore? Can you, uh, we looked back at your first Product Hunt launch. You can show us what was it originally?

Let's, let's walk back from there. So SuperMemory launched as this consumer app that was supposed to be your kind of a better bookmarking or note-taking tool with a few other features. So like, for example, you can take your memories across LLMs and this is from very long ago. So we had an MCP launch and then we had another launch for the app itself. So earlier, obviously it was just the app, which was like a save. You can save things to the app. You can get things from the app. Very simple as that. And then we were like, okay, MCP is a thing now. So we should probably make an MCP out of it. And this was a time when I personally was doing internships. I was working at Cloudflare at the time, and this was a side project that I did as a part of this thing called BuildSpace. And BuildSpace, you might know about BuildSpace. It was like this place where a lot of builders used to build things nights and weekends. And they really pushed you to building something and like talking to users and stuff like that.

Yeah, it's like a Gen Z accelerator that like they've founder just gave up and just like moved on.

Exactly. So I was in both season 3 and season 5 of BuildSpace, and I open sourced Soup Memory during BuildSpace, and it, like, the project absolutely blew up. It got like, you know, it was one of the biggest projects, like fastest growing projects of 2024. And I kept tweeting about how I'm running this on Cloudflare on $5 a month budget. And it was still a consumer app. But then I kept adding features to it. So earlier, the way we, the world used to think of memory and how we used to think of memory was just RAG. So you just embed it, put it in a vector database, and that's all you need to do. And now then I like dug down the rabbit hole. And so vector databases, like at that time, and even now, except a few like TurboPuffer and Chroma, were not really scalable for the scale I was getting to where they were either getting too slow or too expensive. run. And so I had to, you know, find other creative ways of making it really cheap for me to run. And I kept tweeting about all of this. So my tweet was like, I run Soup Memory on $5 a month and I have 100,000 users and this is my architecture. So it was like an entire diagram and that blew up. That was the first kind of, you know, famous Soup Memory event. And that led to not people being interested in the app itself, people being more interested in the infrastructure that I was building.

Yeah, I remember you open sourced it and then I was like, oh yeah, I need to star and fork this. Cause I was like, if I ever need this, I need, I need this setup.

Exactly. Exactly. So after starring it and like after that open source, you know, it blew up to like 10,000 stars in a few weeks. But then I realized like, you know, a lot of companies were reaching out to me. They were like, hey, can you help us set up our memory thing? And I also went and worked at other memory companies and it became like this entire thing that like I was doing pretty much a lot of consultation work around context engineering. And I realized like all of these have very similar use cases, but with slightly different flavors where they, they not only need retrieval, they need like, like true user understanding. And we'll get, get to that as well. Like what is modern memory? You know, like how should AI remember? And then I was like, okay, so to make an LLM truly good at understanding a user, you have to handle 4 things. One is knowledge updates. So you have to invalidate stale knowledge and build on top of it, which is different from the storing vectors. You have to have some sort of temporal reasoning. So you have to have like, give the agent an understanding of time and how things have passed for personalization. And you need some sort of forgetfulness to forget things that are not relevant anymore at all. But you also need this concept we call user profiles to always like, memory is not just a retrieval call. The LLM has this very small profile of the user that it will utilize on every single turn. So I was building all of these components into SuperMemo and all of these apps. And I realized that, okay, there's some really difficult infrastructure challenges to be solved here, which is both on the extraction layer, which is how do you truly like, you know, how do you train a small model that can extract things about the user and maintain the second layer, which is the knowledge graph. And the way we build our knowledge graph is different from how people usually build knowledge graphs. So people usually have this concept of triplets.

Triplets.

Where they have—

Object, predicate, object.

Exactly. Object, predicate, object, or entity, relation, entity. And we think that triplets actually lead to worse performance because you have to traverse them a lot to get to any information. And like, you can just think of it as—

This is the learning of every graph guy ever.

Exactly. Exactly. So you have to Like to, to, to find out like, okay, what do, what food does Dhravya like? You have to first get to the entity Dhravya and you have to traverse 1 or 2 layers deep to find out like, okay, Dhravya likes food, Dhravya enjoys this thing, et cetera, et cetera. And that leads to extremely slow and like, this is just unnecessary. So we built our own knowledge graph. We built our own extraction pipeline. And I was like, this consumer app is probably not useful anymore. This core infrastructure is more useful now, and that is our core business now, and that's what we offer.

Amazing update. I think one of those things where, like, I think we can get more into it. What is SuperMemory today? Like, is it a cloud business? Is it a VC-backed company? You know, you're a young founder. Like, tell people more just about the company itself and what it is today, because obviously we spent a bunch on the background. Yeah.

Yeah. Uh, well, I should have done this before, but, uh, all right. I'll give a little bit of an intro about myself first. So I'm Dhravya. I'm currently 20 years old. Before this, I was working on agents and databases and a bunch of other things at a few companies, including Cloudflare, HypeFury, and others. I had 2 companies acquired and I came to the US. So I built Supermemory again as this thing, raised $3 million, um, last year. And since then we have been a VC-backed team. Cloud business that's mostly open source actually. And we are trying to build a context infrastructure for the AI era. And so what do I mean by context infrastructure? Whatever external context or like that the agent would need, including just retrieval to like user understanding, memory, et cetera, like self-improvement, self-learning, personalization. We provide all of these in this one single tool, which is Soukhman, because the way we structure the data, we can automatically provide all of these facets. Because again, every single memory use case would require a different way of implementing it. And so we do that for developers. We provide an API for people to use, but we also still have a consumer-facing app, which the idea is that you have your own memory that you can utilize across everything you use, or you can add Soap Memory to your agent to make it much better at remembering things. So for example, the OpenClothing was blowing up. or Claude Code blew up as well. It got like 2 million impressions. Both of them were kind of this consumer-facing plugins that we offer.

Yeah, it's really impressive to watch. I think a lot of people, I'm gonna put OpenCLAW in the title of this thing. Let's go right into the OpenCLAW thing.

Let's do it.

There's more to go into SuperMemory. I think more people can check it out. You've published a lot of stuff, but the topic of the moment is, I think, was it Peter Levels or something? Somebody was having problems with OpenCLAW memory.

Everyone, everyone has been.

And then they're like, well, okay, how do you do it? And then it's like, all these different answers and I'm like, wow, this is really not solved.

Yeah, exactly.

Okay. So tell everybody what's going on.

All right. So let's first talk about how OpenCLaw currently handles memory. And I have like this little Claude artifact that I prepared for my article. But so the way OpenCLaw has with QMD or with whatever memory plugin you use, it inherently relies on tools. To search through these memory.md files that it prepares. So, you know, like what did I decide about the API? Then agent will decide to search and sometimes it won't decide to search, which is probably the biggest problem here. And then it will get some results, it will do some scoring, and then it will return it back. So, and the search right now is through the memory.md files or the memory folder. Yeah. So if you, if the agent decides to not remember things explicitly, it will never show up in the results because you don't know what to remember at ingestion time. So there's a bunch of issues here. And because these files are static, they don't handle updates. They don't handle, uh, like the forgetfulness I was talking about. There's no temporal reasoning. You cannot look at particular parts of the file because you don't know what is where. And other problems like that. This, like, you know, this is the reason why everyone is complaining about OpenCLaw's memory. This to me is surprising because when people complain about OpenCLaw's memory or when OpenCLaw even did the memory, people didn't like look into how exactly and why it's bad. So we did all of this digging in. And we were like, okay, what is the right way to do memory? And we already had these plugins for Claude Code, for Open Code. So we had a lot of learnings about how agents should be stateful. So we basically turned the tools-based approach to hooks-based approach that actually is reading from the Sup memory graph, which keeps the content fresh. So it handles updates, et cetera, et cetera. So now, you know, for every tool call or for every user message, a hook automatically will put less than 2,000 tokens worth of information dynamically. Um, so the entire context will not have more than 2,000 super memory tokens. And optionally, it could also do a tool call if it needs more information. And we have like contradiction resolution, we have temporal reasoning and all of that built in. So the information is always truly fresh, no matter what. That's the gist of it, but we can dive more into it because like there's some things that are not in this diagram that we also do. So yeah, that's kind of the simple why Supernanny is just better.

QMD is from Toby Luca, which is like really super impressive. I think it's a very interesting— basically Toby wants to do everything locally. You obviously are Cloudflare maxis, so you're going to do it on Cloudflare. So obviously that's like a fundamentally different thing. I think hook-based is probably right because obviously models make mistakes. And I get very, very frustrated when like I use Groq in xAI and it doesn't search tweets. Like I'm, dude, there's only one reason I use Groq is to search tweets.

Yeah. You have to like ask it, like, please use these tools. And most of the OpenCLAW users don't even know that tools are a thing. So they don't want to think about the tools. So yeah.

Do you think, do you perceive that they're non-technical? I mean, is, you know.

Especially, you know, a lot of people I have been seeing, they are technical in the engineering sense, but they're not technical in the agent's world. I think we're in a huge bubble right now. And like, they won't know that, okay, this is exactly what's going on behind the scenes. Like, they would not even look into it. So yeah.

It's just curious because as an engineer, you should— this is obviously you and I think that this is the only thing that's interesting, but you know, there's many other engineers who have real life. Yeah. So, okay. I think other than this, there's another memory discussion that I think Cursor and Claude, those guys have been saying, which is like sort of file system-based memory, right? Like where you just, You put, you dump a bunch of whatever.md into your file system and then you grep over it. And OpenCLaw has some compaction, right? Like some data journal, some heartbeat stuff. What are your takes on all those things?

So I think file system memory is actually the right way to do things in a lot of cases. Like for most Claude Core users, that is probably the right thing to do. But you still have the same problem of you have to know what to remember in order for it to remember things. So you have to explicitly mention that, okay, please remember this or please bring this up. And it's limited by the files themselves. So the files can get longer and longer. So that becomes a problem. So you have to split between many files and the files still don't have any update logic, et cetera, et cetera, unless you explicitly have the agent do that. They're also really slow to traverse through. So you'll have to do agentic discovery to find out any context to answer any question. Like, for example, if I have something like implement this workflow, it will have to, you know, first look through everything it knows in the memory. All the workflows, etc., etc., and then it comes to the answer. Now I think this is a very hot take, but I think that they could like one is personalization, which I'm talking about, but then there's also code indexing, and they could index the code, and Cursor does, but Plotcode does not. And I believe this is also because they are not incentivized to utilize less tokens by the end user because they're just offsetting the cost of indexing, etc., etc., to. Tokens and time that the user takes to run the agent, which is, which is what they want. So theoretically, you could add like code indexing, you could add personalization. It would be much better and faster. And I'm not just saying it. We actually ran a benchmark just yesterday. I published this blog. I'll share my screen again. I published this like extremely technical report, which also kind of blew up, but we ran a benchmark on long memory LLMs. Against 3 things. Oh, the screenshot is in one of the code tweets, I think. Uh, how do I view quotes? Twitter UI keeps changing. Here. Um, so the Claude code one performed the worst and OpenCLAS slightly more than that and SuperMemory is the highest. And you can see that, you know, it's like a pretty significant difference, like almost 50%. Let me, let me clarify.

This is your reproduction of the Claude code method and your reproduction of OpenCLaw, or you actually use Claude code?

No, it is our reproduction. So it's like pretty much exactly—

I think it's important to say. Yeah.

Yeah, it is pretty important. And, uh, you know, we also think that benchmarks should not be trusted and cannot be trusted. So like, you know, you should not listen to me actually. So we built this open source kind of benchmarking evaluation platform, like Groq's OpenBench. It's called MemoryBench. Which I'll also share my screen. So we, like, this is our implementation of exactly how it's doing. In fact, we also literally like went into it. We implemented OpenCLaw's hybrid method with the same formula, with the same everything. And you can just view the PR to find out exactly what's being done. But in short, this memory bench is a way to benchmark any providers against any benchmarks. Against any judge on the same kind of the base rules. And this lets you quantify all the things. So not only the quality of how the memory system performs, but also the latency, cost, and other, like, you know, heuristics like the top-k recall, NDCG, et cetera. So this is like a really good way of doing benchmarks for memory, and it makes it really simple as well. I actually asked every single competitor. to use this because it actually makes it easier for our competitors to compare against SupMemory as well.

Yeah, I think that's a very good infrastructure. Okay. Yeah. People should try that out. Other than, uh, just, just while we're on the topic of benchmarks and all that, there's LongMem, Mivel. Um, what's not so good, whatever you want to say.

Yeah. So I think we are still not there in the memory benchmarks space. And there's a lot of reasons behind this. I might also write a detailed post about this, but I think that LongMemEval is a very good benchmark because it calculates, it tests for the right things. It tests for the right things. The dataset is great. Everything is really good in LongMemEval, except for the fact that you can win LongMemEval by remembering as many things as possible. Like there's a direct correlation to the number of memories extracted and the score you get on LongMemEval. Okay.

That doesn't seem bad.

Because yeah, it's not too bad, but like in real world you would spend more cost. Like in real world, you don't want to extract everything because extraction is not free to win against the benchmarks. So it's good for like some use cases, but in the real world, that's not how people expect memory to work. So we think that that's fine, but LongMemory also does not test for forgetfulness because how would you test if something should be forgotten or not? It tests for updates, but not forgetfulness. And a few other things. So we still use Longman eval because that's the best we have. Then there's Locomo and Locomo is like really bad because like, and, and people actually, it's not just me. Like pretty much everyone knows that Locomo is not the right thing to, to judge.

No, no, I don't know why.

So the reason is like Locomo is essentially testing for retrieval capacity and it's not testing for the real memory things. Like you can essentially just context dump everything in Locomo and get like 100%. In the latest models. So one is the, there's only 30 questions. Like the dataset is really small. Second is it does not test for like knowledge update, temporal reasoning, multi-session, all of the other things, like the important things, like preferences. It doesn't test for anything but retrieval. And like in Gemini Slash or whatever, like you can just literally, the questions, the sessions themselves are short enough that you can put the entire session in these context lens and it will work right away. So I think, and it's also very old now, like in that age it probably did make sense, but now it's been 3, 4 years since that and the models have gotten much better. There's also ConvoMem, which I actually like. It's a really good one. It's by Salesforce and it tests for more assistanty use cases, like assistants should do both RAG and memory, et cetera, et cetera. So that's pretty good. And. Sometimes you need both, which I also want to come to actually. But that's what I think about the benchmark space. There's one important thing, which is there's no concept of benchmarking user profiles. So all of these benchmarks are very retrieval heavy.

Like factual information.

Yeah. Like you, you essentially want to, you know, fetch something, get an answer, answer the question, but that's not all use cases. I'll show you one. A slide real quick, which will make this very clear. I have it ready right here, luckily. So traditionally you have this retrieval thing that happens. So you get a question, you retrieve something, and an answer is generated based on that. But we think that it will not work for most non-literal questions, like find the best monitor for me. If I've never spoken about a monitor, which I probably have not, it will not return anything. It'll be like, oh, it'll give some generic answer. But agent probably needs to know a lot of default things about me at all times. Like user is a CEO, but it also needs to know some episodic information. Like user recently raised fund, fundraising, moved into a new office last week, et cetera, et cetera. And in which case it will be like, oh, you code all day and you just moved into an office. So this is probably the one. that you should buy. It's a little expensive, but this is the one for you. And this can only be done if you really have like a very extremely small, like 500 token information about the user that captures both the static things and the episodic things, like some recent episodes that's been going on. And we also have this built into SuperMemo, but there's no way to benchmark it.

Yeah, it's basically personalization, right? So it's a bit hard. Okay. I just want to close the loop on anything OpenCLAW related. I think people are using a lot of things. What other issues crop up? Things like secrets, things like context confusion if they've done similar things. There's a trade-off between retrieval and accuracy or recall and accuracy, right? Do you find these kinds of things and what are people doing?

Yeah. So I think that the way OpenCLAW does it is it essentially sends back the last 15 messages in the conversation and it essentially uses that back and forth. And I mean, the approach itself is not ideal because you will, like, you are not doing any, like, you're not utilizing any cache tokens. And because you're not utilizing the cache, you're essentially paying like insanely like 10x more than if you would do it the right way. So that is always a problem. So like, We think so that, but that's like an OpenClaw issue. You know, we cannot really do anything about that. The other thing is people are giving it access to their secrets and stuff like that. And we have to, like, as a memory system, we have to like make sure that no secrets end up to us, like in any case. A lot of users in OpenClaw also have like really like browsers attached and stuff like that. And heartbeats and heartbeats are all going on, you know, every hour. And the heartbeats are also ending up in memory, causing some pollution. And some people want to use their OpenCLAW memories along with their Claude core memories. And we actually don't want to do that. Like it should be siloed, like it's their own things, but people want that. So we are offering configuration. Recall-wise, like, you know, super memory sometimes does not end up giving the right results. And that's why we recently shipped a new thing. For OpenClassrooms, it was a huge learning experience for us. It's called hybrid mode. There's always a trade-off between what you want to remember in a memory system. You always have to choose exactly what you want to remember, which is important because you want to keep them updated. You want to keep them fresh. You want to forget things. But sometimes the user, you don't know what the user will ask about. Sometimes a user will say something that was not remembered or was not very significant. So how do you handle those cases? So the way we do it is we have, because we do both memory and like managed RAG, we can show up the memories if they exist, but we have a fallback to RAG answers. Like it will just return the raw chunk if SuperMemory did not remember it. And then in the background, it will add that to the memory as well. So this, this essentially gives you like, you know, like we say that SuperMemory should almost never forget anything because of the fact that we are always giving the agent the right context as long as the user is asking for it.

Amazing. Are we going to the hybrid stuff?

Yeah. So this was the hybrid stuff. So essentially you will return the memories first, and then if there's any raw chunks that match up, we also return those to make sure the agent knows just enough information to answer the question. And no other provider does this right now. So yeah.

How important is cost? Like my perception is people are very cost insensitive, but you obviously, like anyone who's like a real engineer, um, does care about cache hits and all that. Uh, I don't know. I don't— do people care?

Yeah, I think people care a lot about cost actually.

Okay.

Like, I think we, again, we are in a huge bubble and a lot of our users, especially like, you know, they do care about the cost because like, because people would compare a memory system against the LLM they use. So if the memory system is, you know, $10 for a million tokens and the LLM they use is $2 per million tokens, it will be like, oh, like this does not make sense to use. It's too expensive. I should not use this. And this happens to us. And we want to make sure that like we're giving them a lot of, you know, a lot of satisfaction on like they should be happy about using SuperMemory and it should not be something that, you know, they have to be concerned about. So for us, we have optimized our infrastructure down to it costs us 2 cents For a million tokens, because we have our own model, our own database, our own everything. So we have to price our users also really low, which we do. So yeah.

Okay, cool. That's a really good sort of masterclass in everything sort of memory, state of memory, including a side tangent on evals and all that. What's next for SuperMemory? Like what are sort of the open directions for 2026 that people should think about?

Yeah. So 2026 was the year we realized that The prosumer aspect of super memory is something that people are really excited about. Like this idea of having this one memory that you can connect everywhere, you can interop between providers and stuff like that. And we are continuing to do things like that. Like we're going to launch the plugin with Cursor today, and we already have the Claude, Open Claude, Open Code, all of those. That's good. Next up, we are going to do a launch on voice agents. Partnering with pretty much every voice agent company because of the fact that Soup Memory is so fast at the work it does. So it's really good for voice. Um, that would be amazing. We are doing more on evals. We are building our own eval. We are going deeper into personalization because we think that no one has done a lot of work in personalization yet. And a lot of other, like we are building a new version of our database. So yeah, it's going to be really exciting.

Cool. Well, thanks for jumping on. I think this is a great introduction, at least for Latent Space listeners, to SuperMemory. Obviously, you've been plenty famous on your own with your projects, but I think I already invited you to World's Fair. We'll see you do more talks for that kind of stuff. And I think just generally, I'm starting to see real startups and people taking memory seriously compared to 2, 3 years ago when it was all vector databases.

That was a joke.

I think that's really good. There's more coming. I think LangChain has their own take. I think all the others—

There's LLaMA, LangChain, MemZero, Zep. Yep.

Both LLaMA and Zep have done workshops, so I think it's your turn.

Yep.

Amazing. Looking forward to that. Okay, cool. Good to catch up, and I'll see you online.

Great. Thank you so much.