The Retrieval Loss Problem: Why Your AI Memory System Fails at Scale

You know, the thing that was really interesting is like everyone's trying to engineer context, but we predict it.

Can you tell me more about this prediction?

You need the right context for the right task.

How do you do that?

If you're building a voice, agent speed is really critical. The recommended speed for voice experiences is to respond back to you in 4 to 500 milliseconds. If you want to personalize your voice experience in a RAC pipeline, it's going to take you RAC 5.5 milliseconds. There's been recently like an explosion of like multiple agents working together. And it's really like an exciting opportunity. One of the core problems is when you're stitching together these agents, these agents need shared memory, but also you need to optimize the right set of context that you're giving for each agent. And with hallucination, it becomes a trickling effect.

A lot of people are talking about retrieval accuracy and you're talking about retrieval loss. Can you explain a little bit more about what is that? Hey everyone, welcome back to setai. Today I'm sitting down with Shakat Kabara, CEO of Paper. He's building what he calls as the predictive memory for AI Shoukat, thanks for joining us. Before we dive into your predictive memory, could you tell us a little bit about yourself and what you're building?

Yeah. So I'm Shaukat worked really a lot on machine learning AI at like big tech companies, Microsoft, Apple, Facebook. We're building paper. Think of it as really the memory layer for AI agents. So if you're a developer and you're building an agent, your agent needs to remember like we remember. And that's what we've built. And the unique thing that we do is we do predictive memory basically for agents.

Nice. That's something new. We'll dive into more about what this predictive memory means. But before we go into that, there are a lot of jargons in this space like context engineering, memory systems, maybe retrieval. Can you help me to detect, decode what some of these actually means? I found a really good quote from your blog post and you said something like most AI systems today rely on vector search for retrieval things like rag systems. Right. And you said that it has a critical blind spot that it finds fragments and not context. Can you help me define what is this context?

Yeah, that's a really good question. So when you think of context, you know, the easiest example I give is how we use context as humans. So when we're having a conversation you probably want to have context on who you're interviewing, for example. Or I want to have context on, like, the conversation that I'm having with a person. It could be like personal context. It could be context about the project. If we're working, like in a team, in a company. So context is king, really. Context is, like, so important at work and as leaders leading teams or in organizations. You need to really make sure that your team has the right context so that they're empowered to do business the best work that they can do. So now if you take an example of AI agents, when we're building agents, we need to give agents context. And there's a really popular term there that I really like, which is called context engineering. It actually started prompt engineering because initially when you have an ll, you need a prompt, you need to, like, engineer the prompt so that you can get the right information or context to the agent. But then this really evolved into a context engineering because what we learned is you need to. Normally, when you're stitching multiple tasks together, you need to engineer the context window to make sure it doesn't go beyond, like, the maximum limit, to make sure that you're really optimizing the right context and you're retrieving the right information to help your agent do work. But one of the things that we learned as we went through this journey is that the more data you give your agent to remember, the worse memory gets over time. So this is like a term that we that some coined as, like, context trot. So a lot of times when you try to bring a lot of data and you give that to an agent, like thousands or millions of tokens, it doesn't really work well. And then a lot of times when you try to save a lot of data, like Interact Pipeline. The more data you store and save, the worst retrieval gets over time. We'll talk about this in a little bit, but that's basically one of the core problems that we're trying to tackle.

Does it have to do with this concept of loss in the middle problem? Because how the LLMs works and it prefers.

Yeah, exactly. This is what happens when you bring in millions of, let's say thousands of documents and give it into the context window of Gemini or any of these models. You have this problem where you have a needle in a haystack. It's like you put a needle in the middle and you can't find the needle. With more information, it becomes harder. But I'd say when we work, we're not looking for needles in Information, we're looking for different snippets of information or context that we want to synthesize, bring it from multiple sources and come back with an answer. So actually real world queries are more sophisticated than like the needle in a haystack, like evaluation. But I'd say one of the things that was really interesting is like everyone's trying to engineer context, but we predict it. So rather than trying to engineer the context window, we've built prediction layers or prediction models that help us predict the type of context that you're going to need before you actually do it. And we realized that this helps us overcome the challenges of context drop because with more data, our prediction models get better over time and then we can give you better results.

That's very interesting. I've never seen something like that on the market just yet. Before we go more detail into the predictions, can you define memory for AI? Because we, a lot of people are talking about memory for AI.

The way that I think about it, we talked about context and how you engineer and hopefully create a context in the context window. When I think of memory, I really go back to like our brains and memory that we have because the way you think about it, when we as people, when we see things, when we grow up and we learn, we accumulate a lot of information. A lot of this information is stored in our brains as memory. Some of it is like working set memory, which is basically when you're trying to complete a task, there's a few pieces of context or information that you need, like that we put in your brain that you need to complete the task. Think of a simple to do list. This is like the working set memory. Then you have short term memory, which normally is episodic in nature. Or it's like a form of memory that when you go to sleep and you wake up, your brain tries to predict actually what information you need during the day and tries to optimize for the things that are important, like mirror. And you have long term memory. This is like an experience that you had a while back and you might remember it or you might not remember it. But if you use that as an inspiration, when I think of memory, it goes beyond the context window and it's like every, everything that you put. It could be the chat conversations that you're having with your agent, it could be all of the rag data or your rack pipeline, multimodal video documents, code snippets that you store. It could be shared memory. So it's anything that you store that then you want to Bring into the context window so that your agent has context to do their tasks. So think of memory is like a broader thing that you sort everything in that can help you then optimize and like work with the context window, the generic memories.

How we as humans work is the total storage of everything that happens.

Exactly.

Information we have. However, when we are working on something or making decisions, we don't use all of it. We use a selective part of the memories. We retrieve different parts of the memory that can help the current.

Exactly.

And that memory that you're talking about, what's inside of the context window, that's what's important.

Exactly. You need the right context for the right task. Right.

Exactly. When you think about your product and how you're selling it, do you feel you're selling a memory solution or a contact solution? Do you have a specific way of saying it?

Yeah, Memory solution.

I'm curious, when you explain paper to your users, what makes people immediately say, oh, I get it, yeah.

So I think there are like two use cases, the two type of customers that we talk to very often that get excited about what we're building. One is voice agents. And if you're building a voice agent, speed is really critical. And I experienced that when I was at Apple building like Siri and voice experiences before. Actually, the recommended speed for voice experience is for your agent end to end to respond back to you in 4 to 500 milliseconds. That's normally like the way that humans pause and how fast we speak. That's like the recommended experience. But the problem is, if you want to personalize your voice experience, you really, if you put your data in a Rack pipeline, it's going to take your RAC pipeline 500 milliseconds, then you're going to add a bunch of more to do speech to text and text to speech or even audio. So that's one to two seconds. And that's not like doable.

Not acceptable. Yeah.

Not acceptable. Yeah, it just doesn't work. So with paper, because we predict like the type of context that you need and we put that on device and we cache it for voice agents, it becomes a seamless experience. We can actually retrieve the right context in less than 100 milliseconds. That's going to be the next launch. So we haven't launched that yet, but that's going to be the next major launch that we do is bringing this context and like the predictions that we have on device. So this really unlocks use cases that weren't possible before for Voice. Voice is really like critical and important and people get that. And the other set of customers is B2B. If you're building like an internal, like knowledge agent inside of an enterprise company, that agent needs access to a lot of maybe terabytes of data, like all of the Slack conversations that you have access to all of the documents, like a notion, for example, all of the meetings on Zoom, maybe your code. Imagine giving that Agent360 view of all of the information that that matters to you to help you do your work. That's a lot of information and a lot of solutions today. If you give it all of this information retrieval doesn't work. It like breaks. It's not reliable, it just breaks. Basically ours work really well at scale and with more data it gets better. So those are like the two cases.

That's the aha moment. So basically, in order to have good results, you need to manage the context window really well. And what you do is you reduce the amount of data. You put the best set of information for the context.

Exactly, exactly.

You know, right now on the market, there are a lot of people talking about different ways of engineering context. And you mentioned that engineer context is actually very brittle and it can be very expensive and it can be unsustainable. Can you explain that a little bit more?

Yeah, for sure. There's been recently like an explosion of like multiple agents working together to help you do things. And it's really like an exciting opportunity to increase the throughput or output of agents. Agents to help you do work. One of the core problems is when you're stitching together, these agents need shared memory, but also you need to optimize the right set of context that you're giving for each agent. So imagine you have an orchestrator and you have six or seven agents at the bottom, and each of them are trying to do a task. And let's say they're trying to build software. They want to build like an app that you ask them to build. There are a bunch of challenges today, but one of the core problems that ends up happening, which is more manual, is you need to, when you have all of the memory or all of the data, you want to make sure retrieving the right information and you're then passing the right information to each agent, whether it's shared memories or memories that are only accessible for two specific agents. If you give a lot of data and you don't have a lot of retrieval, doesn't work really well, then you get into a hallucination problem. And with hallucination it becomes a trickling effect. If you have the agent, the main orchestrators hallucinating, then all of the sub agents end up hallucinating and that becomes like a contextual problem. So it's like one of the bigger problems that we see is like how you manage hallucination. And there's like different techniques to help you do that. But I think the first step is getting retrieval working well and reducing hallucinations with that helps with that experience, basically. And then there's like other things that people do to make sure that you're optimizing the type of prompt that you have that you're sending to each agent. Because depending on the type of prompt that you have, it could get your agent to do work really well or not do work well. So when you're engineering context, you're thinking about like the prompting and how you engineer that. You're also thinking about what you're retrieving and reducing hallucinations. And finally you're thinking about like cost because your cost is going to bloat if you pass in a million tokens and send that to each like agent.

The prompting engineering part, we still do prompt engineering. At the end of the day, it is the hard in the.

For sure.

You're thinking of this when you say like scalability of this type of systems. Right. You also created, I remember seeing something called retrieval loss and you were talking about, a lot of people are talking about retrieval accuracy and you're talking about retrieval loss. Can you explain a little bit more.

About what is that exactly? So basically retrieval loss, we built retrieval loss to really help us measure like a few core metrics at scale for memory. So one of them is accuracy. Are we able to retrieve the right relevant context? 2 is speed, can we retrieve the right context fast? And then three is cost. We don't want to shove or add like millions of tokens into the context window because that will impact cost. So if you look at these three core metrics, we put that into the bucket of retrieval loss. And with retrieval loss, the higher it gets, the worse retrieval is, the lower it gets, the better retrieval is. What we discovered is for architectures that are search based, that means that when you put all of the memory into a database, whether it's vector or graph or like a regular database, and then you try to retrieve data, actually retrieval loss ends up getting worse with more memory. So the more memories you add, the worse it gets.

You're talking about if I just purely increasing the context window.

Exactly. So a way to think about it is if you add like 100 million tokens to memory. A billion tokens, 10 billion tokens, the more tokens you're adding and storing in memory if you try to retrieve it, if it's a search based like retrieval system or architecture, it ends up getting worse with time. And actually Google had this really very valuable research paper that talks about the physical limitations of dimensions of your embedding. So for example, if you have an embedding that has a thousand dimen, there's a theoretical physical limit for accuracy for retrieval. So if you end up having like more than a million or a million and a half documents, let's say stored in memory, post that your retrieval starts to become like unusable. So this is like an example of the physical limits that you end up hitting as you add more data to your RAC pipeline. It gets worse over time. One of the core things we discovered which was really interesting is if we treat this as a prediction problem, the more data we get, the better our prediction models, the better then our retrieval gets. What we discovered is actually retrieval loss stays flat or even gets lower with time, which means that with more data we get better at connecting things together and giving you the right context over time.

The traditional approach, it's just top k using ranking re ranking methods, finding what's most relevant. And that's still based on similarities in the vector space. You're saying you're doing some sort of prediction. Can you tell me more about how are you predicting?

Yeah, that's great. Can I share the architecture diagram that we have?

Yeah, yeah, please, I would love to see it.

Yeah. Awesome.

So you're treating it as a machine learning problem, basically.

We are, exactly, yeah.

What's your objective then?

Yeah, the objective is to get the right context at the right time for the user. So the best case is we can predict the context you need before you.

Ask the question, how do you do that? And do you have to label your results such as is this relevant? Is it not relevant? Are you doing some annotation?

So we use different techniques for prediction. It's not only like one thing, and I can give you a few examples to make this a lot more concrete. But like one simple example is if we build a good understanding of the goals for users and then the tasks, we can actually predict the next task. And if we can predict the next task, maybe we can actually do a good job at predicting the context or the questions that you need to ask to help you complete that task. So that's one simple example of how you can think about this. So a lot of the models like that we have, we don't train on like the customer data. But what we look is we look at the basically the questions and the answers and we try to build prediction models that help us get to the right answer quickly. So just to quickly maybe walk you through like the architecture that we have, it's really inspired by how our brain works. We actually have tears. So yeah, it's basically like a brain inspired architecture and predictive memory. So like in the human brain you have like the working set memory which is in the prefrontal cortex. You have the episodic memory, you have long term and then you have neural pathways that actually neural pathways is really interesting because what happens is like when we're having this discussion, your brain is like taking snippets of things that are important and storing it in your brain. But it's not only doing that, it's connecting like neural pathways with this information that you just captured with other things that are relevant that it's going to predict that you're going to need in the future. So we do something like that. And if you think about the architecture, it's pretty simple. We have the application layer where you can have AI agents, tools that need like intelligence or AI embedded inside of them, hopefully in the future robots so that they have memory and we can share their memory securely with them. But that's more like the application layer. And then we have SDKs that are on device, we have an MCP server, we actually also have a CLI tool, we have APIs and we have this durable execution flow that we can build on Device and our SDKs. If you're building an agent and you use our SDKs on device, it has an intelligence layer. And this is where we have a lot of the prediction models that we have. What happens in the background is before you start work, we actually try to predict the type of questions that you need and the working set memory. These could be goals that we think you care about, tasks that we think you're going to do next, but also important context and memories that come from your like long term memory that we think are going to be important that you've used before or like are things that you, we think you're going to need. And this becomes like really fast. So if you come and ask a question like your agent will already know like the goals and tasks that you're going to work on and then can quickly get access to key memories that are important to help you complete a task. Episodic memory is a little bit more interesting. It expands on this notion, but it doesn't look at a filter of goals or like predicted goals. Think of it as more of a cache or citation cache. So this is where if you've used context or a memory before and we've the agent cited it in an answer and you frequently used it, maybe you're going to need it later. So we try to build this cache tier or prediction tier of key memories that we think you're going to need based on things that you've done in the past.

So you're predicting the cache as well?

Exactly. It's like a cache based prediction, but it uses not only cache hits but also more important signals like user feedback. If you came back and you said this is a good answer or a bad answer or even citation. So if the agent came and picked one of the memories and cited it in an answer, the agent can automatically give us feedback and say, hey, we cited this memory. So we use that feedback loop to then increase like the citation hint, if that makes sense.

Do you take feedback from the user in a live conversation when we are chatting? When you are talking, you are sparkling my memories all the time. You reminded me of building machine learning models, things like that. When I'm responding to you, you are also getting feedback, right?

Exactly, exactly.

You tracking these conversation turns and getting feedbacks on the fly?

Yeah, exactly. We don't build models on the fly, but it's a good idea. But we do get that feedback, right? Because when you come back and send your message back, as we're talking, we're able to capture that. And then if that's stored in memory, we get that feedback and we have other signals also. Like if you give a thumbs up or imagine like other signals. If you're building an agent, you might want to get feedback from your users. We give you the ability as a developer to give us feedback in a feedback API as well. So we do get that feedback and that helps us improve the prediction models that we have as well.

Sounds like you're predicting my intent. Meaning that after we talk about one topic on a few turns and you're probably going to know what I'm about to ask next. You probably know exactly, right? Do you do that?

We do that as well. Yeah, we do that for the purpose of trying to get the right context to help you answer the question or complete a task.

Because every time when you're answering my question, you're retrieving different memories, different data in your brain to answer my Next question.

Exactly.

Awesome.

Yeah, I really love how you're taking the brain and inspiring me to even a little bit better product. I think we want more people to use this brain inspired architecture, predictive architecture, because I think this is really how we're going to be able to scale with memory and with AI. The last point that I just wanted to mention is we also have a graph. We are using Neo4J and what's really cool is we give developers the ability to pass in your own ontology or custom ontology. We can then when we add the memory, we add it to these tiers. But then we also build the graph ontology using the ontology that you gave us. Then when a query comes back, if it doesn't hit any of these tiers, we do an agent, a graph search and then can give you back, for example, for example, list of tasks that were in a meeting or if one example question is show me all of the tasks that my customers or feedback that my customers gave on key problems across my 100 customer conversations. You can do that with one shot, with a graph. You build the graph and you can get all of the insights or the problems across all customers. So it's pretty cool. And then these are like the tiers and then we use like a storage backend. So we use MongoDB, Quadrant and Neo4J and that's like the core storage that we have. Quadrant we think is the fastest and we love it. It helps us be very fast. And Neo4J is really good because it helps us build like the knowledge graph.

So Neo4j is for more relationship and connection building. The graph architecture has a lot of its own merit. And the developer can choose to give you my own ontology, right? My own graph.

Exactly, yeah.

Do you build graph like automatically?

Maybe for some of us we automatically. Yeah, exactly. As a developer, it's really easy. You just give us the anthology, you're like schema. And then when you set up the schema, when you add the memory, we automatically add it to all the tiers and we automatically take the unstructured context. Let's say if it was a meeting recording, we automatically structure it based on the ontology that you gave us and save it to the graph. And then when you want to retrieve, you can just use natural language and we can find that context in these tiers. And if it hits the graph, we use the graph search to go and get the right context and get you the result.

Basically, how's your agent make the decision to go which database to retrieve? The information, either graph or the vector database.

Yeah. So the first step is tiers. So we have a model that predicts tiers and then basically if it's the vector one it hits, for example, quadrant. And then we also do a lot of chunking and we do query expansion and we use MongoDB for that. And Mongo is great for that. And then we also, when we store data in the graph, like for example, in a meeting, you might have customer, you might have people, you might have like tasks, you might have insights from a customer. As an example, we actually take all of these, store your ontology in the graph, but we also store it as classes in MongoDB, which is pretty cool. And then basically if it hits the agentic graph, it can go to the graph and get the right context.

Do you somehow take user feedback as a signal to guide you which route to go?

Exactly. Exactly.

So you're doing that as well. So you're having a lot of prediction in your system.

We do. And if you notice here we have like eight prediction models. So we're doing like a bunch of things to stitch this together.

Yeah, that's crazy. But it makes sense. You mentioned about when you wake up in the morning, you might be predicting what you're going to do. And I realized maybe I'm unconscious doing that. Just, I'm just not aware.

Yeah, I think it's very funny. Today I was reading this research paper on like neuroscience and one of the things that was really interesting that I didn't realize is like when we sleep and we dream, our brain is actually creating different, like scenario simulations. I mean depending there's like different types of dreams. But there are this case where like your brain is like putting future pathways in dreams and trying to predict the future. And sometimes it does a good job at predicting your future, but then it uses that dream state to then help optimize what information is needed when you wake up. So there's some research there that I think is really fascinating.

It is fascinating. Maybe part of dreaming is also a realization, a representation of the graph in our brain for all the past memories and maybe the memory of the day. And they were just representing it in a vivid dream.

Yeah, in a vivid state. Yeah.

Well, there are so many things in our brain, how it works, that can guide how we design our architecture. I think what you're doing is really interesting. What do you think of the. How do we balance the short term memories and the long term memories? Like retrieval or you think prediction can solve all of it?

Yeah, I think Prediction can solve most of it. If you notice we have like edge clouds. So edge is tier 0 and tier 1 and this is where we have like a smaller working set memory and then episodic memory. So think of these being like short term and then long term. Memory is everything that you put in the cloud, which could be like vector or graph based. So if we can predict the tiers, we can predict the information in the tiers, then we can try over time with more data, we can improve or optimize what we are putting in these.

Do you think this is how our brains are functioning behind the scenes?

I think some of it is inspired by that, yeah. For sure. We need to build more knowledge about ourselves. I think we need to learn more about how we work. So I think, I'm sure there's like more ways to improve this. But it's really inspired by how we think and how our brain works.

One side of me think that we are efficient in terms of memorizing and forgetting and some part of me, I don't know if we're actually that efficient. Right. When I think of long term memory, I think there is a component that the modern transformer architecture is not thinking about is emotion. Right. Because a lot of the times what triggers my long term memory is emotion. Like a smell of something and a familiarity, like deja vu brings me some old memories. And that's not something that I can imagine machine learning or these things these design architecture can handle. What do you think?

Yeah, it's a good question. Yeah, I'm not sure. I don't know. I think emotions are really hard. You can even in the latest like video models, like they look really good but you still don't feel the emotion coming from the person that's talking. You can kind of feel it and it's really hard. It's not something that you touch, something that you feel. So it's a little bit different. But I do think like the point that you made is interesting and it is like what, what triggers really like neural pathways to connect together to get you to remember something. Because when you're putting tons of things like for us emotion becomes something that's important because our body believes that it's like something important to remember, whether it's something that happened in the past, then it was like emotional, it ends up being triggered really easily. But when we're building agents, what are the right triggers? What are the right neural pathways that you want to build connections with? What's the dopamine version of basically the connections here? If we can predict things that are important for you in the future, maybe that can act like a dopamine hit and can help you build things together or connect things together to then make it easy to retrieve the right context.

I feel the way you are thinking is philosophically one step ahead of. If you think about the attention type of models, what drive attention is based on the end result of whether it's translation or whatever it is you're building connection between A to B and then because you already have the end result, that's how we build the large language models.

Right, Right.

In a sense we are still focusing on goal oriented building. I think emotion is another level. I'm looking forward to new architectures that can actually factor in how that triggers the neural pathway are forming and how we forget and remember things and then make it more. I think maybe that's when the true consciousness is happening in AI. What do you think?

Yeah, it's very philosophical. You know, I like this discussion. It's getting me thinking about like how we can even further add more value into this architecture. I really like the emotion piece and I can imagine, for example in the future when we're working with robots, hopefully we can work with robots safely and we can safely share with consent our memories with them so we can collaborate with them. But I can imagine an experience where if you have a really great experience with a robot and their emotion gets triggered positively that in the future when you come and they see you because you triggered an emotion for them, it's easier for them to remember you versus someone that they just saw randomly walking. So I think there's values in emotion for sure.

I like that once we can quantify and measure that, the way we are realizing it is not so how we are made. I want to ask a little bit about your market positioning and your go to market. My understanding is that your target audience is developers. Target PMs as well. You mentioned B2B.

Yeah, it's a good question. Like we're targeting developers and vibe coders who are building agents. So I think in my mind there are two buckets of people. One is developer and you're building like let's say the next cursor or you're working in cursor, you can come talk to us. It would be like one example if you're building a voice agent, so you're a developer inside of your coding. Rather than you going and trying to stitch together and build a Rack pipeline, you can easily with one line of code use us. We are also announcing it here that we're going to go open source. So excited to share the architecture diagram with you guys. And we're planning to open source our core retrieval. So that's one bucket. I think the other piece that's really interesting is like we're seeing a lot of users that are also like vibe building apps. You don't really need to be a developer, you can be a pm and we're seeing a lot of generalists basically. So you can do anything right now with AI. So if you want to build an agent, even if you don't know how to code, you should use us. Because we have an MCP server, you can actually easily, if you create your agent, you can easily use our MCP server and bring ad context and take it everywhere. So that's like another good example. If you're using any of these tools to do a pipe coding, you can use us. And then for me personally, I'm using Paper as an end user. So I have like over 22,000 memories. All of my customer conversations are recorded. All of our documents are code.

Can you explain to, let's say whoever have heard about this concept about MCP but don't know exactly what it is? Can you explain in a layman term?

That's a really good question. MCP stands for model Context Protocol. And the simple way that I like to describe it is it helps your. It helps you connect with apps. So if you have some data in like Tasks and Linear and you want to get that context, you can maybe use the Linear MCP server to help you connect with these apps. So a simpler way to think about it is it's like a technology or a framework that helps you connect with other apps and get context from these apps.

So basically getting data or you can pass your data to some other apps.

Yeah, exactly. You can do that or you can complete actions. So one example is if you want your agent to mark a task done in your task management tool, you can connect an MCP server and then your agent will learn how to add the task in your task management tool. So it can get data, but can also do things in the app like on your behalf with your consent.

Because you said you're selling memory solutions when you're marketing your product. Do you feel this concept of memory is still new to most of your potential users? Is it still new idea?

Yeah, I think it's a new concept, but it's easy for people to understand. Like for developers who are building agents, they quickly get it because they recognize their agent. Agents need to Remember and they need to get the right context. And if you're an end user and if you've used Claude or any of ChatGPT, any of the tools, it's pretty clear that you need to get your agent to remember your like AI to remember. Especially at the beginning, like the first year, that was like a big pain point. So I think it's easy to explain to people easier than like context engineering or some of these other terms. But I do like the context engineering term because it's a good term for the problem that is focus on which is like helping you get the right context.

Right. When you're talking to your user, do you feel you're focusing more on explaining what memory is or you're trying to explain why people.

It's more why? Why paper? Yeah, it's more like I get what memory is. Okay, tell me why this is better, for example, or what are you doing that's different.

And you have some competitors. I saw the Llama Index and LangChain, they have lang graphics. That sounds similar to some of your features.

Yeah, actually when you think of like Llama Index and Langgraph, we see them more as partners because they're like an orchestration framework. We can actually plug into any of these orchestration frameworks and be like the state or the memory layer. They also have their own memory solutions as well. Like we can plug into their systems. I think there's a few people that are focused on memory. I think Mem0 is one, Zepp is another one. Those are like memory focused platforms basically.

So you're more on par with those guys.

Exactly. Yeah. It's the same way to think about it. Architecture is different because we're building predictive memory. But yeah, we're in the same space.

Let's say today I want to build an AI application. What do you think should be in my AI stack? Obviously I could include you, but what do you think of the minimum viable version of it?

I'd say three things. One is you need some orchestration layer and I think one key product that's pretty hot right now I think is temporal. So I'd recommend like that you look into temporal and see if it can help you drive durable execution. Because that's really important. What it does is it helps you guarantee that when you're orchestrating and giving a task out to a sub agent, that sub agent always does the work that you're asking them to do. There's failures, there's like state where something might break and you want to come back and Retrieve that state. So I think one thing to think about is like when you're orchestrating, how do you make sure that you have durable execution? Temporal is one example, and we can plug it into temporal. So you could have paper be the memory layer and temporal be the layer that guarantees your task execution. The second piece, beyond orchestration, is you need to choose the LLM layer. So are you using ChatGPT, Cloud Anthropic, and you want to build an architecture in a way that gives you the ability to use these differently. You might have different use cases that are optimized for different LLMs. And sometimes you want to switch and swap these. So you want to make it really easily swappable. And then you have the memory layer and the memory layer. You want to make sure that layer is continuous. So regardless of how many agents you spin up or different LLMs you choose, you want to make sure that your memory is continuous and you can use the same context and pass it to any of the other LLMs or agents. So I'd say these would be the three core layers, orchestration the LLMs that you're using and then basically the memory.

Layer, which includes storage and database. What about database?

Yeah, I bucket this really under memory because it includes storage and database, basically, yeah.

Okay, cool. And when you are pitching how do you make people remember paper?

I showed them a demo. Should I show you a demo?

Why not?

Yeah. Let's see if this is going to work. Let's see. So I have a few demos actually that we can go walk through. We've built a few apps. Actually one of the apps that we built is this chat app and it's open source. So if you're building, if you want to get a feel for how memory works, you can try this. You can also like easily put this in vercel and with one button and you can easily give it a try and see how memory works. But over here you have a typical chat experience. You can add context. These are previous artifacts that you've created that you can add as context. But then you have memory enabled. You can enable or disable search and you can pick any of the LLM models to try. I'm just using chat and one of the cool things that we did recently with this and I want to give it a try here is like audio. One of the problems with audio that we talked about previously is it takes a lot of time to get or retrieve context. So I'm going to try this right now with you and see if this is going to work. Can you. I'm assuming, Angelina, that you can hear. But what I'm interested in is help me understand what paper's predictive memory is and why it can scale unlike other architectures. Okay, I'm just going to pause it here, but. And notice how fast that was. And smooth. We automatically bring context and it has information. I wanted to give you maybe another quick demo here. So this is. We talked about retrieval loss. I'm going to use Opus right now for this example, and I'm just going to ask a question of. Show me, like, the retrieval loss. And what it's doing now is it's doing some reasoning to go through and do searches. And it will come back with, in a few seconds, some more information from memory. Here we go. So this came back and it's showing, like, some more information. So we'll go through this, wait for it to finish, and then I can quickly go through this.

I have another question about the graph part of the tiers. If I don't have my own ontology, you automatically generate that.

Yeah, we do.

Is something like the user turns on and off.

Yeah, I think the user can. Basically, when you set up, like the. When you use the SDK and when you set it up, what happens is you can set the schema that you want for the tenant that you want. And then when you do that, we always use that schema when we build the graph for you or when we build the ontology.

Makes sense.

Yeah. Yeah. So anyways, I'm not going to go through all of this, but it gives you a kind of an example of. It's bringing all of the context, putting all of the sources.

Memory.

What did you say?

So this looks like the episodic memory that you're. The memory.

Yeah, exactly. Yeah. I have so many things here. So it grabs a lot of it and it puts it here in context. Let me try to see how I can go back to the screen here. Okay. Yeah. I can maybe stop sharing.

No, it's okay.

Yeah.

The one you just showed me is that retrieval from your 20,000. Like, exactly. That's why it looks so many. Yeah.

Cool.

Do you think you're in a very crowded space?

I'd say that it depends how you define the space. If you define the space as AI, it's super crowded. But if you define the space as predictive memory, we're the only ones there. But I think. Yeah.

Which one is your space? And I think it's either.

Yeah, we are in the AI space, but we are building this predictive memory. And we're the only ones there. But obviously there's other solutions. You can go and you can build your own RAC pipeline. You can go and use a solution out there. So there's like multiple choices for a developer for sure.

What are other choices for developers?

Yeah, I say that there are like three choices. One is you can build your own Rack pipeline as a developer. Like you can choose to go and build your own vector embedding, build your own graph, try to build predictive memory. You think this is the right architecture? So that's one option. You can build it yourself. Obviously if it's so it's like one and then two is you can basically use like third party solution. So we would be an example would be like use van0zap or paper or any of the other solutions. There's cogni as well and a bunch of others there. So those are like the two options is you can either build it yourself or you can choose to like use an existing memory layer or memory solutions.

What's the third one?

I think the third one is you can choose not to do either. You can pick up Gemini and you can say, okay, I'm not going to worry about storing anything. I'm just going to put everything into the context window. And I'd say it depends on the use case. If you're building something that has a very small amount of data and you don't need to have personalization for the user over time. Think about if it's like multiple paragraphs or whatever, you can just put it in the context window and that works. You don't need to build like a sophisticated memory rack solution. But if you're building a voice agent and you need to personalize your voice agent, you need to do something there. Or if you're building like a B2B agent that needs a lot of information from within the company, then you need like a memory layer there.

So hear me out. I have a sense of what is your true competitor. If you say your predictive memory, then your new category then because you're the only one. If your AI is just too way too broad. So what about the positioning is more about better rack systems? Like you're competing with people who want to build better racks. Does that sound about right?

Yeah, I think that sounds right. I wouldn't say we're competing with people building better Rack systems, but we're like if you want to build the Rack system, you can use us and you can build the best Rack system ever.

Okay, okay, that's the tagline Right. That's the pain point, I think.

Exactly.

Okay. I really enjoyed a conversation with you, Shaka. I learned a lot from you. And I think I really like the idea of using predictive modeling and machine learning in these architecture. I think it also makes sense. And I realized when you were talking, I realized I am always kind of predicting. I'm not predicting, I'm anticipating. I could anticipate what you're going to say.

Makes a lot of sense. I also learned a lot from this. So I really appreciated this discussion and it was really great talking to you.

Yeah. Thank you so much for joining me today. Sounds good. See you next time.