95% Prompt Cache Hit Rate: How LLM Cost Reduction Actually Works in Production

This is my, you know, profession. Software engineering. It's completely different than it ever was before. I mean, we are in a whole new world now. Large language models hallucinate, right? They make things up. They— I think I, I think they said this and this is what I think I had for breakfast 3 weeks ago. The problem to be solved now with agents, one of the interesting problems is short-term memory, working set memory. It's my top engineer who uses AI as sort of primary method of coding. He's spending as much on tokens as his entire salary. So we are burning these data centers up. Now that I have this technology on my hands, I can just tell it to go do it. I'll show up tonight, it'll be done.

Are we in an AI bubble?

I wouldn't bet on any apps at this time, but infrastructure is much safer.

Hello and welcome to another episode of Data Innovators and Builders. Today I am with Rowan Trollope, CEO of Redis. Rowan, very nice to have you here.

Yeah, thanks, Saket. Great to be here.

Awesome. So Rowan, maybe give us a little bit of your background coming into your career through Symantec and Cisco and Five9 and now at Redis.

Absolutely. I've been building software since I was, you know, professionally and paid since I was 18, but going even before then when I was a teenager, you know, doing, having for fun projects. But, but since I was 18, I got hired at a startup and have worked in the industry ever since then. I started, did about 20 years in originally consumer software and then the beginnings of the cybersecurity industry leading up to the dot-com, the dot-com boom slash bust. And then have worked as, you know, head of products and engineering at Cisco, running their software company, software assets, primarily was on the collaboration technology group that managed all of their, you know, sort of communications infrastructure and technologies. And that was exciting. And then I was a CEO of a company called Five9 for about 5 years. And now I've been at Redis, back to my roots as a developer. For 3 years.

Awesome. That's an incredible experience you have there. And I have personally been a huge fan of Redis. I initially known it as a memory store, but you have come in and set a very strategic direction for the company as a memory layer for AI and a much broader platform. So tell us a little bit about this evolution of Redis.

Yeah, well, so about 3 or 4 months after I joined, ChatGPT, you know, kind of 1.0 came out and I had been really a big proponent of AI for the 5 years prior to that. So now it's been a total of about sort of 8, 8+ years since the Transformer paper essentially came out right around then. And when I took the job at Redis, it struck me that we were going to enter into this new era of, you know, AI, really GenAI applications really becoming the main thing, whereas the last 10 years has been cloud mobile. And if you look at the last 10 years where Redis really got its start as a, you know, remote Redis stands for Remote Dictionary Server. That's what Redis stands for. And it became a core part of the application infrastructure on the internet in terms of helping, you know, almost every business you could possibly imagine build out their infrastructure and scale it to internet scale. Redis is a core part of that infrastructure, and we've been very lucky to be sort of a very successful open source project and adopted by lots of developers. So that project, you know, that, that was a 10-year cycle, you can call it. And I think we're now heading into a new cycle, which seems obvious to me, uh, around building agents. And those agents are going to need to have a data store, both to serve data to them and also for places where they can store, serve and store information. And so we've been very focused on that opportunity. Just what does it mean to be a data store for agents and, uh, how does that work in the, in the enterprise context? And what are the new problems to be solved? Like memory, you mentioned is a new emerging problem to be solved since you have essentially a, tremendous amount of exhaust coming off of these applications and/or these agents around unstructured data and turning that into sensible, useful information later. So, so solving agent memory. And I guess the big aha moment for me that set us down this path was when really looking at an LLM, if you, if you look underneath the covers or I guess by way of analogy, an LLM is very similar to a human's long-term memory. Right? In the sense that, you know, it's read a lot of things, you've fed it a lot of information and images and text and so on, and it vaguely recalls much of that information, and then it can formulate it into text, you know, sort of answers back to you about your memory, which is very similar in the way that our own brains work in our long-term memory. When we ask questions of our memory, do I, you know, well, gosh, did I ever see that movie? And you start to get some ideas, right? Or what did I have for breakfast last week on Wednesday? But the other part of our brain that's not part of a large language model is the short-term memory and the working set memory. It's what have I currently loaded and what am I currently operating on, right? And that's the kind of stuff that before this interview, you know, you maybe we had a little conversation, you took some notes, that's all in your working set memory. You don't have to, you know, it's, it's fast and it's highly accurate and very similar to a human's long-term memory. Large language models hallucinate, right? They make things up. They— I think, I think they said this, and this is what I think I had for breakfast 3 weeks ago. So that's kind of an interesting analogy. And the problem to be solved now with agents, one of the interesting problems is short-term memory, working set memory, you know, semantic memories, really the different ways of remembering things. And then also, how do you load things from the from different data sources, not just the large language model. Like the other thing about language models is they're trained on the open internet. They don't have your enterprise data. So how do you get your enterprise data into the short-term working set memory, which in the case of an LLM is the context window essentially? How do you get all that stuff preloaded and do it in the most efficient and effective way? A problem that I think you know well.

Yeah, yeah, I know that's a problem that I know very well. And the way I think about this to some extent is that If you were to go to ChatGPT and ask a question and it doesn't remember who you are and what you last talked about, your answer, so the interaction wouldn't be as interesting, right? Isn't that what we are basically talking about when we talk about memory? It is, yeah.

And I think if you look, clearly Anthropic and OpenAI have done some work to do context window compression and pulling in the last 24 hours of chat logs and so on, but that's not really enough to really know a user. We're starting to see products emerge like ClaudeBot. This happened last week, um, you know, around, um, you know, that, that are sort of natively imbued with long-term, the ability to store memories and to record those over time. Now that is using a fairly straightforward, you know, storing MD files on the local file system. But we're also seeing a tremendous amount of, you know, different approaches and opinions about how, how these memories should be extracted, stored, and then retrieved. Later on down the road. So that's one interesting problem set. We at Redis have taken an opinion. We've built out an open source project called Agent Memory Server. That's at GitHub, uh, slash, you know, slash Redis slash Agent Memory Server. And that's our version of what a memory system should be. And we can go into the details of that if you like, but that is a new emerging problem. Separate problem is, well, you go and talk to the agent and it needs to get data that's not in the large language model. Maybe it's your company's travel policy or, you know, what grade are you? You know, I, you know, or what's the weather in Paris? Like all these kind of mod— like, you know, other information that maybe comes from an internal dataset. You need to be able to pull those things up in real time. So it's— so that's the other problem that we're solving.

Yeah, no, absolutely. I think getting that data in real time, getting the meaning with that data, having good quality of the data, I think all of that comes into picture. I think one specific aspect you talked about is like travel policy, let's say. Now these are things in form of documents, and I think one of the capabilities that Redis supports is that of a vector database, correct? I'm seeing the pace of evolution be so fast, so I want up your perspective a little bit on this. So I would say 2 years back, RAG was the hardest thing, vector databases were very hard, and now a lot of providers of vector database solutions still a very critical piece of technology. How do you see, you know, some of these technologies evolving in the coming years? You know, yeah, the pretty fast pace, right?

Yeah, I think RAG for— you're absolutely right. So, you know, the first couple of years since ChatGPT, I think, I think most enterprise developers or business developers anyways went off and tried to build chatbots because that was the obvious thing to do is like, well, let me, You know, we have an IT help desk in the, in the company. Let's create a chatbot that I can talk to for our employees, and that'll make an experience better. And there's been companies that have sort of sprung up around that, this category. So I think in that world, the chatbot modality needs, you know, to be imbued with enterprise knowledge. You know, if I'm asking, you know, how many days of PTO do I have left? Well, it has to get that information from somewhere. And, and so RAG was a convenient way of doing that, you know, okay, let's store the chunked PDF of the PTO policy for your company. And now when you ask a question to the chatbot, we recognize that and we, you know, it's a maybe it's an HR chatbot or something. And so we inject the necessary context in that RAG workflow that worked fairly well to get enterprise data into the context window. And so we were being used for that. What we've seen over the last year or so is a shift towards agentic workflows. I get asked the question a lot just to define it. So maybe start at the high level. What is an agentic workflow or what is an agentic application? Well, and it's not very complicated, you know, it's essentially a process that runs, that has a large language model as part of it and which has the ability to run in an independent and automated way for a long period of time. Doing some sort of work, right? And so this is— I don't know how anyone else would describe agentic, but that's how I would describe it. And so it's a process that's running that has— that uses a language model for reasoning and can be given tasks that it can then go execute. And from one angle, it kind of looks like the way a human being would take on a job, right? You, you give them some information, you provide them with some company data and training, You give them, and then you say, this is your job, and you give them a job description, and then you say, here's a task for you to go solve, please go solve it. And essentially with the information and training you've provided that entity, uh, it can go do the work. That's how a human being works, you know, a junior employee or an intern. That's how these agents are going to work. And they're starting to be built out now. And there's a new set of problems that come along with those agentic applications that are very, very interesting.

Yeah, I think we are seeing that. I would say last year, 2025, was a lot about getting these from like a prototype to, you know, to production stage. And I think we are seeing a lot of improvement on that front. So maybe if you can share a little bit about that, like where are those challenges on going, taking those agents to be something you can actually delegate to give them a job description, let them work on it or yeah.

Well, I think, you know, in the enterprise anyways, agents are very much a matter of being in POCs at this time.

Yeah.

Now the world's changing quickly. We just recently saw the launch of Claude Bot, which has been renamed to OpenClaw. And that's a great example of an agentic application or an agent-based application that developers have now. So it's kind of gone viral in the development community to demonstrate that agents are really feasible at this time. I think before that, if you looked at the landscape, say, you know, before 2 weeks ago, you would say, well, there's a couple of examples, maybe a handful of examples of agentic applications at scale. And those would be the coding products. Codex from OpenAI, Anthropic Claude Code, Cursor, perhaps, you know, their agent mode, uh, you know, Augment, a coding tool, has, you know, their thing called Augie. So they all built these agentic workflows and then looked at, you know, added on MCP to get data in and out, blah, blah, blah, blah, blah. So that has been the area where there's been actually real progress. And now what we're starting to see is, okay, could we actually build agentic applications outside of the coding workflows to automate other types of jobs? And I think what has so many pundits and observers, you know, fairly excited excited/nervous, to be honest, is what we've seen, what it's going to do to the world of developers. I mean, this is my profession, software engineering. It's completely different than it ever was before. I mean, we are in a whole new world now, and that happened very quickly. We— I envision you're going to see Claude Code-like agents emerge in individual call it verticals or specializations that are going to completely transform many of the jobs that we do today. And that sounds somewhat scary on the one hand, and it certainly is scary. But I'll tell you, as a, as an engineer using these tools to write code now, you know, it's extremely exciting and it's much more fun to write code when you're given one of these tools one of these, you know, an agentic, you know, kind of coding tool. I use Claude Code. It's much more fun because you can do way more. You know, the interesting part of software development— I've been doing this for my whole life— was always you have an idea, you can do a bunch of stuff, and then you can materialize and see your idea come to life. Now, the bunch of stuff that you had to do involved, you know, esoteric non-English languages and programming and C and, compiler errors and syntax problems and compiling and all this other crud. But none of that was particularly interesting. What was interesting was you've got an idea and you can create some outcome. And now what's happened is all that stuff in the middle just went from taking, you know, what might've taken you 6 months now takes you 6 hours. I mean, it is a remarkable transformation and will be a remarkable transformation of this job. Now that hasn't rolled out to all the engineers in the world yet. You know, a lot of people are just trying it. But if you look at how engineers are using these tools that are sort of starting off with AI from the beginning, it's a dramatic shift in the workflow and a dramatic shift in how much you can get done.

Yeah, I mean, as an engineer myself, I 100% agree with you. Like when I was learning to code many, many years ago, I think I was building a game and much of it was to just to see how it works. And yes, there's a lot of time that goes in before it's ready to try. And every idea you come up with again takes many hours or days to get there. Right. And I was listening to the interview from the creator of ClaudeBot, I think now OpenCloud Multibot. He was saying that how addictive it is because it's like one more prompt. It's like, I want to try one more prompt and see if it gets me to where I'm looking to get to. And it's certainly like that. It's extremely— I would say, yes, software engineering has completely changed. Completely changed in the past year or so. But I think it's also become a very developer-centric world because a person with an idea, maybe it's a product manager or a developer, we can see that overnight or weekend people are building out stuff that just changes like what is the baseline, right? So, you know, how— I mean, you've been a developer yourself, right? I mean, how do you see that? Is this a phase that we are in while technology is early, or is this the way of things going forward?

Well, I think it's the way of things going forward. And in fact, you know, um, a couple, couple things. I was just recently in Israel last week, uh, visiting our engineering team. We have a big engineering team in Israel, a lot of really smart people, some of the smartest people in the world working on Redis, on the core of Redis. Um, and we were talking about how AI has changed. Our company is really leaning in with AI coding tools, um, and then expanding to the rest of the org. And I was talking to some of the engineers about just how much has changed and what's changed. And a few things really stood out to me. One, and this is kind of from my own usage of the tools as well, uh, it really eliminates or it brings together, uh, some different parts of the organization that have been sort of separate specialties. So product management and engineering and UI design, just take those three, for example, kind of the three jobs you would need to build a product. You need to first Start with product management. What is it? What problem are we solving? How are we going to solve it? What is that problem set? What is the outcome we want? Developer, how do I do it? How do I actually go write the code? UI and UX, what are the interactions? How do we interact? How do we display this to the customer? Okay, so those three functions in the— call it in the, in the development team, you know, and maybe DevOps would be like a fourth one, but just start with those first three. What I see happening is those three specializations really blurring and coming together. I've been talking about this. In fact, I heard Satya Nadella talk about this last week, and I think he said at LinkedIn they've moved in this direction towards creating product builders. So a new role that's not a developer or a product manager or a UX designer, but it's a product builder and it's a more generalist kind of role. So I think that's one big shift. Now, that's going to be quite— that, like, if you look at organizations and you can, you can— in my teams, for example, I go and talk I talk to my various organizations and sometimes some of our most successful engineers, those, those so-called 10x engineers. One example is Salvatore Sanfilippo, the author of Redis. Salvatore is not a specialist coder. He is an incredible coder. Okay. He's one of the best coders I've probably is the best coder I've ever worked with. Uh, but he's also a product manager. He defines the requirements. He works with the community, he works with his users, in this case developers, to understand, hey, what would you like? What would help you do your job? So Salvatore is already one of these product builders, right? He understands his customer, uh, his user, the, the Redis developer. He understands how to write code and he also does UX design, right? In our case, because we're an SDK, the UX is the docs. Salvatore has always written his own documentation. So he's a good example of someone who has always been a product builder and was for a long time considered to be, and is considered to be, you know, one of the, one of the most successful and, you know, prolific engineers, uh, you know, in our industry. So that's a model for, I think, how things are going to play out. And if you look at the rest of the teams, like the kind of classic way that big software companies have been built or startups, we tended to go more in the world of specialization. Engineers that, hey, I don't care what we're building. This is like a— and this is just like she used to piss me off. Like, when I would hear people talk about engineering as if it was a factory. So specs go in, you know, somebody writes specs, they hand it to a bunch of developers who are kind of the factory workers who kind of build the thing and kick it out the other end. And then somebody tests it and, you know, sells it to the customer. And that always struck me as the wrong analogy for our industry. I just didn't— for some reason, intuitively didn't like that. It, and it was very much all about specialization. My model of the world has always been more of the craftsman world where a craftsman is a product builder, right? If you could take a craftsman who's going to make a chair like this and they're a woodworker, they don't have somebody else designing the chair. A craftsman designs the product. They understand the use case, understand the user. They, they're artistic, they're creative, they come up with it, then they build it. Right? And maybe they pick the color, like they do all of that. And I say, I think we're heading— what AI enables is a move back towards craftsmanship in this industry. But it's like superpowered craftsmanship because now you've got, you know, a complete studio of automation tools that can do everything for you from idea to implementation. I'll give you one example. This happened this morning. I was sitting with my CTO, Benjamin. and we were talking about how to make it easier for a problem, which was like, how do we make it easier for ClaudeBot in this case, or OpenClaw, to store its memories, and how do we actually implement it in Redis? And we had, and I had an idea over the weekend, which was, well, what if we had a CLI that translated POSIX file commands into Redis commands? Okay. And then you could just throw that CLI with a, with a, a skill file at ClaudeBot or OpenClaw and say, hey, I want you to store your stuff in Redis now. I don't want you putting stuff in the file, local file system anymore. I think is a bad idea in general. And so we talked about it and in the middle of the conversation I said, wait a sec, why don't we just go spec this out while we're talking? And which we did. Well, here's how it would work. So then we created a prompt. I then used Claude code to create the PRD, uh, looked over the PRD, looked like a good idea. The, you know, in principle, uh, and then basically, uh, tweaked a little bit on the PRD. And this, by the way, this all happened in about 1 hour.

Yeah.

During my one-on-one with Benjamin. And I said, Claude Code, go implement this PRD. Now it's still working on it.

Yeah.

But it'll be done soon. And that means that tonight I will have that product that Benjamin and I batted around in terms of an idea early this morning. In my hot little hands for me to go test with my agent. That is a completely new world. That project may not have been very complicated, but the, the barrier, like, is it important enough that I would actually go do it? Maybe, maybe not. You know, it's one of those like fleeting ideas that you have as a CEO or CTO. Like, I think it would be cool if I had this, but I'm not really sure. You know, it's if you're not really convicted, it's it doesn't, it's not worth the energy to have a developer or for you to spend a day or two writing something. Now that I have this technology on my hands, I can just tell it to go do it. I'll show up tonight, it'll be done, right? My agent's running in what's called a Ralph Wiggum loop. So it's, you know, taking step by step by step. The thing will be done. I'll be able to try it. If it makes sense, I'll, maybe I'll throw it on GitHub. I'll promote it, whatever. Maybe I won't. So what a dramatic change. Now, you can just imagine what this means for every other job function when this technology kind of comes from those functions. And I'd say one last thing on that front, which is the next logical question is like, well, where is it likely to go next? And if you think about what made this so successful from an engineering perspective, it's that the results are provable, right? We can run a lint checker or a type checker, you know, or a compiler against some code. We know if it compiles or we know if it's got lint errors. So there is a provable nature to the world of development. So if we look at all the other functions where the outcome is provable, those are likely the next pins to fall, if you will, because that's where you'll be able to really train these models in an effective way.

Yeah, and I think that's a really good framework because what is provable as outcome is kind of protecting you from, you know, some of the hallucinations of an agent or, you know, let's say it's a business process like you know, managing a procurement or, you know, making a payment, you may not have as much provability there. And that's where the challenges would go to. But I really love what you're saying in the sense that it's all about having the idea and knowing what you want and knowing what is actually good. And then AI kind of can do the work for you. And that I think happened with initially with like writing content, like, you know, AI can just generate a lot of content. But what matters is, is it good and can you judge and can you do the right prompt? And I think here we are saying that, hey, if you know what you want the product to do, previously it was like, give me a detailed spec and I'll build it for you. Like, I'll give the spec to Cloud Code, you know, and let it build it for me. And but I'll have a lot more things to throw at the market or at the users and see what really makes sense. So we are easy to build and easy to discard also, maybe, right?

Yeah, exactly. And I think, you know, Bezos once talked about Amazon's superpower and the superpower of being an at-scale, like a scaled, you know, enterprise.

Yeah.

Was that they would move quickly and their ability to move quickly allowed them to try more things, that they could just try more things. And, and by trying more things, they would find the few things that worked really, really well faster, and then they would go build them. Well, you're now in a world where I think that even a startup has the ability to try way more things. So you don't have to be Amazon with thousands of engineers or tens or hundreds of thousands of engineers to be able to say, hey, let's throw, let's throw 10 people at that for 6 months and just try it. Let's try this. Let's try that. Let's try Amazon Alexa. Let's try this. They've done that very successfully, I think, over the years. Anybody can do that now. And what that ultimately— I think that the next step I would add in this journey, if you will, is I think that the, the ultimately the, the, what you go back to, if you eliminate these bottlenecks in the process, you get to what are the real limiting factors of a business. And the real limiting factors of a business are, you know, in some cases was how much you could do. Like, could you build something in enough time? Right? Uh, but mostly if, if, if that, if that bottleneck is gone, the limiting factor becomes ideas. It becomes, What are the things that make sense to build? Number one. Number two, distribution. So knowing the actual few set of things that would make a really big difference to grow your business. Two, your ability to put those in front of enough people to get traction and adoption. And I think those two problems, I don't know that they are changed dramatically by AI just yet, but I think when you take the bottleneck out of writing the code or building the thing, then it goes back to what should we build in the first place? And I would tell you, look, in our company we got a lot of ideas, but it— none of this technology has made us better at knowing what are the right things to build. That's something that we can certainly get more feedback and faster results by just trying a lot of things. But I think that, that it is interesting because it does let you spend more time thinking about what are the right things that are going to change your business and then, okay, those are the ones, let's go do those.

Yeah. And it is— it takes a lot of context to actually distill that down, like what makes sense. And it's a fast-moving context world, right? I mean, you thought about this idea of, you know, moving that markdown into Redis instead of on a file because last week, you know, Claude Bot came out.

Exactly.

And if things are new, things are coming out every week, you're like, okay, there's so much more context and fast-moving context. So You know, it's all— yeah, it's an exciting time to be in technology. But at the same time, I want to ask a little bit of a question on the platform side. Right. So you talked about the chair analogy, right? We are saying that we used to think about features, right? Now we need to think about product, not just a feature as an engineer or a product manager or designer. Right. And in the same way, you know, we have often talked about tools and platforms, but now the customers are really thinking about solutions. They're not saying like, you know, I want database or a model or this or that. I want something that solves, you know, this particular problem for me. Right. How do you see the role of platforms kind of as this solution becomes top of mind and people go with like FDEs and solution definition and put the right platforms or tools together?

Well, I think we're at a place now that's a— let me unpack the question because I want to make sure I—

Yeah.

Kind of answer. There's different parts to it. We're clearly in a world where— take Palantir and their forward deployed engineers— that model has proven obviously itself out in the— in their case. Right. Which is a lot, as far as I understand, a lot of government contracts and this and that. We have actually at Redis deployed and adopted a forward deployed engineer of sorts in our industry. But I think their observation, as I understand it, is that, hey, you know, a lot of these companies or agencies, they don't have the talent to be able to go and implement these solutions. And that's why you need to sort of include engineers and FTEs kind of, you know, into that picture. For us, we, as we have engaged with our customers, particularly around Agentic, We are, um, we do have forward deployed engineers who work with those customers because it's hard to build these applications right now. Like, companies are quite, uh, the, the, the infrastructure layer is not solidified yet enough to, and it's not, we're not yet in that like turn the crank implementation phase, right? We understand very, very well how to build a consumer app that can get to hundreds of millions of users. You build a Swift app, you build an Android app. You know, we have all the tools for testing it and building it and right in the middleware and right in the back. And we got mobile backend as we got all this stuff. And you could find, you know, in, in San Francisco, you know, or in Bay Area, you, you could find thousands of engineers who could do that end-to-end process for you. In the case of agents, that's not the case. It's gonna be very hard for you to find somebody who can tell you, I got a pattern, I know how to do it, I'm gonna bang this out. Cuz we're just not there yet. So still early in that process. So our forward deployed engineers are helping enterprise customers make some of those early choices. Okay. Because look, for especially in highly regulated verticals, there's a lot of work to do, uh, where, you know, we're probably going to see, uh, we're probably going to see the progress be the furthest out in the sense that you've got extremely sensitive customer data. Everything has to be still on-premises and data center, you know, there's all kinds of hurdles and things you have to jump through. And so I think it's— we're pretty far out from still having production apps on that front. So hence us putting forward deployed engineers into these companies helps them adopt these technologies faster. We've seen a lot. Last thing I'll add is we've seen a lot of progress by integrating with the leading ecosystems. So we have been integrated into LangChain, for example, as Redis has been, has been integrated there into that ecosystem. About, you know, I would say maybe 50% of the deployments that we see anyways in enterprise of agents are using LangChain as an underlying or LangGraph as an underlying foundational infrastructure. Now all of the, now all of the model providers are starting to go there. You saw, I think, I think just today there was announcement from OpenAI on this front. Microsoft shipped an agent framework. Google has their agent framework. We are integrated. Redis is integrated as the default memory storage platform for, or storage platform for Microsoft's Agent Framework. We're doing similar integrations with OpenAI. So the fact that we're kind of part of that underlying fabric and has gotten us integrated into these platforms really, really well. And the second thing that's driving it, because there was a survey that came out recently that showed Redis as the number one by market share for agent data. Well, how do we do that? Well, one way that we did it is one reason that that has happened, because if, to be honest, we didn't push it, it got pulled, is that our technology, one of what makes Redis special is it is so easy to use, right? There's a CLI, the commands are English language understandable. Even the protocol is essentially written in text in English. It's easy to like debug. It's, it's kind of perfectly built for an agent. And agents love using Redis. Agent coding tools love pulling in and using Redis. They're very, very good at it. And so that's another aspect, I think, of what drives success here is, you know, in our industry, we used to build infrastructure tools for developers. Now we are increasingly building tools for agents. Who are developers, the agents of developers. And so that's a different skill set. You know, you're not building flashy websites or, you know, going to open source conferences or what distribution you're putting MD markdown files on the internet that are written as in the most simple possible explanation of how to use your tool. So I actually anticipate any of these legacy tools and APIs and platforms that are hard to use are going to really struggle from an agentic perspective because they generally like if the LLM is smart, it's going to prefer simpler is always better. You know, Hanlon's Razor.

Yeah. Yeah. I think one of the things that I feel is also happening right now is, you know, back to that FDE question you said, right, where building these things is hard. I feel like because AI agents has suddenly created a market where everybody's in market at the same time. So I've talked to like companies that are like let's say in manufacturing sector outside the US and the level of technology or talent that they can have to build something like at the front of it, everything looks easy working with agents, right? But there's a lot of tech stack that goes beneath it. So I agree with you that it is hard to build and making that possible, you know, whether it's integrated into existing solutions and all sort of becomes critical. I also feel like many of these companies that I talk to are looking at like, hey, what I want to do is make this part of my process easy. What does it mean as far as agents and models and context and all of that stuff is not something that they are really thinking about, right? So they need somebody to help them unpack that.

Yeah, and they need to see, okay, has anyone done this before and what tools did they use, right? We're in that front end part where everyone in the tech world now says they're an AI company.

Yeah.

Any new interesting problem, there's probably 10 startups that are going after solving it. So it can be very confusing for a developer to figure out what's the best practice here. Like, there is no best practice yet, right? There's a bunch of people trying things. We're in that early phase. So this infrastructure stack for agents needs to settle down a little bit and get some more maturity before I think we really start to see the volume of applications being created because they have to be created by humans still. And, but I think that's happening again much faster than in any previous cycle. And that's the important thing is don't assume that the formation— let's say, let's say you use history as your guide, like, well, how long did it take for the cloud mobile stack to solidify to where it sort of became easy, quote unquote, to build a scaled application on the internet? Well, that maybe took, you know, 3 to 5 years, right? From the time that somebody built the first app that got to 100 million users and was really good and, you know, scaled in the internet and worked on, you know, infrastructure to where you had, you know, AWS and the whole PaaS stack and mobile backend as a service and identity, like all those hard problems sorted out, probably 3 to, 3 to 5 years, maybe something like that. Um, I think this time it's going to be much faster. Uh, you know, you'll find, okay, that stack is probably going to solidify very quickly. And, and frankly, you know, if you can become one of the apps in that stack, one of the standards, like Redis is the standard in the cloud mobile stack for fast, you know, data storage and operational data platform. Um, a lot of value there. You know, there's a, you can be one of those winners. There's a tremendous amount of value in, in that.

Yeah, absolutely. And you talked about every company calling themselves an AI company, right? One of the things that if you break it down, you know, there are model companies, of course, that have the foundational models. There are just a handful of those, right? And then there are all these companies that are in the enterprise space, technology companies, which I think have varied contexts around them. So, you know, how do you think about this sort of context data-centric company and model-centric company and the opportunities, because I see the model companies also getting into everything right now.

Yeah, well, I'll take the first part of that one. So, so, you know, and I mentioned the news, the OpenAI Codex news about their agent platform this morning. So I haven't even looked at that news yet. I just saw the headline flash by. But I think that the model vendors are certainly going to try to make it as easy as possible. And we're seeing this to build applications using their platforms. The, the CSPs, you know, Microsoft, Amazon, Google, I think they clearly want to have the front-end experience around how do you create an application on my platform. So obviously Microsoft's partnered with OpenAI, Amazon's partnered with Anthropic. So you know, I think we're going to probably end up in a world where the hyperscalers do have a significant foothold in terms of enterprises or businesses that want to build applications, you know, and go to their platform and get an end-to-end stack. Right. It's not clear to me how, whether, you know, where the line will be between model vendor, model companies and, and the hyperscalers. And if I had to guess, you know, I think this expert, the real expertise in the model companies is the models. Expertise in building enterprise software in general is going to be with the hyperscalers and also the distribution. So I would not take Google, Amazon, or Microsoft out of the running for, say, for example, owning that frontend experience. You know, we saw Cursor really take off. We saw Cloud Code take off. So I think, you know, I'm certain that you're going to see the model vendors try to own more of that stack. Um, how much they're going to own beyond the model is not totally clear to me yet. I've been very, you know, I think what they have going for them right now is they're small, they're smallish, they're scrappy, they're moving very, very quickly. Right. The OpenAI agents framework is fantastic. I mean, it's just a beautiful thing. Yeah. Anthropic has got an amazing platform in Claude Code, which they just today upgraded. Actually, I think it's now a native, a native application. So you're seeing that space move super fast. Not clear where the line will be. Between what the hyperscalers do and what the model companies do, you know, that'll remain open. I don't think anyone has really moved seriously in the direction of going further down the stack. So specifically data platforms. So there's obviously the new entrant, the kind of two companies that are pre-IPO in the, in the data world, Databricks and Redis, you know, I think in terms of having the biggest scale and being ready to go out and IPO these companies. We're one of them, and we're seeing a lot of our growth driven by AI workloads and GenAI, you know, sort of things that are happening. So we were well positioned for this wave to happen. But I think you're going to have a— there's, there's got to be a lot of data companies that are not finding a place for themselves in this new stack. And so, you know, that's, that's going to have to— that's going to clearly be a challenge. And then you look in the public markets from a data infrastructure companies and you've obviously got the hyperscalers who have their own solutions. But it's a small list of companies that are, you know, in this space. There's obviously Snowflake and Mongo, right, who are clearly multi-billion dollar revenue streams wanting to make sure that they're part of that next generation AI stack. And you can see CJ at Mongo has been out talking about, you know, some of their great customer wins. Snowflake obviously is a very forward-thinking company. So they've been doing a lot with Agentix. So that's Those are the players, as I see it, who are sort of circling around this. I mean, there's a long, you know, a long list of, of ankle biters and, you know, other new startups that are kind of coming out to solve some of these new problems. But, but I think, you know, I don't see enterprises looking to adopt new data platforms yet. There could be some specialized use cases, you know, super low-cost, reasonable speed vector databases. Probably maybe one, you know, we've seen Google out there with their vector store that's scaled up, some startups kind of coming at that space also. So there's probably like there's some edge cases, kind of interesting, you know, use cases. But for generic data platforms, it feels like the big guys in the category have really responded and have built out, you know, the additional capabilities and are figuring out how to integrate with these ecosystems effectively.

Yeah, and I think memory has a really important role to play. And the way I see this is that the NVIDIA acquisition of Groq, right, paying a premium for on-die memory and especially for inferencing workloads wherein the next level of memory performance is kind of the, the RAM or the DDR that comes with the chip, right? And if I look at the same parallel, then I would say Redis as the sort of fast memory compared to maybe a data warehouse or a database, right? So there is a premium that is there when you can access that information or that context faster at the time of inferencing and sort of stitch that together. Right.

So that's where we are. That is our, you know, of the emerging problems and interesting things to go do and build businesses around. That is the one that I think is most interesting to us. Inference time, you know, inference time workloads. So, so when you are running your inferencing, there was a bunch of things to do. The first interesting opportunity that we saw emerge there, we predicted anyways, was that for many applications, the cost of intelligence, the token costs were going to be too high to put something into really into production. So we built and we, and you know, even if the costs are, and the costs are doing this, by the way, they're plummeting. I mean, orders of magnitude per year that the cost of intelligence is, is, is probably going to asymptote towards zero. But even with that, what's necessary is a tremendous amount of infrastructure to make that happen, to drive those costs down. So you can do caching as a classic computer science pattern to make that happen. So obviously that's where Redis has been one of the killer app on our platform has been caching. Really quite interesting how you do that in the world of LLMs. We built a product called LangCache. LangCache is a web service. It's at redis.io/langcache. And what you can do is you can take a prompt and you can just feed it into LangCache. And if it has seen a similar prompt like that before, it can just feed you the answer that it already has. And if it doesn't have the answer, nothing similar to that, it can go hit the LLM and then populate the cache. So it's a cache that sits at the prompt layer. We saw one company that's very good for certain kinds of use cases. Take a open healthcare, open enrollment for the healthcare industry. In this world, you have like in the months of January and February, whenever open enrollment is, tons of people go to the website and ask questions. We ran a POC for one of the largest healthcare providers here, and they're working on an implementation now where we saw a cash hit rate of questions from consumers at 95%. So very, very significant savings for them on a large language model. We have another Fortune 50 company who just bought our product. And implemented it in all their TVs. And so their TVs now, when you go into their voice search, it's using, I don't know which model vendor they're using underneath the covers, but it turns out a lot of, you know, it's, it's not a, it's not a multi-turn conversation. First of all, it's like, find me this. What's good in, you know, in this genre, you know, can you show me the Grammys, whatever, you know, play the Super Bowl, like that kind of stuff. Those are all very, very cacheable. and they saw a 70% reduction in their LLM charges by using LangCache. So LangCache by Redis is a great starting place that we've identified in that area. The second area that's like, and there's a bunch of areas we can talk about, you know, 2 and 3, maybe you tell me where you're more interested. One is agent memory. Okay. Another is data access, enterprise data access and managing context. So we can go in either of those directions.

Yeah, I think, you know, what I'm seeing is that what we do at Next.js is connect to the different enterprise systems and bring that data because that's the core of the technology. And I can totally see that caching that information because you don't want to hit that, you know, SAP system too many times. Exactly. Rate limits or other things to deal with. It's a real problem. Yeah, yeah, yeah.

That we've been focused on. So this is a very natural— so the problem, as I would describe it, just to restate what you just said in my words, is, you know, how do you get the right context, the right data into the context window at the right time? And how do you do that quickly and efficiently?

Yeah.

And securely. Okay. So the first pass at this, if you will, was hand-building the context. So if I go back to like I'm going back in time to, you know, 3 months ago, But like, how would you build an agent? You would, as a developer, you'd write some middleware like this is in your agent loop, your agent run loop, and you would say, okay, take the prompt, go get my static context that I care about. So like, what is this prompt? Maybe, you know, use an LLM to figure out in the prompt, is there some useful context that I need to bring in? Go do your database query, go pull your thing out of memory or whatever and then stash it in the, in the, in the, in the query. So if it's a travel agent bot, okay, we're going to need the travel policy for the company, probably always need that. So just stuff that into the context window. And with larger context windows, this problem has gotten a little bit easier. But along comes MCP, obviously a different way of pulling context instead of pushing it. So instead of the developer hand-rolling the context, now we have the agent asking for context. So now we tell the agent, here's the MCP tools you have access to, here's what they can do, describe them, and then let it go pull the context. So perhaps you have an API which is, uh, you know, get travel policy. Well, that's pretty simple. Anytime it needs to get the travel policy, maybe there's a search travel policy, whatever, it can just go call that API, get the information, um, bring it into the context window. The problem with that— now that works well in the way I just described something very simple like a travel policy. But when you start talking about complex backend structured data sources, you're not going to want to put— now, by the way, all of those database companies immediately came out with MCP servers. So you might think the best way to solve this is let's take the MCP server, let's stick it in my agent and let it party on my data. Well, bad idea. You know, the relationships in your underlying structured data stores are completely, uh, opaque. You don't know what they are. Um, you don't know what the underlying data is in these tables. I mean, you can kind of guess by reading table column headers, but, but it would be better if the developer or the somebody in the data team could say, no, no, no, this is what this data really is. And oh, by the way, you definitely don't want your agent accessing everything in your backend data store. It can get confused. You can get too much, right? You can overload the context window with information. And I think that we've demonstrated that text-to-SQL, especially in this kind of unconstrained, you know, opaque world of backend structured data stores, not the best way. So we have a different approach. We have, uh, we did an acquisition of a company that brought us a very popular open source, um, project called EnrichMCP that was based on the idea that what is needed to solve this problem is a semantic layer that sits on top of the data and allows you to define kind of like an object model essentially for your data. So take your structured data, denormalize everything, and then expose it in a way that is very, very easy for an agent to understand. So, so we did that acquisition. That was a company called Featureform. We've now been working on our new product, which is called Redis Context Engine. And what the context engine does is a couple of things. The main function is to take structured and unstructured data, uh, and then put it into the, a materialized view, an object model view with descriptions and relationships that are all described in English or in a language that can then be published through an MCP server to the agent. And so what this technology does, so there's a few parts that make that work. Part 1 is how do you get the data into Redis in the first place? So we have a brand new product we call Redis Data Integration, and you can use Redis Data Integration. It's a CDC, you know, pipeline. You can set up a CDC pipeline, you can pick your tables, you can pick your columns, you can pick your queries, and you can, and you can, and you can pick your basic transforms. So I want to take let's say my Workday employee table, and I want to do a query of employee table, and then I want to join that with another table and get the grade level and title of the employee. And I want to just take employee, employee ID, grade level, and title, and I want to stick that into a new table, and I want to then denormalize that and store it all in Redis in a series of keys, or maybe a sorted set, make it really fast. Boom. Now we have that. So that is now a pipeline that's set up to read your enterprise data, just the stuff you want, and push it into Redis. On the top side of Redis, that's kind of southbound, is getting the data. The northbound is now how do we expose that to our agent? So with our context surfaces, the new project we've, uh, been working on, uh, what you do is you use, create a bunch of Pydantic models around the underlying data, what's in Redis. So you describe what is now sitting in Redis as a set of objects and you semantically describe the, the fields and the keys and the relationships between the data and the objects in your model. And then that with the small amount of work on the developer's part or the data engineer, you then expose an MCP tool, okay, that exposes exactly what the agent needs, already preprocessed, already secured, already done in the right way. Really, really easy, which makes the effectiveness of those agents super high. So it's got an MCP northbound interface and then it's got a data gathering and transformation pipeline in the middle. And then you come in, in the, you know, in the bottom end and then you come in the middle and define the objects for your agent. So that's solving problem number one, enterprise context and how you get that. Now, why Redis? Why would you use Redis to do this? One, we're really fast. So Redis is extremely low latency. It's designed as an operational data store, you know, for applications. That's often how it's used. This exact pattern I just described is how people were building cloud mobile apps before. They were just doing a lot of that stuff manually with, with middleware. So all we're doing here is we're saying, look, we kind of know this is a problem that's coming at us. Let's just go get ahead of it. Write a basic product that can do all this, which will make the lives of these developers much easier. So you don't have to implement that old model anymore. And it's solving a new problem, which is instead of the engineer and the data engineer being the ones writing the queries, it's going to be the agents writing the queries. And so you want to do a lot of that work for them so they don't have to do that work. So that's solving problem number one. Problem number two. Does that all make sense?

Like, yeah, yeah, that makes sense. In fact, if I want to bring this back to something that I think you have said once before, which is that AI models don't fail because they're slow, but they forget. I think you wrote something to that effect, and I feel like you're solving both the problems. You're solving the slow problem and you're also solving the forgetting problem.

Well, the forgetting problem is an interesting one because it's slightly different. So this is the like, how do we put the context of our enterprise data into the model? This is a good way of solving it. Redis context engine. The second piece is agent memory, essentially. And how do you take all the data that's coming off of a Redis, off of an, off of an agent and make sense of it? So first and foremost, an agent needs to sometimes store data. What we often see those agents doing today is kind of using the local file system and, you know, using text files because they're very, very text, you know, friendly language models. But memory is an emerging, very challenging problem. So we have an unstructured conversation between a user and a backend agent. We don't know what it's about, but we know that there's a set of logs that come out, a set of transcripts. And let's take that. If we use the travel agent example, there in the conversation, right, you may express certain preferences that should be remembered. You know, you might say something like, hey, I need to, I need to go to my meeting in Paris next week. You know, can you help me book this flight? And the travel agent, you know, go does some work and comes back and says, here's the available options. And maybe one of the things you say is, well, I don't, I don't ever like flying in window seats. You know, I always like the aisle. So you just say that as a, as a paragraph or a sentence to the agent. That's a useful piece of information that needs to be remembered. What you don't want is that piece of information that you like, you know, that you like aisle seats, not window seats. We don't wanna forget that. Okay. But we also, it's packed into a bunch of other text about Paris trips and this and that and the other thing. It's not very, so how do you extract it? Well, so that's one problem is extracting semantic memories that are worthwhile to be remembered. User preferences, you know, experiences, et cetera, et cetera, et cetera. Defining how much to remember, what, you know, et cetera. And then a set of, you know, I would say even more second order problems Like, what about conflicting memories? Like, maybe, maybe you say you don't like window seats today, but maybe in 2 weeks you go, you know, I actually took a window seat and it wasn't so bad, so you can book me in window seats now. Now we have an overlapping and conflicting memory that has to be resolved. And so this is an— and I can go on and on about that, but the memory problem of remembering what the user has said, interesting challenge. Now, to put it into context, how do we used to do this in, in the cloud mobile era? We're just exiting, we would build a SaaS app and then we would build a preferences page with a CRUD-style interface, you know, like a, you know, which is window— like I prefer window seats and some PM would write a requirement and then you'd go save a field in your database that says, you know, user prefers window seats or prefers aisle seats or, you know, is a vegetarian or whatever. Well, we're not going to go build that kind of an app anymore. We don't need to define a priori all the, user preferences, the user can just express them in their normal native language. The LLM can then figure it out. We use an LLM under the COVID so we built a product called Agent Memory Server that you can just throw all of the chat transcripts at. We use a large language model to extract relevant memories and store those in Redis as memories, and we vectorize them and store the raw text as well. We have another side of the API, which is to remember something. And you can pass in the context. So if we start a brand new conversation and you say, you know, I want to plan a trip to Brazil with my family, the travel agent will take that context, use its MCP tool to search the Redis memory server and say, got any travel preferences, right, for Saket? Oh yes, we find through a vector search, you know, doesn't like window seats. And so we can now include that the agent can be intelligent and go, well, Seth Kett, I remember 2 weeks ago you told me you didn't like window seats, so I'm gonna make sure that you are booked in an aisle seat. That's a really delightful experience to have an agent that remembers you and your preferences as opposed to what we have today, which is sort of like these forgetting agents that don't remember anything about you. Now, some of the model vendors have started to solve this in their apps, right? Not at the API layer, but in the apps. ChatGPT, if you tell it explicitly to remember something, it will. Some of the new agents that are being built, like ClaudeBot, are using text files to store memory. I think that's an interesting approach, but I think we're going to see lots of different things. So we've been working on this agent memory server. It goes well beyond just the simple extraction and, and recall of memories. It also deals with more thorny problems like I would call the contextual grounding. So contextual grounding is a problem essentially where if you were to say something like, I, I, I am going— I need to go to Paris this Friday. Okay. If you were to store like, okay, we've learned something about you. The user needs to go to Paris this Friday. Later today, if you come into a chat with us or tomorrow or whatever, It'd be good for us to remember, like, hey, are you still planning that trip this Friday? Well, this Friday is a temporal reference that is, uh, needs to be grounded to an absolute time. It's a relative temporal reference. On this day, Monday, you said this Friday. Well, that's actually February 8th or whatever. So contextual grounding, in this case temporal grounding, would be replacing relative time references with actual times. Replacing relative locations with actual locations. So, you know, I, I, uh, if I'm in San Francisco and I say I need to go, like, that place is on First Street. Well, when we store the actual text, that place is on First Street. We need to know, A, what was that place? And B, what is First Street? Which first? Because otherwise, if you just store that place on First Street, you could be sitting in New York. 2 weeks from now saying, where is that place? And it says it's on First Street and it gives you address to, uh, in New York, not in, in San Francisco. So, so the location grounding. So contextual grounding's an interesting problem that memories have to solve. We solve that problem. We have a, an approach that we use to do that. Uh, and then, and then conflict resolution. You know, you say you're interested in, you know, you're a vegetarian now, but then maybe in a, in a month you decide you're not a vegetarian anymore. How do I resolve the fact that you've said 2 things that conflict? All these areas are solved in the way that inspired very much by the human brain. If you look at our long-term memory and some of the processes that run during REM sleep, there is a process that goes through and takes the comp— the, the, the collected memories in the short-term working set memory of the brain and compresses it and extracts the facts to be remembered and stores them in the long, in the, in the long-term memory banks. Uh, also that does conflict resolution. So you can have two things that relate and tie those together, et cetera, et cetera. So. We sort of inspired by that in terms of how this product works. And by the way, this is similar to what other memory companies are doing. We got our own unique takes. So that's another interesting area. So that's Agent Memory Server. So you got context on the one side, bringing enterprise and business data, you know, or data about the user into the context window. And then you got remembering things like storing memories and then, and then recalling them later to make a much better experience.

Okay. I'll just close that. Yeah, yeah.

Okay.

Thank you. Yeah, I mean, it's such a fascinating conversation, Rowan, and I can see that we have gone from basically a fast memory cache is how Redis started to an entire memory platform, if you will, right? And extremely relevant to AI and the outcomes we can get from that. So really appreciate you going deep into that. Just to very quickly, one second, wrap it up. You said something about, you know, the history aspect of it and you've been through the dot-com rise and fall. Quickly, you know, is this— are we in an AI bubble or is this not one?

You know, I don't know about that. I would say probably there's some bubbly aspects to what's happening now. Certainly on the surface it looks like there is. But, but if the bubble last time was kind of defined by like, hey, we're running fiber that nobody ever used. You know, all of our data centers are running hot right now. I mean, my, you know, one of my engineers, my top engineer who uses AI as sort of primary method of coding, is spending as much on tokens as his entire salary. So we are burning these data centers up. So I don't think it's a demand problem for this compute. I think that that is there. But to the degree that a bubble exists in one way, shape, or form, which I'm sure it does, you know, the way I think about that is I went through that in the dot-com crash and what was relatively safe at that time, what, what, what washed out the, the, the famous examples you hear is like the Webvan and the Pets.com and whatever those different set of apps were. It was not clear that the apps were quite ready at that time. I think we're in a similar state now where it, I wouldn't bet on any apps at this time. It'd be hard, you know, it, you know, you'd be likely to be wrong perhaps if you made those bets. But infrastructure is much safer. I worked at a cybersecurity company, you know, when the dot-com crash happened and our, our sales just kept going like this. So I think infrastructure is a relatively safe bet, and it's one of the reasons I chose to come to Redis and to go help build out this next generation of software.

Awesome. This has been a great conversation, Rowan. Very enlightening. Thank you so much for taking the time and would love to see how the Memory platform takes off.

Thank you for having me.

Thank you.