Write, Change, Recall, Forget: MongoDB's Pete Johnson on How Retrieval Drives Agent Performance

Hello and welcome back to The Cognitive Revolution. Today, my guest is Pete Johnson, a 30-year technology veteran now serving as Field CTO of AI at MongoDB. We start with a brief history of database technology going back to the 1970 introduction of SQL and unpacking how the relative scarcity of disk space at that time informed data normalization as a design principle. And then how decades of Moore's Law and the resulting evolution of constraints ultimately led to the creation of MongoDB in 2007. This history puts today's acceleration into stark relief. As Pete recounts, in a remarkably short period of time, we've gone from being effectively forced to implement RAG pipelines by the very limited context windows of GPT-4 class models To RAG is dead when we first got million token contexts, to the brief token maxing window and subsequent budgetary fallout, to today where RAG is again a major enterprise priority. As company data is increasingly available for AI systems to use, usage itself is scaling, and naively maxing out the context window costs multiple dollars each and every time. Pete's key point above all is that agent performance, and especially cost-adjusted agent performance, depends heavily on effective retrieval. With this in mind, MongoDB has continued to ship the improvements that developers need to take full advantage of AI. Vector search, sorting methods which combine traditional and vector search results into a single result set, a re-rank tool called $rerank, and their own embedding models powered by the acquisition of Voyage AI, which have some pretty cool features like a shared embedding space across model sizes and a Matryoshka data structure. Interestingly, while MongoDB customers are free to use whatever embedding models they like, Pete insists that embeddings are, contrary to popular belief, not yet commoditized. From there, we go on to talk about how developers are using these tools to build memory systems. The write-change-recall-forget loop that Pete sees as the general pattern And why forgetting is currently the hardest part. For now, Pete says it's important to remember that we're only 18 months or so into building AI agents, and as such, there is no established right answer and nothing like a lamp stack for AI that enterprises can confidently buy and deploy without meaningful customization. We also touch on the lessons he's learned from talking to customers in India, Brazil, Mexico, and Europe. all just this year. And to my surprise, he reports that the most advanced companies he's spoken to in 2026 were in fact outside of the United States. Without further ado, I hope you enjoy this speed run through database history and high-level AI readiness strategy for enterprises and survey of the still fast-evolving frontier of agent memory with Pete Johnson, Field CTO of AI at MongoDB. The Cognitive Revolution is brought to you by Mercury, the banking platform loved by over 300,000 entrepreneurs. I use Mercury's virtual cards, which make it super easy to set limits, expiration dates, category, and even merchant-specific spending controls to give my more autonomous AI agents, AID and Clay, the ability to buy and test products. Recently, I asked if they could find a good way to split an AI-generated image into layers, separating the text from the background and so on. 2 of the products they found were behind paywalls, but using their Mercury virtual card, which is limited to SaaS purchases only, they bought a month's subscription, tested the products, allowed me to review the results, and then canceled the stuff we didn't need, all with functionally zero risk to me. This is already really powerful. And now, with Spend, Mercury is making it possible to run an entire company's spending with the same level of ease and control. With Spend, you can set granular budgets for every team, person, and all the agents you like. Plus, you can process receipts automatically and even temporarily auto-lock people's cards if there are ever any issues. The future of spending money is dynamic but controlled. So join me in the future of banking. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column N/A, members FDIC. The IO card is issued by Patriot Bank N/A, member FDIC, pursuant to a license from Mastercard International Incorporated.

Pete Johnson. Field CTO of AI at MongoDB. Welcome to The Cognitive Revolution.

Thanks so much for having me, Nathan. I appreciate it.

I'm excited for this conversation. I've been a database guy in the past, and now we're all kind of everything guys these days with our agents greasing the wheels for us. But the, I think one of the big themes that we're gonna get into is the quality of data is still critically important to Making things work. And I'm excited to learn everything I can from you over the course of the next hour or so. Maybe just for quick starters, I remember first encountering MongoDB like 15 years ago, maybe when a young Guillermo Rauch, who's now the CEO of Vercel, was showing it off in the context of what at that time was like seemingly like crazy futuristic Node.js applications. Like, wow, you can make JavaScript do these kinds of things.

Yeah.

And there's a, wait, there's a database that like works in this different way where I'm not constrained by the schema that I already defined. Like this was mind-blowing stuff. I'm not sure, we've got a lot of technical people in the audience that say AI engineer is kind of our number one profile, but I'm not sure everybody knows kind of the broad lay of the land when it comes to database technology. So I thought maybe it would be helpful just to first kind of contextualize SQL versus NoSQL, schema versus schemaless. Give us kind of the high-level view so we know where Mongo sits in this obviously bigger database space.

Sure. I like to think of this in terms of the history, in terms of the different kinds of problems we've had to try to solve with software over the last 60 years or so. And where it really starts for me is in June of 1970 is when EF Codd, who's an IBM researcher, Wrote the original white paper that gave birth to SQL. So that was June of 1970. I happen to have been born in February of 1970, so I am as old as SQL, Nathan. And if you think about the kinds of applications we were building at the time as an industry and the kinds of hardware we had at our disposal at the time, the applications were very departmental in their nature, right? Not every, Not every company had a computer, and those that did, the people that were using it were often on the same floor or certainly in the same building. They were working 9 to 5 hours. It was okay to have downtime on the weekend. And if you think about the big 3 pieces of hardware at the time, memory, compute, and storage, storage was by far the most expensive of those 3 things in 1970. So if you think about how you might organize data, keeping in mind that the scarce resource was storage, you would of course do it in such a way that's now known as normalization. So you may have in university or in coding school, you may have learned things like first and third normal form, all about trying to lay out data on disk in the most efficient way so that you aren't storing any piece of data more than once. The simplest example I can give with that might be my wife and I share an address and maybe with a retailer might want to know what our address is. There's 2 people at that address. You might have in one table the list of people, in another table you might have the addresses, you might have a 3rd table that links the people to the address so that you're only storing that address one time. And that's the most disk efficient way to do that with, like I said, some kind of normalization. Well, fast forward to 2007, which in October of 2007 is when the first commit was made to MongoDB. And the world is a very different place in 2007. The world has internet, the world has cloud, the world has mobile devices. iPhone came out in the US for the first time that year. So it's a very different world. And the scarce resource in that world, because we had 47 years of Moore's Law between those 2 dates, The scarce resource in that ecosystem became time because the kinds of applications that we would write that were in the cloud or that we would write for mobile devices, you could not have downtime on the weekends. It was not people who were confined to the same building. It was not 9-to-5 use cases. So it was a very different world that required a very different kind of database. And that's what gave rise to NoSQL. Now, our flavor of NoSQL, which like I said, That first commit was in October of 2007, uses that as its core, JSON. So you mentioned the stuff with JavaScript, that's why that was so popular 15 years ago or so, is because what you get back from your API call is JSON. We're storing it in transit as JSON, we're storing it on disk in a binary form of JSON we call BSON. And that's part of why we're so fast is because we're keeping that format. The whole time so that when you serialize it off a disk, when it goes through our servers, when it ends up in your client, it's essentially staying in the same format that whole time. So if you think about the kinds of applications that we've been building since 2007, like I said, disk space is no longer the scarce resource. If you take that address example, if you denormalize that data just a little bit, if you instead of storing it across 3 tables, if you have a JSON structure that has my name and my address, and then a second one that's got my wife's name and the same address, you're duplicating the address. But what do you get in return? What you get in return is one disk read. So that's why it's faster. So instead of having to do 3 disk reads to get that same data, you can now do it in one disk read. Does that mean that every problem is solved by a NoSQL database or by MongoDB in particular?

No.

But it also doesn't mean that every problem has to be solved with SQL. And that's how that sort of evolved out of— SQL was invented at a time where the problems were slightly different, and a lot of those problems still exist. SQL's a certainly a valid solution to lots of different problems that we still have. But when you need speed, when you need scale, and increasingly when you need better retrieval quality, That's really where MongoDB shines.

I'm reminded of the old saying that the problem has faded, but the solution persists. And yes, there's, yeah, see that in a lot of parts of life these days.

Tell me a little bit more.

You were just kind of getting to it, but tell me a little bit more about, I mean, Mongo's a pretty big company, $30 billion market cap, obviously has made serious inroads into some of the biggest companies in the world as customers. Give me kind of the next level of detail in terms of before the AI moment, and then obviously we'll get into that in depth. What are the kind of big decision factors that would push an enterprise to go one way or the other with their database decision? And how are they, do they typically do it on like an application-by-application basis these days? Or do big companies feel that they need to have a single kind of consistent database technology for everything they're doing?

Sure. It, it's a mix. Like you just said, we're in like 75% of the Fortune 500. But if you look at the overall market, you mentioned our market cap, we're about $2.5 billion is what we ended our revenue. That was our revenue number for our fiscal '26. We're currently in the middle of our fiscal '27, and that's about 2-3% of the $100 to $110 billion database market. Where most enterprises choose us is those first 2 things that I mentioned. When you need it fast or when you need it at higher scale, because you're not storing disk, you're not storing that data across multiple places of disk. That's how you get the speed. That's how you can get better scale. If you're, if you're storing, if you're gonna retrieve JSON, because every modern programming language can serialize JSON into some object format so that you could use it immediately. in your business logic. And that's the other aspect of time. It's not just wall time for the transaction, but it's also the developer time of what do I have to do with this WAD that I get back from my API call and then make some kind of useful business decision out of it. But what we see is right solution for the right problem. And increasingly what AI has done is it's opened up the number of problems that we're a better fit for than a traditional SQL approach, especially those at higher scale. And that's why I mentioned the 3 things, the speed, the scale, the retrieval quality has really become an important factor. It continues to, especially later here in 2026, as you start to see people strategize around token maxing to evolve different kinds of agentic architectures in ways that we didn't see in the first half of 2026. Retrieval quality has become more important. For AI, but it does end up being a mix. It depends on the corporation. Some of them will take it workload by workload, and for this workload we'll use Mongo, and for a different workload we'll use SQL. Others are taking an all-in strategy. It really varies based on, in some cases, how much greenfield versus brownfield do you have in your portfolio? In other cases, it's where's my talents already? I was just at, for a major automotive company here in North America, I was just at a hackathon. Or we trained up like 250 engineers and turned them loose for 3 days on a hackathon, and they've got some new product ideas that fell out of that. So there's no single right answer, right? There's lots of ways to sort of walk through that forest. But at the end of the day, it does come down to right problem for— or right solution for the right problem. And where we typically help people is with those modern applications that need that speed at scale. those, those authentic use cases that increasingly need that better retrieval quality.

I'm really interested though in Big 3 tech modernization, and I've had a bit of a brush with it from time to time, just enough to know that there's a lot of appetite, but there's also a lot of friction. What was your experience of, and I'm sure a lot of these app concepts that people were working on are AI apps, right? Without naming names, what more can you Tell us about the kind of gonzo on-the-ground experience in the Big 3.

Some of it was customer experience kind of applications. Others was how can we make the dealers more efficient? How can we make the mechanics more efficient? Some of it was how can we make the in-car experience more efficient? It really spanned the gamut of different types of use cases that a Big 3 automaker might have. But really, it's— I mean, I'll go as far as to say, like, our education system has a bias, a historical bias. That thou shalt always normalize. And for a lot of use cases, that's valid, but there are an increasing number of use cases where these big 3 that I just mentioned matter. And increasingly that retrieval quality one is becoming a much bigger deal for agentic architectures as you start to try to take better advantage of the context window than just stuffing it with a million tokens every time. And we can get into some of the details as to why we're a good fit for that and why that's emerged, like in the wake of what happened to Uber earlier in the year with them burning through their token budget for 2026 in 13 weeks. That grabbed a lot of headlines, and there's been similar stories about where like maybe token maxing wasn't such a great idea to begin with.

Yeah, it was a phase. We've all, we've been, we've all been through our phases. I was really attracted to the schemaless database model way back when, because I'm just kind of a free-flowing person who loves to just, Make a mess now and tell myself I'll clean it up later. And so this is one of the things that I'm really interested in, in terms of how NoSQL DB interacts with AI applications, because it seems like on the one hand, it's more tempting than ever to do that, right? It's more valuable than ever to just be like, let me just dump all the data in here and like maybe an AI will make use of it sometime later. But then you do have interesting challenges downstream of that.

Yeah.

One, with your example of like the 2 records with the same address, now you gotta update it in 2 places. So that's something to keep track of, obviously. I'm also kind of struck that like the advantage of latency, you may have a different take on this, but it feels to me like the advantage of latency is somewhat reduced because now like everything gets put through a slow step anyway. And so the ability to like shave a little time off the DB call when I'm then gonna have to wait for tokens. Maybe matters less. And then I'm also thinking like, if I do start just dropping all my data payloads into a sort of extra field, now I'm gonna be doing like exactly what you're saying I shouldn't be doing in terms of just bringing a million tokens into context all the time. Uh, so take that in whatever direction you will. What do you see as kind of the biggest trends in application development? that have bearing on your database choices as we are AI-ifying everything. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Granola, the AI-powered notepad built for the way real people actually meet. Here's how it works. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup and no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word and still walk away knowing exactly what was decided, Who's doing what, and what comes next? When I had Granola co-founder Sam Stevenson on the show earlier this year, he explained how Granola aims to provide a calming experience for people with crazy workdays. And as a user of the app myself, I have been struck by how streamlined, even minimalist, the Granola product experience is. That takes real discipline, but the result is a product that works not just for AI early adopters, but diverse teams of people who just want to get things done more efficiently and effectively. Listen to my full episode with Granola co-founder Sam Stevenson for a masterclass in designing AI products for mass market adoption. And try Granola for free at granola.ai/tcr. That's granola.ai/tcr. The Cognitive Revolution is brought to you by Diffusion, the AI transformation specialists that help organizations, from traditional SaaS businesses to defense companies to nonprofits, build software factories that can scale not just outputs, but business outcomes. You probably know that the majority of enterprise AI projects fail. In general, that's because leadership fails to realize that AI isn't like traditional software that you can just buy and install. On the contrary, if you want AI to amplify your business's unique DNA, you'll need to make a sustained effort to record, understand, simulate, and optimize your business processes. Building these skills by trial and error takes years, but your business problems can't afford to wait. So here's how Diffusion can help. You identify your most important business problem, fly to Silicon Valley for an intense week of problem-solving with the Diffusion team, and by the time you leave, you'll have not only cracked a critical challenge, but built the core skills needed to do it over and over again from home. Cognitive Revolution listeners receive a 25% service credit on their first engagement with Diffusion. So visit diffusion.io/tcr to learn more about how custom-built software factories can scale critical outcomes for your business.

That's diffusion.io/tcr.

Sure. So let me, so first let me push back on one thing you just said. Most people think that we're schemaless, and that's not actually true. It's not that we're schemaless, it's that we're schema flexible, that you can change the schema more easily over time because more of the data is denormalized and therefore centralized in a way. It's not a strict requirement unless you specifically say so, that every document in a collection, so we don't have tables and rows, Right? We have a— think of a JSON WAD as a document, and you have multiple JSON WADs put together into a collection. Not every document in a collection necessarily has to have the same shape. You can have different shaped documents in the same collection. So because the schemas are flexible and because you can have different document shapes in the same collection, that's why people assume that we're schemaless. But what it does is it gives us the ability to be malleable in a way that a traditional SQL schema is less malleable. I won't say not malleable at all because it depends on how you're laying out the data, but anybody who's ever had to change a SQL schema that's already in production and the cascading effect that has on multiple tables knows the pain of what I'm talking about, that if you had some of that data denormalized, it's far easier to add attributes to a document that's already there and do so selectively in a way that isn't possible nearly to the same extent in the SQL world. And this is where how we've implemented, like the history of how we implemented vector search and the impact that has on application architectures for agents ends up, like the details matter here. So if you'll allow me, I'd like to talk a little bit about that history. Does that sound okay?

Yeah, please.

Okay, so for us, it really started with, in an unusual place, and that is with lexical search. So in 2020, we noticed that a common use case for MongoDB was for people to stand up their own Apache Lucene servers co-located with wherever their MongoDB clusters might be. And the reason they were doing that is they wanted to be able to point that Lucene cluster to different text fields in a document and be able to do keyword retrieval off of them. Perfectly reasonable thing. So we thought, well, as you might know, there's, there's 3 different versions of MongoDB. There's Community, which is you're responsible for the support and you're responsible for the operations. There's Enterprise Advanced, which most customers are using on-prem, where you're responsible for the operations, but we're responsible for the support. And then there's Atlas, which is our managed service version of it, which you can deploy your instances on any cloud hyperscaler data center you'd like amongst Amazon, Google, and Azure. And in that form factor, we will do the support and we'll do the operations. So those are the different choices you have when you deploy MongoDB. And regardless of which one you choose, like I said, We noticed people standing up Lucene servers so that they could get keyword retrieval on what they had. So in 2020, we introduced what's now known as Atlas Search, which is on the managed version, the managed version of MongoDB. You automatically just, you just get it as part of the instance that there's a lower-priced tier where you just point it to attributes, some text attributes that you already have in your data, we'll automatically index them. You can do keyword retrieval search on those. Then the next logical thing was a couple years later, well, if you're gonna have lexical search, you might as well also have vector search. So think about this, what's a vector? At the end of the day, a vector is an array of floats, right? You take some piece of data and whether that piece of data is text or an image or audio or video, you pass it to an embedding model of your choosing and what you get back is an array of floats. So to us, an array of floats is just an additional attribute to a document you already have. So for us to implement vector search was just add an additional attribute to the flexible schemas that we already have, and now build a vector index on top of that array of floats. And that's your vector search. And what it created was this notion of being able to do these powerful hybrid searches as well. So If you think about the sort of the trivial example that most people learn MongoDB off of is suppose you have a book and a book might have a text field for the title and it might have a text field that is a URL that points to the COVID of that book. You might have an integer that's the number of pages, you might have an integer that's the year of publish, and you might have a text field that's the synopsis. So if that's a standard record for a book in, if that's a standard document for a book in MongoDB, and that's put in a collection, suppose now I want to create a vector index off of the synopsis. I could take that text, synopsis text, I could pass it through an embedding model of my choice, and I then store the array of floats that come back from that embedding model. And if I wanted to, suppose I had 10,000 records, I had 10,000 documents in my collection, maybe I only wanna do a vector search on the top 1,000. Maybe I don't wanna do it on all of them. Because of the flexibility of the schemas, because you can have different shapes, MongoDB not only allows that, but thrives on it really well. So we had lexical search, we had vector search, and then that enabled us to do this notion of hybrid search. So suppose I wanted to do combine a lexical search of the title of the book with this now vector search that I have on the synopsis? And what if I also wanted to pre-filter because I've got some integer fields in here? Suppose I want to do a search there where I do a lexical search on the subject and a vector search on the synopsis, but I want to eliminate any documents, any books that weren't published after the— that were published. Oh, I only wanna see books in my dataset that were published after the year 2000. So now I have these 3 levers of query power that together are— give me lots of interesting ways to now query my data that I combine this sort of pre-filtering based on the metadata with the lexical search, with the vector search. So once we had that in play by like 2023, we thought, well, this works with Any embedding model you want, but could we make this easier to develop if we had an embedding model that was part of it? And that's why we purchased Voyage in 2025, which should create this better together story. And we can get into all kinds of things with the better together, but the way that we go to market with Voyage is you can use Voyage on whatever vector database you want. You can use any embedding model and use MongoDB is your vector database, or you can use them together and there's a better together story that's part of that. So that's kind of our history. Like I said, kind of unusual that it started with lexical search, but it kind of went lexical search, vector search, hybrid search with pre-filtering, and now this better together storage with the Voyage embedding models. That's how that built over time for us.

What is it like to query that today? Are you able to, I, I think this is like a, Has been relatively recently advanced, right? That I can now issue a single query and put kind of in SQL terms, I guess, a WHERE clause and like all 3 of those different kinds of data and the engine under the hood, which filter to apply first, like all that stuff kind of, I don't have to worry about anymore. Is that right?

That is. So one of the advantages that we have in the way that we implemented vector search is it gets to stand on the shoulders of the base product that we already had. So we've been running Atlas in production for a decade now. So we already knew— know how to replicate data across multiple data centers, even across multiple clouds. We already know how to do sharding, whether you want to keep data close to specific users for business reasons or for legal reasons. We already know how to do security, not only query at rest and— or not only encrypted at rest and encrypted in transit, but we can also do during the query, we get to stand on the shoulders of all of that for the vector search. And one of the things that we get to do on top of that is MongoDB has always had this notion of aggregation pipelines. And the point of aggregation pipelines is instead of forcing you to do a query, get some JSON wad back of your result set, and then have to massage that before you use it. Instead, with the aggregation pipelines, you can give extra, you can give extra instructions to the backend So that data gets sorted, filtered, massaged in different ways so that what you get back from that API call you can now use immediately and you don't have to do any other further massaging. So because we already had that mechanic in the core product, what you're referring to is we just announced a pair of features. One's called Rank Fusion and one's called Score Fusion. And what that allows you to do is in absence of an aggregation pipeline in these new aggregation pipeline stages, you would have to make the vector search and then independently you would have to make the lexical search and then you would have to combine them yourself. Now with those 2 aggregation pipeline stages, you can do it in one API call. So you only have one round trip call to our servers and then on the backend we'll do both of those calls for you and then we'll combine the results either based on their rank with Rank Fusion Or based on their score with score fusion so that what you get back is an already sorted list based on which one of those functions in the aggregation pipeline you chose so that you could then use that JSON-WAD, that result set immediately.

I'd love to learn a little bit more about how that works because this is something that I have run into and I've, I've, I've come up with pretty simple solutions for myself, right? But like, You sort of have the filter and then I'll just be like, okay, I guess I'll sort by the vector match score subject to the filter, right? Or we'll kind of tier them based on different filter values. It's always struck me that there should be smarter ways to do that than I've come up with. And if you guys have some, so. What are the kind of galaxy-brained approaches for getting the best final sort order? From these kind of fundamentally different things?

So for now, what most people are doing is, and this is the reason why we have the pipeline stages that we do, it's, it's usually based on a rank or by score. And if you also throw in the pre-filtering, that lets you, that lets you reduce the dataset before you do the queries. And then you get those queries back and you can interleave them based on either score or rank. For now, that's what most people are trying to do, but My joke for this, Nathan, is like there's 2 kinds of developers. There's those that love MongoDB and there's ones that haven't tried it. Like we're, we're, we're always in a pretty tight loop of feedback where we— the whole reason we came up with Rank Fusion and Score Fusion is because the community told us they wanted it. And very much the same way that why did we start doing Atlas Search? Why did we make lexical search part of the platforms? Because we saw people were using it and we found a way, we saw a way to make their lives easier. If it's so that you don't have to spend the operational cycles managing your own Lucene servers, what could you do with the cycles that we give back to you? And it's the same thing with Sprank Fusion and Score Fusion so that you don't have to manage that code base yourself so that if you could just throw it in as a verb in the query that you already have, okay, now what are you going to do with that, with that time that we're giving back to you? So Where it goes, community's gonna tell us what other kinds of things they want with that.

Read more novels is one candidate answer for what I'll do with my time savings. And going back to your book example, another thing I've struggled with at times is let's say I have the book, but let's say I actually have all the contents of the book too, right now I've got hundreds of pages of text associated with this kind of one entity. Now I can break that down into, I just read a novel that had acts, so it had like 4 different acts. And then of course there were chapters within the acts, and then there were sections perhaps, and paragraphs. And obviously you're down to sentences at some point. How do you think about, and I think there's some new work on this too, although I, I confess I don't grok it fully. How do you think about like how that raw text should be chunked or otherwise divided up to be represented in vector space. Is it like, am I looking for a single answer to that, or is it, do you think of it more like a trunk and branch and leaf kind of thing? I've experimented with that kind of stuff a little bit as well, but I don't, I guess I, I could, I don't really know what the state of the art is. In that realm today. Hey, we'll continue our interview in a moment after a word from our sponsors. You're listening to Deepgram Flux TTS.

Different voices, same model, all ready to speak.

Flux TTS is a streaming text-to-speech model built for voice agents.

Flux reads the room. It holds context across the conversation, turn after turn.

With consistent tone, interruption handling, and plenty of personality.

So your agents keep it flowing and customers can just keep talking.

Try Deepgram Flux TTS free now until September 12th. Visit deepgram.com/keeptalking. Terms apply.

Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable-5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration. Add layers of meaning, optimize syllable density, and above all, write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking, to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

Fair.

So there's a couple of, more broadly speaking, and I promise I'll answer your question. More broadly speaking, the use case for this level of search, for this kind of semantic or vector searching, is typically RAG pipelines for agents or for a more sophisticated agentic memory for agents. That's when you typically see vector search start to enter the conversation when it comes into an agentic architecture in some way. Specifically, what you're talking about here is chunking and chunk size. tends to be more in that RAG pipeline use case. I've got some external data, I've got some proprietary data. I want to inject that proprietary data into my LLM of choice without having to go through an expensive fine-tuning or training process because the LLMs are trained on public data. They're not trained on my proprietary data. So how do I inject that in a safe way that still gives me some data fidelity so that I'm not, I'm not giving away my proprietary data, but I can still get it to focus on my business problems instead of every LLM knows who the last 30 Oscar winners were. And that's not super helpful when I'm trying to get an agent to help me figure out what my NOC should do next, or my, my IT ticketing should do next. So that's the use case. So if you've got a bunch of documentation that someone who is a human professional in that world You now need to figure out how big are the, how big should the chunks be that I place in the individual documents and what's the fidelity of that? And when you're the application developer trying to figure this out, there's this trade-off that you're looking at. So if I make my chunks too small, let's say to the fidelity of a sentence, if I make my, I potentially lose the context in which that sentence appears. Just if you just read one sentence from a novel, doesn't tell you a whole lot. But if you then also read the paragraph or the page, it tells you a lot more about what the context of that sentence is. So that's an argument for, well, maybe I should make my chunks bigger. There comes a point though where if my chunks are too big, my storage costs start to go up and my retrieval quality starts to go down because If I've got 3 pages, let's say, and I'm trying to get to the fidelity of a sentence, I now have too much information and I'd lose the fidelity of that sentence. So what's the right answer? I, I've been doing this a long time, Nathan. The answer's always, it depends, right? So you have to go through, as a developer, you have to go through this iterative cycle. Like I chunk to a certain size, I run some tests, what's my retrieval quality? Okay, now I try it again. I try a different chunk size. I, Do some tests. I see what my retrieval quality is, and I have to do that 3 or 4 times that for my use case to try to find out what the right balance is between chunk size, storage costs, and retrieval quality. So one of the reasons we bought Voyage is because like those guys, like their side hustle is teaching people at Stanford how to, how to build LLMs. Those guys are all geniuses and they came up with this new concept last summer called contextualized chunking. And we've got a blog article on this that we can potentially link through here. But the idea here is that if left to your own devices and figuring out your own iterations of chunk sizes, there comes a point in a graph where if you've got retrieval quality on your x— on your y-axis, and you've got chunk size on your x-axis, with a traditional embedding model, there comes a point where at lower chunk sizes, you have zero context. So the retrieval quality is low and it, it builds. But then at some point it flattens and then it starts to degrade if your chunk sizes become too big. And that's why you have to go through these iterations to find the right combination. But what if you didn't? What if instead of sending that all as one big text blob, what if instead you send it as 2? What if you sent the fidelity of the sentence you actually want, and then as a second string, you send the size, you send the other contextual information. It's where the name contextualized chunking comes from. And we will figure out for you what the right chunk size is for that combination. And we will give you a vector array that comes back that balances those things for you. When you flip that script, it turns out you can get better retrieval quality with a smaller chunk size, which otherwise is not possible if you're doing it the traditional way. You, the way to get better chunk size, way to get better retrieval quality was with higher chunk size. But if you use this contextualized chunking, we released version 3 of that last summer, we just released 4 of that in the last 6 weeks, you can flip that script. Now there still are use cases where you need to control the chunk size for different things. But if that's something that you wanted to not worry about, or have to learn about, and the way I think about this is more developers will build AI agents in the next 3 years than did in the last 3 years. And in order to make that possible, we have to lower the learning curve, we have to make it easier For the developer ecosystem to learn how to do this. So if you don't want to have to learn how to go through the iterations of figuring out what the right balance of chunk size and retrieval quality is, instead, if you could use contextualized chunking, we'll figure it out for you and you still get good retrieval quality out of it. So that's, that's one of the benefits that Voyage offers that no other embedding model out on the market offers, that ease of use feature. That gets you better retrieval quality.

I definitely appreciate not having to worry about it, but I do wanna learn a little more about it. It, it sounds like this happens in sort of a test-free way at the level of the developer. So like, I don't have to bring a bunch of eval suite work to the table. So how, at the level of kind of principles, like how is it working in the background? And you said also I'm getting one vector back, right? So I pass in the sort of the chunk that I think I would really want to be able to zero in on and context, and that gets converted into a single vector back that represents both of those things in, I guess, some sort of superposition.

Yes, that's exactly how it works. So you get back one array of floats exactly how you did before. It's just now you don't have to go through the iterations of figuring out the size of the chunks you need. And similarly, there's another Voyage feature that all the Voyage embedding models have called Matryoshka reasoning. Because the other place you have to make a decision about storage cost versus retrieval quality is with number of dimensions. So what do I mean by that? Everybody knows what 2 dimensions is if you've taken high school level algebra, right? X, Y.

Right.

But in an embedding space, you tend to have at least 256 dimensions and sometimes as high as 2048. And the more dimensions you have, so each dimension is represented by one of those floats in that array of floats. The more dimensions you have, the richer your embedding space is and the better retrieval quality you get, but at higher Dimensions, that doesn't come for free. You've got a storing 256 floats takes up less space on disk and in the index in memory than it would for 2048. So you again have to go through this iteration of what's the right number of dimensions for my use case, given my, what my storage costs might be. All the Voyager models have a feature in them called Matryoshka reasoning. It comes from the Russian nesting dolls. If you think about how Russian nesting dolls work, right? You've got— You've got one doll and you open it up and there's another one exactly the same but smaller, a smaller fidelity inside. And then you keep doing that over and over. So what the— how the Voyage models work is when we generate, let's suppose you did it 1024. Suppose you wanted 1024 dimensions. You run some tests. Now you want to try 512. With a traditional model, you have to run your entire corpus of data through a second time at 512. But with Voyage models, you don't have to do that. When you've run it through at 1024 vectors, those floats, they're ordered. So if you want to try 512, you just lop off the last 512, and that's— now you immediately can begin testing on the remaining 512. So again, it doesn't completely solve the problem of figuring out what the right combination of retrieval quality and storage space is, but it helps you get to the answer faster. So these are, there, when you take some of these things in sum, there, we've got 3 or 4 of these kind of features that make it easier to use and help you get to your final answer more quickly. And the idea there is to give you time back in your day that you can work on your business logic instead of figuring out the plumbing.

Yeah, that's cool. I'm a huge fan of Matryoshka anything. Uh, when does this stuff become necessary? So for me, I'm basically a business of one and I try to be an early adopter of everything that I can. And I do have a pretty well-working, uh, I call it deep context, but basically a retrieval system that allows my agent to go into kind of all of my history from the last 5 years, essentially. Which is emails and Slack messages and everything I publish online and DMs across all kinds of channels. The podcast of the transcript of the podcast, I should say, diaries. So it knows what I've said and what the guest has said. It adds up to about a gigabyte in my case.

Sure.

And I haven't really optimized it much at all. I just kind of let the agent throw it into a database of its choosing and put whatever Optimizations on it. It felt like it needed to. And then we did get a, at one point it was like, well, yeah, we could probably do better than keyword. So we've got an embedding layer on there as well. I think, full disclosure, I believe I used the Gemini embedding model for that. Um, but it's like not very well optimized. How would I know if I'm really missing out on something? I, I don't have like a huge eval suite, right? It's just like, I'm kind of vibing it with, it doesn't seem to be working well. Is it a matter of like data scale, scale of users? Is it about like cost? Well, I want to optimize my inference cost and that's where I really need to get serious about how much data is, is being returned. Like what are the, what are the thresholds that people are obviously larger organizations cross where they're like, okay, we can't really do it the let the agent choose its own adventure way anymore. We really need to get serious about some of these optimizations.

Well, I'd return to the 3 things I mentioned in our first 20 minutes or so. It's speed, scale, and retrieval quality are the main 3 things. What most people do, and you're not going to offend me if this is what you did, I mean, most people start with Postgres and pgvector, and then they choose their embedding model with whatever cloud they're using. Like Gemini is prominent if you're going to use, if you're going to be on Google, just like OpenAI's embedding model. Is, is pretty popular over on Azure because of the historic relationship that those 2 companies have. But there comes a point in when you're, when you're doing a demo, when you're doing a POC, it doesn't always show itself. But there comes a point where when do milliseconds matter to your use case? When does scale matter to your use case? And that typically comes depending upon chunk size, that typically comes at about 100,000 vectors. Is when is what I mean by scale. And when does retrieval quality matter? If you look at Voyage AI model, Hugging Face has a benchmark out there called RTAB that Voyage AI models are typically at the top of, and we can get as much as 14% improvement compared to some of those other embedding models that we just mentioned. So are there use cases for which a 14% difference in embedding model quality is the difference between a hallucination and a correct answer. And that's before you even start putting re-rankers on it, which is another way that you can boost retrieval quality without having to do anything special to your data. So like I said, it's speed, it's scale, and by scale I typically mean in the neighborhood of 100,000 vectors, and retrieval quality out of your embedding model. Most people think embedding models are commoditized, and that is not true. There is a very big difference that you can get in retrieval quality based on what embedding model you choose. Anthropic does not have an embedding model in market. They recommend us.

It's a great recommendation. You mentioned re-rankers, and, and then there was also this kind of earlier concept of sorting results out of the database. This brings to mind this concept of bitter lesson engineering that I think is kind of growing in prominence, which from my simple point of view is just like every so often you should probably kind of go through your stack and look at all the kludgy extra things that you did to make things work and say like, which of these do I no longer need? Because the model got smarter or the embedding model got better. And so things are just kind of naturally working. Now, or could naturally work now, where in the past I had to kind of do all these artisanal craft sort of things to, to make sure that they worked. Are you seeing examples of that in the retrieval space broadly, where things are in some ways getting easier, or have you not really seen the bitter lesson applied to these pipelines, these production environments?

Yeah, I think there's 2 places that I've seen that recently, and one of them is with the re-ranking that we were just talking about. So in the spring, we released $rerank, which is a companion to the Score Fusion and Rank Fusion that we already talked about. So those 2 relate to doing a hybrid search. The $rerank, again, you would, in a historical way, if you would have to do your vector search once, send your results to a re-ranker to get them reordered in a way that is most optimal for a RAG use case to then pop them into the context window. Just like we did with the score fusion and the rank fusion, we've now got a stage where if you do $rerank, you call the API once, we'll do both of them for you on the backend so that you only have to make one round trip to the server for that. So that's one place that we've seen some additional ease of use. The other one is a feature that we released recently called auto embeddings, which is you tell us— this is a better together story— you tell us what, which collection, you tell us which attribute on documents in that collection, and you tell us which Voyage model and how many dimensions you want, and we'll take care of the rest of it. Anytime an existing document with that attribute in it changes, We will automatically take that new, let's say it's text, put it through the embedding model, update the vector in the document, update the index in memory, and we'll do all that for you. If a new document shows up into that collection that has that attribute, we'll go through the same cycle for you. So again, trying to remove some of the plumbing so that you get some time cycles back as a developer so you don't have to, you don't have to go craft your own and maintain over time. your own embedding pipeline. You just pass us one JSON WAD and we'll do it for you.

Zooming out a little bit from the database to the, the broader systems that wrap around them, you've been talking stuff about memory for AI systems. And this is another thing where I've seen kind of the great complication and then somewhat simplification, probably simplification. has been enabled by things that enterprises are maybe a little reluctant to pay for at scale. Like when I get a million token context window, I'm like, great, now I don't have to worry about all this stuff. I can let the model handle a lot more of it. But I do take your point that obviously you got thousands of users that can add up. How, how, what would you describe as, and of course it depends, but trying to abstract away from some of the details and give me kind of a center of the distribution description. What would you say is like the state of the art in what a good memory system looks like today?

Sure. So I, again, history matters here, right? When ChatGPT came out in 2022, at the end of 2022, it had a very simple application architecture. You took, you took the query, you put it in a context window, The LLM would process it and give you a result. End of story. That's how it worked. And by spring of that year, by the next spring, you would see headlines like ChatGPT passes the bar exam, and it would have these very cool things it could do. We've already talked about why you need RAG, because those LLMs even today are only trained on public data. If you want to inject proprietary data into it, you need to, to specifically do that with a RAG pipeline. What we saw in 2025 though was the second problem that we see with, with the core LLMs, and that gave rise to the need of memory. And that is like in spring of '23, if you had asked ChatGPT what's MongoDB stock price today, it couldn't tell you. And the reason it couldn't tell you was the LLMs have a knowledge cutoff date. There's a date after which they don't know anything about the world because the model vendors at some point have to stop training and put them into inferencing mode. And the way that we overcame that in 2025 or so was with tools and with MCP and being able to have to grant to the LLM, hey, here's this API, or here's this tool we could call to go get additional information. That's why whether you're a Claude person or a Gemini person or a ChatGPT person, that's why today if you ask, hey, what's MongoDB stock price? It could tell you because it can find a tool in its arsenal, go make a web Search, get that information, and report it back to you. The other thing we saw in 2025, though, is this notion of looping, where the result of one loop of that agent then became the input of the last one. The context window is still very dumb, right? The context window is still— you have a fresh context window every time you pass it to the LLM. And that's when we needed agentic memory. And the early passes we saw of agentic memory was Short-term memory was typically, okay, take every response that we got within this session and cram it into the context window so that we can see what happened earlier in this conversation. And then long-term memory was, okay, now do the same thing for every session we've had for the last 3 days and cram that in there as well. And then it had 2 negative side effects that I now see people starting to push against. Negative side effect one, we already talked about, is token maxing, right? If you, if every agentic loop puts a million tokens in the context window, even context you don't necessarily need, your cost starts to add up. But there's also a functional cost to that. There's a couple academic studies that show that like the first 7K and like the last 7K tokens are the most important and the stuff that's in between can end up just muddying and confusing the LLM as it's trying to come up with an answer. So instead of thinking of how could I maximize, how could I cram 100, a million tokens into the context window every time? What if I could choose just the right 200,000 for this agentic loop? So we're starting to see more sophisticated short-term memory that doesn't take everything in the session, but just the things in the session that are relevant for this pass. And the same thing with long-term. And we're starting to see some new memory types, especially in Fortune 500s. There's a new memory type called taxonomic memory. So think about either in a vertical, so like automotive we were talking about before, or in a specific company, there's some terminology that is specific to verticals or specific to a particular company that a layperson who is not in that vertical, those words may mean different things. So you might have a list of, let's say, 100 of those terms. But for this agentic loop, what are the 5 that are relevant for this loop of the agent, and then do that again for the next loop. What are the 5 that are relevant for that loop of the agent? So again, instead of taking all 100 terms and putting them in the context window every time when not all of them are going to be relevant, what if you could instead have a more sophisticated memory type— memory system that instead of just taking everything, use vector search or semantic search or some combination of vector and lexical with some pre-filtering and just got the, just the right, just there's some perfect, you'll never get there, but there's some perfect context window for this pass. And that perfect window doesn't use all of it. It uses just the stuff that it needs to. Those are the kinds of things that we're starting to see in enterprises, both to combat the cost of the tokens and to combat the— relevance of the tokens so you can get the best answer out of this call to the LLM.

And do you have any tips for maintaining these memory systems? I mean, this kind of goes back to some of the challenges with, and maybe there's good solves within MongoDB for some of this. If so, definitely highlight them for me. But I sort of struggle with the lack of normalization or my failure to observe the DRY rule in my own memory system sometimes. I'll have like, you know, I have these like raw exports that combine my email and DMs and everything else right into these kind of monthly log files. And these are typically like a couple hundred thousand tokens for a month. Then I'll like summarize those into a monthly summary. And then I'll take the monthly summaries and I'll put those into a yearly summary. And then I've got like a wiki type Karpathy-inspired structure that sits on top of all that and is like more entity-based and they kind of link to each other. And some of this lives in a database, some of it lives in a file system. But now I've got a challenge where I'm like, some new stuff has happened. Okay, what do I do? What do I change? How do I make sure that it's changed everywhere it needs to be changed? I sometimes also have instances where I've, I've had really funny, I mean, I think the models are getting better where this is less of a problem these days, but in setting this up in the first place, I had some problems where the models would see a project that I like experimented with a little bit or like was maybe gonna pursue and then didn't. And it will like keep it as like an open thread for months and months. And I'm like, dude, I never actually did that. Can you just forget about it? So forgetting is also sometimes a, a real virtue of memory system. Um, How, what are the, what have you learned? What tips would you offer in terms of how to maintain these things so they're accurate, not just when you first create it, but on an ongoing basis?

This is the hardest problem with agents right now, I think. And the way that I put this to people is we've, as an industry, we've been building databases for 60 years. We've been building agents for like 18 months, man. Like nobody knows all the answers yet. We're all sort of collectively figuring it out. But what I do see with these more sophisticated memory types is that as the application author, that you have 2 responsibilities with these memory systems, where with the prior one, you only had one really. So not only do you call this memory system, hey, here's my query, give me the best, maybe you give it a token budget is the way some of these work, so that you don't, you don't have to figure out how many memory types there are and how to query them. You're just like, gimme the best 50K tokens or gimme the best 100K tokens.

Mm-hmm.

You give it a token budget. And then it gets it back from the mix of different memory types it might have. And then you put that in the context window, you get your answer, and then your second responsibility is to send that answer back to the memory system so that the memory system can then curate it and then write it into the memory system. And when it writes it into the memory system, the more sophisticated ones also have this notion of RBAC so that if you and I are in the same job type, that we get to share.

Mm-hmm.

The memories, so that if you come up with a really good memory and then I get to reuse it later, then that ends up being beneficial for both of us. So that's the second responsibility you have. But the hardest part is what you're talking about here, the way that one coworker just wrote this to me yesterday. I want to quote him, write, change, recall, forget, because these things have a, a half-life to them, right? Like things that are more recent are more important than ones that that took place weeks, if not months ago.

Maybe you can tell me about any like Mongo-specific tricks or advantages for these sort of graph structures that I seem to keep making. I have people and the organizations they work for and the ideas that, that I sort of associate them with. And these things all kind of point to each other. But they do it in a pretty loose way where I'm, I'm right now kind of trusting the agent to hopefully notice those pointers and to the degree I'm like running maintenance, hopefully follow those pointers and do the necessary maintenance. And I don't have a lot of guarantees. And I suspect that there are better technologies that I could be building on that would give me a lot more robustness.

Well, there's, there's a couple of different ways to tackle this. Like I said, the forget part, is the hard part of it. But what we see our customers doing is this is where like the retrieval quality of those memory types, that's where that non-commoditized embedding model can make a difference and where a re-ranker can make a difference. You can get, depending upon the use case, you can get like a 5 to 10% boost at your retrieval quality just by using a re-ranker on top of whatever your embedding model of choice is. And that's true of architecturally, that's why we made it so easy to implement the re-ranker on top of the Voyage embedding models when you're doing that vector search. But there are use cases where maybe you have so much data that you might have to take a hybrid approach where maybe you use, maybe use a graph structure for, I don't know, 2 to 6 levels of data. And then you get down to a leaf and you do a vector retrieval inside that leaf. We see people doing that as well. There's all kinds of information you can find on our website about how you can, because we're JSON-based, you can use MongoDB to build graph structures into your data so that you only have to go one place for a graph database, a core database, a vector database, the embedding, re-ranking, all in one place. So you don't have to try to stitch together multiple tools yourself and have to maintain that over time. But we do see people doing that with graph structures. It's typically not as deep as like you would traditionally think of as a graph database need. But like I said, it's pretty common to go, I don't know, half a dozen or maybe a dozen layers that maybe for bigger corpuses of data that are more heavily categorized, like some of our retail customers do that with product databases. If you think about how a hierarchy of products might appear on a website or on a mobile app, They might segment that first by product category using a, using more of a graph style. And then once you get to a particular product category, then do some vector searching within that individual category node.

Are there any applications that you would point to as just being great examples of memory well implemented?

Well, our biggest customer right now, they haven't been very public about how they did it. There's a company called ElevenLabs out there. They have multiple agents per customer. This sort of micro-agent approach where you get multiple smaller agents at the disposal for individual customers and given the number of customers they have. So they, ElevenLabs started life as a model provider that was doing way sophisticated speech-to-text and text-to-speech. And they built a platform on top of that that is more of like an audio editing suite that they do for that. And as part of that, they have a bunch of agents doing all kinds of editing and transformation kinds of things for their customers. And if you think about the kind of context memory that they have there, an important way to do that as well. I would worry about any memory architecture that instead of relying on lower-cost embedders and re-rankers is relying on multiple passes of the LLM to help you categorize and shrink the corpus of data that you might want to then put into the context window. For the larger, sort of more functional LLM call, because then that's just adding up tokens as well. I mean, that's why things like embedders and re-rankers have a lower token cost is again, right tool for the right problem.

Yeah, that adds a lot of latency too at this point. Even the small models can reason for quite some time before you actually get your answer from them.

They can. And so all those things add up over time and it's not just tokens, but you're right, the overall latency for the decision loop that you're into makes a big difference.

I think you also have some interesting takes on build versus buy analysis. I won't even try to summarize it. Just gimme your, your hot takes on how people should be thinking about building versus buying. What is mature enough in the AI realm to buy? And what do you, even if you kind of buy part of it, what do you still have to expect that you're gonna end up building or customizing enough that it kind of feels like building?

Yeah. So I, I have the good fortune in my job, I've been to 7 countries this year to talk to probably 100 different customers about where they are in their AI journey. And they tend to fall into sort of 3 camps. Camp 1 is I bought a license for this one tool and I'm done, right? Like my AI strategy is done. I bought one thing. And while that can be a good starting point, it typically is not specific enough to help solve the problems of your overall enterprise. Then you have a group of people who have done some POCs and maybe a couple of production deployments with varying degrees of success on ROI. Those that have struggled with ROI, I always argue, probably picked the wrong problem. Picking the right problem is really important in this space. And the way that I encourage customers to think about this is like, what are the top 10 to 15 problems that you have going on in your business right now? And of those, what do you have good data for? I mean, you made a comment earlier in this conversation, like data quality is a big deal. Things like bad data quality and bad security posture don't get solved by AI, they get amplified by AI. So what's your, what's your biggest problems? What do you have good data for? And then what do you already have metrics for around the problem? And that's the biggest difference that people tend to skip over when it comes to problem selection. Because if you don't already have metrics for how something is performing, you won't know if it got better. So call center use cases are very popular, low-hanging fruit in enterprises because I already know what cost per call is. I already know what call volume is based on how I'm already bonusing people in those jobs. So if I introduce AI into their workflow and I see those numbers jump, I can attribute that change to the AI and I can do a back-of-the-envelope ROI. We see the same thing with software delivery lifecycle. Early in the year, we saw all kinds of bragging about, well, now because I've got Claude Code or I've got T-Gravity or I've got Codex, now I can produce 5 times as much code as I did before. And anybody who's been doing it any length of time knows that lines of code is a terrible metric to judge the productivity of a set of developers off of. Instead, How fast are you getting from idea to production deployment? That's the metric that matters there, not necessarily lines of code. So the metrics there matter. And that's the difference between the folks that are in that middle state, either stuck in sort of POC purgatory and haven't quite got to production deployments. The biggest difference isn't in how they're applying the tech, it's what problems they chose to try to solve. And then you've got people that are sort of more advanced that are looking at some of these more sophisticated memory types and things where they're trying to optimize the systems That they have. And in some cases, they're, they're making purchases of larger platforms to help them do that. Other times they're doing homegrown. And like I said, we're, we're like 18, 24 months into this. There's, there's no, there's no one way to do this yet. There's, there's no LAMP stack for agents yet in the way that we have with web development. We will get there. Having lived through that, having lived through that lifecycle, we'll eventually get there. But like there's, there's no React and Angular, there's no LAMP stack for agents right now.

You mentioned call centers and this is the second time today that this has come to mind. I just ordered a pizza last Friday night from the local pizza place in my neighborhood. Shout out once again to Greg's Pizza in Detroit. I think it's just a one-location business. And who answers the phone?

Detroit-style pizza is underrated, by the way.

It is. This actually is a traditional circular round pizza, but But I agree, after auto, it's Detroit's biggest export to the world is pizza. But even just this one location business, right? I call and who answers but an AI agent. And it was a pretty impressive experience, I have to say, like very natural conversation, not flawless. There were a couple little things where, especially as I started to get a little adversarial in my testing of it, like I asked it what company Is powering this AI agent experience that I'm having. And it took that as it somehow thought I was ordering pizza for my company. So, you know, there's some kinks still, but I overall was like very impressed. Like the happy path I thought worked really well. And this has me thinking like, boy, there must be just a ton of transformation happening at call center businesses. How, how much are you seeing? Like, there's always this kind of waves of change, right? Where like the first wave is like, we give our agents A tool where they can like get help faster and be more accurate. But then there's this like next wave, or there might be several waves, but certainly one that seems like it's coming is you just call and talk to AIs, right? How far along that journey are the customers that you are talking to? How much like actual disruption have you seen in that part of people's businesses so far?

Most the Fortune 500s that I talk to, are doing employee-facing use cases with human in the loop for exactly the reasons that you just laid out and for some of the ROI reasons that I laid out before. So if you think about the employee-facing use case, you already know what you're bonusing your different employees are and what KPIs, what key performance indicators you're using to judge whether or not they're doing a good job. So like I said, I used the example of call centers before, but you can think of any job, whether you're an insurance adjuster or a line worker in a manufacturing plant or whatever your job might be, everybody's got some key performance indicators. And if you introduce AI into whatever that workflow is and you see those jump, that means that you're, you're, you figured out some ROI. But there's also security, a data security issue for that as well. If you and I are employees for the same company, is it great if my salary accidentally leaks to you? No. But it's way better to have that than if we are 2 customers of the same company and I get to see your data. That is, a VP has to get on a plane for that kind of scenario. So like the risk reward is higher for customer-facing use cases than it is for employee-facing use cases. So at least where we are now in the ecosystem, where I mostly see the Fortune 500, with exceptions of course, But I see a lot more work going into those employee-facing use cases with human in the loop than I see going full autonomous AI with customer-facing use cases.

For you in, in what you're doing at and for MongoDB, are you, you've, of course you've got traditional documentation, right? That human developers can go and read. But I'm sure you're also thinking a lot about how do you make your documentation AI-friendly? And then how do you have an AI that can proxy into that documentation? And then potentially you're like packaging up and distributing skills and trying to think about like how to become the database that is like recommended by the agents when somebody's doing something for the first time. How is, how is your role as like bringing this technology to the world evolved in light of all the different ways that people are discovering and starting their implementation process?

Sure.

So all the things you just mentioned are certainly things that not just us, but every software company is involved in one way or another. The 2 that I would shine a spotlight on are the MCP tools to make it easier for agents to talk to your data in your MongoDB instances. We had an MCP server for a while and we, we're now kind of self-hosting some of those as part of Atlas, as part of the ecosystem. And then the other one is the agent skills. So think of that as kind of curated system prompts for doing things like data modeling, doing things like optimizing operations, things that we have well-organized playbooks for that previously we would have as documentation, but now we have them as a series of doc— of markdown files that you can feed to an agent of your choice. to give it sort of the lowdown on different ways that it should approach different tasks. And there's maybe the, I think it's agent skills we just dropped a couple of months ago has like the top 8. It's like it's 6 to 8 different agent skills that we have for things like operations and data modeling to help you do those sorts of things. So yeah, that's become pretty commonplace these days and we're certainly part of that ecosystem.

You mentioned this acquisition. that MongoDB did of Voyage a couple of times. And I had a kind of just a couple like theoretical questions about that.

Okay.

One is, it's striking that Mongo built its own vector DB, right? And there's like a bunch of vector DB startups out there. So like one could have imagined an alternative scenario where There was an acquisition of a vector DB startup, but there wasn't, there was an acquisition of the model startup. And then also just look at the relative value. And you said these guys are geniuses and I don't doubt that, but the price at even at $220 million, which is, I'm old enough to remember when that was serious money, you know, it's, it's less than a 10th of a percent of MongoDB's overall market cap. So I'm like, Or less than 1%, I should say. So I'm just kind of wondering, like, what should we infer from these observations about value and defensibility in the software business over whatever passes for the long term in your mind? You know, you could tell a story about like infrastructure is the big winner, models get commoditized. Incumbents can defend themselves against startups? Like, what do you think are the right macro lessons to draw from this experience?

I mean, fundamentally what we've always been about has been how can we make the day in the life of a developer easier so that we can make it easier for them to build their business logic and spend less time worrying about the plumbing lower in the stack. I went through a description earlier in this conversation about how we implemented vector search. And for us, because we already had the JSON-based, It was, it was relatively straightforward for us to just, okay, you add an additional attribute. That attribute is your array of floats. You generate an index based on that array of floats. So that was pretty straightforward for us to add to the existing product and take advantage of the things like the sharding, the security, and the data replication that we already had in the base product that some of the net new vector database companies out there have to catch up and struggle with a little bit. So that's why we chose to sort of do vector search the way that we did is because of the flexibility based into, built into the document model from the very beginning. It was, it was pretty straightforward for us to do that, but where it was not as straightforward and why we made the Voyage acquisition was today you can still use whatever embedding model you want as long as it generates that array of floats. You put that array of floats in your document, you go create your index and you're off and running. But we saw an opportunity to, to A, that the market was seeing embeddings and re-rankers as a commodity, whereas we, we saw Voyage really standing out. And like I said before, Anthropic agrees with us. And we could create these better together stories like the auto embeddings that I mentioned before. Another one I didn't mention is being able in Atlas to manage all your API keys in one place so that you don't have to go to one console for your core data, another console for your vector data, a third console for your embedding models, and a fourth console for your re-rankers. You can get all that in one place. The management of it is easier over time. So if you look at the history of MongoDB features and really what our focus has been, we're all developers at heart and it's, it's about making it easier for the developer ecosystem to learn and operate this, these new techniques that we see in application architectures around agents, whether it's a RAG pipeline or whether it's the agentic memory. So how do you make, how do you make vector search more approachable? How do you reduce that, lower, lower that learning curve so that more people can learn it more quickly and start to participate in this ecosystem?

Another story that I've heard that I want to just get your reaction to and see if you think it's true in your experience is I was speaking to a founder of a vector DB startup one time, and I kind of made the case or put it to him that, geez, it seems to me like the incumbents are going to be able to add a vector aspect to what they're doing before you're going to be able to kind of replace everything that they're doing. And so it seems like they're going to have a hard— you're going to have a hard time, like, really displacing them. And his answer was, well, that may be true, but most of the data that is coming into our vector database has never been in a database before at all, right? It's just been sitting out in some data lake or data warehouse or just kind of unstructured piles of documents. And now it's coming into a higher level of infrastructure and it's being made more valuable in a way that just wasn't happening at all before.

Sure.

Are you seeing something like that? And, and what does that look like? I also noticed that there's the multimodal embedding model that supports video. So video would be potentially a great candidate for the kind of thing that has never been in a database before. Are you seeing this like phase shift of just like a much greater universe of data coming into Mongo than in previous eras?

We are seeing, we are seeing this broader ecosystem of data that, that wasn't indexed before. Because it didn't lend itself to a traditional lexical search. The, the cool thing about the vector search is, I mean, vector search at its core is taking some piece of data and mapping it into n-space, into some geographical, geometric space. And then really all a vector search is similarity, like where, what are the closest vectors to this new thing that I'm searching on? And video is a great example. Audio is a great example. Unstructured data. You just take a bunch of PDFs that you have sitting around in a SharePoint. I mean, those are good examples as well. So yeah, I think that there's there's some truth to that. That we're it's not just how do I take the data and put it into a database, but now how do I find it? And how do I find it in a way that is fast and scalable? That's a good quality. And that's why the way that we implemented vector search and being able to within the same platform have the combination. Of the pre-filtering, the vector search, and the lexical search. We feel like that gives us an advantage and gives our developer community more levers to pull from than what some of the alternatives are. And like I said before, because it stands on the shoulders of the core product, with the Atlas version, you can already deploy that in any hyperscaler data center you want with good data resiliency, with good security, And if you need to, to shard that data so that it doesn't leave particular geographies. We got all that for, for essentially for free because of how we implemented vector search on top of the core product. And yeah, we're, we're seeing all kinds of different kinds of data get put into those documents in a way that we didn't before.

You mentioned travels, I think so far to 7 countries this year. Where I'm, I'm so kind of myopically focused on what's going on in San Francisco and Silicon Valley. That I'm mindful that I may be missing important stories or differences in perspective that are going on around the world. I try to fill that gap, at least when it comes to China, but it's a big world out there. What has stood out to you in your travels this year in terms of differences of perspective on AI usage patterns, values, you name it, could be anything, but just kind of what do you think the U.S. Audience in our inward-looking way is missing that the rest of the world is doing?

I think the biggest thing, if I turn the question on its head just a little bit, there's a presumption in other countries that the US is ahead and doing things that other people are not. And I found the opposite to be true. So, I mean, I live and work out of Cincinnati. So US is one of those countries I've spent some time In both Amsterdam and London earlier in the year, but I just did a tour that included stops in Toronto, Bengaluru, Mexico City, and São Paulo. And the two most sophisticated customers I talked to this year were in Mexico City and São Paulo, and they assumed when we started their conversation that they had U.S. competitors that were doing things that they weren't. And like I said, the opposite was true. I think we've reached a point with some of these technologies that like geographic barriers don't matter nearly as much as they did during like the web app era or, or even during the cloud era. Because for the cloud era, like if one of the hyperscalers didn't have a data center in your country yet, you were kind of outta luck. That's not true anymore. Like pretty much every country has at least one hyperscaler data center in it, and by extension, Access to models and vector databases and embeddings and re-rankers, like access for these things is far better than what I've seen with previous technology revolutions that we've seen. So I think I understand why there would be an assumption that U.S. companies would be ahead since so many of the bigger AI companies are U.S.-based. But like I said, the two coolest things I've seen this year were in Mexico City. And in São Paulo. So I think that those geographic barriers to being ahead in the market are starting to disappear.

What do you think are the barriers? Why is that? I guess, is it that the American companies are culturally too conservative to run as fast as some of these international companies are? Is it that— is it like a leapfrog story where they sort of had The international companies, I mean, had kind of less recent technology investment that they would have to get comfortable replacing, or what's, what's driving that surprising observation?

Well, I think it has more to do with the lack of barriers outside the US than it does with any of the behaviors of the companies within the US. It's just access is democratized now in a way that it wasn't, that with it, it wasn't for the cloud and for the mobile waves, right? Like, If you didn't have the radio towers yet, an iPhone's not going to do you much good. But now we have all this infrastructure that's laid worldwide in a way that we didn't 15 years ago that makes access to these things far easier than they were before. So there's just, there's just a wider set of people in a wider set of countries that have access now that they didn't before. So I don't think it's any, it's not an indictment of the way that I think US companies are thinking about it. I just think the There, that greater access erodes those geographic barriers that we've seen with previous technology waves.

Any closing thoughts? Any, anything you'd wanna leave people with or anything I didn't get to that you would wanna make sure we touch on?

Well, there's one technology thing that we have I didn't get a chance to talk about, and that's shared embedding spaces. And then there's, there's also this notion of just like kind of where we are on this journey together. The shared embedding spaces, real quick. Is one of the other ease of use things that the Voyage models provide that nobody else does is in January we released 4 versions of the, of the version 4 of the text model. Everybody typically releases small, medium, and large, and at those different price, each one has a different price point and a different retrieval quality that comes along with that. But we also introduced something called the Nano, which is open weight and free for anybody to download in off of Hugging Face. And what Shared Embedding Spaces does is each one of those 4 models shares an embedding space, meaning that embeddings that are generated with one of those models, they're compatible with the other 3. And what that lets you set up is you could take your corpus of data and you could embed it using, say, the large. And developments Every other embedding model is going to force you to pay tokens during your development cycle to hit against their embedding model. So as you're querying, as you're taking your queries and you're passing them through the embedding model to do your searches. But with us, you can use the Nano there. You can run that on your laptop if you wanted to. So you could, if you wanted to, have zero token costs aside from the embedding of the corpus of data. But for your queries, you could eliminate your, your token costs during development in a way that wasn't possible before January. So that's not for everybody. You do take a little bit of a retrieval hit when you use a different retrieval quality hit when you use a different model. But for some customers that don't want to have to, that want to lower their token costs during development, this is one way that they can potentially do that. So that's, that's shared embedding spaces, which we introduced in January. And more broadly, I want to come back to the idea that we've been building databases for a really long time. We have not been building agents for a very long time. We're, we're gonna continue to see iterative improvement and it's gonna happen fast. Like, MC published MCP on like Monday before Thanksgiving 2 years ago, and by March, all of their competitors had embraced it as a protocol. Token maxing first got mentioned in like early March, and by April it was over as like a conversation piece. Like, The quickness, the speed at which we have these conversations and we go through these cycles of these different features, it just, it's faster than it ever has been before. So the fun part is there's still a lot we all have collectively to learn. We're gonna all collectively figure out how to make this easier for everybody. And you, you'll see more and more agents in everyday life as a result.

Well, as Field CTO of AI at MongoDB, I know you're gonna be racing as hard as you can to keep up with all these cycles, and we'll continue to look for new updates and more great progress from you. Pete Jansen, thank you for being part of the Cognitive Revolution.

Nobody keeps just one thing anymore. We used to have to choose. One picture for a whole December, one letter we couldn't lose. Now there's a record of every hour, boxes stacked to the beams. A house so full of yesterday, it's swollen at the seams. Trying to forget is the hardest part, so I held on to it all. Every summer, every Borrow a song and I can't find you at all. I kept everything, everything but you. I kept everything, everything but you. I went looking for you Sunday down the rows of all I own. Everything lies with its own kind. Everything but you has a home. Your laugh is in a box somewhere under years I never sorted through. I swear I never lost a thing. I just can't I can't forget you. 'Cause forgetting is the hardest part, so I held on to it all. Every summer, every borrowed song, and I can't find you at all. I kept everything, everything but you. I kept everything, everything but you. Last night, looking for my keys, I found that Ferris wheel, a ticket folded in a roadmap next to songs you used to steal. Nothing in this house is gone. It all stays close to what it knows. I quit digging, started following, and the pile became a road. Now when I come looking, I don't tear the whole house through. I start with one bright summer and let it walk me back to you. One thing beside another, each one heavier than the last. Hands me one more down the road, right to your laughter, right where it was before. 'Cause forgetting is the hardest part, and I never needed it at all. Every summer, every borrowed song is a light on down the hall. I kept everything, everything leads to you. I kept everything. Everything leads to you. Everything leads to you. Down the road to Foundry Hills. Everything leads to you. Down the road to Foundry Hills. Everything leads me home to you.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.