AI UX needs more innovation, right now it’s embarrassing - Jeff Huber, Founder and CEO of Chroma

You've got ChatGPT, AI boyfriends. There's sort of this whole like, you know, consumer angle. And then the thing that has hit the most thus far has been coding agents.

It seems now obvious that like coding was going to be like the first dominant use case for AI. It's just such a perfect playground, you know, deterministic software that like it really lends itself well to being able to hedge some of the disadvantages of the non-determinism of LLMs.

We work with a 90 million token TypeScript monolith. There's nothing like that in Sweetbatch.

The amount of innovation inside of AI UX is is like kind of embarrassing. People should do a lot more. And if you're listening to this, you should do more.

Welcome to the first, you know, the inaugural meeting of bald guys in technology working on agent frameworks or agent adjacent things. Such a pleasure to be here.

BGTN, something like this, right? That's the acronym.

I have a guy I know at one point he said, oh yeah, I know Jay from the JBBL. Was it Jeff? The Jeff Bezos lookalike club? I was like wearing a vest or something like that.

Well, it's a compliment. I think it means you've been hitting the gym too, you know.

Exactly. Yeah. And I look like a billionaire. So I think that's— it's a great place to start. Such a pleasure to be here. Thank you for hosting me. You know, it's a wonderful studio we've got. We've been kind of going back and forth on Twitter for about 2 years, I want to say. Yeah, 2 and a half years, sort of since the very dawn of like the GPT-3 API era. So I'm super excited to get your take on a bunch of stuff that relates to agents, memory, storage, things that are sort of non-AI adjacent, but like database construction. I'm also very interested. To get your takes on that and see where the conversation takes us. So, yeah, maybe a good place to start is AI application database. That's sort of the term that's on your website. What is different about an AI database compared to Postgres or MongoDB or anything that, you know, developers more sort of traditionally used to?

One way to describe kind of the modern instantiation of Chroma, you know, obviously you're changing the messaging to reach different kind of generations of developers. Chroma is building modern search infrastructure for AI. Moddery is in sharp contrast to a lot of traditional databases and systems that are really hard to scale, really expensive. You have to think about capacity planning. You have to size your nodes and do a bunch of like workaround backups and warm standbys and just a bunch of like operational overhead. A lot of modern distributed systems now like sort of abstract away all of that complexity and just like give developers a much easier, uh, and more cost-effective, um, while still being very performant. Solution. So that's like the modern piece. And then for AI is in some ways like the more interesting piece of this conversation. We basically believe that there's like 3 things that that means. Number 1, it's for AI developers in contrast to, again, traditional like search infrastructure, which like had to be a sort of a pointy-hatted— pointy-hatted? Is that a term? Search engineer to go and like set up and implement and maintain. Now everybody has to solve search problems to bring the right context to agents. Number 2 is AI workloads. In the past where you would only really index and make searchable certain pools of data, which were like very high value, that lots of people needed to query a lot of the time. If you think about our jobs day to day, a large part of our jobs is information retrieval. I would wager that you and I are spending 15 to 20% of our day-to-day lives just doing sort of information retrieval. Imagine all the different systems you touch, all the different data you touch, right? You have a map of that. You have an index, you have routing, you're figuring out where to find stuff. And so I think the same is true as we want to move more and more labor to agents is all that data needs to be legible to, accessible by, and searchable by agents. The third thing is AI features, which is, you can kind of break this down a bunch of different ways, but I'll give you one simple example to start, which is that in the past, like, your recall at 10 blue links or 10 results really mattered because that's all that a human could digest. Your recall mattered, but also your precision. Um, now with LLMs, uh, precision matters much less, but recall is actually more important an LLM can digest hundreds of results potentially very easily and cheaply, but you need to make sure that within those hundreds of results, the right answer lies. I think there's a bunch of other really interesting things that both our customers are working on, we're working on, where you can really use the LLM sort of as a query planner. It's one way to think about it. The LLM can decide and understand the intent, and they can build the query to go and get the right information out of the system. And again, these are things that like are only possible now, like because like LLMs exist.

That's really fascinating. I've seen an application in doing sort of successive SQL queries that's sort of like this non-deterministic query planner where it'll run a query, it'll like look at the schema of a SQL database and then it will decide, oh, based upon this, I should go query this. And it's sort of like running a non-deterministic SQL query in a way where there's just like determinism interleaved with non-determinism where it decides what the next thing is. It's a very exciting idea.

Broadly speaking, the job of most kind of AI engineers really is context engineering. And you gotta figure out, you know, how to wire up all your systems to like give the agent the access to the right pools of data, teach it how to use those things. You know, in some cases that's gonna be structured data inside of relational databases, data warehouses. In other cases, it's gonna be tools and APIs that the agent, you know, needs to know about, have access to, know how to use. In other cases, it's gonna be unstructured data. You look at like the volume of unstructured data, in most enterprises, it's already 100x, 1,000x structured data, and that's only going to accelerate. You think about like LLMs running all day, every day, like what are they ingesting and what are they outputting? It's more unstructured data. Obviously we think a lot about at Chroma, the unstructured data side of things, but you know, there's not a silver bullet here. And again, this is engineering. We have to do the hard work.

So you mentioned context engineering. To me, that's meant many different things over the course of the last, call it 3 years or so. Um, you know, at the very beginning you could only fit maybe 4K tokens. I think GPT was like 4K initially, GPT-3. And yeah, fast forward to now, we have a million tokens on Claude Sonnet announced yesterday. So can you walk me through how the sort of the art of context engineering has evolved over time and how you see it sort of evolving in the future?

Yeah, in the beginning, as you said, the context windows were teeny tiny and you could do like, you know, very small amounts of work. Um, you know, now I think the largest like commercially available model is like Llama 4 Maverick. That's 10 million tokens in context window length. And then, you know, certain startups have sort of at times, you know, made the claim that they have 100 million tokens or a billion tokens or infinite tokens. You know, one thing that Chroma has done a bunch of work around also, um, for the people that are listening, you can go and check out our research technical report on context rot, which is the question of like, well, you say that you're giving me a million tokens, but how well can you actually use those million tokens? It's a really important question and a question that like the labs are not particularly incentivized to help you think of that question in the first place, nor really to answer that question. But we did the work, we did the research, we had the proof. And, you know, most developers who are practitioners that are really building stuff every day already had an intuition about this. You hear from developers all the time that like, yeah, push past 100,000 tokens with any of the Claude models, things get a little squirrely. And so like, it's a little touch and go, isn't it? Yeah. We just like, I don't know if we want like reliable and performant systems. Like we're not looking to add more nondeterminism. Usually. And so our research shows that across both the ability of the model to pay attention to large parts of the context window and the ability of the model to reason over that context, that we see quite steep degradations as early as 5,000 or 10,000 tokens. And so again, for your use case, it's going to change and vary, like how much reasoning power you need and how much attention you need. But I think these problems are not going away. I think that it's fundamental, potentially, to the transformer and just how these models are trained. I hope that it does get better because it's kind of a pain in the butt to work around, admittedly. But like, I think, you know, for anybody building today, you need to both skate to where the puck is going, but also understand like what the reality is. I guess just in summary, like, you know, we're going to run the analysis on GPT-5. We're going to run the same analysis that we ran on the new like million token Claude model. We'll see what the numbers say. It's a pretty stable bet that like, at least for the next like, you know, 3 to 5 years that like developers really do need to like embrace and get very good at context engineering. If anything, just for speed and cost reasons. Reasons, not even just accuracy reasons, but like, you know, your classic hierarchy of make it work, make it fast, make it cheap. We're still very much at the make it work phase. And like, you know, I think you have to engineer context window to make it work.

For sure. Well, intuitively, so you mentioned that you don't think this problem is going away. And I remember seeing the needle in the haystack setup you had where it's like, I don't know, you put the entire Lord of the Rings series in there and then there's one line from Harry Potter and you say, can you find me the line that's from Harry Potter? Maybe this isn't from your research specifically. That seems to me like the type of thing that you can produce infinite amounts of synthetic data.

Around.

And so if you wanted to run some type of an RL loop, you could really just grind this thing down to the absolute bearings. If you said, find the Harry Potter line for me. So do you think that that's actually going to cap out and we're not going to be able to defend against this issue where it won't be able to like find the needle in the haystack?

A few things. First of all, that's not needle in a haystack. Needle in a haystack is actually strong lexical matches. So you brought up more of an example of a semantic match or even like a categorization matching thing where like, oh, this sentence, maybe it doesn't even say Harry Potter, right? Or doesn't even say Voldemort. It's like, just, I should sort of know based on my training data that like, that's from Harry Potter. Like true needle in a haystack is like one example of this is like the question would be, what did my college professor say is the best thing to do to get good at writing? And then the needle will be, my college professor said the best thing to get good at writing is write every day.

Interesting.

And like extreme lexical matching, like it is a zero reasoning task situation. Even if you ablate that and you just do a simple semantic hop where like you say instead, you know, How do I get good at writing stories or something like this? You see actually, you start to see degradation. So the needle in a haystack strong lexical matches is solved across all token window context lengths. Again, it has zero reasoning power and also has very limited attention. You just pay attention to a needle. I want to just quickly add that going back to the question about RL, obviously RL is incredibly hot right now. It's all that anybody can seemingly talk about. I think there's two questions there. Number one, What can RL do well? And the second question is like, what can the models absorb? And like, what can they absorb without forgetting? And I don't think the jury's out on that, right? Like, we don't actually really understand, like, from an interpretability perspective, like the limits of these things. If you probably can make a model that is extremely good at like, you know, categorizing Lord of the Rings content versus Harry Potter content, will you lose everything else in the process? Maybe not everything else, but most of it.

So if we look at the general distribution of like any possible book and there's like a one line from Harry Potter thrown in there, maybe, you know, it's unclear how good we're going to get. I think when you typically look at long context loads that we're seeing in production today, it's usually like very long agentic rollouts.

Correct.

And I'm curious, do you have any thoughts on like in that specific sub-distribution of data, which is just agentic long-term rollouts, is it better or is there any evidence to demonstrate that like we're better at remembering stuff?

I'm very bullish that like more on a percentage basis, the percentage of fine-tuned models, RL or otherwise, the percentage of workloads served by those models will only go up year over year, probably infinitely, probably in the next 40 years. And today it's quite small, right? Most people are just using base models, maybe not base models, but they're using the RLHF chat models primarily. I don't know the percentage of models that are fine-tuned on a specific task, but I think it's probably single-digit percentage points, if not lower, maybe sub-1%. I think that will go up year over year. And so I'm actually long purpose-built models. I think that's the best way to get to the level of reliability that we want. The part that I'm skeptical about more so is that that's like a panacea. I'll give you one example. We were doing some experiments and we actually continue to do more experiments with Sui Bench. And we looked at long context in the context of Sui Bench, right? Sort of genetic rollouts, lots of iterative kind of tasks. If we gave the agent access to prior success cases and/or prior failures, um, does it improve agent performance? Intuitively makes sense, right? As a human, have the cheat sheet. If you can figure out, oh, I hit this problem before, this is how I'm gonna fix it. That seems like it's gonna, you know, be good. Or, you know, you know, few-shot prompting, right? Like, oh, I've seen X, you know, 10 situations like this before. This is what I did. This is what worked. Now I can just copy that. And so we found, again, this was like a limited test. And so there's a bunch of disclaimers to put on it, but what we found is that like, when you gave this agent access to prior failures, it boosted model performance. Something like, you know, what are the percentage points? It was significant. It was like, you know, something like 25% to like 33% or something like this. When you gave the agent access to only prior successes though, agent performance went down.

Really? That's super unintuitive.

I think it's not unintuitive once you unpack it. We'll see if it's intuitive or not, I guess, once we unpack it. So I think the intuition is that like, models are lazy. The most insidious distracting information is the information which seems highly relevant but is not for a subtle reason. And the models are not always capable of appreciating the subtlety. Imagine you as a human, right? You take a human who's not an expert at doing something and you ask them to do a task, and then you give them access to a corpus of, you know, previous examples. Like, they are also just going to sort of fall into that local minima and sort of resort to copy and pasting. Hey, you already did the work for me. Thank you very much. I don't have to think right now. I'm just going to copy and paste. Um, and so, yeah, that's what we, that's what we found.

And just to clarify, when you say a library of prior successes, it was successes on related tasks, but not the exact same task. Because you could have a copy paste machine that would just take the diff from the success case and apply it.

That would be not an interesting test in some ways. That would just be true cheating.

Yeah.

No, it was like, it was similar, but not the same.

Got it. Okay. That, that is a very interesting result. And I wonder if that carries over to training. As well. You know, I've seen a couple of results that have demonstrated that there's like a continuous, uh, there's a continuum between packing context with in-context examples or few-shot learning all the way to actually doing fine-tuning. And are the negative samples actually more valuable in a SWE-Bench context than the positive?

A million percent. I mean, this is actually in a weird way, like where Chroma started. Um, we were really interested in active learning, which is fundamentally the process of like, the fundamental question of active learning is like, what should I train on next? And what you want to identify are examples that are difficult, out of distribution from what you know already, but learnable. So difficult, but learnable. You know, there's many tools obviously to like kind of gather that signal. But actually Google just like last week released like a new paper like on active learning. I saw that.

Classification of examples, which ones are going to be helpful.

Right. And they actually use, I think they use embedding spaces to like kind of derive these sorts of things. And we are starting to see like a few of our customers use Chroma. Like in like training as well, which is pretty cool. So I think it's still super early, but yeah.

Is there a way in which you can actually embed something like that into the vector database itself? So you have these embeddings, then you can somehow have like an inference layer baked into the DB, like at the C++ level that allows you to do a projection onto a single scalar value that tells you like this is valuable or it's not, maybe not just for active learning purposes, but for other classification purposes.

The term vector database, I've always hated. I will continue to hate. So I'll just add that really quickly. I think it's like a, it's sort of a brain rot term. It doesn't describe what the job is here. But besides that, I think LLMs running inside of databases are really interesting. I think the process that most retrieval systems, you can kind of reason about how they work, is that you first want to gather all possible relevant information. But then again, as we've already discussed, you can't give all of that information to the context window. It can't handle it. And so you have to glean it down. You have to focus it somehow. Re-rankers are a common historical tool for doing that. These are like, you know, sort of dedicated cross-encoder, bi-encoder models, like specialize on ranking specifically. We're seeing more and more people use LLMs directly as re-rankers, or it could be, you know, binary classifiers. Like, is this— here is the prompt, here is a chunk. Like, is this relevant? Yes or no? You know, the output tokens are basically zero. You can just use like the logits directly. But if you extend that, you can go further. Yeah.

One more.

And you can do stuff like, oh, maybe it's not just binary classification. Like, why can't you do dynamic chunk rewriting? Here's the context. Here's the chunk. Like, focus this chunk down to only what is relevant for this query, or why can't you do semantic analysis? It starts to actually look more like an unstructured query engine where you can almost ask like any query of a pool of data and then have like massively parallel LLMs helping you like process and understand that data. That to me is a really exciting future and I think like will come to pass. Obviously I think today like people are still having in their minds that like inference is expensive, which obviously it is if you're using like the latest and greatest models. Though the cost of tokens may not go to zero truly, people running like, you know, Qwen 2.5 themselves on like pre-committed GPUs, et cetera, et cetera. Like they're getting the costs down like really low. I think it's like, you know, a penny per million input tokens or something like that. And so it starts to almost feel like a free resource, you know, it's almost like using like electricity in your house. Like you don't think about it. You just like throw compute at the problem.

Yeah. And in that conversation of, you know, running LLMs is expensive. I think one thing that people don't really mention is that you're already spending a lot of money on running your database. And if you can actually take the LLM across that blood-brain barrier and bake it into the foundations of the DB, you're actually probably going to end up reducing the total number of queries you do because you'll get the answer in fewer number of total queries. I'm very bullish on this idea that you would have a smarter query planner that's actually in some ways non-deterministic running on a GPU that's just bolted onto the database as opposed to expecting external agents to be an expert in how Chrome works.

I think we've always believed that data has gravity. And then eventually compute workloads come home, they come to their data. And that can obviously be like just sort of network egress, transfer cost reasons, as you said, like in the future, maybe build a sort of, you know, sort of page directly the data we want for context or retrieval, like directly onto GPUs. And so all that living on like, you know, one chipset would make a lot of sense.

For sure. And it stands to reason you could do hardware level optimizations on these things. So they're, you know, better specifically at using your data storage format in order to do retrieval better for your tools. You can train a very lightweight one. Et cetera. Can you actually take us through some of the nuts and bolts of how you guys are thinking about that and any implementation efforts you've had to date? Do you have fine-tuned LLMs that interact with the ChromaDB, whether they're outside the DB or inside of it? How do you think about basically like making customized LLM tooling to interact with Chroma?

We've done a bunch of experiments, um, and many of them are very promising. I think like to date, just like, you know, as far as small companies go, we've been really focused on doing one or only a few things very well. So we've been very focused on like making the database very fast, very cost-effective, very performant, very reliable. You know, this is what our customers want us to do. Now we're starting to like see more ways we can help our customers with things like data ingestion, potentially like embedding model inference. And so we're making like some like, you know, sort of baby steps in that direction, I would say. But there's a lot that we have planned and want to get to, but you know, it's a question of like focus and time.

Totally.

Yeah.

Yeah. And it's, you would be probably saving cost and time, but we're still at the make it work era for many of these applications. And so maybe it doesn't make sense to bring it, you know, the LM across that blood-brain barrier as it were. Let's talk about code search. I think this is one of the most fascinating areas right now. Uh, you know, obviously in terms of just the revenue that's been generated from AI applications, you've got ChatGPT, AI boyfriends. There's sort of this whole like, you know, consumer angle. And then the thing that has hit the most thus far has been coding agents truly. Right. I mean, we're seeing. $600 million revenue from Cursor. Cloud Code is not far behind. There's a lot of challenges in coding. One of them is finding the right stuff. You mentioned Sweetbench a second ago. These are fairly large codebases, but ideally we could get these things to work on significantly larger codebases. So, you know, we work with a 90 million token TypeScript monolith. There's nothing like that in Sweetbench. What are the challenges to making this work? What is Chroma doing in order to help? And how do you see code retrieval evolving in the future?

I think like in retrospect, it seems now obvious that like Coding was gonna be like the first dominant use case for AI. It's just such a perfect playground of deterministic software that like it really, I think, lends itself well to being able to hedge for some of the disadvantages of the non-determinism of LLMs themselves. I always go back to sort of like the, uh, Andrej Karpathy, like Software 1.0, Software 2.0 type idea. Like, you know, what do we do when the internet came about? Um, we put phone books on the internet, you know, now that we have like AI, like sort of software 2.0 or 3.0, depending on how you count, like what are we doing? We're primarily using it to write software 1.0. It's like the case today that will, that will change. And it's a very, it's obviously ubiquitous case. Companies, there's been so much, dare I say, like propaganda about like how to do or how not to do code retrieval.

It's called backpropaganda actually.

Backpropaganda. I like that. You heard it here first, folks. People should spend less time on Twitter. People should spend more time with their benchmarks and their own evals and like figuring out what works. There was certainly this era where people were saying that like vector search is all you need. I never said that. Chroma never said that, but like that was the thing that like many vendors were pushing. Obviously we don't know what the future holds. Maybe eventually, you know, these dense representations can, you know, be way more powerful than, you know, sparse representations. But like today it's not obvious that that is a panacea for any application, no less code search. Famously, some comp— companies, you know, whether it's marketing or not, have sort of like made it a key point of their differentiation that like, no, we just grep. And of course grep is just regex, right? You're doing like regex search on top of a set of files, which works well when you have, which can work well, especially when you have textual data. I think like it's actually useful for to unpack this a little bit and to help people understand like when semantic search versus lexical search like makes sense. Also separate out indexing, right? Which is actually like building an index on top of either semantic search or on top of lexical search. You just brute force semantic search. You can brute force lexical search at a certain data scale. It makes sense to index it because you bend the curve as you scale. One simple analogy I use, which is not directly a coding analogy, though I'll bring that to code in a second, is like, you know, in my Google Drive, if I want to search for the spreadsheet that has a list of all my investors, I know what that's called. Like, I made it. It's called cap table. And so I just want to type in cap table. There it is. There's a search result. I click it. Great. That makes sense because I'm a subject matter expert in my own data. I know the grammar, I know the language, I know what to search for. If I'm not a subject matter expert in my own data, then okay, well, I'm going to search for like the spreadsheet that has the list of all my investors. That semantically is going to match very high to cap table naturally. And then, you know, I'll click on it and go forth. So I think you can think about this like for any given use case or user or both, a mixture. There's a certain blend of like how educated is the agent or the human in the grammar and the dictionary of the content that you want to search. There's sort of a blend of, I think, you know, some applications it's in terms of utility, it's 80/20. Other applications it's flipped, it's 20/80, depending on, you know, which of these techniques are going to be very useful. People have found that like LLMs are really good at code. They've been trained on a lot of code. And so they're actually like pretty good at the grammar. And they're pretty good at guessing which terms to search for. They get those terms exactly right or very close. And so regex is a very powerful tool as a result. I think it's, again, the way to unpack and think about this. I think it's a kind of code retrieval broadly. I think the jury is out on whether regex is all you need. Jury is out certainly on whether dense embeddings vector search is all you need. I think that in likelihood, the reality is that it's going to be a mix. And obviously when and how to use what is The interesting part of context engineering. But again, this is like, I think really what does differentiate like the tools people love versus the tools that people do not love is like, how good is the system at pulling relevant context? You know, people I think are, are tempted to like lean on the crutch of like, oh, you can just have the agent agenticly search, agentic RAG. Never wanna say that again. You know, just like keep going back to the well until you get something that like seems like it works. But I think you have lots of problems there too. Like how does it know when to stop? What if it stops too early? Like, in some ways you're just adding, like, more non-determinism to the system, not less. And so while that's useful, don't get me wrong, we already talked about, like, LLMs being great, you know, they can be, like, great query planners. The agent querying 80 times until it finds the thing that you want is, like, not likely to be the dominant feature of the application that users love a year from now.

So then do you have a perspective on If we were to make custom tooling for an LLM, let's say it's like a language server on crack, something like that, that has the structure of the domain baked into it. So it's not just doing lexical search, but allows you to like jump to reference for a given symbol or something along those lines.

Yeah.

Do you think that would actually solve some of the issues you described?

Yeah, 100%. I mean, this is what a lot of the top companies are already doing. So they're using regex, they're using symbol search as like two very like key techniques. Um, they're also using semantic search in some, you know, specific cases. Some of them are going even further. And pushing, you know, more advanced stuff. Again, going back to like the gathering-glean analogy, first you want to like get all the possible relevant tokens or information, and then you want to glean, kind of focus it down to like just what you need. If you think about like, you know, what do even more and more and more sophisticated approaches to like searching, if you will, what do they buy you? They might buy you a faster and cheaper gathering process. But like, that might be on the margin, not that important. It's possible that like brute forcing it at the glean stage, if you will, is kind of most where most of the value lies. And again, you can always make it 10, 15% better, cheaper and faster, which like users may or may not notice or care about, depending on the user and the situation. But I think again, you have to tie it back to the actual difference in the user experience and the difference in also the ability of the agents to I don't know that that will in the long term make a huge difference in the ability of the agent to perform, I guess is what I'm trying to say. I think if you do a good job gleaning, you'll still come back to all the relevant tokens. I think what it will mainly do is make the agent loop, again, on the margin, faster and cheaper. And then whether that matters to your user is a separate question.

For sure. Yeah, in a way, the file system is a database that exposes various search techniques over it, including ripgrep or whatever that may be. It seems like a huge reason for why that has won thus far. So Cloud Code will just use RepRap. It doesn't use any fancy language servers or anything like that. And that is, as far as I can tell, state of the art. It's just super easy to set up. You don't have to know anything about, you know, a specific database or query methods. It's just good to go as soon as you have a Linux VM. I have a hard time believing though that's going to be the case in the future.

Yeah.

It feels like there's so many things that are inherent to the structure of code that you would want to leverage for a hyper-intelligent LLM to be effective. Language server is a good example of that. You might not know what to call the authentication module, but instead of running like 6 ripgrep queries, maybe you want to actually do a semantic search in order to retrieve it. There's obvious complexities associated with creating the, the actual vector database, but that seems like the type of thing that will end up being resolved in the near future. So, and not to mention, searching the entire web's worth of code is not something you can do on a single file system.

Exactly. Yeah. I think I'll also make the point that like, if you're just searching text, then yeah, maybe regex can get you. Most of the way there. Again, if it only gets you 80% of the way there, if adding semantic search, embedding-based search gets you another incremental 8%, that might mean the difference between a user selecting your product versus a competitor's product. Right. And so like, it's not necessarily all about like, you have to think about these things. Text search by definition is only good at searching text, which in the case of code is usually only text, but not all the time. Right. Like a lot of multimedia ends up finding its way into the, you know, public folder, if you will, of your codebase. And so like, how do you search images? How do you search video? You know, that's also, I think, an important consideration if you want agents to truly go that like last mile of like utility and reliability. Another thing that I'll mention is like, today, I don't know of any agent that does this. I'm not like, you know, totally up to date on everybody's latest and greatest features, but like today, it seems like most people only search like the current active commit, the current dirty state of like whatever, you know, active commit is on your local machine. They're not referencing other branches, not historical or other open branches, which seems like it would be very useful actually, especially if you have like two agents working on similar features in separate branches. Maybe they should like coordinate to some degree and not create some ridiculous merge conflict down the road for the human. That requires you, by the way, to be able to query them, you know, sort of in real time simultaneously. Those are separate search indexes. I need to think about how to do that. We've built some features. One feature is called forking, where for a given collection, you can create a copy of that collection in under 100 milliseconds. Which makes you, enables you to skip like the cost and time of reindexing. And then you just land the diff of either the dirty state or, you know, whatever commit you wanna search, like down into that index. You also, uh, save a lot of storage as well, cuz you're not paying for incremental, you're only paying for the incremental storage. Um, and so we see like a lot of AI coding companies using Chroma, like specifically kind of in that way where they're doing like tons of forking, they're doing tons of forks of collections to search. The other thing that I'll, I'll mention is, uh, not just searching, like your code that you're working on, but searching the open source dependencies on which your code relies, right? We've all seen like the famous, uh, is it XACD comic, right?

Little thing on the side. Yeah.

The tower of dependencies, right? Today, and maybe, maybe in the future it'll be by LSP or otherwise, right? But like, it seems like today most AI coding agents like don't like click into references and go look at the actual API reference. And so like, if you're using that version of FastAPI or that version of React, like the, it increases the chance of like hallucinations cuz like I don't think that like models, maybe with huge libraries, like, you know, FastAPI and React, like the model, if you ask it like at this specific version, what is the API? I think it might be able to get it. I've not tried. Um, but like, I think the long tail is really hard. And so like increasing like agent reliability on coding, let it look at the references, let it look at the open source dependencies, but you don't always want to like git clone and do npm install and wait 5 minutes, um, for your agent. Right. And so then like those things. I think also need to be pre-indexed so that you can quickly kind of get in there and start writing code.

Yeah. And sometimes they're already minified, so you don't get good documentation if you do npm install.

Exactly.

So is this an active effort that Chrome is undertaking?

Yeah. When is this going to be published?

This will probably be like a week or two weeks from now.

Yeah. We actually then just released a—

We can wait until you publish this. Maybe that would be—

We call it Code Collections. And we've indexed like all major open source dependencies across npm, PyPI, uh, Crate and for Rust and, uh, Go, um, sorry, Cargo and Go. You can search them across various release tags.

That's amazing. So I can pin a version and say, tell me what the API supported was in this version of this obscure package.

Yeah. I mean, basically you're just giving more tools to the agent. Um, so you say, okay, I want to search like this exact package at this release tag. And you can do a regex search, you can do a semantic search, you can do both. You can do file-based search metadata. It's all in there.

That's super cool. I want to revisit this idea of a forkable collection. I think this is actually a pattern I've spoken to a lot of folks, you know, in the space of agents and coding and whatnot. This is an idea that keeps coming up is because an agent is something that it seems like they're going to be RL'd in the future with just an agent tool where it can fork itself. Any type of an artifact it's working on, if that artifact is also forkable, so maybe it's a VM, maybe it's a Git state, you know, you have obviously like stack PRs and whatnot, or collection seems to make a lot of sense.

Yeah.

And so what you're describing here is you have, you know, a situation where you want to be able to index a codebase and then on different Git branches you would have different collections that you can search that are in some way copy-on-write. So it's very easy to spin up a new copy. Is there a future in which you would actually bring that into Git itself and have some type of a Git extension where it's like a searchable index that has a diff stored for each commit?

I'm not sure that I want to cross Linus. Ever. Um, so I think the integration at that level—

What's the order of magnitude we're talking about here in terms of the size of the index? Is it vastly larger than the actual code itself?

There's so many variables, right? It's like how you chunk it, how large of an embedding model you use, if you quantize it. That's the main, the main difference or challenge. It's configurable, I guess, depending on like how large or small you want it to be.

That's super cool.

Yeah. I think, I mean, as we know, uh, Git is not exactly good at like large file management. Right? Anybody who's used getlfs will never use it again, usually. Moving around, while portability makes a lot of sense, like you do want to like ship like the index with the code, you also don't want to be moving large files over the network at kind of agent runtime either.

Mm-hmm.

Because it's like just going to be slow and expensive. You want to search remotely and like just get the data you need, not move the whole index over. So I suspect that that will not become a dominant pattern.

The other thing you mentioned that I thought was pretty thought-provoking is this idea of searching across different commits. So you're absolutely right. If you told me to go implement a feature, Jay, as a developer, I'm not going to go look at other people's stuff. I'm just going to start looking at the code that's on my machine today. That does seem like the type of query though, where you would want to dispatch somebody to sort of background research for you, right? So maybe there's a specialist agent that just will look at prior instances in a codebase. You know, it'll do Git archaeology. Look through all previous branches and find for you. Oh, we did actually do this in the past, but that doesn't actually block the main thread of the agent going forward. And this seems like a type of thing you could actually bake into the database itself. And so I would say, hey, you know, find me all instances of X, has a fast response to me. So it says, hey, in the codebase today on this commit is what we're looking at, but also just sort of an async query in the background. So I'm curious, do you think about applications like this where there's like a multi-tiered?

It's a good, yeah, it's a good question. I think one way to think about this is like budget. So for some queries you want to spend very little money or very little time or both, and you want to answer very fast and very cheaply. For other queries, you're willing to spend a lot of time or a lot of money conceptually to like go and get the right answer. That is, I think, falls more in the purview of these sort of like background search agents. I think the term people use is deep research. This is kind of like a deep researchy type thing. I don't like that term, but that is what people call it.

Why not? What's the issue with deep research?

I don't know. It's just another dumb term that doesn't mean anything.

I think it's the first company to get it across the finish line gets to coin the term. And unfortunately, deep research was—

Has anybody got it across the finish line? Like you've used deep research presumably, right? Oh yeah. Like it kind of sucks. Like if you run it on something that you know about, you're like, this result is incredibly mid and like wrong in like subtle ways, but meaningful ways in like 10 or 15 different places. And like that should make you very skeptical to use it for any other task that you care anything about. Because if it's not good at something you know something about, like why should you trust it to give you something you don't know about?

I don't know if my experience has been the same. I mean, I guess I'm not asking it questions where I'm expecting PhD-level research. I'm rather looking for something where it's like a quick book report where it's retrieved the relevant links from CNN or, you know, PyPI packages or whatever. And it's verifiable when you look at the output, like you can actually see that it got the right answer. But I have no trouble believing that if I was to ask it a difficult problem that, you know, it would not be able to get it. I think that'll be solved soon. Yeah, I do think it was one of the first products that launched in a full-on consumer manner that actually tried to do like an agentic task that took about half an hour. Like prior to that, really, you only had ChatGPT, which would run for, you know, 5 minutes or something along those lines. And so for that reason, I want to give it to ChatGPT or Google or whoever launched it first and say, deep research, great. You guys get to pick the name and I'll call that sequencing. But I guess there's, so there's deep research, but there's also multi-tiered research. So like you should be able to specify, okay, I want the ability for this thing to come back and interrupt me with like updated information on the question I've asked.

Right. I mean, not just like give you sort of status updates and reports on like how it's going and what it's thinking about and it's kind of streaming those tokens back, but also potentially asking you for clarifications if it needs additional help. I mean, it's something that even like deep research tasks today, like don't do. My understanding of like how deep research in ChatGPT works, for example, is that it's actually like obvious if you use it, but it is like hardwired to ask you a question. Hmm. Like it will always ask you first a question. You notice it never asks you another question, even though it presumably could or should. Um, and you know, that will change. We'll do whatever our customers want us to do is the long and short of it. Um, I think like today that paradigm we've not yet found to be common or popular enough where people want us to do it. And I think that's probably mostly about the maturity of these things where they're not stable yet. And there's like the change is so high that like people rightfully so want to keep it like in their purview directly, like very directly. And then once it becomes like standardized and well understood, then there'll be gains, speed, cost, reliability, operational complexity to like sort of like farming it out. So that's my guess.

But yeah, I would probably add on that to say one of the reasons you don't see this in production today is because The high watermark or like the maximum length you can get an agent to run and actually do something valuable is between an hour and 2 hours. And if a full half of that is just doing research and spawning a subagent that goes off and look for stuff, you should just have the main agent thread end up doing that. But maybe as we anticipate these agents are running for 10 hours or, you know, 24 hours or something like that, it actually does make sense to spawn off little mini deep research agents that come back with verifiable information for you.

Yeah, I like that.

In, I think, yeah, as you see the length of the main thread gets longer than the size of the agency it can support will almost certainly go up, right? Because there's just more things to be done that you can actually, yeah, make use of.

Yep. Yep. There's some like direct analogy to companies. It feels like where like maybe this has a breaking point clearly, but it's like, you know, what you find is the correlation, at least that like the larger the company, the farther into the future, supposedly they at least have the ability to look. Not that all large companies do look far into the future, but like in theory, the larger the company, the farther in the future they have the ability to look.

For sure.

Yeah.

Yeah. So we've talked a lot about retrieval of code or like artifacts that are maybe like information and Slack or something along those lines, which is obviously going to, you know, it's the wave is still yet to crash, I think. And there's going to be huge productivity gains that comes from this. One of the most frequently cited interesting use cases for retrieval is memory. So it's not just the actual read portion, but also the write portion of what do I decide to remember in the first place? I know y'all have spent a lot of time thinking about this, so I'd love to get your thoughts. Maybe as a motivating question is, what do you think are the actual good applications of memory that exist today? Like, where has it been successful? And what do you think those applications will be in the future? And then how do we pull that off?

I think broadly you can split— I mean, first of all, memory is more— it's not a technology. Memory is a benefit of a technology that is legible to the average person. Your mom and my mom sort of can understand what memory is and means and kind of why intuitively it would be beneficial. That may also be true for like the C-suite, right? Oh, memory. I get what that would be. I want my agents to have memory. Like, that kind of makes sense. I think in practice, for the practitioners who are actually implementing this stuff, the read path is just another context engineering path, going and gathering relevant historical information and deciding what's important or not, and then putting that in the context window and then using it. The write path is interesting. So, like, what should the system remember? Broadly, the system should remember anything that allows it to get better at the task at hand. I think that's one way to think about it. So like kind of agent continual online learning, something like this is like the goal of memory in my mind. I think you can break that down. You know, you start to break things down, you create like kind of potentially false hierarchies and dichotomies that like can create more brain rot. But I do think that like there's some useful distinction between agent task learning. So like how can this agent get better at the specific task? And then user personalization. How can the agent get better at serving this user? Now, of course, you know, serving a user is also just a task at the end of the day. It is intuitive to sort of break it into those two big buckets. Agent task learning, we've already talked a little bit about when we were talking about Sui Bench. It actually was that experiment to see past successes, past failures, you know, can the agent reference those and get better? And it was obvious that this is not super— it's not just trivial, right? Like, you don't just give it access and call it a day and ride off into the sunset, go sit Put my ties on the beach? No, it's actually way harder than that, at least today. I think it will continue to be very difficult. User personalization in a way feels more tractable and more accessible. And so, you know, remember that I want my TypeScript formatted like this, or, and that's kind of the coding example, right? Or, you know, always remember to write a test anytime you write a new function, you know, these sorts of things. User personalization also kind of just like user instructions, like my preferences. I'm not, but like, you know, in theory, I'm lactose intolerant. And so if you're planning a trip to London, don't suggest me going to get ice cream, right? Like, you know, these sorts of like kind of user consumer stories, I think also have real utility. You know, the memory feature inside ChatGPT is contentious on some level. Some people, it seems to drive them into a deep state of psychosis and, you know, they literally go mad. Other people, I guess, including myself, like just turn it off because I just want to manage it and I don't want the leakage. It's not a contentious or sort of contrarian opinion to claim that's not been done well yet.

Nothing makes me madder than when my AI waifu forgets what I said to her yesterday. So I empathize with them.

I'll avoid making jokes. That is funny. Yeah, exactly. Yeah, I think that choosing what to remember is pretty interesting. So in some sense, you may not have to choose. You can just remember everything in your gathering, glean, retrieval, information retrieval pipeline broadly later, you can then decide from everything that I know about, you know, like what should I kind of pull into context here? That may work well for like factual information, especially when the user says explicit things like, please remember X or please remember Y. I think this is how like the windsurfs and cursors currently do it. Like they kind of have these like magic words where like if you say something, then it'll remember it. I actually think that like, that's fine, right? Like teaching users a minimal amount of user education about like the UX of the product of like how to remember stuff, like even clicking a button, why not? Like getting that level of structured data from an end user, like solves so many problems for you. We're doing it in a totally background implicit way. Like it's just way harder and maybe, maybe not that much more valuable relative to the users being very explicit about their preferences. That works well for like point facts and like, you know, point information. Um, we do see that like, you know, if you just, just use a text file to sort of put all of these like preference data into like just a text file. And then you're just like throwing that thing into context every prompt. Like that, that's dumb. Not just dumb, but like it doesn't work well. That's what I mean. And so like you shouldn't do that. You should still be discerning about what you put in the context.

It ends up giving it a bunch of deluge that's irrelevant or—

Yeah. And the, the example that I use, maybe I'll find a better example eventually, is like if I'm planning a trip to Paris and last month I went to Prague and like, you know, in Prague I went to a pinball museum, maybe I love pinball and like, so I'm going to Paris now. Like it should probably recommend if there's a pinball museum in Paris, it should tell me about it. Obviously that's a really important thing. But like, if I grew up going to my grandmother's lake in Wisconsin and like now I'm going to Paris, you know, it shouldn't say like, oh, you should go check out this lake because you went to a lake as a child. Like, no, like that's not important. But I think it's like hard for agents to discern like what is important and what's not important.

Yes. Um, gentle pushback on that is, isn't that sort of, uh, post-retrieval? We've already, you know, gotten all these memories. The agent should be able to discern and say like, okay, this is a lake house, but he was 5. Now he's a grown-ass man and he's going to—

Again, one would hope, but it just seems that like, that's not a stable assumption.

Empirically, that doesn't end up actually panning out. Yeah. Interesting.

The nuance I was going to add is that like, you know, sort of point factual information is likely well served by some of the techniques we have today, but like long-term analysis of like not OLTP, but OLAP, right? Not point queries, but aggregations. I, Mr., Mrs. Agent, has now done the same similar task 45 times, like the 46th time, like, how do I get better at this? Like, how should I adjust my system instructions? How should I adjust my system instructions? How should I adjust the information that I have access to or how I access it? Potentially even how should I modify my own weights such that like I can do better at this thing next time? I think that problem is like very much like underexplored and not solved.

So I was going to ask, so, you know, all of the implementations I'm aware of of memory today that are in production, something along the lines of writing it to a to-do list or storing it in a database somewhere and retrieving it later. Do you think there's a good fundamental reason why this isn't baked into the models themselves?

What would that look like?

One example I can imagine is that, you know, you want audibility. And so anytime it remembers something, you should be able to view it in a Notion document or you should be able to view it in, you know, a Postgres UI that says, okay, I can go through and I can cross out things that are irrelevant. And so if it's, you know, actually baked into the weights of the model, it's not a good idea.

Yeah.

However, you know, we're talking about the sort of like vibes-based understanding of how to do a task. And that seems like the type of thing that there's sort of, you're adding a bunch of extra steps if you have to store it in an external database. And if you think about what does the fully generalized version of this look like, that's going to be successful. It probably is doing this like lifelong learning thing that we've described where every time it does a task, you give it some type of an error signal and it's able to just update itself in a way that's not just behavioral. It's actually memory oriented.

I mean, that would be great. The auditability questions are quite important. There is a dearth of research in applied AI broadly. The labs are very focused on the next big model that hill climbs and maximizes all the legible benchmarks because that's just how they accrue eyeballs and dollars and talent. And whether or not those benchmarks really mean anything now and whether they're actually helpful and whether we should care is an important question to ask that nobody seems to ask. You have a bunch of people at like the sort of SaaS layer, like building useful stuff with AI, but like don't have like the time or the org chart or the skills to like do great applied research. And so I think there is like a gap here is the first thing I wanted to say. I do think that like fine-tuning is very promising, especially with that fine-tuning is easily like roll-backable, you know, more like LoRA-shaped things. I think like are very promising. You almost want again, like a tree of state that like you can roll back very easily and also like hot swap very easily, like in production.

It goes back to our discussion of forking an agent.

Exactly.

You have one agent build up some memories, but then you can go back. I think that is pretty key that there should be an external data structure that's not actually the parameters of the model. So it's very easy for you to swap in and out, like LoRA, as you mentioned. What about applications of memory? So I think this is one noticeable thing that I've noticed as an avid agent user. I think ChatGPT and Claude launched it today, I want to say, where it'll be able to respond to your previous things and it will say, oh, I see you're a software engineer and therefore, and it will give me an answer. It's probably doing some form of, you know, retrieving a bunch of documents from previous chats. Maybe they're distilled and shows it to it.

Yep.

The only other applications I've seen of actual memory in agents today are in a couple coding agents. So Cursor just rolled out a thing where it will, like you mentioned a second ago, it has maybe a secondary model that looks at your chat and says, is this memorable or not? And it will propose a memory. It's pretty bad. I'm not going to lie. Like I've been using it for a little bit and the memories it's proposing are like, oh, you always like using this emoji in logging. And I'm like, well, I did in this specific instance, but I'm not claiming that that's not resolvable. It's like definitely giving me the type of thing that with proper context management and engineering, you'll get something good, especially if people are clicking the button and you get this. Concrete feedback. Do you have any examples though, where memory actually does move the needle in terms of the experience of a user interacting with an agent?

I think intuitively we want agents to perform at human level or beyond. And if you just consider how you teach a human to do something, it's not really possible to write the whole user manual and write the whole textbook and have the human read it and hope that they just do it. In practice, any job, is an apprenticeship model. And there's a lot of like tacit knowledge, which is required, like on-the-job training, they call it, right? Where like you are giving feedback to your employees or people that you work with about like, do more of this, do less of this, do this this way, do this that way. I think if we want agents to be as reliable as humans, then like that has to also exist in agent land. It could be agents themselves giving themselves their own self-critical, self-critique feedback, or I think even more likely for the time being, it's going to be humans giving agents this iterative feedback. Here's what to do next time. That seems like a very big and important problem to solve. But if you can solve that problem, I think that the reliability of AI agents will go up massively.

To what extent is that a consumer behavior that needs to develop? And I guess as a precursor to this question, I started working on coding agents 2 years ago. People just had no idea how to use them effectively. They would give them tasks that were like, oh, fix my code base. And it's not exactly clear what's wrong with your code base. And so it would go nowhere. And we've seen in the interim between the launch of GPT-3 and now, there really is this sort of ingrained customer behavior, user behavior of, this is the type of problem I can factorize out and have an agent go and do. And it seems to me like we do not yet have this muscle in users themselves to say, this is something that's worth remembering and building this muscle sort of an active knowledge curation. So To what extent do you think that's actually the bottleneck to successful deployment of memory?

I don't know.

Wait and see.

Yeah. I think it certainly can get better, you know, whether it's going to be like good enough to get across the line where it like reaches some sort of, you know, activation point for a lot of use cases. TBD.

I would imagine that it's the case that if ChatGPT had a thing in it where it would say remember or don't remember and there was just this sort of universally accepted UX mechanism, then we would end up seeing memory much more widely deployed. And you could expect if you're launching a new product, you could have a memory feature baked in. And in fact, people would expect that.

I think that's right. I mean, OpenAI has this ethos of like the most minimal brand and kind of DX or UX possible, which I think probably is the right call for a mass consumer play. Like clearly what they're going after now. Problem is that they're also like the, they're considered to be kind of the forethinkers and like what AI UX should be like. And as a result, everybody just sort of cargo cult copy and paste this exactly what they're doing onto their thing. And the amount of innovation inside of AI UX is like kind of embarrassing. Yeah. I think that people should do a lot more. And if you're listening to this, you should do more.

It's interesting. There was a period where people were saying chat is not the final paradigm for agent, you know, human-agent interaction or human-LLM interaction. Like we need to do, you know, do something better. And we sort of rode that wave up and now we're coming down. People are like, actually chat is pretty good. Like we've tried a bunch of other stuff over the last couple of years. Nothing has truly landed in the same way. And I'm curious, why do you, do you have any thoughts on why that might be the case?

I think chat is good UX because humans understand it because we chat all the time. And so like it is extremely intuitive for that reason. Um, but not all power tools are intuitive by definition. Power tools are usually not intuitive. They require you to learn them to get to unlock their powers, if you will. And with something like coding, that is a good candidate for a power tool and that's a good candidate. Where like, oh, just basic chat, like, you know, may not be enough. And I think you already see this to some degree, right? You can like tag files and stuff. You can kind of give it additional context. I don't know, you can author tools or kind of give access to tools and there's the other things we can do. The first thing people usually jump to when they say that chat is not enough is they say, I want branching chat. And now all of a sudden you're making some sort of like, you know, reactive node diagram thing in React.

Never works.

It just never works.

Has never worked in history.

It just never works. You know, maybe in 2025, wait, no, 2035, uh, it will work, but it just never works. I think one way to think about this is like, how can you give an expert human better information about what the agent is doing at any given point in time and intuition about as well? And then how do you give the human the ability to course correct, refine that agent's actions? In real time. It's sort of anybody's guess on what that should look like. So yeah, for sure. Yeah.

Yeah. It's interesting on the branching agents. I think it's important to clarify that branching agents is definitely going to be a thing. In fact, it already is. Claude Code is a great example of that. But even with that, people have still rendered it into a linear chat history because that's just easier for humans to wrap their heads around.

Exactly.

And so the actual visualization is not one of these fancy D3 charts where you can drag around and react and whatnot.

Exactly.

There probably is a visualization like that that would be helpful to people, but it just hasn't caught on. So, well, I think that's a pretty good note to end this on. So thank you so much for taking the time. It's super interesting to hear about everything you guys are doing at Chroma, and I appreciate you having me.

This has been awesome. Thanks.

Awesome.