Memory, evals, and efficient storage in AI systems with turbopuffer and Braintrust | Full discussion

All right, team, let's get started. Welcome to Research to runtime. This is a session on building AI systems and agents with our esteemed guests, Ankur, the founder and CEO of BrainTrust, and Simon, the founder and CEO of TurboPuffer. For those that I haven't met, I'm Talia Goldberg. I'm a partner at Bessemer in our San Francisco office, where I lead many of our AI investments across stages. Bessemer is a global venture firm. We partner with entrepreneurs from the very earliest days through every stage of growth, and are very proud to work with companies like Perplexity and Anthropic and Fall and Abridge, Canva, Shopify, Clickhouse, many others, and many that are customers of BrainTrust and TurboPuffer. And we've seen the power of these products and what they're enabling firsthand. So we're super excited to share that with a broader group. And with me, I have my colleague Bhavik, who's an ML engineer by training and also works with me to lead our AI investing. But we really started this series because the landscape is a roller coaster. There's a roller coaster of progress, of innovation, of best practices, and everyone's trying to figure out what to do and how to build and best practices and tactics. And so it has been awesome to get this community together to share that and to learn from some of the best. With no further ado, let me let our guests introduce themselves and maybe give a little background on yourself and your company. Ankur, why don't we start with you?

Yeah, sounds great. Very excited to be here and chat with you all. I'm Ankur. Prior to Braintrust, I used to lead the AI team at Figma. And before that, I started a company called Empura, where we did AI document extraction in the Stone Ages pre chatgpt when it was quite hard. And at both companies, every time we changed something like updated our models or changed our prompts or changed the underlying architecture that we used, we would break stuff for customers, and we had to get really good at avoiding that. And to do that, we built tools to help us do evals well. And then it was really hard to get data to do those evals. So we built observability tools to help us actually collect data in a way that was useful for evals. And the third time around, that turned into braintrust. I was reflecting this morning, and I actually, I only know two Simons, and I really like both of them. The other Simon is at Notion, and that Simon was one of the first people that we talked to and he shared how they did evals at Notion and we started working with them. Zapier, Stripe, Instacart, Airtable and a bunch of other really great companies that are building AI products. And we're doing it really early. And by working with them we collaborated and established some really good workflows around evals and observability that are now the BrainTrust product.

Awesome.

And Simon?

Yeah, I'm Simon. I spent almost 10 years building infrastructure at Shopify. So I'm up here in Canada and I spent when I joined Shopify was doing a couple hundred requests per second and by the time I left we had saw peaks of around a million. And I worked on mainly the things that did not scale, playing whack a mole on all the bottlenecks when the Kardashians rolled through and did some of the largest flash sales in the world. And the fundamental bottleneck for most major SaaS platforms is the database layer. So I more or less worked on every single aspect of the database and scaling the compute layer for almost 10 years at Shopify. When I left I was bopping around at some of my friends companies helping them in little infrastructure things. And at one of them I discovered the massive cost of embedding based search. And with this one company, a company called Readwise, we just did some very simple article recommendations over the course of a month. It worked pretty well. But this was a bootstrap company. Spending three grand a month on their postgres and putting all of this into actually operationalizing all these vectors would have cost them 30 grand a month. And they actively did not do it because it was too expensive. And that's what we set out to do with turbopopper is do an order of magnitude cost reduction to unlock a lot of this product that people wanted to ship. And that's what I work on today.

Amazing, thanks for sharing that and giving a little bit of the background, maybe building off of what you just said. Simon, and just as a little bit of context for folks, what are the specific decisions in traditional vector databases that create these cost explosions and what were the things that you guys did to really address that? I think it's known or publicly reported that folks like cursor have 20x cost reductions by switching to your architecture. And it's not just cost, there's also latency and speed. And so what did you do and what were the challenges?

Yeah, I think it became when you start a company you generally have some insight or something that you think could be done differently. And I think that the insight that led to turbopuffer was that there was a new storage architecture in the air where we could use S3 or GCS or Azure Blob Storage as the source of truth. And that only really had become possible in the past few years. So the new storage architecture that was in the air was one where you use NVMe SSDs. NVMe SSDs are about 100 times cheaper than using memory, but the throughput that you can drive through them is only about five times less. So if you can build a database that takes good advantage of it, you can run into some real economic advantage. The second thing that happened was that S3 became consistent at the end of 2020 and S3 got compare and swap at the end of 2024. And this has allowed us, with these three new principles, to build databases that can have a completely different storage architecture than those that came before.1 where S3 or Google Cloud Storage are the only source of truth. You don't even need a metadata layer. I think Ankur and I can both go into a lot of depth of why or how to use the metadata layer. And I think we have some different thoughts on how to do it. But this is a new storage architecture. And I think fundamentally, if you want to build a generational database company, you need two ingredients. The first one is the new storage architecture, because if not, then all the incumbent databases are just going to add on and egli. But if you have a new storage architecture, has fundamentally new economics or new performance characteristics, and you have a new workload, that means that people are out shopping for a new database. In this case, connecting enormous amounts of data to LLMs is the general new workload. If you have both of those ingredients and you have good execution behind it, you have the potential to create a generational database company. And I think that we saw those two things in the air. So we talked about Cursor notion and Linear some of our customers. And for a customer like cursor, it matters a lot what it costs, not because cursor is cheap, but because cursor has to earn a return on all of the data that they have to search over. That's fundamentally what we're all doing, right? We're earning a return on whatever is underneath us. And so if we can change the economics in a way where the products that people can ship on top of search are fundamentally different, then our customers can ship more ambitious versions of their product, right? So with Cursor, we reduced their cost by 95%. And that doesn't mean that we're the H and M of search. It means that suddenly they could index much larger code bases for way more customers on economics that they could earn a return on. Same with Notion. They used to have a per user AI cost and part of them removing that was switching to TurboPuffer to go from a more traditional storage architecture and into one where we could reduce their spend by millions of dollars a year on this new storage architecture and made a lot of sense for them. So I think I'm more in the business of making sure that people can earn a return on top of us with the product that they build than anything else.

Yeah, I love how you talk about this very clear moment of why now? Because it's really hard to build a database company, as you said, and it's really hard to have these switching costs and folks get very built in. What do you see for companies like A Notion or others or even a company that has an existing setup, how they switch over and what that process is like to get going.

On TurboPuffer, there's a couple of profiles, but I think that the there is two, two, two major categories. So the first ones are more net new workloads where these are either newer companies or ones where they only have very simple search and they can wholesale replace it with a simple hybrid search inside of their app. So using normal Lexical search on top of end and vector embeddings as well and fusing it together. So these are more net new where it's very simple for them to do it and they generally are newer adopters and then start on turbopuffer and then go from there. The other type of customer that we see are ones where they have existing Lexical search systems, typically Lucene based search systems that are tuned for Lexical. They see that they can boost their search recall and end precision by 10, 20% by incorporating embeddings, which is huge. Right. Like some of these companies, you'll have people that spend an entire year improving search results by 1 or 2% because that really ends up mattering on average to their customers. So 10 to 20% that we've heard from some of our customers is enormous. Especially when you also are starting to have machines use that same search to try to find and research and all these new baseline SaaS features that we're starting to expect. So in those companies generally they adopt us alongside their existing Lexical search and then over time shift more and more search workloads over to turbopuffer, including some of the net news within the business, but also incorporating some of them and then querying both. But over time, you would like them to move everything to turbopuffer

and Ankur. Are you seeing that companies are also benchmarking, call it latency recall of the vector systems that they're using in your product? What's the entry point to BrainTrust?

Yeah, so I actually haven't seen people run performance benchmarks, like speed benchmarks of different vector databases in their evals. And I think the reason is that you don't really need evals to do that. If you have a set of cases or the shape of data that you want to test the performance on, you can test it once or twice and get a reasonable measure. But if the words inside of a particular paragraph of text change, it's unlikely to affect the speed at which a vector database returns stuff. We do see a lot of people actually eval the quality of search results that they get. And we do see a lot of people find silly concurrency problems where they're like serializing vector database calls or something in their waterfall of a trace. And so there's quite a bit of TurboPuffer API calls that get traced in BrainTrust one way or another. But honestly, I think most people that we work with that use TurboPuffer, when I ask them about it, they say the decision was pretty straightforward and easy.

That's helpful. And even just taking a step back, if I was building an AI agent net new, what's the best time to start creating an email harness or starting to do tracing, production monitoring, all the stuff that BrainTrust provides?

Yeah, I would actually say it's on day zero. So when we started, we talked to a bunch of companies and we found that the people that were Most interested in BrainTrust were people that had shipped their product three months ago. Because you have this, people tend to have a bit of confirmation bias leading up to shipping a product that they've played with it. Let's say you and I are working on a product. We might sit in a corner and play with it until we feel like it's good, and then we're like, we don't need evals. We were able to improve this thing. And then you ship it, and then Talia starts using it and she's, wow, this thing sucks. And you're like, what? I We were sitting in a corner and it was fine. No, it actually sucks. And you're like, why does it suck? And she's like, why do I have to tell you? And that is usually the point at which people realize that they need evals. So however the people like that start doing evals, realize that if you're not doing evals while you're actually specing out or thinking about a project, then you're going to waste a lot of time. And so usually, like, after people get bitten by the eval bug, they start doing evals as they're actually prototyping things. And I think a large part of that is if you use evals from day zero, you can build a prototype that kind of sucks, but you have a feedback loop built into your development process that lets you improve the quality of the product very quickly. And if we were using the metaphor of you and I sitting in a corner, we'd have to sit in the corner for a lot less time to be able to ship something than if we weren't using evals.

That's helpful. I just wanted to flag for the group. If everyone has any open questions, please do add them into Zoom and we'll make sure to answer them as we go. Okay.

There's actually something I wanted to add to what Simon was saying. So he mentioned there's two things that have to change quite a bit for a new database to be relevant. And at BrainTrust, even though we're not selling a database, we actually build our own database called Brainstor. And Simon and I talk a lot about this all the time. It's also built to run directly on object storage, and it literally Wasn't possible until S3 released Compare and Swap, for example. A lot of that stuff is very relevant. But I actually think there's one more thing that's quite different in AI, and I think this is especially true for us. Maybe it's a little bit less true for search, but in traditional observability, Prometheus flavored observability, the information that you track tends not to be tied to pii. The logs that are spit out of your web server, or the CPU metrics that you're tracking from all of your containers, whatever, if that leaks, it's not the end of the world. It's obviously not good, but it's not the end of the world. And if your web server logs are spitting out pii, then that's usually something like if Simon, if Simon discovered someone doing that at Shopify, that's usually something that would be considered a pretty serious bug and fixed quite quickly in AI. The interesting thing is that what you're observing is actually people's raw interactions with LLMs and the information that you're Observing actually inherently has a lot of PII in it. And the scrutiny that people have applied to BrainTrust from a security standpoint is an order of magnitude higher than the scrutiny that was applied to observability products in the previous generation. And actually that's one of the other benefits of using object storage. We make it very easy for customers. We have Fortune 10 customers, for example, using BrainTrust, we make it super easy for them to run BrainTrust inside of their own cloud environment. And part of the reason that's practical for them to do is that Brainstor stores all of its state directly on object storage. And so from an operations standpoint, it's also significantly easier for customers to be able to run BrainTrust inside of their own cloud. And I actually think that is something that is very different than the previous generation observability systems.

I think this actually there is a reason number three and a reason number four in the why now? Of the databases. And you're touching on number three. I just usually simplify it to one and two. The way that I often talk about three is that there's a new deployment model, right? In 2015 I was working on migrating all the Shopify to the cloud, right? Like it was like even a lot of the frontier companies didn't go into the cloud until the late 2010s. Now everyone is running in the cloud, even most of the enterprise is running in the cloud. And it makes it a lot easier to ship things into people's VPCs without ending up as an on prem company very early, which just becomes you're building a support team very quickly. So I think the new deployment model is one, and then I think that the. I would go a step further here and also say that this actually allows even more deployment models than just the byoc where this is easy operationally inside of another customer's cloud. I think this is one of the most under appreciated things about this architecture, but also one of the most important that we see when we're out selling, which is that with this architecture, if you guarantee that everything is in object storage, it means that you can use your customer's key to encrypt every single byte of data. So TurboPuffer exposes this. You can send an encryption key that's managed inside of your cloud to TurboPuffer, which is logically the same, even though it exists in our bucket as if it exists in your bucket. This is fantastic for enterprise because it means that it gets all the governance controls that that they need to have to be able to Shut down any data access, but without any of the operational pain of running it even in their own cloud. Right where you want to lock it down as much as possible. And they can go a step further. A lot of our customers in SaaS do this where they use their customer's key to encrypt the data in the turbopuffer bucket. So not even our customer is able to see their key. It's passed all the way through and that customer has access to it. We see some of our more really advanced customers in their BYOC deployment, which is probably very similar to what you're doing. Ankur store all of the data inside of a bucket and then turbopuffer just has access to a particular prefix to run inside of that bucket, but all of the data is in there. And it gives the customer this warm fuzzy feeling of knowing that all of the data is in their control, either with encryption key or in turbopuffer's case, it could even be in their bucket.

When it's. When the customer managed encryption key, do you search over the encrypted data directly or do you decrypt it in memory and then encrypt and then search?

You have to use the key. Yeah. And then we get it into memory and then you have to be. You have to accept some TTLs on disk cache and things like that.

So there's no homomorphic vector encryption yet.

That stuff is. It's doesn't really help you very much. It just makes it more complicated if you, if you break into one namespace but you can do it like you can do these rotations. It just makes the attack take longer. It doesn't fundamentally change the shape of it. And then the fourth thing that is very nice if you're building a database company, is to get a lot of data very quickly. That's often the trend.

So just to take a step back, a lot of traditional database systems have been built on NVME non volatile memory. They haven't separated compute and storage in the same way. Can either Ankur or Simon, whoever wants to jump in, just outline the sort of broad scopes, broad scopes evolution that we're seeing before. Before we dive into specifics around databases.

Yeah, there's not a lot of database systems that are actually built on NVMe. The problem is that the, if you rewind two or three generations ago, like back when I was working on databases at MemSQL, when in the ancient days people were just starting to get access to SSDs regularly on a hardware like on Prem hardware. Maybe back in those days, Simon, you were like starting to get SSD data center. Yeah, fusion IO. That's good stuff. But most people can't even afford that. And so some database systems tried to build some fancy algorithmic support for these SSDs but pretty much no one did because SSDs are really bad at random writes. And that is like the main problem when you're building an OLTP system. Now fast forward a little bit. And then people got infatuated with the cloud and for a long time there was no SSD support in the cloud. And when it finally came as these NVME disks arrived, they are not durable beyond the lifetime of an instance. And all of the stuff that people were working on before assumed that the NVME would survive and be useful as long term durable storage. And that is still not the case today. And that I think is a key difference. I don't know of any commercial OLTP system that is actually built with the assumption that it can use volatile NVMe. And volatile in this or not volatile is not the right word. Non durable NVMe. And I think that's the big. That's the big difference planned at scale is there we go.

But they've solved it from the.

They're very. But I would consider them in the latest generation.

Sure, I think. But I think there's also. There's two parts of it, right? There's the operational side which PlanetScale is obviously exceptional at. Right. Their Kubernetes operator is phenomenal. Can handle these circumstances because what you can end up with is like 3 NVMe instances are gone and then all your data is gone. So you have to have an enormous amount of trust in your operations to be able to get onto these NVME instances. We can talk about the software side for a second. Ankur kind of alluded to the random write piece which makes OLTP incredibly difficult. At least any B tree based OLTP which Postgres and MySQL are, you need some LSM and only those are also only starting to mature. But I do think that the storage engine that takes advantage of NVME is also fundamentally different than the ones that have been written in the past. Right? MySQL and Postgres were both written for HDDs, right. Just so many generations ago. And even like SSDs are different and then NVMe SSDs are different again in the trade offs you make with memory. And so I think that the nice thing is that the storage architecture that is required to take full, full advantage of Opix storage and NVME is more or less the same. Because what you need to do to take advantage of NVME on the read path is that you need to do a lot of outstanding concurrent requests to the disk in as few round trips as possible, because you can't escape the, say, 100 microseconds of random read latency that you have to an NVMe SSD. In the same way that every time you go to S3, you have a P90 access time of around 100, 200 milliseconds, depending on the region and so on, you can't escape that. And it's sort of fundamentally the same problem where you can max out the Network NIC to S3 and you can max out the NVME port to the disk, but you have to do an enormous amount of parallel requests in every round trip and minimize the number of round trips. That's not how the storage engines that were built in the 90s or the 2000s or even the 2010s were built with that kind of round trip sensitivity in S3 really forced our hand in making sure that we only do three round trips for everything. But the storage engine is capable of doing good cold read latencies on S3 by minimizing round trips and maximizing concurrency, just also happens to be phenomenal for a disk. And then what we can do, and I think we both do this, Ankur, I don't know if you ended up using the code that I sent you, but to get around the random writes, you just fill up the entire disk, which also is an enormous economic advantage because normally if you have a database workload, you're right, running at maybe 50% disk utilization to give yourself and the on call some headroom. But if you're caching and do one big ring buffer of the entire disk, in that case you're doing 100% disk utilization. You just wrap around once you get there. And now all of your writes are sequential because you're just, oh, I'm caching this block, then this block all the way. And then you wrap around and start writing frequent values back. Anyway, we went, yeah, we can pull ourselves out of this hole now.

No, that's helpful just to understand that better, since you're now using object storage and you want to achieve some degree of data locality. Simon, my understanding is that TurboPuffer is best fit for customers that have natural sharding in their data. Can you talk about that a bit?

Yeah, I think that a startup's only mode is focused on. And our focus in the beginning was any large multi tenancy workloads where the P100 tenant was not particularly large, but so the individual shards could not be that large. That's not the case anymore. Now we can do shards that are into terabytes without any issues. But in the beginning, you're right, we took advantage of the fact that the largest code base in the world, I don't know, something like LLVM or Linux or something like that, is still not that large. And even for notion with their, with their workspaces, the largest notion customer was not that large. So we focused on getting very good at handling a very large amount of shards. Now the plan has always been to get very good at handling very large shards because then you have other customers that have extremely large shards. Maybe their biggest customer has a billion or maybe even tens of billions of documents they want to search simultaneously. And then you want to get good at that. Like every database shards, it's just who manages for you and when does it happen.

And so even if we have the exact opposite problem, please keep going. But yeah,

we try to. Then what we're working on now are namespaces that are around half a terabyte. So that works fine. Now in turbopuffer you want to max out at probably half a terabyte to a terabyte of shard sizes. But you want to make these shards as large as possible because fundamentally when you're doing search or any type of database lookup, it's like M times log n where it's not actually log n for a vector lookup, but let's pretend it is. Then the end, you want that to be as high as possible because it grows logarithmically, but you want the M to be as small as possible, to do as many, as few of those operations as possible. So for reference, in elasticsearch, when I ran that in production at Shopify, you go for a shard size of somewhere between 30 and 50 gigabytes. Incredibly small. So the M log n is very large, which means you're spending an enormous amount of CPU cycles doing that search. Whereas in Turbopuffer our shard sizes are trying to get, we're trying to get up to a terabyte, right Successfully. And then at some point you have to like spread that out into multiple machines and build a sharding management layer on top of that, which will be there in Future versions of TurboPuffer. But fundamentally you can just do ID modulus N which is what a lot of their customers are doing. Short answer is yes. In the first version of TurboPuffer we only supported small shards. But the answer is that now we can do state of the art shard sizes.

Yeah, it's quite interesting because we have. If you use one of those customers like Notion as an example, all of the logs across their users are coming into one brain trust collection and the task is actually to look at the information across all of them. So we don't have sharding based on like a collection, but we do many of the same tricks using time as a partitioning key.

Everything shards.

Yeah.

With that, I think it would be awesome if we could go to a brief demo from both of you. Maybe we'll start with you, Ankur, on brain trust and talking through as you go through a bit of a product demo. I think one thing that we heard as folks were signing up for this session is that there's some questions around how to structure evaluations for multi step agent workflows where one failure will cascade through an entire process. Just things around best practices. And so maybe as you go through and do the demo you can just talk a little bit about both of those things.

Yes, I'm very happy to. Give me just one second.

We'll see if the demo gods are with us.

I think they are. I just want to pull up the right project. Okay, great, great. So we have an agent built into BrainTrust and before I talk about evaluating agents, I'll just show you a little bit about that agen because then we're going to look at the logs for that agent and a little bit about how we evaluate it. Here is an example of a question I ran. I can do a fresh one. Who uses this feature the most? And similar to things like cursor or Claude code, this uses an LLM with a bunch of tools available to it, in this case to analyze the data in my logs. And it will run a bunch of searches over the logs, which take advantage of a lot of the things Simon was talking about. For example, like in BrainTrust, if you run a search and we read from object storage, it might be slow the first time, but then all that data is cached in NVMe and the second search is much faster. So this thing will take advantage of that and it's going to try to run a bunch of searches and then figure out who uses the product the most. Now, behind the scenes, as that runs, it's actually generating traces that look like this. And this is like a classic agentic trace in something like braintrust. So here's like a system prompt with some instructions and then the user's question and then a bunch of tool calls and you can see them interleaved. You can understand the timing of each of these calls and you can also do stuff like understand the conversation in a more chat like experience. So you can see, hey, I wanted to see what these results were when it returned. It let me actually go and dig into that. And this is a lot of the debugging that people do when they're actually playing around with something like brain trust. Now the other thing you can do that is quite cool and something Simon helped me figure out when we were first working on BrainTrust is search. And so I can search for something like my email address and you'll see that it will complete very quickly and it will also actually update all of these aggregates as well. And that is again something that takes advantage of some of those properties. Simon was talking about a lot of the data to calculate one aggregate is similar to the data that calculates another aggregate. So once it arrives on NVMe, it's quite easy to actually run a bunch of redundant or similar calculations over the data very quickly. And I think again, it allows us to build like a really cool user experience. The last thing I'll show you is how this actually translates to evals. I have a project and this is a little bit stale. I should probably rerun this. But when we were first shipping this feature, we actually ran a bunch of model comparisons. And so this is a pretty popular thing that people do in BrainTrust. They take use cases like the one that I just showed you. And if we click into this, you'll see that this experiment actually looks very similar. Like these logs look very similar to what we were looking at before. You can do tracing in production, but you can also do tracing when you're evaluating to get all the same debugability. And if we go here, you'll see we can actually evaluate different models side by side and understand the trade offs and stuff between them. So this is the kind of stuff that I think is really powerful. There's a lot we can talk about with questions if people are curious about how to specifically evaluate agents. But the one thing I'll just quickly call out is I think it's super important in addition to running a bunch of really good scores, to also track a bunch of metrics that help you understand the relationship between LLMs and tools. The most useful thing I find is looking at stuff like tool errors and trying to see if I use a different model or if I change the prompt or something about how it runs. Do I suddenly get like a lot more latency in my LLM calls? Do I suddenly get a lot more tool errors? Stuff like that?

Ankur, if I have a production system set up and I come in and say I can see the logs and 95% of them are great, but 5% of them are wrong, how should I start to prioritize what to work on or what to focus on?

Yeah, I think one super simple thing you can do is capture user feedback and Braintrust allows you to do that quite easily. So you'll see here there are some score columns and you can do stuff like filter. Let's see. Sorry, I think I need to refresh the page. Yeah, we can filter and say, hey, I want to find all of the examples where there's feedback. And this is a pretty quick and simple way to just surface things that may be interesting. And I think that if you are early in your application development life cycle, you probably don't have that many interesting things right here I have six. So I should probably look at all six of these and look at the thread view and try to understand like what actually happened here and why was this a good or bad experience and then turn that into test cases to use in an eval. If you find that you have too many, then you really have two options. The first thing that people think about, which I think is a cool thing to do but not often that useful, is to try to come up with fancier scoring mechanisms. So maybe you have a thumbs up and thumbs down and you have 10,000 of those and you come up with another scoring method, for example, like using an LLM to look at the outputs in addition to that and filter that down further. The simple thing to do, which is what I honestly recommend in a lot of cases, is just look at the first 10 of those 10,000 and you'll probably find something interesting. And I think that every time you go to the logs page, if you find one interesting novel case, then it's ROI positive and so you don't really need to think about it or you don't need to overthink it too much. I think as long as it is relatively time efficient for you to find novel interesting things that you haven't seen before, then things are good. And if it's not, then you should invest in more scoring to help you narrow things down further.

Are there any common dark patterns or common mistakes that you see People make either setting up eval harnesses, trying to close this loop.

Yeah, I think a few things. So the first is that people will only do online evals or only do offline evals. I think you should think about your job. Building AI software is to build a feedback loop because you can't predict what a prompt is going to output. And so you need a feedback loop from what people actually experience to what you're developing. And to build an effective feedback loop, you need both offline evals and online evals. And so you need to basically do both that. And I think some people don't. Another thing that we see people do is only trace LLM calls. It's very easy to do that. Like you can use our libraries or there's some proxies and stuff that will let you do that. And I think you get some visibility into what's happening by just looking at the LLM calls, but you get significantly more if you can capture the inter. Interleaving execution of LLM calls and tool calls. So I think it's very important to put in a little bit of work and get traces that reflect the information that you actually want to see. And then I think the third thing is if you're very lazy about scorers and the anti pattern that we tend to hear is do you have a pre built scorer for hallucination? That's probably one of the most common questions that we get. If you're lazy about that, then you're just not going to get good results. And the reason is that I think scoring is essentially like the AI evolution of writing a prd. It's your opportunity to come up with the criteria that are very specific to your use case that if met, will result in a good user experience. And if you're lazy about that, just like you're lazy about writing a spec or writing a prd, you're going to end up with a crappy user experience that regresses to the mean. On the other hand, if you use it as an opportunity to create differentiation for your product and really think about how to capture the attributes of an experience that you think represent a good outcome for the user, then you'll get a really good experience. And so we really encourage people to create and customize their own scoring functions to try to find stuff that is relevant to them.

What's a good example of that in

terms of we have two customers that are very different, Vercel and Stripe. Hallucination for Vercel means something very different than a hallucination for Stripe and if you try to use the same logic to figure out whether something is a hallucination, I think you might. In the case of Stripe, you might miss things that are actually really important that you don't make up. In the case of Vercel, you might not allow the model to be as creative or open minded, if you will, about the code it generates to help someone achieve what they're trying to do and actually trying to think about what that means. In the context of Vercel, for example, you can statically analyze code and if it references a library that is not imported or doesn't exist, that is a very verifiable form of hallucination that is just completely irrelevant for customer support bot in a financial company,

that's helpful. Maybe one more question and then we can chat a bit about the demo for Turbo Buffer. One of the biggest gains of the last year or so with regards to these agents has been tool calling and their integrations with third party existing systems. Now imagine when you're calling tools with an agent, you could get back an image, you get back a PDF, you might get back text increasingly, maybe videos and multimedia. How far do you anticipate braintrust will go in terms of storing storing that and creating that trace?

Oh yeah, we already support that. We have a feature called attachments and I think the way to think about attachments is like in a traditional database if you have a varchar or a variable length string, it doesn't store it directly in the B tree, it stores a pointer to it. And that lives in a cheaper and simpler form of storage with some performance trade offs. We do exactly the same thing for multimedia. So you can upload attachments to BrainTrust and they get stored directly on object storage as blobs and referenced inside of the data that's actually indexed. So it's very cheap to do that as a result. And the trade off is that you can't search the base64 content in an inverted index, which no one cares about, and so it actually works quite well.

That's a ruffle. And you mentioned you use TurboPuffer in searching across BrainTrust docs within this platform natively.

We so we are about to release a feature that uses TurboPuffer and that feature lets you search the BrainTrust documentation automatically. The word doc is probably the most overridden term that Simon and I could refer to in this conversation because it means many different things. We unfortunately don't currently use TurboPuffer to power the search experience and stuff inside of BrainTrust, but Simon and I are always scheming about interesting ways we can collaborate. And I wouldn't say that is a permanent thing.

Hold that. Awesome. Maybe with that. I know, Simon, you wanted to introduce to demo TurboGrep and show the capabilities of Turbo Buffer.

Yeah, for sure. Yeah. It's a database, so I don't have a nice view. Actually, I do have a ui. I could show the UI and then I'll show you turbogrop. But unfortunately we are a lot better at Rust than we are at React. So bear with me. Maybe if someone in attendance wants to come help us with the dashboard, they should come join us. This is what turbopuffer looks like in the dashboard. It's very simple. This is our test account. And yeah, you can see here it has almost 5 billion documents and the invoice is only $700. Now, we don't charge ourselves, but gives you an idea of the kind of scale you can get to with good economics on turbopuffer and very simply just shows you what's going on. You can see the namespaces that you have here. When we test, we just create a namespace for every single one of the unit tests that we run. So there's a lot of them. I think there is, yeah, there's tens of thousands. Some of our customers have millions. But very simple, simple UI here. I will now show you turbogrop. And turbogrop was just something that I've been hacking on when I have time. I think Ankur has more time to code than me. I don't know how. I don't think Anchor sleeps very much.

I can tell you my formula after this.

Yeah, please. TurboGrep is basically just like RipGrep, aspirationally in the limit one day. But instead of searching with regexes and full text search, it embeds the entire code base and then searches over the embeddings. And what that allows you to do is you can do a search like upsert to vector search vector store and it will return the actual function that's doing that. In this case, like writebatch, which is doing that. So inside of my editor I can do upsert to vector store. Right. It doesn't know. It will have to find out that it's servo puffer and that it's write instead of batch and it just finds the function. So I use this all the time because I honestly read a lot more code than I write code. And being able to do a semantic search like this is extremely useful. You could also do something like, how does it do a fast search while indexing to try to lead you to the right thing. And it leads me to this function called Speculate Search because it's creating the embedding and stuff while it's chunking the whole code base to keep it up to date. And then I can go read this from as I map my own vocabulary onto it. I could do something like ASCII emoji for Pufferfish and it will take me to the progress bar here, which is like an ASCII Pufferfish that's inflating and deflating as you're doing things. So it's a very simple demo, but a lot of our customers do this sort of fundamentally connecting data to AI, but whether it's code or documents or PDFs or some kind of unstructured data. So you can see here, if we reset this namespace, then what it will do is that it will chunk the whole namespace, then it will create. Then it will find the closest region, which I'm currently closest to the Toronto region for turbopuffer, and then it will create an embedding. This demo right now uses Voyage and create the embedding and then it does the TPUFF search and it took about 25 milliseconds once everything was written in and then served the search result. So now if you do this, it will chunk the whole code base again and then rerun it. So it's very simple, but we wanted to have something simple out there for our customers to use. And also it's a very useful tool for me. This will work also on much larger code bases. Actually, maybe we can use that. Do I have like Rails or something? I don't know if this is going to work. Split a string. I don't know. Let's see here. Let me do this verbose. So you can see here it's chunking the entire code base and then there is 53,000 chunks and it starts to just pump these into the whatever H100 voyage is running to create the embeddings. It's a bit slow because I'm using a pretty slow model and I don't have all the rate limits Turbopuffer can do. About could do up to 10,000 or more writes per second. So this is currently limited by the Voyage model or my uplink one of the two. And it's just going through the code base and eventually once this finishes, in about another minute, it will return to query. But yeah, that's it. That's my little demo.

Now, Simon, like Once you've chosen turbopuffer or your performant database system, there's a lot of levers. You have the chunk size, the embedding model, the re ranking model to improve search quality. Where do you recommend people start and what has the sort of highest effort to payoff ratio?

My general answer here is find an embedding model that's fast. Because we've seen customers who have embedded tens of billions of do documents with an embedding model and then they get great throughput, like more than the 500 a second. Here I'm getting on my little probably trial account with Voyage and they pay potentially hundreds of thousands of dollars to embed all of their like billions and billions of documents. And then the Query latency is 300 milliseconds. So you want to choose a fast embedding model. And that can vary a lot because you might run all of your workloads in Frankfurt and whatever embedding provider you're using might not give you any choice over the region and rank it all to Oregon. And now no matter what, you're paying 200 millisecond penalty to get that embedding unless you have more control. Okay, finally finished. So we recommend spending a little bit of time, you can use any code agent to very quickly just find the one that's fastest for you and run it. The other thing is that we recommend that people start with something very simple. Just start with a simple embedding based search on a small use case and then get it in production. And then don't try to do anything fancy until you have the evals in place. Because if you start to do something complicated before you have the evals, you end up in this space where you build a complicated system, but you have no idea where in the complexity the value is. So you want to be able to do that, to start building your intuition. So what I generally say is stand up the simplest possible thing that passes vibes and then try to get the evals going as quickly as possible. And then it's about creating the evals. And creating the evals is like you have to create a bunch of evals and then the people on your team have to start creating evals. And you should probably also recruit your cousin to create evals, but you need a lot of evals. When I was working on search at Shopify, we spun up massive teams of just people that sat and created these evals. Today we have other means of doing that, but it's a very traditional way of doing it and everyone knows that's the best way to get this good search result. And then you can start layering on complexity, right? You can do a query rewriting layer where you start rewrite query. You can use lexical search, you can use multiple embedding searches as part of the same one. You can start to use late chunking, late interaction. You can train your own RE ranker, you can try different RE ranking models but do not introduce complexity into the search pipeline until you have the emails in place because otherwise you will get addicted to complexity that you have no idea if it adds any value that's helpful.

And the other trend we've seen is like as we move to these agents multi turn LLM systems doing searches they might. An agent might first try a semantic search then try the traditional a traditional search and it seems like TurboPuffuffuffer offers hybrid search for those types of use cases.

Yeah, that's right. I think we find that there is. Let's say that an embedding model doesn't know what turbopuffer is. It might start returning like fast fish. I don't know if like a sailfish is the fastest fish and that's not helpful. Right, because it doesn't know what it actually is. Whereas a lexical search model will know exactly what it is because it has no idea, it just sees that it's a string. So embedding models are very good at turning strings into things. But if it doesn't know what the thing is, which is often the case in say you have a notion workspace and you're using some internal wording that the embedding model could never know what actually means, then the lexical search comes in really handy. So it's great to use both. And we find that most of our customers have a lot of success with that similar especially if you're powering some kind of command case search right where you want to search for SI and the embedding model is going to think that that's C and that's something agreeable and yes, right. It's span is for yes but really what you want is just to find a document that starts with Simon. Right. So these things have to play in concert and generally for search applications it's pretty idiosyncratic what works well and we try to give opinions about what we've seen worked well on average. But you need to have the evals in place to go from there.

This is awesome by the way. Thank you. And I can see, I love that you both are basically selling each other's products throughout this entire process. It goes hand in hand.

Well, I think we've seen a lot of customers actually be successful with both products together, which is pretty cool and not a super common thing, at least in my experience.

Yeah, we've seen the same, which is why we were so excited to have both of you on together as part of this, because we knew that it would lead to great back and forth and that there was so much overlap from just what we've seen, even across our portfolio and customers. I know we only have a couple minutes left. Would love to wrap on just hearing a little bit from both of you on your own stacks, like your own setup. Whether it's your own coding setup or tools internally that you all use, anything that you can share that you think would be interesting to the broader group. We found a lot of this show and tell and to provide a lot of value. And y' all are experts and have good noses for what's the latest and greatest at the cutting edge. So would love to hear that

I can go first. I think the main thing that has really changed for me is moving from synchronous programming, which I also don't have the bandwidth to do a whole lot of right now, except between the hours of 9pm and 12am and asynchronous programming. And I think asynchronous programming is quite powerful, and I like to think of it as like, for people that are professional programmers, it's like what Vibe coding does for people that are maybe programming not professionally. And I think asynchronous programming is basically working with a really powerful coding agent or multiple coding agents at once to execute something that is very difficult. And it requires thinking a lot about testing in evals, for example, so that you can let the agent iterate really effectively. It requires reading a lot of code. It requires thinking about scope. Like you can burn out an agent very quickly by giving it too much scope, but if you figure out the right abstraction to let it work in, it can be very effective. And so I've spent a lot of time over the past, let's say six months or so trying to just personally hone that skill. And I don't think I'm like the number one, world's best asynchronous programmer, but I think I've improved quite a bit. And I've seen some people who are extraordinarily good at it. So I'm just generally quite excited about this trend and I think it's interesting to see. Like, it'll be interesting to see how truly productive like great programmers who embrace it can be.

We've seen some pretty wild demos and just even things on what people are doing that have really embraced this on Twitter and even are orienting their sleep schedules around sub agents. Yes and yes.

This is the return of the Uberman man. Like sleep schedule where you sleep one hour and then you're up for an hour and a half or whatever it is. No, it's less than that. It's like 20 minutes every two or three hours when the agents wake up. I think I'm landing somewhere very similar to anchor. I. I also don't have a lot of time for the synchronous programming and finding the one to two hours of focus that it takes to get some of these bigger chunks done. So it will be a lot of terminal agents. I'm starting to use the background agents a lot more to do various small things. Like yesterday I was putting up another case study and it's just, okay, here's the markdown file and then I use this program called Whisper Flow. So I can just yap for a minute about exactly what needs to get done. And generally the agents are very good at that. When you can just speak instead of typing, which is a bit of a bottleneck for me. Then I think there was a, there was about like a six month period where you have to use the coding IDEs and if you didn't, that was a massive performance loss. And during that time I got. I started having wrist pain again, which was why I switched to neovim in the first place. So I'm back to reading and reviewing all of my code in VIM and then having a lot of the both background agents and the terminal agents doing the work. But the bottleneck now very much is reading the code and coming up with good instructions. And so that's what I've been doing for the past many years. So that setup is very well set up in neovim. I have the script called Review where you just pass it a GitHub URL and it just gets the whole diff into VIM in a like in buffers and with chunks in the gutter. And that's been very helpful. And now that you can just comment to the agent. Yeah, I can send you the script. Yeah, it's open source, it's in my dot files. Okay, great. And it's like a teaserly coded. This is like a pre LLM so you know it's bad.

I use neovim as well and actually I went through exactly the same thing, although I started using an IDE when Figma acquired us, just to learn about copilot. But the last act of my IDE was actually writing my new neovim config file, which felt very nice and it was like a nice piece of closure. And there's actually a bunch of cool things in IDEs now, like the LSPs have gotten a lot better, and you can recreate a lot of the IDE features in neovim, I think, quite nicely now.

Yeah. And there's some distributions that make that really easy. And TurboGrep, just to sell my little open source project here is very helpful for reading code in unfamiliar code bases, which I also spend a lot of time doing, like reading dependencies, things like that, which is. Which it takes a while to map the vocabulary that you have in your head of how the thing works to how it actually works.

That's very cool.

All right, team, I think we're at the hour. Thank you guys so much for spending the time. This was great. For everyone on the call and following along online, we'll follow up with ways to contact and learn more about BrainTrust and TurboPuffer. Check out their sites, they're also hiring and enjoy. But thank you guys so much. This was awesome.

Thank you for having us.

All right, thanks.

Take care, guys.