Braintrust CEO on Where Engineering Actually Matters in AI

AI is continuous and systems are discrete. Humans fundamentally think a little bit more in terms of systems and, you know, predictability and reliability, consistency than they do non-determinism.

In the past, large companies that received lots of money were basically naturally rate limited by engineering speed. These frontier labs don't have that problem. They can literally just raise money and like build a model based on the money. Like they just throw more compute and more data at it. So you kind of have to ask the question like, what is going going to end up limiting them.

If you're building an agent and you're using pre-trained models or whatever, it's a fool's errand to think that the agent that you're building, the way that you're providing context to it, like all that stuff, that is not engineering and it shouldn't be engineered and it should be bitter lesson pilled. Meaning you should build that system in a way that you can throw it away tomorrow. Right now we're kind of building, you know, like God, and so it's possible and probably economically viable to keep throwing people at the problem to make God like 1% smarter. But when you can't make God 1% smarter, there is like an insane opportunity to engineer God to be more efficient.

All right, thanks, Kevin.

Thanks for having me.

I think people watching this will know you, but let's just very quickly go through your background, um, mostly just to set the stage, because I want to talk a lot about whether AI is actually a systems problem or not.

Great.

So do you mind just kind of giving the rough sketch?

Yes. Nice to meet you, Martin. I'm, I'm Ankur. Prior to Braintrust, I, you know, back in the ancient history, used to work on relational databases and like way before LLMs. I saw deep learning come out and become a thing and started to get excited about how the way that we query and work with data, which was primarily SQL, is not as powerful as what we could do. And so I started a company almost 10 years ago now called Impura, where we did—

It's been that long?

Yeah, it's been— time flies. Yeah.

Wow.

Where we did AI-powered document extraction and that turned out to be like one use case that worked pretty well, although we were using like computer vision models and stuff because they were way more powerful. Yeah, I mean, at that point they were way more powerful than language models. And so there we had like a bunch of customers who were doing different use cases and we might make our invoice stuff better and then make the bank stuff worse or make the bank stuff better and the invoice stuff worse. And we had to get really good at avoiding that. And that's when we built internal tools to do evals and then get good data to do evals and sort of run this feedback loop pipeline. I didn't think too much of it at the time, but then we got acquired by Figma and I led the AI team there and we had exactly the same problem, but this time building on top of LLMs.

By the way, you know what's interesting? Everybody says the word eval. You say the word eval, I say the word eval. I found out very few people actually know what an eval is. So maybe it's worth describing specifically what an eval is.

Yeah, I think evals are the scientific method applied to software engineering with non-deterministic systems like AI systems. So you come up with a hypothesis. Let's say I'm going to try out a new model or I'm going to tweak my prompt or I'm going to throw more context into this agent by fetching from whatever, this API. And I suspect that this is going to improve the quality of my agent, or I suspect that it will make it faster or whatever. So you, you come up with a hypothesis and then you essentially simulate running the system on a set of inputs and you observe the outputs. You might have ground truth, you might not have ground truth, and you try to measure what the difference is. And then you quantitatively look at the difference. And I think what's really important is you qualitatively understand like, hey, this thing, okay, it says it got better, but let me like look at it with my eyes and see if it actually was better or not. And you sort of, by reconciling that, not only do you, you know, you double-check things with your intuition, but you also give yourself the opportunity to improve the next eval. Yeah.

So we're going to get back to that in just a second. But just so everybody knows what we're talking about with evals, So your background before that, you were at MemSQL.

Yeah.

Maybe talk about that very quickly.

Yeah. MemSQL was started right around the time that people realized old school SQL databases were not really built for the web. And I think there was this really big NoSQL thing that happened for a while.

The head fake.

The entire industry head fake. Well, I think the same thing is happening right now with agents and Bash.

Yes. Let's go. Yes. Let's go.

Let's go into that later. You know, imagine the same agents and batch thing, i.e., Hadoop back then. And I think what happened is like many enterprises who are trying to use NoSQL were struggling because, you know, the end users and all this legacy systems and stuff that they had built didn't understand how to speak NoSQL. They understood how to speak SQL.

Well, you think that was the issue? I mean, my experience of the issue was like it didn't provide the guarantees you needed. And so you had to like implement it at the app layer. Well, of course.

I mean, but then like you're like reimplementing an RDBMS. Most enterprises weren't even there though. I think that's what like smart developers who were actually paying for the NoSQL products realized. But the enterprises weren't even able to do POC because, hey, I have Crystal Insights, you know, like, I don't know if you remember that stuff. Okay, I have it pointed at Oracle. You're trying to sell me, you know, NoSQL, whatever database. How do I point this at you?

So, by the way, I think the more pernicious version of this problem was, is like you got sold on this, oh, the web giants are doing this stuff that's eventually consistent. I can too.

Yeah.

And then you end up building this into a system that actually requires strong consistency.

Right, right.

And then all of a sudden you're like, fuck, this is like the wrong thing.

And then like, that's where the rubber meets the road.

Yeah, yeah, yeah. And that may, I would say this costs the industry, companies years.

Yeah. I remember there was this weird moment when I would, uh, I, MemSQL was based in the Bay Area, so we would constantly try to get like the cool tech companies to use us.

Yeah.

Um, and we had a lot more traction with traditional enterprises. So I would often fly to New York. And meet with like banks and stuff and they were using our product and there were, you know, like people who had a relatively basic understanding of SQL were doing like very, very complex financial analysis using our product and, you know, SQL queries that are this long.

Yeah.

And then I'd go to, you know, tech company of your choice and visit them and they had like 50 engineers working on writing a MapReduce job that does like a really dumb version of that, of that, of the same query and just like struggling, struggling, struggling, trying to figure it out.

Yeah. So and also like work Workflow like that is totally different than like database query. I mean, these are almost like two different things. Like a lot of the Hadoop use case was like MapReduce, which is like process a bunch of documents versus like actually something where you would need SQL.

And it makes sense for that use case.

Yeah, yeah, yeah. Which actually brings me into what I want to talk about next, which is from my perspective, working with a lot of founders in this AI space, they tend to come from one of two backgrounds. They're either AI people. Whether systems product people.

Yeah.

And like, you're definitely a systems product person that's becoming an AI person. And like, do you find that there's in any way like a tension between these two things? Yeah.

I mean, AI is continuous and systems are discrete. And so the, you know, the way that we think about stuff tends to be quite different. Like, I think systems people are they're not trying to optimize and they're not trying to necessarily optimize over like a large aggregate. I think in systems like if you're building SQL systems, every SQL query needs to be correct. And so I think where the, like the tension is, is, you know, we always want to build compilers and I think AI people always want to build optimizers and, you know, compilers have optimizers, but they're algebraic optimizers, so they're kind of different. And I think that humans fundamentally think a little bit more in terms of systems and, you know, predictability and reliability, consistency than they do non-determinism. And what has worked well for us is I think understanding as much about AI as we do, which is certainly not as much as like the hardcore researchers, but quite a bit, we're able to provide tools that help AI make sense for people that maybe like to think with reliability and performance, whatever guarantees.

Yeah, let me, let me just try this on for size to see if this makes sense. And which is it feels like a lot of the way that the AI industry has evolved is around this vague notion of the bitter lesson. I know like people use that term like freely, but like, but here's what I mean, which is these models are universal function approximators. and it's distribution in, distribution out. And so they just throw a bunch of data and a bunch of compute at it, and then you get this thing. And it's almost like anti-engineering, right? Where you're like, like, I'm gonna sit down, you know, I'm gonna have a team for 4 months that's just gonna like throw a bunch of data and compute at this stuff. And like what comes out is basically the thing as opposed to like some clever way. Like, see, you read the original Sutton-Bitterlesson paper, it's like, you know, don't, don't do engineering, just do like the thing. And so like, I feel like you come, like, you're a database guy, like database systems product, which is like all about engineering and trade-offs and incrementalism and abstraction versus, you know, here is the God model type stuff. And you're in the position of actually having to rein the complexity of this stuff with things like evals and products or whatever. And so I'm just wondering, like, do you think that over time, like the better lesson wins out here? Or do you think that we're going to have to, like, use a systems approach? Or do you think these things can ever be even reconciled?

Well, I think that even though the thing that we're all fascinated by is this bundle of weights that gets produced and it's kind of like a mini god or whatever, there is a shit ton of engineering surrounding that thing, especially around capturing, cleaning, preparing, distributing, shuffling data.

You mean making— you mean the training part or like just using them in an app?

The training part. Yeah, yeah, the training, the all the, the, even though, you know, the, the artifact, um, is, uh, is in some ways not engineered, right? Um, there's a ton of engineering around it to make the artifact engineerable. And I think the same thing is true in application, uh, use of AI. Like we actually, um, Michaela from Replit, uh, said this once to me, which made a ton of sense, which is a new model comes out, We want to throw away our entire codebase.

Yeah. And I heard a very similar one from somebody that worked at Cursor, and he said, imagine writing an operating system for a chipset every time, like, a new version comes out, like, you had an entirely different machine code or instruction set.

Exactly. Exactly. I mean, I think, by the way, I think that happened today with the new models that came out.

But what do you mean? That's 5.3.

No, no, no, no, no. MX 2.5 and GLM 5. But we will get into that in a second. But, but I think if you're building an agent and you're using pre-trained models or whatever, it's a fool's errand to think that the agent that you're building, the way that you're providing context to it, like all that stuff, that is not engineering and it shouldn't be engineered and it should be bit or lesson pilled, meaning you should build that system in a way that you can throw it away tomorrow. The thing that actually matters, and the difference between the teams that build products that you and I would say work, like Cursor's product, or products that just feel like shitty, is all the engineering work that goes around it to make sure that you can build a feedback loop from what's happening in production to, to what you're actually testing. There's one— I can't say which yet because they haven't shipped it, but one of our customers, which is relatively sophisticated. They are already testing the new Chinese models in their agent harness, and they have very, very precise insight into where it works and where it doesn't work. That harness is— the testing harness is very well engineered. Um, the innards of the system that they're testing is not engineered at all, and that's very much on purpose.

I'm gonna get at the Chinese models in a second, but I actually want to push on this in just one more direction, which is, um, I mean, at some level, like, whatever, you, you pour a billion dollars into a training run and out pops this thing, and then you hope that you've got a good harness that manages it and understands it. And like, that's kind of what Braintrust does as a company, and it reigns in complexity. But I just have this kind of weird, maybe doubt from coming from traditional software, is what if you had a team that spent 5 years and a billion dollars building an operating system, and then it shows up, they, they show up and they just give that to you, like, you just wouldn't have any hope from the outside. You know, like, so is there some way, like, this is the same type of problem, or is the problem reduced a little bit more just because, you know, like, the way that these function is more scoped? I mean, I just, it just feels to me like a billion dollars of compute and data, I'm just using the kind of some rough number, like the current training runs are of that size. And that thing kind of shows up, like, how can we have a hope of even reining in that complexity?

I mean, there's like a hilarious metaphor here for vibe coding, right? Like, you know how people talk about the comprehension debt that you build up while you're vibe coding something and then eventually you need to look at it?

Yeah, yeah.

You're basically describing that, right? So, you know, it's a good question. It's really hard to know what the current models actually are. Like, I don't actually know, and I have no idea what the margin structure is, but it feels to me like what happens is every n days we stop making rapid progress on the quality of the models, and then people engineer them like crazy to make them incrementally more efficient. I don't actually understand whether that means having a deeper, you know, inductive understanding of what's happening inside the weights or not. I sort of doubt that it does. But I do think that when, like, right now we are kind of building, you know, like God. And so it's possible and probably economically viable to keep throwing capital at the problem to make God like 1% smarter. But when you can't make God 1% smarter, there is like an insane opportunity to engineer God to be more efficient. And I do think that these things are in constant tension. Like there's a lot of people I feel like who are, who want to be nerd sniped into making these systems more efficient. But you know, the bitter lesson is, is sort of preventing that.

My somewhat naive view of this is it's really actually a capital flow issue, which is as long as these companies can continuing to raise a ton of money, then you'll always just do the dumb thing. Like there's no reason not to do the dumb thing because it's a lot faster, right? You can't scale engineering. And as soon as they can't raise capital, you'll have to do engineering.

Yeah.

I mean, I think specifically with Braintrust, as a user of Braintrust, the way that I think about it is like you have these things that landed from space, but there's no way you'll— there's no way you'll actually understand them.

Yeah.

So like literally the problem is not to understand them because a lot of people, when they think of e-fails, they think of like, I'm understanding this thing, but like, that's not what's happening, it's like you're almost protecting your app from them. You know, like you're building like whatever machinery is needed to like use exactly what you need to, but like have enough guardrails on so like you can actually have, you can actually tie it to a traditional state machine.

I mean, just like the scientific method, I think evals are actually almost entirely about understanding the problem that you're solving. Like, I think that evals are the, like, product managers should only be working on evals. And I think evals are the natural evolution of like a PRD. By creating a really good eval, you are, you know, making a declarative representation of what your product should be.

Yeah, yeah. No, this is my sense too. And there's another thing that for me along this lines that I think a lot about, which is these models, as soon as they're released, they just sublimate intelligence, right? Just because if you actually look at what's going on, a lot of the AI companies that are not like one of the two big foundation model companies is you know, they're training these smaller models and they're doing all this RL, but they always use the SOTA models to do it. Yeah. It's almost like, you know, like whatever, uh, you know, OpenAI does a billion-dollar training run, that model's out there, and then that's just kind of bleeding out to these other models, which is a form of engineering. Yeah, of course, 100%, you know. But, but, but like, we're kind of in a way, as, as like a, as an industry, very lucky that these things bleed intelligence so quickly. But then it kind of begs the question, like, where does this intelligence go, right? Like, if if, if the engineering, if the, you know, if, if it becomes an engineering solution, then you'll just say, well, now people will kind of engineer solutions until the next model comes. But one place it's definitely going is other models. And in particular, it seems to be going to Chinese models, right? Like in a way, you know, if you just like, you know, you could just wait for 3 months and it'll show up in somebody else's model and you can use that. However, we don't see economically weighted a lot of use of Chinese models.

Yeah.

Right. And so, like, do you have a sense— like, I would love to hear your thoughts. Like, are the Chinese models good? And if so, like, do you have a sense of why we're not, like, economically weighted, seeing a lot of use? And, like, do you think this is going to change?

It's quite interesting. I think what we see is in terms of number of use cases or number of logos, usage of the Chinese models is very low. In terms of number of tokens across our customer base, usage of the Chinese models is very high.

Really? What about dollar weighted?

Dollar weighted, it's low.

Yeah. So interesting. This distribution is so interesting because I think this is probably the most important distribution in the entire industry. It's just very, very hard to get this data.

I mean, just to put this in perspective, the US open source inference providers today, like the day that we're recording, probably by the time this comes out, this information will be invalid. I ran a benchmark. We wrote like a Bash versus SQL benchmark, which turns out to be a pretty good agent benchmark.

Oh yeah, I saw that. That's actually— we need to talk about it.

That was super cool. I got, yeah, I got nerd sniped really, really hard.

Actually, we're going to talk about that next. That was so good.

What's interesting about this benchmark is, you know, among other things, Sonnet 4.5 onwards score 100% with caveats. The caveats are cost, error rate, latency, etc. But they, you know, there's sort of like this saturation point where it's like, okay, the model is actually just smart enough for this use case. And there are trade-offs between the different models that are worth considering. Uh, KIMI-K 2.5 came out last week, very exciting model, or maybe two weeks ago, very exciting model. Um, still in like the 73-75, um, score range for, for this use case. Like, it would frequently get lost and confused. GLM-5 came out, and, and I just benchmarked Minimax 2.5 right before coming here on an, on a, on, um, on their, you know, API directly. And they both score 95%. And the cases that they messed up are just like very small, like arguably grading errors rather than real errors. And so, you know, these, these two models are actually as good as the models that are as good that have essentially saturated this specific use case. And GLM-5 is 3 times cheaper than Sonnet. And then on, on their inference, like Minimax's inference, Mini M2.5 is 3 times cheaper than GLM-5.

Do you think that, that the reason that dollar-weighted these things are a small part of the market is self-cannibalization, or is it because right now the frontier labs are able to like release the next model just to retain the majority of the, the user base, and as soon as like that's done, we're going to see more of a shift towards these open source models? Because it just doesn't economically make so much sense to me why we're not seeing more usage of these. I think, um, dollar weighted again.

Yeah, yeah, no, well, first I think that people use these models, um, which have worse APIs, higher error rates. None of the open source providers give you good rate limits unless you beg them. Like you could sign up, anyone can sign up for OpenAI, um, right now and you'll get really good. I don't know how they do it. Um, but they're just very, very good at managing rate limits and stuff at, at just un, believable scale. If you try to use any open source model, you'll immediately be hit by rate limits and you have to like beg the CEO to get access to higher rate limits. So I think like they're just not as well delivered right now. Another thing is like, that's, that's honestly just saying like it's a systems— very much so. Very much so. A second, a second thing is that that's great. People reach for these models specifically to save cost.

That's the self-cannibalization issue.

Yeah.

By definition, like they're just don't want to pay as much.

Yes, exactly. And not only are the serverless prices cheaper, but if you actually work with the inference provider or you run on your own GPUs, you can get it to be like even cheaper, right? Because you're trading predictability for cost. So if you have like a continuous workload, there's a use case we're going to ship pretty soon. And the rate that we're getting is almost an order of magnitude cheaper than GPT-5 Nano.

Wow.

And so I think that's another like really, really big component.

What do you think? I mean, what do you think happens this over time? Do you think that the frontier models slow down and then like the distribution shifts to like—

Well, I'm a systems person, not an AI person.

So I—

and instead of me— me too.

Is why there's such a biased conversation.

Instead of making— instead of making a prediction, I'll describe this, you know, the system that in which I suspect that the thing will—

and we know adaptive feedback loops, you can't, you can't predict convergence or divergence. So we'll leave that to the listener.

Yes. Hopefully a smart listener will write to us about this. But I think that what we see is, you know, let's say that there's some cadence at which truly innovative models come out. I would say that the last time that this happened was December. Like GPT-5.2 and Opus-4.5 were remarkable. Exactly. There's step function changes and what— no question. And what happens is as soon as one of these things comes out, the entire industry forgets about open source models and no one talks about them and everyone says, why would you waste your time using them? Blah, blah, blah, blah, blah. Except we have a few very shrewd customers who've observed that certain high volume use cases just don't change over time. and they've specifically instructed their staff not to get caught up in this stuff. Some of these customers are like literally still using Llama 3.1, which is like, you know, ancient, ancient news, but they're getting really good results out of it.

This is super interesting. So like, I mean, is this, is this performance, price, predictability, or all the above?

It is, the team that is solving the problem. But remember, like, up until like 2 years ago, engineering teams would spend like years solving a problem, right? To your point, right? So the team solving the problem, in this case customer service, or one, one case customer service is familiar with the quirks of the model and understands how to eke performance out of it for this use case.

And what does performance mean in this context? Is it like latency?

It's both inference latency and accuracy. Yeah. Basically it's like, you know, it's like a, you know, you could say like an RDBMS. It's like a widget, you know, or an alien thing that they understand how to wield. Yeah. They understand how to wield it so well. And the use case isn't really changing, right? Imagine like a high-volume consumer use case for customer service. You know, it doesn't— why change it when you can just make this thing get more and more and more optimized? But anyway, what happens is the new model comes out, everyone forgets about the open-source models. and then, uh, the new models stagnate, and then, uh, the open source models approximate the performance of the closed source models in the, you know, sort of 3-month timeframe that you described. And oh wait, December is, you know, 3 months ago. Um, and, and then everyone says, oh my God, what's the point of the closed source models? We'll just use the open source models. And historically what's happened is that with pretty, uh, consistent regularity, except I think there was maybe a 6-month stretch before '01 was announced, where I think things felt kind of dead and like Deepseek came out and a bunch of interesting things happened.

Yeah. But I mean, isn't Deepseek coming out soon? 14th?

Maybe.

Yeah. Yeah. Yeah. They announced.

Um, should be exciting. Um, but the, uh, with, with relative regularity, the commercial model comes out before the, um, open source models have a chance to actually really disrupt enterprise mindshare. And so it's kind of this, you know, it's like this push and pull thing where if there is a period of time where there's not enough innovation in the closed source models, if they're— if the open source models replicate the performance and, you know, it's enough time, then I think it's just bound to happen.

It's just so interesting. Like, we've just got a different, like, equilibrium state here where in the past large companies that received lots of money were basically naturally rate limited by engineering speed, right? Like it would take 2 years to get the next Windows out or whatever it was. I mean, it's just very, very hard to scale a large software project. These frontier labs don't have that problem. They can literally just raise money and like build a model based on the money. Like they just throw more compute and more data at it. So you kind of have to ask the question like, what is going to end up limiting them if it's not engineering complexity? Is it like maybe there's engineering complexity on like the data cleaning?

I don't think we've reached the point of market inefficiency like that that justifies true engineering investment.

I mean, the specific question is, is let's imagine, let's imagine the following. Let's imagine, you know, let's call Frontier Lab X. Frontier Lab X is able to raise a billion dollars and then it can train a model based on that. And then the model on that billion dollars does really well and it only has to do well for a couple of months to get to the next raise. Then it raises $5 billion and it trains the model and then, you know, and that does really well. And then it raises the next one and it raises $10 billion. Like the question is, I guess, what point in, like in that world, it's able to raise more money than any other company that's downstream of it together. Like Anthropic's current raise is probably more money than the entire ecosystem that depends on it can raise. That's how much money we're talking here. And so in that world, like, when do you ever get to the point of engineering? I think you just have to assume either the money rationalizes, you know, or there is some fundamental scaling limit in building these models, which I don't know what that would be.

And what happens if neither of those things happen?

I mean, it's AGI, man. Yeah. I mean, there is this— sometimes I think to myself, like, so either we're in this one path where, like, one company can keep raising 10 times more money. And then at that point, like, money doesn't matter anyways. Yeah. Because like that just becomes like arbitrarily good at whatever.

Well, money is an artificial construct, right, to begin with. What is money?

Yeah. Yeah. Or, or on the other side is at some point, you know, as the, as the sun gets larger, the surface, you know, has to fragment just because like now, like, you're competing with against absolutely everybody and like, then you're going to see fragmentation. It feels like we're still pretty early and it's like the sun can go for quite a while.

I think so.

Yeah.

I mean, I would probably, um, I'd probably guess that the speed at which companies outside of the coding domain, which is itself, um, a first derivative for real consumers actually taking advantage of AI. but the speed at which like enterprises are deploying and can make use of, and consumers, like, I mean, most consumers I talk to are using ChatGPT or Gemini, um, not, you know, one of many of the interesting consumer AI products that have been started. But I feel like the speed at which the, uh, you know, planets or whatever surrounding the sun can actually ingest the heat, um, is, uh, potentially the first limiting factor.

Oh, that's a very interesting point. So you're basically saying that these models will get to a point where like they just can't even be consumed. So it's actually on the consumer, it's like it's on the demand side because right now it feels like it's unlimited demand.

Exactly. Like use this, this, this eval example I was telling you about where I think Sonnet 4.5 is, you know, it's going to score, it scores 100% on this benchmark and it's going to score 100%. I'm pretty confident. Like it's hard to come up with a question that it's going to really struggle to answer. But it's actually like it's very difficult to integrate that into an enterprise setting and actually make use of that kind of system to begin with.

This is great. Actually, this actually reframes my thinking. I always kind of assume unlimited demand because the demand has been so historically strong for this. Yeah, but like at some point in time we won't know the difference between AGI and AGI++.

Right.

We're like, it's smart, you know.

In my observation, working with a lot of enterprises, I think the demand is unprecedented in that every enterprise is willing to spend a lot of money But the implementation of AI systems is still very, very, very, very early.

Yeah. And do you think this is like endemic to the technology or is it just— No, there's— The enterprise being the enterprise.

There's a natural rate limit at which, you know, human political systems can ingest new, new things, right?

But it's a very secular phenomenon like the internet was, which is— that may be the case for the organization, but every individual is using it.

Every individual is using ChatGPT and Gemini. So I think that's the, the, if you ignore the enterprise for a second, you think about ChatGPT and Gemini, the question or the framing of the question is a little bit different, which is, are ChatGPT and Gemini going to become the everything app for everything possible or whatever? Or maybe one of them wins or, you know, whatever it is. Yeah, it really feels like a two-horse race there. And if that happens, is there, is there sort of a limit or does that just become the unlimited?

Yeah, this is a great point. Like, like for, for individuals, it's like there's a natural saturation point because like, whatever, like, you know, there's only so much I need to do with one of these apps where the enterprise would be like a different saturation point.

But there could be things that there's, I mean, ChatGPT could start doing your laundry and cooking your food. And I mean, no one necessarily, at least certainly I don't necessarily know what these companies are cooking up.

Yeah. But there's plenty of consumer stuff that, you know, these, um, although, although ironically, if you look at solution spaces, dollar weighted, like the ones that actually require automation are, are are not really working in the same level as things that are just basically like, you know, text prediction. True. Yeah. Yeah. I mean, the state spaces feel very different.

Yeah.

It's like the things where you like, like you feel like you're mapping to a constrained manifold, like code or language or whatever, they do a very good job. And then as soon as it's open-ended, it's like, well, we'll see.

I mean, even when you're writing code, if you can constrain, if you can constrain the environment, like what's happening right there, it, you just get them, you get it.

You get to see what's happening right there. Do you actually have an agent going on your laptop? I do.

Yeah, I do.

Yeah.

I'm trying to rewrite something and I, yeah, I wrote a very, very specific test and Codex is, it's doing its best to try to actually there's, it's, there's two things that it's doing.

What's it doing? One thing we have this, just so you know, for those that are watching this, Ankur has a laptop on a chair over there that's half open, so it's not closed because it's churning away at a problem. That is— that's a fucking sign of the times. Yeah.

There's one problem which if we batch a bunch of LLM calls, we can be way more efficient, like almost linearly more efficient. And there's another problem where we have this really cool feature in our SDK Um, but we have to implement it in every language. So I'm trying to work on a way of doing it across all languages and then have a sort of smaller bootstrap thing in every language. Um, so these are, both of these problems are actually really good for agents because you can, we have a bunch of tests that test both of these features and, um, you can sort of define the very specific constraints of what you want the solution to look like. But both of these things are just, you know, they're very hard technical problems. So it's just churning for a while.

Going back to our previous topic on, um, you know, you've got, you know, a billion dollars goes to create a model, they create the model and then the value goes down. Right. And then like, maybe it's like, you know, these Chinese models are, you know, gaining from it. And like, there is some kind of like peace dividend of intelligence to ecosystem. One of my observations has been the companies that grow the fastest tend to be on that token path. Like they're going like, you know, like you know, billion dollars to create this thing and then it just kind of leaks money. If you're like, if you're like on that kind of stream of leaking money, then like you get growth. Now, unfortunately, that, you know, downstream of the big model, like margins are actually very, very tough because it's not even a, it's not even an economics fundamental thing. It's just as an organization, you'll always go towards wherever top line is greater.

Yeah.

You're early on. So everybody goes ahead and does that.

And so I think people also just perceive all the value to be in the model.

Yeah, yeah. I just meant more like, let's say that you're on the token, let's say you're doing a company and then you're on the token path. You could decide to like add value on top of that, or you could decide to like have margin or not. You can actually make that decision and like you can add enough value to do margin, but like it'll come at the expense of growth.

Oh yeah.

And I've just yet to find a startup founder who like, given the ability to like cheat a little bit on margins in order to juice growth.

Yeah.

Won't go for growth.

Right.

And I've yet to find an investor, independent of what investors say, where like they will like give you credit for the margins as opposed to the growth. And so I think like we're all trained to just go for growth and it's so easy to do it with it. If you're on the token path, it's easy cuz you just sell the token for cost. Right, right, right. Like you're selling electricity for cost. So I'm just wondering like, as you build out your company, like how do you think about like being on the token path versus like framing around it, or is this something that you consider?

When we first started, I spent some time talking to Ollie from Datadog, who I really admire and is also an investor in Braintrust, just to absorb some of his wisdom. And he actually described his framework for doing this with cloud. And he said something like, you know, our goal was to be some percentage of cloud spend and try to figure out how we can price our product so that it makes sense. I mean, People have all kinds of comments about their pricing, but whatever. The company is doing really well. I think most people use them would say the product's very valuable.

Yeah, Datadog is awesome.

And I think what I took away from that is we need to, even if we're not reselling tokens, we need to figure out a way to make our product feel valuable on the order of the number of tokens you're consuming. And so that's why our pricing, it's not directly— we don't charge you for the number of tokens, but we charge you a very cheap rate in terms of the number of gigabytes that you ingest. And guess what? A token is roughly 4 bytes. And so that's helped us do, you know, some value alignment there with, with customers. And it also resonates because, hey, our cost at the end of the day ultimately, ultimately can be attributed that way too.

It's so interesting. Like, this is, this, this mirrors very closely what happened when we went from perpetual licensing to recurring.

Oh, yeah.

And everybody had this conversion where they're like, I can sell you this thing in perpetuity, or I can sell you recurring. And basically the recurring was something that would pay the in perpetuity thing off for a period of time, assuming some depreciation.

Yeah.

Makes sense. So like you have to have this equivalency because the entire world goes to like the new model. So right now it's like token-based usage. So whatever your pricing model, it feels like you have to have some equivalency to the token path and you have to know what that conversion is. Based on like whatever the customer is using it or your understanding of it.

Right.

But it's very, very disruptive to like change a pricing model, right? Like, you know, if you're going from like seat-based to usage-based, right? I mean, in your case, like we did that, but even like the billing systems and stuff is a tough problem.

It's a very tough problem.

Yeah.

Yeah. I think like it's hard to change billing that often.

Yeah.

And then billing on the order of tokens, which are super granular, is also very hard. And then of course, everyone's trying to abuse things all the time.

So is there a fraud issue, by the way? Do you see this at Braintrust? Because I mean, a lot of these companies that are basically reselling tokens, that's a lot of part of their business. I mean, it's a, it's a huge issue.

But I mean, our customers deal with it constantly.

Yeah.

So we help.

But you're more B2B, so it's probably less of a—

Yeah. I mean, we also— this is going to change, but we also have a relatively high entry price point, you know, just the set of folks that are attracted to our product or end up signing up. And let's just get it. Yeah, but we, I mean, we do have people that, you know, create ABC1, ABC2, ABC3 or whatever free plans and try to rack them up.

Yeah, yeah.

But I would say that the, you know, the flavor of usage that we offer is at this point relatively well understood. So it's not that easy to abuse.

Yeah, yeah. Listen, so we don't have too much more time. I want to get into the Bash SQL thing because I think that was so funny. So maybe frame up like how you got snared nerd sniped and what happened.

So I think a number of people online have made this qualitative observation that, boy, you know, Opus is so good at using Bash. It— and hey, you know, when I use Opus and I use it with a CLI, it feels way better than when I use Opus with an MCP server. And maybe that's the root of why people are making this comment, but it if only I could just map every problem to be Bash, then because the model understands how to use Bash, it will perform better on this problem. And honestly, I think that's bad quality engineering thinking.

And let me be clear, you're saying Bash, but like my understanding is people are saying I could give it kind of an API instruction data or I could literally put it in any Unix environment and decide to do whatever it wants in that Unix environment.

There's many variants now.

Like people, people, people that I'm familiar with.

Yeah. Yeah. So, so I think what, um, and we benchmarked actually multiple of these things, but what some people started doing is literally depositing things as files on—

Yeah, exactly. Like I will give it, I'll put it in a Unix environment. I'll give it curl.

Yeah.

It can download whatever files it wants. I'll give it access to the internet.

And like, that is like, that's, that's layer 1.

Okay.

Layer 2 is I'm going to create like a fuse type thing. Where, where I create these. I'm not going to— hey, like, you know, I have too many customer support tickets for this thing to look to download every customer support ticket. So I'll create a fake file, one per customer support ticket.

Okay.

And then, and then of course, it gets even more complex. People have now created like virtual environment type things in every programming language. And so you can write like, you know, like a Python Bash system or something. And all of this is basically saying like, Okay, great. Let's assume that the models are really good at Bash and then let's engineer our thing to sort of meet them where they are. And we have an agent inside of Braintrust called Loop. So we benchmark this stuff constantly. We now support SQL directly in Braintrust. So we've of course been benchmarking that. And, you know, again, going back to what I was saying about hypotheses, like To me, intuitively, it didn't seem like models would be fundamentally worse at writing SQL than writing Bash to solve problems. In fact, I think models are really good at writing code, which is harder than writing SQL. And so they have all the facilities to express something as SQL that they could express as a series of Bash statements. And the problem is, how do you organize the data in a SQL environment that a model can actually access? And then, of course, If SQL happens to be a more efficient way of reading the data for the problem, then it might be more efficient. Like part of the reason I think people are going crazy about Bash is that if you're working with code, the most granular abstraction that you can use to manipulate and read code is essentially Bash, right? Because the abstraction that is provided to you is a bunch of files. If you're working with like customer support tickets, the metadata row that describes the customer support ticket has a lot of information. So if you can run a SQL query across those to filter down the set that you should look at more granularly than you should. And so we ran this eval and the results are just like comical. Like, you know, SQL is more accurate, it's more efficient, it's more token efficient, it's faster. The worst models perform better on SQL than they do on like everything.

It feels like this is another dichotomy that's shaping up, which is like there is a whole percentage of population that seems to be like, give it a computer and let us do its thing, right? Which is kind of like this brute force, just give it a computer. And there's another which is like, let's give it computer science fundamentals. And like, really, like, I mean, I had a conversation this morning with like some of the top systems people I know, and they're like, now when I use agents, I like use strong typing, I use referential transparency.

Of course. Like, they're actually like, you're really underestimating the intelligence of the model if you, if you force it to do the brute force thing. Yeah.

And it almost feels to me like the brute force one are people that that don't quite, aren't quite comfortable with like the complexity of like true engineering and computer science. And they actually understand files and they understand like curl and like whatever. So they understand that where I feel like this is a potential for a golden age in CS where you actually understand like what are the tools to make a system reliable and, you know, safe and like improve correctness.

I'll give you an example. So at Braintrust, we have a folder called type specs and it has, you know, all the type specs for The API, the UI, Brainstore, how all these things relate to each other. Now when I write code, I handwrite the type specs because I sort of like pace around our office and I have a bouncy ball. I throw the bouncy ball. I think really deeply about what the type system for Braintrust should be, and I handwrite the type specs and then a bunch of tests fail. And that's when, you know, the agent starts, starts running and it just goes. When I review other people's code, because now people are sending a lot more PRs. I just read the type specs and then, you know, usually I find, you know, we debate about what the type spec should be. And although I still scan the rest of the code, I have a pretty high level of confidence that if we— if me and the person who's submitting the PR agree on the type specs, then it's highly likely that the rest of it will be implemented appropriately.

What about state? Guarantees, or like, do you include that in your specification?

Well, for us, it's challenging because people self-host the product and there's so much data that you can't migrate anything ever.

Yeah. So, um, I'm just saying, like, I can, I can see you doing the types, but that makes a lot of sense. The other thing seems to be like, you know, state management is hard and there are a bunch of fundamental trade-offs that you need to articulate in a formal way.

State management, um, I think as a database person, state is also just types. And I think defining the right types that allow state to flow performantly, that's where a lot of the challenge is.

So for me, things like, there's the simple question of like, here's an index and that would be part of a schema. So I'm assuming that's under your type system, but there's other stuff which is like, It's okay if, if, if, you know, this is, if you read it and it's invalid, you know, you can optimistically update this thing. This requires strong consistency. Like, oh yeah, yeah, that sort of stuff. What you need, I mean, like you've got like, I mean, you process a lot of stuff. So I'm just wondering at what level do you have to get involved for that?

Yeah. Well, personally, I tend to get involved in all that kind of stuff, maybe more than is needed just because I enjoy it. But, I think that the best database systems are the ones that make consistency also declarative and reason about these types of challenges as part of the type system.

Okay, so your type system really is a declarative type system, like the database sense.

Oh yeah.

Includes all of that.

Okay. Yeah, yeah. Now we're not perfect, but my stance is always, hey, we're debating this thing. How do we formulate the problem so that we can express the trade-off in the type system?

Yeah, makes sense. Very cool.

Great.

Well, listen, this has been great to catch up. We've got to do it again. Any last things you want to cover?

No, thanks for having me.

Okay, awesome.

Thank you.