The Great Evals Debate — Ankur Goyal & Malte Ubl

Hi, welcome to the Lean Space Lightning Pod. We are here with a very special episode because this is not really about a company or some launch that someone's doing. This is actually just a debate that I accidentally set off about coding agents mostly not having evals, or they do have evals. It's mostly, it's not really like standardized or it's an entirely, it's a lot of it's vibecoded and I think a lot of it is based on the fact of of this clip that I had from Boris of Cloud Code saying that they basically just vibe it. And that's a $500, probably higher than that, $500 million business that doesn't have evals. So I wanted to invite Ankur and Malte from Braintrust and Vercel to just talk about like the state of evals here and like the evals debate. You guys have your own perspective. Ankur, you got a bit in a bit of like a blogging back and forth on evals. What are you guys feeling? Like what's maybe a position statement from each of you?

Yeah, I mean, I can tell a little bit, like I think the, like one maybe important background on me is also that I worked on Google Search just before leaving for Vercel. It's almost 4 years ago, but like I do vaguely remember it. And obviously that was also before ChatGPT moment and before kind of engineering becoming a mainstream thing. And so I left a world in which I was doing evals every single day and kind of thinking that that would be it, right? Like I wasn't really seeing that in my future as someone joining an infrastructure company. And so it is both kind of fun to see it coming back and I think it does give me a little bit of extra experience for what, like how it helps and how it doesn't help. One thing I tell people for sure is that like you have to put in the work, right? And even vibe checking, like as the eval, like is important. Like if you are a product manager in Google Search, you're probably doing like 500,000 searches a day. And if you are, but then you want to do re-evals because if you want to do 50,000 searches, you're not going to do that yourself. And I think there's something very similar here where essentially essentially you just see like, hey, I want to know if I'm doing well and how fast can I find out? And like, yeah, the vibe check can tell you really fast, but if you want to know tomorrow, then the only way to do it is to have someone do the 50,000 searches or the 50,000 coding exercises, right? And if you want to know in, let's say in 3 weeks, you can run an A/B test. And if you don't care, then you, you know, do neither of those things. But it is like, I think like the way I think about evals is essentially like it's the thing that can tell me tomorrow whether my change is good and I can operate without that knowledge, but it's super, be helpful.

Yeah, I have very little to add to that. I, I, um, I strongly agree. I think like fundamentally, if you're building an AI product, you're dealing with this like non-deterministic magic and you don't, you can't like type whether it's a model or prompt, whatever, you can't really type something and know what's going to happen. So you need a feedback loop. And I think that fundamentally evals are one important factor in helping you build a feedback loop and offline evals, which I think is what a lot of the debate was about, are both the most challenging to build feedback loop. And also the most efficient once built. And then I think A/B tests are a little bit less challenging to build and a little bit less efficient. And then like pure vibes are basically like zero effort to build, but then also the, the least efficient. And you have to do a little bit of all of these things. And I think what I've seen both firsthand from building Loop, which is our own agent and brain trust, but much more secondhand from working with Malta and a bunch of really great companies building AI products, is that the best people are extremely deliberate about the investment in each of the feedback loops and they're constantly questioning whether they've put enough energy into each of the feedback loops and making them efficient. And then I think coding is, it's like such an interesting topic for a few reasons. The first is it is the use case that has the most product market fit in AI, I think, other than ChatGPT. And so it is just an important and interesting use case. The second thing is that to a large extent, it's very verifiable. And so it's like, it's almost insane that it's not evalued more, or at least the fact that it's not eval more is super interesting because unlike something like, let's say creative writing or summarization, et cetera, where there's some nuance, there's a lot of objective stuff that you can, um, evaluate. Now, if you talk to the people that are really deep into it, they'll disagree and say that refactoring a codebase is something that's very hard to evaluate, or the visual quality of an artifact that you produce on Vizier, very hard to evaluate. And I think therein lies the, uh, the subtlety. But then, you know, the, the third point, which I've watched Boris's podcast, which I thought was like ultra ultra insightful. And I think the, the thing that's very clear from, uh, what he said is that based on how people are using Claude Code internally and the enthusiasm in the community, he feels like they have a pretty good feedback loop. And I, I think that's awesome. Like, it doesn't really matter whether they're running a script that has a for loop that iterates through a bunch of test cases and, you know, whatever, that's their feedback loop. What matters is that they have a good feedback loop. And I think that's ultimately what you need to do. And however you get there is, is how you get there. And if you're happy with the feedback that you have, that's awesome. But I do think it's important for everyone to think about, do they have a good feedback loop? And if not, how do they make it more efficient?

Yeah, maybe one thing to add on what Ankur was saying, just yesterday, I think we saw both Cognition, congrats Swyx, and, and Cursart ship RL fine-tunes of unnamed open-source models. And the reason why you can do that is because you can implement those RL loops based on what is effectively an eval that you built. That you run at every iteration. So that's the reason why this coding stuff works so well is because you can write these evals, right? And I have this like hot take, and this might be like not fair to Boris, but I think he comes from a particular position of extreme unusual privilege, which is that he works at an AI lab where like people in the office next door are like writing the evals and are like training the model like every day in exactly that way. And so It might be like there, there, I'm sure there's some pretty deep hand-in-hand work in making the agent work well and be, and the model be particularly well trained to what the agent needs and so forth. Like there's going to be feedback loop like that. And I think we have to consider that when we make recommendations for the world out there, which is different, right? Like where you don't have this privilege and where you have to kind of make do with what you got from the either big model labs or the open source models, et cetera.

Yeah, I don't know if there was like a particular point to respond on that. I will say like, I think all the agent labs, if you can call it that, you know, Vercel has your own benchmarks, Cognition has their own, Cursor has their own, and typically everyone trusts their own benchmarks much more than the public ones, but then they're not comparable because they're not open source, because probably because they're IP or because they use proprietary code.

Vercel's is open source, or at least they have a big open source one. Yeah.

Well, Next.js, the Next.js one.

In and out. Like, yeah, we open source Next.js. We intend to do it for the other tech that we're building. And that's probably because we come from a slightly different permission. Like, our goal is just that all the AI models and agents are just incredibly good at the tech we've put out. Right. So we incentivize self-interest in doing 100%.

Yeah. So, you know, and then, you know, on commission, like a lot, it's a lot of banking customers that like would not appreciate some of the, those evals work being publicized. But I think like, yeah, anyway, so the point being like, actually we spend a lot of effort on public benchmarks and in fact, facts, like the team, like really burned themselves out over doing this. And in the end, like people look at a chart and they go like, all right, like it's, it's, and then let's check the vibes. Right.

So, well, yeah, I mean, I think those evals are, they're not a public benchmarks that people publish are not in service of creating a feedback loop, which is not to say that they're stupid or anything like that. It's just that the value proposition of publishing an eval is completely orthogonal to the value proposition of building evals in service of building a good product. I think the purpose of publishing benchmarks is marketing and it's, it's good marketing. Like I come from database land where TPCC and TPC-DS, all this, those are standard benchmarks. And then people like ClickHouse make ClickBench or whatever, they make their own benchmarks. It's, it's very, it's a very sort of like mostly objective way of communicating the value proposition and trade-offs of, of your technology. But I think it's totally independent from the practice of using evals as a, as a feedback loop. Mechanism for iterating faster.

Yeah. Okay. Awesome. I want to go back to a little bit something that Ankur, you touched on. Let's just, let's just call these like the primary dimensions of evals. You talk about effort, you talk about what is that effectiveness or efficiency. I think there's one thing that I don't know how to capture, which I'm curious how you guys think about it, is open-endedness. This is not like, this sounds like a made-up term, but it's a real team at Google that is actually exploring open-endedness. And I think Malte, to make this relevant to your background, search is open-ended, but it is not as open-ended as CodeGen. And like, I think the reason people lean on vibes more and spot checks more than traditional evals or offline evals is because they actually cannot predict what was— it's very easy to kill-climb things that you've already written evals for, right? It's always the stuff that you don't know that you don't know that is going to catch you off guard, like some competitor is going to come in and eat your lunch or whatever, right? So to me, that is the, I think the debate of like why people are moving from offline to online or online to vibe checks is because it costs a lot of effort to do offline and it probably lasts maybe 6 months if you're lucky.

Well, I think, I think you're conflating two different things. One thing is sitting in a room and enumerating a set of scenarios or tests that you think represent a workload. And then the other thing is choosing to run those tests outside of production. And both of those things are activities that can happen offline, but I don't think anyone who's legit, who's doing offline evals, is doing the former anymore. Like, in fact, when we started Braintrust and we talked to customers, we're sort of very excited about Braintrust helping them create golden datasets. And now I actually think that term is kind of a dirty term. People don't really want to create golden datasets. It's, I think it's, it's, it's often a wasted effort to, to the point that you're making. I think the best teams view offline evals as a mechanism of reconciling what they see in production with real users who are using the product with tests and iteration that they can do offline. Like the best teams that we work with on a daily or even more frequently sometimes basis are discovering use cases from logs online and then pulling them into their environments and then playing with them. And I think the difference between doing offline evals and online evals or offline evals and just pure vibe checks is I think the pure vibe check or pure online version of this is you observe some behavior in your log. You're like, oh yeah, let me, uh, like play with it. Maybe you tweak the prompt a little bit and then you deploy it. And then to Malte's point, like either statistically insignificant, insignificantly now, or statistically significantly 3 weeks later, you observe the impact of that change. And then I think the offline eval way of thinking about it is you discover some interesting behavior from prod and then you have 1 or 2 clicks that allow you to capture that behavior. And repro it offline, iterate. And as you iterate, you not only test that, but you also test a bunch of other scenarios from other users that may also be recent. And you gain a little bit more confidence about the impact of the change. And then maybe you ship it still like today or something. Like frequently when I'm using v0, I will complain to— there's an engineer at Vercel named Max who's like an incredible engineer. And when I complain to him, like, I know that 2 clicks later he's taken my complaint and is playing with it, not in prod, but on his laptop. And I think that is what offline evals are really about. It's not about manufacturing a golden dataset anymore.

Anything to add?

Yeah, I think there's definitely the workflow velocity. There's the understanding of what the eval tells you, right? I think that the common quip that if the vibes and the data disagree, the vibes are probably right is true, right? So you have to be honest with yourself. Do they agree and iterate? On them over time. The other thing that like, I don't know, it's just my weird head who thinks about it in these terms, but like to me often the, I like to say the first derivative controls for methodology. Like if on a, you have your number, right? And it's more important that you know that it's good if it goes up than that you understand what 1 or 2 on it means. That's very difficult to understand sometimes. And so you need to like, if you invested in your evals, the opposite is true, right? Like, so it's so nice to be able to be be aggressive in the changes you make, knowing that your encoding of what you thought yesterday was a good idea for your product hasn't regressed, right? Even though the world has changed, like that is just like very traditional kind of having testing in software engineering that allows you to move faster because you have higher confidence. And I think this is like, you just have to understand the trade-offs between that and potential extra work, obviously.

One of the things that I've noticed along these lines, I like, I hate writing unit tests personally. We, I write them actually Cloud writes a lot of them now, but I've always hated them because they're so brittle. Like I change the design of my system and, and then the unit test slows me down rather than enables me to verify whether my system still works. I've always been like a hardcore integration test person. And I think the same, or maybe an analogous set of design principles and trade-offs apply to thinking about evals. Like if you construct evals in a way that represent the true north star of the problem that you're solving, then they tend not to break as you change the underlying system. Whereas because if you hardcode them to a narrow subset of like an implementation detail of your problem, they can be very, very brittle. And so I also, I think you have to, if evals are not working very well for you, obviously some of that is like the cost of evals and maybe the tools that you're using or the framework or whatever. But some of it, you kind of have to zoom out and say like, okay, how do I, how do I reframe how I'm thinking about what I'm evaluating so that this can be productive for me? Or if you choose not to, maybe your competitor will, right? So maybe use that that as a guiding principle.

Yeah, I think evals as a competitive advantage is underrated. I don't know if that's spicy to say. Okay, so, and I wanted to maybe ask Malte, like, are there any best practices that you guys have developed on eval, maybe around braintrust, or like principles that you think that maybe Ankur hasn't really covered because he is from his point of view, you're from sort of the user point of view. Just, I just want to leave it open-ended as to stuff that you've seen.

Yeah, I think from a just pure workflow point of view, and Ankur kind of already like highlighted some of this, but I can maybe go a little bit more concrete, right? Like the, the actual workflow is that with this tool like Braintrust, right? You essentially are logging and you're, you're collecting facts about your production system. And what's really interesting is then how do I turn the thing that I know people did in production into, into, into something that I can validate in my codebase. And so what we do there is that we try to, on some relatively regular level, kind of upgrade failure modes that we observe in production into something that we then in the future can measure how well we do based on those previous failure modes, right? And so you're in this kind of constant loop that is actually in a way very vibes-driven, right? Because there's the ground truth of like what actually matters, which you can obviously sometimes you envision what it is, but like in practice, What actually happens is that you are driven by the failures and the successes of your production users. And then you lift that into something that's expressible on the code side. I think the other thing that maybe is worth mentioning is that because we're talking about coding in particular, it is in fact possible to do the essentially like evals, like does this program compile? Does this program start up? Does this program start up and render the first page without client-side error? You can create some of these very trivial, like in quote unquote trivial evals, but like what we found is that they're actually, they're extremely valuable, right? That's actually where, so the, like what we then, what we actually literally did is we turned those evals into our RL pipeline, right? And so we essentially trained a model to fix the errors that the eval was previously detecting. That is extremely effective. And it's while, you know, basically what we do is we have this composite model architecture where you run the frontier model and then we run the fine-tuned model after to fix its errors. That doesn't perform better than an agentic loop, but it's orders of magnitude faster, right? Because all the, like, you know, like the agent might all the time find out, oh yeah, I made a mistake. And then it totally knows how to fix it, right? That's not the problem, but it's fixing it using a model that has somewhere between 30, and now maybe the fastest one, 900 tokens per second, which is still snail pace for trivial errors that these models make all the time. Like const a 1, next line, a 2, which doesn't compile, but then it's very, very easy to fix.

Okay. So you're saying you don't do an agentic loop, you only do frontier draft and fine-tuned fix. Is that correct?

That's not what I'm saying. I'm saying we do avoid having to do the agentic loop in more cases so that the total time goes down because you use the the agent loop for fixing difficult things and not for fixing trivial things.

So I'll briefly mention that in the SweepGrep work, part of the RL objective was time to complete. And so you can just actually constrain the number of turns it takes if you have the agentic driver model not be incentivized to take forever because we want to prioritize human interaction.

Yeah, I think there's another point. I'll inject a little bit of spice into this discussion that Malte made, which is is using evals as a way to get product insight. And I, and I think this may be where evals caught the attention of Lenny. I don't know for sure, but we've, we've, we've seen like a massive surge of product manager, product managers, uh, and designers getting interested in participating in the eval process among our customers. And, and I think the reason is that in many ways, coming to the table of product building with representative examples and criteria that articulate what good versus bad is for a use case. Is just a more precise and usable form of product management than writing a spec. And so what we've seen from some of the best teams, um, is like actually leaning into getting product managers very, very involved in the eval process. And if a product manager who has a very good instinct about what the vibes should be wants to communicate that to an engineer who is maybe less familiar with the vibes for a particular use case, and in coding this doesn't make any sense because the engineer has the best vibes. But if you think about use cases like finance, for example, or healthcare, where AI is providing like a massive amount of impact, the product manager has a way better, usually has a way better understanding of what the user actually wants. One thing they could do in like 2015 is write a spec that describes like, hey, you know, the user wants this and therefore blah, blah, blah, blah, blah. Now what they can do is construct evals that help illustrate that point or even construct a scoring functions, or LLM as a judge, as they're called, that are, they're kind of written like specs. They have the same structure and they're kind of looking for certain criteria, but they just end up getting utilized differently. And I actually think there's a lot, I think, I think there's a lot there and it's, it's, it's also somewhat coincidentally interesting that that particular phenomenon is mostly irrelevant to the coding domain, but is like highly relevant to all the other domains in which someone might use AI.

I think that's right. And actually, I think the magic keyword that everyone's saying is rubrics these days for LMS judge.

Really?

Isn't it? I don't know. That's what we use. Okay. So like, yeah, we built this thing called the Junior Dev Eval. We're upgrading the Senior Dev Eval, working with like McCourt and like other, a lot of these like high-end annotator companies to do all these. And coming back to the point on product management, yes, I think it's right. I think it's both the managers love, like, you know, would love to have more of a say in engineering, right? And they write this like 50-page paper PRD, no one reads it, or they have to go engage in this back and forth and say like, well, you didn't exactly get what I meant. But here they can actually come up with a number, and if the number went up, they're happy, everyone's happy, right? And they get to treat the products as a black box. So it's a—

I don't even think it's just the number though. I think it, if you just think about vibes, like forget about the term eval and you're working in the finance domain and like, I don't know much about finance, but if I like, let's say I build something that, I don't know, processes an invoice or something. And Swyx, maybe you're like a, you know, finance wizard and you're like, hey Ankur, the vibes are off. I'm like, I, what do, what do you mean, dude? I have no, I have no idea what you mean.

Yeah.

Um, how are you going to communicate that to me? Domain expertise.

Yeah, exactly.

Exactly. Yeah. So, so you could write a spec or you could just give me some examples. And I think the beauty of AI is that give me some examples is actually a very, um, powerful way of communicating, uh, uh, product intuition. And so again, I think like if you just forget about the term evals or, or, and you, and you don't like, I, I think vibes are actually a form of evals. They're just like a very, very expensive scoring function that is extraordinarily accurate in a lot of use cases, maybe not for compiling code, but certainly for like aesthetic quality, for example. So if you, if you just generalize and you think of evals as including all of these things, I think it's a great way of communicating product intuition.

Yeah, I agree with that. I would say like the, so that's one of the responses to the Great Eval Debate, which is, well, vibes are also part of evals. Then I'm like, well, then you're, then just everything is evals, right? Like you're always doing evals that, you know, because you are an evals person, you obviously see the whole world in your point of view. But I think we're trying to like push towards people towards—

We can stop using the term evals and just call it feedback loop because I think that's, I think that's not there's no feedback loop vendor or something like that's not the perspective I'm trying to communicate here.

No, no, no, no, I'm not accusing you of that. Yeah, I think feedback loop is right. I think that's why there's also all these like RL startups that are coming out. We're working with Applied Compute and like there's like 20 different ones. I just had a call the other day with another one and it like feels like it should be vibe codable, all these environments. I don't know why they're so hard, but I wonder if you guys have interacted with the RL environment startup groups and like, is that a part of the evals movement that is happening or is that separate?

Yeah, I think we obviously are familiar and friendly with a bunch of these companies and keep an eye on it. And I think that I like to think of evals as like really like 3 components, like, you know, data or examples and then something that you're trying to evaluate like an agent or whatever and scores which help you compute compute whether, you know, your thing works or not. And I think RL environments, um, they're, they're like definitely, uh, a powerful form of the third thing and not necessarily a powerful form of the first thing, data. And I think that what we have really tried to solve for at Braintrust is helping people solve the data problem. Like we, uh, we, we, we are really, really about helping you capture data from prod and figure out what's worth looking at and and pull that offline. And I think you can use that to help inform how you use an RL environment down, down the line. I would say, practically speaking, outside of companies like yourself, Swyx, that are at the very frontier and doing like training your own models and doing like really crazy stuff, we haven't seen RL trickle into the average company that's trying to figure out how to just use a few prompts and maybe build like a basic agentic loop. Loop and get it working. I'd be quite excited for that to happen, cuz I think that the RL paradigm is more programmable than manual, and I think that's quite cool. But I think what I've heard and experienced so far is that you have to have a pretty specific set of expertise, which may be what you're alluding to with the vibe coding thing. You need to have like a pretty specific expertise to design the RL environment in a way that's not vulnerable to reward hacking. And I, I think that either you'll end up with some, um, fixed number of very well engineered RL environments, or You need to somehow employ this expertise to apply it to the N+1st problem. And I don't, I don't see any evidence yet that the average company that's trying to build an AI application will be able to recruit that expertise. It's kind of an open question to me how, how far it will propagate outside of the companies like yourself who are at the very frontier.

I'll just briefly interject in case Malte has something to follow on. Actually, I see a lot of it in computer use in terms of the RL environment. And I think you're coming from a very very human-centric point of view of data. Like humans are the source, humans and domain experts are the source of truth and they're effectively distilling their knowledge into the environment using as the dataset. I think that the, what I see the potential of RL environments is, is that these are a lot of encoded opinions in a realistic environment, which would not be conveyable at human scale. So you're trying to scale up individual humans by just having them construct construct realistic environments that embed a lot of normal assumptions that an RL agent can go in there and just work on. Yeah. So I think the difference is kind of trying to decouple from the humans because humans are expensive and slow into an environment which you can parallelize, control, and play back and forward. Yeah, that'll be my twist.

One comment I wanted to kind of dig in a little bit, and it's a little bit not directly related to what we were discussing, but I think there's an interesting link. Is that, uh, Swiggy, I think you were saying earlier in the conversation that there is basically, obviously people have proprietary evals, people publish some notion of leaderboards publicly, um, which are marketing and they're like, you know, people find that interesting primarily, but they don't have a particular purpose. And I think there is a, like the Next.js eval that we released that you already mentioned and which we, you know, we're planning to do more in that space, I think points to an interesting future where the question is not so much like, you know, is there going to be an RL process somewhere or is there going to be a better model, but like, can there be kind of an inversion of control of like who's writing the evals versus who is building the agent and who's building the model? Like right now it's always the same entity. And I could see in the future that that actually changes. Like if I'm a business and I want an agent to be incredibly good at like automating my business, because my business model isn't that someone clicks around my UI, my business model is that something else happens on the backend, then I'm incredibly incentivized to say, hey, you know, Model Lab, hey, Agent Lab, how can we work together so that you have an easy time getting really good at the tasks that are relevant for my business? And so I think that there's sort of this new way of thinking about whose responsibility is making the evals. It might not be the same. Entity that then takes those into production in terms of making a better model or a better agent.

Yeah. I think that's why companies like Artificial Analysis, LM Arena, these are like the independents, like the new gardeners of independent rating agencies for various companies. Yeah, it makes sense.

Yeah, but I think that's a bit of a different take than from what I meant because they are independent, right?

Yes.

But I don't, I exactly don't mean that. That, I'm like, I'm not independent. Like, because I'm Vercel and I publish an eval that I want OpenAI and Anthropic to use to make sure when they ship the next model that they're better at the stuff that I care about. And that's, I think, somewhat different from— now obviously, like, you could have a middle person like Element Arena in there to facilitate that process. But then my main point was like, the creator of the eval is an entity that has— that wants to achieve a goal. It's just that the goal isn't that their own models get better as the goal that someone else's model get better.

Yeah, this, this idea has been around forever, like OpenAI, the evals repo and the sort of expressed, not, they weren't like shy about it. The expressed intent of the repo was for people to contribute evals that they could use to make their models better at. And, you know, the trade-off is, hey, we're OpenAI, we'd appreciate having more evals, um, excuse me, to, to test with. And then in exchange, if you contribute evals that you care about, you'd get that, you know, perform better on those evals with future versions of our model. The interesting thing that we saw, and we've talked to all the labs about this because we work with basically all the commercial customers that they would want to optimize for. The interesting thing is that most customers, or actually I'd say a stronger statement, like all customers are quite afraid and reticent to just hand over the data that they use to do evals on to labs. I wouldn't say I fully understand it, that position myself, and I think that in some utopian sort construct, it could be very valuable for everyone to exchange data this way, but it just hasn't happened. And maybe RL environments are kind of a cool solution to that. I definitely see, I'm like, maybe to illustrate Malte's point, let's say that I'm starting like a healthcare startup and I'm really, you know, my job is to figure out like based on your, like a picture of your tongue, what diseases you might have or something like that. And, you know, GPT-5 is like, okay, at this, but I want it to be like really great at this problem. How do I do that? Well, I could like train train my own model. That's gonna be really hard. I could get a bunch of evals together, like take a bunch of my friends who have diseases, you know, pictures of their tongues, and then upload them to OpenAI's evals repo. Or I could create an RL environment that I publish on some marketplace or something that people can use to actually test. Maybe I have a way of generating photos of tongues that illustrate various diseases. Maybe I could somehow publish that, and then that's a very efficient way for people to get better at that. And this is kind of a white space thing right now. Like if I'm starting this tongue disease business, like over the next several, you know, hours, days, weeks, months, that this mechanism should exist. Like it, it's from a capitalist standpoint, it like makes sense for this mechanism to exist. That's, that's kind of what you're saying, right? Malta.

Yeah, exactly. I think it's always important to track the incentives, right? Like I don't do that if I'm the medical agent startup, because then it's, then it's for me privately. But if I, um, something that I want the world to use through AI, then I'm incentivized to make it, the AI, extremely good at this. So, and that always happens when there's a bit of a detachment in the business model from the, from that task itself.

Yeah, I guess Next.js is kind of like a good subtle example of that, right? Like you don't, you may not want your competitors to be able to generate as aesthetically pleasing designs as v0, but you certainly want every model to be able to generate Next.js with extreme precision.

Exactly. Like that's our explicit goal. And we also want to be able to ship new versions of the framework and then have quick update in the base models to be able to like to essentially learn the new skills.

Have you tried building an RL environment for Next.js?

Well, we just shipped the evals last week, but I think it's a very interesting Well, I think like our own environment for Next.js is just npm install nextjs.

Is it? No, is it? Or can you, can you actually design something that is better? I don't, I don't know. I mean, I think it's, I'm not enough of an expert to, to, to know the answer to that question, but I'd imagine that you can probably apply some creativity and create something that's better.

You know what? The Next.js templates page is the set of our environments because Next.js itself is too horizontal. To be meaningful, but actually focusing on the top like 5, 10 use cases of Next.js, yeah, I would say so. Okay. I don't want to— I'm mindful of your time. You guys are both very busy, important people. And like, but I also, I feel like we come to a good spot in terms of like, you know, if you want to control your AI engineers, you want to control your labs, you should build evals so that it's like a tool of communication and also—

Control might be a little bit of an overstatement.

I just— I'm being spicy. I'm being spicy. No, you know, like he who writes the evals, you know, rules the world or whatever. Right. Like, so something like that. Something of that nature. But no, I think it'd be a very good place to end. I think very educational for people who are like, I haven't really thought deeply about this. Obviously, people can reach out to both of you who are very online and very generous with your time. And yeah, I mean, I think this really is a very deep debate. It's for people who are serious about building good AI systems. And I mean, I fully agree. There's more to have with this debate about how do you push the state of evals. It's something I'm interested in exploring at Cognition. I don't have the time yet or the material. I'm just saying, if we want to define define what's next for coding agents. We have to build the evals and then we have to kill climb the evals. And that's like basically what my job is. So I'm excited for that. Anything to plug for both of you? Anything coming up or any calls to action?

Well, we're shipping crazy stuff every week, so I don't, I'm not sure I have anything specific to plug, but come visit braintrust.dev and join the party. My plug is to listen to the other podcast with you that I recorded this week, which does plug everything we're While we're all here, I will plug the fact that you can now with one click send AI traces from a Vercel application to Braintrust. We are very proud to be part of the Vercel marketplace. It's, it's, uh, it's super exciting.

Yeah. How did that slip our mind? These, these are literally the, the best two guys to talk, talk about that on. Okay. Go test it out. We'll put it, uh, some links to that in the show notes if we can find it. Uh, thank you guys.

Thank you.