From AI Prototype to Production with Ankur Goyal

The most important thing that I like to ask people is, if I use your app and I complain, how much effort is it for you to take that complaint and turn it into an eval? And the best teams, it's like 1 or 2 clicks, if that. The teams that are NGMI, they haven't even thought about the answer to that, right? And so I think that's really the goal. Like, can you get to a point where it's 2 clicks from someone complaining to something ending up in your evals?

You're listening to Founder Mode, the podcast where builders share what's really working instead of just pitching the dream.

Today we're talking about something that sounds easy but turns out to be brutally hard. And that's going from your AI demo to production.

Yeah, our guest today has spent the last decade building the infrastructure that makes scaling actually possible, and now he's applying it to help companies make their AI reliable.

Yeah, I mean, we've all seen the hype, the, the amazing 30-second demo that really hits, but then 6 months later it's still not live.

That's why this episode really matters. It's about the systems, feedback loops, and boring details that make AI work for real teams. Kevin, we're seeing a lot of failure in AI implementations. The MIT report that just came out, 95% of enterprise AI adoption has driven zero ROI. What's your take?

Yeah, I mean, you know, everybody's striving to now be in that 5% because they've all heard that the 95% and they're all looking themselves in the mirror and being like, which side am I on? And knowing that they're not on the good side, more than likely. And so I think we've been fortunate. We found a couple of the 5%ers in our work. And I think what, you know, stood out to me when I read that is like, what were the things that sort of made you show up in that case? And typically it's a lot deeper work, a lot deeper integration, a lot more hands-on sort of like understanding the use case and then really thinking about it as a business and not saying, oh, I've got this cool UI widget or AI widget that I can apply to like make something see or something happen. And so for us, it's really been about finding something that's like materially hard and then spending a lot of time on it to deeply understand it and how the business sees it. Because today it's so easy to, you know, jump into ChatGPT and grab a doc, a piece of paper, or just some data and ask it a question and it will give you an answer. Yeah, but then seeing that show up day in, day out, that's where really the hard stuff begins. So yeah, so I think the real trick there is just figuring out like where the business value is and like being sure that you can build the thing that matters. To sort of like actually achieve and land in that 5%, which is where you want to be.

Yeah, I think it makes a ton of sense. And it's not about just participating in the AI hype cycle, right? It's doing things that drive business value.

Totally. Ankur's built across like multiple generations of infrastructure, I think, from these structured data to unstructured search to AI agents and, you know, done it quietly but at significant scale.

Yeah. He's also one of the clearest thinkers in the space and he has no interest in selling you hype. So should we, should we bring Ankur on?

Let's do it. Ankur, welcome to Founder Mode. You've gone from databases to ML to AI, and now you're building honestly what's synonymous with eval with Braintrust. Great to have you here.

Excited to be here. Thanks for having me.

Yeah, Ankur, we're, you know, we do a little prep here. We kind of look at all of our guests' backgrounds and the things that they've built and done. Super impressive. Your experience and your history, you've built across so many different layers of the stack, right? Like search, databases, AI, and now AI reliability. What's the common thread that kind of brought you here?

Honestly, it's databases. No matter what, I feel like I can't escape them, or to me, everything looks like a database problem. We even, like at Braintrust, we already built our own database called Brainstore. Nice. But you know, I think I've always, I've always been fascinated by this idea that if you help someone kind of create a really good system that lets them capture data, work with it, and then iterate with it, then they can build really good stuff. And back in the day when I was working at MemSQL, it's now called SingleStore, we helped a bunch of enterprise companies that were generating tons of data, like financial companies who, if they could make trading decisions in 30 seconds instead of like, 30 hours, they would be able to make a lot more money. Or industrial companies that were trying to get data from sensors and make better predictions about when things would fail. And it's kind of the same thing. I think in AI, the more effectively and the more quickly you can use data to improve the quality of your product, for example, the much better of a product you build.

Was there a particular pain from your past work where you're like, wow, evals is the point that's going to be the one for AI that's going to fix it and that kind of triggered this, what is now Braintrust?

Yeah, you know, I almost didn't want to start working on Braintrust because I was so fatigued from this problem. So my last startup, Empyra, we built AI-based document extraction. And it was really, really hard pre-ChatGPT. Now it's a much easier problem. But, you know, back then it was a really hard technical problem. And we supported a bunch of different document types, for example, like invoices, bank statements, insurance documents, etc. And we realized after BERT came out that we could train one transformer model to do really well at all of these things rather than train different models for each of them. But what we found is that every time we improved one document type, we ran the risk of regressing another one. So we might improve the invoice performance and then break bank statements. Now you probably see where this is going. Our financial customers were not particularly excited about that. And so we had to get really good at doing evals. And as soon as we built evals, you know, we went— I think we used to argue about model decisions for like 3 or 4 months at a time. We went to shipping a lot faster. And I saw that like really transform how we built stuff. But then it became really hard to get data to do evals. And that's when we started figuring out, okay, how do we link together our observability stack to getting data to play with offline? That felt very weird at the time. And we had to write a bunch of scripts and build a bunch of stuff, but I didn't think too much of it. And then Figma acquired us and I led the AI team there. And that was like right after ChatGPT came out. So we dropped everything that we were doing and we started thinking about, okay, how do we use LLMs to make Figma's product better? And funny enough, we ran into exactly the same problems. Like we kept building stuff and then another team would start playing with it and break it and We ended up basically building the same tooling again. And after doing that twice, I was really sick of this problem, so much so that I didn't really want to talk about it.

And then you started a company to solve the problem.

Yeah. And this brilliant guy, Elad, was like, hey, you've built the same tools at both of these companies, but now it's not just you in a corner working on AI stuff. There's a lot of companies like Instacart who are building AI products. Maybe other people have this problem. And I was like, oh man, I'm so sick of working on this internal tool. But we talked to a bunch of companies, including Instacart, Notion, Zapier, Airtable, Stripe, a bunch of our early customers. And they were like, yeah, we're just starting to figure this out. And that's how Braintrust kind of came about.

That's amazing. So you've said that going from prototype to production in AI is kind of an uphill battle. What do most teams underestimate?

I think, you know, once you ship your product, you start to get feedback from users that you didn't expect. Honestly, that's true with any product. But in AI, what happens is, you know, someone uses your product. Let's say you're building— we'll keep using the Instacart example. Like, let's say you're building a tool to help people construct their grocery cart for the week. What happens is people will find things that don't work, they'll complain to you, and then you'll realize like, oh wow, you know, the user is right. I need to go and edit my prompts, or I need to go and like change, you know, the model I'm using, or maybe try decomposing the problem, whatever it is. Um, and so you try doing that and, uh, you play with whatever the user was trying to do and it improves that user scenario, and then you ship it, and then another user, uh, complains and says, hey, yesterday I was able to create my shopping cart, but now it's not adding bananas to it. You're like, oh my God, what did I do? And basically, when you're prototyping, you tend to have this razor focus on just a few examples or maybe one thing at a time that you're really looking at. And that's a really great way to make progress and iterate on an AI product. But when you actually ship something and you have a lot of users, suddenly you need to take into account the collective wisdom that you've gathered over time. And I think that's where it gets really, really challenging. I think it can feel a lot like, you know, literally like playing whack-a-mole. Like you improve one user's thing, another thing breaks, you improve another thing, etc. And I think that's where teams start to see the benefit of evals. You know, early on, we had this heuristic that if someone hadn't shipped their product at least 3 months ago, it wasn't worth talking to them about Braintrust. And we would often meet teams and they were like, "Hey, we don't need evals. Our product is great. My mom uses it. She really likes it. The models are great. They're getting better. I don't need this." And then 3 months later, they'd come back to us a little bit hungover. "Hey, are you guys still there?" And I think that timeline has accelerated. People now understand that evals are quite useful, but the general thing is still there, the phenomenon.

How do you help teams kind of move faster without losing trust? Like, what's kind of the key thing? Like, if you were going to like, you know, pick the 1 or 2 things that a team should do to say, you know, I want to move faster, like, you know, increase my velocity without like losing trust in sort of like you said, what you built yesterday is not going to break tomorrow kind of thing.

Yeah, I think there's really 2 things that you should get good at. The first is you should have a really good environment for doing iteration. We have something called the Playground in our product that it's really simple. You can test multiple models and prompts or agents, more complex things if you want, side by side with a bunch of data points. You don't need to start by computing fancy numbers and scores. Just running your prompt on a bunch of examples and looking at the output with your eyeballs is a pretty good way to start. If you're sitting in your terminal and playing around with something, you're only working on one example at a time. So I think just getting some basic tool in place that helps you do that. And then of course, the 10 examples that you start with will suddenly turn into like 10,000. And then it's really useful to actually have some numbers to help you sift through everything. But having some kind of like iteration mechanism, I think that's really important. And then the other thing is having a really strong connection between what happens in production and what you're able to test. And I think this is why we've had this really sort of end-to-end experience from logging and tracing all the way back to evals in Braintrust from basically the beginning. It's funny, like one of our first customers was Airtable, and we didn't have logging when we started. And there was an engineer at Airtable who basically created an eval, and then you see where this is going, he just kept adding stuff to it. And soon it became the biggest eval we had ever seen. We're like, hey, what's going on? He's like, yeah, these are my logs from production. And I'm just using, I'm capturing them in Braintrust so that they're in the same format as my evals. And that was a huge, you know, it was like, okay, yeah, that actually makes a lot of sense.

Baby logging.

Yeah. And so we, you know, we turned what he was doing essentially into logging. But having that feedback loop I think is just incredibly important.

So you've been public about comparing models like OpenAI's and Anthropic's. What does good model selection actually look like today?

I think there's two analogies for what models might be. One is they might be like CPUs. And by that, I mean, you can take the same set of instructions and pipe them through a CPU and you ought to get exactly the same result. But different CPUs have different performance profiles. If you run an Intel CPU from 3 years ago, it might weirdly be better at certain things, but likely not other things, et cetera. And then I think another analogy is databases. Again, I think a lot about databases, but every database speaks SQL, but they speak slightly different dialects of SQL. And some databases are dramatically better at certain use cases than others. And I actually think LLMs are a little bit more like databases than they are like CPUs. The same English string sent to one LLM might produce really different things than another. Kind of like databases, I think you need to do a bit of benchmarking and diligence when you decide which model or family of models is your best bet to solve a particular use case for some period of time. But then you need to focus almost all of your effort into making that model work really well. I think the best teams that we work with, basically, if you were building a SaaS product, let's say like 10 years ago, You might reevaluate your relational database every 3 years, or if you're very advanced, every year or something like that. If you think about AI, everything is compressed. So the best teams that we work with reevaluate their model choices maybe once every 1 or 2 months, and they rerun a bunch of benchmarks and sort of understand, okay, well, it seems like now in GPT-5, GPT-5 Nano is weirdly fast. And so I can suddenly unlock this user experience I didn't think I could have done before. Let me go like play with that for another month or two. We do see teams that never reevaluate their model choices. And, you know, that I think they often lose product market fit. And then we also see teams that spend all their time comparing models side by side, and they never ship anything. So we kind of think this in-between point is what works best.

How about like actually eval fatigue? Like, you know, once you get to scale, you're going to get, you know, obviously lots of evals and lots of data coming back, like, What's sort of the best practice to sort of understand what to look at, how deep to go, and to make sure that you're sort of balancing this sort of like tuning versus like shipping and moving forward?

Yeah, I think, um, the short answer to your question is rm -rf. Don't be afraid to delete things. Um, the long answer to your question is I think my framework for evals is not to think of them as benchmarks, um, it's to think of them as time savers. So as humans, um, I think When we do evals, our job is to influence what an LLM is doing to better match the expectations that we have for what a user wants. That's it. That's our role in this weird world that we live in. Let's say that we have 15 minutes a day to spend working on that. Let's say you have one user and that user uses your product 5 times a day, so you generate 5 data points. Then your eval might be you spending 15 minutes looking at what that user did 5 times, remembering what they did yesterday, and then maybe tweaking something so that you have a better shot at improving the 5 times they use the product tomorrow. Now let's say you have, you know, 500 users and they each use the product 100 times a day. You know, now you have 50,000 data points and you just can't look at all 50,000 interactions. And I think the right way to think about evals is, okay, I have these 50,000 of examples, what are the 5 to 10 that are actually worth my time looking at in more detail? And I think if you think of evals as kind of like a prioritization function, then it— the answer to your question becomes kind of obvious. Like, if you have too many evals, um, that means that you're just generating too many things which seem like a priority. And so maybe you more aggressively prioritize within that, or maybe you remove things. I think the mistake is when people start to collect like 50,000 evals and then they generate a bar chart instead of actually looking at the data. And they say, okay, now I no longer have time to look at the example, so I'm just going to look at the bar chart. And I think like with anything, with any example of anyone using data, that is not the right thing to do.

Right. You posted this great line. I want to make sure I get it right. All agents converge on while loops with tools. What actually makes that design work?

I think it is inherently what the LLMs do. If you think about stripping away all software abstractions and really just imagining the most simple architecture that if it worked would be extremely simple, and if it doesn't work, it's only because the LLM is not yet smart enough. That would be a while loop with tools. And I think because LLMs are improving so quickly and because the model, the people who are improving the LLMs are sort of adopting and building with this mindset, you can look at how Claude Code is built. You can look at the OpenAI Agents SDK. You can look at how ChatGPT works when you actually interact with it. It's all the same, you know, while loop with tools idea. That means that this architecture is the only one that really withstands the iterations that models make. Now, of course, the reality is a bit more complicated. Sometimes you need to think about state. Sometimes you need to have one LLM delegate a task to another simpler LLM. But I think that arriving on these more complex designs with a purpose, like, "Hey, I used a while loop with tools and I got stuck because I need to delegate this task to another LLM for cost reasons." I think if you arrive on that from first principles by actually experiencing the problem, then it justifies investing in a more complex design. And then I think if you arrive on it with that, like from first principles like that, if the model becomes better or cheaper, then you'll also have no ego or no attachment to that. So if suddenly the assumption, which is that the model is too expensive to do a while loop with tools, goes away, then you'll just revert back to the simpler design. So that's just what we see, I think, some of the best teams do.

It was interesting. We were having lunch a couple of weeks ago and we were talking about how we were solving one of our problems and kind of went back to a more deterministic solution. And you were like, I think you guys are going to flip on that a bunch. Yeah. And I think this reminds me of that. And like, how have you sort of seen that? And like, you know, when I'm sure there's stuff in your guys's world where you're looking at that and you sort of say, oh, this time, you know, the LLM is not smart enough. I need to kind of like coach it or make a more rules-based kind of code. And then you end up saying, oh, now they got smarter. How do you advise teams to just constantly test? Is it like you have an eval framework that sits on top of that, or what would you sort of approach that problem in a more, like I said, first principles way?

Yeah, I think there's really two things you can do. So the most important thing you can do is you should capture the thing that's not working as an eval. And the reason that's really important is anytime a new model comes out or anytime you have some kind of idea about how to improve things, You want to be able to reevaluate the assumption as efficiently as possible. So let's say like, um, you're trying to build something in the healthcare space and you want to have something— you want to have a model, um, look at, uh, patients' notes. And you found that if the patient's notes are more than 3 pages long, the model gets totally stuck. I'm just making this up. Um, and, uh, and so you're like, okay, what do I do about this? Well, the first thing you should do is capture several examples of the model getting stuck so that let's say GPT-6 comes out tomorrow, you can click a button and then find out if it's still stuck. So make sure you do that. Now to actually work around it, I think there's really two things you can do. The first is you can hack and hacking works. And the second thing, which I've seen a lot of people start to do, and I think it's actually a pretty powerful method, is just to reshape the user experience around what the models can do today with an eye towards enabling more as the models get better. If you think about some of the most popular AI applications, they are simply allowing the user to do things that from the outside, if you look at something like Cursor, you're like, oh my God, they got so lucky because the models are really good at coding. Well, no one actually knew that until Cursor really became popular. And I think the Cursor team is pretty smart, right? They iterated like crazy and figured out like, oh, these things actually work really well. Let's actually figure out how to craft a user experience which highlights these things. I think that mentality is also really powerful.

What does good observability look like in AI apps and where are most teams blind right now?

Yeah, I think observability is a bit of an overloaded term. The reason is I think it's really important to think about the business outcome that you're trying to drive by investing in observability. It's also expensive. And so it's important to think about like, what am I actually trying to solve? So in the traditional software world, I think a really good product is Datadog. And the problem that you're trying to solve by investing in observability with Datadog is uptime, or uptime and performance. So we use Datadog and let's say someone is having, the Braintrust UI is slow, we rely on Datadog to go and figure that out. And I think, okay, uptime is really important to us, therefore we invest in observability and pay Datadog a ton of money, and it helps us achieve that goal. In AI, I think the purpose of observability is fundamentally different. It's quality. So an investment in observability is an investment in the quality of your application itself. And if you work backwards from that, you ask, okay, how does observability contribute to quality. I think the most important thing is that it lets you build a feedback loop between what users are experiencing and what your developers and subject matter experts, product managers are looking at regularly. So the most important thing, um, that I like to ask people is, if I use your app and I complain, how much effort is it for you to take that complaint and turn it into an eval? Um, and the best teams, it's like 1 or 2 clicks, if that. The teams that are NGMI, you know, they haven't even thought about the answer to that, right? And so I think that's really the goal. Like, can you get to a point where it's 2 clicks from someone complaining to something ending up in your evals?

We need to start a not gonna make it, like, name and shame segment.

How, I mean, to go into that one a little deeper, interesting. Like, so when you say like 1 or 2 clicks and that's like, Hey, they report a bug because they shake the app or they put, you know, they click something on the UI to report a bug and then behind the scenes they're sort of capturing that or they're like, are there teams that are actually taking user feedback and just saying every bug sort of captures the input or the sort of change and then just sort of makes a new eval for that?

Yeah, I think there's some people who go overboard. So you asked this really good question about like what happens when I have too many evals and how do I interpret that? I think that One of the lessons I learned pretty early in traditional ML is that while you can be very liberal with the data that you use for training, you have to be very careful about the data you use for validation. And I think that evals are very much an exercise in applying human judgment and taste to an LLM, you know, and what it generates. And so I think at some point, it is quite valuable to have a human curate the set of things that end up in your evals. Now, I think scale dictates everything. If you have one user and your user uses the product 5 times a day, then you should look at every log and then say, oh, this one is fine, but this one didn't look so good. Let me add it to my evals. As you start to scale, there are a lot of different things you can do. The thing that has worked really well for our customers is using online scoring. Online scoring lets you write code or use LLMs to look at your data and then figure out what is likely to be potentially bad and then save it for evals. And there's a lot of really interesting stuff you can do there. And then of course, the other signal, of course, is user feedback. So you can capture explicit user feedback like thumbs up or thumbs down, or implicit, like, are they shaking the phone or whatever? And that tends to be a good signal, but it's very, very noisy. So I think using LLMs to post-process stuff, applying curation with some human insight, all that is very important.

What's next for evals, agents, and AI infrastructure over the next few years?

One of the weird things over the past couple of years of building Braintrust is that we work with teams that are building the coolest AI products in the world. But our product has had like almost no AI in it. And, you know, I feel like we were building like a previous generation product in some ways. And the reason is that until recently, models have not been really good at looking at each other's work. I think like a good metaphor for this is a dog looking at itself in the mirror. So literally, like if you give an LLM some of the work of another LLM, it will get confused and sort of have a hard time understanding what is it versus, you know, the other LLM. Luckily for all of us, I think Claude 4, GPT-5, and not yet Gemini 2.5, but hopefully—

Listen up, Google.

Yes, please, Logan. Yeah, everyone, fam, please. The model is clearly smart enough, but it's just not good at calling tools. Anyway, the most recent vintage of models They have enough of an identity, if you will, that they're able to decompose this. I think with advancements around reasoning and so on, we've just seen in our evals that they're able to now actually meaningfully improve the work of another LLM. We've started unleashing this in our product. We just released something called Loop, which is an agent that's built into Braintrust. It's now accessible as an MCP as well. You can use it in whatever medium makes the most sense. But it is incredibly good at actually looking at the work of another LLM and improving things. And we've seen great performance for things like automatically improving prompts, looking at a dataset and figuring out, okay, what should I add to this dataset to make it better? Looking at logs and trying to figure out what should I pull from these logs that Kevin's not already testing and help him. A lot. Yeah, exactly. And so I think again, if you go back to what evals fundamentally are, it's like as a human, I'm helping to guide a model, do something that aligns with my or user's expectations. The implementation of that until recently was very manual. You look at a dashboard with your eyeballs and then you click around and you're like, "Okay, let me type 65 characters to see if I can tell the model that I'm going to lose my job if I whatever and see if it changes things." And I think that that part of it is really going to change over the next year. The implementation of evals and creating the feedback loop from logs to evals is going to be very, very LLM-driven. But I think we're still going to be guiding the taste and judgment that goes into that.

Where can people get involved and follow your work?

So I'm on X/Twitter. My username is @ankurgoyal. As an Indian person, it's kind of hard to get your name. So I discovered when I was 11 that if I remove the vowels.

That was available? Yes.

And, you know, it's actually worked well. It's universally worked for me across every platform. So none of the other, you know, 1 million people with my exact name have figured it out. Yeah, exactly.

And what's the, what's the Braintrust website?

Braintrust.dev.

Awesome. And then any offer, anything you want to share with Founder Mode listeners for Braintrust?

Yeah, if you listen to the podcast and you try out Braintrust and you want a sweet deal, Hit me up. You can reach out at ankur@braintrust.dev or hit us up on Discord.

Love it.

All right. That was super clear and just like such a refreshing take on what's actually working in AI. So thank you for being here, Ankur.

Thanks for having me. A lot of fun.

That was one of those interviews where the answers really can help us and me tomorrow. So I think, you know, it's not just theory, but it's on the roadmap.

Yeah, you're— I know you're going to call a couple of people after this. What stood out to me was how Ankur really broke down the way the models have evolved and how that has implicated his work at Braintrust. And now he solved this problem multiple times. It's been amazing. What were your top 5, Kevin?

Yeah, so I think databases are everything. I think this idea that evals need to become, you know, mandatory once you start to see regressions, the idea that you ship fast but then have this closed feedback loop. Mm-hmm. LLMs sort of need to be evaluated monthly, right? You know, in the old days you'd look at infrastructure every year or two, and now it's like LLMs are moving so quickly. You need to sort of like take a quicker look on a more frequent basis. And then finally, this idea that as long as you're not using Gemini, models can improve models and Gemini will get there. Google, you know, they're grinding hard. But I think that, you know, was the big ones I saw today.

Yeah. If this helped clarify your thinking on AI development, please share it with your team, especially the people that are thinking beyond the prototype.

And if you're enjoying Founder Mode, like, subscribe, drop us a comment, give us a review, and let us know your feedback. And that's a wrap. Because building AI is hard, but shipping it doesn't have to be.