1 to 100: Braintrust

If we're very deliberate about making sure that we get the right customers together, then the pattern is also really strong across customers. So what I think about is if you're not hearing the same thing from customers, it's not the customer's fault. It's not the customer asking you for something super specific. It's your fault for not identifying a consistent problem across a consistent customer. If you take construct a cohort of customers who share roughly the same problems, then you don't need to spend all this effort like doing Kanban boards and prioritizing stuff or whatever. You just listen to them and things tend to work out pretty well. I would say Notion is a great example of a company that is just way ahead of the curve. And when we work with teams that are, that are like that, we collaborate really closely with them. And what you see encoded in the Braintrust workflow and the UI and stuff are a lot of the nuggets of learning that some of the really advanced teams had early on while they were building AI product. We've kind of codified those into the product. And so what I hope is that a lot of the stuff that people would almost invent themselves and then have to go and build has been codified really nicely into Braintrust.

Hello and welcome to 1 to 100. A podcast that takes you inside the most promising hypergrowth startups in a conversation with the founders themselves at the helm. Our goal is to give you the inside story alongside empirical analysis of a notable tech company that you might want to join. I'm your host, Osman Hanif, the author of the Why You Should Join newsletter. This week's 1 to 100 is all about Braintrust.

For those tuning in, welcome to 1 to 100, where every week we explore hypergrowth startups worth joining with the founders of those companies. As always, my name is Asman. I'm one of the authors of Why You Should Join and the host of the podcast. Today's company, Braintrust, offers an enterprise-grade stack for building AI products. Their platform allows for evaluations, logging, and a playground for LLM prompts. They count some major companies like Instacart, Notion, Airtable, and Zapier as customers, and have raised money from some big-time investors including Greylock, Elad Gil, Adam D'Angelo, and Greg Brockman. Braintrust has around 10 full-time employees that are all based in person in San Francisco. I'm super excited to be chatting with Ankur Goyal, the company's co-founder and CEO. Ankur, thanks for doing this.

Thanks for having me.

Absolutely. Well, you've raised from some of the best investors out there. You count some of the best companies as customers. Clearly what you've built is needed. So let's start there. In your own words, what is it concretely that Braintrust offers and how are customers using Braintrust today?

Yeah, I think, I mean, you described it pretty well in, in your intro. I, I'd say like the, the only thing I'd add is when you start getting into building AI software, you start to see that I think like the old way of doing things just kind of falls apart. And, and, you know, I've been a software engineer for longer than I'm willing to admit, but life is pretty good if you're building like good old software. Like, you know, we're really good friends with the Vercel team. They're also a Braintrust customer. And I mean, they've like, they've made it so nice to build UI software at this point. You have like beautiful IDEs. You know, you can use Cursor nowadays, it's even better. There's wonderful systems for doing CI/CD, even like GitHub has it built in now. And then there's all this amazing observability and deployment stuff. But in AI, actually, like all of that stuff falls apart. It's a really big pain in the butt to put all of your prompts in like f-strings or whatever throughout your code. There's really no analog to CI/CD and observability. It also kind of totally falls apart because you don't really care about like CPU metrics. You don't even have CPU metrics. You're not, you're running LLMs through an API. What you care about is the inputs and outputs that users are trying and, you know, getting a sense of what's breaking. And so if you like imagine that landscape, there are a lot of really good teams that are trying to ship AI product and they're just struggling because all of the tools and methodologies that they're used to using, they just totally fall apart. And so what we did based on my own personal experience working on this, both at Figma and Impura, my startup. Previously is we started by building really, really good tools for helping teams do evals. And the reason we started with evals is one, I think it's the most broken of the entire workflow. And two, if you build really good evals, then actually everything becomes easier. And so we actually like really strongly believe that the right place or the right starting point for teams is to build really, really good evals. And once you kind of figure that out, The speed at which you can build product is much easier. And so very simply, our product is, it consists of an SDK. We, most of our users use TypeScript or Python, actually, like most of them use TypeScript and you can write your evals in a very easy way that almost feels like running a unit test. And then you run an eval, Braintrust, you know, eval, and then it runs the evals for you and it creates this really nice UI. And if you're a software engineer. Who really understand software and you care about quality, but maybe you don't have a PhD in statistics, you can look at that UI and make sense of which test cases broke, which ones improved, how it compares to what, what is on your main branch, all of that stuff. And that, that's like the core of what we do on the eval side. And now we've actually built like a whole platform that basically is like, hey, if you are willing to build good evals, then there's all this other amazing stuff you can do. So for example, you can actually reuse the instrumentation and even the scoring functions that you use in your evals in production. And then when you're running your workload in production, you actually format all of your data in exactly the perfect format to use in your evals. And then you can find cases either automatically or using human scores that you can send through our API or in our UI and filter down to cases that are broken and maybe not covered by your evals. And again, we kind of like encourage slash gently force you to instrument your code so that that data is already in the perfect format. And so you can just click a button and add it to a dataset and boom, it's part of your evals. We also have this really cool prompt playground, which lets you do this like two-dimensional thing. So you can take the prompts that you're using in your application, plug them into the playground, try out different models, and then plug them into exactly the same datasets that you're using to do your evals. and then you get this kind of like 2 by 2, or really it's like n by m matrix of your prompt running on all the different data points. And then you can kind of like visually inspect them, play with the prompts and tweak them, and you can even kick off full eval runs from there and do like deeper analysis. So that, that's basically the kind of stuff that we're all about. Yeah.

Yeah. And what would you then say is, dare I use the word, like the killer feature? Are most customers being onboarded for evals and then sort of being introduced to, you know, logs and this Prompt Playground and other features?

Yeah, I mean, that's definitely how we started. Probably like 80% of the people that signed up, signed up to do evals and then they started doing other stuff over time. And actually, I think it's only 10% of our customers are only doing evals now. A vast majority of our customers are doing both logging and evals and then more than half of them are actually using every module in the product, which I find pretty fascinating. Now, I think when we meet with teams, they always have some burning problem. And I think the burning problem is it's one of the three following things. One is we keep changing our prompts or we keep changing our code and we keep breaking stuff and we just don't understand what's going on. We need to, we need to figure this out. If that is your burning problem, you should sign up on braintrust.dev and then, you know, sign and run some evals. The other burning problem that we hear is like, okay, we're running this thing in prod and my CEO is playing around with it and complaining about some stuff. And I, it's just like, I don't know how to get that information. I don't know where to put it. And if that is the problem that you have, you should sign up and start logging. And actually we see an increasing number of customers doing that. And then the third problem that we hear that's a burning problem is I keep wanting to try new prompts. And every time I do that, I send a Slack message to my engineering team. They manually construct a Google Sheet. They send me the Google Sheet. I copy paste every individual example from the Google Sheet into OpenAI's Playground and Anthropic's Playground and Bedrock's Playground and, you know, Together's Playground. And then I copy the output and then I put it back into the Google Sheet. And then we sit down and look at it as a team, and this just takes us too long. And if that's your burning problem, then you tend to start with the playground.

Yeah, absolutely. And I guess starting with evals, you touched on it a little bit, but how would you say teams are sort of doing this right now without Braintrust? It sounds like maybe some just neglect it altogether. And if so, why? What's the hard part about evals that is forcing so many large teams of big companies to just get it wrong or not do it?

Yeah, I think, you know, a lot of, I think there's a good number of teams that just don't do it, but I would actually say there are a lot of teams that are doing some form of evals. And that often looks like a Jupyter notebook where they have, it's like almost like a runbook, like, hey, I'm going to change my prompt. Let me run it through this Jupyter notebook and then see what it breaks or what, you know, the evals produce. Some people actually even build their own internal tooling. Which kind of looks like Braintrust. Sometimes nowadays, sometimes we meet people and they say, we signed up for Braintrust and then we built our own version of it internally, which, you know, it's totally fine. I think it's, I'm, I'm, I feel proud, you know, when we're able to have that impact on people. But the reality is that these problems end up getting really, really complex, like the type of visualization, the volume of data that people end up processing, tracking all the Git metadata and associating it with the evals and making sure all that stuff works. Hooking it into your production logs, and then you have more people, you need to do role-based access control. Just, it starts to get really complex. And what we find, and actually, like, sometimes we meet people who say, I don't understand why Braintrust isn't just a for loop or whatever. And I actually, I usually encourage those people to go and build their own version. And then like 3 months later, they'll come back to us and say, okay, great. I built my own version. Now I understand why this thing is so complex. I'd love to use Braintrust. And, and actually, like, the, the last thing I'd say on that point is, uh, we have been really fortunate to work with some, some teams that are just way, way ahead of the curve. I would say Notion is a great example of a company that is just way ahead of the curve. And they are, uh, when we work with teams that are, that are like that, we collaborate really closely with them. And what you see encoded in the Braintrust workflow and the UI and stuff are a lot of the nuggets of learning that some of the really advanced teams had early on while they were building AI product. We've kind of codified those into the product. And so what I hope is that a lot of the stuff that people would almost invent themselves and then have to go and build has been sort of codified really nicely into Braintrust.

Yeah. And so if you had to pinpoint, you know, one specific— I mean, maybe it's not one specific thing, that makes this really hard? Would you say it is just the amount of data these companies have and figuring out some way to structure and run tests on it? Is it sort of the analytics and the statistics you run afterwards? What is the one thing that really is killing people?

Yeah, I mean, I think there's— I'll give you the boring answer, which is probably the correct answer, and then I'll give you the more sexy answer. I think the boring answer is that when you run an eval, you need a place to put the eval results. And it's, it's actually just really painful to create an app with a database and set up permissions on the database correctly and make sure your app can query the database. And if the schema of the table is different or you change the eval, your app, you know, the, whatever UI you build sort of adapts to that. I think at the core of it, that's the problem, which is like, where do I put the state associated with an eval? And, and then, you know, there's where do I put the state associated with the dataset? How do I version those things? How do I connect them? It's just, there's a lot. The sexier answer is that AI workloads tend to be very complex and it's usually not just a spreadsheet cell of like, here's the input and here's the output. There's often a very complex. Trace of stuff that happens. And so actually like collecting all the data associated with a trace and then beaming it up somewhere and then actually aggregating all the scores across the spans of a trace and visualizing the trace is a really hard technical problem. That, that's probably one of the harder technical problems that we've had to spend time solving. And that's, I actually haven't seen anyone who's built an internal tool that does that gets to the point of doing tracing. And on the flip side, I think tracing is where a lot of the value is when you're actually debugging an, an AI issue.

Yeah, yeah, absolutely. And, and sort of on your first point about sort of, you know, even just dealing with databases and, and, and sort of this, the data schemas, sort of in the same, same realm, I, I read somewhere that you guys help teams create sort of these golden datasets what that they can use to, you know, then benchmark their tools in the future. I'd love to kind of explore that a little bit more. What, what does that look like and, and where does that fit into the eval process?

Yeah, for sure. Uh, let me zoom out. So prior to LLMs, if you wanted to do anything related to AI and ML, and I think this is why a lot of people didn't, but if you wanted to do anything related to AI and ML, you needed to create a really good dataset. I used to, at Empyra, we used to help people do document-related stuff. And our big innovation was that instead of needing like 1,000 invoices to extract, um, you know, the invoice number from an invoice, you only need like 10 examples from a specific vendor, or maybe like eventually we got it to like 3 examples or something like that. But you still needed some. And with AI, that's not really a thing, right? You can just prompt GPT-4o or whatever, and it will, it doesn't matter what the invoice looks like to a large extent, but people recognize that you still need this kind of like data flywheel thing where you can create a feedback loop. And there's a few reasons for that. One is so that you can incorporate user feedback into improving your application. And the right way to start doing that is via eval. So let me collect examples of my thing working. Let me collect examples of my thing not working. And then when I run an eval, I'll test both of those and then try to understand, do I improve or regress performance? Another reason is that if you collect really good data, then you can do automatic optimization, which could be, um, in-context learning or fine-tuning or training your own models. Whatever it is, you create this asset that you can use to essentially further customize a model or a process around it. And then I think the third thing is that you can also now spin up new use cases faster. And so if you get really good at creating good datasets, then like, let's say you are working on something email related and you collect a really good dataset of difficult to parse emails and maybe like some of the replies that are generated, and you're working on a new problem, which is classifying whether an email should have an auto-generated reply or not. Now you have like a really good artifact that your engineer can go and run with. And that for a lot of AI use cases is like 90% of the, you know, time investment is figuring out the data. So I think everyone kind of appreciates that there's a lot of value in creating these so-called golden datasets. But in practice, it's really hard to create them. I think a lot of the teams that we meet before using Braintrust, they're managing data in JSON files in a Git repo, which has a number of benefits. It's, you know, version controlled and all this other stuff, but it's very, very difficult to collaborate. And there's a lot of friction to add, to go from like what's happening in production, which is where the interesting data originates to the JSON file. And when you create friction like that, you make it challenging for engineers and non-engineers to actually grab good data when they see it and then put it in the dataset. I would actually say like one of the things that kills people when they're building AI product is let's say you build a product, you're like, hey, give me your feedback. And I'm like, oh, I clicked this, this, and this. And then this thing broke. If you don't capture that interaction and add it to a dataset that you are using in your evals and fine-tuning all this other stuff, like if you're too lazy to do that, then you're screwed. And so one way of thinking about it is we, we make it inexcusable for you not to do that. And I think that's, that's where a lot of the value is. And then there's all this other stuff like that. It's hard to version datasets. Datasets in Braintrust are automatically versioned, you know, like all this other stuff, but really it's making it frictionless.

Yeah. Yeah, absolutely. And, and I promise this is the last nitty-gritty question on evals, but I, I'm sure we have an expert here, so I could do this all day, but I am curious when it comes to LLM applications. Obviously this is a whole field of study. How, how, how do you guys sort of go about like evals? Like, and a very concrete example is, you know, I've built some LLM tools before, right? And you know, the, the output is non-deterministic, right? So it can say the right answer and maybe word it differently. How do you go, go about even evaluating things like that? Which is, it said the right thing, but if I hardcoded the answer it was looking for, it said the wrong thing. I'm curious about that. Yeah.

Yeah. Yeah. So I think the first thing is to realize that evals are not about telling you whether your app is perfect or not. They're not like unit tests. And if you think of them that way, I think you're going to have a really hard life. Eval, I think, are really about prioritizing the precious time that you as a human being have working on your AI tool. So let's say in your case, you are working on your app and you have like 1,000 input examples that you're playing with and you use them in your evals. The, the value of doing evals is not running the eval and then getting a magical number and saying like, yes, I'm at 87%. That's better than I was yesterday at 86%. The value is actually running the thing and then immediately looking at the cases. Like, let's say you have 10 minutes and look at spending those 10 minutes looking at the most impactful input and output, like raw data points that you can. And so the right way to think about scoring is a way of prioritizing your time, not to tell you the perfect truth about the thing that you're scoring. So if you use that framework, I think a few things become really obvious. The first thing is that you should not overthink how you do your scoring. So if I were you and I were working on that problem, I'd probably start by writing a scoring function, which is like equals equals. Braintrust makes that really easy, but there's even an exact match score, 11 sheen score, whatever, but just try that and run the experiment and then look at the examples that are zero and then be like, oh no, there's all these examples that are zero, but really like, it's like, it's like roughly the same thing. And then you should ask yourself like, okay, is it semantically the same thing? Or are the strings close enough together that I don't care? And, you know, if it's like, if the strings are maybe a few characters off, maybe you use Levenshtein distance because it's a heuristic, is fine. And it doesn't need to be 0 or 1. It can be any float, you know, between 0 and 1. And then maybe if it's a semantic thing, you use a model-based evaluator. And there's a bunch of LLM as a judge evals that are built into our open source auto-evals library. You can grab one of those and just try it and see what happens. Um, most of our customers actually go through this journey and then they end up customizing the evaluators to really crisply capture the criteria that they're trying to evaluate. But it should always be an iterative process. I think it should always be driven by like, why am I wasting my time looking at these examples? Uh, oh, here's why, you know, here's my idea about how I can waste less time. Yeah.

And so Braintress really operates more on the data side of like you know, here are all the examples, let me structure it, help you visualize it, and then you can use that to then later figure out what is the best evaluation model in place to sort of score what you're doing.

Yeah, I mean, we, we give you— we have a, a really comprehensive library called AutoEvals, which has a bunch of like purpose-engineered scoring functions built into them. But it's, it's open source for a reason. Like, we want you to get in there and mess around with it and try tweaking, you know, the logic and the prompts. You know, like almost everyone who uses Braintrust reads the codebase of auto evals, and it's kind of like it's designed to be that type of accessible.

I guess shifting a bit, I'm curious about what inspired you to even start working on Braintrust, right? Obviously AI is booming, and I think many, especially with LLMs, sort of ran to using them and building applications with them. But you ran to evals, observability, logging. I find that really interesting. Like, why is it that you turned in that direction and not just let's use this and make an app out of it?

Yeah, I mean, I think part of it is I am, I always work in building tools that enable people to do stuff, especially technical people. Like I started my career working on relational databases. Um, and, and then at Empira, we helped like ops and tech teams process documents and I worked at Figma where we enable designers with tools. And so I would say like when there's a problem, I'm generally the kind of person who likes to help enable people to do stuff with tools. I really love it. And so, you know, to me, it almost feels like the obvious answer to that question is, wow, it'd be so much fun to work on the tools that enable this stuff. I think, you know, the reality is we ended up building eval tooling at my last startup, Impura. And then at Figma, we had exactly the same set of problems and we started building exactly the same set of tools. And last summer, just about a year ago, actually, I was talking to Ilad Gil, who works really closely with me on Braintrust and basically incubated the company with me last summer. And he was like, hey, you've built the same thing twice. Other companies are doing AI stuff now too. Maybe we should talk to Zapier and Notion and so on and see if they also have this problem. And we talked to a few of these companies and actually Zapier like reached out after we talked to them. They're like, hey, like we really desperately have this problem. Can you please build something? And, and that was, that was a pretty clear sign that there was a real need for this type of thing. And that's what really, you know, got Braintrust started.

Hey, 1 to 100 will be right back after a quick word from our sponsors. This episode is brought to you by WorkOS. If you're building a B2B SaaS app, at some point your customers will start asking for enterprise features like SAML authentication, SCIM provisioning, role-based access control, and audit trails. That's where WorkOS comes in with easy-to-use and flexible APIs that help you ship enterprise features on day one without slowing down your core product development. Today, some of the hottest startups in the world are already powered by WorkOS. These include Perplexity, Vercel, Jasper, and Webflow. WorkOS also provides a generous free tier of up to 1 million monthly active users for user management, making it absolutely the perfect authentication and authorization solution for growing companies. It comes standard with rich features like bot protection, MFA, roles and permissions, and more. If you're currently looking to build SSO for your first enterprise customer, you should consider using WorkOS. Integrate in minutes and start shipping enterprise plans today.

In terms of sort of, you know, the product, what are sort of moats that you see for this company? You know, obviously you've pointed out this is a hard problem and a lot of large teams struggle with it, but how do you sort of fend off maybe other startups are slowly trying to enter into this space?

Yeah, I mean, I am, to be honest, I'm not an ivory tower product person. I, I, I care about one thing, which is, are the customers who choose to spend their precious time with us successful? And that, that, that's it, right? Often when we meet people or we start engaging with them, helping them be successful doesn't even mean helping them use our product. Maybe it means helping them learn about evals and they're like, I'm not ready to, you know, use your product or whatever, but that's totally fine. And I think getting to Having the opportunity to help really good customers be successful and then working that into our product has led to a lot of really great companies, including but not limited to the ones that you mentioned, use, using our product. And, and that has led to a lot of really interesting problem discovery. There's a really exciting feature I can't talk about yet, but it's something that everyone is thinking about. It seems really easy, but no one has solved. Um, and I was talking to Eden, our head of product, this morning about it. And we were just like decomposing the problem and thinking about the UX and stuff. And it's like, it's super complicated, right? So we're going to figure out this problem and we're going to ship it pretty soon. I'm excited about it. But I think if you really stay focused on making customers successful, then these moats type things, they, they, they work themselves out. So it's, it's just not something I think too much about.

Yeah. Do you, would you say that you think that founders maybe sort of hyperfixate on this stuff a little bit too much? Would that sort of be your take on it?

Yeah. I mean, I think everyone's different and I know lots of founders that are mega successful, like way more successful than I am, that think that the things that I think are stupid are really important. And think the things that I think are important are really stupid. You know, and so I don't want to, you know, if you asked me this question like 10 years ago, I would give you a different response. But at this point, I think I don't want to pass judgment on how others do things. What I think is useful is for you to have some sort of perspective about what's the best way to do things and then do it. And usually what that means is it's the best way to do things for you. And for me, honestly, I wake up every morning and I look forward to every day, like 7 days a week, getting to work with literally the best customers in the world with an amazing team. That's, it's like all I could ever possibly dream for. And I think it energizes me like nothing else. I know people that are motivated and energized by different things and like, good for them.

Yeah, absolutely. I mean, I would say that you yourself have been quite successful. I mean, you've mentioned a couple times here, but Empira was your last company and it was acquired by Figma. And at Figma, I believe you were the head of ML platform. What were sort of the lessons you learned there that you've still carried over to Braintrust? What are some key learnings? I'm sure a lot of people would be blessed to hear some of the knowledge you've gained.

You mean at Figma or Empira?

I guess starting with Empira, what are some like takeaways? I guess that was the startup. Experience, whereas Figma is a little bit more big company like, yeah.

Yeah. I mean, I would say like both were really great experiences and I can't say enough good things about Figma as a company. There's, there's almost like an infinite number of things to learn from just how well they've built the company. But, you know, I think from Impura, there's a few things I learned that, that I would maybe highlight. So one of them is I was on this really weird career trajectory. Where I started, I dropped out of college and started working at a really tiny startup. I was employee number 2 at MemSQL, and then it grew, and I got to be the VP of engineering, and I did that for a few years, and I left. And when I joined MemSQL, I was the most junior engineer, and so I literally spent a year fixing bugs. It was like wax on, wax off, and that was one of the best experiences of my life. Just an unbelievably good education into how relational databases work and how to build software that doesn't crash all the time. And, you know, I think compounding from there, I sort of subscribed to this narrative that VCs will tell you and blog posts will tell you, if you're growing your career, you should like stop coding. And I didn't stop coding just because I think I'm like I'm literally addicted to writing code, so it's very hard for me not to write code, but I sort of subscribe to this idea that, hey, I'm now a manager, I'm a leader, then I'm a CEO. I shouldn't be in the weeds. I shouldn't be writing code. I should be thinking about strategy and delegating stuff. And I think at Empura, the first like 2 years of the company, I actually didn't spend as much time as I needed to on the product. We also got some strong advice, which I understand. I don't want to blame anyone for giving the advice, but really strong advice about hiring certain people, like hiring a VP of product early on and stuff. And I think instead of subscribing to what other people were saying and listening to people's advice and buying into a narrative about who I'm supposed to be, I really should have embraced who I am. And I'm like a very simple person. I like talking to customers and shipping things. And that's it, basically. And, you know, I think I'm pretty good at it too. And so that was one thing, like, don't necessarily listen to the advice that people give. And by the way, I still write like a ton of code. I think I spend about half my day writing code right now. And that's weird. I mean, I don't actually know other peers, whether they're engineering leaders or CEOs or whatever, who do that. But I've kind of accepted that that's just who I am and, you know, it is what it is. And then the other thing is being really mindful about stage-appropriate leadership. So when you have the opportunity to work at a small company and then the company grows, or you work at, as an employee at a company and then you grow, and then you change situations, like you go to a smaller company or you go to a larger company, that what's required from you as a leader, it's not just a function of who you are and how you may have evolved from being, you know, an engineer fixing bugs to a VP of engineering or whatever. It's also a function of the company. And so even if I, in my case, if I was a VP of engineering at a big, you know, whatever big engineering team, and then go to a small startup, it's not just what I am able to do or what I should be doing. It's now I'm at a really tiny startup and what's stage appropriate for the startup is for someone to be way more hands-on. And so I think that, that's like a really, really big just set of lessons. One, I mean, is like just embrace who you are and it's almost difficult to succeed in any way unless you do that. So you might as well do it and sort of take life as it is from there. And then the other thing is to be very mindful of what an organization needs from you given its stage, rather than what you perceive about your own skill set to be. So that's probably like the biggest learning I would say.

I know you mentioned a lot about how to the blood you're an engineer, But you've also definitely done a really good job with sort of distribution and sales, have some really big logos up there. Now that's often sort of the hardest thing to crack, right? You can build and code all you want, but how do you get it to people and have them paying for it? How, how do you go about getting some of these big customers to try you so early on while you're still iterating on your product?

Yeah, I think the, there's a really annoying answer to this question, or maybe it's unsatisfying answer to the question, and It is that I have been working in the Valley for 14 years now and have spent a lot of time cultivating relationships with really good people. And so when we started working on Braintrust, first of all, Elad was involved and like super involved in trying to help us figure out how to validate the idea and talk to the early customers and His network is unbelievable. And we literally sat down and wrote down a list of the 50 customers that we would want and either he knew them or I knew them, or even better, we both knew them. Right. And that's something that there's no, you know, there's no overnight solution to that. There are companies that don't have that network that succeed and they succeed in other ways. And there's nothing wrong with that or, you know, whatever, but you know, as a function of just time and experience and I don't know, like, you know, just working on it for a long time. We had that advantage early on. And so we took advantage of it. And so, so yeah, we talked to people about the idea. Um, it resonated. It wasn't easy to get people to use the product. They're one of our really prominent customers now was one of the people that we talked to. And they were like, hey, you know, we built all this stuff already. You're great, but, you know, love you, but, you know, we don't really need, we don't need anything from you. And it took 6 months of, you know, sort of like reminding them or checking in with them and every single time getting a response along the lines of, eh, you know, it's like cool, but we already built this ourselves. And then one day, you know, they were like, hey, actually we don't want to keep building this ourselves and we think you can, you you can help us take it to the next level. So even, even with that, it's not easy and it takes persistence, but that's what worked for us.

Yeah. So, so would you say sort of the sales cycle or the distribution for you guys has been a little less, I've built this product and now I'm going out and selling it, has been a bit more of a collaborative process, which is maybe from the very initial steps, some of these companies have been here with you?

Yeah, I think we have been we basically, we did two things early on. One is we identified the problem that we're solving. And early on, the problem is just very simple, which is, I want to change my code and I don't know what it's going to break. Do you have that problem? And it, yes, you know, like people have that problem. And so we were very clear about that. And by the way, that implies nothing. There's no, the word eval, you know, LLM, none, ops, none of that is in the problem statement. It's just like literally a problem that people have. And so we got really clear on the problem and we aligned with people that we really respect on the value and importance of the problem. And then the other thing is we were very narrow about the target customers. We weren't on Twitter saying like, oh, hey, you know, blah, blah, blah, blah, blah, come, we're building this thing. I'm not going to tell you much about it. Sign up on this waitlist with 3 words or whatever. We wrote down a list of 50 companies that are all pretty advanced. They're all in product-oriented companies that have built incredible products, and they are trying to figure out how to incorporate AI into their core products and services. Like, very specific. That's not every company on the planet. And so the combination of these two things resulted in a coalition of early believers in the company who cared about the problem, they had some faith that Elad, me, and like a combination of, you know, hooligans that we could assemble would be able to figure out a solution to the problem. And then we did.

Yeah. Yeah. I mean, it sounds like the mission statement or the problem was, you know, pretty high pie in the sky, you know, it was a pretty general statement. I'm curious, how do you sort of build and start narrowing down concretely into what you're building, especially if you're working with so many bigger customers early on? On, I imagine that a lot of these customers have very specific burning problems. How do you sort of balance not being led into one customer too much, which is, this is what I want, can you start customizing and building for me, and sort of maintain your own path as well at the same time, if that makes sense?

Yeah, this is, this is more art than science. And I think like, if you want to read, I think there's, there's a variety of different people who have different stances about this. The way that we did it is, one, I had personally built eval tooling twice already. And so I had some really strong intuitions about what the right workflow is and how to sort of get that from 0 to 1. And it turns out that the 0 to 1 was more structured and opinionated than what people had sort of cobbled up themselves, just as a function of time. Like I've been working on evals for, I don't know, now 8 years, whereas most people were just getting into AI in the last year. So I'd like to just, you know, I just had maybe a clearer sense of what the user experience should be and what the workflow should be. That got us from 0 to 1. And then I think I'm a strong believer in being very deliberate about the inputs to the product development roadmap rather than the outputs. And by that, I mean, We're very deliberate about which customers we talk to and who we ask for feedback about things. And then if you are a customer who is kind of giving really good feedback, then we just listen to you blindly. And if we're very deliberate about making sure that we get the right customers together, then the pattern is also really strong across customers. So I, what I think about is if you're not hearing the same thing from customers It's not the customer's fault. It's not the customer asking you for something super specific. It's your fault for not identifying a consistent problem across a consistent customer. And so you should think about it at that level. And then if you construct a cohort of customers who share roughly the same problems, then you don't need to spend all this effort, like doing Kanban boards and prioritizing stuff or whatever. You just listen to them and things tend to work out pretty well. There's a non-trivial number of things that we've built that have been incorrect, but I just don't care because I'd rather we build a lot of stuff and build it really well and sometimes not quite build the right thing than really tightly manage the process and the relationship between engineers and customers and all this stuff.

Yeah. So you would say if you find yourself in a position resisting the temptation of letting any one early customer take you down a different path, you've— the issue here is that you're talking to the wrong customers or you haven't narrowed down your base correctly.

Yeah, exactly.

Yeah, I see. That makes complete sense. Yeah. Yeah. I guess moving towards a little bit on the team, you have around 10 employees currently. The team is still pretty small but serving a large number of customers. How do you guys go about hiring? I'm assuming you prefer keeping it to a small team as long as you can. And how did you hire the people you have hired?

Yeah, I think we— the way that I think about it is we should be more people. We can certainly afford to be more people. And we hire people that are the right fit literally as fast as we can. The problem is it is very difficult to find those people and then it's difficult to excuse me, recruit them, but we try to hire as fast as we can. So we don't, we don't try to keep the team small for any arbitrary reason. We just keep the bar extremely high. And I think the other thing that we do is we have, I think, a very strong and simple culture. We have no meetings except for one weekly demo session where the engineers give a demo of what they worked on over the last week. And the go-to-market folks, their demo is talking about customers or insights from customers or pipeline, depending on exactly what they're working on. And that's it. You know, everything else is really self-motivated. I don't micromanage customer requests. There's no product management stuff. You know, it's, you know, if you, if a customer requests a feature, there are engineers in the Slack channel paying attention. And, you know, we really don't try to get in the way. Now there's a lot of people for whom that is not a great working setup. So I really apologize if you, if you know, if you hear that and you're like, wow, that sounds terrible. That's totally fine. For some people though, that is a really great working setup. In fact, we do these work trials and sometimes people do the work trials and they ship something and then like maybe Simon from Notion or whatever, like uses it and then sends them a really nice emoji and it's like, whoa, you know, like I built something, this really cool company, the CTO used it, it was awesome. And you know, that for some people, that's just, it's an incredible feeling. And that is, I would say almost what we've, we've designed and built a company around encouraging people who like that to be successful. So, and then you break it down and I think our interview process and our recruiting process, it's all about exposing people to that. Working style. And for the people that resonate with that working style, it almost feels like there's no option that could possibly be better than Braintrust, which is great. And then for a lot of people, it's just not a fit. So that's kind of how we evaluate people. We have, we have an interview, we have a work trial. You can kind of pick what you want to do depending on your circumstances. A lot of people, you know, get scared off by our interview, especially, you know, we have front-end people write C++ and we have back-end people write JavaScript. Again, for certain people are like, whoa, that's really cool. Other people are like, hey, I'm a frontend person. How dare you ask me to write C++ in an interview? Great. You don't have to work here. So that's one layer. And then the next layer is like, how do we find these people? I can't say enough good things about the Greylock recruiting team. They've recruited half the company. It's just, it's unbelievable how good they are, both in terms of you know, like sourcing people, quality, helping us close candidates. It's just, they're, if you're a founder and you like, you literally want help recruiting your team, there's, there's just, there's literally nothing better than, than their, their recruiting team. And then the other half, you know, is people I've worked with before or, you know, my brother works at the company. So you sort of, you pull things together and, you know, it works out.

Yeah. Yeah, absolutely. And, and for the, For folks listening in and they hear, you know, one meeting a week and question, how do you keep everyone on the same page? And also, how do you sort of plan out what the future of the product looks like? What do you respond to that? How do you keep people organized and also working on more than just resolving bugs but building out new features?

Yeah, I don't necessarily want everyone to be on the same page. I think it's important for people to have access to raw data, which in our case is mostly customer feedback, but it also includes metrics and, you know, your own intuitions about what should be in the product. I'll give you a small example. We have these cookbook recipes on our website, which are different use cases that you can try out with Braintrust. And we have everyone who joins, including now non-technical people, they have to build one of these cookbooks early on in their time here. And it's an incredibly eye-opening experience. They always find bugs. They usually fix the bugs or come up with feature ideas, but it's just a super useful way to put yourself in the shoes of the user and understand the product. And so, you know, I don't actually, I think it's kind of a good thing when people aren't on the same page. If two people aren't on the same page and they're sitting in the same room, you will inevitably end up debating something. And there, as long as everyone respects each other, you'll end up with, you know, a good outcome from the debate. So now on the flip side, you have to be really self-motivated. You have to be really independent and you have to kind of get what we do. If you, if you are the kind of person who benefits from lots of review and structure, which a lot of people do, you're just, it's just not a good environment. Like, you know, it's, it's, it's not, it's just not the place. It's not the place where you'll, you'll enjoy yourself. On the other hand, if you like product thinking and you're also a systems engineer or something, it's— at Braintrust, you are unleashed, right? You can— what I describe, people often ask me, am I going to be able to talk to customers? And I'm like, if you don't want to talk to customers, you're going to have to run away. It's impossible not to do that. And then larger stuff, I think We, we have a fuzzy but articulate version of what we think the future should be for AI development. And we talk about it all the time at lunch, you know, via Notion docs and Figma ideas and, you know, building prototypes, whatever. We're constantly noodling on that and talking about it and bouncing ideas around. And every once in a while someone has an insight like the one I was describing this morning. Maybe a customer points out like, I'm okay, I'm finally ready to do this thing. We seed them with some ideas about what we might want to do in the future, but you kind of see the stars align about like how you could actually ship a piece of this fuzzy future. And when we see the stars align, then the troops rally together and we kind of, we figure out a way to ship something. And so we keep kind of chewing at things in these pieces and it is not a trivial balancing act to balance fixing bugs, building features and exploring crazy ideas. But I'm not sure there's any obvious answer to how you do that. I think we have our own sort of fun flavor of doing it. But that, that's, that's our approach. I think you, you should have a— you should have some picture of the future. And we like to, instead of planning long steps to get there, we like to just sort of assume we're going to get there and then chip away at like relatively big bites towards getting there.

And, and, and a couple last questions touching on what you just brought up. I mean, you mentioned Greylock, you've also mentioned Elad, um, you have some great investors. How'd you sort of get in front of these guys? Did you know them beforehand and how'd you choose the investors to work with? Obviously this is something that's top of mind for a lot of founders.

Yeah, yeah, for sure. Again, I have an unsatisfying answer, which is like, it takes a long time to build trust with people. I met Elad at a pizza shop 8 or 9 years ago and a mutual friend of ours was hanging out with Elad and he was like, oh, hey, nice to see you. Come hang out with us. Okay, great. And Elon was still like kind of a celebrity then, and now he's like a super celebrity. But you know, I was like pretty excited to meet him. So we had pizza and then he emailed me the next day and we went on a walk. He like walks really fast. And so I think in like 30 minutes we walked from the Ferry Building to the Mission and we just talked about a bunch of stuff. And then as I was starting Empira, he invested and he was a relatively small investor, but we just kept hanging out and chatting and chatting. And when we were getting acquired, he was like extremely helpful through that process despite the, you know, stake or size of his investment. Um, and so we've just been kind of, you know, building a professional relationship and a friendship over the, the last many years. And then Saam, who led our investment from Greylock, I met him when I pitched Empyra to Greylock and they turned us down. And sometimes you meet people and you just connect and Sam and I became friends and we just, you know, again, obviously I was sad that we got rejected that time, but who cares? Like we, he's such a smart person. He's so nice. He actually recently got married and we went to his wedding and he sent us the invite like before when I was still, I think, I think the invite was like still when I was at Empira or something like it was just totally unrelated to Braintrust and then I pitched Sam on Braintrust 2 months before his wedding and it ended up working out, but we were joking about how it would've been really awkward, you know, at his wedding if, if things didn't work out that time. But, but again, I think I met him in 2017 or something. So we, we met and then built a relationship.

Yeah, I guess there really is no shortcut to hard work and experience at some point. I, I guess the final question I had here is, You mentioned there how, you know, at your guys' lunch and, and through Notion docs and different mockups, you guys are always pondering on what the future of AI development looks like. I think the audience would love to hear from you. What do you think the future of AI development looks like? This is obviously a very loud and noisy space. What do you see it to shape out to be like?

Yeah, I think maybe without going into too much nitty gritty detail, I, I would say one of the bigger debates. Is what does it mean to create an AI, create a piece of software that uses AI in the future? Because right now it's kind of like you have these LLM API calls and you have code and who knows where the code is running and most of the code and all the libraries are implemented in Python, but everyone's building UIs and it's just all over the place, right? And I think there's two camps that are forming. One camp is kind of like prompts are like assembly code and it's like ridiculous that we're writing English by hand. And so let's come up with programming abstractions that allow us to automatically generate or manage or whatever it is prompts. And I think that's a very intriguing perspective. And I'll be totally honest, as a computer scientist who literally studied programming languages and worked on compilers for like 10 years. I would be just so excited about that. Like, man, that's so exciting. The other set of assumptions or the other way of looking at this is actually code is becoming irrelevant and the things that matter are prompts, models, and data and how you manipulate and maneuver those things and encode logic in English and work with, you know, gently and carefully curating data. To feed into a prompt within context learning or testing or whatever, that process of like sort of curating and encoding or coding prompts and data together, that is actually the future. And as much as it hurts me as a programming language person, I actually think the second outcome is, is the right one and, and the, or it's, it's what's going to happen. And the reason, and by the way, I might be totally wrong. I don't want to be dogmatic about it, but you kind of have to pick, right? And I think that the thing is, there are two reasons for it. One is, I think nothing matters more than a human being looking at an AI system and a piece of data and using the human judgment that you have to intuit why the AI system doesn't understand the piece of data. Is it the prompt? Is it the model? Is it the data? Why? That is the key workflow that enables AI progress. I mean, if you talk to researchers, that is the thing that they do, and then they take away from that how to create or manipulate data to better improve model. If you talk to an AI practitioner, they look at that and then they think about how to change their prompt or how to use a different RAG technique or whatever. But it is that moment of like looking at a piece of data in an AI system and wondering how to improve it. And then the other thing is that I've seen a lot of people, in fact, like the, my former head of product from Empira is now the AI PM at Superhuman. And I worked with her at Empira where we were building an AI system. And I, you know, spent some time with her at Superhuman where they're building an AI system. And she's now essentially a programmer, right? Like the, the amount of stuff she's able to do and the amount of impact she's able to have in a hands-on way by working on prompts. Writing English, being an exceptional communicator. It's just unbelievable. And so it feels very clear to me that the predominant method for programming with computers is going to be more prompting, not less prompting. And so that's a lot of what we try to enable.

Well, absolutely. The final note here is for those interested in working at Braintrust, are there any specific roles you're hiring for? And if people fit that description, what's the best way to reach out to you or the team?

Yeah, for sure. There are 3 roles I think that would be really relevant to this audience. We're looking for systems people. These are people that work on databases, operating systems, compilers, et cetera. The interview will test your understanding of these kinds of things. Be prepared to write C++ code and understand atomic operations and all that kind of stuff. If you're interested in this kind of thing, and I would say if you have a resume that illustrates your experience, then that's great. If you don't, just send us some code and we'll take a look at the code and give you a sense of whether you're a fit. And then product engineering, these are people who really, really deeply care about UI and product. We don't actually hire full-stack people. So if you're a backend Python person, we're probably not the right place for you. In fact, I would say if Python's your primary programming language, we're generally speaking, not a good fit. And, and, and so I think on the product engineering side, we're really looking for people that are obsessive about creating good user experience and probably really, really good at TypeScript and the ecosystem around it. And there again, I would say in the absence of a resume, the best thing is to, again, point us to some work that you've done, whether it's code or like an app that you built or something, we can take a look and then give you a sense really quickly. And then the last thing that we're looking for is a DevRel. And as a DevRel, actually, if you are a Python programmer or a JavaScript programmer who likes, is really passionate about education and working with developers, you know, we'd love to talk to you. And so there again, either resume or send us something you've worked on and we'd love to chat. Absolutely.

Well, Thank you so much, Ankur. For those listening, do check out Braintrust. What they're doing is absolutely amazing, and there's a whole list of companies and investors that can second that. Thank you for taking the time.

Yeah, thanks for having me. That was a lot of fun.

Thanks for tuning in to this week's episode of 1 to 100.

Please send it to any talented operators looking to join their next company.

Ratings and reviews are extremely helpful for a new podcast, and you can let us know which other companies you'd like us to profile. 1 to 100 is a podcast from the Why You Should Join newsletter and Turpentine, the podcast network behind Econ 102, Complex Systems with Patrick McKenzie, Turpentine DC, and more shows for experts by experts in tech.