How the Smartest Companies Use AI | Ankur Goyal, Braintrust

what the smartest companies are doing right now is they basically have two product roadmaps. One roadmap assumes that AI is going to keep getting better, but it's not going to be what people scientifically think of as AGI in the next 5 or 10 years. And then they have another roadmap, which assumes that all these crazy things happen. You want to structurally build a product so that if OpenAI or Google, Anthropic, et cetera, if they release a model, that is twice as good as the model today, your product gets better. Most of our customers are forecasting at least 50% of their engineering projects over the next year or two involving AI and probably 100% in the next 5 years. What we're seeing is that the paradigm around building software is totally different. It no longer revolves around code sitting in an IDE. It revolves around data that you get from your users and prompts that not just engineers, but product managers, users, other people are able to contribute to, to control and affect how the software actually works. I hate meetings. So at Braintrust, we have one company meeting per week. Okay. Wow. Per week?

I thought you might say day.

No, no, no. It's one per week. And the longest it's ever taken is 15 minutes. Folks at larger companies will chuckle and they should, but we give people an insane amount of autonomy and You know, I'm actually the type of person who believes it's better to give people a lot of autonomy and for them to mess up occasionally or be slow or unproductive for a week while they're wrapping their head around a problem than to micromanage people.

Welcome to The Peel, where we explore the world's greatest startup stories. I'm your host, Turner Novak, founder of Native Capital, venture capital firm that also hates meetings. Today I talked to Ankur Goyal, founder and CEO of Braintrust, the end-to-end developer platform to build the best AI products. Ankur sits at a very interesting spot in AI. His Braintrust customers include a who's who of companies that were the fastest to launch and continue to develop AI products like Zapier, Notion, Replit, Instacart, Airtable, Retool, and more. We hit on the importance of LLM evals, how the best companies are building AI products, why predicting AGI is impossible, how AI is changing software, advice for selling AI products to large enterprise customers, and why the best companies have two AI product roadmaps. Ankur also shares learnings from his first failed startups, when and why he doesn't follow best practices, why you should be very careful listening to VC's advice, how Braintrust hires new candidates, and finally, why he hates meetings. This conversation is very unique in the sense that Braintrust is used by all the companies at the forefront of launching AI products. And he shares a lot of unique perspectives that you probably haven't heard before. Thank you to Elad Goyal for introducing me to Ankur and to David Song on Elad's team, Albert Zhang at Braintrust, and Song at Greylock who led Braintrust's seed round for all suggesting great topics for this episode. Without further ado, let's jump in after a quick word from Atio.

There's a world where your CRM is powerful, easily configured, and deeply intuitive. Ateo makes it a reality. Ateo is a CRM built specifically for today's modern company. It's flexible, easily configures to your unique data structures, and works for any go-to-market motion, from self-serve to sales-led. It automatically enriches your contacts, syncs your email and calendar, lets you create powerful reports, and quickly build Zapier-style automations. The next era of companies deserves more than a one-size-fits-all CRM with outdated UX. Someone recently described Attio to me as the linear of CRMs, which if you're in the know, that is high praise. Join ElevenLabs, Replicate, Modal, and more. Try Attio instantly at attio.com. That's A-T-T-I-O dot com, or tap the link in the show notes and scale your company to the next level. Ankur, how's it going?

Great. Thanks for having me. Excited to have you.

So your company, Braintrust, you do a lot in AI. You kind of sit in the middle of all of it. Can you just kind of talk about what you're seeing that's going on right now just from your viewpoint?

Yeah, I think the most exciting thing is that every company, whether they're a software company or whether they have software in their company, which is basically every company, is trying to figure out how their products, how their internal processes, how their core business changes with AI. And it's kind of like a primordial soup right now. It's, it's so exciting at pretty much every company. There's a group of product managers, engineers, designers getting together, prototyping things, shipping things into production. And the way that people build software is like changing right in front of our eyes. So I'm super excited about it.

Are there any examples that you can think of, or you're allowed to talk about, just of how customers are kind of using Braintrust and what kind of things they're doing with AI?

For sure. I think some of the companies that I think are really exciting are companies like Zapier and Notion, both really heavy Braintrust users. And, you know, in the timeline of things, they actually shipped their products really, really early into the sort of AI explosion. So ChatGPT came out towards the end of 2022. Both companies had products in market, I think, by January of 2023. And so they're really ahead. And what we see is the stuff that they were doing 6 months ago, other companies are starting to do now. I think really what it comes down to is using evals as kind of this core primitive around which to build AI software. And basically, if you engineer a really, really good system for evals that allows you to change your software, try out new ideas, incorporate user feedback, you see something go wrong with an AI, in production, you can immediately take action on that and make sure it doesn't happen again. And so their velocity, you know, just keeps growing and growing and growing. And the speed at which they're able to do stuff is pretty mind-blowing.

Can you explain what eval is? Because I'm not 100% sure and I just want to make sure everyone's on the same page.

Yeah, absolutely. So, you know, the problem that evals solve is if you— I won't call out any large corporations, but you've probably heard of some large corporations having massive fiascos with their chatbots and, you know, other AI services.

So it would be like answering the question in a way that is probably not correct or polite or something like that.

Exactly. Or if you're, you know, in the context of an agent, maybe using the wrong tool or expecting a tool like a weather tool to do something like check a stock price. Or if you're working in the space of data analysis, you know, generating the wrong SQL query. So there's a variety of different things that can go wrong with AI, and that's because AI is inherently non-deterministic. When we were working on AI stuff at Empyra, my first startup, we learned about this problem because we started using language models and we would increase the aggregate performance of the model by 3 or 4%. But then we had some really big banks as customers and we'd ship them the new version of the model and say, "Hey, this is 4% better." They're like, "Great, but it's—" 3% worse on my use case and maybe it's 7% better on other people's use cases. And that's really what evals allow you to solve. They, it's kind of a workflow and methodology around gathering test data that represents what you expect to happen with an AI system and then testing your AI system against the test data. And, you know, there's in some ways similar to unit testing or continuous integration. With traditional software. The difference is that evals are never perfect. And so you can't just have like 100% green or 100% red result with an eval. You need to do more detailed analysis every time you run an eval to really understand, did I break stuff? If I broke stuff, what did I break and why did I break it so that you can go and try another idea?

And that's basically what Braintrust's kind of core initial product is? Yeah, exactly.

We, so to kind of put this in perspective, evals aren't, it's not a new idea. People that have been working on AI and ML, including myself for a long time, have been doing evals. What's challenging and interesting now is that the core persona who's really driving a lot of the AI innovation is not an ML scientist or a data scientist anymore, or it's not just those people. Now we see a lot of product engineers actually building really exciting stuff with AI. One interesting data point is that most of our users, a vast majority of our users actually use our TypeScript SDKs, not our Python SDKs. And so Python is the language of ML. It actually seems like TypeScript is the language of AI. And so, you know, with that in mind, evals are a very new idea for product engineers. And what Braintrust does is make it super, super easy for any software engineer, regardless of how much data science or you don't need an ML PhD or statistics PhD to do stuff. You can get started with evals in Braintrust and immediately set up this very nice developer loop so that you can build good software.

So what would you do pre-Braintrust? Could you even use this? Could you even do it as a developer or?

Yeah, I mean, what most people do without Braintrust is what's called a vibe check. And this is when you come up with an idea and you hand code it into your AI tool and then you play with it on your computer and you're like, huh, okay, this seems better. The vibes are good. And then you gather a group of friends to get in front of your computer. Everyone's sort of hunched over. And you try out a few different ideas and you sort of come to a group consensus about whether it's better or not. Okay. And honestly, if you're just starting and prototyping something, that's the only thing you can do. So that actually should be the step 0. It just shouldn't be step 1, 2, you know, 3, 4, dot, dot, dot, and—

So that's like the first step or step 0 in the sense of like, there might be something here.

Now we go a little bit deeper with it. Yeah, exactly. I think as soon as you passed the vibe check, you should immediately get your idea or app or feature out in front of some users. And if you've instrumented it with Braintrust, then as people are playing around with it, they actually generate data and you, and you log the data into the perfect format to be able to use and test with later. Okay.

And do you, you use a different format than kind of the industry standard?

Am I right? No, we, so what, what we do is we actually fully embrace the OpenAI protocol. Many of our users use OpenAI and many of our users don't. But what we've seen is that alternative models actually generally speak the same network protocol as the OpenAI models. And so what we do is, and we've actually contributed a lot of open source work towards this as well, but we make it really easy for you to basically treat any model as if it were an OpenAI model and then fully embrace that standard.

So if you're using something from, you know, insert whatever, non-OpenAI company.

Google, Anthropic, Llama, you know, Mistral, anything, you can sort of speak to it as if it's an OpenAI model.

Interesting. Okay. Did not realize that. That's, that's pretty cool. So then what do you think about then just like the future of AI and AI infrastructure? Just where are we going over the next 5 years, 10 years, if you can even predict that far out, but like, how are you kind of seeing all this evolve over time?

Yeah, for sure. Actually, I think what the smartest companies are doing right now is they basically have two product roadmaps. One roadmap assumes that AI is going to keep getting better, but it's not going to be sort of what people scientifically think of as AGI in the next 5 or 10 years. So imagine LLMs are faster, cheaper, the difference between GPT-4 and GPT-3, maybe that happens 2 or 3 times and they just get really good at a number of things, but they're sort of, you know, the way that you work with them, it's kind of the same as it is right now. And then they have another roadmap, which assumes that all these crazy things happen. And most companies, so most of the good companies we talk to are, they sort of have these two roadmaps and they think about how to balance the bet between the roadmap. And I've actually seen some companies shift their own personal bets about what's going to happen. I've been working in AI long enough to know that if you really intuitively think something is right or something is going to happen, you're probably wrong, or the probability that your intuition is probably not that good unless you're someone who works at OpenAI or Google, et cetera. And so I don't have a strong intuitive grounding on which of these two roadmaps is going to play out, but we certainly have two such roadmaps. I think I would encourage every startup, every large company to have two as well.

Okay. So then how would you recommend then you're, you're building an AI, like an AI app, or you're trying to build a product for a large enterprise customer, like as an AI startup, how would you approach that then? Like having two different roadmaps and actually building the product?

Yeah, I think the super generic thing I'd say, and I've actually heard some interviews with like Sam from OpenAI where I think he's given kind of similar feedback is, you You want to structurally build a product so that if OpenAI or Google, Anthropic, et cetera, if they release a model that is twice as good as the model today, your product gets better. You don't want to build a product that if they release a model that's twice as good, it gets worse. And so I'll give you an example. Let's say that you build a product that does some kind of medical recommendation thing and you know, you're able to solve like 30% of the medical questions that come in and for the remaining 70%, you can't, you're like, you know, sorry, the technology is not there yet. And you figure out some really nice UI or whatever to handle that. If the model suddenly gets better, then that 30% number, it goes up, maybe it goes to 40, 50%, and then your product gets better as well. So that's, that's a really good strategic way to position yourself. And I think in those two roadmap With a product like that, it's more about the speed and maybe the TAM that gets affected. So maybe in the roadmap where AI stays roughly the same over time, you start to build a deeper workflow, but you assume that the number is maybe capped at 40 or 50% automation. And in the timeline where we achieve hardcore AGI over the next few years, maybe that number goes to closer to 100% and you don't think about other broader ways to build value because you can unlock a lot of TAM with that.

What's your opinion on AGI timeline? Do you have any views at all?

I think probably my North Star perspective is that I have absolutely no idea and it's foolish to make strong assumptions about Braintrust or strong assumptions about my personal life that assume something is going to happen or is not going to happen. And that's just, you know, when I was building Empira, for example, I literally saw AI accelerate at an exponential rate, you know, in front of my eyes. And we would assume something that was really, really, really difficult was going to be really difficult for the next year. And 2 months later, it wasn't really difficult. And we had to backtrack and, you know, deal with that. So I think I've learned the hard way. It's just, it's not a, we are not, well-equipped to estimate what's going to happen with stuff like this. And so it's hard to use intuition. In terms of my personal perspective, you know, I honestly, I see there's like two kinds of people. There are people that spend all day working with LLMs and there are people that don't. The people that spend all day working with LLMs are significantly less concerned with safety issues or this thing, you know, going rogue or whatever. They sort of, you build an intuition and in some ways a friendship or something with AI and you start to get a sense of what it's like. And it really, to me, it feels, I don't want to reduce it to better Google or whatever, but the way that I remember when Google came out, the way that it unlocked me personally to find information that was challenging before, I feel like that It's the next step function or maybe two step functions from that. So I'm really happy with what we have. I think it'll take us 10 years to really make use of it. And I'm excited for what's to come, but very, very happy and excited even with what we have.

Yeah. Well, the Google analogy reminds me of back when it came out for a while. If you knew how to use Google, you were like, you had insane competitive advantages. Just as a person, just in the workplace, personal life, whatever. Kind of the same thing with AI. It's like if you just understand how to leverage it, you have this insane advantage against people who don't or aren't.

Yeah, I think I see this happening very, very significantly with developers. The very best developers I know are all in on Copilot, Cursor, et cetera. And some of the least technically strong But smart product-minded developers I know are all in as well. And then there's a bunch of kind of middle-of-the-pack developers who have resisted and, oh, I'm better, you know, blah, blah, blah. Those people are getting left in the dust because why? I mean, the super elite people are 5 times more elite or 10 times more elite than they were before. And you can hire a bunch of people who have insane product empathy and maybe they're not the best, you know, computer scientists or whatever, but they're able to be very, very productive now. And so yeah, it's happening, especially with developers.

Just over the next, then, I don't know, as AI kind of evolves, do you see that statement becoming more and more true? Like what do you think is going to happen with developer tooling just over the next decade?

Yeah, I think, you know, I'm a little bit biased here, but I think there are basically two kinds of developer workflows that will matter. And they're both giant areas of opportunity and innovation. One is if I'm building software as we know it today, like let's say ordering coffee from a coffee shop type of software, the workflow around creating that kind of software is going to change dramatically. It's already changing. It's significantly easier to create that software for a variety of reasons. I think Vercel is a huge example of, I mean, they're doing a lot of great AI stuff too, but if you, even if you just subtract the AI stuff, the very, very little amount of friction that's required to spin up an application, host it, have authentication, you know, modern JavaScript, all that stuff. It's just very low compared to what it used to be even 5 years ago. Um, and then you add AI on top of that. If you want to spin up, you know, an app or a website to order coffee at your local coffee shop, it's just very, very easy to do that. And I think we're going to see a lot of innovation there for simple software, for complex software, for getting from zero to one prototypes to software maintenance, code review, helping with testing, helping to understand tests. There's just a ton of opportunity for innovation there. And again, I think there's, if you use this sort of two roadmap framework, I think there's a variety of companies and products that are sort of making different flavors of bets along the spectrum of those two roadmaps. And it'll be interesting to see what plays out. And then the other kind of software that is really going to matter, again, this is where I'm biased and, and I'm really bullish on what we're doing at Braintrust, is the software engineering that goes into building really good AI-powered software. And today, AI-powered software is a relatively small but important initiative at most companies. Most of our customers are forecasting at least 50% of their engineering projects over the next year or two involving AI and probably 100% in the next 5 years.

So this means they use AI to build the products or the product has an AI component? Product has an AI component.

Okay. Yeah. And so I think the first category is about using AI to develop software. And the second category is building really good software that uses AI or really good software that has an AI component. And I think what we're seeing is that the paradigm around building software is totally different. It no longer revolves around code sitting in an IDE. It revolves around data that you get from your users and prompts that, you know, not just engineers, but product managers, users, you know, other people are able to contribute to, to control and affect how the software actually works.

So then Braintrust does more than just evals at this point, right? For sure. What's kind of just like the spectrum of what people are using it for?

Yeah, so we, we like to think of Braintrust as the end-to-end developer platform for people to build the world's best AI products. And I think we're very fortunate at this point to, to work with like literally the companies that are building the world's best AI products. So I feel like, you know, in collaboration with, with them, we've, you know, companies like Notion, Instacart, Airtable, Zapier, Coda, uh, Replit, I think I saw on the website. Replit. Yeah.

Uh, Airtable, if you didn't say that one yet. Yeah. It was just like logos of like the, the, your favorite companies basically. Like the ones you, you know really well.

They're all, they're all really great. And they, they, and many of these companies, the founders are like daily active users of Braintrust and just sending us heaps and heaps of feedback all the time. So I, I think we've been able to productize actually a lot of their workflows around building AI software. And at a high level, I think basically what we like to think about is in the old world of building software, which is pretty good actually, there's kind of 3 big components. There's the IDE, the CI/CD, and observability. And in Braintrust now, we actually do all 3 of those things. So we have a prompt playground which allows you to basically create and test prompts, test models side by side. You can run evals. It's a really powerful multiplayer collaborative version controlled IDE system that allows you to basically build the prompts that power your application. We see product managers sometimes spend more than 6 hours a day in the Prompt Playground working on stuff. And what are you doing there? Yeah, so I mean, for 6 hours, I mean, that's a lot of time. Prompt Playground is essentially the place where you're you're writing the core logic of an application. And so let's say you're working on a new product idea and you have a handful of examples of the product idea. Nowadays, what you can do is, you know, maybe take 3 examples. Uh, let's pick something like creating a, a shopping cart list automatically based on a user's query. You might have like 3 examples of like, hey, I want, you know, I want new clothes, or I want groceries for my trip this weekend. Maybe take 3 or 4 of those, write a few prompts, test out a few models, see does the quality-to-speed trade-off of GPT-3.5 versus GPT-4 versus Claude 3 Haiku, how do those compare today? Because it might be different than yesterday. As people put out updates.

Oh, yeah. One model updates, you want to compare with every other model again, right? For sure.

Yeah. People recalibrate this stuff. Almost on a daily basis. So try out 3 different prompts at once, tweak the prompts, try to get to a good state. You can save the prompt from the playground and then with one line of code, use it inside of your application, ship it to some users, maybe internally to start, and then auto-generate all the logs from their interactions. And then sit down as a team an hour later and say, "Hey, what did you think?" Oh, these 5 interactions sucked. Okay, great. Let's open them up in the playground and start tweaking and see if we can get better performance.

So you'd be trying different models, you'd be trying different variations of, you know, insert whatever approach to trying to do the grocery specific or the clothing specific example of the shopping cart.

For sure. Different tool calls, you know, all— there's so much stuff that you can try. So why is it a big deal?

You talked about how there's these three different components, the IDE, CIDC, and observability? Why is it a big deal to have them all in one place?

It's, it's really because everything revolves around, around data and, you know, observability in, in Braintrust product is kind of like the top of funnel for you to capture really, really good data. Um, you know, the, the classic example is you're building an app and you ship an internal version and maybe your CEO tries it out or something and says like, hey, this, this query sucked. And then they send you a screenshot. What are you supposed to do with that? Most of the time you look at it, you forget about it, or maybe you keep it in the back of your head and vibe check it the next time you change something. But what people who use Braintrust do is the CEO sends that and then within 1 second they find that interaction in their logs. They can see every single input/output, every model call, you can click a button and reproduce the model calls directly in the Braintrust UI, fiddle with the models, all of that stuff, and then save it into a dataset. Now that it's in a dataset, every time someone runs an eval or every time your PM is playing with prompts in the playground, that piece of data is under consideration for what they're doing. And so the fact that all of these things are able to share these datasets that you curate and build up in Braintrust is very, very powerful. That's why they have to be in one place.

Because it's just easier to check and check the logs, see what's actually going on versus you'd have to, there'd be another program for it or something or.

Exactly. Yeah. I mean, in the absence of Braintrust, what someone would do is take, go, maybe go to one logging system and then look at all the logs. And then literally in the UI, like one of our customers used a big data warehouse, a very popular data warehouse product to do this stuff before they used Braintrust. It's probably super expensive. I would assume. It's super expensive. And they would, handwrite an SQL query and then in the UI of the data warehouse product, which is just for testing, basically copy out the JSON and then write a script that transformed the JSON into a simpler format and then copy paste that into a file that they put in their Git repo. There's so much friction there that if someone complains that, you know, the chances that you're going to run all of those steps and run them the same time without errors every time, it's just, it's just not going to happen. It's like an extreme version of I used to use spreadsheets, now I use a really good product.

Using multiple models. Why is that important? Like if somebody just— they always just use one, why is it a big deal to use more than one or many?

Most of the companies that we work with are bought into one model. Okay. If I zoom in further and make a less extreme statement, I would say most of the projects that people work on revolve around one model. And I would maybe offer a very, very technical analogy that I think about a lot, which is, are models more like CPUs or are they more like relational databases? And by that I mean AWS has hundreds of different instance types. All of them have different CPUs. Some of them have ARM CPUs, some of them have Intel CPUs. There's all this variety, but you can take pretty much any piece of software and run it on pretty much any EC2 instance type and you get different performance or, you know, different results. Maybe 1 out of 1 billion times there's some kind of error or something. It's not common, but it's a very transferable thing to go from one CPU to another CPU. On the other hand, you have relational databases. Every database speaks SQL, but if you try to take an app that's built for one database and move it to another database, it requires an insane amount of engineering work to re-encode the quirks of one SQL implementation to another. If you can't tell, I've worked on databases for a long time, so a lot of stuff I think about is in terms of databases. And it seems to me like models, everyone thought they would be more like CPUs at first, like you can transfer stuff from one model to another, but they're actually turning out to be more like databases. They all ostensibly speak English, but the dialect of English or the accent of English that Claude speaks is different than OpenAI. And so what we see is in the beginning of a project, people tend to evaluate a bunch of different models and recalibrate what trade-offs today are best suited to the project that they're working on. But once they settle into a particular model or model architecture, they just try to make the best of it over time. And that tends to yield better fruit than reevaluating your model choices every 2 days.

So someone might be using multiple models at their company or for their job, but each individual project or software or whatever typically will just run on one model.

Yeah. And of course there are exceptions. Like we see some companies, for example, somewhat popular thing to do is to have one model that is sitting at the, you know, the front line of chat and it can be much simpler and do some basic, you know, English, polite English, whatever, back and forth. Maybe it's even fine-tuned to some particular domain, but then have more use case specific models that implement more complex workflows that you sort of drop off to. That's not super common actually, but more than zero people are doing that.

Okay. So there's at least one. So how do you pick the right model? There's a bunch, I think Google just released a bunch of new ones last week. Every time Facebook updates the Llama model, There's a bunch of, there's so many options. How do you pick what you should use?

A few things that I would think may contribute to this, because there's a lot online about blah, blah, blah, long context, whatever. So a few things. One, I think open source is very accessible in certain ways, but it's actually very, very hard to use in other ways. One of them is actually inference capacity. So people tend to really applaud OpenAI for the quality of their models. And they tend to forget that OpenAI actually also, and Azure, have by far the best inference service. Like they have the highest rate limits. So if you're starting a new app, you're unlikely to get blocked as you scale. They have really good availability, a very easy to use API. And so one, one thing I would think a lot about actually is inference capacity ease of use and ease to deploy. And there's a number of companies that actually are aiming to make that easier for open source models and so on. But I wouldn't assume that with an open source model, you actually have the inference capacity that you need to be able to build a good application. Another thing that's really important, which probably not the most fun thing to hear, is that there are commercial and structural reasons that you might choose one vendor over another. So common version of this is I trust Microsoft, so let me use Azure. Another example of this is I have an enormous enterprise deal with AWS. I might even have a most favored nation clause, which says if AWS offers something, I have to use it. Interesting.

I didn't know that was a thing. That's crazy.

It can be. Yeah. And so in that case, you should use Bedrock because, you know, it practically speaking, you can do all the things that you need to with the variety of models that Bedrock offers. I think Bedrock is really hard to use. However, we've actually solved that problem with Braintrust with our proxy, so you can actually communicate with Bedrock as easily as you would use an OpenAI model. But in that case, you should use Bedrock. And I think there's a similar set of reasons structurally that it might make sense for you to use Google's models. And then of course there are technical merits. You might have a use case for which it makes sense to use a million-plus tokens context window, you might have a use case where you need extremely low latency and what GPT-4o or Claude 3 Haiku offer is really, really appealing to you. And so I think that there's also technical considerations, although at this point I personally think you can achieve the technical things that you need to with more than one option. And so the structural stuff ends up being more important.

Interesting. So if I'm somebody listening to this where I feel like I can't keep up with AI over all, because it's things are changing so fast. What would you recommend or maybe what's your process for just staying on top of everything that's going on?

My recommendation might be a little bit different from my personal process, so I'll give both. My personal recommendation is to pick one major vendor and become a fanboy. And it's okay. Or a fangirl. It's okay. I know and think extremely highly of the people at OpenAI, Azure, AWS, Anthropic, Llama, they're actually all wonderful and they're competing. It's great for the industry and you can build really great stuff with any of these technologies. I meet a lot of people who waste time debating the merits of one or the other, and I meet other people who instead are like, okay, I'm using OpenAI or I'm using Llama, I'm going to make the best of it. And you can be fairly certain that if there's an innovation with one model, the other models will pick it up. So just stay focused, pick one path and try to actually build a good product rather than spend a lot of time trying to keep up with what other model providers are doing. In our case, I think we are very, very specific with trying to work with companies that are innovating and building extremely good relationships with the engineers and the executives who are driving that innovation. So I, my news diet is firsthand anecdotes from people at, you know, the folks that we've talked about a few times who are our customers just asking questions or sharing observations. And that's been really great.

What's like the most interesting thing that you're hearing or seeing from that? Maybe you've already hit on it, but just, or maybe something else, like anything interesting anecdotes that you're seeing from all those conversations?

Probably the number one thing That I would say is everyone uses function calling or tool calling, which is probably not the most popular feature of models. If I asked you, "Hey, tell me the difference between this model or that model, whatever," I'm not sure whether you'd say tool calling is a thing. But if you ask people like, "Hey, you're using OpenAI. What would you need to be really, really good for you to be able to use Google's models or to use Llama 3 or whatever?" The number one answer among people that have stuff that has product market fit in production is tool calling. And the reason is that not because they care about calling tools, it's that tool calling allows you as a programmer to say, I want the model to spit out data that matches this exact format. And the reason that's important is if the data fits that exact format, you can use it in your code. If it doesn't meet that exact format, you're basically, the only types of products you can build are chat products. And that's it. I can't stress enough how important tool calling is and how I think really, really significant majority of all use cases are going to be on OpenAI's models until or unless other folks actually take tool calling seriously.

So that's why when you think, I'm thinking examples like Zapier had this most recent launch was a Chrome extension that basically can do things while you use the internet, essentially. It's kind of like this agent almost that lives in your browser. That's obviously not a chat. So that's one of your customers. Yeah. That would be an example of something.

100%. And that's even a more obvious example. I would say if you're a company that's building document-related software, you might expect everything to just be coming out as plain text, but the reality is it's all actually coming out as data. And so even the companies that are building freeform question answering stuff that's interesting, they're all structuring the outputs in terms of tool calling. I mean, even with Braintrust, our eval stuff, it all uses tool calling. So we have a scoring function, for example, called Factualiti, which tells you whether the answer that a model spit out is factually correct with respect to a reference answer. And that thing uses tool calling to help the model organize its thoughts into a few different categories. And then just as importantly, we're able to surface that data back to our users who are developers so they can make sense of maybe why the answer was incorrect or why the model got lost.

So it's just critically important. And then if I'm non-technical, I've never heard this word tool calling before. Can you quick just explain what that is? Because I want to make sure everyone's on the same page.

Yeah, for sure. Tool calling is basically an option that you can provide to a model or a configuration parameter that you can provide to a model that says, I'm going to give you a prompt and I want you to spit out your answer, but I want it to match this exact JSON format that can be used elsewhere. Exactly. Okay. Yeah. Yeah. It's basically like the equivalent of an API spec, but for a model.

So tool calling is like APIs for LLMs. Exactly. Essentially. Exactly. Yeah. That's a good way to think about it. Yeah. Okay. One other question, just in terms of, I've kind of seen there's like a little bit of this debate of LLMs are going to replace software engineers. I don't know how true that is, but What would you say to somebody who maybe is worried about that? Like maybe they're a young person, they're in college, they're majoring in computer science, they want to be a software engineer and they're like, I'm not going to get a job. How should I think about that?

Yeah, I think the near-term advice I would give someone is times may be more difficult than they were before. Like you can't be a shitty software engineer and get a job as easily as you could in 2021. So if you want to be a software engineer, take it seriously. Like actually study computer science. Learn computer science, take the compiler class that everyone fails, take the operating systems class, you know, but like push yourself because it's not going to be— you can't do the things that LLMs can do now and expect to just ride the wave. And, you know, I think the very best software engineers are insanely more capable than the mediocre software engineers. And currently the mediocre software engineers are very much more capable than the sort of pure AI engineers. But I'm not so sure that that's going to be the case for too long. So that's the near-term advice. I think for the long-term advice, I think about this every day because I love computer science and I don't personally love pottery or other artisan crafts, but it may go that way, is you got to think about the two roadmap things, right? In the AGI roadmap, what is your personal career roadmap? Probably should build some other skills, whether it's talking to customers, designing things, recruiting, maybe take a plumbing class because that job is probably not going to be displaced for a long time.

You can easily make six figures doing that stuff.

Yeah, we're actually thinking about adding it as a company perk. So— What? Yeah. So many companies have a voluntary educational perk. We're thinking about having a plumbing class perk, seriously, like a vocational perk.

To learn how to be a plumber.

Yes. Or another vocation that— Of your choice.

Yeah. Interesting. Yeah. Is this like kind of like as a tongue-in-cheek perk or is it very serious? Like you should probably learn how to be an electrician.

It's one of those things that at this point that feels like a joke that we won't be laughing about in the future, but it is our way of pushing ourselves to think about the two roadmap sort of thing for the future. And you like love programming.

I just know I've like, all this stuff, like you feel like, like I would put you in like the top 1% or 0.1%, very, you know, top whatever percentage. How did you get into it? When did you first start programming?

Yeah, so I have a friend in high school. I wanted to be a doctor my entire life because my parents are doctors. They're the first people in their family who really did higher education. You know, it's a really big deal to them. So I was influenced by that.

Did you grow up in the US?

I did. Yeah. Yeah. Yeah.

Did they move from somewhere or did they also grow up here?

My parents grew up in India and like around New Delhi and then they moved to Scotland. I was actually born in Scotland and then we moved to the US when I was really young. Two things happened my senior year. One, I took biology and I just, I like really did not get along with my AP bio teacher. This is high school. This is high school. Yeah. Bio is, if taught incorrectly, is all about memorization. I'm not a memorization person. And so, you know, I mentioned that because I think one teacher can have a huge impact on someone's life in either direction. On the flip side, I took linear algebra classes. I grew up in Pittsburgh, at the University of Pittsburgh, because I ran out of math classes and I learned how to program to help with the linear algebra assignments. And then I had a friend who told me, He read a book which taught him how to do symbolic derivatives in chapter 3. And that blew my mind because I was doing a little bit of programming with this linear algebra stuff, but the thought of writing a program that could symbolically manipulate an equation, I was like, "Oh my God, that can't be possible." And I understand probably for 99.9% of listeners, they're like, "What is wrong with you?" But that motivated me like you could never believe. And so I, I read the book, which is called SICP. It's actually a famous book that MIT used to use to teach people how to program. And I read through chapter 3. I wrote that program about how to write symbolic derivatives, and I literally knew what I wanted to do for the rest of my life.

Was do algebra and derivatives or program?

Write programs that powerful. Okay.

And then did you go to college for engineering? I did.

I went to CMU and studied computer science. And you dropped out. I did.

Yeah. Why'd you drop out or what's the story there?

Again, influenced by my parents, I had this narrative for my career, which was either work at a big company or do a PhD. And I interned at Microsoft after my sophomore year of college and I hated it. And I realized that it's kind of like pick 2 of 3. The 3 things are impact, creativity, and sleep. And at Microsoft, every character of code that I typed was actually pretty significant. I was working on Bing on the distributed compute infrastructure back then. As an intern? Yes. Okay. And the characters of code that I typed were processing billions and billions of rows of data from Twitter and all this other stuff to power search ranking. It was pretty cool, but I worked for 4 hours a day. I was constrained from working more because interns weren't allowed to run batch compute jobs after 5:00 PM. I felt very constrained. So no problem, I'm constrained in this way, so let me do research. So I started doing research and started doing that through to the next summer. There I realized I picked creativity and sleep, but there's no impact. The research we were doing was really fun and cool, but I remember Facebook released a paper about— I was working on low-power database stuff. They released a paper about how they were doing low-power databases at scale, and I was like, "Okay, the stuff I'm doing is cool, but they're actually running this stuff. That's so much cooler." And so I had this kind of existential crisis that summer, and I moved out to San Francisco and interned at a startup. Which didn't end up working out. And I sort of found my way to the MemSQL founders from there. But there I realized like, wow, okay, this startup thing is cool. I can choose not to sleep and have a lot of impact and creativity. So that's—

so the three elements of your life, that's the one where you kind of cut it out a little bit? Yeah. How much sleep would you say you get a night? Are you comfortable sharing?

Sure. Yeah. I think my target is 6 hours and 20 minutes.

What's so important about the 20 minutes? I've just measured it over time.

And if I sleep for exactly 6 hours and 20 minutes, I'm the next day is optimal in terms of my awake— awakeness. If I sleep more than 7 hours, I get groggy. And if I sleep fewer than 6 hours for 2 to 3 days in a row, I get fatigued. Yeah, that makes sense.

I'm the same way. If I get less than 6 for 2 days in a row, and then generally I feel like it's either 7 or 8:30. You know how we sleep in like 90-minute REM cycles? If I wake up mid-REM cycle, I'm screwed for the rest of the day. Yeah. I'll drink 3 cups of coffee and still it'll be like 5 o'clock. 'clock at night. So yeah, it's important. So you've chosen like the optimal 6 hours, 20 minutes of sleep. That's actually still a decent amount. I feel like you got like 2.5 of the 3, I feel like.

Some people— Maybe. Yeah, yeah, yeah. Okay.

And then when you were at MemSQL, it was acquired by SingleStore, was the name.

Was it acquired or was it the same product? No, they renamed the company from MemSQL to SingleStore. Oh yeah. Okay. That's what it was.

And then, um, uh, your wife Alana said this is where you like learn to be a grown-up, basically. Like, like, because I guess you were still technically at college age. Can you just talk about that? She said this is something we got to hit on. I—

okay, so first I would say Alana definitely is the one who helped me be a grown-up. So I'm not so sure I even was— I— maybe it was enough of a grown-up to, to, to interest her, um, when we met. But, um, one thing I often think about, I remember it was June of 2012, maybe. Yeah, 2012, 2013. I was like, yeah, it must have been 2013. Yeah, I was 22 and we were about to close our first big deal and we just hired a salesperson. This was a single store? At MemSQL, a single store. Yeah, yeah. We just hired a salesperson and he was calling the customer and they were like, we don't want to talk to a salesperson. So they sent me out to New York. Imagine a 22-year-old wearing a suit that the sleeves are longer than—

The NBA draft pictures where they're all wearing these long suits. Have you seen those?

Exactly. Yes. And it's like 95 degrees in New York. I went on a red-eye because I didn't know what I was talking about, or I didn't know what I was doing traveling. And I showed up and spent all day debugging their database stuff with them. And they ended up being our first big customer, which was awesome. But Nice. You know, I just had all these formative experiences. I really— Eric, who was the CEO, Nikita, the CTO, they really gave me, I would say, an unfair amount of autonomy relative to my age and experience. It was, it was an incredible opportunity. And so, you know, I got to meet and work with a bunch of customers, many of whom I'm still in touch with today and use Braintrust. I got to hire a team. I learned how to manage people. I learned how to hire people, fire people. I met a bunch of VCs, many of whom I'm still in touch with today. And so it was, it was just an incredible foundational experience. I think, you know, anyone who's lucky enough to work with founders early in their career who give them just this insane autonomy to do things that they've never done before, you should just jump on that opportunity.

Like if I'm 22, I'm looking for my first startup job, like maybe I just graduated or maybe I'm thinking about dropping out. Or wherever I'm at and I'm trying to find that opportunity, what should I look for in terms of the founders that I'm working with?

I would just be super upfront about the things that you want to do and it will be very clear. So I remember most of the companies I talked to, I said, I want to start a company someday and I want to talk to customers and I'm willing to work as hard as humanly possible. And some of the companies I talked to said, oh no, you shouldn't do that to yourself, blah, blah, blah.

It sounds like it's probably not a fit for you.

Exactly. Yeah. Some companies I talked to said, oh yes, you know, as an engineer, our product managers, blah, blah, blah, blah, blah, blah, blah. And you have, you can measure your impact and blah, blah, blah, blah, blah. I remember when I talked to Eric and Nikita, they said, dude, there is so much shit to do. Like if you want to do something, just do it. Like I don't, I just don't care. It's pretty obvious.

Because I might be scared of saying to a founder, like, hey, I want to start my own company one day. Like that might indicate I'm going to leave and like 2 or 3 years. So you're saying that that's actually okay to go say?

Well, if you're thinking that, you should say it. Yeah, right. I mean, most, most good founders, they want to be part of that.

I mean, and they want to hire other autonomous people.

They want to hire autonomous people. They want to invest in your company. And when you start it, they want to help you start it. They want to collaborate with you after you start it. I'm still extremely good friends with both of those, with both Eric and Nikita, you know, and I ended up staying at MemSQL for 5.5 years. Every 6 months we had a discussion about career and is this the right time for me to start a company or whatever? And maybe if I wasn't upfront with them, I wouldn't have had the opportunity to be the VP of engineering. Because I remember I had a dinner with Nikita at some point and I was like, look, I feel like I've built a— and by the way, I was totally I was totally wrong about how much I thought I had learned and stuff. So, but I was like, you know, I feel like I've learned all this stuff. I'm not sure how much there is left for me to learn, blah, blah, blah. That's just totally wrong. But, you know, for me to continue doing this, I want to understand what's the next step in my career. And Nikita, I remember he was like, well, there is an opportunity to lead the whole engineering team. It's not going to be easy, but over the next 6 months, we can work together and try to set you up for that. And if it works, it works. And if it doesn't work— and I need you to commit, if it works, to staying for at least 2 years. And if it works, that's great. If it doesn't work, you can do something else, no problem. And it worked. And then we had the same discussion a couple years later, and I don't think anyone wanted me to leave. And there was a lot of discussion obviously around that, but the conclusion from the discussion is, hey, it's been 5 and a half years, maybe it's time to do my own thing. And, and it was totally fine.

And this is what Sam— he's— is he on your board or one of your investors?

He led our seed round.

Yeah. Yeah. At Greylock. And he, he said the— this was before Braintrust, this was Empyra. Yeah, yeah, yeah. He said you made the very first version of the product. You like hacked on a flight basically. What's the story there?

I was starting to think about Empyra, which was basically we basically helped companies use Stone Age nowadays AI to understand unstructured data. This was like NLP, natural language processing. Yeah, but back in that day, the only thing that worked was computer vision. So we were at first, I mean, I was just super fascinated by the idea of AI and ML changing how compute and data talk to each other because I'd been working on that with SQL for a long time at MemSQL. And I very, very prematurely saw that AI was going to change all of this. And so at that, I was like, how do I just get into AI given the skills and whatever I have? So I thought, okay, let's try to help people understand photos. And then eventually Empira started helping people understand documents and that's where we started to get commercial traction. But yeah, this flight basically, I'm the kind of person who I see a moment of inspiration and then it catalyzes me and I go. So I was on a trip with a really good friend of mine to Nairobi, and we spent a few days helping a kind of like the Amazon of Kenya there with doing some SQL stuff to analyze their supply chain and so on. And then we spent a few days on safari and I had just started getting into photography then too. So I took a bunch of pictures and we were flying back and it was like, okay, how do I find all the pictures of lions or those are the coolest pictures or the pictures of elephants? It was really hard. And so on the flight from Nairobi to Munich, I built the first version of Empyra that basically solved that problem.

Nice. And then what would you learn? I think this is a question from Alana. It's like, what'd you learn from Empyra?

So there's like an infinite number of things that you just sort of learn with life and maybe the best reflection of that is gray hair. But, you know, I think some of the key things that I learned, one of them is to sort of embrace who you are and not try to change who you are to match a company. I think there, especially when you work in another company for a while and you have some success doing that, you really learn how to adapt who you are and how you function to the sort of vibe and nature that the company operates with. But when you're starting a company, you actually need to do the opposite, and it's really, really, really hard to do that. You can't always meet people where they are, which is what most early founders attempt to do.

So you're saying you stick to, if you have a certain way of working, you don't bend to mold into maybe what the blog posts tell you you should do?

Yes. So I did that at Empira. We're not doing that at Braintrust. But that sort of leads me to my second point, which is I had this really interesting career arc where very early on, I became a VP of engineering and then a CEO. And if you just kind of Google those titles and read blog posts from the best who've ever done it, blah, blah, blah, those people shouldn't be coding. They should be delegating, hiring, solving escalations, whatever, not in the weeds of a product. And I made the mistake early on at Empyra of sort of continuing on that journey. And it was actually the wrong thing to do for a couple different reasons. The first is that the only reason I had the opportunity or deserved the right or whatever, however you want to frame it, to work on Empira is that I'm a pretty good programmer.

And then the blog post would have said, stop programming. Yes.

Yeah. You're a CEO now. Many people, our VCs, et cetera, were telling me like, hey, you know, you're an engineering guy and, you know, blah, blah, blah, but you shouldn't be writing code. You know, you should be doing other stuff. So I listened, and this will be the third piece of advice, but, and so I over-delegated a number of things around the product and engineering stuff, and I think that was the wrong thing to do. It sort of, we didn't end up building a product at first that was really true to me and, you know, how I wanted to create a product, and that just made things really difficult. Then the second part of that is the stage of Empyra was very different. Than the stage I was operating at MemSQL by the time that I left. And I think you have to adjust your leadership style to be stage appropriate. And at time t 0, when you're starting a company, you don't need to be delegating anything. You should be actually doing everything so that you understand intuitively and deeply what the product is, what the challenges in the software are, whatever, if you have the skillset. And then as the company scales, of course, you change what you're doing. So yeah, I think that, that was probably my biggest learning. I, I couldn't be more in the weeds at Braintrust than, you know, especially compared to that. And then I think the third thing is I really enjoy and feel positively about all the VC working relationships that I've had. They've, you know, been super supportive through a number of things. Um, so I don't mean to undermine that by any means, but I think listening to VC's advice is actually something that you should be very careful about doing. And honestly, most VCs, when they're giving advice, they, especially the good ones, they try to caveat that you should be careful not to necessarily listen to their advice. However, as a first-time founder, you don't have a boss for the first time, and it's very disorienting.

You can almost think the VCs are your boss.

Exactly. So until you have a discussion with yourself or your therapist or whoever, and you really unwind this and sort of deeply come to terms with the fact that the VC is not your boss, you tend to have that perspective and they're not. And so you have to learn not to listen to what VCs are telling you to do. The VCs, when we started Empira, they said, you are an engineering person, you're not a product person. So we need to find a really good product person. We ended up hiring someone really great, but it took a year and a half and we ran retained searches and all this stuff that occupied a ton of time searching for this person who was going to find us product-market fit instead of just working on the freaking product and doing it. And so yeah, I think you have to really understand innately what kind of company, product, et cetera, you want to build. Accept VC advice almost like customer feedback and prioritize it, triage it, politely reply but not always listen to it, et cetera. Or maybe not, whatever your style is.

Yeah. Yeah. So it's almost like when you're thinking about delegation, you delegate the things that you're specifically not good at, not necessarily the things that everyone else delegates. Is that a good way to think about it? Well, yeah.

I mean, I think there's phases. So at Braintrust, we have this amazing person who joined, I think he was employee number 4, including me, named Albert. Albert previously worked in venture and then before that worked in investment banking. And I think compared to other people in his generation, probably because of his upbringing, investment banking and so on, this guy grinds like crazy, right? I mean, he works super hard and he works super hard without strings attached. He's very, very philosophically aligned with me on that respect. And Albert does pretty much everything that I'm really bad at and don't find the energy to do. And that's one form of delegation. Actually, Elad really helped me think about that early on and why it was important to find someone who I could partner effectively with who would do a number of the things that are really, really important for a company, like make sure that we publish a blog post every week, but if left to my own devices, I'm just not going to find the energy to do that. And so that's one form of delegation that I think is really powerful. The other form is the more nuanced type, like in engineering, for example. I think trying to find engineers, designers, product managers, et cetera, who have really specific skill sets and are exceptionally good at things that maybe they're as good as you are. Maybe they're a little bit worse. In most cases, they're actually much better than you are at any individual thing. And then figuring out how to enable them and equip them and delegate to them effectively is really important. So, you know, with a number of our engineers, my job shifts from understanding customer context and writing the code to trying to be the frontline customer support and pull all the customer context into one place that a really good engineer can look at. And then figure out what to build, why to build it, whether to build it, and actually build it.

Yeah, you were telling me also you don't really do a lot of meetings. Can you kind of talk about how you structure your day that's maybe counterintuitive from the blog posts?

I hate meetings. So at Braintrust, we have one company meeting per week. Okay, wow, per week. I thought you might say day. No, no, no, it's one per week. It's scheduled for 30 minutes, and the longest it's ever taken is 15 minutes. Uh, we're actually going to make a change to the meeting starting next week that adds, because now we have 7 engineers on the team. So we're actually going to add a new component to the meeting where everyone does a demo. So I think it'll actually take the full 30 minutes. We've timed the demo slot so that it will. I'll report back next week, but yeah, it's only taken 15 minutes. And I think, you know, we, we give everyone on the team and we're very small, so I can say this, you know, and folks at larger companies, you know, will chuckle and they should, but we give people an insane amount of autonomy. And, you know, I'm actually the type of person who, who believes it's better to give people a lot of autonomy and for them to mess up occasionally or be slow or unproductive for a week while they're wrapping their head around a problem than to micromanage people. And there's this huge temptation to see someone take longer than they should be taking to work on something and start to add process to solve that. But actually, if you hire the right people, they know that too. And they might be working through their own journey of why something has taken them longer than it should. And then the next time they work on something, it won't. And I think when you give people that trust and space, they're just way more effective. So that's, that's one part of it. The other thing is that we're just able to do a lot more per IC at Braintrust than we would be if we had a lot of meetings. So I think every IC, including myself, is probably 2 or 3 times more productive without the interruptions of meetings all the time. And so that's, you know, another big part of it as well. You know, on the personal side, I just don't— I don't like meetings. Yeah, that's totally fair.

I definitely tilt more towards that direction. I guess I have to take them because I am a VC, but it's kind of like controversial. I'll talk to people. I'm like, I really only meet like one new founder a day. And some people are like, you know, I'm meeting 5, 6, 7, 8. And I'm just like, that sounds crazy. Like that's almost overkill. I don't know.

I can relate to that. Like early on at Empira, I thought that was my job. I need to fill out my day with meetings with customers and candidates. So I would literally reach out to people that were not great customers and meet with them because I thought I needed to meet a certain number of people per day. Yeah, that seems like on paper. And then, and then guess what? You do this. You practice a few times and you realize how to be persuasive to these people, and then you sign them up as customers, and then they're the wrong customers for your product, and then you're screwed. Or you meet a bunch of candidates and they don't meet the bar inherently that you need for your company, but you talk to all these people and like any other human, you start empathizing with them and you think, oh, this type of person, whatever, they're not so bad. Maybe we should hire them. So I think it's actually very, very dangerous to do that.

So with customer stuff, how do you determine someone is the right or wrong customer? Are there things to look for specifically, maybe Braintrust, if that's easiest to give examples?

When we started thinking about Braintrust, Elad and I wrote down a list of 50 companies that we wanted to talk to, to initially get feedback. And then if the feedback was good, get them to either use the product or invest in the company. I think another area that we've been really successful is actually building a coalition of really good investors. Everyone from Greg from OpenAI, Olivier from Datadog, folks that are really in observability, AI, et cetera, and really know what they're talking about. Many of our customers, the Zapier founders, Howie from Airtable, et cetera, Simon from Notion, they're all founders and, or they're all investors in Braintrust as well. And so we were just very deliberate about the companies that were ahead in AI, in technology, and we wanted to work with them. And then the other thing that we do that sort of naturally helps select the right people into the product, you have to be careful about this, is have a very self-service product. So this kind of happened by accident. If people from Zapier hear this, I'm sure they'll chuckle, but Zapier actually also has very few meetings. And so when we— they were our first user. And when we started working with them, I tried to get a bunch of them to talk to me, to give me feedback about the product. And Brian, who's the CTO, was like, this is just not going to work. Like, you're not— none of our— I was like, hey, can you please convince some of the engineers to talk to me? And he said no and said, like, you know, you need to make a video that sort of pitches the dev workflow that they can have with Braintrust, and then maybe they'll give you some feedback. And I made like 30 versions of that video, and then Brian kept giving me feedback, and finally it was good enough. So I sent it to the team, and then a few days later they started using the product. And by the way, creating the video actually helped create the product. So that was a pretty useful exercise. But we basically had to be self-service from the beginning because of who we were working with. And now that helps some people get our product, and for some people it's still really difficult or challenging to wrap their head around or they need more tutorial content, whatever. And we're always adding more content. We're always making the product easier to use. But in some ways, the people that use the product today have self-selected because they sort of get what we're all about.

So you said something really interesting earlier. You said you and Elad sat down. What customers do you want? How did Braintrust start?

Like first day, like what's the story there? Elad and I, have been pretty close for a while. He was one of the first investors in Empira, and then when Alana started her fund, he was the first investor in her fund and anchored her first fund. And Empira had its ups and downs. One thing I'll say is when we were working through the acquisition, which was not a straightforward process, Elad, who was one of the smallest dollar and equity investors in Empira, which is the furthest from the case in Braintrust, was one of the most helpful, probably the most helpful person through the acquisition process. I mean, just like I mentioned the list of 50 companies, when we were working on the acquisition, he helped me come up with a list of companies to work through the acquisition with. And I talked to him daily between, for probably 3 months while we were working through the acquisition. I think you all know how busy he is, that it's crazy. And so I think super highly of him and I enjoy every interaction I have the opportunity to have with him. And so when I was at Figma, we would continue chatting about various startups and, oh, this company is working on database stuff, what do you think? And so on. At one point we were talking about AI as it started picking up and he was like, hey, what kind of stuff have you guys been building? And so on. And I pointed out that at both Figma and Impura evals were a big problem and we had to build our own solution to it. And that we had tried using other kinds of products that were built for ML research and data warehouses and stuff, and it just, it never really worked. And so we had to build our own tooling. And I was kind of joking about it, like, this is annoying. And Elad picked up on that and was like, interesting, you've built the same thing twice. A year ago that maybe didn't matter, but now everyone is trying to figure out how to build with AI, and so probably everyone's going to have this problem. So we talked about potentially incubating a company that solved for it, and then we wrote down the list of 50 companies. We talked to a bunch of them, and Zapier and then Coda and a couple others, Airtable, started using the product and they started sending so much feedback that I looked at that and thought like, wow, okay, there's something very real here.

Yeah, they obviously care if they're giving you so much feedback.

Exactly. And that's kind of how Braintrust got started.

How did you choose the list of companies like design partners? You might have already mentioned, but maybe a more broader question, just how should you think about design partners as a founder starting a company?

Yeah, I think the big question, and I probably should have mentioned this when you asked about learnings from Empyra, The big question you need to ask yourself is what customer or type of customer do you want to slave the next 10 years of your life away for? The relationship that you have with your customers, it's not slavery, but it is— You're serving them. Yeah, it is servitude for sure. And you have to enjoy it. You have to understand them. You have to care about the same things that they care about. Because if you don't and they care about something and they can't communicate it to you in a way that deeply resonates with you, then you're not going to build the right product for them. And so, you know, I loved the customers that we worked with at Impura, and I especially loved the developers at those customers that we worked with at Impura. But I realized over time that I wasn't, I wasn't sort of naturally relating to them and I wasn't empathizing with their needs the way that I did when I was working with developers at MemSQL. And so yeah, the first thing I would say is you just have to be really deliberate about what type of customer you want to work for. At some point I was talking about Braintrust to one of my friends who's an advisor, and he was like, "Yeah, blah, blah, blah. It sounds really interesting, but developers are the worst market to sell to." And there's a lot of truth in that. I think grass is greener for everything, but My thinking was, yes, that's true. There's a number of challenges, but there's just nothing else I can do. And so I'm going to do it. Sometimes that's the best route. Yeah. And then once we sort of determined that, I think what we realized is that, and every market is different, if you're disrupting a legacy industry, I think your calculus might be a little bit different. But what we realized is that what the Zapiers and Airtables and Notions of the world are doing, today other companies are going to be doing in 6 months. And so how do we work with the companies that are doing today what people later on are going to be doing? And to solve for that, it was actually quite simple. We looked at all the companies in the tech ecosystem that had already shipped AI products, and there were very few of them at that point. And so we added them to the list, and then we added people that, you know, we just kind of knew them. So we knew that they were cooking up some interesting stuff. We added them to the list and there you go.

Straightforward, but also a lot of thought that went into it.

Yeah, I think, I think that this is the kind of thing that sounds very straightforward and obvious, but very few people do.

Yeah. So then in terms of like, you, you talk a little bit about just like intentionality of hiring, like what kind of hiring are you doing right now and, and who do you wanna bring on board?

Yeah, I'm, so our primary focus is hiring engineers at the moment. We're, we're also actually starting to build out the go-to-market team, so we're looking for a DevRel, our first DevRel, and we are also starting to look for more folks on the sales side. So those are the primary areas that we're hiring. I think Braintrust is in person in San Francisco. We are very low on meetings and structure. You have to really love extreme autonomy and ambiguity, and I would say you have to be motivated by that kind of environment. You know, I will say that compared to other companies I've worked on, the product market fit that we're experiencing is pretty insane. And so I don't want to overpromise, but you will have the opportunity to work with some of the best companies in the world. And the commercial traction that we're seeing is, it's pretty unbelievable. That's, it's really awesome. Maybe one thing not to be worried about, people actually use the product.

Yeah.

Yeah. Yeah. I mean, We do this really fun work trial thing actually for engineers who are willing to do it. And it obviously is a great way for us to figure out fit, but it actually ends up being a great recruiting tool as well because you might work on something that an engineer at one of these great companies is asking for, and then you get to ship it during your work trial because we ship the product more than 10 times a day. And then you get feedback from them on Slack, very excited, and then maybe a screenshot about how they used it. And you're an engineer, you know, you don't get that opportunity that often. And getting to do that multiple times in a week when you're actually doing a work trial with us, it sort of makes it, you know, like, wow, why wouldn't I want to work here?

Why would I go back to the other, whatever I was at before? Exactly. So then that work trial, how do you set that up? How did you, like, if somebody else is like, oh, this is an interesting concept, I want to also do work trials, How would you recommend doing work trials?

Yeah. So I think we've been very inspired by Linear, who wrote a guide about how they do it.

We'll put it in the show notes for people.

Awesome. Yeah. Awesome. Awesome. I would say the biggest surprise to me about the work trial is you have to separate how much you like the person from how productive they actually are. And so what we do is after a week, we run two mental thought exercises. One is during this week, was the person less than zero, equal to zero, or greater than zero productivity? It's actually very hard to be greater than zero productivity within a week.

Or you're saying in terms of what they contributed to Braintrust? Correct. Correct. Yeah. Okay. So did they add something?

Did they add something? Were they neutral, like a no-op, or were they negative? Most people end up being neutral, a few negative, but mostly neutral, and very few people are positive. I think we've hired maybe around 20% of the folks that have done work trials with us full-time, we've extended an offer to. So it's not a perfect process, and I'm sure we can do better. And I think we're a little bit better now than we were before in terms of the qualification before the work trial, but that's okay. It's actually been a very net positive experience for us. So yeah, one is really to ask that question. And if you believe the answer is greater than zero, debate it a little bit, maybe have someone on the team who's a little bit more skeptical and work through that puzzle. And then the other thing that we do is the product market fit test, which is if you took this person away, how disappointed would you be? And because the work trials are often a week or many of the folks that do work trials with us, they're in this sort of transition period between jobs, so they end up traveling. I think almost every work trial, someone has traveled in the middle of it or at the end of it just because It is their time off, and so they're sort of traveling too.

And they fly in or they just come into the office the full week or?

It's all over the place. So we have people who do the work remotely and then come in to hang out with us for a little bit. We have people who come in, we have people that are in San Francisco, and so they're just already there. We're not too crazy about doing the work trial itself in person because we don't want to create too much of a barrier for someone to try it out. But you, you can tell pretty quickly whether you sort of miss having the person around or miss having their presence. If, if you're sort of triaging issues and you wish you had this person around to help, like just throw something to because they can, you know, that they can drive it to completion. Sometimes you feel a bit of relief that the person isn't there because, oh, I don't need to like come up with a piece of work for this person to do. So I think the product market fit test is actually a pretty good way of assessing it as well.

And the work trial candidates are pushing code into production that customers are using. How did you get comfortable with that as a founder of like, you know, just kind of letting go of that with a candidate that's not even working at the company?

Yeah, I mean, we're very careful about code review and we leverage this really nice workflow on Vercel. I, I saw just to get somewhat technical for a short period of time, one of the cool things about Braintrust is that we allow customers to deploy the infrastructure in their own cloud environment, and we host the UI in our environment. And so what that means is if you're a customer, you know, like the Notions of the world who care a lot about data security, you don't want to send all of your AI data and whatever to a random startup who's helping you. We enable them to store all the data in their own cloud environment, but they're still able to use our UI that's updating all the time. And it's actually, there's actually not many products that do that. And it's, it's, it's not easy to build. However, once you architect and build a product that way, it allows you to kind of have the best of both worlds between security at scale and rapid development. So with that context in mind, we can change our UI constantly and we can do all kinds of fun stuff. One of them is actually powered by Vercel. We do these preview links. And so if you're working on a change, let's say one of our customers is like, hey, I have this piece of user feedback. I wish this button had this third option or whatever. You can actually work on the change on a branch and then send the customer a link to that preview thing. And it is accessing in a very secure way the data in their cloud environment, but they get to use software that's unreleased. Okay. It's pretty crazy. And so if you're doing a work trial, you can actually work on fairly novel changes. And while you're working on them, you can get user feedback without necessarily landing or shipping the change. And then once the feedback is vetted and the code is vetted via code review, then we can land it much more safely. So you're basically—

people push things into production that's used by the customers, but it's not actually the final product.

So it's like kind of this in-between state. Yeah, I think we've been very deliberate about engineering the engineering so that we can have a lot of velocity with the UI and the product, but do it in a way that works at enterprise scale. And so this is just an example of how we kind of take advantage of that.

Have you learned anything just in terms of selling to some of these enterprise customers that want AI products or that are trying to build with this stuff? Like any lessons on just like what they're looking for right now?

Let me start by saying, if you're trying to sell software to an enterprise company right now, and it doesn't have to do with AI, then it's really hard to get their attention. Really? Okay. Everyone from the board to the CEO, to the CIO, to the software engineering teams, the software engineers who themselves care about their individual resumes and want to make sure that they're working on things that protect their jobs over a long period of time, they see the opportunity with AI. And they are prioritizing it. And so if you are able to help a company with their AI agenda, then you have an unusually high shot at doing business with them. If you are working on something unrelated to AI, then you have an unusually low shot with— on doing business with them because it's going to be very hard to galvanize the amount of attention that's required for an enterprise company to actually sign a deal with you. It's a non-trivial process to do business with any company. And so I would just be mindful of that. And I, every founder I talk to, I basically say like, I understand you're passionate about whatever, but, you know, rewire yourself to be passionate about AI because unless you are, you're going to have a really hard time selling to the enterprise for the next 10 years.

Would an analogy be like cloud versus on-prem or like— For sure. Like internet versus no internet? Just like it's kind of one of those paradigms where it's going to change and—

Cloud versus on-prem is an awesome example that I have thousands of scars and wounds dealing with via MemSQL. At MemSQL, we were selling on-prem and there was a lot of opportunity there, but the insane amount of technology and engineering that we had to do to get baseline opportunity and attention from customers was significantly harder, more expensive, slower, et cetera. Than our rivals who were just killing it by selling cloud software. And by the way, because of how cloud software works and because of how it was new, the quality bar was lower because on-prem software, you can only redeliver it once every several months and you have to convince people to upgrade all this other stuff. Cloud software, you could actually ship broken software and then fix it immediately. And so that's, it's radical. And then If you're working with a company and you're trying to sell them AI stuff, I would be very careful about selling software that competes with internal software engineering efforts to accomplish the same thing. The reason is if you go to a company and you think about all the things that a company can do better because of AI, maybe let's use a canonical example like marketing, ad placement or whatever, something like that.

Automating just creation of ad.

Let's use that. Perfect. Yeah. Automating creation of ads. You might think as a founder, okay, great. I can help every company automate the creation of ads better. But guess what every software engineer at that company is thinking too? I need to figure out how to use AI to help my company do better. What are the opportunities in my company to do that? And so there are a lot of companies that are trying to sell to line of business AI-powered software right now. Or AI innovation, and there's nothing else to it other than the AI innovation stuff, and they're actually competing with the software engineers at that company who are trying to do the same thing and trying to engineer AI software to do that because they have the same goal as you do, you do not want to be doing that. And so whether it's this software engineering dynamic or whatever, I would really study the dynamics of how customers, software engineers, the internal forces within a company relate to the problem that you're trying to solve and then make the solution that you're selling something that enables the human beings at that company to be successful by betting on it.

So it's almost like solving new problems in a way, or something that just people aren't really using software for at the moment, or?

Potentially. Although I think there's a number of problems that people aren't using software for that the software engineers inside of that company might want to solve. You have to be careful. The thing about Braintrust that we've been somewhat deliberate with, and by the way, we do sometimes compete with internal efforts or internal tooling, is no one really wants to build logging eval UIs, like graphs, tables. It's just like— That's not really fun. It's not fun. It's really not fun. It has nothing to do with your company specifically. Yeah. And so why would we be building this? On the other hand, If you do something that those folks do want to build, then it can be very hard. There's no one-size-fits-all answer to this, but it's— and it's kind of generic sales advice. Sales is very much a human thing. So you have to understand how you are helping the humans at a company in a repeatable way with your product and almost engineer what your product is or why it is that way to help those humans.

It's funny that that's the takeaway when we're talking about like AI and tech software. Yeah. Okay. Makes sense. Do you have a favorite, just like AI product that you've come across recently or like something that you've liked? Like, I mean, it can be something everyone's heard of, but any more like under the radar too? Just like cool ones that you've seen. Yeah. Good question. That's from David Song on Elad's team had that question.

Oh, great. Yeah. So, okay. I'll give you my thing that everyone's heard of, which is Suno. I earned a lot of brownie points with Alana and her family over Christmas break by making a Suno song for every member of her family. It wasn't the only gift, but it was part of it. So thank you, Suno, for that.

They're like the, you type in a prompt and it creates a song out of it, basically?

Yeah, exactly. And then under the radar stuff, it's actually really hard for me to answer that question because I have access to what everyone may have access to in the next few months from almost all the cool companies. Let me put it this way. I think the workflows around AI, around productivity, around data analysis, there's really, really exciting stuff coming, just really exciting stuff. Our internal workflows at Braintrust are very, very, very different than if we had started the company a year or two earlier. The way that we, for example, do project management, the way that we collect customer feedback, the things that we choose to spend our time organizing, it's all very different. And yeah, I would say in general, I'm probably most excited about the productivity space as a whole, partly because I hate meetings. And I think the AI innovation that's coming with productivity will actually enable most companies to have far fewer meetings.

That's great. I love that. So to kind of recap, it would be just like there's a lot of big things coming that just no one's really come up with yet. It's new features. It's not just the stereotypical ChatGPT wrapper or whatever. It's new form factors.

What I find very exciting is that there are, I mean, there's some things that are very hard to do with models, but there are some things that are, once you see the ideas, you're like, wow, why didn't I think of this? But really the innovation is the UI, the UX, the grind work of collecting good eval datasets and then tuning everything to just work often enough that you're not upset with it. And there's some just really great product UI surface area coming out.

So it's basically when you talk about evals, fine-tuning, that's what you use Braintrust for. Exactly. So you guys are sitting right at the middle of all of it. Yeah.

I mean, if you want to build a really, really great AI product, you use Braintrust. Okay.

Well, that's, that's a good way to end the conversation, I think. Well, yeah. Thanks for doing this. This is a lot of fun.

Awesome. Yeah, I really enjoyed it.

Thank you for listening to The Peel. If you wanna support the show, share this episode with a friend and check out our wonderful sponsor in the show notes. If you don't wanna miss future episodes, subscribe to my newsletter in the show notes and you'll get new ones plus the transcripts in your inbox every week. Thanks again for listening. See you next time.

Hey everyone, Eric here at Turpentine. We're building the first media outlet for tech people by tech people. We're the network behind the show you're listening to right now. We have a slate of hit shows across a range of topics and industries from our AI and investing cluster of podcasts to shows that drive the conversation in tech with the most interesting thinkers, founders, investors, and influencers like Econ 102 with Noah Smith. We're launching new shows every week and we're looking for industry-leading sponsors. If you think that might be you and your company, email me at erik@turpentine.co. That's E-R-I-K at turpentine.co and let's partner together.