In this talk, Zapier senior engineer Olmo Maldonado and Braintrust founder Ankur Goyal walk through Zapier’s recent AI product development, focusing on the AI Zap Builder and Zapier Copilot. They describe Zapier’s philosophy of rapid prototyping and iterative improvement, and how they embed product‑manager input into the development loop. The core of their process is an in‑house evaluation framework built with Braintrust that generates synthetic data, runs continuous integration tests, and uses custom logic‑based and LLM‑based graders to monitor over 800 unit tests (up from 7). Observability tools from Braintrust provide fine‑grained tracing of multi‑tool agent flows, allowing the team to pinpoint regressions. The presentation details challenges in prompt engineering when migrating from GPT‑3.5 Turbo to GPT‑4 Turbo and later GPT‑4o, including system‑prompt handling and function usage, and how small prompt adjustments restored accuracy. Metrics show a 300 % accuracy improvement, latency reduction from ~14 s to ~3 s, and expanded test coverage. The talk concludes with reflections on the partnership with Braintrust, emphasizing the value of integrated evals, observability, and prompt tooling for building reliable, high‑performing AI features at scale.
Chapters
My name is Olmo Maldonado. I am from Zapier. I promise I am not an open source LLM. My name is Olmo. I've been at Zapier for over 7 years. I apologize for all the bugs that I've introduced that may have affected you. And if you happen to have any bugs that you want to report, I'd be happy to take that in and start working on them. You name it, I've been part of the team, so, you know, feel free to come by and talk to me about it. I'm a lucky husband and father, and I've been playing some golf lately, and I don't know why I'm doing that.
Hey everyone, I'm Ankur. I actually went through Hamil's journey, I think. I built eval tooling at my last startup, Impura, and then when I led the AI team at Figma. And that's what led to Braintrust. Actually, Braintrust started kind of as a collaboration with our friends at Zapier, who were our first users, and it's been a lot of fun since. I'm also a husband, not yet a father, but a proud older brother, and also a reluctant golf player. I hope to play against Olmo and beat him someday.
All right, cool. So today it's gonna be a lot of storytelling. I'm not here to prophesize what you should be doing. I'm just gonna share what we've done that has worked for us, and I'm hoping to actually learn a lot from you all as well of what is working for you and something that we should try ourselves. I know I've already learned a lot from the conference, so I'm hoping that you all learn something from this talk. We'll go over what we're doing at Zapier, the tech that is going on at Braintrust, and a couple of examples of how what they've done has really helped us to make a good product. So if you're not familiar, we're actually just at a quick poll. How many folks here use Zapier? All right, nice. A good number of folks, appreciate that. Thank you for your support. If you're not aware, there's actually over 7,000 apps of your favorite ones online. We make it very low-code to no-code to integrate with all of them. On the right, you can kind of see how the workflow works, and we'll do it in a reliable way to make sure it's mission-critical and everything. Now, we have a lot of AI integrations at this point. I'm happy to say per day we're doing over 10 million tasks. So if you haven't tried us out, please give it a try as well as use all the integrations with AI. Here are the apps that we— the products that we have built with AI. I'll only talk about the first two with the AI Zap Builder and Zapier Copilot, but I would strongly encourage for you guys to explore all the other new products that we have available. Central in particular, shout out to my colleagues that are here. It's a bot framework, so you can make your own bot connect to over all of your apps that you have online. So, yeah, if you want to learn more, please go to zapier.com/ai.
Really quickly about Braintrust, we'll keep the propaganda brief. Braintrust is the end-to-end developer platform that some of the world's best AI teams use, including Notion, Airtable, Instacart, Zapier, Vercel, and many others. And basically, if you break that down, there are 3 things that we're really focused on today. One is helping you do evals incredibly well. Olmo's gonna talk about how they do evals, which I think is probably the best way to actually learn about that. We also help you with observability, and I think it's really important that you build your stuff in a way that evals and observability, there's actually kind of a continuum across them. And so we are really kind of focused on that problem. And then the last thing is we help you build really, really great prompts, and there's a bunch of tools around that.
Yeah, so this is what it looks like to work at Zapier. We want to get the prototype as early as possible to the user. We take an iterative approach. We will get some things wrong, and we hope to learn from them and just keep improving the product as fast as possible. We make adjustments through our evals, and evals are the way that we make decisions. It didn't used to be that case. So if you haven't played with it, this is the AI Zap Builder. You give us a prompt as you see there, we'll do our best to make a Zap for you. And when you click try it, there's your Zap. Um, it will do, uh, many other things as well, uh, like field mapping and so forth. So what we learned from this experience is how do we go about knowing How well is this product, you know, delivering Zapier to the customer? Over 7,000 products, the long tail of integrations is vast, so how well are we doing it? One of the things that we did that has worked for us that I would encourage for you all to do as well is involve the product managers, involve your product side of things to be part of the conversation. This is an engineering problem as well as a product problem, And you can see here in this screenshot our P0 priority, the things that we wanted to make sure that our AI Zap Builder was able to produce. We wanted to make sure the triggers, actions were working as you'd expect. We don't want the, like, the wrong step in the wrong place. And we wanted to make sure the top 25 apps were supported and that they were done in an eloquent way, like, correct way. And yeah, we have even internal Zapier, Uh, apps like paths and, uh, filters, we want them to work as well. So all of this had to be in our eval suite in some way or form. This is our, um, framework that we built in-house with the help of Braintrust. Um, we have synthetic data from our corporate accounts that we use for seeding the evals and we use that to get going with all of that coverage that you saw before. What we do is we load that data from Braintrust, actually, that is hosting us, and we, we take that and run that on a CI basis as well as a manual basis. And we have our own little runner that essentially kind of does a load test against all of our AI providers every single time that we run this. So it's been really incredible to take all of this data, run it, report on it, and start acting on the things that we've seen. Uh, we also have these custom graders that the previous speaker had mentioned. They're both logic-based as well as, um, LLM-based. Uh, and in general, what we're trying to do is make sure that that criteria that you saw before is being tested among and that we are actually acting on what we wanted to see. So here's an example, uh, of all the different runs that we've had. Uh, as you can see, it's pretty often that we run it. We want to make sure that if any regressions happen, that we act on them quickly. I'll actually go over one of those cases in a bit. And this is really easy for us to act on it. We can see this is a screenshot of Maggie's project for the, for the Zap AI Builder. And, you know, as mentioned earlier, we have observability thanks to Braintrust. We can see within it what happened, what were the inputs, what were the outputs, as well as compare the pink and green. Hopefully you all can see it. Is actually comparing against previous runs as well and trying to find, huh, what, what went down, what went up, and so forth. Um, and yeah, this is just showing that even further with all of our different graders, the scores, if you will, of like the different things we're looking for. We want to make sure that the ones that we care most about are being highlighted and that we do something about it. So after all this work of creating the eval suite and running them continuously, I can say that before this, we just had 7 unit tests. So 7 unit tests that were run manually by devs, and now we have over 800 of them, and they're all run part of this merge request as well as on a continuous basis, and we get alerted on if any regress. So a lot better coverage there. This has led us to improve nearly 300% of our accuracy. I will say that is not saying that we're at 100%, We still have a lot of work there, but it is fortunate that we were able to improve with this process that we created. Now, we're very thankful for our customers. These are just a few shoutouts of how they have received that product. This is using an older UI, but it's essentially the same product. Now, one thing about that product is a single-shot approach can only take us so far. So this is the next iteration. As you all might imagine, it's a chat interface. We want to allow the user to interact with the editor as they're happening. So a progressive iterative approach. You can see in the demo here, the GIF, that we're— we not only did the same prompt that you— that we did before, but we're also testing steps. We're also configuring fields all as quickly as possible to the user. This is actually not sped up. So we're really happy so far with the performance that we're getting out of this thing. But yeah, the problem with it though, now that it's in a kind of like an agent framework with multiple tools that it calls, we couldn't see what was the critical path. What is the things that we need to improve now to make the accuracy even better, to make the experience better? And this is where again Braintrust came in. They have tracing capabilities. This allowed us to break down the request from very granular observability to a very gran— fine, fine look into the problem. And just as you would expect, you can actually see the inputs and outputs of a chat completion, the tokens, the time to response, you name it, it's available. And we can quickly iterate on that one as well with the, with the playground that they have. Not showing that, but just wanted to showcase that it's really easy as developers to go into it and really understand what is going on with with the performance of the Copilot. So one of the things that ended up happening with the Copilot is, you know, early on, because we wanted to get to market first, we wanted to just do GPT-3.5 Turbo. And, you know, we, we started testing different models. And this view that you're seeing here was manually made for us so that we can get a better sense of you know, the performance of different models across different tools. And as you change those tools, what is the performance characteristic overall? And we settled on GPT-4 Turbo for our message router that we call. And unfortunately, that came at a cost of performance. That was a lot slower than before, but at least we were able to provide some guarantees to our customers on accuracy. That led us to want to make that better, and we— our evals started to show a regression. We were wondering what happened, and the reason for that was we switched to 4.0. Now, you might have already heard from the conference that 4.0 is— you need to be careful with it. And yeah, we actually stumbled on that one. But I can say that with few changes that we had to make and a couple of parameters that we added to the OpenAI calls, we're kind of back to the same performance that we were before, and that's what I want to talk to you about right now. So as you know, we can see all the examples or all the different runs that we've had with experiments. We noticed beforehand that we were 80% or better in most of our scores. Um, after the change to the GPT-4o, all of that regressed below 80%, so we were really worried that You know, we wanted the performance benefits and the cost benefits, but we didn't want to lower our scores. So what should we do? This is showcasing some of the work that my colleague Maggie Cody had. So shout out to her for her hard work in this. You can kind of see that all of our scores went down, and we can clearly see that there's a pattern to this. Like, what is going on with it? Like, drilling in further, we noticed that, you know, the 22 regressions that happened in here were all related to OpenAI deciding to forget our system prompt in some way or to give an answer back that we didn't want to see or do. It just, in some ways, we had prompts that were too fine-tuned to GPT-3.5 Turbo. So we had to regress our engineering, if you will, our prompt engineering. And that ended up allowing us to actually go back to the numbers that we were at. So this is the example of that prompt, how we were a lot more elaborate with how we were asking the GPT-3.5 to, like, respect our wishes. And afterwards, we actually just relaxed it a lot more. And again, trial and error, really quickly iterating on that loop that I mentioned earlier. And that is what led us to make these discoveries. Now, the other thing that I mentioned just to showcase that is, you know, we made a change to the tool choices. We deprecated the functions usage. Over to the tool choice. Um, that tool choice auto is actually an asterisk there because we are also going to experiment with making that required. So that's coming up next. So yeah, overall after those changes we can see immediately that most of our scores went up. Uh, we're really happy that for us it's really easy to compare back and forth with, uh, previous runs. That's what's going on here on the top. Since then, we've been able to adopt GPT-4.0. We still have more work to do there, like I mentioned, but it's, it's an iterative approach. Before this adoption, we were around 14 seconds, and now we're at 3 seconds for stream-based Copilot. And of course, we had a lot more reduction as we did before, or with this. So in conclusion, I just wanted to share some of our stories of how we've worked with Braintrust from the very beginning to make a great product. Really, we, we couldn't be happier working with them, and I don't know, there's not much more to say to that.
