Error Analysis: The Highest ROI Technique In AI Engineering

Hey everyone, in today's video I'm sharing a conversation with Jacob Carter, founder of NurtureBoss, and Jacob is going to walk you through his experience with error analysis, which is my favorite technique by far when building AI products because it has such a high impact. What's fascinating about error analysis is that many people I would go even as far to say most people don't know about it, especially if you're not coming from a machine learning or data science background. If you've read my recent blog post, A Field Guide to Improving AI Products, you'll notice error analysis as a key theme. While the blog post has several case studies in it, I'm excited that you get to hear from Jacob himself.

Quick background of Nurturboss and what we do. We build AI products for, for multifamily or the apartment industry. AI products for us means virtual leasing assistant. So imagine, uh, imagine the folks that are working on site at apartment communities. They're answering phones, uh, having conversations with prospective residents, helping to get tours scheduled, uh, answer questions around amenities or pricing and availability, uh, and then also following up with those prospects via text and email to try to drive them to convert to become residents. And then once they are residents, there's all kinds of work around collecting rent on time, chasing down overdue rent, boosting online reviews for that property, facilitating maintenance requests and service requests. So all these things, NurtureBoss has AI products that do that on behalf of the on-site team. I'm so worth calling out the main channels that we communicate with our voice email, and text. And those are all gonna operate and work a little bit differently, but they do ultimately all run on OpenAI. So when we released our products, you know, it was a giant black box for us in terms of what was happening, if it was going well, if it was not going well. In an effort to self-educate, I took to the internet and stumbled upon Hamel Husain's blog, which I learned a ton from and ultimately just led me to reach out to him and ask if he could help us kind of become experts in developing, iterating on, and measuring AI products. And that's when we got into the eval conversation and the look at your own data conversation. So for context, I had no idea or understood at all the value that would come from quite literally just looking at our data. Hamel Husain, I don't know if I ever expressed this with you, but it was kind of frustrating when you kept telling me that, like, what is the secret here that I'm going to learn? And then I got into it and realized how impactful, insightful, and important it was that we do that, and we'll continue to do that. So to this point, you know, I've looked at thousands of conversations via phone, email, and text that our AI has had with residents or prospective residents, and a few things have stood out. That I think are worth mentioning really quick. One of those is getting an understanding of what we should be building. So my background, I've been a software engineer for about 15 years. I've spent a handful of those years on the product side of the house versus the engineering side of the house. And when you're trying to decide what to build for your customers, you do discovery, you do, you know, beta testing, you do journey maps, you have all these processes. Being able to actually see what people are asking your product every day and identifying gaps where they're looking to get problems solved that our product does not know how to solve, it builds your product roadmap for you. So that's been really exciting and really helpful to be able to get that out of it. Equally as important though, you start understanding how people assume your product is going to work, which never lines up with how you thought people were going to assume your product would work. So it identifies a lot of bugs, adjustments, and changes that you need to make based off of what users are looking for out of your product versus what you've actually built. And those patterns really just surface and become obvious when you start reading more and more conversations. You also start to get an idea of how the AI model is going to respond. So you make assumptions there as well, But when you see it respond to the same type of questions again and again, you really start to be able to kind of put yourself into the mind of what the model's gonna do, which informs how you build features in the future and how you work on different things like tools and RAG when, you know, prompt engineering when—

Jacob, would it be helpful to share your thing, your data viewer?

Yeah, yeah, I'll bring that up now. Yep. So when we started getting into looking at this stuff, you know, we built our own tool like a lot of people do to quickly iterate and go through and judge different conversations. So these are text message conversations, and then we can read through this conversation and see in detail what happened with the AI. The human said this, the AI said that, the AI called the tool. This is the response that the AI got back from the tool, and then I can come through here and say if this conversation was good or bad, and then I can quickly type up notes explaining why it was good or bad. And the act of doing this, I learn a lot, but then also it's really easy to build tools that are also powered by AI saying, hey, for all of my text messages, categorize the negative results that I get. And by clicking that, I see, okay, I'm getting tour scheduling errors at the most, rescheduling, which I would group with tour scheduling, Handoffs were not triggered. What that means is we wanted the AI to call in help from a human and it didn't. That's what a handoff would be considered. So it's really easy for me to see this and get an idea of what's going well and not going well. And now that I have that, I can just get to work, right? That's enough for me to just get to work as a product owner. I can start creating bug tickets, creating feature tickets, instructing the engineering team on what to focus on and what to build. So that alone is incredibly valuable, but what we then can do is take this data into a tool like Braintrust and start running actual automated tasks, you know, the variety of evals, whether it be unit tests or other, to start understanding how changes we're making are actually implementing or solving those problems. So one thing that I guess I'll just bring up because it's specific that we use Braintrust to solve Everybody deals with this problem, but I think nobody expects to deal with it, which is that AI models are not very good at understanding dates, what day it is, what the future is, what the past is. So we would have a lot of customers say, "Hey, we want to schedule a tour for 2 weeks from now." And if they ask that question today, the AI would say, "Okay, 2 weeks from now, that's February 28th or February 29th, if we're on a leap year, March 1st, but definitely not 2 weeks from now, right?" or we'd get somebody say, "Hey, we want to schedule a tour for March 1st," and the AI would look for tour availability for March 1st, 2022. So this date handling issue was a big problem for us, and we were able to use the playground in Braintrust to create a bunch of use cases or tests of different ways to reference dates and then see the output the model was giving us for what it interpreted those dates as. And we went from a 33% success rate to a 100% success rate through iterating on that problem through, through Braintrust. So I think that's another just hyperbole.

One thing I want to highlight here is like we're not using— so we're not, you know, we're not talking about generic stuff like conciseness score, hallucination score, all that nonsense. You know, we're talking about real pain points that Jacob identified and now he is addressing, and, you know, we're going to start measuring that soon. We just got off the, you know, we're just ending this like error analysis stage, but he's making real progress. Like, he is moving forward. It's not just some, you know, exercise, academic exercise, which is a lot of people go through when they try to do evals. He's actually making real progress. You can even hear it in his story. Like, he has an understanding.

Yeah, yeah, absolutely. Every, everything that we've been doing really from the start immediately started exposing action items that our product team could operate on immediately and start improving our product, creating a better experience for our customers, and also educate us internally on how to build things moving forward based on the learnings that you get by just looking at your data, which again, while it sounds so simple and silly, I can't stress enough how impactful it's actually been to to just look at the data and see what's going on with the conversations. The other thing I just, I would mention, last thing I'd mention really quick around kind of those generic comments that Hamel's making and the lack of value in the generic evals is that inevitably your system's gonna be set up in a way that not only makes it so the generic eval doesn't make sense, but even, you know, a lot of the generic eval tools won't work perfectly off the shelf. You have to understand the nuances of your setup so that you can use these things in a way that's valuable for you. So what I mean by that is, I think Hamel Husain, you know, one of the big realizations that we had is when getting into our data is that we really have evergreen threads, if you will, right? Because these conversations don't just start and then they're over. They go on for the lifetime of somebody looking for an apartment, to renting the apartment, to living in the apartment, to renewing the lease. So on and so forth. So it's not like I just have a conversation. I can say, here's a start and stop completed conversation. It's evergreen. There may be more, there may be less, you know, we don't really know when it's gonna end. And that's something we had to think about and consider. By getting into the data, by looking at the data, you start to realize why it's not working or making sense, what your unique use case is, and you can start to either build tooling around that or know how to leverage existing tooling in a way that makes sense for the way that you have things set up.

Okay, so we heard from Jacob about his journey with error analysis. There's a couple of things that I think we should take away from that. One, Jacob didn't use off-the-shelf metrics or generic approaches to diagnosing his AI issues. He started by looking at his own data, and you heard from Jacob himself, he used what he found to drive his entire product roadmap. It was that powerful. Second, Jacob created his own data viewer, and I think this is really important because it removes all friction from looking at data, and Jacob was able to render the data in a way that made sense for him. The process of doing error analysis involves looking at data, writing detailed notes about problems you find, and then categorizing those notes and counting which errors are occurring more, most often. And that's the simplest form of, of error analysis, and that's what I recommend starting with. What you saw is that Jacob even automated a bit of this by using LLMs to help categorize those notes and then aggregating those categories and counting them. Lastly, I'd like to take— I'd like to thank Jacob for sharing his experience. Not all companies are that open. I hope you learned something about error analysis today, or it has piqued your interest. Either way, I'll drop a link to the blog post that accompanies this video in the show notes.

Thank you.