LLM Eval Office Hours #3: The Importance Of Starting With Error Analysis

I, I've prepared a few things, um, just to make sure you are well apprised. I'm going to share this. I think it's just the briefest of like slides, and we can look at some code and stuff like that just so you have context. I think definitely, um, the app is an SMS-based, um, Let me get this out of the way. Make this maybe bigger. SMS-based caregiving app. So if you can kind of see the architecture here, Twilio to FastAPI is a Qdrant vector store, Memzero for memory, Azure OpenAI and Helicone for observability. So all those kind of pieces. The personas that I'm working with are early stage, steady state, and kind of day-to-day caregivers. Eventually, I'll be looking at something that is a little bit more sophisticated. So it's a little bit more basic right now. Features are very much focused on self-care. I think of this middle section really as kind of memory and retention around that. So recall. And since self-care is a big component of that, there is, built in the system prompt, a kind of a reflection point. So it always kind of folds back on that. And there's a bit around, like, it's not about any medical advice. Um, I just laid out some V2 features because I know they're coming and they get a little bit more sophisticated. And, um, so I'm just trying to figure out where they fit because they get into like structured output, RAG type stuff. Um, but not, not important for now, I don't think. Um, scenarios, you know, someone comes in looking for suggestions. So, you know, the format I tried to think about here is, you know, what they like, you know, a typical kind of user story and then what the app does, the expected app behavior. So I went through a few of those and then same thing with like v2 expected ones. I wrote out a couple test cases and I think this is one of those places where, you know, Yeah, let's see.

What is the caregiver? Is it like a doctor? Is it any kind of caregiver of any kind?

Yeah, yeah, and I probably should have probably explained that. Yeah, this is an unpaid caregiver. So this is like you taking care of your mother, you taking care of your father, or this is a parent taking care of like a disabled, uh, child. Gotcha. So, and that represents about like 50 million Americans. So it's on a small, Tim. Test cases here, I'm trying to think about, you know, again, like, you know, when I'm trying to wrap my head around evals, I think the biggest stumbling block I'm running into as a multi-turn conversation experience with memory, I'm not even trying to get into like the grounding because it's, you know, maybe medical and things like that. But the multi-turn bit and the memory bit are the things that I'm tripping up over. And maybe I'm overcomplicating it. But this is where I was just trying to get to the most simple evaluative heuristic that I could, you know, in terms of, you know, taking some of the sample user data that I have. So I have 10 beta users that have been using it for the past, like, 2 months. 3 months, and the expected behavior, and then how I would kind of create it. So I wrote that up for 5 cases. And then last night, because I reread your 2 blog posts, I wrote up some unit tests. I haven't run them, but this is also where— and we can look at them if it's helpful. Um, but you know, again, this is like, you know, trying to figure out if I'm, if I'm doing these things right.

Um, okay, yeah, that's a good question. So we have some unit tests. Uh, can I look— can we go back like to slide 8 or 9? Just wanna just look at it one more time. Okay, so you have like some of these test cases and okay, have some scenarios or like I see passive, supportive. Okay, so you have some ideas. Can we look at slide 9? Just want to remind myself.

Yeah.

System provides general self-care. Okay, go to slide 10 again. Okay. Okay. Yeah. Okay.

So this is all using, you know, not— it doesn't go out, doesn't hit any APIs, right? I ran it once and, you know, I think thankfully, right, it failed a few. So like it gives me something to work with. And then we'll hop into Cursor in a second, right? And the last thing I want to show you here is in Azure AI Foundry, I exported out from Helicone my dataset, which was like 223 rows. And I imported that into Azure AI Foundry. They have an eval thing, which I think is very similar to what OpenAI has. And then I ran it through. And I know this is one of those probably I shouldn't have, but it was what I had provisioned. So I just ran with that. Sure.

Yeah.

So like I had, I used the same model, you know, 4.0 as my judge. And I know you want to typically write, use a smarter judge, a smarter model as a judge or something like that. But other than that, I set the same parameters. And then it ran through. It does a, you know, a standard set. I think you can also set some other criteria. Um, and you know, there's a CSV that, that accompanies this as well. Um, yeah, I mean, I have some questions, but then we can just—

yeah, thanks for preparing those.

Yeah.

Um, can you— so is— so before you get into like these evals and stuff, Are there specific failure modes? Where are you at in the app? Are you kind of still building it, still prototyping it? Do you have anyone using it already?

Yeah, I have 10 beta users that are using it on a semi-regular basis, right? Like, it's SMS, so it's very hard to kind of keep them kind of in the loop because I don't have anything that is, um, I don't have any kind of push out to them, so it's very reliant on them picking up and using it.

Okay. And then that's a common thing. Yeah, like a lot of people do SMS route. Makes sense.

And building out a separate, like, production version, which is slightly different, but, um, but yeah.

Okay. And, um, The evals that you think you're thinking of, or that you have right now, like, are they based on failures that you're seeing from those users?

Not exactly. I think they're— they just represent really more of, I think, the, the general criteria that I, I think the, the app or experience should adhere to. Right.

Um, are you looking at that data from the users, like their interactions and— yeah, kind of seeing how it's going?

Yeah. And that's, that's where some of the, you know, I think that's where some of this, I think, came from. And when I was looking at it as well, um, so I, and I know some of, like, I think the easiest stuff to pull out is, um, You know, and I think what I do like about this is that the coherence and the fluency bit, or the coherence and the relevance, because this is conversational, is how friendly, because that's how, you know, I think a big feature, right?

Have you looked at the individual conversations? Yeah. And Have you been, um, have you done error analysis yet on those conversations to try to categorize like what, where, where they might be failing?

No, I wouldn't even know how to kind of like define that. Yeah.

Okay. That's good. Yeah. I think that's one thing that's like, uh, being surfaced in the office hours actually. Um, that is, uh, interesting is like a lot of people they don't know where to start with the evaluations. There's a lot of different things you can evaluate. You can have all these different metrics. You can come up with a lot of different tests. Um, you can come up with an infinite number of tests. You can brainstorm all kinds of stuff. And then ultimately, like, that, you know, that can— you don't want to just keep doing that because you need to develop your application. So like, you have to prioritize what you're evaluating based on something, um, just because it's an infinite space. And the way you prioritize that is, okay, if you have users, if you have a fair number of users, you have 10, hopefully you're getting some, you know, consistent data, um, you know, because you have usage. And what you do is like you read every single— like the order of 10 users, I would say you can read everything at this point, which is fine.

Yes. Yeah.

And, um, you know, is to go through each interaction very carefully and just categorize like what are the different kinds of errors you're seeing. And you can do that in like— just think of it, it's like in a in a dumb way. Like, it doesn't have to be anything fancy. It could just be in a spreadsheet where you just add columns for different categories and you just put an X like, okay, this is that, this kind of error. It's not retrieving the right context. It's not friendly enough. Whatever, whatever. And after you look at, let's say, I don't know, 50 interactions, you will know. You can say like, oh, you know what, like, this is my biggest issue here. This is like the most important thing. You might not even need to get to 50. Like, you might get through 25 and it would just be extremely obvious that like this one thing that you need to do. So when you first start doing that, um, the first phase is like you don't even need to do evals maybe because you will see something like so obvious that you're just like, okay, you know what, I'm just gonna fix it. Like, forget evals, let me just work on that right now. But then, like, after you get to some point where it's like, okay, you kind of fix some of the bigger things, then you have, like, some other things that are, like, a little bit more difficult. You're, like, not sure how to fix those, or, like, that, you know, you kind of pluck some low-hanging fruit, then you can, like, focus on creating evals for that. And like kind of let that help you prioritize. I mean, what you're doing is okay. Like, it's okay to have ideas of like stuff you want to test for, especially if like those are important to you. Um, but let it— if you can, let it be driven by what you're seeing that's wrong, and that will like, that will, um, influence everything else. Like You know, like, cause this coherence and fluency, groundedness, all this stuff, like it may not mean anything to you, you know? Like, I don't know what this means. Like if you get a score of 3.72, tomorrow you get a score of 4.2. Does it really mean that it's better? We don't know. That's the problem with this, you know? And if you, you don't wanna waste time on this cause like you have limited time. And so, um, That's where it should start.

Okay, so like even where I start with like the multi-turn bit of the— is a non-issue, right?

Multi-turn is fine. Yeah, I mean like multi-turn is— yeah, there's nothing wrong with multi-turn. In fact, most applications that I work with are multi-turn.

Um, but I mean like evaluate it because like each, you know conversation response is a row. So just evaluate each of those and see how— and judge those on an individual basis.

So when you do the error analysis, yeah, okay. So when you do the error analysis, you're really looking at it as a whole? Yeah, like does this session— I'm just going to say session.

Yeah, like a trace, right?

Yeah, kind of like a session. I know it's like SMS, so it's like what is a session? You have to have some way of kind of breaking up things into sessions, you know, maybe it's like some time cutoff. I mean, there's always, there's some logical aspect to it, like, yeah, but like, okay, in a given session, there might be X number of turns. Did, you know, did your assistant accomplish what you wanted it to with the user? And like, if not, if you're not doing it in the right way, we need to like, categorize that. And you look at it like more holistically.

Yeah.

And then based on the failure that you see, like, oh, okay, it's not— has not retrieved the right context, or it's just, you know, or it's like doing something dumb like emitting user IDs into the output or doing something other. It's like any kind of error you see, then we can decide like how you evaluate that, like how you write tests for that. Sometimes you have to look at the whole session. Yeah, you kind of evaluate it. Sometimes you can do something depending on the error. You can like, you know, look for a certain message or only look at like the last message or the first message, you know, depending on the error. Um, you know, kind of drives how you test it.

So yeah, I think what I'm hearing, this is really fundamentally like a bottoms-up type of a thing as opposed to like a like a top-down, right? Like I should look at what's there, right, as the basis to determine, um, what is, you know, what are the areas to build around, as opposed to come in with a preconceived notion, right, of what I believe to be, um, you know, the the core features or experience of the app and then shape it around that? Or is it kind of like bring those two together?

Excellent question. Um, I, so different people have different flavors that they like. I really like, I'm, I'm kind of a little bit more biased towards the bottoms up, like you said. Yeah. There's a way of being a little bit more top-down, which is instead of writing the tests up front for what you want to— what the behavior you want to be— simulate that behavior that you want to test and let it appear in the data so that you can then decide, is this worth testing? Because what happens is you have this idea of what you want to test in your head Yeah, however, it's often— it's only until you can see something and react to it that it refines the idea of, you know, what it is that you want to test. Like, it's easy to like write a test, but then— so like, you have to, in my opinion, you kind of have to see it. So you, you know, like, if some of these things are important, like the things that you kind of outlined on slide 8 and 9, you know, you can try to simulate those user interactions with an LLM, and then you can, like, then you can see.

So this is the go down a little bit of a synthetic data and then simulation route, and then, you know, determine, like, is— are those, you know, scenarios and evaluation criteria useful in, you know, the feature set in the app, right?

Like that type of a Yeah, absolutely. Because it gives you something tangible to work with.

Yeah.

Because, hey, if you want to do— if you want to accomplish those goals that you laid out in slide 8 and 9.

Yeah.

You know, if you generate synthetic data that tests those goals, then you have something to work with and iterate on.

Yeah.

You know, it's like, make those interact— like, make those interactions work well. In, in response to those synthetic users.

Is there— I mean, have you— maybe you've written about it, like, have you written about just general good guidance on, like, number of runs or how to generally approach, like, that from just, you know, I, I guess, starting, right?

Like, uh, in which part, the synthetic users or synthetic data, or which, which aspect?

I guess both parts, right? Like setting up the synthetic users, the simulation to arrive at like your, you know, a conclusion presumably, right? Like on—

Yeah. Okay. Yeah, this is a good question. Everyone asks this question and just, just start small. You don't have to have a set number in mind. People really stress out about this. They're like, okay, how many should I look at? Should I look at 1,000? Should I look at 100? I don't have time to look at 100. That's not going to be fun, blah, blah, blah. You know, just look at— just start looking at it.

Yeah.

Um, you will get— the heuristic is keep looking at data until you feel like you're not learning anything new.

Hmm.

And what that means is either A, when you look at data, you're gonna see like one glaring problem that overshadows every other problem, and then you'll know, okay, let's just stop looking at data and let's work on this problem. Or as you're looking, you'll be like, wow, okay, like this is getting like really new information as I look at each row of data or each instance of data. Let me keep going a little bit. And then, um, you reach a point where you're like, oh, I kind of feel like I kind of have an idea, and you can follow some of your intuition. And it doesn't often have to be that much data. And then it's like, for synthetic generation, just start really simple, like just generate like 1 or 2 test cases, you know, for each of these things. And you may decide when you look at it, you may look at it and say, okay, you know, um, Just iterate like slowly from there, and then you can just generate a little bit more and more, but don't make it overwhelming. Um, you got— there's some amount of faith, I guess, you have to have. I mean, but it's surprising, like when you start looking at data, you, you get— it's valuable much faster than you think.

Yeah, no, I mean, it makes perfect sense. So I think that that's helpful, and I think that I think helps me. I think that eases, I think, a lot of anxiety I had just, you know, thinking about just the, the, you know, when I, when I hear evals around, you know, what I'm doing, right? Um, so approaching it that way, I think makes it much more approachable. Um, which is, I think, good. And just so I'm super clear, right, and it's probably the dumb question, right, that I'll ask, which is these— what we're talking about now, this is distinct from unit tests and assertions, right?

Yeah, I mean, it's distinct. It's going to be like— so the errors you see will motivate you to write the errors and assertions. Don't write the error assertions because you're trying to eat your vegetables and trying to be— do some kind of thing that I wrote about. You know, when you look at the data, you're going to be— you know, you'll find something like, oh my goodness, I want to— this error, I want to make sure that I know about it. Yeah, you know, kind of like you'll want to write the test. Be like, this, this error keeps me up at night, or like, I don't want users to go through, see this, you know, this is embarrassing, let me write a test. Um, if you do it in advance, you're just going to— is— yeah, you have to go the other way.

Yeah, no, I think that that's helpful. I mean, some— it was also helpful for me to do it this way too, because now I know not to, right?

Uh, yeah, yeah, no, I mean, uh, I think, uh, yeah, I think people— this is not uncommon, like, yeah, that people have skipped a step and have started right doing evals, like, just by writing tests. Yeah, so I also have to maybe write something on this or, you know, let people know. Yeah, I mean, I think that, that really the whole thing, the whole eval thing, it's like the most important thing is the looking at data is like, it's almost like you will, you will understand like Even the eval, the test themselves, is like pales in comparison to what you will learn from looking at some data. It seems like unreasonable, like, oh, it's such a clerical task, or feels like, okay, why am I look— I'm gonna like look at like logs. But yeah, it's like really the highest leverage thing you can do.

Yeah, I mean, eval is just a second order, you know, bit from from it. So yeah, look at the first order. Yeah, yeah, great.

Yeah, let me know how it goes, you know, after you kind of start looking at stuff. Let me know, let me know how you, how you go. Be really interested to, to hear about your progress.

Yeah, really appreciate it. Thank you so much.

Yeah, thank you.