LLM Evals: Common Mistakes

So first of all, just set the stage. You've been writing about evals for more than a few years extensively.

That's a lifetime in LLM land.

And, um, like a lot of your papers, I find that the workflow that you describe is actually pretty innovative and many vendors have not picked it up, but it is the right workflow. And through that, I think. I found at least one person who I felt, okay, this person really knows what they're talking about. I'm talking about you. And I realized people have a lot of misconceptions about evals. So let's get into it. The one that nags us the most is how foundation model benchmarks are so different than domain-specific evals, and people tend to conflate the two. You want to talk about that?

Yeah, it's kind of crazy whenever I talk to people. This is application developers. All right. So it's not people training foundation models. It's like people building chatbots for their internal company documentation or chatbots for their own clients to interact with the data or so forth. And in these foundation model evals are not for you. They, they are for OpenAI's Anthropic's people who are training general language models, general coding assistance, tool use, um, and so forth. And then I like to think of this as like the SATs or the GMAT for the LLM. You're not going to hire someone because they had a great SAT score. Like, it's—

I hope not.

Interesting to see how, if they didn't tank the SAT, sure. But you're going to give them tasks that are specific to your role that you're hiring them for and see how they perform on those tasks. And you can't outsource those tasks, right? That's called evals. That's like figuring out what matters for the role at your company, hiring for that. And it's such a terrible process, sure, upfront, but when you make that investment, then you can reap all the rewards of using LLMs. And I think people, it hasn't hit them yet. And, and partly it's because Twitter and the news are just flooded with all of these foundation model evals. So it's very misleading and there's not that many people talking about what are the crazy, what are the bespoke evals that you need to have and they're really boring, right? Make sure that your summary isn't within a certain length or you have 3 instructions in your prompt. Make sure each instruction is followed even in turn 8 of the chatbot. These are very boring evals, but they make or break business.

Yeah, and I think people have this tendency, they want to just have a play. Tool or plug and play thing to take that same mentality of foundation model evals and say, hey, let's just plug in some evals that someone gives us. So I think the space is pretty flooded with vendors who offer your generic off-the-shelf evals. So things like hallucination score, toxicity score, conciseness score, all that stuff like that. I keep— even me repeating those words feels— I feel like I've said that 100 times. People just want to take that, you and run with it. And, uh, it doesn't work. I think it's very, it's very harmful, uh, in most cases. Um, you can use it in a very skilled way, but most people don't use it in that way, and it, it just bites people.

That's a good starting point, but no more. Like, don't, don't use something that can't pass a really basic eval, but don't rely on that, I guess. Um, this, this is very interesting because it reminds me of another really failure mode around these custom evals is people like to outsource all aspects of labeling, even upfront. Curious if you've seen that.

Yeah. Uh, this one drives me insane. So people think of, there's a lot of companies have siloed technology departments where they outsource all their development. And they've taken that same mentality to building AI. They say, hey, this is a technology thing. We're going to give it to our developers. A lot of times the developers are in a completely different country. Are in different location, very much separated in both time and space from the people who know the business and the domain experts. And it's kind of like, kind of a second-class thing, like this technology development. And what people, the mistake they make is they say, they make the developers write the prompts and they make the developers do the labeling. And that just is a recipe for complete disaster. That doesn't work at all. Because basically you're trying to outsource your taste and your business judgment completely and solely to developers. That's like something you weren't delegating before. People don't realize they're delegating it. They're just kind of packaging it together, lumping it all together with this, oh, this AI is so complicated. It's a developer's job.

And prompts are the moat of your business. The prompts, the label data that you're labeling. If you outsource that, you're, you're outsourcing your moat.

And so I've tried to fight this in many different ways. It's actually been a little bit of a struggle to get people to back off of this behavior because there's a very strong bias towards, hey, just outsource AI. It simply doesn't work in the way that if you try to develop AI like traditional software. One thing that I found that has worked well is these integrated prompting environments where you make sure that the person's application they have an admin mode that allows a domain expert to change the prompt and see what the user sees. But you don't— the idea is that the developer is not the one writing the prompt, it's the domain expert. You just make the— make sure the prompt is extremely accessible. It's almost like one step away from the user being able to edit their own prompt, except you just let someone internally do it. And that tends to remove lots of friction and bring the domain experts in.

Related to that, another useful thing that I actually saw this last week was having the traces made very, very accessible to the domain experts, giving them to 5 domain— not even 5, 3 honestly— domain experts to like label or verify if it's correct or not correct or how they would change it. And then just see how they disagree. It's a very common for even domain experts to disagree on what the right behavior should be and what is correct and what's not correct. And I think oftentimes it just points to ambiguous specifications and prompts. It's just things that result from if you don't really carefully outline what your pipeline is going to do, what your tasks are. If you outsource this too early, you just, you can't make progress basically in resolving any level of ambiguity. You can't send any ambiguity to an LLM. That's rule number one.

And I think, uh, one thing you've— we've talked about a lot is, hey, to make sure that these annotations are being done correctly, there's a lot of upside in building your own annotation tools so that you can make those traces accessible. Um, I think people— because there's a lot of tools out there that will collect your traces and they're good. They're like these observability tools. You can read your traces, but that can still be a little bit complex for a domain expert and building your own. You wanna talk about—

It makes me so happy from that Lightning lesson that we did, um, where we vibe coded the labeling app. And then now there are people on, on X or on Twitter or whatever coming out of the woodwork showing screenshots of their vibe coded app. It makes me so happy. I think interface design. So let me back up actually. What do we really want in these early stages of pipeline development? We want access to like a lot of labeled realistic traces. that we feel confident are accurately labeled, right? They can be 99%, even 95% correctly labeled, but certainly not 50%, which is what you would get from an outsourcer, or 70%. Um, and you want enough to feel like you, like, have a really good picture for what the pipeline is doing. Um, and the way to get there is really to have a good interface where you can elicit feedback from domain experts, from yourself, even from other team members. And these interfaces have to make, it has to be very simple to give feedback, very fast. So thumbs down, ranging from, you know, the feedback that you elicit, like thumbs down to like open-ended comments on failure modes or feedback on just the behavior. That's one aspect of it, just the type of feedback that you elicit. And the another aspect is, you know, how you present the data to the human to make sense of. Humans are very visual people, right? When we're trying to assess the quality of an AI-generated email, we need to see it like an email. It makes it just so much faster to read. If we're trying to assess like a plot that the LLM generates or a chart, we should see the chart, right? We should not see the code that the LLM came up with. So for these two reasons, right, like just the diversity of feedback that you want to elicit, as well as the data type that you're trying to assess and how it needs to be visualized, I think you need— both of these mean that we should to try to build our own labeling interface. And you could argue, oh, it's so much effort to invest in building a custom labeling interface, but we have really good AI right now. You can— we vibecoded in 10 minutes, we had our own custom labeling interface.

I think this is really important. And just from another perspective, everyone's data is so different, and it's really important you render that data in a way that's specific for your exact problem. And you really dial it in and you make it super specific to that and you remove all the friction possible. You have filters that make sense for you. You render it exact same way. Like you said, like, yeah, it's Markdown. You render the Markdown. If it's images, you render the images. Um, you may wanna like have things like collapse certain things or do other kinds of things.

It's just all bespoke.

It's very bespoke and there's just like, AI is really, really good at creating this specific genre of web applications that show data.

Truly.

So I think, I think we're in agreement about the build your own annotation tools.

Definitely. That's not to say that existing observability tools are bad, right? I think they're great, but so first you should definitely try to log all of your data, no questions asked. And a lot of the existing tools are optimized just for everybody, right? So they're going to show you your traces in a table. Sure, you will be able to filter by, say, trace ID. Um, but to get that second layer of, you know, making it very easy to annotate, you would need to build your own thing. You don't have to annotate every single trace, right? You can annotate a sample. You can have domain experts annotate.

And you can build your annotation app on top of these observability tools because essentially they're like databases and you can pull from them. You can annotate and you can push back to them.

Absolutely. I didn't want it to come off as we're like, never use the off-the-shelf.

Definitely.

Yeah. That's not what we're trying to say, but hopefully people understand that. I think that interestingly, it's now made me think about when you are labeling data, I think a lot of times also a common misconception people have is humans are very slow at labeling data. I'm very slow at labeling data. Sure, I can build a great interface, but I'll still be slow. Maybe can I just use an LLM as judge to label data? You look like you're smiling, but I think there's a billion misconceptions we could probably unpack and help people know how to misjudge.

There's a lot of misconceptions about that. The one, my favorite kind of line I like to throw at people is I ask them in a Socratic way, how do you trust your LLM as a judge? And then they look at me and they say, I don't know. And then I ask them, how do you trust anything? And then they say, I don't know. And then I say, well, you have to check it, right? You have to check it. And then from there, we start the conversation of there is no getting out of You have to label some data and then, because people resist it quite heavily, but when you realize it's a matter of it's, that's the only way you can build trust fundamentally is important. And so I think I have found that to be effective. It's a funny conversation, but it works, tends to work.

Building on that, there's a couple of things that I'm going to try this explanation and see if it resonates with you, but two mistakes that I see people make is One, trying to program an LLM to judge the entire trace as is, just be like, is this correct? And I liken that to if you hired an intern and asked them to check the trace, is this correct? It's not likely that they would do significantly, so much better than the original generator. Like maybe it'll catch some mistakes, sure. But there's so much work for them to check. You haven't told them what to check in the trace. So it might not be good. And what works a lot better is if you are very specific about what the judge should check. For example, just check the tone. Have one judge check tone. Have another judge check that it used the tool correctly. Have another judge— think about all of the different failure modes you could have and just program a judge for each failure mode, as you would hire a person to check an individual thing of your output.

So I think that's one of the things we're going to teach also is don't just brainstorm a priori all the possible things that can go wrong, because that's infinite. Like, try to ground that in the errors you actually see.

Yeah. And a judge per error, really. And then the second mistake that I see people make is once they have identified the failure mode, for example, the— an email gener— an email assistant, the tone needs to be a certain way to reflect you know, this, the person that you're sending the email to. If it's a friend, you should be more casual. If it's an exec or a VP, you should be more formal. The tone shouldn't be like, is the tone appropriate? 7 on a, like, or a 1 to 5 scale where 5 is appropriate and 1 is inappropriate. LLM judges just have no idea. Even a human would have no idea what a 5 means. What is a 4? What is a 1? Right? It's going to like come up with its own thing and they're not going to be consistent with each other because it's grading each trace in isolation, that obviously, like, this kind of rating can be unhelpful. Whereas if you take the time to figure out, you know, what is appropriate and what is not appropriate, maybe even segmenting by the type of recipient for the email. So if it's a professional recipient, if it's a friend recipient, family recipient of the email, and have a different LLM as judge for each one of those, and you give examples of acceptable tone and unacceptable tone, and honestly, sometimes just 2 or 3 examples is fine, then you can have a lot more trust in the LLM as judge. It'll be more accurate. It'll literally be grounded in your own labels.

Um, you can study the agreement empirically between yourself and the judge, and you don't have to— it's not a touchy-feely thing. It's a scientific thing. You can say, hey, it agrees with me in this amount. I think people get really bogged down by the idea that, oh, let's, let's, uh, linger on this LM as a judge thing. So one thing that I see everyone doing wrong is assigning Likert scales. Also tell them as a judge.

Likert scale from the 1 to 5.

And it seems like a good idea, but I think we've both found that this tends not to work. Um, you wanna talk about?

Why? I'll start by giving history on the Likert thing. So there is a paper called Judging— I can never remember the full paper title, but Judging LLM as a Judge. Basically, it was from the chatbot arena, folks. It was NeurIPS 2023. Something Judging LLM as a Judge. It's a good paper. They— but the idea that they espouse is— well, first, they do use a Likert score. To rate certain qualities. But in their case, they're also exploring the concept of training LLM judge evaluators. And anytime you are training a model, you have to collect data to train on. And in that sense, right, you are going through this procedure of grounding the judge. Maybe you have examples for 1s, examples for 2s, examples for 3s, and so forth to 5s. And that's good. That's fine. I don't have a problem with that. I think what I have the problem with is no concept of grounding. You're not fine-tuning your LLM judges, you're using off-the-shelf, which is fine. Um, but if you're trying to use those on a 1 to 5 scale and you haven't grounded it in any way, you don't even have examples in the prompt for what's a 1 and what's a 4 or whatever, then there's no way that it's going to be aligned with you. It has to read your mind. It can't do that. Um, the other thing is having examples for every one of those 5 points is very time-consuming. If you just have binary evaluators, you only have to have pass and fail examples, much easier.

And the argument that resonates with me most for having binary scores instead of 1 to 5 is that in a business concept, in a business sort of setting, you will have to likely make a go or no-go kind of judgment at some point. And you should force yourself to encapsulate that in the score.

Yep, exactly. A lot of times I will, I've literally seen this in production where it's like score greater than or equal to 3 means send an alert. So you, you could have saved yourself all of this hassle and just trained or figured out like a binary LLM as judge yourself.

The next thing that I think is worth talking about, and we're going to discuss all of this in a lot of detail in our course. One thing that people get hung up on also. In addition to all the things we described is when I tell them to label data, they kind of break out into hives and they get really nervous because they think it's like me telling someone, okay, go to the beach and tell me to start picking up one grain of sand at a time. And they feel immediately like this will never be fruitful. And I think the disconnect is as machine learning engineers, we know We have all lived and discovered this experience of just looking at some data, even if you have millions of data points. For some, we all know that by looking at data, even 100 examples, you just learn so much. And we've been through that. The first time we did it, we're like, what, why are we doing this? Because someone told us to, my mentor told me to, or we heard that this is a good idea. The very first time we do it, we're like, wow. This, I'm gonna do this every single time. This is where all the value is right here. This is, no one talks about it. It seems it's the most unglamorous part of the whole thing, but it—

I think it's, it's cuz it's not sold well, right? If you want it, people don't realize that we're not asking you to look at every grain of sand, every data point. We're asking you to go to a point. Where there's significantly diminishing returns. And often that actually comes pretty fast. You look at enough data, you've seen all of the failure modes under the sun for your prompt, and it's done. You don't have to keep looking at more, right? There's diminishing returns. There's a phenomenon in qualitative research called theoretical saturation, where you could imagine as you're looking at all of your data points, you're kind of aggregating in your mind, like, ah, I learned this thing. Here's a new failure mode. I've uncovered all of them. And at some point you saturate. Your set of observations does not grow as you look at more data. And that's when you're done. You don't have to keep looking at data anymore. And people don't realize how quickly this happens, right? Sometimes even 30 data points, you stop learning more. People think that, oh, they're building a chatbot and they can take in open-ended natural language queries from their customers and pretty much anything could happen. Well, Turns out I've never seen a chatbot that has more than 10 types of queries. When you really boils down to it, people have lookup queries, people have a general knowledge question-answer queries, people have like navigational queries. They want to go to a specific page in your product. There's not that many, right?

I agree.

And I think you're not going to uncover this whole vast world. You won't be labeling forever, I guess is my point.

And. I think that it's not just about you're not learning anything new. Usually after just a handful of examples, even 30, you have a lot of actionable things that you can go do right now to make your application better. And even if you stop and come back later, keep looking, like that first X amount of examples you look at are really valuable. And then we'll talk, we'll be talking about in our course how to do that. There's a systematic way of how to do that. And how to do that very effectively. And the name that we, in, in machine learning for that is error analysis. Um, I think there's some other names that I recently learned about. I think like coding is one.

From coding is what the quality social scientists call it, but they do the same thing, right? Their job is actually the same job as the eval people, which is like they've collected a lot of survey responses in the field and now they have to like write a paper on it.

And I think the other thing that is not intuitive is, okay, you look at, so okay, fine, look at data. I can look at 30 examples. Everyone listening to this is like, great, I'll look at 30 examples. Now what? And the next thing that's not intuitive is like, so how do you look at it? And there's simple techniques like taking notes in the right way when you look at that data and you can take notes in a way that allows you to actually aggregate those insights, and that's easier than ever with LLMs. You can use LLMs in the loop to help you—

Yeah.

Figure out what the patterns are in your own notes. So that was like a little bit more tedious before.

Totally. How do you merge like insights across different traces? People, that's a skill that like you take a class to learn as a social scientist. But now you can dump all of it in an LLM and it'll tell you, here are your top 3 failure modes. And here's how each trace is assigned to it. Like, it's kind of crazy.

Let's talk a little bit about offline eval sets and what uses are and what their limitations are and where you think people go wrong.

Where, where should I start? I think, okay, so maybe this is also part of the MLOps history, but typically when you're training machine learning models, right, you collect the label dataset as we've been talking about, you split it into train, dev, or val or whatever one of the people call that, and test sets, so 3 different splits. You train your model in the train set, you do any hyperparameter tuning and adjusting to observe performance on the validation set, and then you have— the point is here that you're trying not to overfit your meta process of hyperparameter tuning. If you try to tune hyperparameters just to a specific validation set and your test set doesn't match that distribution, then when you deploy to production, your parameters might might not work, or you might experience a performance drop. And I think—

so concretely, let's say you are sort of tuned— in our course, we'll talk a little bit about prompt tuning. There's different kinds of algorithmic prompt tuning. One of them is sampling examples that you can use in your prompt, and a lot of those techniques use some kind of hill climbing to sort of maximize or minimize a metric of your choice. But if you see that your validation score is a one quantity and your holdout or your test set is a completely different quantity, then you're obviously overfitting. And for some, and I've seen that quite a lot in LLM land. Machine learning people know better because that's kind of beat into you from 101. When you first start doing machine learning, they're like, no, test train, test validation training set. You split it and they kind of beat it into you. Please don't overfit.

So maybe let's talk about why it's so odd phenomenologically here in, in LLMs. So first thing I think is, um, there's just not enough labeled data. In, in traditional machine learning, we like had the benefit of having lots of data to be able to draw any sort of statistical conclusion, right? There's so many textbooks that tell us like minimum 10K points to like fit your random forest model. Today people are working with 25K. 30 labeled traces and then they, they split that into 80% train, 20% test. Forget this holdout set business. So if your test set is 6 examples, I don't know, in my mind, can't really draw any conclusions from this, right? So what if you get 100? Some people, so what if you get 83%? You, you missed one. I don't know. I can't, can't tell you anything. I got such a big performance boost because I got one more example correctly. I don't know. So I think there's, there's two takeaways you can have from this. Like one is maybe you just have more labeled data? Sure, fine. You could try. I think it would be very hard because the scope of the pipeline is now larger, right? You could take in any sort of natural language input, right? So it's, it's not like traditional machine learning where you have feature vectors that you know what they represent or mean. Now anything kind of holds. But the other thing to keep in mind is you shouldn't rely on these validation accuracies or test set accuracies. To tell you exactly what the pipeline performance is. You have to do online monitoring. Otherwise, I see so many blog posts of like, I use such and such prompt optimization tool and now I got 87%. And it's like, is that what your online performance is? I have no idea. How big is your validation set? I feel like a lot of this becomes very murky and also very misleading for people. And it's not just like random developers who put out these blog posts, right? AWS, I was reading blog posts from big companies. Last week where it was like, how big was your validation set? Are you sure you don't have any leakage? You're not monitoring.

And sometimes the machine learning people that publish these things of, oh, I, I just overfit to the validation set and now I have a celebration of sorts.

Yeah. And it's, I think it's just want to stress how easy it is to overfit to validation in LLMs because there's only 5, 6 data points. If you have some rephrasing of that in your training set, You're virtually guaranteed that, right? So, so I really tell people you have to monitor your performance online. Don't use holdout sets to tell you what your pipeline performance is. And the other thing is a lot of times these offline validation sets are hand curated, right? You are man— you are putting in error modes that occur. You're finding edge cases. It's more of a robustness set. It's not a random sample of your production traffic.

Which is reasonable, actually.

Which is good, right? It's just like unit tests. Tests in CI, right? You design your unit tests to be 100% passing and prevent against regressions in the future. But certainly 100% unit test passing doesn't mean you have great latencies and perfect online performance, right? It's night and day, two different things.

And we're gonna tease apart all of these things in the course. Hey, how do you construct your test set exactly for like these unit tests, but then also how do you estimate your actual error and what the difference between those are and how do you go about it. Um, Trehya said a lot of times you want your hard cases, your failures in your test set, and you almost want to anchor on them, especially in the beginning because you're trying to make progress. But there's ways to go about it. Um, there's a lot of nuances. So the course is exciting. Um, there's going to be pretty robust materials there. I think it's going to be the only course of its kind that I know of, at least at this point, going into domain-specific evals, application-specific evals, which is really exciting. There's a lot of stuff out there about foundation model evals and training on that. So I'm really excited.

I hope everyone takes the course.

See up here somewhere in the YouTube, there'll be a link and go to that link. We're going to provide a 35% off discount code. Here in the YouTube. So just find it, you'll see it on the screen, and you can all see it in the show notes.