LLM Evals: Common Mistakes

So first of all, just set the stage. You've been writing about evals for more than a few years, extensively.

That's a lifetime in LLM land.

And like a lot of your papers, I find that the workflow that you describe is actually pretty innovative and many vendors have not picked it up, but it is the right workflow. And through that, I think I found at least one person who I felt, okay, this person really knows what they're talking about. I'm talking about you. And I realized people have a lot of misconceptions about evals. So let's get into it. The one that nags us the most is how foundation model benchmarks are so different than domain specific evals, and people tend to conflate the two. You want to talk about that?

Yeah, it's kind of crazy. Whenever I talk to people, this is application developers. All right, so it's not people training foundation models. It's like people building chatbots for their internal company documentation or chatbots for their own clients to interact with the data or so forth. And in these foundation model evals are not for you. They, they are for OpenAI's anthropics, people who are training general language models, general coding assistance, tool use, and so forth. And then I like to think of this as like the SATs or the GMAT for the LLM. You're not going to hire someone because they had a great SAT score. Like, it's, I hope not interesting to see how if they didn't tank the sat. Sure. But you're going to give them tasks that are specific to your role that, that you're hiring them for and see how they perform on those tasks. And you can't outsource those tasks. Right. That's called evals. That's like figuring out what matters for the role at your company, hiring for that. And it's such a terrible process. Sure. Up front. But when you make that investment, then you can reap all the rewards of using LLMs. And I think people. It hasn't hit them yet. And, and partly it's because Twitter and the news are just flooded with all of these foundation model evals. So it's very misleading. And there's not that many people talking about what are the crazy, what are the bespoke evals that you need to have? And they're really boring. Right. Make sure that your summary isn't within a certain length or you have three instructions in your prompt. Make sure each instruction is followed even in turn. Eight of the chatbot. These are very Boring evals, but they make or break business.

Yeah. And I think people have this tendency, they want to just have a play tool or plug and play thing to take that same mentality of foundation model evals and say, hey, let's just plug in some evals that someone gives us. So I think the space is pretty flooded with vendors who offer your generic off the shelf evals. So things like hallucination score, toxicity score, conciseness score, all that stuff like I keep even me repeating those words feels, I feel like I've said that a hundred times. People just want to take that and run with it and it doesn't work. I think it's very, it's very harmful. In most cases you can use it in a very skilled way, but most people don't use it in that way and it, it just bites people.

Good starting point. But no more. Right? Don't, don't use something that can't pass a really basic eval, but don't rely on that. I guess this, this is very interesting because it's reminds me of another really big failure mode around these custom evals is people like to outsource all aspects of labeling, even upfront. Curious if you've seen that.

Yeah, this one drives me insane. So people think of, there's a lot of companies have siloed technology departments where they outsource all their development and they've taken that same mentality to building AI. They say, hey, this is a technology thing, we're going to give it to our developers. A lot of times the developers in a completely different country are in a different location, very much separated in both time and space from the people who know the business and the domain experts. And it's kind of like kind of a second class thing like this technology development. And what people, the mistake they make is they say they make the developers write the prompts and they make the developers do the labeling. And that just is a recipe for complete disaster. That doesn't work at all because basically you're trying to outsource your taste in your business judgment completely and solely to developers. That's like something you weren't delegating before. People don't realize they're delegating it. They're just kind of packaging it together, lumping it all together with this, oh, this AI is so complicated. It's a developer's job.

And prompts are the moat of your business. The prompts, the label data that you're labeling. If you outsource that, you're, you're outsourcing your moat.

And so I've tried to fight this in many different ways. It's actually been a little bit of a struggle to get people to back off of this behavior because there's a very strong bias towards, hey, just outsource AI. It simply doesn't work in the way that if you try to develop AI like traditional software. One thing that I found that has worked well is these integrated prompting environments where you make sure that the person's application, they have an admin mode that allows a domain expert to change the prompt and see what the user sees. The idea is that the developer is not the one writing the prompt, it's the domain expert. You just make the make sure the prompt is extremely accessible. It's almost like one step away from the user being able to edit their own prompt, except you just let someone internally do it. And that tends to remove lots of friction and bring the domain experts in.

Related to that, another useful thing that I actually saw this last week was having the traces made very, very accessible to the domain experts, giving them to five domain, not even five, three. Honestly, domain experts to like label or verify if it's correct or not corre or how they would change it and then just see how they disagree. It's very common for even domain experts to disagree on what the right behavior should be and what is correct and what's not correct. And I think oftentimes it just points to ambiguous specifications and prompts. It's just things that result from. If you don't really carefully outline what your pipeline is going to do, what your tasks are. If you outsource this too early, you just, you can't make progress basically in resolving any level of ambiguity. You can't send any ambiguity to an LLM. That's rule number one.

And I think one thing you've, we've talked about a lot is hey, to make sure that these annotations are being done correctly, there's a lot of upside in building your own annotation tools so you can make those traces accessible. Um, I think people, because there's a lot of tools out there that will collect your traces and they're good. They're like these observability tools. You can read your traces, but that can still be a little bit complex for a domain expert and building your own. You want to talk about it makes.

Me so happy from that lightning lesson that we did where we Vibe coded the labeling app and then now there are people on X or on Twitter or whatever coming out of the woodwork showing Screenshots of their vibe coded app makes me so happy. I think interface design. So let me back up actually, what do we really want in these early stages of pipeline development? We want access to a lot of labeled realistic traces that we feel confident are accurately labeled. They can be 9%, even 95% correctly labeled, but certainly not 50% which is what you would get from outsource or 70%. And you want enough to feel like you have a really good picture for what the pipeline is doing. And the way to get there is really to have a good interface where you can elicit feedback from domain experts, from yourself, even from other teams members. And these interfaces have to make. It has to be very simple to give feedback very fast. So thumbs down. Ranging from the feedback that you elicit, like thumbs down, to like open ended comments on failure modes or feedback on just the behavior. That's one aspect of it, just the type of feedback that you elicit. And the another aspect is, you know, how you present the data to the human to make sense of. Humans are very visual people. Right. When we're trying to assess the quality of an AI generated email, we need to see it like an email. It makes it just so much faster to read. If we're trying to assess a plot that the LLM generates or a chart, we should see the chart. We should not see the code that the LLM came up with. So for these two reasons, just the diversity of feedback that you want to elicit as well as the data type that you're trying to assess and how it needs to be visualized. I think both of these mean that we should try to build our own labeling interface. Um, and you could argue so much effort to invest in building custom labeling interface, but we have really good AI right now. You can we vibe coded in 10 minutes. We had our own custom labeling interface.

I think this is really important. And just from another perspective, everyone's data is so different and it's really important. You render that data in a way that's specific for your exact problem and you really dial it in. You make it super specific to that and you remove all the friction possible. You have filters that make sense for you. You render it exact same way like you said. Yeah, it's markdown. You render the markdown. If it's images, you render the images. You may want to like have things like collapse certain things or do other kinds of things.

It's just all bespoke.

It's very bespoke. And this is like AI is really, really good at creating this specific genre of web applications that show data.

Truly.

So think. I think we're in agreement about the build your own annotation tools.

Definitely. That's not to say that existing observability tools are bad. I think they're great. But so first you should definitely try to log all of your data, no questions asked. And a lot of the existing tools are optimized just for everybody. Right. So they're going to show you your traces in a table. Sure. You will be able to filter by say, trace id, but to get the second layer of trace, you know, making it very easy to annotate, you need to build your own thing. You don't have to annotate every single trace. Right. You can annotate a sample, you can have domain experts annotate, and you can.

Build your annotation app on top of these observability tools because essentially they're like databases and you can pull from them, you can annotate and you can push back to them.

Absolutely. I didn't want it to come off as we're never used the off the shelf.

Definitely. Yes.

That's not what we're trying to say, but hopefully people understand that. I think that interestingly, it's now made me think about when you are labeling data. I think a lot of times also a common misconception people have is humans are very slow at labeling data. I'm very slow at labeling data. Sure, it can build a great interface, but I'll still be slow. Maybe can I just use an LLMs judge to label data? You look like you're smiling, but I think there's a billion misconceptions we could probably unpack and how many.

There's a lot of misconceptions about that. The one my favorite kind of line I like to throw at people is I ask them in a Socratic way, how do you trust your LMS and judge? And then they look at me and they say, I don't know. And then I ask them, how do you trust anything? And then they say, I don't know. And then I say, well, you have to check it. Right? You have to check it. And then from there we start the conversation of there is no getting out of you have to label some data. And then because people resist it quite heavily. But when you realize it's a matter of it's. That's the only way you can build trust fundamentally is important. And so I think I have found that to be effective. It's a funny conversation, but it works.

Tends to work Building on that, there's a couple of things that I'm going to try this explanation and see if it resonates with you. But two mistakes that I see people make is one, trying to program an LLM to judge the entire trace as is just be like, is this correct? And I liken that to if you hired an intern and asked them to check the trace, is this correct? It's not likely that they would do significant so much better than the original generator. Like maybe it'll catch some mistakes. Sure. But there's so much work for them to check. You haven't told them what to check in the trace, so it might not be good. And what works a lot better is if you are very specific about what the judge should check. For example, just check the tone, have one judge check tone, have another judge check that it used the tool correctly. Have another jud think about all of the different failure modes you could have and just program a judge for each failure mode as you would hire a person to check an individual thing of your output. So I think one of the things.

We'Re going to teach also is don't just brainstorm a priority all the possible things that can go wrong because that's infinite. Like try to ground that in the errors you actually see.

Yeah. And judge per error. Really. And then the second mistake that I see people make is once they have identified the failure mode, for example, the an email generation, an email assistant, the tone needs to be a certain way to reflect. Know the person that you're sending the email to. If it's a friend, you should be more casual. If it's an exact or a vp, you should be more formal. The tone shouldn't be like. Is the tone appropriate? Seven on a like or a one to five scale where five is appropriate and one is inappropriate. LLM judges just have no idea. Even a human would have no idea what a five means. What is a four? What is a one? Right. It's going to like come up with its own thing and they're not going to be consistent with each other because it's grading each trace in isolation that obviously this kind of rating can be unhelpful. Whereas if you take the time to figure out what is appropriate and what is not appropriate, maybe even segmenting by the type of recipient for the email. So if it's a professional recipient, if it's a friend, recipient, family, recipient of the email, and have a different LLMs judge for each one of those and you give examples of acceptable tone and unacceptable tone and honestly Sometimes just two or three examples, a sign. Then you can have a lot more trust in the LLMs judge. It'll be more accurate. It'll literally be grounded in your own labels.

You can study the agreement empirically between yourself and the judge and you don't have to. It's not a touchy feely thing. It's a scientific thing. You can say, hey, agrees with me in this amount. I think people get really bogged down by the idea that oh, let's, let's linger on this element as a judge thing. So one thing that I see everyone doing wrong is sending like art scales. Also tell them as a judge, like.

Our scale from the 1 to 5.

And it seems like a good idea. But I think we've both found that this tends not to work. You want to talk about why?

I'll start by giving history on the current thing. So there's a paper called Judging. I can never remember the full paper title, but judging LLM as a judge basically was from the chatbot arena, folks. It was NeurIPS 20, 23 something. Judging LLM as a judge. It's a good paper. They. But the idea that they espouse is. Well, first they do use a Likert score to rate certain qualities. But in their case, they're also exploring the concept of training LLM judge evaluators. And anytime you are training a model, you have to collect data to train on. And in that sense, right, you are going through this procedure of grounding the judge. Maybe you have examples for ones, examples for two, examples for threes, and so forth to fives, and that's good, that's fine. I don't have a problem with that. I think what I have the problem with is no concept of grounding. You're not fine tuning your LLM judges, you're using off the shelf, which is fine. But if you're trying to use those on a one to five scale and you haven't grounded it in any way, you don't even have examples in the prompt for what's a one and what's a four or whatever, then there's no way that it's going to be aligned with you. It's has to read your mind. It can't do that. The other thing is having examples for every one of those five points is very time consuming. If you just have binary evaluators, you only have to have pass and fail examples much easier.

And the argument that resonates with me most for having binary scores instead of 1 to 5 is that in a business concept in a business sort of setting, you will have to likely make a or no go kind of judgment at some point and you should force yourself to encapsulate that in the score.

Yep, exactly. A lot of times I will. I've literally seen this in production where it's score greater than or equal to three being send an alert. So you could have saved yourself all of this hassle and just trained or figured out like a binary LLMs judge yourself.

The next thing that I think is worth talking about and we're going to discuss all of this in a lot of detail in our course. One thing that people get hung up on also, in addition to all the things we described, is when I tell them to label data, they kind of break out into hives and they get really nervous because they think it's like me telling someone, okay, go to the beach and tell me. Just start picking up one grain of sand at a time and they feel immediately like this will never be fruitful. I think the disconnect is as machine learning engineers, we know, we have all lived and discovered this experience of just looking at some data. Even if you have millions of data points for some, we all know that looking at data, even 100 examples, you just learn so much. And we've been through that. The first time we did it, we're like, what? Why we're doing this because someone told us to. My mentor told me to. Or we heard that this is a good idea. Very first time we do it, we're like, wow, this I'm going to do this every single time. This is where all the value is, right here. This is. No one talks about it. It seems it's the most unglamorous part of the whole thing.

But I think it's. It's because it's not sold well, right? If you want it. People don't realize that we're not asking you to look at every grain of sand, every data point. We're asking you to go to a point where there's significantly diminishing returns. And often that actually comes pretty fast. You look at enough data, you've seen all of the failure modes under the sun for your prompt, and it's done. You don't have to keep looking at more. Right. There's diminishing returns. There's a phenomenon in qualitative research called theoretical saturation, where you could imagine as you're looking at all of your data points, you're kind of aggregating in your mind like, I learned this thing. Here's a new Failure mode. I've uncovered all of them and, and at some point you saturate your set of observations does not grow as you look at more data. And that's when you're done. You don't have to keep looking at data anymore. And people don't realize how quickly this happens. Sometimes even 30 data points, you stop learning more. People think that, oh, they're building a chatbot and they can take in open ended natural language queries from their customers and pretty much anything could happen. Well, turns out I've never seen a chatbot that has more than 10 types of queries. When you really boils down to it, people have lookup queries, people have a general knowledge question answer queries, people have like navigational queries. They want to go to a specific page in your product. There's not that many.

Right, I agree.

And I think you're not going to uncover this whole vast world. You won't be labeling forever, I guess is my point.

And I think that it's not just about you're not learning anything new. Usually after just a handful of examples, even 30, you have a lot of actionable things that you can go do right now to make your application better. And even if you stop and come back later, keep looking like that first x amount of examples you look at are really valuable. And then we'll talk, we'll be talking about in our course how to do that. There's a systematic way of how to do that and how to do that very effectively. And the name that we in machine learning for that is error analysis. I think there's some other names that I recently learned about. I think like coding is one from.

Coding is what the quality the social scientists call it. But they do the same thing, right? Their job is actually the same job as the eval people, which is like they've collected a lot of survey responses in the field and now they have to write a paper on it.

And I think the other thing that is not intuitive is okay, you look at so okay, fine, look at data. I can look at 30 examples. Everyone listening to this, like, great, I'll look at their examples. Now what? And the next thing that's not intuitive is like, so how do you look at it? And there's simple techniques like taking notes in the right way when you look at that data and you can take notes in a way that allows you to actually aggregate those insights and that's easier than ever. With LLMs, you can use LLMs in the loop to help you figure out what the patterns are in your own notes. So that was like a little bit more tedious before.

Totally. How do you merge like insights across different traces, people? That's a skill that like you take a class to learn. It was a social scientist, but now you can dump all of it in LLM and it'll tell you. Here are your top three failure modes and here's how each trace is assigned to it. Like, it's kind of crazy.

Let's talk a little bit about offline eval sets and what uses are and what their limitations are and where you think people go wrong.

Where should I start? I think, okay, so maybe this is also part of the mlops history, but typically when you're training machine learning models, right, you collect the label data set, as we've been talking about. You split it into train devices, or VAL or whatever one of people call that, and test sets, the three different splits. You train your model on the train set. You do any hyperparameter tuning and adjusting to observe performance on the validation set. And then you have. The point is here that you're trying not to overfit your meta process of hyperparameter tuning. If you try to tune hyperparameters just to a specific validation set and you're test set doesn't match that distribution, then when you deploy to production, your parameters might not work or you might experience a performance drop. And I think so.

So concretely, let's say you are sort of tuned into a course. We'll talk a little bit about prompt tuning. There's different kinds of prop. Algorithmic prompt tuning. One of them is sampling examples that you can use in your prompt. And a lot of those techniques use some kind of hill climbing to sort of maximize or minimize the metric of your choice. But if you see that your Validation score is a 1 quantity in your holdout, or your test set is completely different quantity and you're obviously overfitting. And for some, and I've seen that quite a lot in LLM land machine learning, people know better because that's kind of beat into you from 101. When you first start doing machine learning, they're like, no, yeah, test validation, training set, you split it and they kind of beat it into you. Please don't overfit.

So maybe let's talk about why it's so odd phenomenologically here in in LLMs. So first thing I think is there's just not enough label data. In traditional machine learning. We like have the benefit of having lots of data to be able to draw Any sort of statistical conclusion. Right. There's so many textbooks that tell us like minimum 10k points to like fit your random forest model. Today people are working with 25, 30 labeled traces and then they, they split that into 80% train, 20% test. Forget this hold out set business. So if your test set is six examples, I don't know in my mind can't really draw any conclusions from this. Right. So what if you get a hundred some people? What if you get 83%? You, you missed one. I don't know. I can't tell you anything. I got such a big performance boost because I got one more example correctly. I don't know. So I think there's two takeaways you can have from this. One is maybe you just have more labeled data. Sure, fine. You could try. I think it would be very hard because the scope of the pipeline is now larger. Right. You could take in any sort of natural language input. Right. So it's not like traditional machine learning where you have feature vectors that you know what they represent or mean. Now anything kind of holds. But the other thing to keep in mind is you shouldn't rely on these validation accuracies or test set accuracies to tell you exactly what the pipeline performance is. You have to do online monitoring. Otherwise I see so many blog posts of I use such and such prompt optimization tool and Now I got 87% and it's like, is that what your online performance is? I have no idea. How big is your validation set? I feel like a lot of this becomes very murky and also very misleading for people. And, and it's not just like random developers who put out these blog posts, right. WS I was reading a blog post from big companies last week where it was like, how big was your validation set? Are you sure you don't have any leakage? You're not monitoring?

Sometimes machine learning people that publish these things of, oh, I, I just overfit to the validation set and now I have a celebration of sorts.

Yeah. And it's, I think it's just want to stress how easy it is to overfit to validation and LLMs because there's only 5, 6 data points. If you have some rephrasing of that in your training set, you're virtually guaranteed that. Right. So as I really tell people, you have to monitor your performance online. Don't use holdout sets to tell you what your pipeline performance is. And the other thing is a lot of times these offline validation sets are hand curated. Right? You are, you are Putting in error modes that occur, you're finding edge cases. It's more of a robustness set. It's not a random sample of your production traffic.

Which is reasonable, actually, which is good.

Right? It's just like unit tests in CI, right? You design your unit test to be 100% passing and prevent against regressions in the Future. But certainly 100% unit test passing doesn't mean you have great latencies and perfect online performance. Right? It's night and day, two different things.

And we're going to tease apart all of these things in the course. Hey, how do you construct your test set exactly for like these unit tests? But then also how do you estimate your actual error and what the difference between those are and how do you go about it? Treya said, a lot of times you want your hard cases, your failures in your test set and you almost want to anchor on them, especially in the beginning because you try and make progress. But there's ways to go about it. There's a lot of nuances. So the course is exciting. There's going to be pretty robust materials there. I think it's going to be the only course of its kind that I know of, at least at this point. Going into domain specific evals, application specific evals, which is really exciting. There's a lot of stuff out there about foundation model evals and training on that. So I'm really excited.

I hope everyone takes the course.

See, up here, somewhere in the YouTube there'll be a link and go to that link. We're going to provide a 35 off discount code here in the YouTube. So just find it, you'll see it on the screen and you can all see it in the show notes.