LLM Eval Office Hours #4: Taming Complexity by Scoping LLM Evals

So I work at Sunday, which is a lawn and garden startup. We sell lawn and garden products and plans that are on a subscription service. And it uses tech and e-commerce to customize these to the unique needs of your lawn. And I'll share, I wanna share my screen here.

Yeah, I'm a customer, by the way.

Thanks for saying that. Yeah, I actually, I took a look at your, some of your account and I have some questions for you about your lawn, but I'll start with this.

Yeah, I don't even know what I'm doing. I just buy stuff and apply it.

Can I ask you one question?

Yeah.

I saw that you bought Bermuda grass seed. Why did you buy that grass seed?

I don't have a good answer. I think like I have some weird places in my house. There's like in, I have a backyard that is north-facing, that's shaded like 100%. And then like my front lawn is south-facing. So maybe it's some, one of those.

I think there are better grass seeds for that, that type of setting. Yeah, you should buy our seed called Shade Select. I'll follow up with the email after this.

Oh, cool. Okay.

Yeah. Yeah. For your location. But can you see this is our website?

Yeah, I can see it. Yeah.

On our website, we have a yard assistant. It's a chatbot situation. It looks like this. So meet your yard expert, Sonny. And there's some prompts to get started here. What can you tell me about my yard? Send that. And then it has some, it has some information about this user's yard location. Your cool season grass is in your lawn size of 2,700 square feet, plant hardiness zone, and a little bit more about your lawn and your soil test. So this has been out for this fall, like 3 months now, and we are just putting together a very holistic evals, I guess, approach. So we did topic modeling and there's probably, there's a lot of surface area here. I think there's probably 40 distinct topics and it's yard guidance, subscription questions, or how to use Sunday product type questions. And we're trying to use real user data as our eval questions or dataset. And I've gotten to the point, I went through your blog post, I had that open for 3 days and you know, in a tab on how to do the evals, write critique, the critiques. And then what we're trying to do now is use this to set up the LLM as a judge and allow us to scale these evals and go a lot faster. And this is where I'm having trouble. This is what I've been doing. So I, I really followed your blog, so it made like a Python Shiny app. And then this is input, the output, and I can write if this is a pass or fail, the critique, and then some, any other issues or the error analysis stuff. And then what I get to is I will have, I have something like this. So I have my, this is like a, just a, the topic area, airification, discounts, shipping. Moving. This was the input query. This is the output from our LLM app. Score, pass or fail. Here's my written critique. And then some of the notes here are optional. But I can't get these critiques to, I guess, work. I can only get up to the 80% or so alignment between my critiques and using LLM as a judge. And I think my problem might be the prompting, the prompt that I'm using for this. It's—

Let me stop you there for one second.

Yeah.

Before you started this process was, because 80% sounds like it kind of good.

Okay. That's nice to hear, but I think, okay, if 1 in 5 are wrong, that's a lot of wrongness about, I worry about just letting that run in an automated way or end it. Okay.

What was it before you started? What was the ba— what's the baseline of—

Oh, probably like 50 or 60%.

Okay. This is a huge improvement.

Oh, okay. I mean, well, that makes me feel better, but it's also very inconsistent. To. Sometimes it will pass and sometimes it will fail based on my prompt and my critiques that I have listed. And I think that's because our questions are very open-ended. It'll be like, what, what can I do for dandelions? There's many possible answers that would be acceptable there. And so it's hard to have them all articulated in the critique. And I just have to keep watching it and keep adding and adding to the critique. And if that's what it is, then that's what it takes.

No, I mean, um, Yeah. Okay. So you've already made like significant progress. It sounds like you, you have something like LLM as a judge is not perfect and it's hard to sometimes get it to be really high alignment, especially if your surface area is large.

Yeah.

And if you have a lot of different open-ended questions, it's very difficult. And proxy in a way. Of as a judge, it's going to be really hard to— in this situation, it might be really hard to get it, you know, the alignment like super high because you have like just insane surface area. You can ask anything about your lawn.

Yeah.

And you're expecting it to give you a coherent answer about anything someone throws at it, which is— and you're doing like, you're doing everything very you're doing it correctly. You're even like segmenting it by topic. I can see that. And you're trying to analyze what's happening. Have you, do you notice, is there er— like errors in certain topics more than others or anything?

Yes, there is. For example, we do really good at answering like er— questions about airification because we have information about that in our knowledge base. So it's retrieved and it does a really good job answering anything about that. But we are having a hard time with like next shipment questions right now because we're in 2024 and the subscriptions are mostly done. So the next shipment is until spring 2025. So it's the way our shipping data is formatted is not easy to articulate to the LLM. So that's an example of, yeah, there are differences.

Okay. One thing you could do is, do you, is your prompt, so are you using a, is, okay. I think from talking to you a little bit before, like you have some, you're doing some RAG in here with the examples that you're putting in the prompt. Are those static or are those like dynamic based on the topic or what?

Right now they are static. That's another area of work we're undergoing right now is how to basically do topic classification and have different routes or workflows for different types of topics where we could be a little more custom with the prompts and, and break down the tasks to be be more specific, but right now it's all one way.

You could try it like on the things that, like, next shipment, for example, since it's like the highest, has the highest error. You could try having a classification that routes that, or like that changes the prompt if detects that, okay, we're talking about next shipment, and brings like that information, that kind of information into the prompt.

That, okay, so that I think is right. That's just logically the next move that you would take on this. What I'm trying to get the eval set up so I could run them quickly and not have me have to read every question and score it. So before we make that change, I want us to be able to establish here's our baseline. We can run it quickly, then go and start to make some of those changes to our system and be able to rerun evals and monitor progress. Does that sound right? Does that, or does that not seem possible?

No, it's a really good question. It's tricky because you, okay, so the question is like, how good is enough in evals? And sometimes it sounds like you're doing a lot already and you're hitting a plateau and I don't see, you're doing more than 99% of people are doing. Like most times I have these office hours and. No one's even looked at the data yet. So you're like way beyond like most people. And so like you have to balance it because you can't just put, you can't just put all your effort into evals either. Even though that's all I talk about all day, that's not really what you want to do. You have to see sometimes you already know from all this work what's wrong. Yeah, it's likely like that. Yeah, you can go into evals, but like it's supposed to be a proxy of, and even though it has 1 in 5 are completely off and I know it's like inconsistent, you can iterate it on it over time, but you should like go back and forth a little bit and make your product better. Go back, tweak your evals, make your product better. It's like a process. You don't wanna just stuck too much in evals all the time. I know it's a little bit counterintuitive just because and see, okay, is it a good proxy or not? If you try to improve next shipment, does it actually, can you see a difference in your evals? Maybe you do, even though it's noisy, maybe you see some signal from there.

Okay. That's what I would, I can, when I, as I've gone through this, I can see, oh, this is very obvious, like things that we need to, to change and make it better. And I'm pretty confident will work, but that's a hypothesis and it would be an experiment. And I feel like in order to get the engineering work done for some of these experiments, I do have to articulate this is what the value of this would be. So that's something we're going through with the group now too, is just the idea of this is applied research, iterative, experimental, and not just build and release features and expect a certain level of performance.

Yeah. Yeah. You could have specialist judges as well, but you have a lot of topics.

Yeah.

How many topics do you say there were?

Well, about 40. And can I ask this too? So I was using, I was basically using a template like this for my LLM as a judge. So this is from the Braintrust AutoEval's library. So I was using something like this. And I also thought too, oh, should I be using more customized prompts for the LLM as a judge? Not just using the same, like the same prompt for every question or every topic.

Can you say that again? So, okay. Okay. So your critiques should be shaping the, like what the, what is in the prompt? Like, not exactly. To some extent you want to see if, so one trick I use and I, I put in the blog post is I get my judge to critique before having a judgment. But then also, if you want to use those critiques in other ways, you want to see, okay, can you use that to come up with some kind of guideline that you can give the judge as well? If there's something that you uncover in those critiques, okay, you should be doing XYZ, you should be telling the model, okay, you need to consider XYZ as a general rule when evaluating this output. Does that make sense?

Yes. And I think that when I use this type of a prompt, it's here's the input, here's the output, here's the criteria. Does submission meet the criterion? That this might be too, it's called a close QA. So I can say here's acceptable responses, but if it happens to have a new response to the question, it, it will grade it wrong or, or fail when it could be acceptable. You know what I mean?

What do you mean by new response to the question?

When I run the evals, I'll be using the same query, but it might generate a new response that I haven't written a critique for yet. Like if it's like how to control dandelions, it might come up with some new way to say how to control dandelions that is an acceptable response, but I haven't seen it yet and I don't have a specific critique for this is this is bad.

Okay, so correct me if I'm misunderstanding, but when you write, you shouldn't have a critique for every possible thing the language model might say.

So that might be my problem. 'Cause that's what I, that's what I'm doing. Yeah.

Okay. Okay. Well, 'cause if you do that, you can't, like, this impossible. The first of all, I think this is impossible in a way. Okay. Because like, how are you gonna do that? That's, it's gonna be hard. And then you don't want to overfit the— it's gonna— yeah, I don't know, it sounds difficult to try to come up with every possible thing you could say. You want to try to generalize it a little bit somehow. I figure out like what are the general principles you want the judge to follow rather than trying to figure out like what are the every single like specific scenario that can encounter. Does that make sense? Otherwise it might—

Yeah.

If you go with the other approach, it becomes a little bit more brittle, I think.

Yeah, that's, and, and that's what's happening where it's, like I said, it's inconsistent, sort of run after run. I think the word that you use, overfit, might be— that makes sense here. Now I think I might be doing that, but then if I write it generally, like, here's some, here's generally good, or, or what you need to say or what you need to not say. Then I'm like, that's basically the main prompt of the actual product.

No, for sure.

Okay.

It does share a lot. Like it can share a lot of the same, same thing. And so what you want the judge to do is you want to have really good guidelines that you can't, you feel that you can apply in any situation. It's hard because first of all, it's really hard when you have something that can just do anything. It's really hard to evaluate it. But that's just the overall, it's, it's very difficult. But then you want to show examples of, oh, how do, how you applied those guidelines. In the prompt, you want to show, okay, because of XYZ, this is a good example and why. Or because of ABC, this is a bad example and why. You have enough of those examples that are drawn from your critiques that you feel comfortable that it's showing, it's giving the language model enough information to like approximate whatever is in your head.

Yeah.

Like, it's like something you could follow. Sure. Or that you could follow before you joined Sundae.

Yeah. Yeah. Okay. And I haven't been using like examples in the way of a sort of question-answer pair, and I could probably do that a lot better. I've just been putting, here's critique, here's all the critiques, dumping it in.

Yeah, it's closely related. Like, I think in my blog I showed like something like dumping the critique in, but yeah, really what you want to do is you want to show the model the reasoning of why you want to get to a particular kind of judgment. And yeah, you just want to teach the model, like, just show like their general reasoning. Patterns. And the critiques, you want to try to see if you can do the more tricky ones. Okay, this is how you apply this guideline. These are like different ways to apply this guideline or whatever. You should find that you're like editing your guidelines as well while you're doing this, because yeah, usually you'll find something that is not working.

Okay. Related to something you said earlier of if this, if we're doing okay, if we're having like 80% alignment, is that, I guess I'm trying to get away from having to read questions, not look at the data, but like having to read and score questions. I have been spending daily time doing this and that's just not scalable.

Yeah.

I guess how good could it get with LM as a judge? Maybe I have an unrealistic expectation that we could, oh, it will just be automated and we can just hit a button, run evals, and we can know if we're making progress or not.

Yeah, I don't think you can ever get completely away from looking at data because at the end of the day you have to trust it. And there is no other way to trust it other than checking it. Yeah, there's just no— there's no way out of that. Doesn't mean you have to look at every single thing, right? You know, you can sample but you still have, like, someone has to look at it. Now the things that, now the thing that you're doing about segmenting into topics, that's where it's going to be more and most interesting because you will be able to see, you know, in this area, there's like high alignment. In this other area, there's not that much agreement between me and the judge. It's like kind of, I can't really rely on it here. So like you might find it's very uneven and you have to account for that potentially in your evals and also in your review process. You might have to say, okay, you know what, in this area where you can't quite automate myself, I might have to look at it more. Okay. And you probably will figure out how to automate yourself more in that situation. It's hard, like, um, a lot of times it might make sense to try to focus Um, these different judges, but it's a little bit of cat and the mouse game. You don't want to over-engineer your evals either, because there's a lot of things you're going to find when you're doing this. The real magic of this entire process is that you're looking at it and you're thinking about what's wrong.

Yeah.

And then if you find something obvious that's wrong, just go fix it. That's what I would do because it's, it's a little bit circular. In a way.

Okay. Then I'm, I'm thinking about like scalability. If we have thousands of conversations happening, like in the springtime, we get a lot of, of traffic. That's a very important time of the year for us. We're trying to get ready for that. So if there's thousands of conversations happening, you can't possibly read them all. How do you ensure quality?

What do you mean? What do you, how do you define quality?

Like getting the questions right, like I'm doing with the evals now.

Yeah. There's different ways you can think about it. One is you can, so you can, okay. One thing you can do is for things that have high alignment, have, you can rely on that a little bit more. You can sample in areas of low alignment heavier to like spot check like what is happening. There's other advanced methods you can use, like you can try to— you can, once you figure out like what is going wrong, you can decide, hey, um, you want specialized tests that go beyond just like general judges that like evaluate things like RAG. Or evaluate factual correctness or hallucinations, we want to do that kind of based on some error, not just like for the hell of it.

Okay.

Because it can become distracting. But what you want to do is you also want to know if you can trust that too. So it has to be grounded in some real error. So like when you roll this out and you have all these thousands of conversations, like whatever, in the spring and the summer. Have some topic areas, like, or many topic areas that have high enough alignment where you can use it as a proxy, but then other ones, like, yeah, you have to look at it. You can also see if you can— another thing is you can see if you get user feedback. Yeah, I don't know how much, like, what the dynamic is. Like, some people don't trust their users.

We do. We have this, the thumbs up, thumbs down thing that's often recommended.

What, how often are people providing feedback? Is it—

I wanna say 8 to 10% of messages, something like that. Okay.

Yeah. What is the actual sort of, is there a business metric that's being tracked? Do you know if people that use this, these features, what the impact is on their purchasing and stuff?

It's still quite new, but I would say the goal is engagement, get people to have conversations and tell us more about them. And then it will make product recommendations, but that's one-off product at a time. It will probably be a long, a longer-term metric for retention and reducing churn.

Yeah, the real advice that I would say here is you might not wanna hear this, but it's Can't tell you. It's like this product, it's like a little bit, the surface area is like way too large. If it was, okay, if you were my client and I would like, I was like consulting with you, I would say, look, it's going to be really hard to evaluate this. It's going to be just like all over the place. Let's pick like a few topics and have something that specializes on that and really understand that and get that to work well before trying to solve everything.

Okay, this is great.

No, because it's actually very— it's very difficult. Like, it's very difficult to just have something that can— you can ask it anything and evaluate that. Like, it's, it's quite hard. Like, it's— you're trying to essentially have— yeah, it's usually not the best idea. Everyone tries to do it. But it's, you want to start somewhere else usually.

Mm-hmm. Okay. I didn't think, like I said, there's 40, probably 40 things people ask. I didn't think that was too, too much necessarily.

No, it's good to be, I don't want to dissuade you, but I guess what I'm saying is like, even the 40 topics, like 40 is a lot of different. It would be good to focus on something and say, okay, we nailed it. Like, this is like, hey, what seed— like the problem I have, for example, what seed do you need? I don't know. Like, I'm just like reading descriptions and I'm just like buying different ones. I have no idea. I'm just like, let me just put it on my grass and see what happens. I don't know. Whatever. Like some like target areas. Like it doesn't have to be 40, maybe it's 5 things. Um, that's my intuition anyways. Okay. Is to like start like scoped and like really make it. Good.

Okay. This is great.

Okay.

Yeah. So another kind of related to this, the large surface area. One of the questions I would, I had for you was the topics change over the time of the year. The questions people are asking in fall are a lot about, is it too late to do this? There's frost. Can I do this or that? That's gonna be very different from what they ask in the springtime when the weather is different. And I, in the past few weeks, I've figured out, or I was concerned that, oh, I, we don't know what people are going to ask at different times of the year because we haven't experienced it yet or had this out. I actually think I can probably do some type of synthetic data generation about that. It's, they're going to ask different questions in the fall about seeding timing, it being too late, or what about frost? But they're going to ask the same. They're still going to ask about seeding timing in the spring. It might just be, is it too early or is it too hot? But it's all just seeding timing. And I think. In the last few weeks, I think I've realized I can probably address that with some synthetic data generation.

What's the distribution of, of like topics? Is it like, let's take a specific season, let's say whatever, spring.

Yeah.

Are the, is it like 2 or 3 questions that are driving like 80% of the volume, you think?

Or is it, I would say it's probably 5 topics, 5 or 6 topics driving most of the Conversations, but they're very different.

Okay.

It's like this fall it was seeding timing. When is the renewal? When does the plan start or what am I getting next year? Those are like the top two, some more subscription questions and maybe a little bit about weeds, but yeah.

Maybe you can focus on those and say, and have a compromise in a way where you're like, hey, most of traffic is here. We're going to focus on this, make it really good, make the LM as a judge as good as it can get, make it work as good as it can be. Yes, there's the other things, but in order to drive progress, like, we're focusing on the things that people ask most about so that we can learn. And then as we go along, we'll refine this tail, the, the tail. That way, like, it'll be more tractable. Because 40 is a lot. Okay. And it just, yeah, it'll just help also to make progress on, makes something really good.

Yeah.

Okay.

Great.

I don't know if you have the ability to do that or not. If you can say, hey, we're focusing on these things.

Yeah, we can, or I, I feel like I could, but how do we articulate that to The user, it would just be like, this only answers questions about seeding now. And if you ask about this, we'll just say, oh, I can't answer that. Is that what you do?

You could do that. You could also just leave it and say, and just know, hey, this is something that you are tolerating as a business to say, hey, these other questions, they're not going to be as polished as these ones that people ask about a lot. That's okay. We're learning. And that's okay. It's just that just internally you're like focusing on those buckets so that you can refine those. That's all.

Okay. This is great. I can't wait to send this conversation to, uh, my colleagues.

Yeah. No, it was really nice to connect with you. I don't buy the products really anymore cuz I have a, I have a landscaper now.

Ah.

So I'm not even doing it myself.

Okay. Okay. If you ever have any lawn or garden questions, please let me know.

Will do. Yeah. Okay, cool. I'll send you the, I'll send you the recording. It's recorded and then I'll just email it to you. Yeah. Sounds good. If you wanna send it to other folks.

Okay, great.

All right. Sounds good.

All right.

Thank you.

Yep. See ya.

All right. Thanks.