LLM Eval Office Hours #1: Multi-Turn Chat Evals

Hello, hello, uh, Max.

Yeah, I'm good to meet you.

Yeah, nice to meet you. Welcome to the office hours.

Thanks for having me.

Yeah, so let's get right into it.

Yeah, so, um, so we're building a company called Windmill, uh, which we're trying to help managers automate a lot of the kind of what we call chores that are traditionally associated with management. So there's two main things we're doing today. One is we help you collect peer feedback, so we see who's working together, and then we have a Slack agent that reaches out. I can show, I can share my screen quickly. If we go to, it will reach out and basically say, hey, how's it been working with this other person about a specific work item? And then it has a little conversation, turns that into feedback, and then make sure that gets back to the person so that they know where strong or where there are areas to improve. Um, another thing we do is every week on Fridays, we reach out, um, and have a little conversation about basically where'd you focus your attention this week? Uh, it already knows a lot of information about where the person is spending their time because we hook up to all the different productivity tools. Um, but they can kind of go in and be like, hey, here's where I'm focused on. And then you can imagine for a manager, they can get a better sense of where people are spending their time, what they're focused on, are they focused on the right things without having to set up a ton of meetings and stuff like that. So overall, we're trying to make it a lot easier for managers to automate a bunch of the stuff that is usually pretty time-consuming. And increasingly, we're deploying these kind of via an agent that works with your team. So this, we call it Windy. Windy will go out to everyone on your team and ask them questions or get feedback or stuff like that at different times of the day. And so we're going a lot deeper. We've been using LLMs for a variety of things, but increasingly we're going a lot deeper into these little chat conversations. Primarily inside of Slack. Um, and, uh, we're now at a scale where, uh, we've been kind of just eyeballing and improving prompts by hand, and that was working pretty well. But now the prompts are pretty good, and, uh, we need some more robust evaluations, specifically in the context of like a multi-turn chat conversation. And so we're currently looking at— we have some in-house tooling which I can show you, um, and we're looking at some vendors, I don't know, like Human Loop and Langsmiths and those kind of people, and just would love some guidance on like where you think we should be started. What are the right areas to focus on specifically in this kind of multi-turn chat type of approach? So yeah, that's kind of some background.

Yeah. So okay, like multi-turn chat is very open-ended. You know, you could potentially evaluate infinite number of things. And a lot of people get stuck at this phase when they're like, okay, where do I start with evals? Very understandable. And so the counterintuitive thing is like one thing that's really good to do is to do an error analysis first. And so what that means is you gather your data like, you know, in whatever way that you collect it in a way that you can read a lot of examples.

Do you mind if I record this? Sorry.

Yeah, it is being recorded on my side, but, uh, and I'm gonna post it.

Okay, cool.

Um, so, um, essentially what's gonna happen is like you do an error analysis, and an error analysis really is— and let me, I could share my screen, uh, cool, to give you kind of, yeah, some pointers. Um, let me One second.

How do I— am I still sharing?

Uh, I think it'll still let me share. So yeah, you won't block me, so no worries. Um, is it— can you see my screen?

Uh, hold on, I cannot. Wait, let me— what's going on? Okay, thank you.

Oh, here we go, here we go.

No, I can— oh, here, so my screen.

Let me close that out.

Okay, cool.

Okay, so what you want to do is like, you want to like perform an error analysis. So basically what you want to do is like, you want to look at your data. There's this video by Andrew Ng that kind of tells you what it is. It sounds a lot fancier than what it is. Basically what you do is like, you look at a bunch of data and start seeing, like, kind of categorizing by hand, uh, like any errors that you see, just roughly. Okay, it's nothing, nothing fancy at all. It's like very, like, like very manual in a way.

Yeah, so we just did, uh, one of the things our users can do is basically saying like, hey, I don't have any feedback, um, which is usually a sign that like it wasn't a good question. Why'd you ask about this thing? And so we just went through that and did analysis of all the different reasons why people essentially dismissed the chat. Um, so it was kind of similar. We didn't do every single chat, but we did the ones where it was negative. Um, and so we kind of have a good sense of like, we have a bunch of examples that are not good, but we kind of, as we, it's kind of, you, you get into that whack-a-mole game where you fix one and then other stuff gets worse. And it's just at the point where we need something more automated. So, but yeah, sorry, I didn't want to interrupt.

Yeah, no worries. Okay. So if you've already done, it seems like you've already done some error analysis. Which is great. So what you should do is, like, focus your tests on those errors.

Yep.

There's a lot of ways to go about it. One is, like, okay, if you can narrow down, like, the problem and kind of figure out, like, why— what's triggering it. Like, oh, okay, when the user is asking about XYZ, it's always doing this, or it's always, like, giving, like, boilerplate answers and— or something. I don't know, I'm just making it up. Um, what you want to do is like start, you know, crafting evaluations for that. Now it's a little bit of a like back and forth. It's not like— there's no like linear workflow through this. So, okay, like first I just want to say, like, if you find something that's like obviously really wrong all the time, just go fix it. Don't get too— don't get nerd sniped by evals, like in the extreme sense.

Yep.

Um, so like after you've kind of solved like the really like obvious stuff that you're like, okay, let me just go fix that right now because it's like whatever, um, then you can start thinking about, okay, you know, can you write tests for those? And like, okay, how do you write tests for something that's like multi-turn conversation? You don't— like it's subjective maybe, um, like you can't necessarily write an assertion for it. Like, and so that's when we get into LLM as a judge potentially, and that's when you want to like sort of see if you can craft an LLM as a judge that matches your opinion of things.

Yep.

Um, so that's what this blog post kind of walks you through in a way. I mean, this blog post is one attempt at describing a possible workflow through doing that, so don't Yeah, don't like just take it with a grain of salt, but essentially like what you want to do is you want to write down specific critiques of like what is going wrong. Like you take, like you drill into one of those categories of the error analysis, you write down like, okay, what went wrong like exactly. Once you have like 10 of those or so, you can try to see if you can go update, like craft a prompt for LM as a judge. And what you want to do is you want to go back and forth and observe like the agreement between you and the LM as a judge until you bring it into like a high amount of agreement.

So you want to basically prove that can— yeah, so, so the first thing is to make sure that the LM can match you as a judge, like a human judge. Yeah, like can, can we treat it, can we teach it to make sure that it can evaluate Okay, so like, can we trust it as a judge is the first thing to try to figure out. And the way to— this makes sense. Okay. Yep. I got it.

Yeah. Yeah. That's the key part people miss. They're like, they kind of just throw the LLM as a judge. They write their prompt and then they go about their day. But then if you don't do the exercise where you try to align it with you and measure it, then you can't ever trust it because you don't really know if it's like better.

How often do people run into issues where they just, the LLM judge is not good enough? Like, generally, can you get it to work?

Or, I mean, maybe that's too vague of a question, but, um, so the problem is, yeah, like, um, this is a very interesting question. Every time I've done this exercise with people, LM as a judge, people change what they're looking for significantly because, like, the process of, like, doing it changes your mind about what should be done. Like, it's like, if you're gonna write critiques, it like forces you to like really think about it.

Yeah.

And every single time, 100% of the time, people are like, oh, you know what, like, actually it should be like something else.

So this is just going through the process is sometimes enough to help improve.

Yes. Yeah, yeah. It's the joke. It's like the inside joke. Amongst like, like, uh, ML or AI folks is like, it's kind of like we go through this process, but it's really the process that actually ends up benefiting you more so than even the LM as a judge. Uh, because it's like just forcing you, it's just like forcing you to look at the data very, very closely.

Um, yeah, like, and it's much easier. It's much easier. I mean, the prompt we have is like a little tricky where it's, it's definitely easier to critique than to actually write like the perfect, like given a result, you can critique it, but obviously the hard part is trying to write that. So, okay, so this makes sense.

All right. Okay.

So, so, so let's say we do that and we have a judge that's working well.

And there's some nuances in here, like, so like one is like, you know, try to make binary decisions about if it's good or bad. And then like write a critique of why it's good or bad. Like try to distill it. Don't, don't be wishy-washy. Like, oh, it was like kind of good here, but it was kind of bad here. And maybe, and blah, blah, blah. All that stuff is like, you know, it's not actionable and you're not going to go anywhere with that. You kind of have to draw some line in the sand. Like, is this passable or not? Yes or no. And you want binary over like multi, like a, I don't know, 1 through 5 or something. Yeah, absolutely. 100%. Because if you get a 1 to 5 score, what are you going to do with that? Like if you get a 3, like what are you, that's not going to mean anything to you over a 4. You're going to be like, okay, like whatever.

Yeah.

Yeah. Um, okay.

Yep. That makes sense.

And, uh, basically like when it comes to the prompt, um, you know, what you want to do is like, you want to put examples in the prompt as well. So like there's some, you know, exact, like you want to put examples of your critiques. In the judge, because what you want to do is like you want to have your judge use chain of thought to do the judging, and like your critiques that you wrote are your chain of thought. So you want to train it to think like you. Does that make sense?

Yeah, yeah, we're doing this in like all of our prompts now, uh, very similar pattern. It's pretty effective, so this makes total sense.

So yeah, okay. So that's kind of like how you can think, start thinking about this, like, you know, this eval for that case.

Yep. And then in terms of, okay, so we have that eval. What about in terms of like the, maybe you're saying don't worry about this for now, but like in terms of like the tooling and the infrastructure to be able to, we're basically trying to decide around building stuff in-house versus like moving our— like right now all the prompts are in the codebase and, uh, all version of the codebase, that kind of setup. Um, and there's obviously a lot of vendors that promise good stuff if you move the prompts to their system and use their evals and logs and observability and etc. And those are definitely problems we're facing. But, um, we have a question around like, do we continue to build stuff ourselves? Do we try to use some sort of like an open-source eval framework that we kind of just plug into R.

Tell me a little bit about your tech stack. Are you Python, TypeScript? What kind of— are you have like machine learning, like data science type people, more developer type? Give me a little bit like—

Yeah, just a few full stack engineers, all TypeScript.

Okay.

Small team. Not really trying to solve the problem where we have like different people doing the prompting versus doing the actual like engineering stuff. More just like we're struggling with observing the world.

Are you using any frameworks internally? Like— LangChain or something? Okay.

No. So we call, we call the, we have our own like homebuilt wrapper around, uh, so we can easily like swap out Claude and OpenAI and, uh, most of the stuff we're using SONNET 35 for. Um, and so we have a little bit of logging and tooling that we built in-house, but pretty much just making the API calls directly, pretty raw.

Okay. Are you using anything already? Are you just shopping at the moment?

We're looking, we're looking at some, some different vendors. Um, and I don't know if it's— yeah, we need to figure— we need to get better at this, um, because we're, as I mentioned, like, we're hitting the wall of, um— so it's not like— I think the setup I want to start to have is where we have a nice eval system, and then when we're doing prompt engineering, we can kind of quickly iterate on prompts and see how those are changing. Um, and then also there's some sort of probably like always running thing that's looking at logs and doing evals, and then maybe some CI/CD like system that kind of like runs before deploying, also running some like evals. So something, I mean, that's like high level, but yeah, some stuff like that we're trying to figure out infrastructure.

Okay, so, um, and is it, are you okay with paying for tools?

Or yeah, yeah, we're okay paying.

So, um, the ones that you mentioned are all very decent, they all do those things. Um, you know, so like my favorite vendor so far Um, Human Loop is really good as well. Um, so I guess for you, since you're like a TypeScript shop and you're like mostly full-stack developers, um, you know, like Braintrust is worth looking at. I don't know. Yeah, no, we haven't. Yeah, definitely take a look at them. Take a look at Human Loop.

Is Braintrust your favorite, you would say?

I like them a lot. Um, they're, they're pretty good. Yeah, it's hard to like have a favorite, honestly. Um, they're like different things, like for different purposes, like, or they're kind of like people different, uh, good at different things. But like, it's definitely— you should definitely take a look at both of them, get a feel for like what you think resonates with you the most.

Um, yep.

That's, that's the one I would add to the mix.

Okay. We looked at LangFuse, LangSmith.

Uh, yeah.

So a lot of Braintrust.

Yeah. I think it's cool. I mean, I really like Harrison, so like their team has amazing support. Like they're all over it. Um, but just to add Braintrust to the mix and kind of choose.

And with the people you've worked with, the people who have made the jump, I'm guessing there have been people in our scenario where they're kind of some half-baked internal tools and then they're like, hey, let's use a vendor. Like, is that usually a good— I mean, I've experienced—

yeah, yeah, okay. So, so yeah, yeah, like, um, for eval stuff, you want to use a vendor. For a lot of other stuff, I don't think you, you should— like, I'm not a fan of frameworks that much. Yep, I agree. You know, LLM frameworks or like, you know, orchestrator type stuff. But evals is things like, I don't know, it's kind of like observability and stuff like that. You don't want to be building that. You know, it's probably really far away from your core thing.

Definitely is. Yep.

And so it's kind of like, yeah, I recommend just using—

and with the, like, so I, uh, I don't know how related LangChain and LangSmith are. I didn't have a great experience with LangChain. Is LangSmith, like, even if you're not using LangChain, that's still worth looking at?

You can use it, you can use it with a Lang without any Lang Chain, LangGraph stuff is 100% compatible. I would say like, it's kind of the reverse is true. If you are really bought into LangChain, then just use LangSmith because like the integration is gonna be like best in class. So like, but if you're not, then it's kind of like maybe it's worth like getting a look at these, at like maybe these 3 that you pointed out. Yep. Just because like also they keep changing and they're like kind of, yeah. So just, and you'll like, you'll like one more than others.

Do you have an opinion on, um, where to store prompts? Because they all want you to store prompts in their framework and version them there, or at least the ones we've looked at, and use their prompt templating system, which obviously I understand how it can make all the tooling they have a lot better because they have the template itself.

You don't have to, like, you don't have to. I know, like, I was looking at LangFuse the other day and they have like a prompt management system with versioning and stuff. Um, I would say it's up to you. There's some, there's some drawbacks to doing, to having your prompt in a separate thing that is not a part of your code base.

Yep.

And you need to reason about how you feel about that. Like, do you need lots of non-developers iterating on your prompts? Maybe the answer is no. And then if the answer is no, maybe just leave it in your code base. Because once you have something outside your code base, then you have to deal with all then you have a shit, you know, you know how it is. Yeah, you basically are having some code outside your code base in a way.

Uh, yep, yep, yep. So that all makes sense. And do you see most people versioning their prompts alongside their just like general code versioning, or are they having like a whole separate scheme where you're kind of storing multiple versions of the prompts and using some sort of a system to kind of like, I don't know, roll stuff out?

Some people are, um, So some people are versioning it like separately, especially people that have like domain experts and non-developers, like they need to iterate on it. I will say that like when you get, okay, like a lot of times your prompts don't live in a silo. So like they need access to applications.

Not at all. Yeah.

Yeah. They need to like query a database. They need to do something. And so it can be a little bit difficult to house the prompt like to test the prompts outside your codebase.

Yep.

And you kind of have to like design a dataset that can like be templated in. And we like, you know, it's get a little bit tricky. So you, there's a trade-off there, but, uh, yep.

Yeah.

You have to think through that. Okay.

All right. Uh, all super helpful. Thanks for taking the time.

Yeah. If this was, uh, helpful for you, please, uh, I'll send you an email with like a feedback. Uh, just let me know, you know, awesome. It could be more helpful.

No, it's, uh, super helpful. I appreciate taking the time. Just make sure I didn't share anything sensitive, um, before you post.

Yeah, if there is, let me know.

Uh, yeah, I don't know. I don't know exactly what I— I guess if the— I showed you Slack, so I don't know what, what was in there.

So, okay, I can edit that bit out, or if it's—

yeah, I mean, if it's just— I don't know. If I can see it before, I can tell you. It's probably—

Okay. I'll send you, I'll just send you the video. Yeah. Just send me the video. Like privately. And then you can let me know. Okay.

Perfect. Sweet. Great talking.

All right. Thank you. All right.

Thanks.

See you all. Bye.