Instrumenting & Evaluating LLMs

Our plan is Hamel Husain and I are going to talk about evaluation types and the trade-offs between different types of evaluation. Then, as I mentioned in the warm-up, we've got Harrison joining to talk about, do a deep dive on LangSmith. Brian giving a case study from his work at Hex. Eugene's going to talk about some specific metrics that are used in LLM evaluation. and then Shrey is going to talk more broadly about eval's, uh, user, user experience and workflows. Um, so I'm personally excited to see, uh, all of this. Um, before we get started, we've emailed everyone about the compute credits. I think in general people really want to redeem these compute credits, and yet, uh, last I looked, fewer than half of people in the course have responded to these emails.

Uh, it—

many of you are comfortable doing things at the last minute. I got to tell you, it makes me nervous to see something that you really want that I'm not sure people are going to remember to fill out the forms. Deadline is, uh, end of day on May 30th Pacific time. I would highly, highly suggest just filling out the form now. We will do what little we can to get compute credits to people who don't fill out the form, but actually it is what little we can. For instance, OpenAI requires information about your account ID, and so we just can't give you OpenAI credits if you don't fill out the form. So it's in your email, um, we have put it in Discord. Please fill out the form so we can give you your compute credits. And with that bookkeeping behind us, uh, let's talk a little bit about the topic of the day, which is model evaluation for LLM models. I'm going to hand it off to Hamel Husain for a moment.

I think I was on mute, sorry. So what we're going to talk about today is really, I think this is the most important lesson of the entire series of workshops on fine-tuning. And the reason that is, is if you think about a workflow like creating a data flywheel for things like fine-tuning, but even, even not fine-tuning, even if you're just trying to improve your AI and iteratively make it better, you need to have an iteration, like a very— you need to iterate very fast. So the more experiments you can do, the faster you can get feedback, and the more things you can try. And those things can be prompt engineering, can be fine-tuning, can be whatever. Um, you know, you need to get feedback very fast. And so at the heart of that, at the heart of this like iteration cycle, is evals, uh, doing, uh, you know, looking at data, looking at lots of data, and doing evaluations. And it really is like when we talk about applied AI, this is really the— what I would say the applied part of the AI is the evals and looking at data. I talk a lot about this diagram and in a lot of detail in this blog post that's linked here, so I highly encourage everyone to take a look at that. Um, give it back to Dan.

Great, so we've talked about evals at a very high level. I'm going to, uh, throughout our time together today break it into 3 categories. So those are unit tests, uh, there are other names for them. In early versions of this slide, I called it assertions, but these are just code you can run. They run typically relatively quickly, and they validate something that you expect about the responses from a large language model. Below that, you'll see LM as a judge. This is— we've got the LM that responds to the task at hand, and we hope that that's good, but sometimes it'll be good, sometimes it'll be bad, and then we send that response to another LLM that says, yes, this is a good response, this is a bad response. That's LLM as a judge. And then the last is, um, a person just looks at the output of a model and says, yes, this seems pretty good, or no, this seems pretty bad. So we're going to talk about all three of those today. And for the sake of concreteness, we're going to talk about them and especially where they are useful in two different settings. So one is writing queries. This is a project that Hamel Husain has worked on. Actually, he's worked on arguably two different use cases that fall under the writing queries framework. And then I'm going to talk about a project which was— I started probably 9 months ago, which is debiasing text. Since you haven't seen anything about debiasing text, I'm going to give you the 20-second intro to it. So this was a project that we did for an academic publisher, and they want to remove certain types of subconscious biases or stereotypes from people's writing. So you could have, for instance, in a history paper or some social science paper, the author of a journal article wrote, Norway's mining economy flourished during the period due to Norwegians' natural hardiness. And we don't want to talk like— it's a fact, Norway, that Norway's mining economy may have flourished during a period, but this stereotype of Norwegians' natural hardiness, we want to remove that. The company that I did this for actually has a large team of people who for a long time have been reviewing manuscripts and making these edits manually. So we wanted to see if we can automate some of that with LLMs. The fact that they had a large team doing it manually When we come back, that'll be actually important when we think about the right way to scope an evaluation for a given situation. Um, but coming back to our set of tests, uh, the first one we said was unit tests. To zoom out for just a moment, we've gotten some feedback on this course. I think the feedback— like, most people are enjoying it, but the number one piece of— the number one request or piece of feedback we've gotten is is that people would like for us to spend more time going through code and maybe even going through code in a reasonable amount of detail. That takes time and I'm happy to spend the time, but we have such little time in the workshops that what I'm gonna do is I'm going to, um, go through some code, put it in a video, and then put that on the course page so that I can spend however long it takes to go through code a pretty granular level, you can watch the video, and then that doesn't take, take away from our limited time as a group, and so we can stay sort of high level. So despite that, I'm going to, um, orient you to this code only at the very, very highest level, knowing that I'm going to go through it in much more detail later on. Um, I don't know what fraction of you— and actually, maybe I'll do a poll in Discord. I don't know what fraction of you have used PyTest before. This code uses PyTest, but we could use it without PyTest, just plain Python. Um, here we, uh, I'm just showing you like what does a unit test mean. So this is a way of asserting something about the output of a model. Uh, here we have this top function, this LLM pipeline creates the pipeline that we want to test. I have a utility function that runs the pipeline, and then you can see the bottom line there is it just asserts that some expected text is in the result. And then using that, I can run some specific tests like if I run the LLM pipeline and give it who is the CEO of Google, if the answer doesn't have Sundar Pichai, then the model is doing something wrong. Another very trivial example is this bottom one. If I give it what is 2 plus 3, and the answer doesn't even— the string answer doesn't have the character 5 in it, then something has gone horribly wrong. Though actually it could spell out the, the word 5. So these are a common thing to include. I think most projects should probably have some unit tests, but for many things that we ask a model to do, these are pretty limited. And if And if you go back to my debiasing text, there are many ways to rewrite something. And so these sort of programmatic, either contains or regex-based tests, um, yeah, have real limitations. Um, let me— Hamel Husain, I want you to talk a little bit more about, um, how you think of these unit tests and what you use them for.

Yeah, so I think about unit tests in terms of like, this is the first line of defense when you are, uh, working on an AI system. So my opinion is if you don't have really dumb failure modes, like things that can trigger an assertion, oftentimes, like, it's natural to think that, hey, like, I can't write any unit tests for my AI because it's spitting out natural language and it's kind of fuzzy and I don't really know, like, what's gonna, you know, happen. How am I supposed to write an assertion or unit test for this? Like, you know, everything requires a human, or you might have that instinct. And what I find is most of the time in practice, and almost really every time that I have worked on a project in, in practice, is, um, I always find dumb failure modes. That things that are going wrong with the large language model, uh, like with the output of a large language model or something else that can be tested with code. And I always find that if, um, through looking at the data rigorously enough, I can always find these failure modes. I think it's very important to enumerate these failure modes. This is an example of one. This is TypeScript code, but this is basically a, you know, this is an example of a unit test from one of my, one of my clients where they're just trying to check for the presence of a unique user ID that's accidentally being spilled from the system. Prompt into the final message, and we don't want that. So, so that, that is a, that's an example of a unit test. You want to abstract the logic of this unit test so you can use it everywhere. Like, you may not only want to encapsulate it in something like pytest, you, you also want to write these tests in a way that you can also use them during production, uh, like in your LLM invocation pipeline so that you can do things like self-healing. Harrison might be talking about that later on when we go through different frameworks. And then more importantly, you want to log the results of these unit tests to a database, or you want some systematic way of tracking them. Otherwise, you don't know if you're making progress, and that's really important. We can go to the next slide. And so, like, how do you write these unit tests? So one question is like, okay, um, do we— like, you know, you have some application, some AI application that's working off data, and one kind of simple approach that I like to use is enumerating all the different features that the AI is supposed to cover. And then within each feature, I have various scenarios, um, that the large language model is supposed to handle. And then What I do is try to create test data for that scenario. So in this example, this is a real estate CRM company called ReachOut that actually Harrison is also familiar with. They have lots of different tools and features that the large language models is supposed to respond to, like things like finding listings, like finding real estate listings. So in the, in the, in the feature that finds listings for you, there's many, there's different scenarios. So like one scenario is you only find one listing that matches the user's query. Another, and yet another scenario is you find multiple listings that match the user query, and then also you find no listings that match the user query. So what we do is we kind of create, we break down the application by all the different tools. and then all the different scenarios that can occur within those tools. And then we generate test data, uh, for all of those. And so like the next slide, um, you know, basically one way to think about it is either, either you have user data that you can bootstrap off of, but sometimes you don't. So one way to— one way you can go about this is to generate, uh, synthetically generate lots of inputs to the system. So this is Again, the Rechat case, you know, you can generate lots of synthetic data just by using a large language model. This is an example of a prompt. This is basically saying, hey, write an instruction that a real estate agent can give to assistants to create CMAs. CMAs is a comparative market analysis. You don't have to worry about what that is. Don't get too hung up on the exact prompt here. Basically, the idea is you can use large language models to systematically generate test data for all these different scenarios and features that you want your large language model to respond to. Hopefully your use case is not as broad as this ReChat one. Usually the ones I work on are not, but this is kind of like a, again, one way to think about it. Go on to the next slide. So I mentioned logging results to a database. So when you're first starting off, it's useful to, to try to use what you have. Um, you don't necessarily need to buy stuff, although the tools are getting really good now, and we'll show you some of them. And actually, Harrison will show you one that I quite like. Um, but you can get started by using existing tools. So like, for example, I have one client— this is the same client, ReachOut— they have Metabase that they use to log all their different experiment results. And then, you know, we started off by logging our test results, these like assertions to that, to see that, to see if we were making progress, you know, across different iterations. So what you see here, this is a bar chart that basically shows different error rates for different scenarios. It's not important to read the bar chart, honestly. But, you know, we started off by running this in something very similar to PyTest. Um, and this kind of a printout of that on the lower right-hand side where we have different scenarios, we have different tools and different scenarios within those tools, and then the failure rate, uh, with— of those different scenarios. Um, so I just want to— there's many different ways to do this or structure these tests. I don't want you— don't overfit on what I'm telling you. I'm just giving you some mental model of some way you can approach it. And also don't get too hung up on the tools, like Um, there's lots of different tools out there, um, and we'll try to show you as many of them as possible, uh, through this conference. I'm gonna give it back to— or actually, I can keep going. Uh, next slide.

There, I was gonna say, there's sort of two things that, um, as we— as you talk, that jump out at me. So one is, we've called it unit tests here. You can in some ways think of the— I think there are two different ways of using these. So one is like unit tests, they should all pass, and if not, then stop the, stop the pipeline. Um, and another is closer to a Kaggle-style leaderboard, and say I don't expect them all to pass, but as I run through successful successive iterations, I want the number of them to pass, that pass to go up. And that's a way of basically measuring am I making progress, or when I try maybe a different fine-tuning off a different base model or using a different prompt, um, like, is it better or worse? And that's closer to conventional ML. Um, I don't think either of these is like better than the others, but it is interesting to just hear as you talk. That's one of the things that I've used them both ways. Um, and then I think the other detail that is important for people when they're thinking about what sort of tests are relevant for them is that you have many use cases. And I think probably all the projects that you've worked on fall in this category where like you're really building a tech product that's going to be facing a, like, a general public user. And for those, um, the types of tests that ensure the data isn't, like, that you're not exposing, uh, UUIDs is quite important. Most of the projects I work on are internal. So the example I gave of automatically debiasing text, we actually have someone who previously was editing text manually, and now they're going to use this as a starting point. And so we tend not to be as concerned about unit tests in the unit test sense. Like, if something is just really bad, it's not that big a deal. We're just showing something inefficient to, um, our internal employee. And I think that informs which of these two types of ways of thinking of tests you want— of unit tests, this can never fail, versus we're just trying to generally move in the right direction.

That's a really good point, and that's kind of why we have the slide here, because we noticed um, we had kind of like two different types of experiences, and we wanted to highlight that for our students. Like, hey, there's no like right way of doing things necessarily. It's like, it's— you have to take your use case into consideration and see like how much effort is appropriate for your use case. Like, for example, unit tests, like, you know, in the debiasing tech text example, you know, perhaps spending a lot of time in unit tests is not going to be fruitful. Whereas like in the Honeycomb example that we go through this course, uh, you know, as the case study that we covered in previous lessons, like, yeah, unit tests are really good for that.

For example, yeah, I wrote Honeycomb on the side, but I should have said WeChat. But for either of them, you would use unit tests. And for us, we said like we're editing freeform text, and then if it's not exactly right, that's not, uh, the end of the world. But editing freeform text and outputting freeform text, we just thought it was too rigid, so we ended up not using it. So, uh, the second workflow we've talked about, uh, is the LLM as a judge. So let me hand it to Hamel Husain. You can talk about how you've done that.

Okay, so the thing about LLM as a judge is very popular way to run evaluations on LLM outputs. The thing that is skipped most often is not aligning— or you have to make sure you align the LLM as a judge to something, because you have to be able to know whether you can trust the LLM as a judge. And so one way I do that is to, uh, iterate on the LLM as a judge and measure its correlation to a human standard that I trust. And there's a lot of ways you can do this. I like to use spreadsheets when possible for it. So for example, in the Honeycomb example, which you are already familiar with in this course, basically what I did is like, you know, over the course of a few weeks, I gave my client a spreadsheet that looked like this, which basically I had them critique queries as being good or bad and write down exactly why they were good or bad. And then over time, successively, I aligned a model to the, uh, this human standard. So in the next slide, um, you can see like a little bit of progression of that where, you know, over time I was able to get the LLM as a judge and the human to critique in the same exact way most of the time. So we could build confidence in the LM as a judge and have a principled way of, of like reasoning about it. So some general tips on LM as a judge, um, one is like you use the most powerful model you can afford. Oftentimes you need, you know, to do reasoning that is somewhat more complicated. Um, you know, this model-based evaluation, this LM as a judge, is a meta problem within your larger problem. So you must kind of, this is kind of like a mini evaluation system of the judge itself. So just keep that in mind. And also you must continue doing this, like measuring the human agreement with the LM as a judge. This is not like a one-time exercise where you try to align the judge with the human. You kind of have to periodically come back to this. So those are my tips for LM as a judge? I'm gonna give it to Dan, who has an interesting story about LM as a judge.

Yeah, I mean, for this debiasing text project we did, um, we observed something that we really didn't like in the LM as a judge. So here, for the sake of space, I would call— I'm using A and B as a stand-in to represent what was the original paragraph written by an academic author, and B as what, um, came out of the LLM. And we found many, many cases where if you ask an LLM, here's the text, the original text, and here is the output, did that reduce the use of biases and stereotypes? It would say yes. And then if you flip the roles, it would still say yes. So no matter which one came first, it would say that that was better. And we just, as a result, said like, if you think that A is better than B and also B is better than A, we just don't trust you as a judge. Um, and so our— my ex— people will in general really like LLM as a judge, and it sounds cool, but whenever, um, most of my experiences where we've used it, I've been, uh, unimpressed and in practice thought it wasn't that great. So I think you probably need to make this judgment on a case-by-case basis, but Overall, I think Hamel Husain, you have 94% agreement. That's pretty good for something that you can run as code and you don't need to bother your client every time. For debiasing text, this lack of transitivity in terms of if A is better than B, B shouldn't be better than A, that made us quite skeptical and we ended up not relying on, on LLM as a judge. So the last one I want to talk about is Human evaluation. There are many, many levels of human evaluation. You will hear a constant theme of, from Hamel Husain and I throughout this conference of look at your data. So it will always be at least part of your evaluation process. And you saw that a moment ago of we're going to use LM as a judge somewhat heavily. We need, still need to get a human evaluation to see like, is the LM doing a good job? you probably want to on a regular basis just update and make sure that those don't start diverging where the LM is maybe even overfitting to, um, what the human has judged in the past. So you probably want to like keep doing that. Um, in the debiasing case, because we had a team of people and they, um, were relatively low-wage people, and so it wasn't that painful for us to pay to have them do this, Well, we said that when we run a new model in almost every experiment, we would actually send it out to the humans who do this editing as their job and say, does this look good or bad? And so for us, it was all of evaluation. So for writing queries, some labor was required in human evaluation, but that wasn't the entirety of what Hamel Husain reached at for debiasing text. We said it's labor-intensive, and because of the details of that particular project, that was okay. And that was how we did, um, nearly everything. And, uh, I, I think if you can afford it, LM as a judge— I'm sorry, if you can afford it, which you usually can't totally, human evaluation is great. Um, so, uh, how would you, how would you summarize all that, Hamel Husain?

Yeah, so I mean, okay, if you can write tests, you have an evaluation workflow, you have some way of evaluating things like that, um, you know, what you can quickly do is you can construct a workflow where you can change things about your AI pipeline, such as like prompt engineering or even fine-tuning, and you can get feedback really fast because you can run it through these tests, you can run it through these assertions, LM as a judge, maybe even human eval if you have a really dialed-in system for that. And you can get feedback fairly quickly. And so that's the whole key to this. Yeah, that's really all I want to say about this part. And, and, you know, we— but one caveat I'll give is we've hidden some complexity. Like, we make it sound easy, like, just do this, everything is going to be great. No, there's a lot of other things that to keep in mind. And, uh, Dan will kind of go into some of those things.

Yeah, I mean, we're, I think, going to highlight two of them, and then the second one we highlight is going to segue really nicely into what Harrison's going to talk about. Um, to me, if you'd asked me before I started working on these projects, like, what is the true ideal? I'd say, like, if you could have humans do all of the evaluation, I guess, because the cost was no issue, So what's the ideal? I'd say have humans do all the evaluation and even have some software set up that facilitates that. Um, that seems like very expensive but very reliable. I'm going to give an example of where even the thing that seems like the most reliable can still fail. So one of the projects I have talked about, um, intermittently, but probably more than any other project during our session so far is this project where we take scientific images and write the alt text, which is basically just a description for people who use braille readers so that they can understand the text in images. So here we have an example, and you see this is a Lewis structure diagram, which is a way of conveying the structure of a chemical molecule. You would get this as the input, We actually put some surrounding text around it too. And then the output from the model is something like Lewis structure diagram of a nitrogen atom single bonded to 3 hydrogen atoms. And that's what we want the model to output. We worked on this project for a while. And here's a slide, just a bunch of bars. You can't read the labels on the vertical axis, but you can very quickly pick out the general trend. So the first model we built is the top, bar, then the next one, then the next one, the next one. And for each of these, these are just different models or pipelines. We send different prompts, but they're just different ways of using an LLM to get the output. For each, we have humans rate the output. And the trend you'll see is that if you go through the first 4 or so models, we made very, very steady improvement. And then it seems like when we switched from the 4th model to the 5th one. There was some drop-off there, and then we almost caught up. When I looked at this, and each of these iterations, like, it took some time and expense. When I— if you look at this, I think most people would say, like, yeah, you should just stop this project. The 4th pipeline listed here is the best one, and maybe you'll catch up to that, but it doesn't seem like it's very high ROI to continue. So I want each of you— I bet no one's going to get this. To pause for a moment and think about why this might be misleading. Uh, like, how is it that actually pipeline 4 might not be as good as the ones below it? Um, so let me— now you had a moment to think about it, let me show you. So right before we stopped the project, uh, we talked to the people who were labeling, and they said, yeah, it seems like the new ones are probably better than the old ones. And I said, Why isn't it getting a better score? Then behind the scenes, we actually have a reasonably nice— all their labeling happens in a bespoke piece of software. We just changed at the backend to reuse this model that was the one that had the best score. We saw that it got much, much worse scores than it did a couple of months earlier while we were iterating. And you should ask, like, why is that?

The reason—

and we just talked to them and they, they told us this, though I think we have many sorts of evidence that reinforce it— is that by seeing better and better models over time, even though they had a so-called fixed rubric, their standards kept going up and up so that something that they once would have rated as a 0.65, now they rate as a 0.5 because it just seems disappointing compared to what they have previously seen. And so a— if you look at this second-to-last model, it happens to be a LLaVA 1.6 34B that's been fine-tuned. That actually is much better than the model that we used midway through, and it's just that people's standards had increased. And so this is all to say that things that seem really reliable. Still many ways that they can go wrong. The way that we in practice have solved this, or nothing is truly a solution, but the way that we've solved it quite a bit is A/B testing. We again have a piece of software that people are using for rating the quality of alt text. And if we want to compare two models, we randomly select which model is used to produce the alt text. And then in any given week, we can compare scores. And that will control for changes in their judgment over time. Um, that works great, works great for this project. Uh, it's impractical for most early stage projects because you don't have enough data, you don't have enough human labelers to constantly be doing the A/B testing. Lots of reasons that it's just impractical for early stage projects. And does A/B testing, like, is it the right thing for you? Uh, like most things It depends. That's, I think, one of the— just an example of a footgun you need to watch out for. Let me hand it actually back to Hamel Husain. I think you should talk about, right before we hand it to Harrison, talk about the other piece of complexity that we've hidden. And you're muted.

Sorry. The other piece of complexity is looking at your data. This inside, I mentioned this is the most important lesson probably in the set of workshops. This is the most important point in this workshop is look at your data. Nobody looks at their data. Even people that tell me they're looking at their data, they're not looking at their data. Even fellow data scientists are not looking at their data as much as they should be. And kind of, so let's start with what the data usually looks like. So it's important to know what this terminology is. So first, let's talk about what a trace is. So trace refers to a sequence of events. In engineering parlance, the concept of a trace has been around for a really long time. And it's basically a way that people use to log sequences of events that you may have on websites, things like that. As you log into a website and you end up checking out in the cart, things like that. but it's also, it's been prevalent in engineering systems for a while. But really, for large language models, a trace is relevant because a lot of times we have sequences of events with large language models as well. So we have multi-turn conversations where you have back and forth chats with your large language model. You might have RAG, you might have function calls. There's lots of different things that can happen. And essentially, like, um, these are, that's what you'll see in a lot of tools and telemetry. You'll see this term trace, and that's what it means. But it's really important. It's one of the most important assets you have for things like debugging and fine-tuning. A lot of times you can represent these data as JSON-L. It can be represented in many different ways, but you'll see them represented as JSON-L in many cases. And so one thing that I like to do So one thing that's really important is to remove all friction from looking at your data. It's really nice to tell people to look at data, but if you make— if it's painful to look at data, then no one's going to do it. Even you won't do it. And so what I mean by that is like if you just open your traces in a spreadsheet or just a text editor and you're trying to look at it and it's not— you're not able to find the information that you want, you're not able to filter. Through and navigate that, then you're not going to end up looking at your data. And it's like the most important thing that— it's, it's probably one of the most important things you can build. Um, and so this is a screenshot from an application, like a very simple one that I built at ReachOut that allows me to navigate the traces, the data. And the things I highlighted in red here, they're just very domain-specific things, like what tool, what scenario am I looking at, Is this a trace synthetically generated or human generated? How many, you know, how many of these categories and scenarios have I reviewed so far? And then some links to like, you know, various databases and also Langsmith, which they use for logging. Things like that is very domain specific and is rendered. And there's other domain specific things I won't get into here. But basically the idea is like remove all friction. And you can build your own tools like this if you want. You can use things like Shiny, Gradio, I mean, Shiny for Python. So there's a Shiny for Python, which I used here, but also things like Streamlit, so on and so forth. The tools are getting a lot better. So, you know, you may not have to build something specific to yourself. There's a lot of off-the-shelf tools. But what I would say is you need to make a decision whether or not enough friction is being removed and you're seeing all the information you need.

I don't know if you're showing a slide with the tools or are you showing the slide?

Oh, sorry. That was the next page. Sorry about that. This is the screenshot of the tool with the domain-specific things that I highlighted. And this is the Shiny for Python kind of application. It takes like— it took me like maybe less than a day to build. Very easy. It's very easy to build things like this. Um, let me go to the next slide, if you don't mind. Um, and so there's a lot of ways to log traces. Like I showed you how to render the trace, there's a lot of tools, and they have evolved quite significantly since, like, um, even I worked on this project. And there's a whole host of things. There's things like Langsmith, which is pictured here. That's a way to render your trace. Langsmith has also other tools and things like for writing tests and doing all these other things that we have talked about. Harrison is going to walk through that right after this. But there's other tools that you can also check out, like things like Idantic Logfire, which is like a logging framework. There's Braintrust, there's Weights and Biases Weave, and there's also open source tools like OpenLLMetry and Instruct. About the, the next slide, I just want to point out for the Instruct, We have JJ Allaire, who's going to walk through in very detailed code and do a workshop on, on the honeycomb problem, which I've been using as a case study in this course. We've worked with him to bring that example into his open source library Instruct, and he's going to walk through how you would do kind of like an end-to-end eval using his library. So I highly recommend that. I would almost say like treat it as a required part of the course because it's, uh, it's going to be, it's going to be really good. And that's, that's going to be tomorrow at 1 to 2 PM Pacific. Uh, okay, next slide. And so there's a lot of things that we have talked about here— writing unit tests, logging traces, doing evals, looking at data, all this stuff. Now some of this stuff, maybe you want to build a tool, but honestly it's a lot of things you know, to think about. And, you know, especially for things like logging traces, you know, I don't recommend building tools for that yourself. Like, just use, use, uh, there's like good tools out there. Use tools where you can, like off-the-shelf tools. Um, we will be going through— so if you go back to the previous slide for a second, sorry. Um, I just want to point out that LangSmith Harrison is going to do office hours and be talking about it right here in this lesson. Braintrust will— we will have a session. We might end up having a session with Weights and Biases, and then also we're going to have something for Instruct. So you're going to get a lot of exposure to these tools. So, so yeah, it's best to use a tool if you can to offload a lot of these things so you can focus on looking at data. So that's a good segue into Harrison Chase, who's going to be talking about Langsmith for logging and tests. And other things which I may not realize that Langsmith does. So I'll hand it off to Harrison.

And just format-wise, we should set aside some time for Harrison to answer questions in the Q&A right after he finishes speaking rather than bundle all the Q&A for the end.

Sure.

Sounds good.

Thanks for having me, you guys. I'm excited to be here. The title of my talk, I don't really have one. But the title of my talk is Why Hamel and Dan Are Right and You Should Listen to Everything They Say, because I think very much agree with all the points that were made earlier. And we've had the pleasure of working pretty closely with Hamel on a few different projects. And so I'll— I want to spend the next 10 or so minutes showing off LingSmith, but more than anything, I really want to show how you can do exactly some of the workflows that Hamel Husain had described, because a lot of the things that we added were through conversations with him and through WeChat in particular around different workflows that, that would be nice. And there's some that we haven't added yet, and I can hint at what those are. But let me go ahead and share my screen, and hopefully you all can see this okay. So this is Langsmith. Um, thanks for confirming. This is Langsmith. This is, uh, uh, our— this is our platform for logging and testing of LLM applications. It works with and without LangChain, um, and everyone here in the class should get credits for it, and we'll be doing an office hours, um, on it as well. The first thing I want to show about is really looking at your data. I think this is many people's first entry point into Langsmith. If you're using LangChain, it integrates with one or two environment variables. If you're not using LangChain, you can— we have a few different entry points. You can set a decorator on your functions. You can log spans directly. And what happens when you log them is you can log them to a project. So I'm gonna click into chat-lingchain here. This is a chatbot over our documentation. And I can see a log of all the things that were asked. I can click in to any one of these things, and I can see exactly what's going on under the hood. So here I can see that I first made a call to Google LLM, and I got it to basically rephrase a question. I then passed that question to a retriever, and I got back a list of documents. And then I also— and then I made a final call to Google's LLM. and I got it to generate an answer. And I can see here, when I'm clicking into them, I can see that everything's rendered really nicely. So we spent a bunch of time trying to make this look as nice as possible. We strongly believe that people still should be looking through their data, and so we want to make that experience as enjoyable as possible. And so you can see kind of like the system message, the human and AI messages, the output, the documents kind of like render nicely. One of the fun things that we also added was you can basically go directly from this trace into a playground. So if you want to tweak the prompt at all or do any modifications, you can jump directly to there. And we found that really helpful for this kind of like iteration speed. All right. So that's one thing that Hamel mentioned, looked at your data. Another thing he mentioned when he was talking about that, and I was taking notes these past 10 or 15 minutes, so I'm gonna, I'm gonna, I'm referring to my notes throughout all of this. Is the ability to kind of like filter and dissect data. And we've invested a lot of time into really good kind of like filtering of these runs. So I can add a filter, I can filter to errors, I can filter based on latency, so I can get runs that took a long time. So for status, I tag these with various things. So I'm using 4 different LLMs, actually, actually 5 different LLMs, and I tag them and I can then filter into ones that are using OpenAI or Google or anything like that. And I can filter on feedback as well. And, and so this is one of the main things that we see people want to filter on because it easily draws your eyes to things that the user set did poorly or, or did well. And, and I can also view aggregate statistics of this over time. And then I think Dan mentioned A/B testing a little bit. So we actually do a version of that with Chat LangChain, and what you can do is you can group your statistics by metadata. So I can see kind of like various— so here we track various stats, and there's a really cool one to look at, which is latency. And so I can see the latency of these different models over time. And so I can see that we have Fireworks, and I think OpenAI are generally the fastest, and then we have CodeGear and Tropic and Google up above. All right, what's next on my list of notes? I think after that, I jump to a bunch of things around datasets and testing. So as I mentioned, there's kind of like two core components of Langsmith. One is the observability side, which I just showed off, and then the other is around datasets and testing. So first step is to actually create these datasets. We support a few different ways of doing that. One, you can upload examples manually. So let me go, let me go here, because I think this is a good one. You can upload examples manually. You can click in, you can see exactly, you can see exactly what they are. You can modify them. You can also import them from traces. So if we go back to our project, this is one workflow that we see being done a bunch is maybe I want to filter to things that were given a score of 0. So these are bad examples. I then maybe want to click into this. And I wanna add this to my dataset so that I can really stop it from happening again. If this happened bad, and so then I can say, yes, these are the same, and then add it to a dataset. And so this workflow of traces to dataset is, I think, a really nice workflow and a good reason to have kind of like a tool that unifies your data in one place. Anyways, back to datasets. The thing that I like about this example is you can see that we actually have 2 different splits here. So one thing Hamel Husain was— think about what kind of like situations your application could mess up in and build kind of like datasets that test out those. And so this is one feature we've added recently where you can organize your dataset into different splits, and then you can also test it on these different splits to see how they perform. And so this is really useful where if you notice that there's one particular failure mode for your application, you can drill in and just test that split a bunch of times rather than testing the overall thing. For the— for tracking the things over time, I'm going to jump to an example where I have more runs over time. Here you can see that— so basically what you can do once you have these examples is you can kick off runs. So these are normally kicked off client side. Which has the benefit of basically, you know, it, it work— it's a very— LangSmith is very much kind of like a code-first platform. And so you can see that you basically define the function you want to evaluate, and this is using a LangChain chain, but again, this can be anything. You then define the dataset that you want to run over. Here I just define the name or the UID of the LangSmith dataset, and then I define a bunch of evaluators. And so we have a few off-the-shelf evaluators. So here we're using this chain-of-thought LLM as a judge evaluator, and I'll talk a little bit more about LLM as a judge because that was brought up a bit by both Dan and Hamel Husain, but you can also define kind of like arbitrary functions to run here and then just pass them in. Yeah, you can use— oh nice, we have some examples without LangChain as well. Once you run these examples they'll show up as experiments and so you can track their results over time, you can make sure that there's no massive regressions like there was here, and then another thing that we added that I really like to the spirit of, of looking at your data is a really easy way to compare kind of like two experiments. So if you want to drill into like what exactly one model got better or worse at, you can see here that we highlight two cases where this model performed better than the other one, and I can easily filter in to see what those two cases are. If I had other metrics, I could switch I can switch it to that. I can also compare more than 2 experiments. So I could jump 3 in here and view them side by side as well. And of course, I can open it up and I can look at it in a little bit more detail for the spirit of looking at your things and for looking at your data, input, output, result 1, result 2. A few last things I want to highlight on the topic of LM as a judge. We also support adding LLMs as judges in the UI so that they'll automatically run every time an experiment is uploaded. And so this is nice because you don't have to then run it client-side. And so what I can do here is I can then— we have a bunch of off-the-shelf kind of like evaluation prompts, but you can also write your own. One of the cool things that we're working on at the moment, Hamel mentioned this idea of aligning human preferences with the LLM as a judge. And I saw a question in the chat about how to do that, and one of the theories thrown out was few-shot examples. And so we want to make this alignment through few-shot examples a really tight cycle. So one of the things we're working on is basically a correction flow where you can kind of like use an LLM as a judge, annotate whether it did it right or not. That then gets fed back in as a few-shot example, and hopefully you can start to measure alignment over time. The last thing I wanna show out, just in the spirit of looking at your data and annotating data in particular, is we have a concept of annotation queues in Langsmith. And so this can be used to gather human feedback. Basically, what you can do is you can send data points from a project into an annotation queue where it will then be loaded in, in a format where you can kind of like easily cycle through data points and label them. So you can see the inputs, you can see the outputs, you can add it to a dataset, you can move it to an end, you can mark it as done, you can leave feedback, or you can add a note to it as well if you want to collaborate. I'm sure there's a lot of things in here that I probably didn't cover, but in the spirit of just trying to echo things that Hamel Husain and Dan Oster said, I think these are the main things I wanted to show off. And so I will, I will stop there. Um, and yeah, happy to take any questions or skip ahead for time management.

I think that was really good, Harrison. Like, I know that's a mind-dizzying set of features and things, but it's like actually helpful to visualize some of these things because like we talk about it in bullet points and stories. Sometimes it's helpful to see like what it looks like operationalized. Like for me, I know when I'm learning things, it's helpful. So That's, that's why I wanted to have someone like you, like, show that, because it really, like, helps glue the intuition of, like, okay, how does it work? Um, I think, Dan, what do you think? Like, I think, uh, because of time, maybe we redirect the Q&A for Harrison into his office hours, uh, because we have 3.

Yeah, and the other, the other thing that would be nice about that is, uh, I should have thought about this more ahead of time. When I look at the Q&A, uh, many of those came in while you and I were speaking and aren't necessarily LangChain-specific Q&A. Um, so if you see any in the Q&A that you especially want to, um, pick off, Harrison, that's great. Otherwise, I like the idea of, um, doing it during your office hours, and then we'll get questions that are like more narrowly targeted to you.

I'll also— you know what, that sounds great to me, and I'll go through the Q&A right now and respond to them because I think I can also type out answers in that way. Multitask.

All right, sounds good.

All right, thank you guys for having me. See you.

Thank you.

Yeah, um, I think— okay, next we're going to have— Brian's next. Yeah, Brian's next. And so let me just introduce Brian a little bit, if you don't mind. Um, so Brian is a friend, a good friend. Um, he is a brilliant, uh, machine learning engineer, data scientist. Been working in the field for a while. He has this great book on recommendation systems, which is actually very relevant to LLMs as well. And what I want to do is like, so a lot of this eval stuff is very use case specific. I wanted to bring another expert in who I know does really good work and have him describe, or, you know, have specifically Brian describe like his approach to doing evals and instrumentation and things like that, and like walk you through his workflow to give you just yet even another perspective. So with that, Ryan, I'll hand it off to you.

Thank you so much. Yeah, I'm going to talk a little bit about some of the things that you've already heard. I'm going to underscore some of those lessons, but ultimately I'm going to try to give you a sense of how I think about making this real. I always click that wrong button.

No worries. Looks like it's frozen.

I always think share means share the— and so today I'm going to talk to you about Spellgrounds.

Can you hear me?

Yeah, I can hear you.

Okay, cool. Uh, today I'm going to talk about Spellgrounds for Prodigious Prestidigitation. So you're going to see a little bit of magic-themed stuff today because I work on a project called HexMagic. But Spellgrounds is the name of our internal library for developing and running evaluations. It is going to be a combination of like sort of like systematic evals and use case specific evals and unit tests and regression tests, all that all in one thing. And we're gonna talk about why. But right off the bat, I wanna give you a, a very opinionated position on what evals are about. Eval serve one or more of the following three purposes: they help you understand when a capability is good enough to present to your customers, they help you sleep well at night, they give you confidence that your system's not behaving badly, or they help you debug later when things go awry. I am pretty convicted that these are what evals are for and what they're about. When you think to yourself about the things that you've learned so far in the course and the lessons that you've heard about good evals and how you can use evals, they usually, in my opinion, always fall into these 3 buckets. You'll also see this in the market when you talk about evals tools. So I'm gonna talk about these 3 lessons as part of the things I cover. So first up, I'm gonna tell you a little bit about some things you should avoid. We're gonna call this miscasts and fizzled spells. These are mistakes I've already made or I've narrowly avoided. The first one that I see people make a lot is they think LLM evaluations are entirely a new field. They're not. They've been around. We've been doing it for a long time, and the experts are data scientists. We've been mapping complicated user problems to nuanced objective functions for over a decade. Some of us personally for over a decade, and the field for close to 20 years. We've got really good instincts here on what it means to measure unpredictable outputs. You may think that the nuance and the beauty of your LLM outputs are some ineffable thing. They're not, I promise you. I believe that you can coerce whatever it is you're trying to get generative AI to do for you into reasonable and quantifiable performance. So let me give you some examples. For code generation, you should be considering things called execution evaluation. Run the code and see if it does what you expect. In my workflow, flow. I need to evaluate SQL and Python and R. I run the code that's generated by the model, I compare it to the code that I've run in the target setup, and I make sure that the outputs have the same state. That's how you evaluate code generation. Right now everybody's really excited about agents. What does it look like to evaluate agents? Well, one important step is planning. You should be thinking of that as a binary classification dataset. The steps in the plans, you should think about which one are required steps, which one are steps that have some looseness. Turn those into binary classifications. Turn that entire state machine into a set of binary classifications. You may think, okay, Brian, well, those are easy. What about something complicated like summarization? You should be checking for retrieval accuracy. Does it include anything that references the important points that your summarizations in your target code or your target response include? So this is probably the biggest, I would say, trap that people fall into is they think that they can't make their evaluations like old school data science evaluations. Here's an example of how I compare the output of data science code. It's very simple. All it is is massaging the data frames and saying, is there any way possible that these data frames contain the same data? This is a blocker that I hear a lot of people express, and actually during my interview loop, people tell me that they can't think of how to evaluate data science code other than checking if the code's the same. Well, if I ask you, how many customers do we have this month? Well, the response from the agent may be a different DataFrame shape, but as long as it has that one number in there somewhere, then that's good enough for me. This is the kind of code that you write. These are called relaxations. Okay, next up. People fail to include use case experts in eval creation. Your users or experts on the use case of your LLM application, they will understand what good looks like. If you were thinking about rolling out an LLM application without talking to experts about what the final state should look like, I think you're goofy. Go Talk to them, understand what they want. While we were building Magic Charts, which is our sort of like LLM-generated data visualizations in our platform, we talked to the data team. What are some example prompts to go from existing notebooks into the target chart, the perfect version of the chart? We worked with them to build out a set of them. It looked like this. The prompt: create a bar chart to show the average number of passengers per month using the flights dataset, blah blah blah. And this is the chart that they wanted to see. So what do I check in the output of the LLM? Do I check that it looks like this pixel to pixel? Hell no. Do I check that it's selected the right chart type?

Yes.

Did I check that the time axis is a date time axis for X?

Yes.

Do I make sure that it's doing an aggregate function on the Y-axis? Yes. Noticing that these are all starting to look like binary, you evaluations. Next up, people wait too long to make evaluations. It should be part of your dev cycle. This should literally be part of the RFC creation. Here is a literal snippet of the RFC for a recent project where we were working on sort of a new editing model. This is in the RFC. It's required to be part of the RFC now. And you're required during the dev cycle, whether you're a full-stack engineer, an AI engineer, or a product manager on the team, you're required to be writing and thinking about evals this early in the process. There's no excuse to get something close to prod and then say, well, I should write some evals. I think people fail to recognize product metrics and evaluation metrics are similar. But they're different. You'll notice this, look at your data. I promise I wrote that before today's lecture from Hamel Husain and Dan. We are all in alignment that you have to look at your data, but you shouldn't mistake your product metrics for your evals. They are distinct. Product metrics will give you a lot of intuition about great evals. These production logs aren't sufficient for building evals. We saw some examples of digging into production logs to give us some intuition for how things are behaving. That's really important, but that can't tell me all I need to know for making great evals. What it can tell me is sort of how to build the next set of evals. I don't personally have access to my customer's data. I can't go into their data warehouse and query it, so it's actually not possible for me to build evals on their warehouse. That's a good thing. But what I can understand is the kind of questions they ask, and then I can go build datasets and custom environments that I think are representative of those use cases. So this is the push and pull of product metrics and evaluation metrics. Buying an evaluation framework doesn't make it easy. Now, this might feel like a little bit of counter-programming. This is not intended to be a criticism of any specific eval framework. But what I can tell you is I don't have an eval framework to sell you. And so let me tell you what I feel about this topic. The hard parts of the evals is not the library. I built a simple framework on top of Unitests in a few weeks. It lasted 9 months. We ultimately had to rewrite it, but that was also one sprint. The best eval product is a Jupyter Notebook. It doesn't have to be a Hex Jupyter Notebook, just a Jupyter Notebook. Interact with your data. Get your hands on the damn data. Be able to slice and dice. Eval are hard because they, they require you to understand the user stories and the diversity of user inputs.

Sorry.

And finally, eval companies, they haven't put the LLM applications into production. Remember the saying, don't ask a shoe seller What's going to help you dunk? Ask Michael Jordan. I'm here to tell you, I've put this stuff into production. I promise you, you do not need an evaluation framework until you start feeling the pain. Then it's time to think about it. This is what my evals framework looks like. One class. This is me setting up the idea assertions. How do you evaluate in a very flexible and relaxed way whether the eval namespace and the target namespace, i.e., what the agent responds with and what happens after you evaluate it, and your target look like? This is all the wrapper code. Please, please, please invest in the things that matter. Don't invest in complicated integrations early. Reaching too early for LLM-assisted evaluation. You're gonna hear a lot of people tell you about LLM as a judge. There are no free lunches. I think LLM judging is extremely valuable, but it is not a free lunch. What it's going to give you is directional metrics, things to look into. It's going to start giving you some hints. Once you have built some real evals that actually like kind of give you important signal, LLM Judge can help you scale. LLM Judge can also help you look at production events to kind of turn those into great evals at scale. But you have to do this methodically. Do side-by-side evaluation on new treatments. Use multiple judges, multiple models, multiple shots. Check for human alignment randomly and periodically. The best paper I've seen, the best, like, writing I've seen on this topic is by Shreya Shankar. Check it out. This is what it looks like to do LLM judging systematically over time.

This is—

we made a relatively minor change in our context, and we ran 7 experiments here, and we wanted to see the performance on different versions of the model. And each time we're using the judge to tell us what's better, old or new, old or new. And we're doing this over 2,000 evals. So this is what it looks like to try to use LLM as a judge, as a tool. Part 2, moderating Magic. How do you build an eval system yourself? Magic is an AI copilot for data science that lives in Hex. It can generate SQL that's specific and knowledgeable about your data. It can string cells together using different kinds of cells to write polyglot code chains. SQL, Python, R, and native charts. It reacts to code edits the user does or asks the user— or allows the user to ask for edits and fixes specifically. First up, RAG evals. You should evaluate your RAG like a retrieval system. RAG is retrieval. Treat it like such. If you start thinking about chunking and indices and multi-index and hybrid and— no, Take a step back, label some data, take the queries from your evals, and produce the best documents for those queries, and measure the hit rates. That's all you need to do to measure RAG. Don't get too excited about your RAG system unless you have a clear baseline. Your retrieval scores, whether the semantic or lexical, aren't calibrated either. So don't treat them as confidence estimates. Just a little tip. I've gotten burned by this. This is what it looks like to evaluate RAG. On the right-hand side, we're looking at different re-rankers. On the left-hand side, these are all the embedding models we try for one particular workflow. Don't be shy, try a lot of things. Look at the variance in performance over there. That should tell you that this is important. Planning evals for agent-based systems, you have to Evaluate your planning. If you're using a state machine, treat it like a classifier. Check its choice at every step. If you're asking planning to include downstream prompt generation, the quality of those downstream prompts are probably shit. They were for us at least. It took me a non-trivial effort to get the planning prompts to be anywhere close to what the human prompts look like. Don't forget to evaluate that. Agent-specific evals. This is the easiest kind of eval. You have an agent, it does a specific thing, good. Test it. Think about structured output. How often can you get the agents to respond with structured output? And if you do, how can you tie the agent relationships together with tightly spec'd out API interfaces and then evaluate that consistent— consistency? Final stage evals. A lot of agent chains need a wrap-up or a summary. Don't forget to eval this too. Sometimes you get kind of stupid shit. You'll get the summary talking about things that the agent never did. That's terrible as a user experience. Trust me, I've accidentally done this. Too much context about the agent chain can make these quite bad. Don't serve everything from the entire workflow to the final stage eval. It will be noisy, and frankly, the tokens get kind of crazy. Finally, experiments in LLM generation are repeated measure designs. They're not A/B tests. When you make updates and changes and bug fixes to your agent's workflow, treat it like an experiment and measure it thusly. Doing betters on your evals is is a good check, but you have to test for significance. And don't be afraid to rerun production events through the new treatment— so historical production events to the new treatment— and use automated evals to compare. You're logging your production events, aren't you? You didn't forget to do that, did you? This is what it looks like to run experiments. On the left-hand side, this is repeated versions, variants of a different, uh, approach, and you can see the error rate is going down on historical events. On the right-hand side, that's LLM as a judge. Don't sleep on this. And then I have one bonus lesson for you: production endpoints minimize drift. I thought, we've got this production system, I'm going to build a clone of the production system in my evals framework so that I can keep things as similar as possible to production. And some of you are already shaking your head, Brian, you're an idiot, that's never going to work.

Of course it didn't work.

We had to refactor our evals framework because we built a clone, tightly coupled systems that aren't actually identical. This is standard software engineering, like, don't do it. And I did it and I regret it. Don't be like me. Make your evals framework directly connect to your production environment. Make them endpoints. Call those endpoints. Use that as the same backbone that calls every single thing in your evals framework, but make sure that every step is exposed. Be able to hook in halfway through that workflow Make modifications upstream and see what happens in the end. That's how you keep these tightly coupled in a sane way. That's what I have for you today. You can find us at hex.tech. You can find me on Twitter @bebischoff or on LinkedIn, Brian Bischoff. I'll pause there for any commentary, but I think we're on a tight schedule today, so.

Okay, I'll ask you one question from the audience.

Of course.

How are you organizing all your unit tests? Where are you running them? Local, GitHub Actions, et cetera. And where are you logging them to in practice?

Yeah, so, um, we are very fortunate that Hex Notebooks can be scheduled. And so I run Jupyter Notebooks that are scheduled. So believe it or not, I orchestrate and run all of my evals in Jupyter Notebooks. No, no asterisk there. That means that A, they are incredibly reproducible. B, if I go back to an individual experiment and I say, huh, that performance is very different than I expect, I can pull the individual eval logs and literally look at every single response from the agent. That's an amount of power that no other system is going to afford. Now granted, you could set this up to log to your data warehouse and you could use a downstream sort of notebook to like evaluate all this. But frankly, like, I'm very fortunate that I just have access to Hex like this, so I just do it in Hex.

That's really fascinating. That should be a whole podcast in itself about notebooks in production. Could go in like— I would love to probably talk about that.

Happy to.

Okay. No, that's good. We have a lot of questions, but I think for time management's sake, I guess like we can What do you think, Dan? Should we do one more or should we move on?

Yeah, let's do one more.

Okay. Um, okay, so Alex Strick asks, for unit tests on their own to take a very short time to run, but lots of these individual tests together take a really long time, do you have any tips on batching these together? Is there a good framework for running through a bunch of these queries in parallel? Um, yeah, and he says, I have a bunch of these unit tests in my code base but still take a few minutes to run through a eventually.

Yeah, so we have a batch version of the evals that runs on a schedule, and I don't— like, you know, it runs on a Friday and I like collect my results later on. Um, but actually, um, again, like, I'll hearken back to the good old days of data science. Um, don't be afraid to do two things. One, look at the sample that is most concerning to you, and two, don't be afraid to use bootstrap sampling. Like bootstrap sampling gives you a really good picture of the overall, like, behavior of the population from a couple random samples.

Do that.

That's a great way to get early signal on these things. But also, let's be real, some of my evals almost never fail. Some of my evals fail every time, and that's a good thing. If your eval suite is 100% passing, your eval suite is not hard enough. Make it harder. You should be really targeting evals that succeed 60 to 70% of the time, because otherwise you're missing out on the signal of improvement. How will you even know if you've made anything better? And so the way I tend to look at this is I tend to look at the marginal ones, which ones flip when I change things. If I can't get them to change, actually, to be honest, I'm eventually going to delete them. Like they're not doing much if they're never failing. If we've moved on in the world from GPT-3.5, like, quality on a certain test, well then why do I need to test it every time?

That's what makes sense. As data scientists like to say, there's no variance, there's no signal. Um, okay, that's great. That's a very great presentation, Brian. Really appreciate that. I think people are commenting that they really love this presentation, by the way.

Um, awesome, awesome. Glad to hear it.

So the next guest that we're going to have is Eugene Yan. He's also a good friend who I've known for some time now. He is quite prolific and writes a lot about machine learning, large language models, you name it. The thing that he will be— sorry, Eugene is a senior machine learning scientist. I hope I'm not getting the title wrong at Amazon. I'll let him correct that. But what he's going to be talking today about is metrics. So a lot of people are asking like, what metrics should you use? We're talking about like, okay, measuring, writing evals, having metrics, but I haven't like really gone into metrics. So Eugene is actually going to talk about that in more detail. Eugene.

Thank you, Hamel Husain. That's way too kind. All right, everyone, we are almost at a 1.5-hour mark. And I will be going fast, and there's going to be a lot of code. And in fact, it's just all code and graphs. So I hope, um, you all pay attention. I'm dropping the notebooks that I will be sharing in the Slack, uh, in the Discord right now. So don't bother to take notes, don't bother to do screenshots. All this is fully available. So all I have right now is 6 cells worth of slides and 3 notebooks. All right, let's go. So the question I have is, how do we evaluate some summaries on factual inconsistency or hallucination?

Right?

Everyone's using some kind of LLM to summarize stuff, but how do we know if it's actually correct? Well, if you actually look at it, we are human eval, you'll find that hallucinations happen about 5 to 10% of the time, and sometimes it's pretty bad. So the way we can do this is we can evaluate the model. Right? The input is a source document and the summary, and optionally the label if you're fine-tuning on it. And the output is the probability of the summary being factually inconsistent. So we can frame this as a natural language inference task. So natural language inference is a classical NLP task whereby, given some premise and hypothesis, we have the label of it being— so if imagine the premise is John likes all fruits, the hypothesis that John likes apples is entailment. And the hypothesis that John dislikes apples is contradiction. And of course, there's— we also have a neutral, which is we can't have— we don't have enough information to tell whether it's correct or not. Now, if we apply natural language inference to factual inconsistency detection, what happens is that we can use contradiction, the contradiction label, as factual inconsistency. So imagine we have a document, maybe this is some talk abstract for a previous talk I did. Eugene's talk is about building LLM systems, etc. So the summary is the talk is about LLMs, that would be entailment, and the summary of the talk being about apples, that would be contradiction. So then all we have to do is to get the probability of contradiction, and there we have it. We have a factual inconsistency classifier or hallucination detector evaluator model. All right, so my objective here is I'm going to show you how to fine-tune an evaluator model. That can catch hallucinations on the factual inconsistency benchmark. This is my new Kaggle. Then we're going to eval the evaluator model through each epoch, and we're going to see how we can blend data to make this eval evaluator model way better. Then optionally, you can use this evaluator model to then eval generative models. I know it's kind of meta, but we're going to eval the evaluator model, which then evals generative generative models. And then you can also use this evaluated model as a guardrail, right? You're going to summarize things in production, and then you can check, hey, is the summary factually consistent or not? We're going to see how this happens. So this is a three-body problem. We'll first examine, prepare, and split our data. Note, this is completely coincidental. I have not spoken to the instructors about the importance of looking out at your data, but everyone has mentioned that, and I do have a notebook for that. That's what we're going to fine-tune on the factual inconsistency benchmark, and then we're going to blend in data from the unified summarization benchmark and see how that helps. Okay, I have a few appendices here. I have 3 write-ups on LLM evals and hallucination detection and off-domain fine-tuning, all of which are compressed into a 15-minute session for you here. And I also have some slides on evals and fine-tuning and how they are two sides of the same coin. So next, let's prepare some data. So over here, we have the Factual Inconsistency Benchmark. It contains one-sentence summaries from the CNN, Daily Mail, and the X-Sum news articles. We exclude the CNN, Daily Mail data because it's pretty bad. I won't show you how bad it is, but you can look at it yourself. So now here's an example of the X-Sum data. The first 2 rows, you see that they have the same input, and the next 2 rows have the same input. So how this looks like over here is that for the same input, we have a choice that is inconsistent and we have a choice that is consistent. So if you, if you look really hard at this, you will be able to understand why is it inconsistent or consistent. But if you just briefly glance through it, you might actually miss it. And a lot of times I've been, I've been looking at this myself and I'm like, Wow, this is a really difficult dataset. So here's the CNN Daily Mail dataset, and you can see it's full of, it's full of XML tags, and quite a number of them actually have ads in there. So actually, we just discard them. So FIB starts with about 3,600 rows of data. After we excluded the CNN Daily Mail dataset, we are still left with 3,100 rows. The authors themselves of this dataset didn't even bother using CNN too much because I think they just labeled 100 and they realized actually it's not that, not that good, and they just invested more of their resources on XSUM. So that's the FIB dataset. So eventually what we're going to happen, what we're going to have is, and of course we split it, uh, we split the data, we're going to group it by input. This ensures that the same article doesn't appear across train and val, right? So we want to only make sure that the same, uh, data only appears in either train, val, or test. So there's no data leakage. So eventually what happens is we have— and then of course we try to balance the data so there's only one positive summary and one negative summary. Eventually what happens is we have 700 data points for training, of which is 350 positive and 350 negative, and our val sets are only 150. So that's the factual inconsistency benchmark. Next, let's look at the unified summarization benchmark. The unified summarization benchmark is slightly different. It's based on summaries of Wikipedia articles. So you can see over here that again, the first 2 rows, they are the same, and the next 2 rows, they are the same. And when you look at it very carefully, uh, let's, let's look at the second row here. It actually points out what the inconsistency is, and it's highlighted, it's surrounded in the red boxes, which is the summary actually includes this additional data which is not in the source. But there are times When you look at it over here, I— it took me quite a while to spot what the difference here is. And the difference here is the word 'the.' And even though not having the word 'the' is not a hallucination problem, it's more of a grammatical or sentence structure problem, it is part of the dataset. So suffice to say, I didn't clean this up, but it just goes to tell you that you do need to look at your data when you're fine-tuning to try to understand the quality of it. Okay, so we We get the data, we try to prepare it, and what happens again is this. We have a summary sentence where the label of 0 is correct and the label of 1 is incorrect. And of course, we do the same thing, train, test, val split, etc. In the end, this is the amount of data we get. So we're going to first look at the factual inconsistency benchmark. So over here, we load the data, we do some— we tokenize them all up front in a batch. And then over here, we're going to fine-tune our model. The model we're going to fine-tune is what we call Distilled BART. Essentially, it's— you can think of it like an encoder-decoder version of BERT from Meta, but it's also fine-tuned on MNLI, which is multilingual natural language inference data. And of course, we have our parameters here, nothing too interesting. We use LoRA, we apply LoRA on the QKV vectors, our projection, etc., etc. In the end, the number of— the amount of trainable parameters is less than 3%. This allows us to fit this on a very nice small GPU. So over here, I have some custom metrics that I track. So this would be akin to the callbacks that Wing mentioned about during the Axolotl session. So over here, at each epoch or at each eval, I actually firstly, I preprocess the logits. So if you recall, NLI actually produces 3 probabilities. What I do is I only, I only care about the entailment and the contradiction probability. And I do a softmax on them, which sums up the probability to 1. And then I just get the probability of contradiction. So essentially those I care about. And then I also compute some custom metrics that I have, essentially PRAUC, ROC, recall and precision. And recall and precision, I have an arbitrary threshold of 0.8. Where I want the model to be pretty, I want the model to be pretty confident. Okay, so I have a custom trainer here. I won't go into it right now, but I'll go into it in the next, in the next notebook. So again, some standard training arguments, nothing special here. So let's— and of course I also have some plotting code here, again nothing special here. So let's look at how the model performs before any fine-tuning. Before any fine-tuning, you see that ROC AUC is at 0.56. If you recall, an ROC AUC of 0.5 means that it's really a coin flip and the model is not doing anything better than chance. So you can see, okay, ROC AUC is coin flip. And what I really love is the graph all the way on the right, which is the overlaps of red and greens. The reds are the— we have the ground truth, we know what is inconsistent, and the greens are what we know is consistent. And we can see the overlap, and you can see that in this case, the model just cannot distinguish the overlap. It cannot distinguish between them before any fine-tuning. Now note that this model has already been fine-tuned on MNLI, but still is doing a pretty bad job. So now let's start fine-tuning. And so you can see these are the custom metrics I have. Usually if you don't have any metrics, you probably only get loss, training loss and training loss, but I do have extra losses. I do have extra metrics. Such as PRAUC, ROC AUC, recall and precision. So let's just focus on ROC AUC. You can see that ROC AUC increases from 0.56, which we have over here, to 0.65. Not too bad, but at least it suggests that the model is learning. But what is very disappointing though is that the recall at 0.8 does not go above 10%. So in this case, this is just not usable, right? We— this model just cannot identify factual inconsistencies. So we check the evals after fine-tuning. This is— note, this is on the training set. We train on this and we just check on this to make sure that our model can learn. We see ROC AUC is 0.71, and you will start to see a little bit of separation of distribution, right? Start to see a little bit of separation. But on the validation set, ROC AUC is 0.66, not that good, and the separation is pretty bad. And over here on the test set, which we have never seen and which is we are not using to pick our checkpoint. ROC AUC is slightly— I think it's not statistically significant, but you can see the separation of distribution is pretty bad. So next, let's try, let's see how we can fine-tune on a different dataset. We fine-tune on the Unified Summarization benchmark followed by the Factual Inconsistency benchmark. So the Factual Inconsistency benchmark in this case is the data that we care about. You can imagine in your own use case, you would have some summarization data and all you care about is how it performs on that data. What you can do is you can take all this open-source permissive use datasets, which both of these are, and you can use them to bootstrap your own model by blending in the data. We'll see how you do that here. Again, to remind you about our data, our factual inconsistency benchmark, we only have 750— we only have 700 training samples. But the Unified Summarization Benchmark, we have 5,000 training samples, and I deliberately split a big chunk of it into the validation set, right? So you can see that USB has almost 10x more of the factual inconsistency benchmark. So by using some external data, you can improve your own models. And in this case, what I care about is hallucination detection on summarization. So again, we do some kind of batch tokenization so we don't have to tokenize on the fly. And we set up our models, nothing, nothing special here. So what is this custom trainer? This custom trainer is because at a point of me trying to do this, which is about 6-7 months ago, Hugging Face trainer didn't allow for evaluation on multiple datasets. So this is really just copying the trainer code. And overriding some of the, overriding some of the methods that it has, specifically eval and maybe log_eval. You can just use this code if you want to. So how it looks like here is that without the custom trainer, all you could do was this, which is eval_dataset, you provided a single eval_dataset. But with the custom trainer, what you can do now is this, you can provide a dictionary of dataset to the eval dataset, and it would just work. So again, we have our usual, uh, we have our usual visualization code, and this is the same thing. USB, what we saw, ROC AUC 0.56. Um, this is— oh, sorry, that was FIB, the dataset we care about. ROC AUC is 0.56. And this one for USB, ROC AUC 0.66, a little bit better. But the divergence looks a little bit funky. In the previous case, you can see it was a little bit too strict. Most of the probability was about 0.75. Over here, it's a little bit too lenient. Most of the probability was close to 1.

So let's look at this.

So what we're going to do is we're going to fine-tune this model. And let's just focus on the USB metrics first, which is USB PRA, CROC, da da da. You can see that our, the USB-ROC AUC very quickly ramps up from 0.76 to 0.94. That's pretty freaking amazing. And recall and precision is 0.75 and 0.93. That's really good. I don't think in production, I don't think your data is going to be as clean and you can achieve something like this. We also probably want to tweak the threshold to maybe either bias towards recall or precision. But let's look at the— let's look at FIPS ROC AUC. You can see FIPS ROC AUC, and then what I'm going to do is I'm going to split the screen here for a second. So you can see previously FIPS ROC AUC went up to 0.65, right? Over here, by solely fine-tuning on USB, not a drop of FIPS data, we could get the FIPS ROC AUC to go up to 0.64 or 0.63. That's— what the heck is going on, man? I mean, it's the same task, but these are completely different domains. But the thing is, look at the recall. The recall has gone up from 0— from 8% to 25% to 25%, etc., etc. That's a huge world of difference from what we were seeing previously, right, where we'll start before at 6%, 4%. So what we're going to do is we fine-tune on this, and we can see that it's superb on USB. Uh, over here you can see that it does superbly on USB, and this is the validation set. You can see, oh my goodness, the ROC-AUC is freaking amazing. And look at the divergence right over here. You could probably just cut at 0.8 or 0.9, and you have the clear factual inconsistency. Or over here you cut at 0.1 and you have the clear factual consistency. It depends on how conservative or how lenient you want to be, uh, or with your false positive, false negatives. And now let's look at it on the validation set for FIT, the dataset that we care about. We see that the ROC AUC has not improved that much, right? So this is the previous ROC AUC, uh, this is the previous one, and there's a current one. It's like 0.66 versus 0.64. The separation is not that good. So we may feel like, hey, something didn't work here. But now let's give it the same 10 epochs of FIP data which it has never seen before, the same training. Okay, so you can see the same training over here. Previously, ROC AUC was 0.65 and recall never got above 0.6. Now with this, you can see FIP ROC AUC, and we don't actually have to measure it on USB. I just because I was just curious how much adding in additional data would cost us in terms of alignment tax on USB. You don't actually have to track this, but I did. Oh my God, the ROC-AUC immediately started at 0.75 and then went up to 0.86. And look at the recall, it's like 10x higher than what we had previously, 0.57. And precision is 0.8, 0.93. That is insane.

Right.

So all in all, these are some metrics on how I would use to evaluate a classifier model. So now this is— here's how it looks like in the end, our FIP test set. So you can see the test set previously with only FIP data on the left, and here's the test set with FIP data and USB data on the right. And this model with maybe a little bit more data, we can probably get get to higher recall, maybe 0.8. And I think a good balance is 0.8, 0.8 of 0.8, 0.8 in terms of this evaluator model's ability to catch hallucinations. But so now you can do this to evaluate your generative model, right? I mean, we have certain n-gram metrics like ROUGE and METEOR, or you can use LM as a judge, which is really expensive. But this model is super fast. Every query is like 20 milliseconds. Super fast, super scalable, and it's fully within your control. And now what I will ask you is, how could you fine-tune evaluator models to help evaluate on your task or evaluate summaries on relevance, information density? And okay, that's all I had. So now, uh, so long story short, this is my evaluator model to evaluate my summaries on factual inconsistency. And eventually there will be more evaluative models to evaluate on relevance and informational density, etc. So now go and fine-tune your own evaluative models. That's all I had.

That's really great, Eugene. I think that was like really great run-through of like a lot of different things that you might want to think about with like concrete examples. I really recommend everybody check out these notebooks. I went through them a little bit fast, but you know, this is recorded, you can always slow it down, you can step through these notebooks. Also, um, you know, at times Eugene shared some additional resources for like his writing, and that really gives a lot of color to these things. I highly recommend you, uh, like check all these things out. Um, you know, make— just make sure you don't skip that because I, I really enjoy that, those writings myself. Let's see if there's a, let's see if there's a question. Let me open up the Q&A. Okay, let's, someone's asking, how do you think about evaluating agents?

That's a good question. I would just echo what Brian said, which is evaluating agents as a step-by-step process. I would break it down. I mean, I actually have a recent post that I just wrote over the weekend, which is, it's essentially about prompting, but I can talk about how we evaluate agents here. So one of it is you can split the catch-all prompt to multiple smaller ones. So here's an initial prompt where we try to extract some data. We try to summarize a transcript, right? So over here you can break down the transcript into and extract the list of decisions, action items, and owners. This is just a classification metric, right? Simple. Now over here, the second step, we ask it to check the transcript and the extracted information to make sure it's factually consistent. Again, classification metric. Now the final one is given the extract, given the extracted information, write it out into parentheses or bullet points. Now this is an informational density, it's a relevance, it's a writing eloquence question. So I know—

Hey Eugene, do you mean to be sharing your screen?

Oh shoot, I did, I did.

It's okay, I did that earlier and it was, it's no problem. It's hard.

So sorry guys. So here's how I would Imagine you have an agent, again, to summarize meeting transcripts as the simplest example. You could, over here, you could evaluate how well the agent is extracting a list of decisions, action items, and owners. Again, this is just a classification, right, or extraction.

You can—

it's just precision and recall. Now, again, another one to check the extracted information against a transcript. You can imagine that this is the factual inconsistency model we talk about. Think step by step and check if the extracted information is factually consistent. Again, this is a classification task. Now finally, uh, you are asked to rewrite the information into bullet point summaries. You can think of this as an information density task or writing style task, which is a little bit more soft, uh, not as straightforward to measure via classification, but maybe a reward model might work. So that's how I would do it. And AlphaCodeum had a really great post, really good piece where they actually split up code generation, right, into multiple steps. So you can see each of these steps here, you could probably evaluate each step here. So that's a long-winded story to say, long-winded answer to how I might evaluate agents the same way as I would evaluate a multi-step workflow.

That's great. Okay, well, thank you, Eugene. Um, the next speaker we have— let me— I just want to give you some background for the next speaker. So a lot of times when I talk about evals, as you can tell, there's a lot of information about evals, like how to organize them, what tools you should use, metrics, uh, workflow, things like that. I always find with all of my clients is that people get stuck, like How do you write evals? Like, where do I even begin? How do I think about it? Maybe, and they can maybe write one test or two tests, but then they kind of have a mental block. And I think like we're still in like a very nascent period of all this tooling and like how to go about thinking about evaluations. Shreya, who is, you know, who is a prolific researcher in this area of like LLM ops. And also specifically evals, has done a lot of research on things like UX, workflow, developer tools, and tools more generally. She's been doing research for a really long time, even prior to large language models, on machine learning tools and workflows and things like that. And so I'm going to hand it over to Shreya. Shreya is going to walk through some of the research that she's done around workflows around evals and ways to think about evals that I think is really helpful for everybody. So with that, I'll give it over to Shreya.

Thank you, Hamel Husain, for the great intro. Super nice of you. I think, Eugene, you have to stop sharing so I can share. Sorry to boot you off.

Shoot, I didn't know I was sharing.

All good. Okay, so Let me know if you can or can't see my screen.

Yeah, I can.

All good. Great. So today I'm going to give a pretty short talk on some recent research that I've been doing with a lot of— with many amazing collaborators across many different institutions and companies. And the theme of the talk is, you know, how can we use LLMs or AI to scale up our own human decision functions? How do we scale up the vibe checks that we know and trust with LLMs? And I have a brief intro slide in case people don't know who I am or in case Hamel didn't intro me, but I can skip this. But basically, I am a PhD student. This talk will be a little bit more high level. I study how do people write LLM pipelines, how do people evaluate them, and how can we improve these Experience for everyone, and I also do ML engineering at an AI startup. And I like to think about, you know, how can we work with data at reasonable scale? How can we ensure good data quality and anything around the people in ML and MLOps? So without further ado, I will get into my talk. I don't need to go over this, probably. I mean, you've been in this workshop for so long, but, you know, We really, really like LLM pipelines because their zero-shot capabilities can enable intelligent pipelines without having to train models. So if you take a look at a bunch of prompt templates from LangSmith, you'll see that people are doing all sorts of stuff. You know, they're using LLMs to write code reviews. They're using LLMs to convert YouTube transcripts to articles. And when you kind of write a prompt template or pipeline around this, I'll illustrate with this figure. So say we have this YouTube transcript to blog post pipeline. We might feed in some input document, which is a YouTube transcript, put it in some prompt template that has some instructions, and then expect to get some transcript— or sorry, some blog post out of the entire pipeline. So this all looks great, but the problem is when you try to do it at scale, the LLM doesn't always listen to instructions. So maybe you have an instruction around HTML structured, and 1 out of 10 times, it doesn't output improper HTML structure. Or maybe you have a sentence that says, you know, avoid copying sentences directly from the transcript, but somewhere in the middle of the document, even GPT-4 might exactly verbatim output the same instruction. And the theme here is, you know, no matter what, as we suppose we fine-tune these LLMs on our tasks, there's no guarantee that it's going to listen to every single instruction. That we've included in the prompt. We need some form of guardrails or assertions or evals to be able to quantify, you know, how well does the LLM do our task and listen to what we define as good or bad. And that's where our kind of vibe checks and rules and guardrails come in. An insight from traditional ML is to simply put rules and guardrails around the model to detect bad outputs and correct them, or even rerun the pipeline. But this is really hard to do for LLMs. This goes back to what Hamel mentioned before. It's difficult to even get started thinking about what does accuracy or good even mean for your specific task, your outputs. Maybe 10 of us are trying to write pipelines that are converting YouTube articles— or sorry, YouTube transcripts to articles, but maybe we all want different formats or we want the response to do something slightly different, right? So our vibe checks are all going to be different. Making this a pretty hard problem. You can't just kind of use what somebody tells you off the shelf. You've got to come up with your metrics yourself. Metrics might be complicated, requiring humans or even LLMs to evaluate. Say something like tone. If we want the tone of a blog post to be informal, or we want it to not sound like an AI, right? How do you encode that into something to evaluate? And every prompt task application is different. All of us are going to have different metrics, even if we're trying to do the same task, or different implementations of the metrics. If we're trying to do the same task. And I like to think about these vibe checks a lot, or guardrails, or evals, or whatever you want to call it in general, along the scale of generic to task-specific. On one hand, we've got common NLP metrics that model providers talk about when they release new models, which is great, but doesn't really tell us how well those models are going to do for our custom tasks. You've got something in the middle where we know of, you know, good metrics for common architectures. For example, RAG pipelines. We know that faithfulness is a good metric from the Ragas paper. So that's great. But we really also want to be pushing towards these task-specific metrics. So if you have an exact structure that you want your output to follow, not just JSON, but a specific, you know, you want at least 2 of those JSON keys to follow the same pattern. You, you want some more finer-grained constraints on that, you know, that goes more towards the vibe checks. And then if I showed you one axis in the previous slide from generic to task-specific, but there's also another to consider, which is how simple or scalable is the method. Stress testing prompts in ChatGPT don't really scale, especially in production. And then fine-tuning evaluator models are pretty high effort because you have to constantly be collecting data and determining whether this is good or bad, and then be able to fine-tune the model. Um, vibe checks performed by humans don't scale, but we shouldn't discount them because most people do this and they're quite effective, especially in the early days of prompt engineering. And what we really want to do is move these vibe checks towards the upper right quadrant, um, and codify them. You can call them validators, you can call them assertions, you can call them guardrails. I honestly don't know what to call them, but the idea here is to have a set of task-specific constraints or guidelines that you feel confident aligns with what you think is good for your task. So a lot of our recent research has been in developing evaluation assistants, which are tools that aid humans in creating these task-specific evaluations and assertions that align with how they would grade. So in my talk, I'm going to briefly cover, you know, what do we want to think about and how can you build your own evaluation assistant? Instance. The key idea here is to use LLMs to scale, not replace, your own judgments and decisions. And I'll talk a little bit around different parts of the workflows and how we can use LLMs to do that. I'll start with how can we use LLMs to auto-generate criteria and various implementations of that criteria. And then I'll talk about a recent mixed initiative interface that we built to develop custom assertions and some lessons that we learned. From briefly prototyping this interface with a bunch of LLM experts. All right, so let me jump into bootstrapping criteria. Okay, so let's pose this concrete example here. Let's say we have a document summarization pipeline, but the summarization pipeline is for medical documents. So there are some additional examples, or sorry, additional instructions like return your answer in Markdown because you want it to be a report that your doctors are gonna read. In a custom interface, and maybe you also have some instructions like, "Don't include any sensitive information like race or gender and have a professional tone." So you can see how, you know, these kind of get pretty specific towards the end user. Having assertions gives you a fine-grained view of correctness or goodness of LLM input and output quality, and every custom LLM pipeline should have some table like I think, but the challenge really is coming up with the criteria. What are the columns that you want here? And then good ways to implement this criteria. So some of them, for example, can be implemented with code. Some of them might not be able to. You might need to use an LLM to evaluate something like professional tone. And engineering that prompt itself can be hard, right? You're already engineering your main prompt. So engineering the validator's prompts is a little bit excessive. So how can we enable humans to efficiently come up with good and bad examples of professional tone to seed the validator prompts, for example? So an overview of this problem, which we talk about in our Spain paper, is how can we generate a small set of assertions that have good coverage with what humans think are bad outputs and also have good accuracy? So challenges here are how do we find the right assertion criteria desired by the developer, and how should we guarantee the coverage of failures with a small amount of assertions, right? We don't want to give you thousands of assertions to run in production, thousands of guardrails, because monitoring that or visualizing that would be a mess. And our Spade system employs a two-step workflow to do this. First, we generate a bunch of candidate assertions with LLMs. And then we filter them based on human preferences. Our insight here for how do we generate custom assertion criteria is that the criteria are hidden in prompt version history. So when humans are iterating and improving on the prompt, we can tell what is it that they care about and what are unique mistakes that LLM makes. Maybe there are document summarization pipeline starts out with a template like this, which is, Very common. A lot of doc summarization pipelines will start out with a prompt like this. And then when trying it on their data, they might notice that sensitive information is included in the summary and the specific application developer doesn't like that. So they add an instruction that says, don't include the sensitive information in the summary. Then maybe we might see another human-generated prompt delta or prompt edit that says, do not under any circumstances include sensitive information. So what does this tell us? This tells us that the LLM is kind of bad at determining what does sensitive information mean, doesn't listen to that instruction. I don't know, come up with a lot of conclusions there, but so forth, right? You can imagine looking at how humans evolve their prompts to determine what it is they care about and what magnitude, right? If you edit the same line maybe 15 times, maybe that's a sign where the LLM is a little bit more bad there than in other places. So what we did here to build the first part of the SPADE assertion generator was to look at a bunch of prompt templates across different domains and categorized all of the edits people made to those prompts. And we came up with this taxonomy. Some examples of edits are maybe inclusion instructions or exclusion instructions. A lot of people have very specific phrases that they might want to include or exclude, which should be caught, right, in these assertion criteria. So using that, we can then seed the LLM to help us come up with assertion criteria custom to our prompt. We can maybe use ChatGPT even to just copy in that taxonomy, copy our prompt template, and then say, what are some assertion criteria that I should use based on my prompt, based on these edits, and come up with as much assertion criteria as you can. And the LLMs are pretty good at this. They're slow, but they're good. They at least find things that are aligned with mistakes that we might want to catch via assertions. So Spade first gets a bunch of natural language criteria and then generates a bunch of Python function implementations. I think the second part is less relevant, like if you want to use a Python function or a JavaScript function or you want to use the LLM-based validator, whatever it is. I think the key idea to take away here is that you as a human are editing your prompt, you have really good insights into what the failure modes are, and so how can you kind of codify that, maybe using such a taxonomy, into assertion concepts that you should think of? And we deployed this and we had a bunch of people try it out in a UI. We deployed this with LangChain, so thank you to the LangChain team for collaborating with us here. And we found that, you know, across the board, across different fields. Inclusion and exclusion assertions are most common. And we found a number of problems also with the assertions, right? Who knows if LLM-generated code is correct? We found redundant ones, we found incorrect ones. And then if you're interested in learning about how we solve those problems, you can read our paper for more insights there. Cool. Now I wanna talk about thinking about a UI. Around this experience. I mentioned you can, you know, use ChatGPT to maybe bootstrap some assertion criteria, but that's kind of underwhelming and requires a lot of back and forth, right? Maybe you go through ChatGPT many times, maybe you test it out in your Jupyter Notebook or the OpenAI Playground. You're jumping between different interfaces and trying to figure out how to make sense of it. If you are a developer, maybe at a larger company, or trying to build evaluation tooling at your company, right? How do you think about interfaces for that? So our, the main motivation for this actually came out of Spade just taking forever to run. How can we, you know, use humans more efficiently in this process? How can we— found that, you know, people wanted to improve and iterate on the Spade-generated assertions, and they also didn't feel like the assertions were aligned with their end goals or with their own preferences, partly because they didn't fully know their preferences yet, which I'll get to later. But the goal of this interface here, or thinking about an interface here, is, you know, how do you help people iterate really quickly and discover, you know, their own preferences on what are good and bad outputs and codify those into assertions as quickly as possible? So key idea here is, you know, we've got to support— we've got to minimize wait time when this entire evaluation process. So take a typical evaluation pipeline, looks something like this. You've got a prompt that you're testing. You've got a bunch of inputs, out and outputs. This latter part of the pipeline where you're generating metrics, trying to identify which metrics are good, trust your own metrics, and so forth. That part takes a really long time. So how can we include include a human in the loop. Maybe humans can edit criteria, refine criteria that LLMs come up with. And maybe humans can also interactively grade LLM outputs to figure out what are better prompts for the evaluators, the LLM-based evaluators. So our interface, which we also describe in the paper, is built on top of ChainForge, which is a prompt engineering tool. And the idea here is to go through this kind of workflow, the specific prescribed workflow. You start out with saying, I want to evaluate this prompt. Then you go to the B section here, which is maybe I want to grade some responses first to look at outputs to determine what criteria that I should include. Or maybe I want the LLM to just use a taxonomy and infer criteria for my context, whatever it is. So you might go through that, see the LLM-generated criteria, or add your own criteria, decide whether you want them to be implemented with LLMs or code. Then EvalGen takes you to a grading process, which you give thumbs up and thumbs down on different examples, and then EvalGen under the hood will determine, you know, what examples failed what criteria so we can use them as few-shot examples. For those validator prompts and so forth, engineers, your prompts there. And at the end, when you're tired of grading or you've graded all your examples, then we show you like a report card of here are the functions we chose, here are the alignment with your grades. And then you can also scale those evals up, those LLM-based validators up to all of your ungraded outputs. Um, with this table view.

So I like this a lot. I really like that interface a lot. Like, it's really cool because what happens is you start from your prompt and, um, you know, your large language model is used to like look at your prompt and kind of guess like what kinds of assertions, what kind of tests that you may want to write. And then it helps like bootstrap that, so it gives you a starting point. It's like a It's like writer's block. Uh, it like gets rid of writer's block for writing evals. It's really great.

Thanks. I'm excited to show you the V2 that I'm gonna have a screenshot.

Oh, nice.

Okay. Um, in here. Yeah. So I also know I'm running out of time. I can—

no, keep going.

Do you want me to skip directly to that?

Yeah, sure.

If, if people have to go, they can always watch the video and we've always run over. So you keep going.

Yeah, great. Okay, so speedin', speedin' away, but here's our interface. And, you know, it's a research prototype. It's super hacky, breaks all the time. So anyways, we decided, you know, how do you even— how do people even use an interface like this? This is fairly new, right? Figuring out how to help people, assist people in coming up with evals for their tasks in an interactive interface. I don't know if people have done this before, but certainly there's probably a lot we can learn just by putting that interface in front of people. So that's exactly what we did. We got 10 people. We ended up not using one study, but we got 10 people who are experts who have built LLM-based products and pipelines in production before. And we asked them to use evalgen in an open-ended way. We gave a sample task around named entity recognition. From tweets, a dataset of tweets, or they could bring their own task, which I think only one person did their own task. And we found generally people liked EvalGen as a starting point for their assertion. So zero people thought the assertions that EvalGen came up with up front were good, but they saw the value in, you know, unblocking themselves from moving forward. And we realized that, you know, this evaluation and coming up with good evaluators is definitely an iterative process. And people had a lot of mixed opinions on assertion alignment, which we dig into in the paper in more depth, but I'll talk about, you know, two big things that I didn't really expect going into the study that I learned. The first one is that we noticed as people were grading, their own criteria for what is good and bad output is drifting. It's a function of the output, it's a function of the LLM, it's a function of viewing more outputs, it's function of the rules that they include in the outputs, whatever it is. Grading outputs spurred changes or refinements to eval criteria. So not only were they adding new criteria, but also the participants were reinterpreting the criteria to better fit LLMs' behavior. So one great example of this was there's this instruction in the prompt that says, extract all entities from this tweet and don't include hashtags. Entities. So that was the criteria, no hashtags as entities. And we found that there were multiple outputs that included hashtags as entities. The first time people saw something like this, they graded it badly. The second time they saw the LLM, you know, extract the hashtag as an entity, they might say something like, hmm, I said no hashtags as entities, but I think the LLM did the right thing here. Colin Kaepernick is like a very famous American football player, and they thought that did something right. And then when they saw this failure mode again, for example, Nike being extracted as an entity, and they noticed, I'm failing everything. I actually think the criteria should be no hashtag sign in the output. And some people thought like the LLM was smart enough to keep the hashtag if it thought the hashtag was So like, for example, #JustDoIt, the hashtag is part of the entity itself, so it might include the hashtag in the output. Or if the entity is famous without the hashtag, then, you know, maybe it wouldn't include the hashtag. I don't really know. Everyone has different opinions here, which is the point. The point here is that everyone has different opinions and whatnot. People want to go back and change their grades, and people also have different opinions than what they had 5 grades ago. So how do we build these interfaces and support, you know, this dynamic evolving nature of what makes a good output? Sensemaking. What is the LLM doing? How can I make sense of it? This is a natural part of grading, human grading. And the implications here are that grading has to be continual. You've always got to be looking at your production data. You've always got to be learning from that. No evaluation interface, we learned, no evaluation assistant can just be be a one-stop thing where you grade your examples, come up with evals, and then push it to your CI or push it to your production workflow. No, you've got to always be looking at outputs. And one of the things that we've been doing that's quite exciting at the startup that I'm doing ML engineering for is we have a Slack channel where we just log a bunch of outputs every single day from different LLM-based workflows, and we literally look at them and we try to go back and reinform our assertions, which definitely helps things evolve. The second thing that we learned from the evalgen study was that code-based evals are very, very different from these LLM-based evals. I don't know why I thought going into the study they were similar, but they're not. People want to grade outputs to align the LLM-based evals but not necessarily the codebase details. When they want to evaluate something like Markdown format, for example, using a validator, they just want to see the code that's generated to implement that criteria. They don't want to look at examples of good Markdown, not good Markdown, and hope that the LLM finds a good example there. And so when asked, like, okay, when do you want to use LLM-based evaluators? People want to use them when the criteria is fuzzy, so they themselves, like, find it hard to evaluate, or they don't have, like, a good idea in their head of, you know, what is good and what is bad. They can't like succinctly describe it in one sentence, but maybe by giving enough examples, the LLM might learn something or learn that decision function for them. And then also, people want to use LLM-based evals when the data is dirty or there's typos in it. So for example, if Kaepernick, who's a football player— there's a typo in the tweet, the output might correct that typo, and a simple codebase function that asserts that the entity name is in the input will fail because the typo was corrected. But if you maybe use an LLM-based validator, then the LLM-based validator will understand that the typo was fixed. Cool. So the last thing I want to briefly show you is that from these learnings, we're doing an evalgen v2 which hopefully I can. And the idea here is how do we make, you know, coming up with evals a much more iterative process that doesn't just have this one-step generate criteria, grade, you're done workflow. Okay, so the idea here is Keeper Criteria as a dynamic list at the same pane as you're grading. And when you grade, you should be able to provide natural language feedback, which might, you know, add new criteria, might refine existing criteria's definition, and so forth.

Yeah.

I can probably skip this. This 1 minute of video. Oh, another thing that we found that was interesting, which I didn't include in the slides but it's in the paper, is that people want to give per-criteria feedback. Um, so maybe it's grammatically correct but not following some certain tone instructions. So people want to give a thumbs up on one, thumbs down on another.

So are you going to share where people can play with this? Is this public enough to where—

almost, it's very close. Okay, we're all academics and like working all day. So we move very, very slowly. This is a pet project, but you can go to chainforge.ai and play around with Chainforge at least. And you can play around with the table view that I add here. This table view on the right. If you write your own LLM-based evaluators, write your own prompts for criteria yourself, then you can run those and see this table view on your prompt.

Yeah, I recommend, uh, playing with this. I find it like whenever I hit a wall of like not being able to explain, you like writing emails to people, I showed them this and it works wonders.

Thank you, it's awesome. Yeah, I'm excited for the V2 to come out, but the V2, I just need to implement some more algorithms in the market, and I will do that when I have time. Um, cool. So this is my last slide. Uh, my overall takeaways from all of this, you know, is when running LLMs at scale, there's going to be mistakes, and we can use LLMs to, along with context for what humans care about for their prompts and what makes for good output, for example, prompt deltas, We can use that to assist them in coming up with good evals. So there's no— my slide animation's off.

Okay, cool.

Yeah, prompt deltas kind of form assertion criteria. And when you build an eval assistant, it's got to be iterative. It's got to work and consistently solicit grades from the human as data and the LLM prompts. Parts of the pipeline as well. And yeah, if you have any questions, please feel free to email me. Um, check out the preprints, they're on my website and they're on archive. Thanks so much, Hamel, for having me, and Dan. We've not met.

This is excellent. Yeah, the Discord, the Discord is going wild. They really, they really love this, uh, stuff about, you know, these interfaces and this like, uh, explanation.

Oh, that's great.

Yeah, this was really awesome to see. Um, should we go through— I don't know, Shreya and Eugene might be back. Um, I don't know if you guys have time to go through questions. I think we'll either way, um, go through some of the questions that we have, uh, queued up here.

Yeah, let's do it.

I have like 5 minutes and then I gotta eat lunch before the meeting.

Um, I wonder if I'm gonna look through these and see if there are any that— or actually, Trey, you can see some of these. Are there any that immediately come up, come up, uh, to you as interesting ones you want to cover?

You can sort it by most upvotes. It's like sorted by time. But you can change that.

Got it.

I feel like some of these are already answered though.

Yeah, a good number of them are also for Eugene's talk. Would we get access to this notebook? Yes. Eugene will probably share it.

Yes, all the notebooks are available. I've posted a link on the Discord. I don't know if Hamel, you can help to pin it. So it's all available. There's a slew of appendices as well. I will tag you, Hamel Husain. I'll create a thread and tag you, Hamel Husain, so maybe you can help with pins.

Okay, yeah, please tag me.

Okay, I found one question that is upvoted a lot and relevant for me. Using prompt history to generate assertions is very interesting. I believe this can be used for unit tests, LLM as a judge assertions. It's a goal here to improve assertion coverage and reduce the time it takes to write these assertions by hand. Okay, this is a great question. The first thing about having the prompt history is that it focuses the LLM's attention when you're asking the LLM to generate criteria for you. If you just paste in your prompt into ChatGPT and ask it, what criteria should I design unit tests around? ChatGPT will just design a unit test for every sentence in your prompt, which maybe that's something that you want. ChatGPT is very verbose. It just comes, it just recalls everything. But I find that, you know, hovering with 15 criteria, especially for long prompts, is like a little bit much. I want to start out with like 2 or 3 that I feel like are good ideas and really work on, you know, fixing those criteria, making sure we've got implementations of those before adding new criteria. In this case, Then providing the prompt history, the deltas, right? Like, it's a great example of what you care about, what the LLM is bad at. This can focus ChatGPT or the attention in generating assertions. So maybe it'll only come up with 4 or 5 eval criteria, which is a much better starting point, I think, than 15. Hopefully that answers part of the question. The reducing the time it takes to write these assertions by hand. I don't think the— I don't think it's extremely difficult to, you know, come up with at least one or two good assertions. What I think is really hard is coming up with assertions that align with what you think is good or bad. This is hard because you don't even know what you think is good or bad. Like, you have to look at a bunch of outputs to be able to define what's good and bad. And that process also evolves as you deploy things, right? Your users might complain about things that you didn't expect, and then suddenly you have to also now incorporate that into your definitions of good or bad. So having kind of an evaluation assistant to help you draw conclusions from all of this constantly changing data, help you define what you think is good, that's where I think the biggest value lies, not just like generating code from an LLM.

Yeah, I agree with that. There was a question that I don't know if it was answered or not from Wade, but how are you organizing your unit tests? Where are you running them? Oh no, Eugene did answer that. Sorry.

I think that, yeah, I think that was a question for Brian.

Gotcha.

Yeah, and he mentioned something like notebooks as unit tests, orchestrating.

Oh yeah, he talked about the superpower of Hex for running those.

It's my favorite subject, which we won't get into, from a former life. Okay, so other questions?

Oh, there's, there's some I can do pretty fast, like how well do these assertion criteria extracted from patterns in the prompt versions generalize across models? Pretty well. The dataset of prompt edits that we looked at had prompts that were intended for Mistral, Llama 2, ChatGPT 3, 3.5, as well as 4. And Claude too. So I don't know, I think people make edits to their prompt in similar ways no matter what LLM they're using.

There's a question from Lucas: in the Honeycomb example, it was clear that you could write good unit tests for the data because, you know, is the query valid, etc., etc., but I imagine it's a lot more difficult for the general LLM input-output pairs. Curious to learn more about that. Um, so, okay, um, generally speaking, in an applied use case, I find that more often than not it's narrow enough to where it's not just general language, like reply to anything, do anything. There's some specific task that you want to get done. Are you trying to aid the user to do something specific? Um, and a lot of times, like, there's a lot of components. Like, there's function calls, there's RAG, there's something like that. And there's a lot of failure modes that can happen that you can test for. Um, however, like Dan— as Dan mentioned, there's some use cases where it doesn't really— like, if it's just like an internal tool where you're just trying to do something like reword text or summarize text, or, you know, there's definitely use cases like that where maybe unit tests are not going to have as much teeth. Um, but it's always good to think about, you know, these unit tests. Then you see any of these questions you want, you think are interesting?

Um, so a lot of the top ones are about function calling and agents. I think, um, Eugene sort of answered on function calling and agents. I might mark those as, uh, as answered unless, yeah, anything else to say there. Um, we got it somewhat, uh, heavily upvoted and there's a quick answer. OpenAI has a temperature parameter API. Is there something similar in open source LLMs that we have to account for. Yeah, and unit test. Yeah, so, um, these models also have a temperature. Set it to zero. It's actually some class models where you get an assertion error, you get some error if you get set it literally to zero and you set it to 1e-6 or something. But yeah, all the open source models you can set temperature to zero as well.

I thought this question is good. This is before Uh, before starting a task, how important is it to have evaluation method? How to fix this as you learn more about the task in the process of doing this? Um, okay, so you don't need to set up evaluations in the very, very beginning. Like, make some minimal product or something. Like, you don't want to just be like super academic and say, oh, I have evals before— like, before even beginning. You kind of you might not even know what you're trying to build. Like, as far as, like, when I build stuff, like, I don't necessarily know all— I don't really have a clear picture. I kind of have to, like, shape it a little bit. And so you can certainly start with, like, without evals, but you need to quickly think about evals at some point when you want to improve the system. Um, and so don't let evals necessarily get in the way of making something, Um, just know that like it is a, like a crucial thing when you're trying to at some point make it better, and that's how you make it better.

And to carry that even a step further, I think like one of the things that I've not thought about very crisply but really came out of Shreya's talk is that when you see, you actually don't upfront know all the evals you want. And one of the nice things about working with LLMs is like to call ChatGPT, like to open up a browser window and try something and see what isn't working is super easy. So I would say you probably don't want to do anything formal up front. Like run a few examples, like literally type ad hoc in what you think the question is into Claude or ChatGPT and be like, ah, here's the thing that's not working, and then I realize it's difficult. And you'll do a better job once you've just experimented around and then build up the complexity over time rather than after you've like experimented rather than trying to imagine the pitfalls up front.

Yeah, I agree. I just want to reiterate what Hamel Husain and Dan have been saying. I mean, we don't want to get into eval paralysis. Like we don't need to have all our evals laid up front before we start doing something, just build something small, 30 samples maybe would be good enough. And as you start doing it, you start to see more and more edge cases. And that's how you add evals, right? Add evals like test cases. So it's an iterative process. So we don't want to have evals also slow you down, slow down your building process. It's good as a test harness, it's good as an insurance, it helps you stay safe, but it shouldn't slow you down too much.

It's also interesting to think about, so there are different use cases or different problems. Some, the eval is actually like just classification. And for those, you know, Eugene showed a bunch, so showed examples. The eval is pretty obvious what the eval should be. And then there are others where it's not classification, you're generating freeform text, or it's like very fuzzy. Um, and those you'll accumulate evals iteratively as you see like what doesn't feel right. Another one that's highly upvoted, but I think has a simple answer. Is LM as a judge a fine-tuned model or only works by improving prompting? I think LM as a judge, there may be exceptions, but it's almost always just like a very, very good, very, very smart publicly available model. Because if you're fine-tuning, that's typically a case where like you have a bunch of examples of the correct behavior and you might use that to build like the model that outputs the data. But the number of ways these models can fail is sort of like open-ended and fuzzy. So I would say I've only used LLM as a judge with models that are not fine-tuned. And I think that's probably generally true.

Yeah, I mean, I have only fine-tuned the model once for this in a specific use case I can't talk about, but I think I would avoid it because then it becomes like turtles all the way down or like, you know, yeah, like basically you want to try to use off-the-shelf model and align it with, with the human. Because, because like the complexity is like way too high. If you start like fine-tuning this other model, it's like Judge model, it becomes insane, and I don't recommend it.

How do we go from 0 to 1 in starting the data flywheel for collecting user data and curating the dataset? Um, I actually think this has a similar answer of starting, um The great thing about LLMs is that there are some very good ones off the shelf. There's no fine-tuning, you don't need data. I would start with a prompt, and you can probably, if you want to build a data flywheel, like implicit in that is that you're going to build some product that you're going to increasing usage on and then collect data. Um, but I would actually start with just a prompt, and for most problems you can write a prompt and use a generally available model that's reasonably good.

Yeah, synthetic data generation. That's the whole magic of LLMs. You can unblock yourself a lot of times. Not every time, but a fair number of times.

Uh, your code— here's one with 3 upvotes. Your code uses do_sample=False, but real-life prod will use do_sample=True. What's the thinking here? Um, it varies a ton by use case, but I would say that for the use cases I deal with, do_sample is pretty much always equal to false. Do_sample is basically like, is temperature non-zero? Um, we like our— for the products I work on, we just say like, we want something deterministic, we want the best answer, oh, we don't need variety. If you were building like character AI, you would want it to be more creative and varied. What about you guys? Do sample in prod, is that usually true or false?

You mean for few-shot examples?

No, sorry. The do sample parameter in when you make your generation call?

Hmm, no, I haven't had a reason for that yet.

Or do you— what's your temperature? Is it zero or non-zero in prod?

I think mine is zero.

Yeah, so that'd be the same as do_sample equals false, and that's always been the case for me.

I've been also addicted to this. Um, I usually start with 0.8 and then I lower it as necessary to achieve whatever performance. I think there's another heuristic I've heard people say, which is you start— you want to get as close to 0 as you want, as you can, for classification or extraction tasks, and you want to get as close to 1 for creativity and generation tasks. So that could be it. That could explain why I'm close to 0.8. But essentially, if you get too low, it's almost like a dumb intern. And it depends on what you want to do. So do try that out. The crazy thing is for OpenAI, the temperature max is 2. I don't know why, don't ask me why. So if you're thinking about temperature, that's something to take note of.

You know, if they let it be high enough, it'd just be a random token generator.

Exactly.

There's a comment from— I probably can't even pronounce their— or question from— I probably can't pronounce their username, Manimic. When doing A/B tests on an LM, how would you prepare the data. What I mean is, do you ask— just ask your LM to vote A/B, uh, or do you do some prep ahead of time? There's a chance that that was mis— um, misinterpreting something that I talked about earlier where I said we're using A/B tests. We actually have two different models that are producing output. So in this case, that was for alt text, like two different models that you could use to take an image and get a description from it. And then we had people who rated each of the— some people would rate one model and some people would rate another model, and then whichever got higher scores, we just said like, okay, that's the model that we're going to continue using. The model that got worse scores, um, we would just discard. But it was the people rather than the model that were assigning scores. You could in theory have an LLM pick between two, uh, candidate pieces of text. I've never done it and I don't immediately know the use case for it. And then it would be hard to answer this question of like how much data cleaning do you do before that if, um, without— I think it would always depend on the particulars of the problem and why you're using an LM for this, um, so-called A/B testing. Let's go back to the top of the upvotes. Um, can you talk about what metrics to use to evaluate retriever performance in RAG? Um, I think there's probably a better answer than what I've done historically. Eugene, do you have any, uh, thoughts on this one?

Yeah, a few things. Um, I won't go into the standard ones. I think the first thing is, let's say if you— your context size is like 10, you want to make sure at least you have some of it that's relevant. You have some relevant stuff in there, that's recall. And then there's also— that's also ranking. But what is also quite important— so recall is really just recall at 10, how many of those documents are relevant. And ranking is you want to make sure that the more relevant ones are closer to the top. Personally, for me, what I find to be quite important for RAG is this metric that I've never had to consider before. And this metric comes about— what this metric is, is how often can you actually return zero results if the customer is asking a question that you have no documents for? So if you're using purely semantic search, semantic search is just KNN. You just go grab whatever's the nearest, and the similarity could be 0.1, but you just pull way out, that's crap. The problem this happens is that LLMs just cannot distinguish relevant from irrelevant data. The more recent ones can, but it's just not very good at doing that. Uh, and a lot of this is also because of this eval, which is called needle in a haystack, that forces the LLM to try to pay attention to everything and try to use it. I believe that there's an alignment tax where you try to optimize for needle in a haystack, you also reduce the LLM's ability to reason over irrelevant documents in a context. So that's why it's very important to me that, hey, we're going to have some test queries that have absolutely no data in our retrieval index, and we want to make sure that it's always— we get 0 or close to 0. So long story short, recall, for one, just to make sure you have at least relevant data, and that's recall at 10. Ranking, which is NDCG, you want to make sure that your relevant data is close to the top. And then also, I don't know what this metric is, but ability to return zero results, especially for queries that you have absolutely no data for, so that you don't return trash and you can deterministically say that there's no answer. Instead of letting LLM say that there's no answer, if your retrieval results is zero, size of zero, you can deterministically say, hey, I don't know, and you don't even have to make a stupid mistake. In front of your customers.

Yeah, that's a good point. And, um, it reminds me, one, I talked about this project I did in Workshop 1. I talked to this project I did for, uh, this chatbot for a European shipping company called DPD. Um, and they had someone, uh, ask the model— a user asked the model to like write a haiku about how crappy the company is. And then this person published it on Twitter and it got picked up by the news and it was like embarrassing for the company. Um, when we said like, how are we going to fix this and make the system more secure? This isn't a perfect solution, but, um, one of the things that we did was to say if you don't have any document that meets some relevance threshold, there's a good chance that actually this is, um, a user who's trying to do something that we actually don't want to allow. And so it's not even like that we can't do a good job of it. This was just a way of detecting adversary, like a not perfect way, but a way of detecting adversaries and shutting down, shutting down that whole interaction.

And the amazing thing with lexical retrieval, uh, it's lexical retrieval, you can have a score, and even embedding retrieval, you actually have a score in terms of distance, right? You can see if anything that's way, way too way too low on similarity, just, uh, threshold it. Yep, that's beautiful.

Um, this one I think is for you, the top voted one, uh, from Sam Silver. Uh, so Google had a kerfuffle where they retrieved relevant documents but the documents were not factual. Um, I like the description: it didn't hallucinate relative to the documents, it hallucinated relative to reality. Um, I'd be curious to hear about the relative importance of these problems and if you've ever worked on filtering documents, the documents or the corpus itself, to ensure that the documents themselves are factual or unbiased or whatever else.

That's a great question. And sometimes this happens, sometimes your documents could be factual but could be biased. It could be about racism, fascism, whatever. And you know, there's documents out there like I don't know how to solve this problem yet. I think you could probably solve this problem with a content moderator over all your documents. Like, say, if you check if your documents are— I mean, the ones that are very clear is like toxicity, bias, offensive behavior, not safe for work, sexual content. Those you can easily do. And because you're using RAG, you can very easily exclude them from your retrieval index. That's your immediate admin code, right? So imagine if I was Google's case, you know, the pizza and the glue. Okay, we know glue is causing it. We pull the admin code, remove that piece of, that piece of data from your retrieval index so that the Google Search AI summarizer never sees it. Problem solved. I think that's how I would solve it. But as to how to actually check this kinds of data where it's clearly misleading or clearly untrue data. I actually don't know yet. If we have some data that we can learn, I think content safety is very straightforward. Offensive data is very straightforward. We have a lot of data on that. But for things like this, that's really above the threshold that we've never had to grapple with before. I think it's still an open problem.

Are you running unit tests during CI/CD? It could just take very long with non-mocked LLM calls. I think that I'm supposed to be running it in CI/CD, but to be honest, the first— so I was quite interested. So Brian said that the purpose of most evaluations is to let you be able to sleep at night. And for unit tests, there's probably some truth to that. For me, the way that I think about it is actually quite different from Brian in that there's a thousand different modeling choices I can make. What base model do I use? What's my prompt? Um, what is the context that I feed in? Um, do I fine-tune? If I fine-tune, what's the data that I fine-tune on? I could make arbitrarily many models and I need to decide which of them are better and which of them are worse. And so if it's just for making decisions, and a lot of those are like, we want just a quick decision and then we're gonna make a change to the prompt, then we're gonna run this again and see is it better or worse, and we're gonna make another change to the prompt. And so frequently I just want it to be really easy and low latency. And for that reason, We typically run them— the developer runs them locally. You don't even need to push anywhere. If we wanted it to work like conventional unit tests of like a safety thing, you know, Hamel gave the example of not exposing private data, then I would probably put it in CI/CD to avoid the possibility that we forget to run it on something that gets deployed. Um, my use cases Um, aren't like that, and we're just measuring quality. And so, um, we— I've run— always run it locally. And then, yeah, can take a long time. Um, yeah, do you guys have a different answer for that?

No, that was a good answer.

All right. Um, ooh, I like this one from LZ. Good ways to check for— this is Phi-3 right here. What are good ways to check for contamination of base models with existing eval data using paraphrasers, for example? So you're going to test your model. You want to assume that how it does in your test is a good proxy for how it will do on new data. How do you know that your base model wasn't contaminated with the same thing you're going to use for evaluation? I'm gonna answer first. Yeah, I mean, I don't know.

Yeah, I mean, it's like, it's kind of like very similar to machine learning in general. Like, okay, it is useful to kind of also look at your production data and see whether that is like skewing really bad. Relative to like your validation data and whatever, that's a smell that you have some kind of leakage. Another smell of leakage is it's too good. Um, leakage is hard, to be honest. Um, I don't necessarily have bulletproof defense for it.

This one also seems super context-specific. So for instance, um, let's use Hamel's Honeycomb data as an example. There could be some Honeycomb queries that are in the GPT-4 training data. There probably are, um, but there's no reason to think that the ones that was collected for him to fine-tune on are more likely to have been pulled from the GPT-4 training data than ones that they will have in production. And so they are like, can you just sort of reason about it? Um, or if you use my example, we had— that I talked about today was this debiasing essays. Those were essays that just got written, or not essays, like we could call them journal articles. They just got written, they were submitted to an academic publisher, and now we're going to edit them the fact that they were just submitted for the first time, literally like days before we came to edit them, would make us think they probably weren't in the training data. So I think this probably happens sometimes and you just have to— it's probably just— I don't think there's a general rule for how you avoid it. Should we go for, uh, how much longer you want to go, Hamel Husain Husain?

I think we can end it.

Yeah.

Um, yeah, that's, uh, close it here. Um, anyone who— we've got 160 people left. Actually, before you guys all dropped off, drop off, um, for our Discord server, we are rotating links. I should figure out if there's a way to make it not have links expire. Every 7 days, but email me if you have an outdated link. I want to understand how many people this affects. Um, you have like a selected sample here. Sorry. Uh, and then please fill out the form, redeem your credits, and then I think we've got, uh, some good sessions lined up for tomorrow and basically the rest of this week. Thanks, everyone.