LLM Eval Tools Compared: LangSmith

Hello everyone, I always get asked what my favorite evals tools are, and that's a difficult question. What we can do is take a look at some of the more popular ones, and the most popular ones I see in practice are LangSmith, Braintrust, and Arise. So first, let's take a look at LangSmith. Welcome to Mystery Science Theater, evals tools edition. With us today, we have Shreya Shankar, Ryan Bischoff, and myself, Hamel Husain. And we are going to do a little fun battle, sort of. What's the right word, do you think, folks? Like, I don't even know what to call this.

Just bake-off, really. We're just trying to see, see what's, what feels the best to us. As, you know, esteemed judges. Shreya, what are you— what are you looking for? What are you hoping to see in these eval frameworks?

I'm looking for good vibes. Unfortunately, I don't know how to explain what good vibes means. Okay. But if you force me to explain what good vibes are, I think I'm looking for the ability to look at my data as a human and not have another LLM look at all my data for me. So hopefully that's like the basic premise, but learned not to keep my expectations too high.

Yeah. And we don't have any favorites, by the way.

That's true.

We have no dog in this fight.

Everyone can be a winner.

It's true.

Yeah.

And so we're going to present them in random order. This doesn't mean anything. So first up is Harrison from LangChain, and he's going to be talking about LangSmith. And so The way this is going to go is like we have homeworks in our class and each vendor is going to present a short video, about 15 minutes, might go slightly over, of Homework 1 and 2 of our evals course. And what we're going to do is like, you know, we're going to see 3 side-by-sides of these different vendors and we're going to see how they stack up. So let's just jump right into it. Let's start with Harrison.

Hello, this is Harrison from LangChain, and in this video we're going to go through Homeworks 1 and 2 from Hamel Husain's eval course, and we're going to do it with the help of LangSmith. So LangSmith is our platform for evaluating, observing, and doing prompt engineering on LLM applications. We're going to take this eval course that Hamel Husain have put together, and we're going to show how to do all the things that they lay out there except in LangSmith. And LangSmith is going to be this platform that brings all of these components together and makes it easier to do evaluations, build confidence in the LLM applications you're building, not just for developers but also for product people or subject matter experts or anything like that. So before jumping into this video, you're going to want to set up a LangSmith account. So go to smith.langchain.com, and here you can see that I'm logged in with GitHub, so I'm just going to click that. But you can sign up for an account if you haven't done that already. It's free to sign up. And then here you can see a bunch of our projects and workspaces, and we're going to dive into this later on in the video. The other thing that you're going to want to do is fork the LangChain version of the evals-course recipe chatbot repo. So this is a repo that Jacob from our team took and added in a few things to make it specific to LangChain and LangSmith. So the main thing that's changed in the code is we decorated the agent that we're going to be building, or the chatbot that we're going to be building, with this traceable thing. And this traceable will log stuff to LangSmith, so we'll be able to see what happens inside of the chatbot, and that'll be super helpful for starting to build some of these eval sets and just see what's going on. Okay, so if you've signed up for a LangSmith account and if you've cloned this GitHub repo then you're good to go. So what are we going to do in this video? We're going to do Homeworks 1 and 2. So let's go into the Homeworks folder and let's take a look at Homework 1. There's 3 parts to this. First, we're going to do some prompt engineering, so we're going to write an effective system prompt, and we'll see how we can do this in LangSmith. Two, we're going to expand the dataset, so we're going to build a dataset, and we're going to do this in LangSmith. It's going to be created from examples and we're going to add to it over time. And then 3, we're going to run a bulk test and evaluate against this dataset. So we're going to run the prompt that we've modified over this dataset and get some results. And we're going to do this all on LangSmith. So to get started, let's go back to this GitHub repo that we forked, and you'll want to do 3 specific things. So first, you're going to want to clone this and set it up and install all the requirements. Next, you're going to need to configure the environment. This is going to be getting your LangSmith API key as well as whatever other API key to use a model that you want. Then we're going to run this chatbot and we're going to run this chatbot just so we can see how some of the tracing in LangSmith works and we can get a prompt from the code into LangSmith. I'm going to go here to my terminal. I've already done the first two steps, but I'm going to run this snippet here. Once I get this endpoint up and running, I'm going to go over to it and I'll see this nice little front end that Hamel Husain created. I'm going to ask it for a recipe. I really like pasta, so homemade pasta recipe, please. And let's see what happens. So the assistant's typing, and we can see that it will respond in full with a homemade pasta recipe. So what happened under the hood? In order to see what happened in the hood, we're going to go to LangSmith. So let's go over to LangSmith, and I can see that I have this evals course tracing project, and this is where I set up traces to be registered. In order to see what happened under the hood, we're going to go to Langsmith. So let's go over to Langsmith, and I can see that I have this evals course tracing project, and this is where I set up traces to be registered. If we click on this, I can see that I have a few runs here. These are from previous places where I was testing it out, and if I open it up, I can see the exact inputs and the exact outputs that came out of the LLM when I called it as part of the recipe chatbot. So what I'm going to do from this trace is I'm now going to open it up in a playground. And what this is going to let me do is this is going to let me iterate on it here and do some prompt engineering here and see how it changes. So the first thing the homework wants us to do is modify the system prompt. And you can see here that's a pretty basic one. So what I can do is I can modify.

So what I like already is Hey, you can go straight from the trace to the playground. You don't have to like set up a playground, go to a separate playground, something. It's just like, let me play with this thing that I'm seeing on the screen right now. That's, that's pretty cool.

I things here. I can edit stuff like that, but what I'm actually gonna do is I'm gonna press this little button and this is gonna open it up in a full screen kind of like prompt canvas. And then I'm actually gonna use an LLM to write a better version of this prompt because I'm pretty lazy. So let's go back to this homework. We can see here that there's a bunch of suggestions for how to replace it. So I'm just gonna copy all of this. I'm gonna go back to LangSmith. I'm gonna put this in the instructions here, and I'm just gonna run that. And now let's see how it modifies this prompt.

Making AI do your homework.

So we can see—

Where did those suggestions come from again?

Uh, both of OpenAI has a system prompt guide and Claude has a system prompt guide. And so I read both of them and then came up with the ones that I also use, this intersection of all of them. Okay, I don't know, maybe now it's going into the training data, so.

Hmm, okay, very interesting suggestions here.

And just putting it in here, and if I wanted to tweak it more—

he didn't read the prompt.

He might have written the prompt.

He may, he may have read it. We don't know what he did, but it doesn't seem like maybe—

yeah, he did. Yeah, I think if I was doing this homework personally, I would have spent more time like reading both the prompt and the instructions to be a little bit more like anxious about the quality of it.

Maybe he'll read it now.

Let's see. Or I could, I could type in more instructions, I could edit it here, but I'm actually just going to use this version. And now I can see that it shows up here. If I rerun this, and in order to do that I just press this little start button here, I can see that it generates a response that's exactly as I described. It starts with kind of like this header 2 thing, it goes on for a little bit, and, and then has, uh, you know, ends with some tips. So it follows a very similar pattern as this example here. One thing that I'm going to do as well is I want to try this out on more things. So I'm going to highlight this and I can see that I get this little button called Convert to a Variable. And so when I click on this, I'm going to enter a variable name. I'm going to call this query. And I can see now that this moves this to the input section. So what does this let me do? This is— so this is setting the stage for testing the same system prompt over more queries in the future. And so immediately I can, I can just change this here. So let's say instead of a homemade pasta recipe, I want a pizza recipe. Now let's rerun this and I can see that it changes that. So it's setting the stage for where I have this system prompt and it's gonna take in a query and return a recipe.

The—

that's really sick. That's like very notebook-esque, like, mm-hmm, vibes.

Template your system prompts. I really love that.

Yeah, I use prompt templating a lot and I, I think this makes a lot of sense. I assume we'll see him go down to the prompt at the bottom and show how you can use a mixture of the template components.

And yeah, last thing I do before— oh, it's still going. The last thing I do before I'm going to go on to the next step is I'm going to save this prompt. So let's click save. Let's name this Harrison's food prompt, and let's save this. Great. So now it's saved and it's committed, and I can return to it when I want. The next step of the homework is to create examples that will—

we never got that reflection step. We never got the sort of like, how did this do? Did it improve anything? We never went back and compared. So I think on the workflow, I'm a little bit anxious. The other thing that I'm a little bit anxious about, if you go back just a few seconds, I think, uh, right before the saving, if you get rid of— yeah, I have some anxieties about this UI. This UI is, is giving me a lot of anxiety in terms of what it prioritizes. I feel like it prioritizes, um, a lot of like automation stuff, but not a lot of like human-in-the-loop stuff. And so for instance, like, the output isn't super easy to see here. I'm not really seeing like the connection between the input prompt and that output.

Like, what do you mean it's not easy to see? You can't tell?

Yeah, like, let me give you an example. So we've now had two different prompts that we've tried. I don't see side by side the different outputs, and I don't see a linking between those two things that we tried and the output. So now I'm like going through this experimentation workflow, and I want to see what's changing as I go through it, and that's not super easy to do here. And maybe there's another view that shows that, but I feel like that's like tantamount because, yeah, like, I assume that when your students are working through this homework The point is for them to like look at the before and after as they implement each of these individual like ideas to try to understand and grok what's changing.

I think in this, I, I would agree with you in the theory, but in this particular homework, the objective is to just get to some system prompt and then run it on a bunch of traces. And homework 2 is all about the error analysis.

Got it.

So maybe it's one of those things that it's like, because he read both of the homeworks, he like realized that, oh, we're just trying to get to a system prompt here and like here's something. But I would agree with you, like you do wanna read it and consider or reflect on the outputs. Cool.

Cool. Hmm. Yeah, it's something worth paying attention to for sure. Okay, let's, let me go back.

Will live in this query dataset. So what we're going to do is we're first going to create a dataset in Langsmith, and then we'll add more things to it. So if I go here, I'm going to click this button on the side, Datasets and Experiments. I'm going to click up here, I'm going to click New Dataset. I'm going to import it from CSV because Hamel Husain have nicely provided us with a few default ones. I think it's called— what is it called? It's called sample_queries.csv. Sample queries. You can see that I found it here. Let's go back here. Let's upload this here. So it automatically recognizes the columns. I'm going to move the query field to the input. Great. You can see a little preview of it here. And let's name the dataset Harrison's Food Dataset. Let's create this. Awesome. Now I'm brought to this view.

I think every tool has their own definition of a dataset, an input, an output. And when we are teaching the class, we're trying to be agnostic and we're just saying, you know, write your traces to a CSV file and then the schema of that table is what is important to you. So like query means user query. If you wanna have like each LLM response be its own row, fine. If you want a row to be an entire end-to-end multi-turn trace, fine. Like whatever it is, it's your job to figure out. And I wonder in these tools if there's like an opportunity to be more explicit about what like a dataset means here, what should be a row and what should be a column, especially because you're forcing us to like make these decisions on what is the input.

I don't know.

It did seem like he could pick the column where the queries are coming from, which is a good first step, because ultimately there's going to be some user input almost every time.

Yeah, that, and my other comment was, and this is not specific to Lane Smith, everybody has this problem, like after I've tried my prompt, like I didn't, I wouldn't know to go into the datasets tab. So if there's like a way to to tell me like what to do next. Oh, you should create a dataset. Arguably the vendors might think, oh, they should take Shreya and Hamel Husain's course to figure out what to do next. But it would be nice if the tool told me.

Like guided, like guided way. Hey, like you wanna do your first eval, this is what you do. Yeah, that would be good. Yeah.

Or like run, make, create your first traces, like go here. And, or even just like in each one of these page, like what do you do after you've created a dataset? Like, I don't know, it's not easy, not intuitive on the UI.

Hmm. Okay. Yeah, that's something to think about.

Where I can see that it's in datasets and experiments called Harrison's food dataset. I have 3 examples. So these are the 3 examples that were in the CSV. And so this wants me to add a few more examples. So what I'm going to do is I'm going to click create example, add an example. It's expecting the input as JSON, and we'll actually see how we can make this a lot nicer in a second, but I'm just going to add in JSON for now. So I, There needs to be a query field, and then I'm going to say homemade pasta recipe, please. And let's add this. So if you notice, I added that in JSON. That's a little bit ugly. Let's make it a little bit nicer. How are we going to do that? We're going to click here and we're going to edit this dataset, and we're going to give it an input and output schema, and that's going to let it show things a lot more nicely. So let's click Add Input Schema. Let's add a property. I know that I want this dataset to have a query column, and, and remember, I saw that when I uploaded the CSV, it had a column called query. I'm going to make this required. And then if we go to output schema, let's add an output schema. But there's actually no, uh, fields that it needs, but let's save it that way.

So that's cool. It seems like it addressed somewhat some of Shreya's comments, like, hey, like, it's kind of guiding you to applying a schema on your data and it's fairly flexible. It's doing it a really nice way. Like, like really nice. Actually, I've never had any UX like this where like I upload my data and it like guides me through the schema. Usually you're just like on your own.

Yeah.

With, so yeah, I'm impressed by that part.

Great. If I go back to add another example. I can now see that it's nicer. And so I can use this to basically create these schemas for my datasets. So if anyone's adding to them over time, they easily know what fields are required and, and what types they should be and stuff like that. So let's add in another query, uh, like pepperoni pizza, and let's save this. Awesome, perfect. So now I've got 5 data points. I think it said to ask to, to add 10, but I'm a little lazy. Perfect. So I've got 5 data points here. What's next? Let's go back to the homework. It said to add 10. I'm a little lazy, so I just added 2 new ones. And so now the thing that I need to do is run the bulk test and evaluate. So they have a script here. I'm actually going to run this from the front end. So what I'm going to do—

oh, nice—

this button. Called Add Experiment, and I'm going to click Run in Playground. And so I'm going to go here, I'm going to now select the prompt that I want. I'm going to go to Harrison's food prompt. And so this is the prompt that I had edited before, and I can see now that I have these data points down here. And so when I click Start, it's going to run it over all these data points, and it's going to create what we call an experiment. If I click this button, it's going to take me into a new tab where I have the experiment results from running this prompt over these data points in the playground. I can do different things so that I can see it all either nicely collapsed or in a full mode.

One thing that— can you go back a teeny second?

Yeah.

One thing that surprises me, if you look right there, You see the queries are different. So on the far left it says inputs. Where's the rest of the prompt with like the system prompt and stuff? I would expect to see— I guess the inputs is just the user query here.

I— yeah, I guess the like system prompt is like a function that's part of the trace, or so like this is like truly the most upstream user input.

Yeah, I think I would like to see the, what I call the compiled prompt here, where you get to see sort of like the user stuff put in, or maybe an easy view. If you click that example number 3741, does that take you to the full prompt?

You know, if I could, if I could, uh, if I could like reach into like another dimension, I totally just tricked myself into thinking insane.

Anyway, I, I— maybe that's what that does.

Um, really funny.

Okay, great.

Running this prompt over these data points in the playground, and I can toggle a few different things so that I can see it all either like nicely collapsed or in, in a full mode.

The hardest problem in the world is like auto-sizing columns.

If I reload it, I can see that it's now finished running. So it has all 5 examples now. And, and if I want to get a view of what's going on, I can actually expand it and I can see the, the full result here. In Homework 2, we're going to dive a lot more into these results. So let's switch back and see what Homework 2 has us doing.

I wonder if there's a render markdown for outputs, or if you can do like custom, custom renderers.

Yeah, I want to do that too. It's a good, it's a good question.

We're now doing this RecipeBot error analysis. So what this means is that we're going to be doing some error analysis against a bunch of different failure modes on how this bot is doing. We're going to try to see how it's doing, where it might be failing, and then eventually how we can improve it. So it's got a few different parts. So first is to generate some basically synthetic data. So we're going to add— want to add more data points to this than we already had. And there's actually a really cool way to do this in LangSmith, so we'll, we'll show that. And then part 2 is we're going to do some initial error, error analysis. So we're going to run the bot on the synthetic queries. We're then basically going to go through and just like annotate them in natural language with, with what we see happening. And then we're going to try to come up with some taxonomy for the errors that we see happening. And so how are we going to do this? Great, let's go back here to LangSmith. Um, let's go back to this food dataset.

So I got to say, you know, he's coming into this, he seems very comfortable and poised, which is very interesting. Yeah, so we're going to see, because like usually when you get hit with all these terms, you know, error analysis, whatever, like a lot of people don't, you know, vendors may not know. So this is gonna be, let's see, let's see that we have here.

Let's go over to the examples that we have. If we click add example, we now see this button called AI, add AI generated examples. And so in, in, in the homework, it has you doing this thing where you identify key dimensions and then you create kind of like these unique combinations of all these dimensions and then you generate natural language user queries. There's a way to do that in, in, in, in code and do that kind of like step by step. In LangSmith, what we're going to do is we're going to do that a little bit more freeform, and we have this button called Add AI Generated Examples. And if this is grayed out, that means you haven't added the input and output schema correctly for the dataset. It's going to need to know those schemas in order to generate it. When I click on this, I can see that I have these this option to generate examples. And so it's going to use the examples in here to generate new ones that seem similar to it. And I can either automatically select some from here randomly, or I can manually choose ones. And what this will do is it will use those as few-shot prompts and basically say, hey, generate more examples similar to these. And so I'm going to increase this to like 10, and I'm going to click Generate. And I'm now going to be brought— if this doesn't work, by the way, that means there's going to, there's going to be a little thing that pops up asking you to input OpenAI API key. And so you should put in your API key there, and then, and then this should work. And so I can see here that it's generating a bunch of these different, uh, input queries. And so what I'm going to do is I'm going to select— I— you can choose which ones manually you want to include if you want, but I'm just going to choose all of them, and I'm going to click Save. And now these are all added to the examples that I have.

So this is really interesting, like I think we're going to see this a lot where vendors, the tools will like try to take a big step in automation and say like, let me just remove this part for you. And inadvertently, this is where people will shoot themselves in these big steps. It's not to say that these steps are bad. I mean, you know, it's just that I suspect that Okay, what kind of synthetic data are you going to get? You might want to think— the things that we teach in the class, like, okay, you want to be thoughtful of what dimensions to explore. If you put a large language model across some initial examples, your expansion is not going to be— it's going to be very homogeneous amongst those examples. You're not going to really be exploring.

They really all sound the same.

What's your feeling?

Yeah, I mean, you know what I'm gonna say, like the first thing I say in lecture 2, is it 2 or 3, I don't know, is when you generate synthetic queries, do not just ask an LLM for synthetic queries because it's going to give you queries that all sound the same. Then you're going to run your system prompt on it and all the outputs are going to sound more or less the same and you'll be like, there are no errors. And then, well, yeah, of course, because you didn't really cover any sort of diversity of inputs.

So one thing that I think pushes these vendors to do this is because they get a lot of requests for it. I think there is a certain frustration that we all have with sort of only focusing on like the local minima that these things are going to generate because we know that it doesn't supply good diversity. But these vendors are getting this request a lot, like very, very, very frequently. And so I wonder how should they be like reacting to customer after customer saying, oh, I wish there was a way that I could like automatically do this or something like that. I have my priors, but I'm curious what you two, you know, would have them do.

Well, I don't know if it's that difficult to go through the process that we teach in an interface. Like tell the user, hey, list 3 dimensions of a query that vary. Okay, so I would think like, oh, maybe ambiguity is one of them. Maybe like how long I have to take— how long I have to cook is another one of them. Maybe like my skill level, like I'm a beginner, am I an expert? So I would just like list these 3 things and then like feel better because LangChain is now covering— or like whatever synthetic query generator is now covering that space. I don't think that's that much work from like the tool, the vendor point of view.

I think like to zoom out even, you know, I think like the UX could do a better, do a, do some work around capturing user intent better. Like meaning, you know, like just ask the user like, okay, let's take you through a flow like. What kinds of, you know, let it ask you like what kind of things you wanna test for and kind of force you to generate some tokens. So you're thinking, and then even behind the scenes, maybe it's doing the dimensions, but at least like it's getting some dimensions in some form out of your mind, even if it's not those like tuples of like the strict dimensions, like fine, but like Okay.

One concrete suggestion I have for every vendor, if you're gonna ship an AI-generated feature, think about what is a form of like inputs you can get from the dev that you can use to seed the AI-generated whatever. Like just have some form there. Just like spend some cycles thinking about what would I put in a form here that will get some custom input from the human. So I don't go off and like automate too much, right? As Hamel Husain says, like don't automate such a big step.

But it's a good question, Brian. Like, what should vendors do? Like, I don't— I know that they get the requests all the time because I get— we get the requests and we get the question all the time too in our class. Like, why do we even have to learn all this? Can you just like automate the whole thing? Um, and it's like yet to be seen. Like, is it better to just like let people go through it fully automated first and then back off into the right, into like something so that at least they get started?

Maybe.

That's a pedagogical question.

My take there is no. And the reason is because actually there's very, very little value of kind of getting end-to-end on totally like trivial self-similar data. I actually would rather them focus on a smaller set of data and get end-to-end on like that high-quality data than like big batch of mostly useless diversity. That's my, my preference. Um, and there are ways to take LLMs and use a small set of data and kind of like multiply it a little bit. You can say these are the different way— like what Jerry was talking about, the ways that you expect things to vary. Like, you can use those as coordinates to the LLM to mix it up a little bit. We did this at Hex where we would take some, some really high-quality evals and we would multiply them by 5 or 10 by very, very controlled prompting on those eval queries that we all knew had the same answer. And so, uh, it still required a lot of human in the loop to kind of like evaluate them. But yeah, my take is like high-quality smaller dataset wins the day over big messy dataset.

So yeah, makes sense. Okay. Um, Okay, I'll keep— continue. Yeah, this is the first like big automation thing, so it was worth lingering on for a moment.

Have in my dataset. So again, this is a really easy way to generate synthetic examples. And if I want to, I can even be a little bit more pignate. So I could even select these three, which if you notice are a little bit different than the other ones, right? The other ones are all like all lowercase, and that's because the two that I added manually are all lowercase. And so it probably saw those and generated more similar to those. But let's say I want some more that are like this.

All right.

Yeah.

Tell us why you like it.

He literally said how— I was like, they're all the same. And then he was like, yes, they're all the same. Anyways.

So close.

So let's do this. Let's go to add example. Add synthetic AI generated examples. Now manual selection. So now you can see that's using the 3 that I have selected. Let's just add 3 more. We can see that when it generates, it's going to be a little bit more of this, you know, like well-structured sentences with punctuation and asking questions. And so let's, let's save these examples now. And so now I have a bunch of things. And so I'm going to run a new experiment now because the next step in the homework is to do this in error analysis. Let's go back here and let's run a new experiment.

I feel like there's way too many ways to get to the same outcome in this whole flow. And I'm like so overwhelmed by how to get to the end state, which is I want like 50, not even 50, 25 high quality queries, as Brian said. I don't know how I'm gonna get there doing this. Like, do I generate batches of 3 at a time? Like, which examples do I select to generate my 3?

But anyways, I mean, what I would say is like, yeah, like in real, in a more like, let's say real situation, you'd probably be using this enough yourself to like maybe have these, you know.

I find myself wondering when he's gonna put the, the correct answers in, cuz if it's gonna be error analysis, then don't we need some targets?

Oh, we, um, we just teach them to do like open coding and like labeling failures basically without having references.

That's fine. That's fine.

So I don't think he's gonna do that here.

Okay.

But it's a good point. Maybe we have a lecture that'll do that. Anyway, sorry, keep going.

Oh, no worries.

Let's choose the prompt I want, Harrison's food prompt. Let's choose that. I can see that it's this nice long one I had before. Let's click start. It's gonna run all of these. Awesome. And so in order to do this error analysis, Harrison is not a chef is what I'm—

my number one takeaway.

Sorry, keep going.

Yeah, I mean, neither am I, so there's definitely some—

he's a CEO of one of the biggest, like, tech companies in the world.

So I'm going to use something in LangSmith called an annotation queue. What an annotation queue is, is it's a place where you can put a bunch of traces and then annotate them. And so it's specifically designed for that. And so in order to— what I'm going to want to do is I'm going to want to send all of these traces, or all of these basically predictions, to this annotation queue. And so I'm going to do that by going— by clicking on this, I'm going to open this new experiment tab. So this is the one I just ran in the playground, and I can see that I have all these— there should be like 15, 16 different examples at this point. Let's go back to this food dataset. And so here I can see that I have a list of the experiments that I ran, and I can see here that, that, you know, the new one that I ran, which has 18 runs as opposed to 5 from before. On average, it took a little bit longer, so some of the questions were maybe a little bit longer. And so I can click this, and then I can click this button here called Annotate. And so what that's gonna do is it's gonna let me send it to an annotation queue. I'm gonna create a new annotation queue. Let's call this Harrison's food labeling. And if I wanted to, I could give it, I could give it some instructions for folks. And so I'm just going to say for now, like, write comments with high-level feedback. And then let's create this. And so after it creates it, I'm going to be brought here. So this is the annotation queue. This is a dedicated experience where I can see a very nice visualization of the inputs. And the outputs, and then I can leave feedback. And so if I wanted to, I could set up a rubric where I, where I leave kind of like specific feedback scores. And I'll actually maybe do that in, in, in a little bit. But first I'm going to add this like natural language, kind of like just reviewer notes. And so I'm going to add notes about how it fails. So let's see what is wrong with this data point here with the output here. Let's say that for this I actually want it to include kind of like recipes to make the pizza dough itself. And in general, when I ask for recipes, I want it to— for things like pizza and pasta where you could make the dough or, or not buy the dough, I want it to always kind of like default to, to making the dough. So includes option for store-bought dough, it should always default to 'homemade dough' and should include instructions for making said dough. Awesome. Let's leave this comment. I can add more if I want, by the way, so I can leave more than one note, but I'm going to click this Done button. And let me make this a little bit bigger so we can nicely annotate things. So now, okay, let's see another data point here. Um, what could be wrong with this? There's no tips. So in this one, you can see that there's no tips here. So let's just say missing tips section. Leave this. Let's go on to the next one. Simple chicken and rice skillet. I'll do one more live and then I'll fast forward to the end. Let's say for this one, I don't like adjectives in the title. And so I don't like The adjective.

He's pretty good at zero analysis. People struggle with finding anything wrong. Like, people— our students are like, yeah, all of them are good. No, so many of them, they're like, they're like, yeah. And I'm like, okay, like, be a little bit critical, you know? Like, let's— so, you know, he at least like had some idea of like, okay, let's, let's figure out something we don't like.

CEO vibes.

Yeah.

The name, no adjectives, titles. Awesome. So I'm gonna, I'm gonna just fast forward through the next however many. Great. So when I get to the end of the annotation queue, I'm all done labeling. And if I go back to the datasets and experiments view, I can see Harrison Field dataset. And if I click on this experiment that I ran.

By the way, I really liked the annotation interface. It had like hotkeys everywhere. It was really nice. It like made the data full screen. It was really just like, I don't know. I really liked it. What did, how do y'all feel about it?

Agreed. I did find it confusing what the difference between feedback and notes was, but.

Hmm.

Yeah, if you go back a teeny bit to show it, I think I had a couple of things that I was like, like the input and the output, like this, like this left bar does not feel well designed. It feels like it's like all the things that are there, but it just doesn't feel super well designed in terms of there's a lot of like extra useless space. The buttons are in like varying places. There's a lot of like, I would say, unnecessary iconography. I'm being nitpicky like I would do for error analysis, but, uh, I just I don't know. I think a lot of the goods are here, but, um, yeah, my, my design aesthetic is not tickled in this moment.

Yeah, fair enough. Okay, see, let's see what he gets up to.

Experiments view. I can see Harrison's few dataset, and if I click on this experiment that I ran and I open it up in this view, I can now see that I have this notes column, and you can see a few that I left, right? So no adjectives in title was a pretty common note that I had to leave. I also had to leave a bunch around when it, when it responds with multiple things, it would sometimes ask like a follow-up question. I just want to give me multiple recipes. Some I don't have any feedback on. So if we go back here, so this is kind of like the open coding where I review the recorded traces and I, and I leave a bunch of comments and I can go back to here and I can see kind of like all the notes that I have. The next step is to basically group the initial open codes into broader structured categories. One thing that I want to do for this is I actually want to do this with an LLM because there could be a bunch of things here and I'm a little bit lazy and I think it would be fun to show how to do this with an LLM. What I'm going to do is I'm going to get all the runs for this project and I'm going to pull them down in code. I have a Jupyter Notebook over here, so I'm going to pull them down in code and then run them through an LLM. Let's view this project. Let's open this up in project view. Now, so these are all the runs that I had. Let's copy this ID. Let's go over here. This is a code snippet. We'll put this in the README as well. Let's initialize the client. Let's get all the results. Then from these results, I'm basically just getting the comments from the notes. If we look at what feedback is, so these should look very familiar. These are all the ones that were included that you saw in the frontend. Now let's just run them through an LLM. Here I'm just going to invoke an LLM and I'm going to ask it to analyze these errors and identify the top error modes. I actually don't care too much about the prompt here because I'm going to open this up in LangSmith and use it to do some more prompt engineering as well. If I go back to LangSmith.

There's so much going on.

I logged it to a slightly different project. If I open it up here, and let's go to the playground. I can see that I've got, uh, so, so these are all the feedback that I—

so that, so that invoke thing must be logging to LangSmith. Is that what you gather too?

I think it's a call or like a completion endpoint.

Yeah, okay.

Got— this is my system prompt. Um, what did it— it gave, uh, maybe too many "Let's give only 3 return in the following format." What I'm gonna do eventually is I'm gonna make these into evaluators, and so I wanna give it kind of like an evaluator name. So let's do something like evaluator name, all lowercase spaces.

[Speaker:ASHLEY] I love how there's more failure modes than distinct open codes. That's like classic example of like LLM has no idea what you're talking about out of context, right?

That's a really great call out, Treya.

What is this? What we teach in the course is certainly you should use an LLM at this point to kickstart kind of grouping together open codes, but we definitely have like a more complex or comprehensive system prompt here. That's kind of giving it context. Like, you're trying to make sense of all of these errors from all these traces for my error app— sorry, my AI application. The other thing is you can totally use this open and axial coding lingo, like grounded theory, social science lingo, because LLMs have been trained to do this. Like, go ask ChatGPT what axial coding is. It knows what it is because it is like well established in the literature. But I would encourage people to do that. Like, you don't have to have this like prompt of like analyze these errors.

Yeah, Sapir-Whorf is your friend in this case. Like getting it closer to where you want it to think is really useful. The other comment I would make is back when he was doing the like natural text, one thing that I would love to see is for a given dataset, once you enter the text once, it's a dropdown selector because a lot of times you, you realize the same error many, many times and actually that's good enough. And so just turning those into dropdown selectors is plenty. Uh, that's usually how I do it. Um, so I, I think that would have been a simpler UI. Um, also just a quick call out, this is the first time that we've seen part of the homework not natively supported in Braintrust. He kind of has to go off the rails.

Thanks, Smith.

Thanks, Smith.

I clearly know that there are different products out there. Um, lang buzz.

I'm surprised he got this far, actually. Doing the whole thing in, yeah, NLIME sets. It's very impressive.

I mean, with this axial coding step where we categorize the errors, like, for me, cognitively, it's really important to see each, you know, example with its axial code just for me to review, almost like a spreadsheet style or in a data frame or something, because I just want to check, like, okay, is this is the correspondence between the two, like, is it making sense? When it's all in a blob like this, then that's not ideal. Like, you know, I can't really debug it, let's say.

Failure mode. Um, let's run this. Let's see what happens. So the ones that it decided were the biggest errors are missing tip section. So there was a bunch where it didn't have a tip section in the recipe. No adjectives in title. So it should not have adjectives in title. And then includes options for store bought dough. So I don't really agree with this because there's actually probably more around, if suggesting multiple, only return the recipe, no other response to users.

But— [Speaker:LUKE] So I mean, it looks like he really did error analysis 'cause he has in his mind what the errors are. So he kinda, he actually like, even though this is like a demo and whatever, he's like trying to just, do his best. Um, I feel like he actually did their analysis.

I feel like he's discovering in real time to like not use an LLM to do axiom coding.

You know, this is good enough. This helps me identify and, and trudge through a lot of labeled traces. And again, I only have 18 traces here, so you don't really need it, but when you have more, it's going to be really helpful. So this brings us to the end of Homework 2. I've created a larger dataset by using LLMs to generate synthetic data, and I've done this error analysis by sending it to an annotation queue and then labeling it with notes, and then, and then actually using an LLM to run those notes through and come up with these kind of like top areas that it's messing up. And we'll see what to do about that in future homeworks. Thanks for watching.

Okay, we made it through the first homework of the first vendor.

Okay, so Hamel Husain Hussein? What's your score out of 10?

It's hard, man. I think I need to reserve my scoring until I see them all because I need a relative thing. It's really hard at isolation to like say something.