LLM Eval Tools Compared: Braintrust

Welcome back to our series on evals tools. Today we're going to be taking a look at Braintrust. So now we're going to go with Braintrust and we're going to see what they got going on. And we got, uh, Wade, our friend Wade from Braintrust, who is also a student in our class. The first cohort, so you might have a slight advantage, but we will see how it pans out.

Hi, Wade Gilliam here from Braintrust, back again with another walkthrough, this time on how to accomplish Homework Assignments 1 and 2 with Braintrust. And in this video, we'll look at Homework Assignment 1, which is all about writing a good system prompt for our recipe chatbot, and then also providing it with 10 or so hand-coded queries to test things out with and get things going. With that, let's go ahead and get into Braintrust and see how to do this. The first thing I really wanted to do was define my intent for this iteration of RecipeBot, and I thought it would be interesting to attempt to build a recipe chatbot tailored specifically to the interest of my family. So I recruited my 3 sons and my 1 wife. I asked them to provide me 10 to 20 example queries that they would, uh, think of issuing to such a chatbot, and I imported those into a dataset to use. So you can see here's an example of one of them. We have the query, and then we also have some metadata, an ID. We have source is hand-coded. This will be important when I start adding in, uh, synthetically generated user queries into this dataset. And then I also thought it might be interesting to include who submitted it to further analyze if any problems were related to the queries of a particular person. Creating a dataset is actually super easy. If you click this button and create dataset, you'll be taken to this.

I will say the, the UX of the trace viewing, for some reason I just like it more somehow. I don't know why, but we can reflect on why that might be as we keep seeing it.

You can go ahead and use a CSV or JSON-L file. I'm going to use a CSV, and then you can drag and drop things wherever they're supposed to go. So The input we have is the actual query. We don't have any expected. And then what I did was include these in the metadata, hit import, and the dataset was created. Super easy.

It is nice to have the metadata tags, I will say.

Let's go ahead and get to actually working on the system prompt. Great place to work on prompts is the playground. And you can see that we have our system prompt in here, and I started this with coming up with the best guess following the recommended format for system prompts in the course.

I didn't see him do this step where he used the app that you provided. I wonder if he's hooked that in at all, or if he just went in and grabbed that prompt and pasted it in here.

Yeah, I don't know.

Yeah, no idea how to—

In a production use case, you probably have your real application, and it can be really nice to bootstrap from what your real application is doing. So one thing I liked about the Langflow approach was sort of automatically started with an existing application. Langsmith, Langsmith.

Damn it.

It automatically started from an existing application and it like using the traces, it was able to kind of load all that in and then you can fork from there. I kind of like that flow. I don't know what this does, but at least it looks like maybe he just pasted it in.

It's a good question. I don't know either. We'll see.

Recommended format for system prompts in the course. And when I ran this against the queries for my actual, uh, users, it did pretty well, but I really wasn't sure.

Okay, so this is a prompt playground. So it's like it has— he has system prompt in a playground and then the data, and he's like running, running that in a playground. So yeah, I guess it's like this is their version of like a similar playground. And to kind of piggyback off your comment you know, they didn't like arrive at it through a trace. They arrived at it, you know, through this like other pre-baked kind of thing. Um, so it is interesting. Um, there's this Loop thing though, so that's— I kind of— okay, let's see what this is. It's interesting.

But what I did was ask Loop, our AI agent, uh, to help me create a RecipeBot relevance score. Because I wanted some means to assess how well the outputs were, one, following formatting instructions. Were they trying to recommend a recipe regardless of the input? Were the, were the outputs indicative of including follow-up questions that would be required based on the user query? And so Loop created this recipe bot score, ran it with my system prompt, and like I said, it did all right, not great. So the next thing I did was ask Loop to—

so here he's asking the LLM to generate evaluation rubric, and then he's running that rubric with the LLM on the output. Is that the correct—

that's what it looks like. I'm looking at this and it's like, okay, I see. I'm looking at the reasoning trace, which I mean, it looks like a reasoning trace anyway. And it's like, um, created a code score, ran an eval. So he's kind of tried to do the whole eval loop, no pun intended, in one go. Um, which I'm not sure, you know, I mean, like I'm biased against it, let's say, you know, like, of course, like, as we talked about earlier. I'm just curious like how he's gonna use it, 'cause it seems like a really large step at this point.

Totally.

Yeah. Generally I'm fine for these kinds of, like, I would call them meaningless automations if they're not gonna make a business decision for you or a critical decision at this part of this slope. Like if he's running these evals and then he is aggregating the scores and then saying like, oh, like a certain kind of instruction is like kind of hard to follow. Okay, let me synthetically generate more queries of that nature. Like, I'm super all for that, right? Like, great job. But if he's trying to say that, oh, the specific instruction was only— was followed like 2.2 out of 10 times, I'm like gonna make a business decision on that and like not even have shipped this product. I'm like super against using these kind of core screen.

I mean, like, how did you— how did he come up with the rubric, you know? Did he get it from error analysis?

Um, yeah, there's also like the question of like stacking abstractions here. Like, where did he get the system prompt for Loop itself? And where is the— where is the system prompt that is built into Loop? So you've got like the user prompt that Wade has written to Loop, there's the system prompt that is in Loop, there's the system prompt that we never even actually addressed yet. Let alone look at the outputs here. So I think this is like stacked abstractions, at least level 3, which starts to really make me sweat. Um, I don't know, I think it's a little early for this big of a hammer, we'll say that.

Yeah, yeah, surprise. Yeah, we're all very aligned, uh, it seems like in this, in our feeling, which is— yeah, it makes sense.

Optimize the prompt based on the score and the data.

Yeah, he said, he said the magic, he said the magic phrase, optimizing the prompt.

All the context I don't even remember.

Okay, it made some changes, uh, to the prompt, and then I went in there and added a few more modifications. For example, my family, uh, doesn't want to have recipes with harmful chemicals. Like red-die-5 or no new—

which one of us wants to say the phrase, you can't optimize to an objective function that you don't know?

Okay, you said it.

Okay, perfect. Is there not a diff that was applied? Like, oh, there's a diff button up there, um, you know, there's a little diff slider on the top. Ah, so maybe there was a diff like possible, but like still I don't I just have no idea what the objective function is. I don't think this is going to— this is going to lead to more consulting business for me.

So I'm serious.

I think so.

Like, I think there is— I don't think this practice is wrong of like trying to improve your prompts based on your evals. You should certainly use your evals to improve your prompts. I think we are just like viscerally reacting to it's not done in the right time here. We've not reached that stage yet where where we know what our failure modes are, built the evaluators that we can trust, and to Brian's point, gotten an objective function.

Yeah, I agree. You know, and you know, just to be clear, Wade is my friend, by the way. So I'm not, uh, picking on Wade. He's trying to be very objective.

All the synthetic data generation that was done by his family and the subject matter expert family members.

That is, that is the best way. Yes, I agree with that.

That's not even synthetic data, that's real data.

Yeah, it's the most important data.

Yeah, contain microplastics. Also, another interesting thing looking at some of these queries is they weren't really, uh, right, requests for a recipe recommendation in so much as they were actually questions about protein and how to get, uh, so many macronutrients, uh, into a meal or to do meal prep.

Okay, but green flag here.

So I wanted to include He's now doing exactly what we want.

He's reading the prompt. He's thinking through how it relates to his, like, family's needs. So, like, massive green flag here.

Definitely. That, that's, yeah. Yeah, he's doing the inherent product work that comes with error analysis that forces you to do.

Some information about how to respond to those. With that in hand, I reran the evals and you can see things got substantially better. We are hitting an average score of 99%. I went through and spot-checked, everything looked great. Nice thing with Loop, when you create these scores, it's going to tell you how it actually created the criteria for the score.

99%. Sorry, keep going.

Yeah, I mean, there's just so much I have to like reflect on what I'm seeing.

Even give you examples to look at that might merit further inspection.

And so it's also green flag.

Yeah, with that in hand, we now have a really good prompt to get going with, because being able to go from those examples to high scoring on different dimensions of the rubric, that's the whole point anyway. That's great. That's like basically like sub sub-clustering and, you know, slicing and dicing. Yeah, exactly.

We love that.

Do you think— this is really interesting— like, do you think there's some good things in here, some things went too far, too big of a hammer? Is this like a necessary part of the journey of like getting to the right place, of like people going too far and then zigzagging their way to the something that makes sense?

Here's what is the problem, and this is my experience from the teaching. Like, you know, when you're making this video, I'm sure Wade knows, like, he's not gonna index too much on this 99.1%. Like, that number is actually meaningless to him as it is to us. He's using all of that to get to, like, the slicing and dicing that Brian was talking about. What are the examples so I can go and start thinking about error analysis? Someone else is gonna watch this video and only focus on that 99.1%. And then say they're done. That's like the biggest dissonance that I feel like this is— I see all the time with like AI evals.

Yeah, problematically, some executive who's going to want the number to be higher than it is because if number not good, number must be bad.

Yeah, yeah, I think those are good points.

So I saved this prompt, which We can see right here there's our recipe bot prompt, and now we're ready to use it to start actually creating some traces. So that's Homework 1. The system prompt is good to go, and I'll see you in Homework 2.

Quick interjection on like design and UI and stuff. Generally, I found the Braintrust like UI and design to be easier for me to guess where things were. Like I was finding myself looking in the right part of the app before he clicked into it, whereas with the previous one it was a little harder for me to like, even, even after he clicked it, to understand how that was where I was trying to go. So I think there's, there's definitely like a little bit more, um, understandability in that UI.

Okay, let me get Homework 2 pulled up. Um, yeah, I agree with you. I felt the same way. Some, something about the UI, it feels more intuitive to me as well. But there's some, there's some places where I like the, there's some places where I like the Langsmith UI more, like the annotation queue. I like that more. Um, I like the, I like being able to enter the playground from a trace. I think that's really powerful.

I think the concrete things that I think were a little bit better here. It was less of like a border radius around everything, less jargon. So like everything in LangStar has like a term for it that I cannot remember. And I'm sure it's fine if you're like entrenched in that ecosystem, but I'm not. And then there's a lot of padding. In that UI versus in Braintrust, it's like extremely compact. I'll say roundedness, I feel like is out of fashion now. I don't know why.

Yeah, so I have really bad design. I just know if it's bad, but it's good that you like can call out certain things because I just feel like it's soft. I can't even like name why, uh, so it's good that you But you know why. Um, okay, shall we go— should we do part 2 of Brain Trust?

Homework 2 is all about error analysis, and it's broken into 2 parts. Part 1 is all about the process required to generate good synthetic data that we can combine with our hand-coded queries to run through our recipe chatbot and get traces for. With that in hand, we can move to the core bits of error analysis, which is engaging our subject matter experts in the process of open and axial coding to eventually arrive at a taxonomy definition of our failure modes. And with that information in place, we can prioritize where to put our efforts in terms of building evals. So in this talk, I hope to go through some of the more interesting ways that we can use Braintrust to—

Does Wade live in a rainforest?

He lives in San Diego. I don't know exactly within San Diego where he lives, but yeah, I do hear birds in the background all the time, and it's very pleasing.

Um, I think like, you know, bonus points for bird sounds. Um, yeah, that's, that's really all I have to say on that.

Yeah, no, I'm enjoying it too, actually.

Accomplish both parts of this homework. So I'm back in Braintrust, and let's talk about how we can accomplish part 1 of the homework assignment, which is really all about getting some high-quality synthetic data that we can combine with our hand-coded data to then generate traces for. So just like I did in talk 1, which if you haven't watched, I encourage you to do so because I go through this in a little bit more detail. We went through the process recommended in the course, and we codify that in experiments. The first one is our dimension tuples. So we go through the process of generating a bunch of different tuple combinations that are candidates at this point, because at the conclusion of each of these steps, human review is required to go through and make sure that these tuples actually are representative of a realistic potential query. We can do that by clicking up here in Review, and you can see that we've already gone through this process, going through all these different unique combinations that were created, and we have our two scores that we have configured for this.

I'm slightly confused. I think I know what's going on. It seems like he used an LLM to generate his dimensions and now he's going through and labeling which tuple, like, uh, you know, these dimension pairs or dimension tuples are good or bad. Is that your impression?

That's what I think. So same.

It seems a little bit overcomplicated, but let's keep watching, I guess.

Terminology is overloaded. Like to call this a trace is going to confuse 90% of students in the class. However, let's carry on.

One quick opportunity he had was he could have gone back to the subject matter expert with a couple of the generations, and he could have been like, what do you think of this recipe? What are some things that you don't like about this recipe? And he could have used those to generate the initial, like, issues.

Yeah, the, like, the exact thing around the plastics, like, that's not everybody cares so deeply about that.

Totally, totally. There could even be a failure funnel of like, you know, early stages of where it goes wrong versus later ones, but I don't know anything about that.

That's fun. Nerd sniping us all.

And a is good score. The only one that we care about at this point is, is good. And so we go through the subject matter expert and determine whether these tuples are good or bad.

This is extremely difficult to make sense of.

Good.

Tuples to generate some candidate user queries. And that's captured here in this experiment. And then we go through that same process of working with our subject matter experts and going through each of these and marking them as good or bad.

Nice.

And finally, we take only the good user queries, combine them with our hand-coded queries, and update that original dataset to include both of them. And at that point, now we're ready.

Why can't I have like a Tinder interface at that last stage? If you're already at the sort of like class of just quickly iterating through a bunch of examples with your subject matter expert, give them a Tinder interface plus plus, like one that like shows them the card. Here's the sort of like request, here's the output, here are the options for how you want to classify it. Like, I think people under— undervalue these like rapid iteration UXs, and I think that's a perfect use of it right there.

Hmm. Yeah, it would have been cool for sure. Even make it a mobile app, why not?

Yeah, absolutely. I actually think these apps could have sort of like a different subdomain that you send users to. It doesn't show them all the innards, it just shows them what they need to give you that data. That's what we've built internally at Theory as a just like a quick internal app for ratings for that reason because they don't need to see all the nasty bits.

Hmm.

You move.

I have one comment that people may or may not care about. But there is, in UI, there's a law called Fitts' Law. Have you guys heard of Fitts' Law?

No, please tell us.

Oh, man. OK. So the time it takes for your cursor to travel to the target is proportional to the size of the target. And how far away your cursor is from the target. So if you want to speed up button clicks or workflows involving button clicks, you need to make the button bigger, or the distance from where your cursor is needs to be very close to that button. None of these interfaces know anything about Fitts' Law is my takeaway.

Incredible.

Incredible. I forget that you are a, you know, human-computer interface researcher as well.

So I mean, that's why you know about these things, but it's like so applicable and I feel like my day-to-day life now. Anyways, analysis.

Another thing I want to note is that as you work through the code in the repo, you'll notice that we build the prompts for each of these steps in code and then upload them to to Braintrust. And one of the big reasons we do this is so that we can go ahead and play with these prompts and in particular use playgrounds just like we did in Homework Assignment 1 to optimize them. The next thing I wanna show you are some changes that we needed to make to the recipe chatbot application in order to properly instrument it with Braintrust. And so I copied the code into my repo And you can see here I'm in the utils.py file. And the first thing I had to do was call init_logger, and this will initialize our Braintrust logger so we can start sending information to Braintrust. And right afterwards, I create this LiteLLM wrapper. And so we have a method called wrap_lightllm that allows us to essentially create a drop-in replacement for LiteLLM But importantly, what it does is regardless of how deeply nested our code is and wherever we call LightLLM.Completion, for example, it ensures that that information is properly nested in spans within the trace. Additionally, it ensures that any usage metrics are reported in Braintrust as well. With that in place, we can go ahead and see some changes we made to the system prompt, We've set it up so that it attempts to load the prompt that we created in Homework Assignment 1, which is the improved system prompt for recipe chatbot. If it can't find that, it falls back on the default system prompt presented in the course repo. And now we can actually get to figuring out how to set methods up to log them to Braintrust, and it's really simple. So here we have our get_agent response method, and all we have to do to get the— this particular method call, the inputs, the messages metadata, and the responses sent to Braintrust is decorate it with our trace decorator. And by default, as I just mentioned, what this will do is create a trace named Get-Agent-Response. It will set the input to messages and metadata and the output to whatever the method returns. But because we want to exert a little bit more control over what that looks like so that it's more intuitive when we look at those traces, especially with our subject matter experts, we can go ahead and take full control over what traces look like by passing in this no_trace_io and setting it equal to true. Argument into traced. With this in place, we can use currentSpan.log to specify exactly what we want set as input, output, metadata, or anything else.

And that's a little too much for me. I preferred Langsmith. It's one decorator and I'm done.

But anyways, one thing I wonder is like, does currentSpan.log make sense? The thing that is not I find the default surprising because I would have assumed that the default would be that you specify what these things are. So I don't have any guesses as to what this default behavior is going to be.

I think it's as Weights and Biases as you can get. You can't do better than that in terms of there is a minimum amount of information you need to capture from the application. Right, you got to get it somehow. I don't know how LangSmith magically got it all.

I wonder if it's maybe he just didn't show us this piece and it is similarly— because this makes very sense to me, but also W&B, um, I'm a little biased.

Um, okay, I mean, I think, yeah, I mean, it makes some degree of sense. Like, okay, you want to know what's the input, output, and metadata and control that. Yeah, I kind of think I would always, I would always want to do this. Yeah, so I definitely feel you. Um, yeah, I think I always want to do this. Maybe it's the cut, like, yeah, I don't, I'm not really sure.

I was gonna ask you to scroll up.

LangChain. Yeah, LangChain. I, I don't know, I don't think he was using LangChain. We can definitely go look at the code if we wanted to, but like We can keep going.

We don't have too much time left anyway.

Yeah. Let's keep going.

Run through our synthetic user queries, our hand-coded user queries, and look at those traces to perform the next step of error analysis, which is open coding. Application traces can be viewed in the Logs view. If we go into Logs, we can see we have a bunch of GitAgent response traces. And clicking into each of these, we can inspect the exact spans that make up each trace. So notice that for the inputs, at each turn we capture the system and user message there, and then the output is the AI assistant response. Also, you can see that our calls to lightllm.completion are properly nested and the usage metrics reported back from LightLLM captured here in Braintrust as well.

Really clean UI there. Like, that trace—

I don't like that there's so much space. There's like so much white space on the left though.

Yeah, that's a true— it is in this right panel. It probably doesn't need to be all hugged over there, but these like collapsible, uh, what do you call it, um, span, or yeah, spans basically, like they, they're, they're working for me.

Yeah, it's like a human-readable sort of, uh, trace viewer. It looks like it's rendering Markdown, actually. Like, if you look at— you see like these Heading 1. Totally. So some, you know, it's trying to render it, it seems like, which is a comment you made earlier in the Langspace review.

Yeah, Braintrust as well. So with these traces in hand I worked with one of the subject matter experts to work through about 50 or so traces and perform the open coding exercise. And I wanted to show you a little bit about what that looks like. So of those 50 traces, we actually made comments on 21 of them. And if we go into the review UI, we can kind of look at some of the comments that we made. And in particular, we're trying to identify things that surprised us, identify errors, or identify ways that were maybe unexpected and can be improved in terms of how the AI responds to the queries that it was given. And once we had all of this in place, we were—

Yeah, I mean, just like having the subject matter expert do the labeling, that's just so beautiful.

Beautiful.

You get, you get so many points for that, right? Like, yep.

Able to then do the axial coding exercise and come up with this failure mode taxonomy document.

I wish they showed how they used Braintwist to do the open coding. Like we kind of just took it for granted that somebody open coded it in some magical way.

But anyways, it looks like the Comet UI is on the left-hand side where that's where they're entering it, but yeah.

But it's not clear to me, like, what are the hot— how do I get into my flow state was not answered anyway.

Yeah, I mean, I know that there's hotkeys here for everything. Like, you press R, it goes into the notes, and you can go left and right key. I know that. He didn't necessarily show that, but it's fine. Um, but yeah, that, yeah, I presume he just, uh, use these comment fields.

Is given. And once we had all of this in place, we were able to then do the axial coding exercise and come up with this failure mode taxonomy document.

Actually, let me just go back for a second. I do agree with you, like, I want to know how he set it up because like if you just push R, it's not going to have this comment thing, you know, it'll have just notes like we have— we're trying to use in class. And he had this like thumbs up, thumbs down. So it'd be good to see, like, yeah, we can dig into that later, like how we set this up. It's interesting.

We were able to then do the axial coding exercise and come up with this failure mode taxonomy document. And again, all the code required to do this is in the repo. So you can walk through this and recreate the same artifacts yourself. And after that, thought, hey, it'd be interesting to see if we can come up with a way to use AI to predict for each trace what failure modes were exemplified in, in it, in them, and then kind of come up with a way to then take that information and make it available for human review and correction so we can come up with a golden dataset of failure modes. And so in the repo, you'll see how we actually created those predictions and saved them as a JSON file.

And with that JSON file in hand, I was able to upload it as a dataset here into Braintrust and create this particular view, which So I, I think I heard that he downloaded something and then he uploaded it to the same tool, which is kind of funny, uh, but okay, um, you know, maybe, maybe you don't have to do that, but whatever.

It might look really interesting to you because it probably looks unlike any view that we've seen thus far in Braintrust. That is because I created a custom view just for failure modes called Trace Failure Modes.

This is such a beautiful table. It's crazy that they don't help you get here. This is the money table that you want for your AI application. Anyways.

You have your inputs and then you have your failure modes.

You have your custom bespoke failure modes.

Yeah.

And label it for each trace. Like, what more could you want? Yeah.

This is the thing that, like, we were saying we wanted last time, like, to debug the Axial code. We wanna see it, like, almost like a spreadsheet. Totally.

Brian, you should build this for us. Thank you.

I already have one. I already built this.

Important for us to do the review of the predicted failure modes for each trace. And also the ability to add custom columns. And we can go ahead and select from a field in our input or metadata or use BTQL if the source of those custom columns is deeply nested in our traces.

So I just want to predict, like, what are you thinking, Brian? It's like, I would— so use a notebook is what we feel— what I feel like when you do this, like, data manipulation, pulling apart things, blah, blah, you know, create calculated fields, that's when I'm like, I'm out.

Like, totally.

I mean, as soon as you said BTQL, I find myself wondering how verbose is that? How, like, what does that support? Um, if it's, if it's just DuckDB, then like, why do I need it in this like custom UI? Like, I definitely would rather just play with this in a notebook. Also, the more I get these metrics, the more I start wanting to make charts and slice and dice on different groupings. I want to do some pivot tables. So yes, I definitely agree with you, Hamel Husain Husain Husain. I built some summary statistics into our app because there are some certain things I just want to quickly know, which all my summary statistics are pivots. They're just literally like, you can add tags to the prompts that you are evaluating or the experiments that you're running or the examples. Those are all things you can tag. And I just do pivots on those different tags so that you can say, oh, in general, we suck at writing summaries for transcripts related to pitches, just to quickly do the roll-up. But yeah, absolutely agree. Where's my notebook?

Where is my notebook? Yeah. And then like BTQL, history is riddled with all kinds of QL.

Okay.

GraphQL, whatever QL. I don't wanna learn this other QL stuff because—

I don't know. I haven't looked into it.

Yeah, maybe it's good. I'm just being funny.

Is it just DuckDB though? Cuz it's probably just DuckDB.

Well, can you click on it in the video?

Sorry.

Okay. Okay. That's so funny. Okay.

All right. We'll play.

And with this in hand, we now have a way. For us to again engage our subject matter experts and go through these item by item and make corrections as necessary to come up with a golden, a golden dataset of failure modes that we can then use to prioritize the evals that we end up starting with.

Wade read the course reading. Thank you.

One brief thing I don't love in the previous— uh-oh, notebooks mentioned. I'm getting excited. Can you go back like to— 3 seconds. Yeah, just one thing that I'm noticing here is he's got this like metadata and it's a YAML file, and the YAML file is keys and values. That just, that doesn't appeal to me.

I, I'm looking at engineers versus data scientists, very classic situation, right? It's like, oh, I have something to show you, I'm going to make a YAML file, and data science is like No.

Yeah. I mean, I love YAML. I just, I don't feel like for this purpose YAML resonates with me. Like to me, this is like the most clear, absolute, like obvious, that should be a JSON blob. You've got—

I think you can make it a JSON blob if I remember correctly.

Presumably it's already hidden values.

Yeah.

No, yeah.

You can definitely, but you, um, I didn't think you would. You surprised me as saying you wouldn't think it's a JSON blob. I thought maybe you want to show it in like a table or something.

Yeah.

Certainly. I mean, I definitely think that these are binary values. Show them to me as binaries in a list. As in the UI, like, duh. But it's weird that even the— let me give you an example of why I would expect this to be in a structured— I don't know. YAML also supports nesting. So, it just feels like a weird choice to use YAML here. And definitely not display it.

A final thing I did was actually create these plots in the notebooks to look at the predicted failure mode distribution for across all of our traces.

Nice.

And then also to understand that same distribution but only looking at the traces that we created comments for. So that's in the notebook as well. And with that, Homework 2 is done. If you have any issues or questions, please feel free to ask in the Discord. Thanks.

Well, all right, so I love how at the end of both Harrison and Wade's error analysis, not one of the evaluators was like toxicity hallucination.

They were automatically generated though, which is—

well, at least there was some attempt. Um, but it makes me feel vindicated.

Yeah, I feel like both left me cold in what they chose to, to evaluate. Coming back to this point of like, they're not using like off-the-shelf things, but none of them felt really like good evaluators, right? Like if you ask me like, oh, you want to evaluate like a recipe generator, the things that are going to be in my like evaluator are going to be like How complex are these recipes? Are they too complex for me? Are they too complex for, like, you know, the ingredients I usually have around? Another one that I'm going to be looking at is sort of like, do they contain, like, certain allergens? I think that's something that's, like, highly relevant. He kind of talked about that with, like, the plastic thing. I just— I felt like, yeah, these evaluators didn't feel really respectful of the, of the, like, task. They felt really obscure and canned.

I feel like it was a jagged competence between the two that we've seen so far. I, I don't feel like there's a landslide victory yet. I don't— I have to, like, reflect on it, honestly. But, like, It's just like certain things were good about each thing and they went wrong in different ways in both places, potentially.

Yeah. Okay.

Part of it is like, you know, both of them were doing a task where they like have absolutely no stakes and it's really hard to like, for unless you have practice, like in an educational setting, it's really hard to force yourself to be the person or like put yourself in that person's shoes. Right. So I do give them a little bit of credit for like trying at least. But I understand that it's hard. You gotta live this like teaching life every day to really be that person.

Yeah, it's really interesting. I mean, I definitely give Wade credit for like the process thinking of like, let me involve a subject matter expert. I love that narrative, even though it's like outside the tools, like who cares, like it's still like an important idea. So ultimately, like, you know, he's, he's probably more likely to get a recipe bot that is useful. Um, you know, not that you, you know, they're both optimizing for different things. Like, you know, he was maybe trying to make a recipe bot. Um, but yeah, I did find the part confusing about the, the synthetic data generation was definitely something went wrong there, uh, to like debug with, uh, Wade at some point.