Can Claude Build Your Evals?

All right, so this is what we're going through. Anthropic released this thing today, Automating Eval Design and Hill Climbing with Claude. Now we did check out this thing previously, this plugin eval thing, and it was a really bumpy ride, and I'm hoping that this one will be better. I did take a quick look at it. It does look a lot more promising, at least from the blog post. But let's like go through the blog post real fast, just like super fast. So the name of this blog post is like Automating Eval Design. And hill climbing with Claude. Okay, great. And then it's like, gives you a little bit of an intro about what evals are. We don't need to read that. And then they have bundled this skill under the Claude API skill. So there's a build eval and a hill climb. And so like in this article, we highlight the principles of good eval design and hill climbing first, then show how Claude code with the Claude API skill applies those principles. Okay. And it has eval design. They kind of had this eval design thing where like, okay, eval tasks. Mirror production. I agree with that. Performance improves with stronger models and more thinking. Yep, agree with that. There is passable headroom at the frontier. So what they're trying to say here is you should have evals that are hard. You shouldn't just score 100% all the time. I totally agree with that. I like that. Like, you know, you shouldn't be saturating your own evals. And, you know, you have a— okay, they want you to say low run-to-run variance. To make sure your evals don't have too much noise. So I like that. Actually, did I, did I mischaracterize there's a passable headroom at the frontier? Hold on a second, let me see. I think I mischaracterized that. It says the most capable model at the highest effort should be well below 100% on evaluation, otherwise you can't reliably judge how changes impact performance. Okay. Importantly, the gap should not be explained by impossible or ambiguous tasks. common tell is that task fails on every evaluation run regardless of the number of replicates. A good task is one where 2 domain experts reach the same verdict. Okay, so what it's saying is like you should not saturate the eval, but you shouldn't just have it be impossible either. It should be like a really hard task that you as a human could conquer, but the AI hasn't conquered it yet. So that's— I agree with that. Then they talk about adversarial sampling. It's like, okay, model capability is jagged and you should try to focus Where like maybe if you— there's a way to sample where the model is failing, you know, you can sort of— that'll be more sample efficient, like you'll have higher information potential. So it's like, okay, model capability is jagged. If you pick cases because today's models fails at them, you're sampling in the valleys of one of the model's capability surface. Evaluation can end up measuring that model's failure fingerprint rather than what's intrinsically hard. I was telling you about sampling. You say, so like pick hard cases because a human judged them to be hard. Okay. Interesting. So pick cases because a human judged them hard. I don't disagree with that. But I mean, they also have to actually fail. Yeah. Yeah. You have to find, I guess, like before you judge anything, maybe you've already labeled the data of what's hard. I don't know. Like, I guess. Yeah. Well, there's a lot of things that are extremely hard for a human that these models do just fine, you know, vice versa. So I guess I don't know that I completely agree with that. Yeah, I'm not sure exactly. I would say I definitely want to sample things that appear like failures. And I definitely want to sample diverse— sample across the space, diverse user interactions or diverse sessions that do completely different things so I can understand, get a mental model of the kinds of different things people are doing. But yeah, I mean, yeah, I don't know what this means exactly yet. I think I'll hold my judgment. Okay, so they finally get into this skill. So you have like called Claude API build eval. The build eval command in the Claude API skill turns these principles into a guided workflow. And then basically what it says is like Claude interviews you, builds the eval inside your code base, and pauses for approval at specific points. Claude helps you sample inputs. To build evaluations in this order. So it like looks at production transcripts, uh, then it looks for bugs reports, then 5 to 10 cases you write by hand, cases synthesized from your code base. And then it says prioritizes production traffic, but it can also generate synthetic data anchored to a few real examples. Skills instructs Claude to generate a simple page that shows you every input and waits until you can confirm them. It's like below we show you an example of, of a sample set of inputs for an email router application. Okay, so like these are some inputs, and then it's like you validating the grader after the inputs. Claude proposes cheapest grader that fits your application's output. So either they give you programmatic verification, which is code, or LLM as a judge if you can't use code. That's great. Then it says Claude grades a handful of cases and asks whether you would have scored them any differently. That's interesting. It tries to— it just goes, well, we'll see. I think we have to try it. It says So Claude will like grade it first and then ask you if you would have done anything different. And it says it's important to read a sample of scored transcripts before believing your evaluator. Okay, I mean, speaking my language here. Scored failures are among the most common ways an evaluation is misconfigured. Okay, says when you validate the grader, the skill tells you the size of the evaluation set, runs the baseline, and prints a score with a confidence interval. Curious what the baseline is, but we can— maybe we'll find out when we run it. It says what you get back: the cases, the grader, the runner, one JSON line, and one full transcript per case, and a plain page that lists each case's score with the link to its transcript. The writing could have been clearer here. I feel like that's written by Claude, but it is Claude. I guess they're allowed to write with Claude. I mean, if anyone's allowed to write with Claude, I don't know, it would be kind of weird if they didn't write with Claude, maybe. I don't know. If you want—

I do think they've done a lot of editing on this piece, at least, Yeah, it's

not

slop.

It's not slop. I was able to read it just fine until like right now.

Claude 5.5 does not write this good.

Yeah, it's definitely— it's not, it's not slop. If you want more than that, that page shows, it just has Claude and we'll build extra page next to it. So these are static HTML files, so they have like some kind of viewer. I'm not in love with this viewer, but Then again, it's illustrative and it's for a blog post. So how much can you possibly show on a page? Just because it's showing you like the raw trace is kind of— I don't know. But then again, let's not nitpick it too much. This is a blog post only. Okay. Diagnostic checks during the baseline.

Also kind of curious what this viewer is because it's not inline in the terminal. So is it like writing an HTML file? Is it like building a new app or something? So it'll be interesting to see.

Yeah, that'll be interesting.

Like, does it wire it up to Vercel or is it doing like a SQLite database or just saving them back to JSON files maybe? I don't

know. Yeah, we'll find out. Diagnostic checks. Okay. It says like during baseline runs mentioned above, Claude checks a number of things. So Claude runs the grader twice on the same output and reports whether the verdict changed. Plumbing. Claude checks for timeouts, API errors, and cutoff. Answers to ensure infrastructure noise doesn't pass as model variance. Okay, I guess this is for LM Judge then maybe or something. I'm not sure. Headroom. If the baseline already scores 95% or higher, the skill warns the user if— if it alerts the hill climb should aim to explore the cost or latency rather than quality.

Makes sense.

They have this hill climbing section. Let's, let's try it out. Let's try this like build the eval first, this build the eval thing. So let's try it. So let me share my screen here. I

guess we probably don't have production traces or the production tickets to give it, but we can work without it maybe.

Yeah, so actually I prepared something here because I know we're doing this livestream. So I have here in terminal these traces, and I don't have a production application. I can't like share publicly this production application, but I have 100 traces from a real application, this like a real estate AI assistant thing. I can show you what it looks like. Let me actually open it in my browser. I should have shown it here. So let me change the window back to the browser here. So here's what the traces look like. There's 100 of these. I don't know if you can see this. This is like from a real estate or this like apartment leasing assistant. It's like a real production system. And basically these are what the traces look like. This is just the HTML viewer of it, just so you know. Okay, so it's like the user has their system prompt, which I hidden. It's like, you are an AI assistant working as a leasing team member at some apartment complex. Your primary role is blah, blah, blah. Okay, I know you've seen this before, Isaac. I'm just doing this for the humor of the audience. So it's like, okay, so you have a user ask a question, you have maybe tool calls, you maybe have like the assistant gives you some information back, the user responds, blah, blah, blah. And like, you know, we can look at these traces. Some of them are different. Like this one's like, I'm running late for my tour appointment, blah, blah, blah. You get the idea. There's like hundreds of these. There's 100 of these traces. Some of them have issues. I am very familiar with this data already because I've worked with it so much. It's like my teaching example for evals because it's like from a messy, like real situation. And I know, I happen to know like where the errors are. And like, where are the tricky ones? So, you know, in some ways, but I will try to simulate like what it is like, like, let's say I haven't ever done evals before. Okay, like, can I point Claude, this skill, at this data? So let's try that. What do you say?

Yeah, I think that's good.

Okay,

we'll see if it, if it guides us, if we kind of follow it as a smart person would, if it kind of guides us to find all these issues. That'd be cool.

Yeah. Okay, so let's see here. I'm gonna, um, so this is the directory where that stuff is. I'll just show you. Um, so like, yeah, it's in traces.json here.

Okay.

We were just looking at this viewer.html file. And basically what I'll do is I'll do like, so the name of that command, I think it's called, or no, Claude API. And then what was it? They want you to say build. Um, and I'll say like, so I voice dictated a prompt into my terminal and I'm using like their build eval command. And the prompt says I'm trying to get started with evals. I don't have application code, but I have traces in the traces.json file, which we know that I have here. These are 100 traces from my apartment leasing assistant. So this is what I got.

And we got Herder going. It's perfect.

Yeah. It's probably overkill for the situation, but whatever. Okay, so I'm gonna go with it. I'm gonna see what it does. Let's see what happens. I'll read the build eval guide and take a look at the traces. Let's check it out. Now, the way the blog post was written is I think it kind of expects application code somewhere, but I specifically told it I don't have application code. Like, can you still help me? And I feel like it should be able to. I reviewed the build eval guide and profile traces, 100 conversations, 1,400 messages, 13 tools, 27 distinct community system, Prompts. Okay. Um, it says, which, which behavior of the leasing assistant do you want a number for first? I'll build one eval for one behavior. Oh, interesting. Um,

I feel like we should do the recommended thing. Grounded answer, call transfer, tool scheduling, but find failures first. That's what it recommends. And that seems reasonable.

Sure. There is no application code here, only traces. What should the eval run against? Grade the recorded outputs. Yeah, I I think so. It says—

It's interesting that it had an option to try and like replay them all through Claude. That was interesting.

Yeah, I don't even know what that means.

It's like, I guess, rebuild the application.

Yeah, I don't know. I was just like, I don't know, that doesn't seem like a possibility. Okay. Do these traces run— contain real caller data such as names, phone numbers, emails? Is it fine for them to live in this repo and be sent to Claude? Yes, it's fine because I've already anonymized them. This is totally fine. And submit answers. Oh, it says I will render the traces compactly. System prompt shown once per distinct prompt. Tool results trimmed so I can read them all. Okay, I'm not quite sure, uh, I'm not quite sure what that means, but you know, then again, it's just— it's a thinking trace, so I won't read into it too much, I guess. Um, I wonder what model I'm using. I don't even know. Oh, I'm using 5.5 Sonnet. That's fine. I'm not trying to go broke here because I'm not really sure how much, how many tokens I'm going to turn. And plus I'm not really testing model capabilities, I don't think. I'm trying to test the UX and workflow of this. And also you don't want it to run too slow because we're trying to just test this out. I haven't noticed this before, deep in thought. Of course, like I don't really use, I don't stare at Claude like this usually for this long, but it is thinking, it's doing some kind of thinking.

You just do it in async. You kind of kick it off and go to the next thing, or?

Yeah. I mean, if it's doing something like this, like I'm gonna do something else. Maybe it's not a good thing because then I get distracted. But yeah, if it's thinking like this, I'm not gonna sit here and just necessarily stare at it. It looks like it's using some kind of scratchpad. Okay. It's— I reviewed all 100 traces. Now I'll verify my notes with some scripted counts and check the text channel system prompt. Which differs from the voice one. Okay, that's interesting. It kind of knows that there's a different modality. That's cool that it noticed that.

Doing some code calculation. What is this doing? Checking dollar and square foot.

No idea. The scripted accounts confirmed the transfer and placeholder issues. Okay, it caused some transfer and placeholder issues. Okay, great. But grounding needs a closer look since voice traces answer policy questions without any log tool call. Meaning the knowledge base text isn't in the trace. Okay, I'm not sure about that. I'm now spot checking 2 suspected number mismatches along with the weekday date claims. Okay, so there are definitely, um, these issues in the data. Like, there are the things that it found. I forgot what it— where— what it said, but there are definitely— yeah, there's definitely problems like that. Okay, it says failure modes from least From most to least clear-cut, call transfer issues. 14, great. 14 traces called transfer call and 10 break the prompts protocol. 6 re-asks, would you like me to connect you after the caller already said yes? 3 talk around the transfer. 1 transfer without no confirmation at all. Okay, I happen to be very familiar with this dataset and I can tell you that yeah, this is like one of the top problems. And it actually, actually nailed it. It nailed this specific error. It like knows like the nuance of there and the kinds of things that are annoying, like the, like the user keep— has to keep asking to be transferred. So it picked up on that. I like that. Okay. Voice call hygiene. Trace 93 says triggering end call allowed. And 56 says triggering transfer. Okay. I would want to see these traces to like know. I want to review that. I'm a little bit suspicious of that. Tour scheduling, 35 traces book tour, 7 tell the caller that the tour was rescheduled, but no trace has a cancel or reschedule tool. So the old booking stays. Date errors, trace 79 gives Saturday, January 27th. Okay, this is actually true. This is like one of the problems that are hidden in this dataset. There is no rescheduling tool. And this tour scheduling is actually gets it. So it picked up on this and nailed this. Very impressive that it found this. I like that. Okay. Made up capabilities and facts. Virtual tours offered with no tool support. That is a problem because there's, there's some apartment complexes that don't have virtual tour and then the agent just promises it. Bathroom layout claim with no data, blah, blah, blah. Email placeholder, 12 replies across, blah, blah, blah. End with a little literal your name. Says findings that didn't hold up. I suspected wrong prices in traces 10 and 94, blah, blah. Okay. So which— okay. So give me some data caveats. Fine. Now it's saying which one becomes the eval.

So I don't really like this. Like, I feel like from here, if I was doing this easy vals, this would be great. But I feel like the next step would be to show me the traces so I can look at them. Uh, yeah.

I mean, like, I happen to know that these are reasonable, but I would not trust it like this. Like, I would want to, like, I would want to, like, verify that what it's talking about makes sense, you know? And some of these could actually be off. Like, I still don't 100% trust it. I want to know, like, what it's seeing is right. So there's some other sentences in here that I'm not sure about. Like, there's date errors. I'm not 100% sure about that, but okay. Yeah.

Yeah, so I guess I wish instead of it saying which one becomes the eval, I wish it was recommending it like creating a viewer of just all the failures to look at. Like I wish that was the step it was giving me.

Yeah, I think so too. I think you need to, I think this is the point where you need to look at the data. Like why haven't we looked at any data yet? But hopefully we will end up looking at some data at some point. You know, we'll see. Now it's saying which one, becomes the eval. It's a little bit confusing wording. Like, what do you mean one becomes the eval? Like, can multiple of these become the eval or just choose one first? You know, I don't know.

One of the problems here is, is like, I guess this is like you should have a course so you learn about evals because it's like which one becomes the eval? It doesn't really tell you how to pick. Like, should you pick the most prevalent one? Should you pick one that is harmful to the user? I don't see how you know which one's harmful to the user if you don't see the trace. So it's like, it's like if you don't know much about evals and you're just I'm gonna use this tool, then it's like, how do you pick? I just like, why is that? Like, why is call transfer rules recommended? Is it just because it's like the most clear? Or is it like, I don't see any reasoning about like, this one is like super detrimental to the user, or this one is just super common. And it's a nice starting point because it's an easy fix. Like, I don't, I don't see any, like, why, why would I pick one of these?

Yeah. I mean, there's some stuff going on here, like tour scheduling. They're saying that you need a judge and a code-based thing. I'm not entirely clear that that is— this is like a judge covers the reschedule and date claims. I don't think you need a judge to cover the reschedule and date claims, but that's just because I know the data. But it's no— but okay, let's just— so let's just humor it. Let's like, okay, let's go for what it recommends, the call transfer rules. Like, I think this is the wrong time to be jumping into evals for sure. Like you said, but whatever. Let's see. Hopefully it's making us look at something. Yeah. Okay. So I clicked on the call transfer rules and now I guess if it makes

us look at data now, then I would be okay. I'd be like, oh, okay. It's going to focus us on one problem.

This is the last chance. But if it's going straight into like building the evals. So I remember in the blog post, they mentioned they build the evals and then they tell you to look at the data. after building the evals, and then they ask you whether you agree with it. But I think that's a little bit backwards. I think you need to be like, look at the data and like think about like, should I build an eval for this? Does this even make sense? But I don't know. Let's see. Let's see. Maybe I'm open-minded. Maybe if you look at it, maybe it's fine.

Like it might be okay to label them and confirm in some cases, but you, I don't think that prevents you from looking at traces from scratch because you still need to look at— you can't just confirm what the judge sees because if there's a whole bunch of problems that the LLM didn't catch or is like a product problem that it doesn't even have context for because it's in your head about what you're hoping to release in the future, like it can't— like anything it misses, the human has no ability to catch it, right? If you're only reviewing stuff that it found.

Yeah. I think it's a little bit of a dark, could potentially a dark pattern. Okay, so let's see what it's saying. It says the inputs are in this folder, Claude Hill Climb Transfer Protocol inputs.md. I don't know why it's in a folder called Hill Climb because I'm not trying to hill climb anything right now. With labels in cases.jsonl, it has all 33 conversations. So these are like 33 conversations about the voice transfer rules, and it found like 11 should transfer, 19 should not transfer, 3 unclears. Then it has like These like unclear things. It says, okay.

Labeled them.

Yeah, it did some labeling. It says, please skim inputs.md. Tell me whether these look like the calls you care about. Okay, let me see what inputs.md looks like.

I'm getting, I'm getting more skeptical by the second. Telling me to look at the data in a markdown file. I'm a little skeptical.

I really don't know about this. I really don't know about this, but let's see. Uh, let's see this. Okay.

I've never— I've never decided to look at, you know, traces and data by putting it all into a markdown file. But—

Okay. So this is the markdown file. And basically what you have here is like, you have the trace number, I guess, the section, the tags, the turns. Okay. So you have some trace ID and what— in case you have a trace.

I don't know what any of that above means, really. I'm like, I don't know if that comes from the data. Like you probably do, but as someone who has not looked at this before, just starting evals, I'm like, did it label, did it say should not transfer and should transfer? I think those are LLM labels. I don't like, I guess the turns, the number of turns.

I would say this needs to be HTML. Like, come on, man. Like we have agents, you know, like we can do it.

Yeah.

We can have something nice here if we wanted to.

Most of that top table doesn't help me do anything. So.

Yeah. The top table, I agree. I'm like, I don't know why this is— oh, it says open viewer HTML, including the system prompt. Should I try that?

Yeah. Let's do that because this looks horrible. I don't wanna read a trace in markdown. Like, why would I? Okay. Open viewer.html.

Where is viewer.html? I actually don't— I'm, like, really skeptical that there is— let me go into this folder. What is in that? Oops. Uh, what's in that folder? There's no viewer.html in that folder. I think it's, you know, I think it's getting confused by the, uh, my viewer.html. See, because I have a viewer.html that I created, so I don't think it's referring— it's not something that it did, it's conflating the two. So okay, fine, you have this markdown file. Now in this case you could say, I mean, this kind of sucks to read. I wouldn't, I wouldn't encourage reading data like this normally. But okay, it says like, this call may be recorded for training quality purposes. How can I help you today? User says preview program. It looks like you might be interested in learning more about our community and possibly scheduling a tour. Would you like to schedule a tour to see Meadowbrook Gardens? No. Let me just keep scrolling down. It says like, would you like me to— it says, seems like you might have a specific question or need assistance with something. Okay, so there's a failure here. There's a different kind of failure here, which is the bot is being annoying as hell. Like it's saying, it seems like you're asking about this. Like, looks like, yeah. Yeah. It's like you're getting therapy. I don't want— yeah, it just— yeah. You want to not talk like that to people. It gets really annoying. Oh,

that next one is really bad. Great. I'll connect you to— it says, do you want— I'll connect you with a representative who can assist you. Do you want me to do that now? Yes. Great. I'll connect you to a representative who can assist you. One moment, please. Would you like me to connect you with someone who can help you? So it asked again. So that was, that was a bad failure.

Yeah. It says, I just want to talk. It says assistant. It finally, at some point it did transfer the call. Now I think like this specific trace, this is trace 1. This is marked as should transfer. So I'm thinking like it, yeah, it marked this as a should transfer. And I guess we're supposed to look at these and decide whether or not we agree.

Oh, God. Trace 18. That tool result inline is just like, how am I supposed to know that's good? OK.

You shouldn't— you shouldn't have to read data like this. Again, this is not weird. This is not 2004. You know, we should be able to make HTML apps. And why did it truncate the tool call tool result? It's completely and wholly unnecessary, but fine. So give me a moment. We have several 1-bedroom lofts available here, a few options.

So I'm a little lost as what we're supposed to be doing. with this. Um,

yeah, I'm a little bit lost. So let's see, what did they ask us to do? It says, please skim the inputs, tell me whether these look like the calls you care about, whether any label is wrong. Oh my God. So we have to like tell it like label, like, no, you should, again, you need, we need an HTML app that allows us to just like change the label. Like, why are we editing like Neanderthals? Like, what's going on?

Yeah. I don't understand why you would. Yeah, this is, I guess, what are you supposed to say? Like, Trace 1, yes. Trace 18, I disagree. Trace 14.

Trace 1 is wrong. It should be— I mean, like, no, I'm not doing that.

Yeah,

give me a break. They're in this markdown block and it's like, okay, these are the turns. You can see the tool call result is truncated. It's kind of a mess to try to read this. And what they want you to do is to manually decide which traces you don't— that could be labeled incorrectly. And this is the label up here. And they just want you to like tell the agent what to do. Like, hey, trace 18 is, should be, doesn't need to be transferred. And trace 19 is wrong too, because blah. And they want you to just act like you don't have any technology available to you. Like you're, you're like HTML hasn't been invented yet or something. So like, I'm being like overly harsh. I mean, whatever.

No, I think it's fair. I think it's pretty fair. I, you tell me, we've only looked at a couple and it's painful. There's no way I would look at 100 traces that way and then Do this.

So we could say, okay, so it's telling us to skim this inputs.md and say whether the labels are right, whether these look like the calls I care about, whether anything is missing. I mean, how the hell am I supposed to know if anything is missing? I guess it assumes that I looked at the data. Okay, so what should we do from here? Should we just say yes, it's fine and see what it does now?

Yeah, I guess.

That's what most people would do. So let's see.

I mean, I guess you looked, you didn't see any issues. So yeah.

I spot checked it. It seemed like it's probably fine. So I'll just say yes, it's fine.

I don't see what else you can do with a Markdown file other than just spot check it.

Yeah. Markdown file terminal. It's like we're on a— yeah. What you should be doing is to redirect this tool to be like, hey, I'm not looking at a fucking Markdown file. Create an HTML thing for me. And then you would have it do that and then have it do whatever it needs to do. I'm not trying to go down that rabbit hole because I don't want this to become like a 4-hour exercise. Okay. So grading first, there's nothing to bill here because traces carry no usage or latency. Okay. I mean, now the grader and the runner, it reads the recorded traces, grades each with code and writes results JSONL and per-case traces in the reports usage. I'm not entirely sure what it's doing now.

I'm very confused.

It wants to run some rm -rf command. Fine, whatever.

Okay, you're very brave. I saw the rm -rf. I thought you were gonna kill it, but you just, you just accept it.

Okay, that's fine. We're good. We're good. Because I have that. I have actually like the certain directories are blocked, so it looks funny, but it's fine. So what's this saying? Let's scroll up and see. Oh my God, I have to read all this. Okay, let's see. Um, I built a code grader from the prompts transfer rules with no judge model. On the 14 calls that transfer, it checks one confirm, exactly one, would you like me to connect you before the transfer? I don't know about that, man. It's gonna— it's like checking that string, would you like me to connect you? Really? Okay. That, that's weird.

That seems like a really brittle test.

Yeah. That's going too far in the code-based approach, I think. Then it says immediate. Nothing is said between the caller's yes and the transfer. Okay, but there's a lot of ways to say yes.

Yeah, so checking explicitly for yes as well, I guess.

Silent, nothing is said on or after the transfer. I don't think that's the right thing to say. If you transfer someone, you can just say like, hey, I transferred you, thank you. There's nothing wrong with that. You don't have to be overly like, I don't know where it came up with that rule from.

The assistant never says triggering or other tool mechanics allowed. So I guess it's— That feels like, um, That no leak though, it feels like it should be a separate eval to me rather than— like, I would split out. I would say like, oh, well, let me make sure that let's do an eval on are we transferring calls properly? And then that one feels more like some sort of tone thing that I would split out.

Yeah, absolutely. Yeah, this should be 2 separate evals. I 100% agree. Okay, protocol okay is the headline. This is It's like so hard to read these sometimes. A case passes only if all 4 checks pass on should not transfer calls. Protocol OK is 1 if no transfer happened. Baseline results should transfer to protocol to— okay, wait, what does this mean again? Should transfer. You have 11 cases. And what is protocol

OK? Protocol OK is 1 if no transfer happened. Okay. Not really.

On should not transfer calls, protocol okay is 1 if no transfer happened. Okay. So it's like, okay, I guess like 2 transferred happened when it shouldn't happen.

Wait, 2 no transfers happened.

I don't know what the hell is going on because it's like,

I don't know why it's this vague. It's like, okay, so protocol okay is 1 means that there is no transfer. So I don't know why.

This is why you shouldn't bundle so much shit into one eval. See how uninterpretable it gets so fast? Because we have all these rules. One confirm, immediate, silent, whatever. I'm like,

see, I feel like if protocol okay is 1, if no transfer happened, they could just call the variable no transfer, and then 1 is true and 0 is false. I don't get why it's protocol.

It says protocol okay is the headline. A case passes only if all 4 checks pass. But then see, this sentence is confusing. It says on should not transfer calls, Protocol okay is 1 if no transfer happened. But I thought all 4 checks have to pass. Why is it 1 only if no transfer happened? That's like, see, that's the problem with slop. Like, you can't read the reasoning because it's so unclear and murky. You're like, it's hard to understand what the fuck is happening.

So I'm starting to feel like they wrote a good blog post, and that's the takeaway.

This is better than a lot of things I've seen. This is getting better. This is way better than the last thing that we did. The

last one we did was a dumpster fire.

Yeah, this actually like, it actually like is getting— it has some promise to it. Like I could see some life.

It did try and get us to look at the data. It did. So that's good. Yeah, it wasn't, it wasn't in a good format or a good way or good interface, but it did try.

So it has like 2, okay, 2 of the 14 transfers pass, a trivial never answer transfer, blah blah, 2 limits to keep in mind. Would you like Would you have scored these any differently? I don't know if I would score these any differently. Like, what the hell are you talking about? Is there something you care about that isn't measuring? How am I supposed to know if there's something I care about that's not measuring? How am I supposed to know if I would have scored these differently?

Yeah, I don't know which ones they are. Yeah.

Yeah, I don't know what the hell is even going on here.

What are some useful applications they probably had in mind for this? I mean, I think they were just trying to do general AI evals.

No idea. Uh, so like, would I have scored? Yeah, so we don't know the answer to this question. Is there something you care about? Okay, I'm just gonna be truthful with it. Okay, I'll be like, I have no idea what the fuck you're talking about. I don't know what cases you scored specifically. And I have no way to look at them. And so how am I supposed to know, would I have scored any case differently? Because I don't even know how things were scored. You're showing me aggregate statistics. And don't even understand what the aggregate statistics mean because the rules are confusing. Is there anything that you can do about this? Like, you know, what is, what are the next steps here? Or what can you show me like a more detailed view of this or like an HTML file or some application that I can look at or something to help me figure out what's going on? And I, you know, I'm just humoring now. I'm like, I'm not trying to act like a smart user who like is stuck. Because I don't know how anybody would go beyond this point right now. All right.

I think this is all you can do. You can either abandon it, but it feels like it's already gone a bit too far without being grounded in what's actually happening in these traces and this data. It probably should have at first point where we talked about it would be nice to look at the data even before we chose this one eval would have been a better place. You're familiar with the data. And we're already getting lost. And so if you weren't here and you weren't familiar with the data, then like, I don't see how you would even know like what call transfer they're talking about if this was a brand new dataset and you weren't here to explain what some of this data is.

Okay, so this is where it gave me this. It gave me this thing. It gave me this thing. So like I complained, it gave me this like call transfer rules, what was scored. Okay.

Better. Okay.

It's better. This call may be— Okay, so fail.

One, trace one should transfer. 4 confirmation questions before the transfer and needed exactly one. So that's clear. For assistant, or I don't know what was at the top, but—

So, okay, that's interesting. Um, so what is it? Fail. I agree with the verdict. Okay. Um, wait, I clicked agree with the verdict and then it just took me to the top. Okay, so we need to basically click on this next one. So I said it says agree here. This one says should transfer. Um,

okay, so the bad, the bad web design might be that you're using Sonnet, I guess.

I don't know. Yeah, yeah. I mean, it's, it's not— so it says, uh, this one fails. Zero confirmation questions before the transfer. No, I don't agree with this. Like it says, you want to speak to a live person and they, they just did it. So I'm like, I would score it differently and I would say like it transferred right away when asked. Okay. Um, this one, uh, so like, you know, this is like backwards, right? I would have preferred to Like, go through this myself. Like, at least show me the traces and have me open code the traces or have me make notes on the traces so I can understand like what good looks like. By having it score it, it's trying to bias the hell out of me and saying like, you know, so you have to fight it and say like, okay, yeah, so like I agree with this verdict, it's fine.

So that's interesting. Let's look at those last 2. So Trace 18, you said it was— you disagreed it was a pass because they said, they said, speak me to a live person, and it just did it, and that was good. And then the next one, they asked for a person, right? And it did a confirmation before it. I guess either of those is fine, whether they— there's one confirmation or, or none.

Oh, either case is fine. It said, call a receptionist please. Would you like me to connect you with someone who can help you? Okay, I guess like Oh, you're right. It still feels okay,

but

no, it's like it should have just transferred it. Actually, it feels okay. It's like I would say if you just say a receptionist, please, I think I would say I agree with the verdict. Fine. Okay, this one fail. Would you like me to connect? Yes. Would you like me to connect? Yes. Yeah, I agree. This is fail. Okay, would you like me? Yep, yep, yep. Okay. Yep, I agree with the verdict.

So if you refresh this page, does it lose? Do you, do you have to hit copy all notes as JSON?

I think you have to copy all notes JSON. Yeah, you have to copy all notes JSON. I'm pretty sure. Sure. I'm afraid to refresh, then I'll lose my stuff.

Okay.

Fail.

Fair enough. Okay.

Fail. And then why did it fail? It says, let's— can I talk to a customer service agent? It didn't annotate that. You see it annotated ask to connect. This should be annotated ask to connect. So there's some issues with annotation here. Connect me to representative. Let me connect you to representative who can assist you further. Please hold for a moment. Ooh, interesting. Okay.

So something Didn't seem right.

This is something else. This is interesting. It failed, but

it feels like there should have been a tool call right after that. There wasn't, right?

There wasn't. No, it just duped the user. It's like, yeah, don't worry about it. I'll do it. It basically lied to them. So I would say I want to fail, but I also want to note that This was a false confirmation. I would actually split this into— I'll maybe have a different eval for this because this is like a different kind of issue than the constant like badgering the agent to like transfer you. This is like, you're telling me I'm transferred, but I didn't get transferred. Or like something funny is going on here and I would want to, I would want to dig into this particular thing. So maybe I'll just disagree because like I don't really know. I guess I'll just say disagree. Okay, we'll do like 2 more. This one traces. Okay, this one should transfer. Would you like me to connect you? Yes.

Um, no, it doesn't. Okay, I agree with the verdict. That seems okay.

No, but it did this thing afterward. Let me connect you to representative. It just—

So it did the tool call and then it said, let me connect you. Triggering a representative.

So I think I agree with the verdict there. Something, you know. Well, pass.

So when it transfers, it should transfer and then say nothing is what you're saying, right?

It just say nothing or say something that makes sense. This says like it transferred them and then it said, okay, let me connect you. It already connected me. Like, why are you— why is it saying let me— oh, maybe it—

Well, I guess there was— it was a transfer.

Maybe there's a delay.

Success message.

Yeah,

I don't know. I guess it depends how the system is working, but I guess there was a success message. So I would say if— Actually, you know what, I think you're right.

I would say this. Yeah, it said— it's basically saying like Please hold on for a moment. Triggering transfer to a representative. I'm forwarding your call to a representative. It's a little bit too verbose here. Like, what? Just say like, okay, I'm doing it. What is all this stuff? Um, so I would actually say like, um,

it's maybe a bit of a tone thing, but I guess if, if it transferred me and I got that message as a user, I don't think I would be super upset about it. I'd be like, okay, it's doing the thing. I'm getting a person, I'm getting what I want.

I would say like, I, uh, yeah, the too verbose of a confirmation. Okay, so let me copy the notes. Let me just say like, hey,

it's a little—

I disagree with some of the ways you label things. I looked through some things, here are my notes.

Yeah, it's a little interesting because you're saying I would score it differently, but it's like what you're saying here is like you would still fail it just for a different reason. You know,

um,

yeah. And I

don't know if this data that I pasted in actually like, okay, it only gave the ones I labeled. It said trace 18, caller said speak to a live person and assistant transferred.

Yeah, see your terminal. We can only see the, uh, oh,

got it. Thank you. Okay, so I pasted that stuff into the terminal and now it's starting to ask me questions. So it's like, see, I pasted the terminal up here. So, hey, I disagree with some of the ways you label things. I'll look through some things, blah, blah, blah. And then it's like, okay, the caller said speak to a live person and the assistant transferred at once with no confirmation question. You marked that fine. So when a caller clearly asks for a human, okay, I would say—

You can say, okay.

I'm gonna say type something.

It seems

like if there's any chance of it being ambiguous, then I would say ask a question. If it's unambiguous, then don't ask a question. Okay, it says Tracy, ATPs both say while transferring, 18 says one short line, I'm forwarding your call. 56 says several sentences. I'd say one short line is okay. Submit answer. I don't like this approach. You're like building the eval and then you're correcting the eval versus, versus like looking at the data first, building like a good, building a mental model of like what good looks like for you. And then going into the evals like this is You're trying to like create— it's basically like create some slop and then fix it.

Yeah.

I don't know. I mean, like, I think it's not— I would prefer to look at the data first. I like that it can help you find interesting data potentially. Like it did find interesting data to look at. It just didn't nudge you to look at it correctly.

Yeah. Well, there was— there were kind of like 3 different errors mixed into one eval, which is going to make it You know, kind of annoying to fix. And so I'm not sure if it's gonna split them out into multiple evals or if it's gonna keep them all as the same eval. And I guess it refreshed the app, so you lost all your annotations.

Yeah. So the question remains, like, is this— which way is more efficient? Is it like— let's, let's keep an open mind. Is it more efficient to create your UL first, and then like, start fixing it? I feel like strongly like the answer is no. I feel like you don't know what you want until you look at some data. You don't know what good looks like. And it's really helpful to look at some data before you jump into evals. Also, the way of going through it has a lot of friction, like the markdown file, and then like the labeling. It encourages you to be lazy because you're like, I don't want to go tell the agent all the instances of where it's wrong. There's just so much friction, right? And so you have to have— you have to know, hey, this is not a good process, and build your own HTML apps and labeling apps, annotation apps, go through it. But if you're going to go through all that work to build an annotation app, might as well just forget the eval and just look through some data and introspect what is wrong. Because When it's trying to create an eval first, it might be doing it with a really bad design. Like, like we saw, it might be trying to bundle in too many things at once and make it very confusing for us. Even in our annotation, we're like, we found like new things like that are wrong that we would actually split up into other kinds of evals or like something else. And so, um, I feel like it just nudges you in a bad direction.

Yeah, I agree. It just I think it comes down to like the way it organizes all the information makes it hard to understand what's going on. So there's lots of examples of that. Like, I don't know how many evals we have right now. I don't know if we still have one eval or if it's split it into like 3 evals. And then when I think about what are the problems that this product has, I guess there's the tone issue. There's this transfer issue. I think there's actually 2 transfer issues. Like, I don't know what the problems with this product really are because We kind of just started looking at the call transfer, but we don't actually know if that's like, you might know, but anybody who's just starting to look at this dataset and starting to do evals, like they just click this recommended thing. They don't know if this thing is important or not.

Yeah, I found it very confusing. And I'm an expert in this dataset. Like I've been looking at this dataset for like more than a year, like very closely. So

I feel like if you had sent me, if you had sent me this, this the, this dataset and said, and I didn't know anything about it, and you were like, tell me what I need to fix with my product. I would probably just have immediately opened a data viewer, just been used to Claude code, make a data viewer and just gone through. And then I would have been like, what are the important things? Because to me, I'm like, okay, we're looking at these call transfer things, but I don't actually know, like, I don't know how often they happen relative to the other issues. I'm still not clear. Like, are there cases where A person never gets successfully transferred? Because that would be what I— like, if that's the case, I would want to prioritize that over everything else. Like, yeah, sure, asking twice is annoying, but if every time it asks twice they do get connected eventually, then— but there's also, you know, a bunch of cases where they never successfully get transferred to a rep and then they fall off. I'm like, well, I'm gonna go focus on that first. And maybe they're the same thing and maybe not. And so I think it's a I think that's, that's to me the problem is like we're already beyond the friction with the data. We're already going down and focusing on one particular issue without actually having any idea if this issue is the important issue to be working on. Because like from this, it feels like, oh, I should just go keep working on this eval. But I don't know, like there was other issues that it came up with. I don't even know what they were.

You went into it, we start, you go into like this deep rabbit hole and there's one eval Like, you're right, it made you pick the most important eval, like, very early, and then you just went straight into, like, polishing that one eval. Um, but we didn't really know, we didn't have enough evidence to know, like, that's the most important one that we should be focusing on.

Yeah, we just like, I don't know, Claude says it's important, so I guess we'll click that. Like, that's, that's what I felt, because I was like, it just says recommended, but I haven't seen anything, so I don't know.

Yeah. Okay, so what's your verdict? Fail? Or pass?

I would fail it. Like, to me, I leave with this, I wouldn't use it myself. So that would be the thing. I feel like if, yeah, I feel like you would get further if you didn't know, if you didn't know much about evals, maybe someone watching has never done evals before. I feel like someone who's never done evals before would get further if they just got the data, they made a data viewer, and they just went through 100 traces and just took notes. And that was all they did. I feel like they would be in a better place than we are in with this dataset now. Even if they didn't do any fancy axial coding, they didn't do any grouping, they didn't do any automated testing, I feel like they have a much stronger grasp on what they should do if they just like went through 100 traces and just wrote some notes and that's it. That's my take, I guess.

Yeah, I agree. I agree with this verdict. I mean, yeah, I don't really have anything to add to that. I think we said everything we wanted to say here. So I think, I think that's a wrap, man.

Thank you for joining. All right, great.

Take it easy. All right,

bye.

Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products, and it's customized to what stage you are at. So this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.