Trying the new Claude Eval tool

We're going to talk about the new, the Claude evals. So let's like, let me share my screen and let's bring it up. Let me share my screen.

So I do not know where the chat is for this.

So neither do— let me give you the link.

Are you sure you're— oh wait, hang on. Okay, you're I found it. All right, I'm good.

Okay, you got it?

Yep, I'll keep track of the chat.

Okay, great. Um, all right, so we got this new, uh, this new thing. It's called Claude plugin eval. See what value of your— see what value your plugin is adding or if it needs more work. And create test cases, run your plugin or skill against those test cases, score these runs, and then run each case again without the plugin to see the difference. And hopefully they make you grade it or do something. So it's like start in your plugin's run folder, or start in your plugins folder and run Claude plugin eval init. You tell Claude what good and bad output looks like and bring in a few real prompts. Uh, Claude drafts test cases and checks, pilots the suite, and tells you what a full run will cost.

Wait, then what does it mean, uh, bring a few real prompts?

I have no idea. Okay, um, we're gonna find out. We're gonna run this, and then it says you'll see each case's score with and without your plugin in your terminal, plus HTML report. Okay. Evals cause them. Okay, fine. Just so let's just go right into this. Let's do it. Okay, so, so we need to plug in Claude plugin eval runs your plugin. Okay, so let's go back here. I think you have to start it with this init command, I guess. Where's the init command? Yep, Claude plugin eval init. So let me share my screen to my terminal. All right, this is my terminal. This is Herder. You have Herder open. Um, and I guess like there's a cloud folder, I guess like plugins. Um, where would be— where is my plugins? Is it like here?

Do you, do you have any plugins?

I thought I do. There's like plugins, like Bundle of Skills. Those are plugins, right?

I don't know. I don't have Claude code, but—

Okay, I could just do this. I could— let's ask Claude, like, um, let's say like, uh, I'm gonna get this blog post, okay, to Testing plugins. Let's say, okay, I want to test plugins. See this thing? Where are my plugins? I understand that I need to run it from the root of the plugins folder. Probably don't need to be using Fable for this, by the way. It's like overkill. Let's, uh, just go to Opus. Still overkill, but whatever. All right, um, I don't need fast. Okay, I don't want fast. Okay, so I need AI to figure out how to do like what I want to know, like Also want to know what plugins I have.

Like, I think in Codex I have like a Figma plugin, but I don't know, is this for— I guess you eval, you can eval maybe existing plugins or make your own plugin to eval.

I think the best way to find out is to try it, which is what we're trying to do. It says, okay, you run it from the plugin root The directory holding blah blah that matters here because several of your repos are marketplace roots. The plugin one level down. You can also skip C. Okay. Okay, fine.

Oh, gives you the okay. Get Hamel plugins, Hamel tools.

Okay, so you have like this plugin. Okay, this Hamel tools one. You have evals one. Um, review loop, I don't even use this anymore. I should delete that one. Website to API, I don't really use that anymore. I— well, uh, that's not true. I think that one has a skill. And this one, I don't use that anymore either. So this is a good time to clean up. Uh, I, I'll clean it up later. Um, hmm, is, uh, the plain writing skill in a plugin? Let me see.

I don't know, might be a little too meta to eval the eval skill plugin.

No, but no, it's like— but it's not an eval skill, it's just a plain writing— it's just a— so okay, like there's a skill here, uh, that we love talking about, um, So, like, can I use that eval thing to eval this skill? Needs to be a plugin, huh? But let's see, maybe this livestream will be over before we know it.

Well, Claude should be able to make a plugin for us if we need one, right?

Yeah, but it would be stupid to make a plugin. Like, you should be able to eval a skill. Why can't you eval a skill? Like, that repo is a plain skill, not a plugin. Its evals directory is already taken by a different harness. I mean, okay, I don't know what that means. What do you mean taken by a different harness?

Well, the scaffolding works. That's good. Whatever that means.

Scaffolding works. Let me actually run it to confirm it. Oh my God. Okay. Okay, I'm gonna exit this part here. You might as well just use—

This is us getting Opus 5'd in real time.

So it's installing the plain writing skill somewhere. And yeah, I don't really know what it's doing, but we can find out. I want to run the eval thing on the plain writing skill. Please let me know when it's ready and where to run the command.

Oh, I think on YouTube our livestream is called Welcome to Our Live Show.

I don't even know how to change it. Um, do you think it's possible to change it?

All right, we have a, we have a question from the chat. Um, Brian Bischoff wants to know, will the recording be available later?

Oh, I could always get Brian to trigger me. He knows.

I don't know, maybe we should turn these into slides so we could distribute some slides.

Distribute some slides. Yeah, where the slides be available. You can always cook up some slides. Brian, we're trying to do this eval thing and we're just getting stuck in the very beginning of like— because it wants— it's like the Claude eval thing that just came out. Apparently you can't eval a skill, you can only eval a plugin. Um, I hate this. Already ran, it works. Let me move it to a durable path.

It's like the vocabulary Yeah, what's, what's an undurable path? What's the opposite?

Yeah, I mean, it's in a temp directory right now, so I was trying to move into something else. I was just reading the traces. So I mean, yeah, we can make fun of it, but it makes some sense, makes some amount of sense.

Um, yeah, he wants to know where the slides are too.

Yeah, yeah, we need the slides. Need the goddamn slides. Okay, um, it's taking a really long time. Maybe we shouldn't be— maybe you should like use something else here. It's just taking away a really long time. Like, why is this taking so long? Oh, you know what, because it's running. Okay, don't—

It's running the evals.

Run the command. The evals. Just set it up so I can. Okay, fine. Okay, so like to run it yourself. Okay, good. Okay, so I'll do that. Okay, I will. I'll do what it says. I'll go there, then I'll run— uh, wait, it says— what were the instructions again? It says run yourself. Okay, let me try that.

Okay, plug in eval. Okay, trust this plugin directory, only evaluate plugin. So we gotta guess.

Okay, what is this saying? It says ablation defaulting with without a plugin something. No idea what that means.

2 arms by 3 cases, 18 runs.

Um, okay, what— why is it— why do I have a score? Score of what?

It's 100% accurate. Like, what more do you want?

Based on what?

Uh, so, um, okay, so someone's asking what the goal of this session is, so I'm just gonna recap. So Claude Devs, the Twitter account, they posted out that there is now this thing called Claude Eval for Claude Code, and it's to eval plugins. And we didn't really know what it was, and so we're now trying to use this Claude Eval plugin and then figure out if it's good or not. What we've done so far is we've taken a plain writing skill. which is a skill from Shreya, and, um, uh, and tried to set it up and it just started like running and giving us scores even though we haven't, uh, given it any examples or told us what it's looking for or anything. It just like started printing out a bunch of scores when we ran it the first time. So we don't know what's going on. We're trying to figure out what this Claude plugin eval is.

Yeah, so you go into your, like, your plugin folder and you run this Claude plugin eval and it's supposed to help you eval a plugin. But like, it just starts giving you a bunch of scores. It's like, okay, what are we doing here? Like, presumably it's like generating some kind of writing and then it's like, it's generating some writing with AI, then it's like generating also an LLM, it's using the skill maybe to like form an LLM message. I mean, we don't really even know what's going on.

Yeah, so we don't, we don't know if it's skills from their market. We believe it's just for plugins, but we don't, we don't know a whole lot yet. We just started. And then Bryce from Cohort 6 says hello to you, Hamel. You're Cohort 6. And he is using Herder. Herder is, it's kind of the same idea of tmux, except you have like mouse control. It's easy. It's very nice. And yeah, you can run like multiple agents with it. Um, so it's, it's Herder. It's what he's using for terminal. It's really nice.

I'm just telling, I'm like, I'm just trying to see if I can get, uh, um. Plain Writing Skill plugin. So I said, what the, what the F is going on in tab 1? Please explain using your Herder tools. I'm just trying to get these sessions to communicate with each other.

Um, yeah, I'm just, I'm really confused if— so I guess what I'm not sure is when you asked it to get the scaffolding in place, can you see, is there a, uh, a diff in the Plain Writing Skill repo? Like, did it, did it like auto create some eval locally, or is this just the Claude plugin doing this?

Um, there's this evals directory. So this evals directory— so if you look at the plain writing skill on GitHub, there's not an— oh, there is an evals directory. Okay, so there's an evals directory there.

Can you do like a git diff so we can see like what files have changed? Okay. So evals/outputs and results.

There's nothing in outputs. It's just this. Wait.

No such file or directory. Well, shouldn't— why should it say no such file or directory? Okay, well, I don't know.

Um, but like, yeah, I think it's putting stuff in here. Um, okay, so let's see, like, what, what is this file?

It's a directory there. You'll have to go into the directory and see what's— yeah. In there, what's inside the directory?

Oh, that's the directory. Yeah, sorry.

Yeah, aggregate_result.json and report.html. Maybe you could probably open the HTML in a browser. You can jump to the beginning on your terminal. Like if you hit up, you can use Ctrl+A. It'll jump you all the way to the left without you having to like—

Yeah, I know. I forget all these things when I'm live streaming.

Oh, okay.

But no, that's good. That's good. No, you should definitely roast me for all this shit. I know you're not trying to roast me. I'm just saying, just roast me anyway. All right, let's see here. Can you see this? Let me see. What is this? Okay, we have a slop eval dashboard. What is going on?

Drafts are 100%. That sounds good. What's that 100%?

I have no idea. No em dash skill fired. No. What are we reading? Okay, partial results. The run was interrupted before the case finished. Average includes non judgments. Baseline no plugin. Okay. No em dash fail. Okay. Prompt to rewrite this paragraph so it reads cleanly. A robust caching layer blah blah.

Okay, so is this is is that a is that a prompt that Shreya had included in her skill, or is this in the plugin, or is this something it came up with?

God knows. We don't know because it's all okay.

Let me see. We'll look at her repo.

Here, I'm gonna say how oh. Where was this prompt created?

Let's see. Let me see what's in there. I'm just gonna look around her evals thing here. Sources. There's a lot in here. Dataset.jsonl. What was the prompt that you were looking at again? What was the—

the prompt was rewrite this paragraph so it reads clearly. There are some evals in here though, like, uh, yeah, I'm not sure. Are robust.

I don't actually— I'm not seeing that specific thing.

Explains a workflow in order. Okay, let's see. So if you go to like the skill itself, I bet you there's something about explaining a workflow in order. So there's this markdown file. Okay, so yeah, like When you present a workflow or sequence, walk through it in order. Use first, second, third, and give each step its own sentence. That's probably what it's trying to do here. So like, um, so explain the workflow in order. Okay, release from— okay, uh, so here's a paragraph to drop in. So it's like coming up with this, right? Releases are cut from tag commits to ship, blah blah. Um, type the commit you want to release. It's trying to give you a process and it's saying like, okay, I don't know what that— it kind of seems meta to me. Uh, the working directory is empty, no README to edit. Feels very haphazard to me, man. Like, first of all, Um, paragraph walks through 3 steps in order, which each step is its own sentence. It does not cram all— okay, that's the criteria. What does weight mean here? What the hell is that? Ordered markers, skills fired.

Yeah, it feels very— so there's a few things that I think are common AI smells that I kind of get rid of because I hate it, and one is like it likes to add had lots of like small gray subtext and like subtitles and labels that just clutter things up. And I see a lot of that here. I think the biggest problem though is like there's all these numbers and criterias and skills. And for any annotation app, like most of the time I want to be able to look at them and know where they came from. And this doesn't tell us at all where anything came from.

Um, I mean, this like doesn't feel like it's meant for you to look at the data even. Like, okay, you see like this is some kind of test, drafts the README plainly. We don't know what— how this test— like there is a criteria, the reply rewrites the paragraph in plain language, it avoids The words leverage, robust, unlock, seamless, landscape, a testament to, and quietly drops the praise. Is that like an LM judge? Maybe. Um, and also, okay, it has these like 3 graders, I guess. No M dash skills. Uh, and then there's like, you need to fire a skill. So that's like, I guess, one thing. Um, so there's 3 graders. Okay. And then like you see, I guess, the 3 graders here. So like, okay, here's the criteria. One, it's like this LM judge. So like we have run 1. Okay, what's inside run 1? Like what you want to know, like just for to check your own sanity, um, what example is run 1? Like what was the output, I guess, of run 1? You don't really get to see that, I guess. Let me see. Oh, oh no, you do get to see it. Okay, our caching layer uses something or the other.

Can you zoom in a little bit?

Yeah. Okay, uh, so like this is, this is like some kind of what— okay, the criteria again is this, like avoiding these words like leverage, robust, blah blah. I don't know why it has to be an LLM, but I guess Maybe it's like semantic matches to those words. This one is easier. This is like no em dash, which is fine. But yeah, I guess like this is the rewrite. What is criteria x2? I guess that's the weight of it. Yeah, weight is 2. Okay, I'm starting to understand by like looking at it. It's not like explained to me very well, but I guess okay. I don't know what 3 checkmarks mean by the This thing?

There's 3 pass, pass, pass. So I guess there's 3 judges.

Okay.

Yeah, I don't know the difference between the check and the pass. I guess maybe check means it voted, it didn't, versus pass being the result. Are there any of these tests that failed? I guess I'm kind of curious.

Yeah, there is. So here's the baseline. The baseline is, I guess, if you don't invoke this skill. That's my— this is my guess, okay? If you don't invoke the skill, what happens? Okay, um, so it failed on this, and you probably can see an em dash in here. Maybe. Actually, I don't know if I— is this an em dash? I don't know.

I'm not sure.

I mean, it might be rendering improperly, but It did fail that. Um, yeah, it did fail that. The— I don't know why there's not the— so there's supposed to be 3 checks, right? Where's this pattern found? So it failed this, like, uh, no.

Oh, it's the baseline.

Oh, it's the skill.

You don't have the skill fire.

Okay, got it. Yeah, okay.

Um, I'm Okay, so I'm not completely sure that this is giving us an eval that helps us improve this or know how good the skill is right now. I mean, I guess it did do some stuff, but I'm— I guess after looking at this, I'm unclear. Like, I guess if the goal is to make sure there's no em dashes, I guess we can say there's no em dashes. And some of these keywords.

Um, so I think the thing that's missing here is like, this is not that actionable. Like, what are you going to do with this? Like, let's read, uh, let's read the documentation because like, look, okay, so like the baseline is without the skill, but then, uh, the experiment is when you add the skill. But of course when you add the skill Then it's going to do the things, and it's going to do the things that Ray better than without the skill. Like, that's— I mean, that makes sense. Um, it's a little bit kind of weird of like an eval to run, I would say. Um, it's like basically, are you testing— does your eval do exactly what it— or does your plugin— okay, in this case, does the skill help do exactly what you said. And, um, this might be the wrong way to be using this. Like, maybe—

Well, I'm a little confused because in their tweet thread they said that we would pick examples of what we liked and didn't like, right? I thought that was in their thread, and I don't see it. We— it didn't prompt us to do that. You can create test cases. So what else? Okay, so I guess it did create test cases. Is there anything further down on the thread on, on this? You tell Claude what good and bad output looks like and bring a few real prompts. So we— it just did that all for us. I'm not hearing you.

Can you hear me now?

Yes. Yep.

Okay. I don't know why I got muted. But okay. So, we ran Claude— we ran this command. This like second command. Claude plugin eval, I'm pretty sure. If I go here to the history, yeah, see, we ran Claude plugin eval. But maybe we should be doing this Claude plugin eval init thing.

Yeah, let's do that.

Okay, let's do that.

We can't see your terminal.

Oh, okay. Let me back out of this. Let me start from scratch so you can. Share different screen. Okay. Claude plugin eval init. Okay. So I'm like trying to— let's set up evals for this plugin.

Okay.

It's probably reading everything.

Howdy, hi, hello from Rocky Mountains. Close to them from Denver. Well, hello, welcome. Okay, so there's a partial suite, three cases previously piloted. It's missing a real should not fire case and has a few fire has fewer fire cases than the floor requires.

I can't.

I hate this fire case versus fire case.

I accidentally hit a button on my keyboard. I don't even know what I.

Uh, uh, you said start fresh, so that's good. I think start fresh is a good answer.

Okay.

Okay.

Okay, step 1, defining quality. Before I write source input, I want to pin down what good plain writing rewrite looks like versus a bad one from our own— from your own experience. What's the most common way this skill has actually failed or annoyed you in practice? Over-editing, missing dashes, jargon? Yeah, I would say like over-editing, definitely. Um, um, uh, transitions are incomplete or abrupt is a big one. Um, and this is like, this really applies to any skill, any writing skill that I've tried to use, and like it doesn't really matter. It's the same problem with all of them. So like over-editing, transitions are incomplete or abrupt, introducing terminology all of a sudden that hasn't been discussed. Anything you could think of that still bothers you? That's top of mind? I don't—

yeah, I mean, I don't really use— I haven't really tried the plain writing skill a lot, but I guess the biggest thing with AI writing, one of the big things that really bothers me with AI writing that skills don't always fix, is they try and make inanimate objects like do things. So they'll be like, oh, the words on the page survive, or they'll use this like analogy of stuff instead of just saying what it means.

Um, okay, using overly dramatic language, throat clearing.

Um, should be good. See how that goes.

Um, and there's this not sounding conversational enough. Okay, I'm gonna press enter on this one.

Wait, wait, wait. You're supposed to answer 2 questions, right?

Okay. Okay. Sorry.

Is there a failure mode you've specifically seen that isn't obvious from just reading skill.md? I think we kind of answered it, right?

We can take a look at skill.md if you want.

So— I think we kind of answered both questions at the same time, maybe. So maybe that would be fine. Someone asked, can we zoom in?

Yeah, we can. Let me let me do that now. Is that better? So this is the plain writing skill. It's just like word choice and tone. Use simple everyday words. No jargon. No puffery. Use consistent terminology. It's okay to use contractions. Not invent hyphenated. Adjectives keep the writing boring. Descriptive and explanatory. Write complete sentences. When you're— when you present a workflow or sequence, walk through it in order. Use first, second, third. Organize a paragraph as topic sentence, then support. Never write 3 or more clauses in one sentence or 3 or more example sentences in a row. Prefer long explanatory sentences over short punchy ones. Be precise, unambiguous, Cut unnecessary clauses. Say exactly what changes in chat. Give context on the problem. No dashes or middle dots. Don't use colons to join clauses. Ordinary. Okay, you know, one, one interesting experiment would be to keep to like, um, minimize the skill to be just like the way I'm reading it. Like, you notice I'm just skipping— I'm just reading the first sentence of every rule.

Yeah.

And, and maybe, maybe it's the case that like that's enough. Like maybe we can get like delete all this stuff. Maybe not, but maybe we can. I don't know. Uh, patterns to avoid: don't give inanimate things fake agency.

That's what I was saying.

Yeah. No analogies or imagery. Never use negative parallelism. It's not X, it's just Y. It's the one that everyone hates. Do not stack rhetorical questions. Don't use vague demonstrative pronouns. I mean, this seems like very comprehensive, honestly. There's also this like, uh, the de-sloppify thing, which is—

Yeah, I mean, I'd say I agree with most of it. Most of it I like.

So, okay, why? Okay, it sounds like there's something you might not agree with, which could be interesting.

Um, oh, I think the, um, the long explanatory sentences. I try and keep my sentences as short as I can, and if they have to be long, then they're long, but I don't, I don't go for long explanatory sentences over short ones. Yeah, so that one I don't know that I agree with. But yeah, I think if AI actually did all of this, I would be very happy.

So it's not— it's not doing all of that. Like, you can't— it's not gonna do it. So it's like, got it. So the real failure modes, grade against our blah blah, that's sharper bar than SkillsMD own rule list. So, so, okay, it's asking you to come up with this from memory, which is like kind of difficult. Like Well, you— I mean, you know what I mean? It's like you need to go through writing samples and then like edit. There's no way you can recall these things from memory. Like you have to do some error analysis first.

Um, there was really no— there was really at no point in this process where it would have been like— it's like not directing you to create any kind of, uh, annotation app or dataset or anything.

Yeah.

Um, Okay, so the real fail modes to grade against are— just read, we're saying that's a sharper bar than skill.md's rule list. I don't think it is, but that's okay. Um, step 2, do you have any real prompts or past bet outputs?

Okay, look at Isaac Flats' blog, um, Shreya Shankar's blog.

But those are for the good, right?

For good outputs. And then, okay, can we pick on somebody for bad outputs? Like, is there somebody that just writes slop on purpose that won't be offended?

Um, I mean, a lot of people that write slop come to mind, but I don't know any of them that wouldn't be offended, I guess, if I told them.

Yeah, yeah, I know.

Um, uh, Does anyone— does anyone write?

Okay, what about let's ask the audience? Give me a blog that you think is slop, and we won't ask any questions. It's not— I don't want to just be— I'll just say it's from the audience, so like there's, um, some anonymity there. We won't call you out, we won't say who you are, we'll just like— actually, that's kind of mean too, I guess. It really doesn't matter, it's from the audience. Hmm. Let me think about this. Okay, you know what? There are some people on X that deliberately. There's one guy who is, you know, deliberately writes with Claude, and I don't think he is. He's like trying to hide it, or he's not. He's like, yeah, I write with Claude, and he's like doing his. It's like according to his goals, you know, and everybody knows it. He's actually very famous for this. His name is Akash, and like he has these tweets and he like— you've seen them, right? Like, uh, you know, like, okay, he'll take a subject, right? And he'll write it with Claude. Mad Max Fury Road Fury Cut tweets in 2 hours. You see what I'm saying? Never shot. And he'll just— he'll take like some trending tweet and he'll just summarize it with Claude. Uh, and then like he gets a lot of engagement and it's like serving his purpose and he's like, whatever. Like, I'm not saying you hate any of this, like whatever it is, like it is. But I'm saying like, okay, uh, let's just say, okay, here are some examples we are trying to avoid with respect to the style. Um, so just give us some of these. You know, honestly, and like, not to, you know, some of these summaries are not bad. Like, they're kind of like, people like them to some extent. Like, I'm not— don't get me wrong, I'm not trying to hate on anybody, okay? Just don't want to get pulled into any drama. Um, I just know that like, you know, it's a thing that he's doing, is trying to write with AI. Um, okay.

I mean, I actually don't mind it. Like, if you're open about the fact that you're using AI and you're doing it with this, I don't mind. You know, it's when it's like you don't say it and then you're trying to sell something and constantly like replying to everyone trying to sell something with your AI bot. That, that annoys me.

It's really weird that you can't— why does Claude not able to read tweets? This like happens all the time. I'm like, is there some problem? Like, I don't know.

I, I have, I have the Cloud API and I just, you haven't used the paid because I found it wasn't, uh, working very well.

I was told to use the browser. I'm like, okay, look, like, what do you mean it's not connected to anything right now? It is connected. Okay, I have to log into this. It's like, dude, let me log into— It's connected now. Okay, so it's getting good, it's getting bad, and I don't know, let's see. Like, maybe, maybe this is like a tortured example. Maybe it's not really good for writing. Uh, it— I can imagine it might be better for coding potentially, where it's like less ambiguous. Um, do you have real prompt— okay, so do you have real prompts, past transcripts, or bug reports where plain writing skill actually ran and you saw one of these failures: over-editing, abrupt transitions, introduced jargon, dramatic language, throat clearing stuff, uh, forest fire cases? Um, I'm just gonna say like I don't have any ready examples because I correct all the bad examples as they come out myself. Um, so it's gonna be hard for that. You have to work off the examples I've given you. I mean, maybe I could, I could point it at like my sessions maybe and tell it to dig for it, but it seems like that's going to take a long time.

So yeah, this is, it's really interesting because it's just having it as a chat interface. It's kind of like you start this process, it's like Claude eval init or whatever it is, and then it asks these questions. And these are like, this, this type of data on, on most skills is not something you just have handy or ready, you'd be like, yeah, it's like, yeah, what would the process be? Would you like leave this open and then be like, well, I'll just like, no, spend a day collecting this and come back? Like, it's not—

You have to be— it has to be in situ, has to be while you're using the skill, you know? Yeah, it's like you need to— there's no way you can do this like outside the context in the moment. It's just not going to work that well. Um, so I totally agree with you. I think this is a little bit— this is going to be hard. It's going to be hard to make this useful because like, like, well, it kind of— yeah, it's pretty handicapped.

And it's mostly kind of asking you for all the stuff that's a lot of the hard parts of the, um, eval process anyway. It's like this plugin, it's going to guide you through it, but it's like Okay, all you have to do is have a dataset of positive and negative cases with what the failure-specific failure modes are, and then you can give— and it's like, well, if you have that, you're almost all the way to evals. But it doesn't help you with any of that part, you know, of the error analysis or process. It just kind of assumes that you kind kind of have the results of an error analysis process on a dataset that you can give it.

I'm gonna say, can you be more expansive with the tests given the skill? I mean, how many rules this skill has? Like 25 rules. Given the skill has 25 rules, can we have at least 25 tests? It's really odd, right? Like, why would it come up with like seven things, like five things, really, when this this particular you know this particular skill has twenty-five concrete rules? Like, it's just being lazy. You know, it's like just trying to give you something. Someone said, "Can we get? Can we have at least one of us to match?" The 25 rules and leave the should not fire examples as well. Come up with others.

I think the fire/should not fire seems like the most useful thing so far. I mean, I know it's a small thing.

Yeah, no, it does.

Just testing, being able to easily test if it's firing at the right times and then if it's not firing, if it's not overfiring, seems like the concrete thing that stuck out as nice.

I would call these— I would call this vibe evals. That's like my term for this, vibe evals. It's like vibe coding. It's like, well, we don't really know anything, but like let's just look at more things and kind of see if we can like generate some shit, maybe like talk out of our ass a little bit, generate some stuff. Maybe we'll like find something. Maybe we'll, you know, yeah, we'll like feel like we're making something. Um, you know, it might help a little bit. Uh, let's see. I mean, but is it evals? I would not call this evals.

So who's the first— what, what vendor do you think is going to be the first vendor to replicate this in their product. You don't have to call anyone out.

I actually just got a text message right now from a friend who may have come close already, but I don't want to call the— I don't want to call them out just yet.

Um, let's see if, if a lot of this stuff had like small— if there's a way to have this but have small widgets in there, like do you have examples?

No.

And it could like you could like put in a little prompt and then you could like judge it. Like if there's like widgets in line, it could be really good because then I could just say I'm just going to spend 4 or 5 hours going through this chat and it'll help me create all the examples that I need with widgets. And like with— if like if it did that, that could be really useful.

Um, Hold on one second.

So wait, skips pure math. Oh, that's a should not fire catch-up.

Okay, so we got all these things. Every prompt will embed a couple of concrete facts so players can catch the blah blah. It says 30 cases total. This will cost mostly more than the blah blah. Okay. So what do we do when we proceed with writing al-ṭarīdī? Yes.

Your video got super blurry for some reason.

Is it still there? Still blurry?

Yeah, the screen share is fine, just your, your video.

Oh yeah, I mean, yeah.

Are you still there?

Yeah, I'm here. See if I can fix the video.

There we go.

All right, good. Shreya is asking us what's the verdict on the Claude eval thing.

I tell her, I was like, I don't think it's looking good right now, but it's like, You know, uh, oh, I missed some of the chat, but the ideas for, uh, finding examples of slop were LinkedIn articles with emojis. That would be a good one.

Uh, there's plenty of, uh, yeah, the corpus is quite big for that one.

Let's see, the docs have an escape hatch to let you directly customize the graders, but the auto-generated graphers definitely feel vibey. I mean, I guess to me it's like, if you're gonna have to direct— I mean, maybe, maybe like if you're directly customizing the graders, I mean, maybe it's like a cookie cutter thing where it like kind of helps you structure it. But if you have to do it manually, that's— I don't know. I feel like I'd need to build like a little annotation app and all that other stuff too.

So Okay, it's cranking. I mean, it's trying to do something. Let's see how much usage I have.

I feel like this— the main thing that this thing could do that would be really valuable is if it guided people through and helped people through like a good— maybe not best practice, but like a good reasonable way of evaluating the skill. Like, how do they choose examples for their dataset? And like, how— what do they look at? Do they look at 1? Do they look at 10? Like, how do they do this? So if it guided them through a little bit of that process, I feel like that would be the main value of using this agent here. But it's not really guiding us through a process or like pushing us to look at actual examples very much. Because the interface doesn't support it. And it did ask for some examples, but it didn't really make it clear that it really needed them. And it's a chat interface, so it's kind of like, I don't know, I don't have them. So I don't know, I feel like that's my— that's my issue with it.

It's still cranking. Uh, what is it doing? It's writing a generator script to create all 30. So it's like, okay, it's creating synthetic data, I guess. That's what it's doing.

With, without ablation. The pilot's running in the background.

All right, let's see, what is it doing right now? Okay, okay, so all 30 cases are written, no MCP tools are involved, blah blah. Okay, whatever. Pilot's running in the background, pilot's running in the background. Okay, wait, it's still running in the background? What's going on? Are we done? I thought it used to show the agents running in the background in the bottom.

It says one shell still running.

Where do you see that?

Right.

Oh, right here. Okay.

Yeah. Okay.

I missed that. Okay. It's taking a while. Not done. I already found a bug. Several regex graders. So it's writing the graders. I guess it's writing the evals.

They're throwing YAML parse errors.

Okay, let's do a git diff so we could see like what is actually happening here. Um, okay, so I guess they're in this thing. That's strange. Still okay. Hmm, I'm not sure.

Maybe that's what's in the directory.

It's just the same thing, right? It's just like this, like more stuff in there or no? Yeah, let's see.

Is it all the same stuff?

Same, looks similar.

Oh, maybe it's just putting it all in that JSON. I don't know.

Okay, I don't know, man. I would have given up already.

Me too. I feel like there's— it's just like, it's a weird, like—

Like, I just, I feel frustrated already. Like, what? Why? What are we— what does it do? Why is it trying? Like, what is it doing? Like, we don't—

Yeah, it's like you have to be in the loop because it keeps asking for feedback. But it doesn't tell you what the hell's going on, and so you don't know what to do.

So I think this is like a failure. Yeah, fail. If they're like a— if there's like a special effect, like a stamp, red stamp, fail, fail. So that's all right, like it has to fail.

Okay, if you work for Anthropic and you, uh, made this skill, I know a course that you can take.

Hey, I mean, you know, emails are hard, man. You know, like, they are hard. Um, You know, I'm, I'm glad they're like trying to do something about it.

Like, yeah, I mean, it's only a first pass. I'm sure it'll get better. Yeah.

Um, but yeah, I'm just super frustrated. I don't think the terminal— maybe, I don't know, I'm not sure it's the like right thing to do. Um, like I said, it's like vibe eval. So yeah, I would just call it a fail.

Yeah, it would be interesting to see this, uh, kind of thing built into like, uh, like say the Codex app or like an app where it could like create annotation interfaces and open it in the in-app browser and like have you like interacting with that. That could be interesting.

Yeah, I agree. That could be more interesting, but it needs to put you in the loop. So there needs to be like an eval mode that you can turn on Where like while you're using the, the plugin or the skill, it then asks you for feedback. That's the best time to ask for feedback. And then it can like try to like make some tests based on that. It can try to do something and like record your feedback and like they can show you some scores like based upon what you say. Like, hey, like you thought this is like a failure that keeps happening. Boom. Okay, that would be super cool. Somebody should make that. Maybe we should like— I mean, I should stop everything I'm doing and try to make something like that. But, uh, yeah, it would be cool.

The other thing for like writing that I would like to do, which, yeah, I guess I built it into my writing tool, but I've been doing so much video that I haven't been doing as much writing. But it's this idea of like a delay between Like a delay between when the skill fires and it captures. So for example, this plain writing skill, there's actually— I want feedback, but I also want it to capture multiple states because I want it to be like it used the plain writing skill and it made some edit or wrote a piece. And I want to capture that AI output of like what it did. But then often I also want to capture like 5 minutes later, what does that file look like? Because it wanted— I wanted to know like, what did I manually change as well, um, as a diff? And I think that would be really helpful to have, um, for evaling this particularly, is like, what did I think? And then like, what did I manually change? And, and even stuff like, did it make it into the final blog post? Did I ultimately delete that sentence? Would be helpful information too.

Yeah, no, totally. Um, you want an annotation interface, like you want to annotate while you're— I mean, at the least you want a side panel to come up while you're using, when that skill is invoked, and like write something there. So like something, right? Like it can't be, it can't be out of the loop. I will go ahead and end this, end the stream. I think we have the verdict. We agree on the verdict here.

Well, thank you everyone for watching and chatting.

Is anyone anyone there? Let me pop into the YouTube.

See, yeah, there's been a few five five people.

All right, that's more than I thought would be here. Got Kelvin. Hi, Kelvin. Daniel.

Okay.

Dreams API. Oh, sorry about the zooming thing. Um, hopefully we fixed it. But all right, see y'all, see y'all next time.

See ya.