8 Claude Code skills I use for building better AI evals

It's gotten much easier to ship AI products, but figuring out whether the AI actually works the way you want is still hard. And one of the biggest mistakes I see are people jumping into evals before they looked at any failures that are actually happening in their product. So my colleague Shreyashankar and I built 8 eval skills to help with that process. We've iterated on these skills by working with over 50 companies and teaching over 5,000 students in our AI evals course. I'll walk through all 8, what they do, and when to use each one. Okay, so you can find the eval skills on GitHub. I have it pulled up right here. I'm going to share the URL up here above to the GitHub repo, and the GitHub has a nice README in it. You can see some information. It's important to start with this table here. You'll see a bunch of skills. The entry point is the eval start skill, and the eval start skill Just lets your agent, whether it's Claude Code or Codex, route to the appropriate skill. So there's this table that your agent can read. My favorite skill out of all of these is the error discovery skill. It's the most valuable skill because it guides you through looking at your data in the right way so that you can build good evals. You will get directed to this skill if you already have traces and you want to find failure modes and you haven't discovered what your failure modes are yet. There's also this eval audit skill, which is really handy if you already have evals and you want to find out what's wrong. And there's a whole bunch of other stuff like getting help with writing a judge prompt, how to validate your LLM as a judge and make sure your LLM as a judge is good, as well as some other things like how to generate synthetic data or how to write a custom annotation interface or how to evaluate search for your RAG pipelines. This is what comes with this evals plugin. I really want to encourage everyone To start with the error discovery skill. It's the most powerful skill in the set. And let me walk you through what it does. We can take a sneak peek at the error discovery skill. Now, the error discovery skill does a lot of things, but it goes through different phases. So first, it lets your coding agent understand your data and instructs it to read and parse the data and to identify what the structure is of that data. Then it tells your agent to identify what are the dimensions of variation. which is just a fancy way of saying identify what's the most important things to pay attention to in the data, what dimensions in the data account for the biggest differences. And it also instructs your agent to think about what bad means. Phase 2 instructs your coding agent to design the annotation interface that you're going to use to look at the data. And there's some design principles here. There's these core Gestalt principles. This is design terminology. It's not something we made up, but this just lays out some design principles around color, spacing, backgrounds and containers, opacity and contrast, as well as some visual encoding rules, things like typography, border container, things like that. And then phase 3 is where we instruct your agent to build the review interface or use an appropriate architecture, and then finally build the HTML app. And we don't have to read every part of this, but essentially what it does is it allows you to quickly get an interface for annotating your data. And let me show you what that looks like. This is what an annotation interface might look like. What I'm showing here on the screen is an annotation interface for a real estate assistant. And the real estate assistant is a chatbot that has multiple turns. And what we're doing is looking at the logs. And we can see here, I can see the system prompt. You are a leasing team member at the Meadowbrook Gardens apartment. Your name is Taylor. Your goal is to answer questions and encourage customers to schedule tours. And that kind of gives you an idea of what this agent is about. And then we see multiple turns. We see an assistant followed by user turns. And you can see like this seems like it's a phone call because this call is recorded is the first thing the assistant says. And you can see there's a back and forth conversation. There's even a tool call where a transfer is taking place and then a success. And we can page through some of these and see the different conversations we see. A user is asking a question about a specific apartment and when it's available, and we see there's several tool calls being made, and there's this tool call that's getting apartment availability and a tool response followed by some other conversations. So you can see this allows us to look at the logs, and this is exactly what this skill allows you to do, is create something that has good ergonomics and allows you to flip through the data really fast. Not all annotation interface will look like this. The agent will create an annotation interface that is customized to your data. So it's likely that your annotation interface is going to look much different than this. The next phase of the skill allows your agent to intelligently sample data. Looking at data is very tedious. You can imagine if you just were to go through logs like this one at a time randomly, it would be very monotonous and you'll get tired quickly. It's really important that you're looking at data as efficiently as possible. So what that means is it's important to look at some data randomly, so to make sure you're exploring the space, but also making sure that you are looking at diverse samples so that you are exploring the breadth of your product and sometimes also drilling into failures that you already uncovered so that you can learn more. And so that's exactly what this part of the skill does, is it instructs your agents to cluster your data and sample from the clusters, but then also random sample. so that you can do that exploration. As you see here, in this phase, we prioritize diversity. The goal is discovering failure modes, not estimating how common they are. And then finally, there's this interactive review loop. This is my favorite part of this skill. This instructs your agent to enter an interactive review loop so that it samples differently as you annotate traces. So as you discover new errors, the Agent will update its sampling mechanism and show you appropriate traces. So we can click on this review loop to see what's going on here. See that there's breadth and depth. The goal of breadth is to cover as much of the dataset as possible, and depth is when once you found an error or a failure mode, examine it thoroughly and find more traces like that. And the way it works is it monitors your annotations in the background. and process them as they arrive. This skill talks a little bit more about what death mode means, how to actually go about that, how to do breath, and some rules around re-reviewing things. And it's instructing to keep serving you samples until you converge or you're not finding any new kinds of failure modes. So that's the error discovery skill. We have a video walking through the skill in a lot of detail where we do it live. We'll put the link here so you can get to it, and if you want to dig into how it works, you should totally watch that. By the way, if you're learning AI evals, I put together a set of flashcards that walk you through the most important concepts in our AI evals course. I'll leave the link to them below. The next thing I want to show you is the evals audit skill. This is my second favorite skill out of the plugin, and basically what it does is allows you to quickly identify common mistakes that people make. In their eval setup. We've been using this skill successfully for years to uncover real problems in people's eval pipelines. And it's really fast and easy to use. Basically what it'll do is it'll check against a list of common mistakes to make sure you are not making those mistakes. So here are some of those mistakes. Let's speedrun them really fast. This skill is going to connect your eval infrastructure, do some diagnostic checks, but what it's going to do is going to make sure that you've done error analysis. Which is the skill we just covered before with the error discovery. It's going to make sure that your failure categories are ones that you observe, so that you have annotations on your data, that any evals you write are based on data and not brainstormed problems that might happen. And it'll check to make sure that your evals are binary, pass/fail, and not things like Likert scales or letter grades. You often don't want letter grades or Likert scales because those tend to be very unwieldy and often add a lot of noise. It also makes sure your LLM judges are not overscoped. You're using code-based checks when you can and you're not over-relying on LLM judge. It also checks for the case where you're abusing similarity metrics. A lot of people try to abuse similarity metrics for evals and similarity metrics are rarely ever useful. For this purpose, the eval audit will also make sure that you're validating your LLM as a judge. In other words, make sure that your LLM judge is good. How do you test if your LLM judge is good? You have to check it against human labels, and that's what this part of the audit will do. Even if you are checking against human labels, it will make sure that you are splitting up your data properly and that you have good data hygiene so you're not overfitting. and you're not peeking at the human labels and allowing the judge to cheat in your evaluation of the judge. This eval audit will also make sure that you're using good review processes when you're annotating data. Next, it will check to make sure you have enough label data for every failure mode. What we have is we target at least 100 traces for every failure mode you have. This is just a suggestion, but it will look at that And if you don't have enough, it will try to suggest ways that you can acquire those. Finally, it'll look at your overall eval pipeline hygiene. It will make sure that you're running error analysis on a consistent basis. It will also make sure you are maintaining your evaluators properly. And then finally, it will generate a report for you to review. The eval start skill and the eval audit skill, those are the 2 skills I would spend most of my time on. If you get through just those 2 skills and you apply those 2 skills, you'll be far ahead of most people on evals. Let's quickly touch on the other skills. So we already covered the error discovery skill and the audit skill as well as the start skill. The generate synthetic data skill is useful when you can't acquire any data. Let's say you don't have any users, you are pre-product, and you want to stress test your product. You might want to generate synthetic data. There is a right and wrong way to generate synthetic data. This skill will help with that and maximize the diversity of that synthetic data so that you are pushing the boundaries of your application in the right way. Then there's a skill that will help you write an LLM as a judge prompt and audit those prompts to make sure they follow best practices based upon what we've found working with many companies on this topic. Next, there's this validate evaluator skill. That makes sure that any LLM as a judge you have is vetted and is checked against human labels in the right way so that you can trust it. There's also this evaluate RAG skill, which will help you build evaluation metrics around search and retrieval. And finally, there's this build review interface. The build review interface is actually leveraged by this error discovery skill because in error discovery, you will have to review examples. So this is kind of a skill used by other skills. There's a little bit more about how this skill works further down in the README. If you want to know the general process of how this all fits together, you can see this diagram or read this paragraph. If you want to see how a real team is using these skills in production, I recently talked with one of our former students, Lucas, about how his team uses them with Cloud Code in production. You can watch that conversation next.