Using Claude Code with Eval Tools

Coding agents can now write code, run experiments, and debug software. So can they do your evals for you? In this session, we put that question to the test live. We give Claude Code the full eval lifecycle, pulling traces, error analysis, and designing experiments. Some of it works surprisingly well. Some of it doesn't. You'll see exactly where automation helps and where human judgment still matters. Mikio is the lead developer of my favorite open source eval tool. It's called Phoenix. Phoenix or AIs, you can Google it. It's great. It's really easy to use, self-host, has really good APIs, really good documentation. And as coding has become a lot easier with models or with coding agents, especially since November, December with the new Opus 4.5 and onwards. Yeah, that is a new territory. I don't think you can just hand over your entire application to an LLM and just say, eval it. But I think there's a lot of things that you got to instrument your code, you got to get it into Phoenix. And there's like a lot of little things around even looking at the data, filtering it, navigating it that you might want to think about. And then since Phoenix has all these APIs, I think the way that Mikio has engineered it is that it's very agent friendly. Like you can connect it with your agent. And so I saw that Mikio posted that Phoenix now has support. He's been working on sort of making it really good with agents. And so I think a lot of the stuff that was more manual before can now be automated. And so I'm really excited to talk about and show how to use coding agents with Phoenix. And I'm really excited to see what Mikio shows us today.

You know, that I think was then weighing on my mind and probably weighing on Hamel Husain's mind and a bunch of your minds too, hopefully. As Hamel alluded to, coding agents are really in production now. Specifically, last December is the big turning point with Opus coming out. But, and even hardened practitioners like Andras Karpathy are saying that before December of 2025, they were, you know, coding maybe 20% quote unquote vibe coding and 80% using tab complete and things like that. But that completely inverted, right? He went completely to 80% vibe coding and 20% manual coding. And I think the conclusion that he came to during that was spot on, which is that right now we're at this interesting point where inference— sorry, the intelligence of these coding agents is a little bit ahead of the infrastructure. And today I want to explore that a little bit and what it might look like to have self-evolving software. I think the left-hand side, we all know if you've been, if you're coding in the 20, 2010s, if you're familiar with deploying an application, maybe you have Log Monitoring or Grafana dashboards set up, and then you as a human dev, something goes wrong, you get paged, you, you look at that glass pane and you look at the dashboards and you come up with a hypothesis of what to change. Then you make a change in the IDE, write some unit tests, and deploy that. And we're in this middle phase, right? Where you deploy an application, you probably do have an observability platform, maybe, hopefully, Arise Phoenix, and then you, as a human dev, look at that screen and figure out what might be going wrong, and then maybe now use Cursor or Clogcode to come up with a solution, deploy it, and then hopefully you can see the changes in the observability platform. And I think we're still going to be in this phase for some time, but let's put on our, like, futurist hats on for a second and think about this third phase, which is what if coding agents do all the coding, right? Then you're still going to have an application and you still need an observability platform because really the agent needs to see the same things that the human was seeing previously, but instead of using a glass pane with dashboards and heatmaps and things like that, it needs things that are, you know, efficient for it to pull things. And I think an agent's eyes are very different than a human's eyes, right? It is really good at parsing out structured data, putting them into files, using grep, using search tools, things of that nature. And so we really need to let coding agents gain access to observability platforms so it can iterate and validate what it's doing. And so it can deploy. The application. Now that like the dashboards and stuff still important because human oversight is obviously really important, but it doesn't become the primary use of something like an observability platform. I think there's so much of like just human way of thinking and stuff that has to shift. I think the hard thing is like each individual has like this tribal how you work. And we all need to change. There's certain arguments about maybe previous, the focus was more important, but now parallelization is important. I think obviously the human role in all of this becomes we shift more into the validation and reviewing and ultimately the harness building, the instructions that these agents will, because the more you, I think the OpenAI paper also particularly talks about this is really about constraining these agents. So we're, I think this new, what they call agentic engineering groups are going to start forming to make coding agents have a little bit more autonomy. I don't know if that like fully answers your question, but yeah, I think so.

You're talking about the harness engineering. Paper, I think, right? Or is there something else?

Yeah.

Okay.

Yeah. I think the, so that, that picture is just talking about like how much vibe coding is out there. But I think what we have to acknowledge also that there's actually two things shifting at once. One is that the way in which code is being shifted, shift is changing, but the other thing that's changing is the types of applications, right? So like on the far left, the reason why, you know, a lot of that stuff worked the way it did is because to some extent, there was a level of understanding just through the code. And this is why coding agents are so good, is if it has visibility into the code, it knows how that code executes, right? Through types, through if statements, through while loops, all those kinds of things. It can create like this state machine and get, get an understanding of how your code executes. So previously, if it was just code, the code documented the execution. But the thing is now, like, we're not just deploying one type of code, right? We're deploying code, we're deploying natural language prompts, and it also is reliant on the weights. And the way that the code looks is not necessarily how the agent operates, right? The agent might loop. The agent, depending on which model it's using, might loop 3 times versus 4 times, all these kinds of things. If you think about it, to some extent, agent behavior is more documented by the telemetry and the traces than it is with code. So if agents are going to be coding, agents are basically going to have to write code. To steer agents in one direction or another, they actually have to have access to the traces. Just to reiterate what a coding agent is, like a coding agent is not just made up of, it's not just Opus 4 or 5, right? It's all the harness that surrounds Claude code. That's things like skills, which we'll talk about today. Maybe you have an MCP server connecting to documentation. Maybe you have agent browsers set up, tools, maybe it has to have access to things like the Phoenix CLI. You know, if you're using Cursor, it has access to the editor and the LSPs, the prompts that are inside of the harness, and its own memory management system. So in order for coding agents to really thrive in your codebase, you need to tune this harness for your particular use case. And the two things we're gonna talk about particularly are skills and tools. Okay. So this is what we're gonna do today is like the idea of a coding agent having access to its own telemetry. And one of the cool things about Phoenix is you can basically give it a Phoenix for a coding that you want. You can have one per workspace, but we're going to have an agentic application. It's going to be streaming its traces, its evaluations, its experimentation, its feedback, all into a local Arise Phoenix instance. And then we're going to create a CLI that's basically connected to that Arise Phoenix, and it's going to be able to pull traces down, log them to files so that you can use things like grep and jq. And then the coding agent will have like eyes into what it did or what is happening. And be able to come up with proposed changes, make changes to the code. That telemetry will then just continuously flow back into that observability stack, and the coding agent will be able to make adjustments. And this is exactly what the harness engineering that is being done inside of OpenAI as well. So today we're going to try to make a coding agent take the AI engineer loop. We're going to go and tell the agent to look at its data, Open codes, a taxonomy, and then we're going to skip the annotate the data just for time's sake, but we're going to get it to, based off of those codings, hypothesize why the output is bad. We're then, we're also going to take like its hypothesis and then let it plan out an experiment and design out like the evaluation criteria that's going to make it measure its outcomes. We're going to force it to measure its outcomes, and then we're going to force it to apply its changes and explain its reasoning. So that's what we're going to do today. Let's jump right into it. So here I have an agent on the right-hand side and I have the Clog code on the side and they're wired together like I just showed you. This is an agent that's connected to mTCP with Phoenix. So let's do the first step of the AI engineering loop just for sake. Of trying to make this coding agent take my job. Okay. So I'm going to clear the context just to prove something. And then I'm going to explicitly pull the, the Phoenix eval skill just to force it. This is an interesting topic we can delve into, but I'm going to paste this and I'm going to force it to do Axial coding to try to understand what's happening currently in the traces that are already in my Phoenix. And I did seed it with a bunch of questions.

Okay. You're going to do open coding and axial coding with the agent?

Yeah. So the way that it works, and I'm going to let this rip, you can see that it's pulling down the traces with the Phoenix CLI. So it's pulling down that data and then it's logging it to files and then it's going through each one and doing the open codes and it's going to go through that open code step. And the reason why, like, I had to prompt it very little is because I actually mounted the, the skill there and happy to just show what that skill looks like.

So it's worth, it's worth lingering on this point for a bit, I think.

Yeah.

Do you think this is a good idea to just have an LLM do your open code Axio code for you without looking at it yourself? Like what's— I know like this demo is for a specific purpose, but we should get nuance. Okay.

Yeah. So I think while this is ripping, we should probably do the open coding ourselves and see what the difference is. Right. Because I think you're right. Like you're the tastemaker and a lot of the open coding and stuff is about you looking at the data. And I think what you're going to notice, and it will be an interesting nuance here is that there's a difference between a human doing the open coding and the astrocoding than what an LLM does. That doesn't mean, I think, if you put a lot of your human tastemaking into skills, it couldn't do a lot of interesting things. Whether or not you fully trust that process or not, I think is obviously up to, 'cause like, how much are you letting it rip? I'm not here to say, I think it really maybe depends. I'll maybe show some examples and they'll be more clear and we'll see some examples of that. I think it's agents give you the ability to get like a second opinion about stuff too. And so a lot of times that can be good. So maybe you should like not look at what it's doing and then you let it rip and you also look at what it does and it might actually discover things that you hadn't thought about because I think, you know, if I did open coding and then you did open coding and axial coding, we might come up with a different coding strategy. And what you'll find is that definitely the agent comes up with a different coding strategy. Than I do. To, to your, I think the interesting debate now really is going to be, is the agent going to have taste? And to some extent, it's not there yet. But if you look at what OpenAI is doing, you can see within their Harness, they basically have created that taste-making inside of here about how they think design should work, how they think the front end should work, how how it should think about user onboarding and things like that. So they've basically encoded that in the codebase to let the agent rip. And I think the thing that you're going to see here is that this codebase is not mature, right? This codebase is one that wasn't designed with this harness in mind. And so it's going to do a lot of things wrong. And I think OpenAI also claimed to do that. That happened, right? The, basically the getting started is really tough because the agent's going to go off and come up, come with wrong conclusions. I'm doing this more as a strawman exercise to see like how much of this can be possible, not necessarily like advertising that like this is what you should do. That it went in and said there's a lot of things that are off topic in the 2 out of the 15 traces, which is 13%. So it came up with a bunch of coding and it's citing the different traces of what it's doing here. So I'm going to prompt it one more time to just come up with a hypothesis and we'll see how well it does. So, I mean, okay, let's do, let's just, we can also do the same exercise, right? We can come in here, we can look at each one of these and do the open codes, right? How do I handle file uploads in Phoenix LiveView? Okay. So it's talking about, it's a little bit of a fun exercise. It created its taxonomy in its own memory system right now, but you could definitely have it sync it back. Those are all the kinds of things that, that you could consider. Do, how much do you want agent-generated notes in your telemetry? And that's obviously— so let's clear the context here. Just let it not bias itself to its previous actual coding. And let's also bring in the Phoenix eval skill again. So we'll still look at the system prompt. So it's saying Phoenix evals the system prompt lacks grounded context about the topic Phoenix, Elixir Phoenix at Chessedai, you, the scientific experiment to validate the root cause step 1, or state a single false viable, like basically grounding it and giving it the process it should go by to build out this experiment and validate validate it, results, and loop on, on that a few runs to validate it and for it to take a, a prove me right or prove me wrong through Eval's approach. And we're gonna let it rip as it does that. So an experiment is similar to the scientific method experiment. So you have your, you know, your controlled variables. In this case, I'm saying the controlled variables, keep the model the same, but change the system prompt of the agent. To, to see if we can steer it to have a better, better grounding in, in the topic that it's supposed to be talking about. We're treating the system prompt as the uncontrolled variable. Then we basically can run evals on the agent before the system prompt has changed and then evals after the system prompt has changed so that we can basically prove that that change in the system prompt had like a marked improvement in the agent behavior itself. And it's using some of the datasets that are already there in Phoenix. I have the evaluate the evaluators here. I have some dataset fixtures. I have the evaluators, which are like, I have this eval called terminal safe because like we're outputting to the command line. I want it to use ANSI characters rather than Markdown. So I have these evals and stuff, and then I have these experiments, which is, does it conform to the terminal format that I want? And we're hoping that because I've You know, created enough of an eval harness in here. The agent can discover the best practices that I've put in here for it to create an experiment. It's going to come up with an experiment to say, because basically we said the system prompt is infecting how it's not grounded. So it needs to come up with the dataset that it's going to run that over. So the examples and the splits that it needs to run on, and then it needs to create the right evaluation criteria. So we put that a little bit into the prompt itself, but it's also in there. Which is after each run record. So I said dataset, create defined fixed set of Phoenix evaluators measuring your success criteria. So it's actually just created an evaluator. So again, not advertising, but this is just, I'm just giving you a peek into what could be coming down the line, right? Is coding agents going to write evals? We'll see how well it does, obviously. So you can see that it's created an eval, right? So let's just look at it. I mean, it should be here soon already. So I can look at it right now. So it's created this new evaluator. It's called topic handling. It's using Haiku. You are evaluating whether an AI assistant correctly handled the user query. The assistant is Arise Phoenix Assistant, an AI observability platform. Handled correctly. Query is Arise Phoenix AI observability. Handled incorrectly. And so it's giving it some few-shot examples of what it thinks is good and what is not bad. Obviously. We shouldn't. Building evaluators like this requires benchmarking, which we can do also if we have some time. So we can look at the experiments that it's formed. How about that? Let's, let's look at the experiments. So it's run this, it's created this dataset called topic hypothesis experiment v1. And then you can see here that it's created an eval called topic handling, and it thought it handled the topic at 0.93. And then it's run an experiment 3 times to try to change the system prompt. And so in this experiment, maybe we can compare the two. Let's compare just two. So it's terminal app, but it's a little bit hard to see. Sometimes it constructs experiments in different ways. Sometimes it does it in the experiment itself. Sometimes it makes the changes. So I think depending on how you prompt it, it will either make changes and then revert it and do it like manually, or it will do it within the experiment itself. It might be in the experiment itself. Again, this is what happens when you let it rip. So yeah, so it has basically put the prompt inside the experiments and it's run two different set of instructions in here. So unfortunately it didn't change the agent itself. It just changed the prompt inside of the experiment itself. And that's just one of those things of like coding agents figure out a way. Let's let it rip on one last thing and then we can just talk over it about all the interesting things we just saw. So I'm going to let it rip one more time. I'm just going to tell it, like, evaluate the evaluator, basically. So I'm going to say benchmark the new evaluators you created and show the confusion matrix of its accuracy. Okay. So we're just going to try to force it to do some best practices about creating a resource dataset for the evaluator and for it to tune it a little bit before we, we accept it. Yeah, to your point, I think I definitely recommend everybody read the OpenAI, we're not admitting is like, we think that there's these repository of skills that are just going to magically make these agents do things, I think it's very different. If you think about like the way that the Claude Code team, for example, keeps track of their ClaudeMD and the skills and the hooks and all that stuff, all these kinds of, for lack of a better word, like restrictions to keep the coding agent on track. That's an iterative process. And so a lot of the, even Claude Coding supports this now where you can say, looking back at your current session, how would you adjust your skills? How would you change your instruction set to to better do not make the same mistakes you made during this session. And they commit that back. And then what the Claude code team claims to do is they take that prompt and then they put it through the optimizers that they have inside. I think it is available in Anthropic desktop app, the optimization, but they basically periodically re-normalize the skills, but they, the skills is a self-evolving thing. Every time the agent makes a mistake, you basically say, oh, You did something wrong, put that in your, your system instructions in your agent MD or in your skills and make sure that you don't make that mistake again. So I think that's the, the harness, if you will, is like the self-evolving. The determinism aspect, I think, is another thing that is talked about a lot in OpenAI paper. And obviously if you use Claude, there's hooks, right? So there's hooks that basically enforce very specific things. So they're like pre-commit hooks, like stuff like forcing the agent to follow very specific patterns with determinism, not just through prompts. And the skills spec, you know, that everybody's converging on also has code snippets they can run as part of the skill. How likely is like an agent be able to one-shot a solution? And that I think is both good skills in the skills, but also best practices in your code because I don't know if this is your experience, but once the code base is mature, agents actually do a lot better or a lot worse depending if your code base has a lot of bad practices in it, it's probably just gonna copy those existing patterns and start replicating them over and over because that's what it sees in the files and it thinks that that files is the way that it should be. So you have to mature both of them at the same time because otherwise the agent's gonna start just replicating bad practices and taking the shortest path to victory. And so it's actually like the URL is incorrect, for example. And I can tell that as a person that's like basically trying to build this agent. That the links that it's generating, it's directly taking the link from Midjourney and just regurgitating those links. But those, that, that those links actually need to be translated into something that is satisfactory for me. So that's like an example of there's absolutely no way an agent without me telling it that is a problem. If you encode best practices, they'll try to do some interesting things like produce a confusion matrix and try to make these evaluators do things. So at least like it will scaffold the benchmarking for you. And then you can go back and add examples in which maybe, or maybe swap out the dataset, but it did create, I think, a useful evaluator as part of this practice. Like at least now I know that I need to at least put something about that there's a huge name collision with the word Phoenix. Like Phoenix is a very generic word, right? If I'm me personally, I'm at the mercy of the Phoenix, Arizona, there's Phoenix Elixir, there's Phoenix, Phoenix LiveView, there's the Phoenix, the old email client, right? And so I need to at least put that. If you really wanted to let it rip, you would say make these changes. I think this time around it didn't produce something as interesting as I was hoping. I've had, honestly, it produced some really interesting results on the same, on the same codebase. So I think that's another lesson learned for me too, is like these harnesses. Sometimes it'll look at the skills, sometimes it won't. It's really hard for me to determine when and why it will do things. I'll give you an example, like last night that I was like, oh my gosh, it did something really amazing, which is it actually started looking at the number of repetitive tool calls that was being made for MCP and forcing actually for the system prompt to reflect on the topic and optimize the turns of the number of times it should look at tools. And it actually created some pretty novel evaluators for me that actually I thought were quite interesting and it actually let it rip for much longer and it came to a really pretty insightful— because what you see right now, like one of the problems here is, okay, even though this is about Phoenix LiveView, it's hitting my MCP server 3 times. Like why would it even pull in context? From my Phoenix MCP server to get— do perform RAG on Phoenix documentation when at the onset it already should know that this is basically off topic and it shouldn't have to make the tool calls. So that was an example of it really understood the topology from the traces and was able to identify something that I could have come up with that conclusion, but I just let an agent rip and it figured out that I should be optimizing the number of tool calls specifically when it was off topic. Let's say these coding agents are making a lot of changes on your system. They need to basically provide evidence that they have made things better or worse, right? And now the burden on me is to review this stuff. And if I have the agent connected to, to come on my eval harness, as well as my observability platform, this is basically providing at least to some degree, the evidence that it's improved the system. Now we can obviously question the validity of the data and the validity of the eval itself. Those are all things that are can be, should be scrutinized, but those can be scrutinized obviously during the pull request. What the agent has currently basically said that it's done is it took the, you know, it ran it with the first set of prompt. It was currently, it was handling whether or not it was on topic correctly at a 0.93 rate. And then it's claiming that it accomplished like a perfect score on keeping the CLI tool on topic. There's a lot of things that I sort of contrived here, right? We're talking about 15 examples. We also made it do axial coding on 50, 15 examples. So that's another prop issue too, is like I didn't really have time to let it rip on much larger data, which maybe it would've figured out a lot more interesting problems, um, looking at more data. The skills are available. So skills are, the distribution of skills is super early right now. They can be distributed in a lot of ways. If you want to just try out these skills, you can do npx add-skill from our repository. And then basically what this does is it pulls our skills that we have been honing over the last month. So the one you saw today is of teaching the agent how to pull down traces, introspect experiments, stuff like that is Phoenix CLI. That one I definitely recommend it. That one's like non— pretty much non-controversial, right? That's just a treat. For example, the Phoenix CLI is not in the pre-training of Opus 4.6. So you might need this one to basically let Solid Code be able to understand how to use the CLI. Phoenix evals is like a combination. So these are different composable pieces that we're hoping that you use them or don't, maybe you don't, maybe you hard fork them, maybe, you know, what you do with them is obviously up to you. The, all of these are basically registered. Currently Vercel's distributing some of these through skill.sh. Some people put them on their website in the well-known domain. So there's a lot of different ways that they're currently being distributed. Word of caution, there's a lot of bad skills out there. So just because there's a lot of downloads of a skill doesn't mean it's better. And I think the other thing about skills, be careful of the skills that like loop and go search the internet for more skills, cuz there's def— just because you can read the skill in Markdown doesn't mean that there isn't malicious HTML in there telling you to do bad things. So there's definitely prompt injection in there. I mean, the great thing about this is maybe you were previously— you should always be reviewing the prompts. And the good thing about the prompts is that they're really things that you can diff. There are things that you can hook into CI pipelines. You can run a regression suite using the experimentation. So basically on a CI step, you can validate that these new system instructions that it just committed, right, are in fact better than the previous ones. So this is where like the continuous eval process comes into play, right? Because you now need to— we only ran it on 15 examples. It's pretty important to run this prompt through a lot more examples and make sure that it's hardened against the other criteria that you might have, like properly outputting to the terminal or properly resolving links from your MCP server, whatever those other evaluation criteria is, it needs to prove itself as well. Even the console logs are being logged as Markdown so LLMs can interpret that stuff. There's just so much happening right now to give coding agents more access to this stuff. I think really it, the call to action right now is like getting your codebases ready and have the right opinions so that, you know, when people come on board with coding agents that you really can scale out your productivity using coding agents when appropriate. Obviously when appropriate is like the big question right now, but there are definitely areas where things are appropriate, right? If I need to make a small UI tweak. And I have best practices of how I, my, my UI should look like, yeah, let coding agent rip. If I'm trying to change the system prompt of a very critical agentic app, should I let my coding agent go rip on it? Probably not, but I should have guardrails through evals that if it does make a change, for whatever reason, it knows that it did something wrong. So those are the kinds of signals that we need to connect coding agents to more of these signals. And the infrastructure needs to evolve, but you need to connect your infrastructure to your coding agents too. There's the Andras Karpathy prediction that 2026, the tail end will be the slop-cop-lix. I don't know what the, how to say it, but like the slop, like there's going to be a lot of slop that we're going to have to reconcile with. But then there's also like the OpenClaw stuff, like has massive security problems, but he probably just went and got hired by OpenAI for, you know, $100 million and produced an amazing amount of value. Yeah, it has a lot of security problems, but he created a very interesting product of a self-coding agent that provides a huge amount of value. I don't want to maybe say don't— obviously I think our role as coding is going to diminish. You better get on board to some extent just because you're going to be outcompeted by the companies that are doing that. But the level of tolerance of what's good and bad, the tastemaking still is like in the engineer's jobs, the PM's jobs. The MLEs, right? That still is like hugely important. Let's just be honest, like these coding agents have no taste. These coding agents don't know what best practices are. There's a lot of things that they don't know. And skills, I think, is the first step to that. But let's be honest, codes, like skills don't fire. What is the Vercel benchmark? It's like they fire 52% of the time right now. So they're unreliable, right?

I'm always naming my skills. Like I even tell it to use the agent browser for testing, like manually to make sure it fires. Yeah, I agree with that. And I agree with— I think you should definitely be preparing, like, to your earlier point, like, you should prepare your code base. If you want to do evals and measure things, you should be distilling your understanding of evals and how they fit into your domain. What you're showing with the OpenAI people, they have, like, lots of documentation, like, this is what's happening in this code, this is how to navigate it.

Like I created this example over the weekend, the codebase that like OpenAI, that took 1,500 commits and a couple million lines of code to get that self-evolving harness into a shape where they are letting it rip by itself. Right. So there's no way that I can create it, but they're proving that basically you can get to a certain level of autonomy, but you basically read that thing and they're not trusting that to go make changes with the OpenAI website, they're doing that for an internal tool, right? They're still also being careful about level of autonomy. That's the TL;DR for sure there. All right.

This has been very useful. Cool. I think people know where to install the skill, how to get the skill. You could visit the Phoenix website as well. I think this was really useful, but yeah, thank you so much for coming. Yeah.

Thanks for having me and thanks for forcing me to at least prove that Warship actually makes some semblance of sense and hopefully it gets better over time. So really appreciate your time, Hamel.

All right, thank you.

Cool. Thanks. See ya.