Jonathan and Robert explain how to build autonomous AI agents using a stacked harness architecture with an LLM judge that verifies quality without human oversight. They collect trace data, annotate samples, and use a feature called "the walk" to surface ambiguous inputs and turn annotation into part of the workflow instead of a separate task.
Jonathan, I know you've been working hard on the Clarity Harness server-side stuff and Clarity API and evolving that. I want to give you some space to talk about that. Actually, one thing I throw out there as a flavor or lens for you to respond to is Hacker News, Y Combinator, all like OpenAI's thought leadership. Like they're all talking about harnesses being the moat. A few months ago we were talking about harnesses being the moat. I think more and more people are waking up to that. We obviously have an opinionated take with the way we think about harnesses and subjective expertise and annotation, like the data science side, meaning the philosophy of subjective experience. I want to hear more about your takes on that and how the server-side stuff is shaping up to meet the moment of what's going on with the AI world today. Any thoughts?
Yeah, I mean, I guess we've had our own opinion about like how harnesses could really benefit for the use cases that we're talking about. And we started this off with a stacked harness architecture. And this is a way that we can really kind of like isolate an individual agent in a harness and like use it to incubate the agent to go from something that has no training to something that can have an LLM judge, right? And I think the LLM judge is really a key concept to center on in these harness activities. And it's basically saying, have we trained our, our agent and our prompt well enough that we can, without human oversight, assert that it is acting well and that it's trained to like generate our stories, it's trained to generate our standups, it's trained to generate like whatever artifacts we need to in a way that's trusted. And in order to get there, we've had this process using the VALS for some time where we try to, to Get about 150 samples of traces, annotate those traces, and then split them into datasets for training and for dev and for test— for testing. If you can use some percentage, say like, you know, 15 to 20% of your datasets as like a training set, and then use it to test the LLM, use another set to test the LLM judge, you get to a point where you've incubated through your curation of expertise an agent that's going to work well and can be verified by a judge. And that's where the self-driving organization sort of takes off, right? Is that we want to create these closed loops with LLM judges that tell us like, does This is good and this is working. That's the whole idea between the stacked harness architecture. And so then it's a question of like, how do we, how do we like just build each harness in a way that we can incubate these agents? And there's a user experience there. And I think what you're showing here is the web interface for the harness. And so this is a way to say like, these are all the agents that I have in my harness. I can center in on each one. I can see the dataset that I'm building for it. And one of the things I've been focused on is just how to make it easier to like sort of extract domain knowledge from me. And That's, uh, some of the interface experiments I've been working on are like the walk. And this is where the harness itself can say like, well, in this process of the incubation, here's the most ambiguous items that are like, we need to learn. And let me point your attention, Mr. Expert User, towards the input that I need to train this agent and to get towards an LLM judge the most quickly. That's what I've been doing is I'll walk the artifacts. You know, like Claude Code using our harness is actually saying like, okay, look over here and tell me what you think. And it has asked me a specific question to say, now look over here on the artifact and tell me what you think. Answers a specific question. And then I'll submit like, you know, like say 5 to 7 different inputs, you know, as I review an artifact, you know, it doesn't feel like this monotonous sort of like annotation activity. It doesn't feel like just pure data labeling. It feels like I'm being pulled into the learning process along with the harness. And I think that's something we can just make this part of our day-to-day work where there's not a distinction between labeling data and getting work done. It's just, we're doing both all the time. Right. So like, you know, I think that's where we're trending towards.
That's the new norm of work. And I love what you say about, it's really about incubation of the agent and improving the expert review loops in the flow of work. I kind of think of a KPI as annotations over time, annotations from run date. You know, the further away you get from a run, it's more stale, the context is lost from your head, right? So like ideally we minimize the expert review time to be within the flow of work. So like when they're reviewing as well, the context that they're reviewing with and the lens they're reviewing with Is less biased, is fresher, it, it's closer to, to truth. And so there's an important thing there, which is you're incubating these agents towards the overall goal of these things are autonomous and they're just shipping for you in whatever part of the business, marketing, sales, product, whatever. And whatever the output to meet those workflows are, like whether it is a prospecting, like win-loss analysis, or it is an artifact like a story or an artifact like a prototype or an artifact like a pull request in the areas of tech teams, enterprise software. Jonathan's just saying Like, we want that to always just get better and better. And so the goal of any organization is minimize the expert review time loops, maximize how good they are, and then be able to incubate agent to get to high-quality artifact. And then it's on autopilot now. Great. We don't have to worry about it. And technically what we're saying is, great. Okay. So we've had enough human training or annotations by the experts to where the LLM judge is just, it's got it from here.
