Robert Ta explains evals as grading rubrics for LLM outputs, using Camel's open-source error discovery tool to automatically analyze failures, build annotation interfaces, and organize failure modes into eval datasets. Speed to annotation and speed to eval are the real competitive advantage.
Camel put out this really nice evals skill open source repo that has skills to guide coding agents to help you build product-specific or domain-specific AI evals. If you don't know what evals are, think about this analogy for grading a non-deterministic thing. Remember when you were in like, I don't know, English literature class, I don't know, philosophy or whatever, when you were a kid, like more subjective areas like arts and humanities? How does your teacher grade you? How do you grade like 40 to 50 students fairly in a subjective domain? You have to put out a grading rubric and you're like, this is for grammar, this is for prose, this is for if you have a bibliography with links that work that are well-cited, this is for XYZ, formatting, how well do you make the argument, how concise, how verbose. All these things are subjective evaluation criteria. And that's exactly what evals are, is how do we grade the output of the LLM in an unbounded space where it could say anything? And can you take that and put that in your pipeline so that when the models change, we can regression test against those evaluations where we bring in that subjective reasoning or that taste component to critically judge the quality of the AI product? That's what evals are at a high level. And if you're doing anything around agents and you just have a system prompt, some tools, some skills or whatever, which I've seen so many people have, you don't know it's good. Like you're running in production, you don't have traces or whatever. You don't know if it's good. Like how do you know it's objectively getting better over time measurably? And that's what Hamel teaches. He has some of these awesome eval skills. One of the key skills I thought was super cool was this error discovery skill. It runs an error analysis on an existing dataset and builds an annotation or review UI bespoke to that dataset, selects diverse samples from the dataset, monitors your annotations, and then organizes failure modes. So failure modes tie to how do I get a grading rubric, right? Just to explain this really quick, when your teacher looked at your thing and you didn't have a bibliography, Failure mode. When your teacher looked at your thing and you wrote an essay on Prometheus versus Frankenstein, which is what she asked for, failure mode. You wrote the essay on a completely wrong thing, right? So that's what we mean by failure mode. And that informs the rubric, which informs the eval, which evals are really the moat for any domain-specific agent or AI product or AI business against the Frontier Labs. That's my true belief. This is the essence of what it does. It brings up a dashboard for you to review a dataset and be able to get to that failure taxonomy as fast as possible. And I think of a KPI, it's speed to annotation, speed to evals. And going through this whole process, it's open coding, axial coding. You get to failure taxonomy across a dataset. You get to your V1 grading rubric and boom, you have your first evals, like dataset. You have to manage your dev, test, and golden dataset.
