In this conversational video, Robert Ta and Speaker B discuss why consistently achieving a 100% pass rate on AI evaluation suites is a warning sign rather than a success metric. They explain that such perfect scores indicate evaluation saturation—meaning the tests no longer provide useful signal for improvement and merely confirm that the model has already mastered the trivial challenges. The dialogue introduces the concept of hill‑climbing, urging teams to design increasingly difficult, meaningful evals that surface failure modes. To illustrate, they use a recipe‑chatbot analogy, describing how an eval should catch issues like ignoring vegan preferences or giving unsafe diabetic advice, especially for commercial health‑focused products. The speakers warn against the cultural pitfall of self‑congratulation (the “monkey in a suit” metaphor), emphasizing that teams need evals that reveal weaknesses, not just reassure them of competence.
Chapters
When the dashboard shows 100% pass rate on your evals, this is a monkey in a suit and tie.
Yeah. So like, look, if you have a bunch of evals and they're all 100%, it's just like 100% across the board, no mistakes, nothing. You should feel really bad about that. Like, cause it's not telling you anything. It just keeps saying 100% all the time. Even if you reach 100% across the board one time, just means your evals are weak. You should try to construct a situation where you're not saturating your evals. You're giving yourself challenging problems in which to hill climb against.
Say more on saturation and hill climbing.
Yeah, saturation is just like, hey, you've exhausted the eval. It's not giving you any more signal. You've totally, you know, conquered that thing. It's not providing you with a challenge. It's not guiding you towards improvement because you already conquered it. And so it's not a good— it's unlike unit tests. unit test that you have in software, you want it to pass 100% of the time. Eval are meant to challenge you. It's a goal that you're trying to move towards, and so you shouldn't sandbag it.
So gimme an example of saturation and it being a challenge to the agent building work for your, your courses. Uh, analogy is always this recipe chatbot, right? So tell me more about recipe chatbot as an analogy. And then grounding your message here in that.
Yeah. So if you have like a recipe chatbot that's like help you with creating a recipe for your daily cooking needs, okay. Um, you might have different evals, like not adhering to vegan preferences or giving people diabetic guidance when it shouldn't. You want to stay away from that. Something like that. Like you have all these things that might be going wrong. You don't want to just have 100% pass rate on all of those. You want to have, try to think of some more like challenging evals. Now, if your product is super simple, it's fine. But if you have like some commercial product,
like a health,
yeah, then, you know, if you're truly challenging yourself, you should not have 100% evals. And yeah, like if your product is trivial, okay, fine. But then like, who cares anyway? Like. You just like, maybe you shouldn't have evals. Maybe you don't need that many evals or just, you can kind of be loose about it. So that's all it means.
I see. Okay. So if you're reading 100% pass rates on your evals week over week as an AI builder team, you're probably this monkey in this team at this, with this.
Definitely.
And this, it kind of reminds me of this saying of just the idea of smelling your own farts, you know, like you're just into that. Like, hey dude, like we just want to know we're doing a good job. It's like, no, you're not. Like, we want to know where we're doing a bad job and we could do better.
