A 100% eval pass rate is a bad sign
20 Oct 2026 · 1 min
0:53
In this brief dialogue, Robert Ta and Speaker B discuss why achieving a 100% pass rate across a suite of evaluations is a warning sign rather than a success. They explain that such perfect scores indicate that the evaluations have become saturated—too easy or insufficiently discriminative—so they no longer provide useful feedback. The speakers argue that when evaluations are saturated, they fail to reveal model weaknesses, making it hard to gauge true performance. To avoid this, they recommend designing more challenging problems that force models to ‘hill‑climb,’ i.e., incrementally improve against increasingly difficult tasks, thereby keeping the evaluation signal informative.
