Why you shouldn't write 500 evals first

22 Oct 2026 · 1 min

Watch on YouTube

0:53

In this brief discussion, Speaker A warns against the practice of creating a large number of hypothetical evaluation tests (e.g., 500) for a product or application. The main arguments are that evaluations are costly to maintain and validate, so time should be allocated to the most valuable tests. Speaker A advises prioritizing evaluations that reproduce actual user‑facing failures, which deliver immediate return on investment, rather than speculative scenarios. While it’s acceptable to keep some hypothetical use cases in mind, they should be turned into concrete, reproducible tests—especially if they involve competitor behavior—so the evaluation remains grounded in real‑world issues.

But you can brainstorm 500 things like that, and you can sit there and write evals for all of them. And you should not do that because number one, evals are expensive. You have to maintain them, you have to validate them, et cetera. And then number two, like, because they're expensive, you need to prioritize your time. And it's best to prioritize it towards failures that are actually happening in your application. So you get more immediate ROI instead of just trying to like all these hypothetical use cases.
Interesting.
It's okay to like have some hypothetical use cases in mind. If you do have hypothetical use cases in mind, you should try to actually reproduce it. If you're afraid of it mentioning a competitor product, you got to try to make sure your product is doing that and then use that as an eval. But generally speaking, you want to ground it first in like actual user errors.