Why You Shouldn't Write 500 Evals First

22 Oct 2026 · 1 min

Watch on YouTube

0:53

In this segment, Hamel Husain explains why writing a large number of hypothetical evaluation tests (e.g., 500) for an ML product is inefficient. He emphasizes that evaluations are costly to maintain and validate, so teams should prioritize building and maintaining evals that target real, observed user failures rather than speculative scenarios. While keeping a few hypothetical use‑cases in mind is acceptable, Husain advises that they should only be turned into formal evals after the team can reproduce the situation—such as confirming a competitor’s feature is being mimicked—so the test is grounded in actual product behavior. This approach maximizes immediate return on investment and keeps the evaluation suite focused on genuine product issues.

But you can brainstorm 500 things like that, and you can sit there and write evals for all of them. And you should not do that because number one, evals are expensive. You have to maintain them, you have to validate them, et cetera. And then number two, like, because they're expensive, you need to prioritize your time. And it's best to prioritize it towards failures that are actually happening in your application. So you get more immediate ROI instead of just trying to like all these hypothetical use cases.
Interesting.
It's okay to like have some hypothetical use cases in mind. If you do have hypothetical use cases in mind, you should try to actually reproduce it. If you're afraid of it mentioning a competitor product, you got to try to make sure your product is doing that and then use that as an eval. But generally speaking, you want to ground it first in like actual user errors.