Hamel Husain distinguishes between foundation model benchmarks like MMLU and SWE-Bench, which don't predict real product performance, and product-specific evaluations that actually matter. He compares foundation benchmarks to the SAT: they measure generic ability, not job performance in your specific use case.
What is the difference between foundation model evals and product-specific evals? And so when you hear the word evals, it's important to differentiate which one we're talking about. And so for foundation model evals, you may have seen benchmarks reported by OpenAI, Anthropic, things like that, things like SWE-Bench or MMLU or HumanEval. Now these benchmarks don't tend to correlate to your product-specific performance, so you don't want to necessarily use those benchmarks for your product. It doesn't make any sense. You need to have metrics specific to your product. So you can think of it as, uh, these foundation model evals are more like a standardized test, whereas product-specific evals are like, what is your job performance? There can be a very big drift between how you scored on the SAT and how good of an engineer or product manager you are.
