Short YouTube monologue (speaker Robert Ta) advising caution about "eval-driven development." The speaker argues there is no single correct way to run evaluations, warns against over-reliance on vendorized, push-button evaluation dashboards (e.g., "hallucination score, toxicity score"), and emphasizes that evals are a tool to produce actionable value rather than a checkbox to tick.
People have asked me, like, oh, should we do eval driven development? And I often say, no, don't do that. The answer is nuance. You have to have, like, a product. Product should work. It should be compelling. You don't want to get carried away in any one dimension, whether it's evals or really anything else. Evals is just, like, an important tool that you should know about. Also, there's not one right way to do evals, but there's a lot of wrong ways to do evals. So the wrong ways to do evals is try to find a vendor that will just do evals for you, and then you just push a button and say, my evals are done. It sounds good on paper. Hallucination score, toxicity score. You get score. And you can put on a dashboard and you're like, okay, I did my evals. Yeah, but that's not the point. The point is not to do your evals. If you do an eval, you get some immediate value out of it. Otherwise, you shouldn't do it.
