Why everyone’s confused about evals

You go meet a data labeling company they tell you our experts are writing evals and then you have folks saying that PMs should be writing evals. What are evals? Why is everyone saying evals? And these are actually different parts of the process and nobody's wrong but when a data labeling company is telling you that experts are writing evals they're actually referring to error analysis. Lawyers and doctors write evals. That doesn't mean they're building LLM judges. And when you say that a PM should be writing evals doesn't mean they have to write an LLM judge. That's good enough for production. If you make a bunch of practitioners sit together and ask them is it important to build actionable feedback loop for AI products, I think all of them will agree. Now how you do that really depends on your application itself. One statement that every ML practitioner will tell you is it really depends on the context. Don't be obsessed with prescriptions. They're going to change.