Hamel Husain explains why evaluation datasets go stale and need refreshing through error analysis. An eval that consistently passes no longer challenges your model. For tracking real progress, use product metrics like churn and revenue instead.
What do you do when your gold dataset becomes stale? Eval datasets naturally become stale over time. You have to refresh them. And the way to refresh them is to do error analysis periodically to get fresh data. Eval can be very expensive to maintain and build because they often involve running an LLM to judge the output of another one. And you want to make sure that every eval you have has high signal. So if you have evals that are passing all the time, That is a smell that, that eval is no longer as useful, and maybe you don't need to run it as often. One of the main purposes of evals is to give you a challenge you can hill climb against, and you should always have a new challenge as you get better. So it's a moving target by definition. If you want a stable baseline to compare yourself against, you should be using product metrics. Those are well suited to more longitudinal comparisons. Things like churn, revenue, monthly active users, so on and so forth. If you want to learn more about AI evals, check out this link. I've put all of these questions and answers in one place so you can learn everything quickly.
