Hamel Husain argues against writing a rubric before annotating data, saying it causes criteria to drift away from what your actual data needs. Start with open-ended error analysis instead: note problems, group similar ones, count frequencies. Then build a rubric that matches what you found.
Do I need a rubric or reference answer before I start annotating data? And the answer is no, you do not. Actually, having a rubric or reference answer too early can hurt you. A reference answer is just an example of a correct response. A rubric is just a set of criteria that you use for judging a response. It's often better to wait until after you look at some data before you specify a rubric or have reference examples. And that phenomenon is called criteria drift. That you have a support agent that handles refunds and escalates those refunds to a human per your policy. But when you look at those traces, you realize this process of escalation is really clunky and really frustrating for the user. So what you need to do is do error analysis. This involves writing open-ended notes about what looks wrong in those traces. And then what you want to do is group similar notes together and count what is wrong so you can see what the patterns are. After you do error analysis, you'll be able to write a much better rubric and provide better examples If you want to learn more about evals, here's a link to a post with all of these questions and their answers that you can go through.
