When analyzing errors, label the first failure that causes others downstream, not every mistake in the chain. Experienced annotators develop intuition for spotting truly independent failures and label only those, saving time and avoiding redundant work.
What does the first failure mean, or what does an upstream failure mean? We're using these terms interchangeably. And as I mentioned before, focus on the first failure because a lot of downstream failures that you label might be dependent on that first failure, so you can just, you know, save yourself time and energy in the labeling process. But once you become more experienced with error analysis, and we really saw this in the last cohort as well, in their second and third time around doing error analysis, people kind of developed an intuition for what failure modes were independent. So if they kind of saw 2 or 3 different failure modes in a trace that didn't have to do with each other, they would just label all of them an open-coded failure.
