Hamel Husain explains axial coding, a method to cluster error annotations into failure categories. Most errors fall into a few patterns: transfer problems, handoff errors, scheduling mishaps, incorrect information, formatting mistakes. Export your data to CSV, feed it to an LLM, and let it build a taxonomy of failure modes you can refine by hand.
what's called axial coding, which is you take your open codes and try to build a taxonomy of failure modes out of them. So you might think of this, if you want a visual diagram, every failure mode a Post-it note that you put on a whiteboard, and now you want to organize it into clusters and build a hierarchy of failure modes. There are all these different clusters of failure modes that came up. For example, transfer, we saw an instance of that, right? Transfer, handoff issues, scheduling issues, right? Maybe it didn't call the reschedule tool. It scheduled a new tour. Incorrect information provided. Formatting error. There might be a very long tail of failure modes. That's fine. But probably like some 80/20 rule, 80% of the failures are going to be explained by a small number of categories.
Actually on the slide, these are the actual results of the error analysis that we did in real life. This is where we knew where our real errors were, and then we could like prioritize those.
This is where I say use an LLM, totally. Export all of your annotations and your traces to a CSV, upload those to ChatGPT, and then ask it to like come up with a hierarchy of failure modes based on your open codes. Now it might miss some open codes, the hierarchy might not be perfectly accurate, but it's a really great starting point to then go and refine that from.
