A framework for analyzing LLM errors using open coding and axial coding, borrowed from qualitative research. Two rounds of each coding step takes a day or two but catches failure patterns that save weeks of debugging later. Teams naming things differently still find the same core problems.
What you should take away from error analysis: the whole point is about iteration. You do open coding, you iterate on open coding until you don't learn anything new. You do axial coding, iterate on axial codes until they don't change. But you stay at a step, one of these steps, until you are theoretically saturating. My tasks, I find that like 2 rounds of open and axial coding is probably enough. Feels like it can take a lot of time to do this, but I think the important thing to note is once you do this, your product is so much better and you won't have to open code and axial code with that much intention and energy ever again. You will only have to do it for a few samples of traces. You won't be surprised by anything new, right? So it is worth spending a day or two in this open and axial coding to really get your product to where it needs to be. But ultimately the failure modes are going to be the same, right? Like handoff tool. It might not say the row label is exactly handoff, We might call it tool handoff, we might call it rescheduling tool, but high level, we are all going to get to the same failure modes.
