Hamel Husain introduces the transition failure matrix, a debugging tool for multi-tool AI agents. Build a matrix with agent states as rows and columns, then fill it with error counts at each handoff. This reveals problem spots like high failures when moving from generating SQL to executing it.
If you're building an AI agent and it has lots of tools and lots of steps, how do you make sense of this spaghetti ball of complexity? One of my favorite tools is a transition failure matrix. This is part 11 of a 12-part series on evals. There's 3 steps that you need to take in building a transition failure matrix. Step 1 is to define the states your agents can take, and they can even just be list of your agents. The next step is you can create a matrix that has a from state in the rows the to state in the columns. The next step is to count all the failures that happen in the handoffs. It doesn't have to be the handoff strictly, it can be any error that occurs in that state, and it can really be any metric. But in the simplest form, you can just count the errors. And if you arrange the information in this way, you'll see hotspots. For example, in this transition failure matrix, what you'll find is there's a lot of errors when you're going from the generate SQL step to the execute SQL step. So you have 12 errors there, and that is a hotspot. So this is a really good way of taming all of the chaos of agents. If you're really interested in evals and learning more, I'll put a link in the comments to more resources, including all of these flashcards.
