Hamel Husain shows how to debug complex AI agents using a transition failure matrix, a visual tool you customize with rows and columns like tool calls or state transitions. Start with error analysis, spot patterns in your trace data, form hypotheses, and iterate. Each failure tells you what to fix next.
How do you get to a transition failure matrix when you have an agent that's dynamically planning with a react loop or dynamically choosing subagents, tools, that type of stuff? How do we do error discovery and iteratively improve such a product? So maybe if we are going to talk about transition failure matrices, you can remind us what that is, Hamel.
Okay, so this is like a transition failure matrix. One principle from data science is you need to simplify your data You need to find ways to reduce the dimensionality of the problem and figure out, like, what are the most important things so that you can visualize it, so that you can sort of make sense of it. And it requires a lot of massaging, right? So that's where the data analysis part comes in, is like, okay, in the transition failure matrix, these rows and columns can be whatever you want. In this particular example that I'm showing on the screen, These are various tools like GenSQL, ExecuteSQL, Plan Calculation, Execute Calculation, but that might not be the right shape for what your problem is. You need to figure out what is the shape that makes sense for you to dig into. There's no one-size-fits-all answer. Basically, it requires a lot of exploratory data analysis of your traces to figure out like what types of signals are most important. Is it like tool calls? Is it transitions between various states? Is it some other kind of latent signals like number of turns, latency? Is it something else? Is there other some kind of latent signal in your data that allows you to like start digging around? And so that is like very exploratory. Like there's no like button you can push to automatically make sense of everything. It requires like kind of fiddling around a bit. Now you can fiddle around with an agent, like you don't have to build these visualizations yourself. You don't have to write all the code to like summarize the data in this way, but you have to like develop some hypotheses. You have to have some business understanding and some systems thinking about like, how does your system work? And then like, how can you simplify it? You start with error analysis, looking at your data, and it's through looking at your data that you start to develop hypotheses about what is important.
I love it. And in particular, I do think of it as, you know, failure-driven development in some ways, right? That you really look at all the failures and allow that to guide what, as you said, what state transitions you're talking about here, and notice where the hotspots are to build out something like this.
Yes, exactly.
