Hamel Husain shows how to debug AI systems when you can't access user data directly. Watch latency, conversation length, and tool-call success rates. Use transition-failure matrices to spot where your workflow breaks, and let coding agents handle the visualizations.
A lot of times you won't be able to access data depending on what kind of product you're working on. In those situations, you have to be a little bit creative. So data analysis is gonna be more important, like looking at metadata, looking at patterns about how users are using your application, seeing which subset of that trace you may still be able to look at. Mm-hmm. You may not be able to look at the user's private data, but there's maybe some aspects of it that you can look at. And so yeah, you can do a lot of like statistical analysis, things like latency, number of turns, tool calls, where tool calls are failing. It's important to do that. It's important to explore the data, still look at traces, kind of constantly update your hypotheses on what could be broken and sort of think about it like a data scientist. If you might be familiar with that term, is like kind of put your data analyst, data scientist hat on and say, okay, like where is this application, like where might it be broken and where can I find pattern? So that's something to keep in mind. And there's all these analytical tools you can use, things like using transition failure matrices or just matrices that show like, okay, where the hotspots in the failure are. You know, when one step is getting handed off to another step, okay, like where in that flow are things tending to break? There's different ways of visualizing the data and analyzing it. It's getting easier to do this with coding agents. So you don't have to be an expert in Matplotlib or some code, you know, like programming languages to do this. You used to have to. Be sort of very proficient to do this effectively, but you don't necessarily have to be now. Now you kind of have to be able to ask the right questions and have a good analytical mindset. You know, it's more accessible to everybody than, than it was before, which is really good. But those are things to keep in mind. Okay.
