A talk on why AI agents fail in production. The speaker argues that every customer support ticket reveals an edge case you didn't test for, and that building robust systems requires understanding failure modes and setting up evaluation loops to catch what you missed.
Ever build an AI agent that worked perfectly in test and then completely fell apart in Prague? Yeah. That's not your agent breaking, that's your assumptions failing. You see, agents seem deterministic in controlled test cases, but in the real world, users don't ask what you're asking. It's ambiguous, it's messy, it's unpredictable. And without a solid eval loop or evaluation loop, you're basically debugging shadows and myths. And let me tell you what I mean by that. Take this for example. You tell your agent to summarize a calendar invite for an event application. Someone attaches a selfie. What does your agent do now? It hallucinates because that's an edge case. It wasn't in your eval set. So your agent isn't broken, but your understanding of your failure modes and your failure space is. So here's the move. Start with your error traces, look at your data, build your taxonomy and eval before you vibe code.
