Neil Howard argues that AI testing misses what matters: users bring emotions and context to every conversation. Current methods catch hallucinations and unhelpful responses, but miss the human layer that determines whether an AI actually works.
Are you finding that your AI in internal testing is being mean to your users? Or it might be hallucinating and not providing the most helpful responses. A lot of teams are going through this because LLM pipelines are inherently subjective, it requires a layer of product taste and also digging through and doing the proper error analysis, looking at your dialectics, looking at conversation terms to see what are my common failure modes and what are the user queries that I need to solve for. And that's how you actually get to the point of rigorously evaluating your agent from an application, application, user and belief centric eval point of view. And this is the input complexity that goes into your agent at time of eval to see what is the next best recommendation or response to this user. Given this context.
