LLMs often fail to catch their own mistakes because they lack product context. Hamel Husain's example: an LLM approved a hallucinated virtual-tour scheduling output as correct. Manual human review is essential for error analysis, not automation.
One common question that we get from people at this stage is, okay, I understand what's going on. Can I ask an LLM to do this process for me? And I loved Hamel Husain's most recent example because what we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be bad product smell. For example, the hallucination about scheduling the tour, right? I can guarantee you, I would bet money on this, if I put that into ChatGPT and asked, "Is there an error?" It would say, "No, did a great job." But Hamel Husain had the context of knowing, "Oh, we don't actually have this virtual tour functionality," right? So I think in these cases, it's so important to make sure you are manually doing this yourself. Number one pitfall right here is people are like, "Let me automate this with an LLM." There are parts of error analysis that an LLM is suited for, which we can talk about later in this podcast. But right now in this stage of freeform note-taking is not the place for an LLM.
