Hamel Husain argues that tweaking prompts to fix AI products is whack-a-mole. Instead, build a simple human review app in 15 minutes, have people annotate 50 outputs, then feed their notes to an LLM to find systematic failure patterns. This tells you what's actually broken.
If you're debugging your AI product by tweaking prompts, you're playing whack-a-mole. Here's what to do instead. Build a simple review app. With Claude Code, you can make one in 15 minutes. You don't have to be technical. Every domain needs its own. A recruiting AI and a property management AI need completely different views. You want all the context a reviewer needs on one screen. Show your data the way people read it. Emails should look like emails. Conversations should look like conversations. Add a freeform notes field and write your own notes. Don't try to automate this in the beginning. You'll spot problems you never would've thought to check for. Make sure there's keyboard shortcuts and a progress bar so you know where you are. Once you have notes on 50 outputs, feed them to an LLM. It finds the patterns. Now you have a list of what's broken. I built a free skill that makes sure that you're using the right approach when you build yours. The link is in the comments.
