Hamel Husain describes a simple method for improving LLM products: have your team manually review outputs, rate them good or bad, then count and categorize the errors. This low-tech approach delivers real quality improvements and helps you target the right problems before building fancy automated systems.
You have vibecoded an app that makes it really easy for the team generally to go in, annotate these, rate them sort of like good quality, bad quality, automatically categorize them, count them, and then you have a prioritized list and you're like, here are the problems that I need to go solve. And what I love about this is, you know, I'm sure our listeners expect some like magical system that does this automatically and you're like, no man, just spend 3 hours of your afternoon, go through, read some of these chats, look at some of them with your human eyes, put one-sentence notes on all of them and then run a quick categorization exercise and, and get to work. And you see this have actual real impact on quality and reducing these errors?
Yeah, it has an immense quality. It's so powerful that some of my clients are so happy with just this process. They're like, that's great, Hamel Husain, we're done. And I'm like, no, wait, like we can do more. Um, you know, you've paid for more. This is so great. I just feel I know what to do. And so they find so much value in this process. This is something that no one talks about. How do you write an eval? What eval do you do? What tools should you use? Before you get into all that stuff, you need to have some grounding what eval you should even write.
