Hamel Husain walks through when to write an AI eval and when to skip it. Only build evals for errors you've actually seen, not hypothetical ones. If the fix is trivial, a cheap code assertion works better. Save evals for hard problems that need many rounds of experimentation.
You may have heard that AI evals are really important when you're building an AI product, but you don't always need to write an eval. Let me explain why. So here's a decision tree you can use to figure out whether or not you should write an eval. One is, have you actually seen the error in your data? And the way you can find errors in your data is to do error analysis, which we covered previously. And so if you haven't seen the error, don't write an eval for it. Can be very tempting to brainstorm different failures and write evals prospectively for them. But don't do this because evals can be very expensive and the surface area is infinite. So you want to stay away from this zone. The next thing you want to think about is how much iteration is required to fix an error. So once you've found an error in your data, how much iteration is it going to take? Is it a trivial error where it's really obvious, like, oh, you forgot to put the today's date in the system prompt and the user is asking about creating an appointment? Well, that's really trivial. Know how to fix it. Do you need to go through all the trouble writing eval? Maybe not. If it's not going to require that much iteration, that's a sign that your eval is not going to give you that much value. It's unclear whether or not you should write an eval. Consider not writing an eval, or see if you can write a very cheap eval that's not an LLM as a judge, that is just a code-based assertion. And the last thing you want to consider is if you have a problem that will require lots of iteration, meaning it's a very hard problem, you don't really know how you're going to fix it right away, that's a really good sign that your eval is going to return a lot of value to you. Is going to help you iterate and give you something to measure against as you explore how to solve a thorny problem. If you like this flashcard, I've created an entire series of them. Take a look at the comments for the link.
