In this conversation, Robert Ta and Hamel discuss why writing a large number of evaluation tests (evals) for AI agents without first examining real user data is a wasteful practice. They emphasize that evals are expensive to create, maintain, and validate, so teams should prioritize building them around actual failures observed in logs rather than hypothetical scenarios. The dialogue covers how both pre‑product‑market‑fit and post‑product‑market‑fit teams can still gather pilot user data to ground their evals and achieve quick ROI. To make the habit of data review and annotation stickier, they recommend building custom annotation applications that render JSON trace logs in a user‑friendly web UI, hide irrelevant details, provide pagination, filtering, and AI‑assisted search, and can be tailored to specific domains (e.g., PDF parsing). Such tools reduce friction, speed up error identification, and turn the otherwise "boring" work of annotation into a more efficient and enjoyable process.
Chapters
When someone suggests you write 100 evaluators before you've looked at data, and this bear's holding out his hands and he's saying, how about no?
Really,
to sit there and brainstorm all the ways your app can fail and be like, hey, I'm just going to write evals for all the ways apps can fail. Like, like you brought up the competitor thing, but you can brainstorm 500 things like that. And you can sit there and write evals for all of them. And you should not do that. Because number one, evals are expensive.
Eval are expensive.
You have to maintain them, you have to validate them, et cetera. And then number two, like you, so you have to prioritize what you, because they're expensive, you need to prioritize your time. And it's best to prioritize it towards failures that are actually happening in your application. So you get more immediate ROI instead of just trying to like all these hypothetical use cases.
Interesting.
It's okay to like have some hypothetical use cases in mind. If you do have hypothetical use cases in mind, you should try to actually reproduce it. If you're afraid of it mentioning a competitor product, you gotta try to make sure your product is doing that and then use that as an eval. But generally speaking, you wanna ground it first in like actual user errors.
So talk to me about that concept, but applied to say a pre-product market fit team versus a post-product market fit team? I'm guessing the philosophy—
You still want some users, like even pre-product market fit, like you want to try to have some pilot users or some like friendly users. And you want to try to see if you can ground your evals in actual problems they're having instead of hypothetical problems, because again, you need to get ROI quickly from evals. Because like, it's, it can be expensive, it's time consuming, it can be kind of boring. Uh, you know, and it's not boring if you find issues and start fixing things and you're like, oh, this, this is actually really useful.
So is that actually just digging into that? So you, you mentioned this a lot, which is like, yes, this is the boring stuff, but actually, and the core message is, but actually looking at your data is the most important fucking habit. Like do that, your agent product will improve. And that's something that I, I see in my own work is just part of it is evangelizing the unglamorous part of just looking at your data and like doing the annotations and recording expert judgment. So like, there's a question there, which is, have you thought about how to make that a stickier habit or more enjoyable habit?
Yeah. I mean, the best way to make that more stickier is to build your own annotation apps to remove all the friction and the pain of looking at data. So you can go through data as fast as possible.
So for an AI builder that like doesn't even know what annotations are, give a quick flyby of annotations, why they're important, and then why you would like build a specific annotation app.
So you might have logs of your product, typically traces. These are like JSON logs of what happened in your user sessions, like all the back and forth turns with the AI tool calls, retrieval, whatever. And basically you want to be able to review all that data. But okay, you don't want to just put it in a spreadsheet because it's going to be painful to look at. You don't want to just open the JSON file in VS Code or Cursor because that's going to be painful. What, how can you make it as least painful as possible? Make a simple web application that renders those logs and those traces, hides information that usually isn't helpful, shows you like the highest value things. Um, it allows you to paginate through these log files quickly and then annotate where you think an error is happening. Um, and then also like help you search through your logs because AI is really good at that. So that helps make it less painful. And sometimes you need other context on the screen. You might need some information from outside the logs to judge whether or not something went wrong, or you might want different kinds of filters. Mm-hmm. different kinds of navigation depending on your domain. You might want to render the data totally differently depending on your domain. Like if you're trying to parse information out of PDFs, then you want something that is like, shows you a document with bounding boxes where the data was parsed from, et cetera. Again, agents are really good at this. You should totally be making something customized to your data so that you can remove the pain.
So in summary, if you're an AI builder, Annotations and error analysis of where your agent is not doing well, annotations of why it's not doing well. That's so important and it's unglamorous and not sexy. So how to make it less painful, more frictionless is to personalize annotation apps around your log data and upleveling the highest value things to apply human judgment. Cool. And that way you get to the agent being more aligned with human labels. Cool. Uh, so yeah, when someone suggests you write 100 evaluators before you've looked at data, how about no? Also, we're learning from Hamel, the best in the game at evals and objective measurable agent improvement, that doing this is also from a product perspective, just really expensive before you even understood where the heat map of most of your errors are. Yeah. So it's, it's kind of a prioritization problem.
