AI Evals Explained With Memes

All right. Welcome back to the Weekly AI Clarity Podcast. From Hamel, evals can be a dry subject, data pipelines, metrics, LLM as a judge, calibration, but that doesn't mean we can't have fun with it. I agree. I've been collecting these memes while teaching our AI evals course, Hamel says. If you've ever stared at a 100% pass rate and felt suspicious, or watched someone reach for a new vector database before running a single eval, You'll probably relate. Let's go ahead and start with this one. This is Hamill's favorite.

Oh, it's a

video. Let's watch it together.

It's good.

It's so good.

The background music makes it really like. Oh, you made this back when you were at 3,000 alumni?

Yeah.

I see. Yo, okay, wait, so tell me more about that meme. Clearly there were so many industry insider level things, the terms and phrases. What inspired this meme? And

yeah, what inspired that meme was there was, there's a framework called HTMX.

Oh, HTMX.

So HTMX. Had this, some person created a meme for HTMX and it was exactly this. It was like all the things that HTMX is trying to like avoid. And you could probably see it on Twitter somewhere. It's like HTMX. Um, you could see like the original, like it was just like this. It was the same video,

same video,

but it was for HTMX.

Okay. So you clearly, as this is going, started making memes as a way to make light of, but also pointedly center on the problems and solutions of evals.

Yeah.

Analysis. And this is, so you saw this HTMX video and this inspired you to make an evals?

Yeah. If you see the HTMX video, you'll get it. You'll be like, oh, I can see. I mean, it's just amazing.

Okay.

It might be hard to find, but it was just like, you know, it was like front-end web development, all the paper cuts that you get. And like what HTMX is trying to solve for. They had like this exact same thing. I was like, oh, I can do this for you. This is a perfect video. It's hilarious.

It is so hilarious. Did the background music come with that original?

Yeah.

Yeah. Dude, that background music just makes the video where it's like this holy, like pure, like amazing solution to all of your problems. The silver bullet. Yeah. Yeah. It's so perfect. It's great. Okay. Uh, I especially like this, like where he slaps the tape.

Yeah,

right there. New model dropped, evals. Fuck it. Incomplete prompts. Fuck it. Eval. I need a graph TV. We just talked about that. Fuck it. Eval. Unpredictable inputs. I need more agents. I think we should buy 2 evals. I, I wonder if this tape would actually be good. Like I have usages for

it. Tape is, yeah, I think tape is good actually.

The tape's actually fire. Unlike, unlike HMX, I guess. All right, next meme. When you're asked, how can we trust the LLM judge? And there's 2 monkeys sitting side by side, one in a brick, one's real, one is a stuffed animal. Tell me more about this meme.

Yeah, it's like, okay, most people, they have an LLM judge, they just have a prompt that says like, hey, is this good? Or they might have a better prompt than that, but they, that's all they got is a prompt. You need to align an LLM judge with human labels. And so this is just making fun of the fact that most people, when asked their LLM, how they know their LLM judge is good, they have nothing to, they have, they have no answer. They're just like,

they'll just ask

them how they know. They're really sheepish about it. They'd rather go hide.

Yeah.

And that's where the meme comes from.

I see. I see. I thought it was something about looking at itself and how the LLM judge—

No, it's like, we got you. We caught you. We got you.

Yeah.

You know, you can't hide. There's nowhere to hide.

There's nowhere to hide. Especially when users are knocking on the door and generating random user queries, or when you made this as a demo and you think it's sick, and then you bring it to your executive and they ask it something that you never expected.

Yeah. Like, how do you know if it's good though? Like, how do you know if the judge is good?

How do you actually measure this thing? All questions that AI builders should be asking. All right, next meme. Our vows need to be application specific. We can't just use some generic helpfulness score. But all the vendors sell them. It's an industry standard. It's fast. Generic metrics create a false sense of confidence. We need to do real error analysis. The guy throws over the table and chair. You're overcomplicating it. Our users don't care about a coherence score. They care that the application actually works. Wow. That's a very powerful statement that just like when you put it out there, it's like, yeah, nobody gives a shit about helpfulness scores, coherence Your users don't care. It's if they got value from using the thing.

Yeah.

Right? Did it improve my life? Is it good?

Yeah.

Right? Yeah. Okay. So what do AI builders need to know about evals and error analysis from this meme and what it—

Yeah. I mean, that meme is like, hey, like don't use off-the-shelf metrics. Don't use off-the-shelf frameworks. Like look at your data, make metrics that are specific to your problem.

And I like this sort of backwardsness, which is The person that's saying, let's go with the generic thing is saying you're overcomplicating it, but it's like, but you actually are. You're, you're trying to bring in all these fake scores.

This meme is the kind of argument you are going to have as an AI builder, as a practitioner, someone on your team is going to be like, let's just use the framework. And you're going to have to fight them just like this.

Just like this. So expect maybe tables and chairs thrown over, but hold your ground and say, our users don't care about a coherent score. They care that the application actually works. Very good meme. High quality meme. Actually, okay, let's start rating these memes.

Okay.

How would you rate this? Would this outta 10 in terms of, let's say,

I mean, I

may— funniness?

It's like 8.

'Cause you were saying like the other meme, the pompous one, it wasn't like as strong.

No.

What would you rate that one?

That one's like a 5. That was more educational.

Okay. Okay. So now we're getting to a rubric around memes. All right, wait, that's kind of hilarious. What about this meme? Going back to, uh, going back to your favorite eval meme. This is 10 out of 10.

This is 10 out of 10? I mean, that just has to be the gold standard. This is the gold standard.

You can't beat that. You really— this is—

yeah, I'm laughing. That thing is a work of art.

I'm laughing even just thinking about it. Wait, I just want to see the model drop thing again. Oh yeah, it does. I love it. It's great. All right, next meme. When the dashboard shows 100% pass rate on your evals, this is a monkey in a suit and tie. What do AI builders need to know about the message of this meme and what inspired this meme?

Yeah, so like, look, if you have a bunch of evals and they're all 100%, it's just like 100% across the board, no mistakes, nothing, um, you should feel really bad about that. Like, because it's not telling you anything. It just keeps saying 100% all the time. Even if you reach 100% across the board one time, it just means your evals are weak. You should try to construct a situation where you're not saturating your evals. You're giving yourself challenging problems in which to hill climb against.

Say more on saturation and hill climbing.

Yeah, saturation is just like, hey, you've exhausted the eval. It's not giving you any more signal. You've totally, you know, conquered that thing. It's not providing you with a challenge. It's not guiding you towards improvement because you already conquered it. And so It's not a good, it's unlike unit tests. Unit tests that you have in software, you want it to pass 100% of the time. Eval are meant to challenge you. It's a goal that you're trying to move towards, and so you shouldn't sandbag it.

So gimme an example of saturation and it being a challenge to the agent building work for your, your courses. Uh, analogy is always this recipe chapter. Right. So tell me more about recipe chatbot as an analogy and then grounding your message here in that.

Yeah. So if you have like a recipe chatbot that's like help you with creating a recipe for your daily cooking needs, okay. Um, you might have different evals, like not adhering to vegan preferences or giving people diabetic guidance when it shouldn't. You want to stay away from that. Something like that. Like you have all these things that might be going wrong. You don't want to just have 100% pass rate on all of those. You want to have, try to think of some more like challenging evals. Now, if your product is super simple, it's fine.

But

if you have like some commercial product,

like a health thing,

yeah. Then, you know, if you're truly challenging yourself, you should not have 100% evals.

Yeah.

And yeah, like if your product is trivial, okay, fine. But then like, who cares anyway? Like you just like, maybe you shouldn't have evals. Maybe you don't need that many evals or just, you can kind of be loose about it. So that's all it means.

I see. Okay. So if you're reading 100% pass rates on your evals week over week as an AI builder team, you're probably this monkey in this team at this

time.

It kind of reminds me of this saying of just The idea of smelling your own farts, you know, like you're just into that. Like, hey dude, like we just want to know we're doing a good job. It's like, no, you're not. Like, we want to know where we're doing a bad job and we could do better. So great monkey meme. Another monkey meme. Could an LLM just do my evals for me? Super hardcore thinking face.

Yeah.

So what should AI builders understand about the message of this meme and what inspired this meme?

Everyone wants to ask this, like, hey, this evals shit, like, Can we just get an LLM to do it for us? After all, LLMs do all kinds of other monotonous stuff for you. And certainly evals sound monotonous. They sound like a chore. Just let the LLM, just let the AI do it. What could possibly go wrong? And that's why this, this is the monkey meme, because it's really just a dumbass idea. Like, you don't want to do that because you're going to miss a lot of errors if you do it this way. You have to look at your data. Because you don't even know what you want until you look at your data. There's certain types of things that an LLM has no chance of finding, things that take product judgment and taste, um, that maybe look okay on the surface, but they're not okay when you go through and read the trace. For example, okay, like if you have a sales assistant and the user has an objection, like, I can't afford this. The AI might say, okay, thank you. No problem. And like, as far as like LLMs are trained to be helpful, but that's not what you want for a sales assistant. You might want to handle the objection or think about alternatives. So it's a common example. There's all kinds of examples that have come up.

Could I share with you one example I thought was super funny? I was doing some work for, this is a small piece of work, like a year ago. And this AI builder and I hopped on a call. And they were like, hey, we have this sales assistant, customer support agent on the website, their e-commerce platform. And they made this sick demo and they thought, oh great, this is gonna work. And it took 'em, you know, all of an hour to make a sick demo. And then they decided, oh my gosh, because everybody around them didn't know how an agent worked, especially last year. Nobody really knew. It's just markdown file, system prompt, like, and then it evolved to harness artifacts and management and evals and datasets and more rigorous engineering, but this is before the before days. So then fast forward, they, they pitch it to their executive team as like, oh, we completed the project in record time because they thought it would take forever. And then the first thing the executive did was ask it about its competitors. And then the agent started recommending competitor products. And so this is a perfect use case of like that engineer or that AI builder only prototyped something, but to go from prototype to production, You have to consider all those things. And like you said, an LLM wants to be helpful. So if you ask it, hey, even if you dress it up with a system prompt of like, hey, you are this platform's customer support agent, and unless you have a grading rubric and evals and a dataset of examples where, how do you handle a user query centered around the competition? Like, you're not going to cover that case, right? So yeah, that's the case where they literally just had the LLM do the evals for them. And bootstrap this whole thing very quickly, but that removes the whole expertise of how the customer support team would normally handle that case. Yeah. Anyway, just kind of hilarious, like customer support agent for a product recommends competitor product. Like, that's like, that's such a, such a meme, dude. Like, all right. Next meme. Claude, find all errors. Make no mistakes. Is this Chris Williamson? The super buff, like Strongman guy, or?

I have no idea, but it's like a meme for like, yeah, someone trying to, I mean, it's like,

yeah. Wait, okay, wait, what should AI builders know about the message of this meme and what inspired this? What will explain that?

Yeah, it's really the same thing as the last, the monkey memes. Like you can't just let AI do your evals.

So is this guy supposed to be like dumb? Like, is that what the—

No, he's just like, it's kind of this guy that's like super locked in.

Yeah.

And he's like, okay, but this

is what he does.

This is, and he's like, You know, just like a power user.

Yeah.

He's like aggressively using the computer.

Super aggressive.

And he's like, okay, make no mistake. Yeah.

There are no mistakes. No, I will not tolerate any mistakes. Okay. Sick. All right. It's meme. Ooh, I like the, I like animals. When someone suggests you write 100 evaluators before you've looked at data, and this bear's holding out his hands and he's saying, how about no? All right. What should AI builders understand about the message of this meme and what it's saying?

It's really tempting to sit there and brainstorm all the ways your app can fail and be like, hey, I'm just going to write evals for all the ways apps can fail. Like, like you brought up the competitor thing.

Yeah. Yeah.

You can brainstorm 500 things like that and you can sit there and write evals for all of them. And you should not do that because number one, evals are expensive.

ELOs are expensive.

You have to maintain them, you have to validate them, et cetera. And then number 2, like you, so you have to prioritize what you, because they're expensive, you need to prioritize your time. And it's best to prioritize it towards failures that are actually happening in your application. So you get more immediate ROI instead of just trying to like all these hypothetical use cases.

Interesting.

It's okay to like have some hypothetical use cases in mind. If you do have hypothetical use cases in mind, you should try to actually reproduce it. If you're afraid of it mentioning a competitor product, you got to try to make sure your product is doing that and then use that as an eval. But generally speaking, you want to ground it first in like actual user errors.

So talk to me about that concept, but applied to say a pre-product market fit team. Versus a post product market fit team? I'm guessing the philosophy is the same.

So you still want some users, like even pre product market fit, like you want to try to have some pilot users or some like friendly users. And you want to try to see if you can ground your evals in actual problems they're having instead of hypothetical problems. Because again, you need to get ROI quickly from evals because like it's, it can be expensive. It's time consuming. It can be kind of boring. So, you know, and it's not boring if you find issues and start fixing things and you're like, oh, this is actually really useful.

So is that actually just digging into that? Because you've taught over 5,000 students now, AI Evaluates, how to objectively, measurably improve your agents and LLM-based products. So you, you mentioned this a lot, which is like, yes, this is the boring stuff, but actually, and the core message is, but actually looking at your data is the most important fucking habit. Like, do that, your agent product will improve. And that's something that I, I see in my own work is just part of it is evangelizing the unglamorous part of just looking at your data and like doing the annotations and recording expert judgment. So there's a question there, which is, have you thought about how to make that a stickier habit or more enjoyable habit?

Yeah. I mean, the best way to make that more stickier is to build your own annotation apps. To remove all the friction and the pain of looking at data. Wait, real quick.

So, so for an AI builder that like doesn't even know what annotations are, quick flyby of annotations, why they're important, and then why you would like build a specific annotation app.

So you might have logs of your product, typically traces. These are like JSON logs of what happened in your user sessions, like all the back and forth turns with the AI tool calls, retrieval, whatever. And basically you want to be able to review all that data, but okay, you don't want to just put it in a spreadsheet because it's going to be painful to look at. You don't want to just open the JSON file in VS Code or Cursor because that's going to be painful. What, how can you make it as least painful as possible? Make a simple web application that renders those logs and those traces, hides information that usually isn't helpful, shows you like the highest value things, um, and allows you to paginate through these log files quickly and then annotate where you think an error is happening. And then also like help you search through your logs because AI is really good at that. So that helps make it less painful. And sometimes you need other context on the screen. You might need some information from outside the logs. to judge whether or not something went wrong. Or you might want different kinds of filters, different kinds of navigation depending on your domain. You might want to render the data totally differently depending on your domain. Like if you're trying to parse information out of PDFs, then you want something that is like, shows you a document with bounding boxes where the data was parsed from, et cetera. Again, agents are really good at this. You should totally be making something customized to your data so that you can remove the

So in summary, if you're an AI builder, annotations and error analysis of where your agent is not doing well, annotations of why it's not doing well, that's so important.

Yes.

And it's unglamorous and not sexy. So how to make it less painful, more frictionless, is to personalize annotation apps around your log data and upleveling the highest value things to Apply human judgment. Cool. And that way you get to the agent being more aligned with human labels. Cool. Uh, so yeah, when someone suggests you write 100 evaluators before you've looked at data, how about no? Also, we're learning from Hamel, the best in the game at evals and objective measurable agent improvement, that doing this is also from a product perspective, just really expensive before you even understood where the heat map of most of your errors are.

Yes.

Yeah. So it's kind of a prioritization problem. Well, one thing I wanted to ask as well, that's a bit more nuanced is I've always thought of software development just like developing a house. Like there's a tax for the maintenance, right? So you could think about this in terms of, well, if I have a Scrum team of let's say 4 engineers, I have 40 hours a week, that's 4 times 40 That's 160 times another, let's say 2 weeks is a sprint. So that's 320 hours. Let's say like 20 hours is a feature. That's 16 features. Usually I always thought about early in my product management career days as like I should reserve some bucket, like, I don't know, 5 to 30%, something like that for like technical debt, errors, bugs, support, maintenance, right? So when we think about evals, I sort of have this question, which is how can an AI product team estimate that tax or that maintenance? Like you're saying like, okay, hundreds of evaluators, probably like I could imagine already, like, yes, that seems very painful. But if someone's making one here, two there, it's harder to understand. So do you have like a measure that you recommend?

No, there's not like a formula thing. You kind of have to practice it and you'll get a strong intuition. There's different kinds of evals. There's code-based evals like unit tests, which are practically free.

Yeah.

And then there's an LLM as a judge, which is very expensive. You got to validate against human labels and do all this stuff. And that is way more expensive. And so you have to think about what is the benefit of the eval and weigh that against the cost of the eval.

I see.

And the benefit of the eval centers around like how hard the problem

is

and how much you think you're going to need to iterate. And therefore, like, how much are you going to be using this eval?

Yeah.

The, the hill climb.

I see. Okay. Say more about hill climbing. I think we went off before you—

Yeah. Hill climb just means optimize a metric or conquer a goal. Like you have, you set a goal for yourself, you set a metric and you just wanna hill climb. Just means you move towards that goal.

I see. Okay. Yeah. That really adds up with the latest Anthropic article about evals and hill climbing. Okay, cool. Thanks for answering those. All right, this one's already hilarious. Next meme, forcing your PMs to look at traces instead of reporting coherent score. And we see here a young man prying open the— Yeah,

that's Nikita Beer, the former, like, I think head of product of X.

Wait, is this an AI-generated?

I think it's AI-generated. Okay, but it's—

You can actually do that, yeah. Okay. And he's just prying, for folks listening, he's prying, this is Nikita Beer, prying open the eyes of an old lady who is in shock of what she's seeing and trying to actively resist. So that's the caption again, is forcing your PMs to look at traces instead of reporting coherence score. All right. Tell me more about the message behind this for AI builders and the inspiration.

Yeah, look, like it's not intuitive that you need to look at your data. Okay.

What, wait, say more about that. Why is it not intuitive?

Because it feels like a manual task. It feels like you're gonna go to a beach, you're gonna pick up one grain of sand at a time, you're gonna look at it.

Yeah.

And you must, you feel that this doesn't seem like a good idea, right?

Yeah, yeah,

yeah, yeah. Because how can looking at one piece of data move you towards your goal?

Yeah.

And the, the answer is it absolutely does. You know, there's a lot in, there's a lot of alpha in your data. You're not just looking through it randomly. You know, like you can sample your data in ways to maximize the chance.

How does an AI builder sample their data in ways to maximize chance value of their error analysis activity?

Yeah, we have a skill called the Error Discovery skill on GitHub. You can Google HAML Error Discovery.

Or drop the comment HAML in whatever you're seeing this, we'll DM it to you.

There you go. And that will help your agent. Like sample the data, you know, it does some clustering, it tries to get diverse sort of traces, then also looks at what you're annotating and like looks, gets clues from that on what else might be interesting. Um, or if you haven't annotated enough of that particular error, it'll get more things like that error. It'll make judgment calls. It'll

try

to do a little bit of, you know, it's like—

Prioritization work.

Yeah, prioritization and searching for you. It's not perfect. Looks like anything else, but

yeah,

you know, it's one way to try to have a more fruitful search through your data.

Like colors your sand so you could look for it. Yeah. Okay. I like that analogy, by the way. I do think that that's how people feel when you tell them to look at the data.

Yeah.

Because it's so overwhelming. Like, it's like, I have millions of rows. Like what? You know? Great. Oh, this one. Obama meme. LLM output. Being rewarded a medal by LM judge, not grounded by human labels. Both Obamas, one is happier than the other. All right, Hamel, what should AI builders get from the message of this meme? Yeah,

it's like, look, you know, you don't want to just use another LLM to tell you whether or not your product is good. That's not telling you anything. You're going to be like this meme. You just reward. patting yourself on the back.

What

you need to do is have human labels or some other trusted source where you know it's the ground truth and give yourself a reason to trust the LM as a judge. Don't just go on it by faith.

And do you have like a typical amount of human labels you recommend as a minimum viable eval sort of?

It's not like a bright line rule. Try to tell people, hey, like have at least 50 labels, but that's just because people love to need a number. Yeah, it will like not be able to sleep at night if I don't say like a number.

I can't personally.

Yeah. So, um, that's why I say 50. At least it gives you some mental

hill to climb.

Yeah. It gives you like a goal that you can achieve. But you know, like the less labels you have, the noisier your statistics are. So that's all you have

to keep in mind. Wait, say more about that. The noisier.

I mean, like if you only have like 3 human labels, then the, the noise, the variance is going to be super high. Because either sample size is low. So when you go to compute the statistics of agreement between your human and the judge, you know, you, you want to measure precision and recall, not just agreement.

Yeah.

Then it's going to be super noisy.

Yeah.

It's going to give you some estimate.

Super skewed.

And whatever estimate you get is going to be, it's going to have error bars like very large, right? If you don't have a big enough sample.

Because it's just 3.

And this is something you can measure.

Yeah. Yeah. Yeah. Definitely check Hamel's blog and his course for how to measure in more detail. All right, let's go over a couple more. Okay. I'll keep going on the Obama meme game. You, we check our AI with coherence score. Them, how can, how we can trust this coherence score? You. And this, he's sweating bullets. Like he's—

Yeah.

Have you seen this play out in real life?

All the time.

All the time.

And sometimes I am the person that is invoking the stress.

You're the them.

So like someone will come to me very excited with a dashboard, be like, hey, look at—

super hype.

Look at all these scores, like helpfulness, coherence, G score, whatever score. I'm like, I don't even know half the scores mean, honestly. And I'm like, okay, how do you know that this is good? Like, what is that score telling me? And how, how do you know? That this— that we can trust it?

Yeah.

Crickets.

Crickets. This guy.

And then that's when people get stressed out. They're like, oh shit. They're like, we— you know, like, that's— they're like, that's a good question,

actually. And I'm like, all been fucking vibes. Yeah, yeah. Nobody in the room wants to say that, but that's actually what it is.

You can

see, like, dude, you know, the craziest shit is like, I've just seen millions and millions of dollars Down the drain. Like, and, and, and what it is, is like, this is some engineering leader wanting to be like, oh no, did we just throw away millions and millions of dollars? Like, we didn't answer that simple fucking question. How do we know it's improving? Right? Like, that's what it is. There's sort of this— I think that plus the combined, oh my gosh, to do the right thing has to go, we have to go to the beach, pick up grades of sand. It's just, these things are like drugs, like the coherence scores and stuff. It's like, you don't want to go to the beach and go through Ransom Crans? Here's a Mai Tai and we'll give you a score. Like, it's, yeah, that's, and give us money for the right. Like, and then when somebody real comes along and they're like, how do you know that this gets better week over week? And they're like, uh, this guy.

Yeah. Yeah. It

happens all the time. Happens all the time. Oh man. Send us a note if this is happening to you. Hamil does have some, he has a big FAQ. I would recommend going to it. And I think part of it as well is how do you convince your team to— yeah. All right. Next meme. What are your superpowers again? I read traces logged by AI products. And this is Batman and Robin for context, right?

Yeah.

So I think 2B.

The original, I think, is like, what are your superpowers? And he just says, I'm rich. And in the core layer, it's like, okay, if you're an AI engineer, what are your superpowers? If you're really good, it's like, I look at data.

Yeah. I'm rich with my error analysis, bro. Like, I'll be error analyzing all day, every day. By the way, what, as, as I go through this, what is your meme? How do you like, what's your meme creation process? Like, how do

you— Man, I had this whole—

Just out there memeing? Like—

No, I had this whole meme creation process. I use this thing called Memelord.

Memelord.

Yeah, it's a software. Does that have MCP? We can get memes. I'm not doing memes anymore. I think I like saturated the meme.

Oh, you dominated the eval meme market. There's no more memes to be made,

Jimmy. No, just like—

There's more, no more laughter to be mined from this eval's content market fit.

Yeah, maybe I'll come back to it one day, but—

So this is Memelord?

Yeah.

The home of the professional unprofessional. So wait, you had a, I just, for the pod audience, what was your meme creation process?

So I had this whole thing where You know, it's like, what are common mistakes people make? And it would mine like memes. I would also pay attention to memes. There's like people that meme a lot on, on X. I don't even remember who they are, but there's people like have memes. I would like pay close attention to memes and like, this is really funny. Maybe I can, you know, do that, do something like that. And yeah, I just paid attention to it.

So these are memes created with expert care and love.

Mm-hmm.

I see.

Yeah.

All right. Using Meme Lord. This is, that's actually great. I love it. All right. Next meme. Reviewing every single trace instead of using a transition failure matrix. This is Bill Gates

and Warren Buffett, I think.

And Warren Buffett with giant paddle. What's that? What's that? Yeah,

it's like, okay, like, you know, you should use good analytical tools. This is a little bit technical, so we may not be able to Cover it.

No, let's try it.

But, um, yeah, there's good analytical tools. There's something called a transition failure matrix, which is an analytical tool you can look at for long-running agents with a lot of tool calls and a lot of steps. You can kind of put things in like this grid and see like what's failing, like a heat map of what's failing, like from one state to another state.

Oh, okay.

What state tends to fail the most? And then you can use that to help you prioritize like what to zoom in on.

Ah, I see. For your long-running agent tasks.

Yeah, for your long-running agent tasks, or if it's just like highly agentic in general and it's like, has a lot of moving parts and tool calls and stuff.

Okay.

Yeah, it can be really useful. So it's like, okay, you know, don't brute force everything. Like go about evals in a smart way. Use analytical tools.

Use the right tool for the job. So when you have lots and lots of traces, use transition failure matrices.

Yeah, that's one approach. And there's all kinds of other things we talked about, like how to sample earlier. So just use, you have to put on your data science hat in the limit and say like, okay, how do you look at data efficiently?

So maybe that's an interesting thing. What are the top 3 pieces of advice you would give to AI builders that are discovering they have to also be data scientists? Because that's actually the name of the game.

Practice. Yeah. Practice looking at data, exploring data. There's a lot of different like exploratory techniques with data, but it's also just Having practice, like building metrics, knowing what goes wrong with those metrics, knowing like all the biases that exist. That's the best advice I have.

Okay. Got it. Cool. So practice makes perfect. You heard it here, folks. All right, let's do 2 more. This one is this AI evals data analysis of traces is a little butterfly. Okay. What should AI builders take away from this meme? And what was the inspiration for this, Hamel?

Yeah, I think people get confused. When people think of evals, right, they think about what are the scores? Give me, just give me the scores. Just give me the test. Give me the scores. I want the metrics. But evals is an entire process. And the most important part of the process is looking at the data and it's analyzing data actually. It's not, uh, it's not really like the metrics that are, the metrics are important, but I think there's a lot of emphasis on metrics. There's not enough emphasis. of doing data analysis of your traces to see what is going on in your data. Uh, and so that's what this meme is about.

Got it.

Is the surprise that a lot of AI engineers have.

Got it. So you're saying most AI engineers have this belief that actually just give me a score, give, try to help me.

Absolutely. 100%.

But you're saying you gotta look at this differently, philosophically. It's about defining and prioritizing where your agent is weak. And then figuring out, should I even put in a VAL here? Is it worthwhile or valuable to the end user and my business to do that? Because that's a tax, like we talked about, it takes effort to even bootstrap one, validate it, improve it. And so it's much more of this mindset of use the right tool for the job and get into your error analysis, but in thoughtful way, like you were saying earlier with the error discovery skill or with the transition failure matrices. Yeah. As ways to look at your data more intelligently. Okay. Yeah, I like this as well. This is, this comes up in my feed a lot in other, other types of memery. Okay. Is there one you want to end with? You want to pick one of these last ones to end with?

Let's do, uh, the snow leopard one.

Okay. It's, so there's a snow leopard here. It says pass on everything. And then there's this skier, snowboarder woman. My judge is 95% accurate. She's really hyped. The snow leopard looks equally pretty hyped.

Yes. Yeah, yeah,

yeah, exactly. So tell me more about what AI builders should take away from this.

So look, like when people have their LLM as a judge, they love to compute agreement or they love to compute accuracy, accuracy, helpfulness. So it's not about, yeah. So accuracy is a very tempting metric. It's like, let's say you have 100 examples. And you have 100 labels, a lot of people naively will say, what is the accuracy of my LLM judge? By saying like, what is the, like, how many of those labels did I get right? That is the wrong way to look at it.

Why is that the wrong way to look at

it? Because a lot of times, like most errors are not balanced. It's not like most errors you can find in your data, they're not like half of them have errors and half of them don't. No, it's like, it's probably like a few errors or like maybe 10% of your traces have errors, 20%. That meme is just showing, is just driving home the point. Like, look, if your error only happens 5% of the time, it is super trivial to have 95% accuracy because all you got to do is just say it never errors and you will be 95% accurate. So instead you got to measure precision and recall and you got to compute

Say more about precision and recall.

Yeah. So precision is out of the examples that you identify that has a problem, out of just those that you say this has a problem, how many of those are correctly identified as having a problem? That's basically trying to measure like how much noise are you introducing, you know, as an alert. So if you have like high precision, then it's not that much noise because you are, everything you say has a problem actually has a problem.

Lower error bars.

Uh, not really low error bars. It's just saying like, when you say something is an issue, it's an issue if it has consistency.

Consistency. It's

not really consistency either.

No,

it's like, it's just saying like, okay, so let's talk about recall and come back to precision. Recall is how many errors can you find in the first place? Like, you know, and not miss any errors.

So

that's recall. How many errors can you recall? How many can you find? In this like messy bucket of like a lot of non-errors and errors, how can, like, how good are you at getting all of the errors?

Yeah.

Precision is like, okay, well you can just get all, you can just say all of them are errors and have 100% recall.

Yeah.

You just say like, you all have errors, then you'll get 100% recall.

Yeah.

Precision is, okay, out of the ones you said is an error, how many of them actually are errors? That's precision. There's a precision recall trade-off. There's always a precision recall trade-off. You can always get more recall by just saying this, just relaxing your criteria and saying like, look, everybody's guilty.

I see.

And that'll give you higher recall, but then that'll lower your precision. And you can always get less precision by just saying like, I'm only gonna go when I'm like, I'm only gonna mark something as failing or guilty or whatever when it has, I have like 100% confidence. Then you're not gonna, you might, you'll, you know, the one person that you mark or whatever will be able to have a problem, but you'll miss everyone else, right? So there's an inherent precision recall trade-off that you have to make depending on your product, your product goals, and you have to measure that. You can't just measure, have agreement. This is a big mistake to like say, to put the word accuracy. So if you ever go into a meeting that has evals in it and someone puts up The word accuracy on the slide, on a dashboard, you need to stop the meeting and be like, what are you doing?

That might be a fucking snow leopard behind those guys.

Yeah. I mean, it's like, yeah, that, that immediately, that number doesn't make any sense.

Yeah. Yeah.

Yeah. That word doesn't make any sense in the context of evals.

I see.

You have to have a really good damn reason to be

using

the word accuracy.

Got it. Okay. So for AI builders. Don't use the word accuracy when it comes to evals and error now.

Yeah. And it's not just, don't just skate around it by like trying to like find a synonym for accuracy, but don't, that metric itself doesn't make sense. Yeah. Think about, again, precision recall.

Yeah. Yeah. Think about precision and recall if you're an AI builder. Otherwise you're going to have snow leopards on your ass.

Yep.

Yeah. You could be happy thinking you're accurate and then the competition—

And you can fool. Everybody in the room by saying it's 95% accurate, everyone's gonna nod their head except when shit hits the fan and your product isn't good cuz you're not actually measuring anything.

Yeah. Or this person comes in, how can we trust this score?

Yeah.

And then you'll be like, wait, shit.

Like, I never thought about that.

I never thought about that.

Like, I thought like, yeah, I was like,

hmm. I thought numbers go up. Thought we were good.

Just like trust the LLM, like whatever it says, it's like, it's God, right?

LLM judge, bro. Like, yeah, bro. Like, get at me. Like, what are you talking about? Yeah, it's validating its shit. It's another one. Yeah, that's what it's doing.

Exactly.

Exactly. Oh, that's rich.

Okay. So, I mean, that's why I created these memes, right? Like, this is all fairly comical when you think about the kinds of errors people make, but this is like real.

This happens on the daily. Yeah, this happens on the daily at multimillion, multibillion dollar companies. That's what people need to understand.

Yeah.

And that's why I really appreciate your work, Kamal, where you're trying to share truth around how do you actually objectively and measurably improve these systems, which is something we all need to learn about as AI builders to go from prototype demo all the way to production and users actually love it and they come back and they recommend it and the business makes money. So with that Snow Leopard meme, I think that was a great last one. Think about precision and recall. Don't say the word accuracy. when it comes to your agents and objective measurable improvement. Thanks a lot, Hamel, for being my co-host and guest on this podcast today.

It was fun to be here.

No, it was sick. It was great to interview on your expertise at the intersection of memes as a medium to share that expertise. I think that's like so perfect. And I've seen these pop up in my feed and I've definitely liked them and reshared them. So I appreciate you putting that out in the world. One last shout out for Hamel, if you see this, join the AI Eval for Engineers and PMs course. This is the most popular AI eval course available. You could learn how to build, evaluate, improve AI agents that actually work in production. Check it out. Hamel has taught over 5,000 certified builders now, including builders at OpenAI and more. Thanks a lot for joining the pod. We'll catch you next time. Bye-bye.

Thank you.