Intro To Error Analysis: Creating Custom Data Annotation Apps (4k version)

And just a quick introduction. I'm Hamel Husain. I've been working on machine learning and AI for over 20 years. I've worked at a lot of different startups like Airbnb, GitHub. I've been working on LLMs for a really long time. I was really early at GitHub, you know, working on LLMs. I led research that led up to GitHub Copilot. So I've been working on AI products for a while. Uh, and I'm excited to be here with Shreya to talk about error analysis.

Yeah, hi guys, I'm Shreya. I am a PhD student in UC Berkeley in the EECS department, but before that I was an ML engineer, so I did a lot of MLOps work, a lot of evaluation and deployment and monitoring, um, of large-scale time series production. And now in the age of Gen AI, I am super interested from a research perspective of You know, how do we rethink how we do evals when we're not training models, we're using off-the-shelf foundation models, we haven't explicitly encoded loss metrics or have too much labeled data to go off of. So what does it mean to kind of build confidence in how we evaluate our systems? So I'm very excited to be here with Hamel Husain today to give you a preview of the curriculum we're building around this. I will move to the next slide.

Yeah, good. Um, so just the motivation of this is, okay, a lot of people, they, you know, they jump straight to frameworks, tools, like, you know, what RAG database should I use, what agentic framework should I use, so on and so forth. And it's quite rare that people are looking at data, and this is, uh, what everyone is stuck on. And, you know, I'm currently an independent consultant who helps people with AI products. And I would say, yeah, the thing that people are stuck on is the reason they're stuck is they're not looking at data. And so this is like conventional wisdom in machine learning circles. This is a tweet by Greg Brockman. You know, manual inspection of data is the highest value to prestige ratio of any activity in machine learning. I would go even further than that. I would say it's the highest ROI activity in machine learning, period. And it's probably, or it is, the highest ROI activity if you're building any AI product, period. It doesn't feel like it would be. It feels like, oh, looking at data is grunt work. But actually, when you look at data, you, you get a lot of value very fast. So yeah, that's just the motivation. I think, you know, this is going to be time well spent. Um, so yeah, let's get into it. So today's agenda, we're going to learn a technique, a structured technique for looking at data. So like, what do we mean by looking at data? And like, how do you look at data? Well, there's a way to do that. It's called error analysis. It's not something we made up. It's been something that's been around for a really long time in machine learning. You know, in machine learning, we've had to deal with systems that have stochastic errors, or, you know, stochastic outputs and stochastic errors. And, you know, we've had to deal with these kind of systems for a long time. And error analysis is kind of a time-tested technique that you can use to quickly find What, number one, what should you invest your time in? There's so many different moving parts in AI. You know, there's so many different components of your AI. Like, should you tune your RAG system? Should you, you know, should you do prompt engineering? Should you invest in a framework? You know, people have these questions. But really, like, error analysis will guide you and help you know what to invest in and also what to measure. So like, what do you measure? You can measure any— everything. It's infinite. And so, uh, it'll, you know, it guides you and it almost like helps you create your roadmap. Um, and then I already talked about, okay, error analysis is not new. It's not something we were making up. Is this something that a lot of people don't know about, especially if you don't have a data science background? Cool.

So we're not going to be able to go through all of these steps of error analysis the day, but we're going to go through the first and the second, as much of the third as we can. But we like to think about error analysis as multi-stage. How do you go from zero to having some set of evals that are custom for your product? First, you start with literally looking at your data and annotating it. What are example inputs your system might have? What are example outputs? Um, what, how do you, what are the vibes that you have? Can you write those vibes down on an output-by-output basis. Then the second step is to take all of your freeform notes and categorize them into groups. So you, you can use LLMs to do this. We found that that's somewhat effective, especially at a smaller scale. Maybe you can try really advanced clustering techniques if you would like, but some way to kind of group these notes together. Then what Hamel Husain likes to call turning your outputs into a pivot table. But basically, in lay speak, that's which outputs correspond to which categories and how many errors you have per category. Given this kind of view over your data, it becomes much easier to figure out, okay, how do I ship improvements to my pipeline? Maybe I change my prompts, maybe I fix other parts of my system. Maybe that in turn creates new metrics. And then finally, I've got to iterate, go back to the beginning, keep looking at data. We find that people kind of do this entire process several times, even before kind of deploying their system. And at some point, you become a lot more— your system becomes a lot more robust. It doesn't really change throughout cycles. You get newer— you get fewer and newer metrics over time. So today's lesson, we're gonna have a case study where you kind of put together an AI recruiter, very basic AI recruiter that has a position that it's trying to hire for, scraped a bunch of candidate resumes. We actually got them off Kaggle and it's trying to write personalized emails for these candidates when applying, basically inviting these candidates to apply for positions. And the problem that we have when we try to write this pipeline, we use Llama 3.70B to generate these emails. We just weren't really satisfied with the vibes. And I think this is really common in a lot of people's cases when they want to build evals. Like, they built something, but they kind of feel like the vibes are off and they don't know how to turn that into something actionable. And we did try a few rounds of prompt engineering, but it felt like we We fixed one thing and we broke another. We didn't quite fix all the things we wanted to. We were just kind of ad hoc iterating on it. And Hamel Husain has this excellent meme here. Maybe I'll let him briefly mention it in the backstory.

Yeah, so this is my friend Greg Saccarelli. And you know, he's playing this game of whack-a-mole. And this is exactly what you are doing if you try to build an AI product without evals. Or especially without looking at data. You know, like you see a problem, you're like, oh, let me fix it with prompt engineering. See another problem, you're like, oh, let me fix that with like some, you know, tuning my RAG or whatever. And you just see a problem, you just keep hammering these different problems. But when you fix one thing, a lot of times you're breaking another thing. You don't really have a way of knowing if you're making any progress. And so you want to avoid this. You want to avoid this endless cycle that goes nowhere with whack-a-mole.

Cool. All right, so back to our case study. So I mentioned I got a dataset from Kaggle for this lesson, and we'll, we'll post these slides later on so you can look at if you would like. But it's simply a collection of resumes. Each resume is— think of it as a blob. It's a text string. The only field that we really care about here is this resume_string. And we take a bunch of HR, HR category resumes and try to write the customized emails to encourage these people to apply for a university admin slash HR coordinator. So when we wrote our pipeline, the prompt is very straightforward. It just kind of details the job description. And ask the AI to write a personalized email that references the experience, that references the candidate's experience, we get some good examples like what's on the left, and we get some bad examples like what's on the right. And I want to emphasize a point here that even this is subjective. People write email AI systems all the time and they have their own criteria. But, you know, I'm a basic person. I just want to make sure it doesn't sound bad. So maybe what's on the left seems pretty good, but what's on the right, like, doesn't have good vibes. I was reading it, Hamel Husain and I were reading it, and I was like, okay, maybe it first pulled the wrong— I don't know why it pulled the flight attendant part of their resume, if at all it's a flight attendant part. It says, you know, although your background is mostly in management and customer service, that's already— that feels like bad vibes. Hamel Husain said it felt like a rejection email.

Yeah, it's like, yeah, yeah, definitely does. Yeah, this is like, although your background— any email starts with although your background, I'm like, okay, I just got rejected from something. Uh, yeah, it's actually pretty funny.

Yeah. Oh, go for it.

No, I mean, it's like, I feel like I get really bad emails from recruiters all the time, uh, that are just like maybe even worse than this. So this is very interesting.

And sorry, can I just ask, is it supposed to be to the HR manager or from the HR manager as well?

It's going to be, it's supposed to be to an applicant. And another issue is probably that it shouldn't say dear HR manager, it should say dear that person's name. But a lot of these resumes are, they don't have the candidate's name in them. So I guess the LLM just fills out some role there.

Honestly, I've noticed OpenAI is, it will switch recipient and sender quite a bit. Gemini 3, sorry. Gemma 3 is a lot better on this, I've found.

Interesting. Well, that's a good failure mode to keep in mind because when we look at the data, I'm— we're gonna see that switching the roles. Um, well, all right, so hopefully this kind of illustrates the pipeline that we're building and illustrates that there can in fact be bad emails. Um, and, and I don't have a great definition of bad. That's another thing you should keep in mind right now, as we know, we trust ourselves to identify bad vibes. We cannot articulate what these vibes are at this moment. All right, so let's jump into annotating some data. Um, whoops, I accidentally skipped the slide. And I think one of the things that I want to do first, I think I have to pull it up Maybe I can't— oh, I'm only sharing the window, so let me share my bigger screen altogether. Share cursor. Share. All right, so I generated a bunch of data, and I could try to open this in Open With I don't know why it doesn't let me open it. Oh, I see Zoom is in the way. Open with Numbers. Does that not work? Guess not. Okay, well, I wanted to show that just simply opening this in Excel spreadsheet is not the right way to go, which you can if you want. It's fine to totally just create new columns to annotate your data in Excel, but a lot of this is just big text blobs. So I, what I'm gonna try to do in this lesson is vibe code an interface for us to label this data. So I hope it works. I've never vibe coded live before. But part of this is also gonna be me trying to come up with the right prompt to give us a good labeling interface. I don't wanna show you all of the code that I used to generate the outputs, 'cause I did generate outputs with some known bugs. Okay, so I want a custom data labeling interface for me to annotate my outputs of this pipeline. I want there to be a view with a single input/output on the page. This— maybe the resume on the left and the email in the center. And then an open-ended text feedback box on the right where I can write any failure modes I find in the data. And then some other things that I find when I like to build custom interfaces are I want there to be some shortcuts. So Have left and right hotkeys to quickly iterate through the rows. Another thing that I really like is have a table of feedbacks persistent, maybe at the bottom right. I typically find this helpful because sometimes when I've written failure modes for previous outputs, I just like to see kind of a list of failure modes at all times that I've generated. So far. Then what else do I want to have? Generate this app in Flask and HTML. I just don't really want a whole React app generated at this moment, but—

That makes sense.

You could do that if you want that. It's just a static app. It can be pretty chill. Don't use— I do have like an old version of this annotation app just to make sure that I could have vibe-coded this correctly. Don't use any other— don't use any files from annotation. This is pretty hacky because you probably won't have an annotation app old. Oh, and the last thing that I want is make sure that the there should be a save button. It's important. Save export button to save the feedbacks along with the rows.

Do you have to tell it where do you want to save it?

Oh.

Does it usually?

I don't know, vibe coding. What do you tell it?

Yeah, I think like before you didn't have to. So maybe we.

Yeah, I don't know.

We'll see.

I mean.

I think the key here, if you're gonna do this yourself, is to use, uh, you know, Claude 3.7 or, you know, Sonnet 3.7 thinking mode in Cursor. Make sure you have that turned on if you're gonna vibe code it this way.

Well, I don't know if I've never tried it outside of— okay, so let's see. The interface will display one resume email in the center. Open-ended feedback box on the right, arrow navigation key. Yeah, sounds great. Oh, and I do have a custom system prompt that like parrots back what I said. I don't know how effective that is, but I just like the idea.

No, I like it. Yeah, it's always good to have a plan.

So yeah, make this, why not? I'm not in the YOLO phase yet of my life where I can just have it execute arbitrary shell commands. Right. I'm really excited.

I mean, so the point of doing vibe coding, might be wondering like, why are we vibe coding it up? It's to drive home the point that like, okay, the first step that you want to do is you want to have a data, your own custom data viewer. And we can talk about like why that is, but you're like, you know, nowadays it's so easy to create your own custom data viewer with AI-assisted coding. And like probably more than half of you already know that, but some people they, they don't. They're like, oh, like why do you, what do you mean, like, you know, create a custom data viewer? Well, um, yeah, it's just really straightforward. Like AI is really, really good at custom data annotation apps, and then you'll see like why. Um, and then I'll show you some screenshots of actual data annotation apps that I've built for several other companies in a similar amount of time that Shreya is gonna do it. So Shreya is doing this like in under— probably like round trip should spend like significantly less than 10 minutes. Um, I mean, we'll see. Maybe I don't want to jinx it.

Let's not jinx it. But we have some questions. All right, so we see, why not use tools like Label Studio or YOLO? This is an excellent question. I used to try to use tools or even Excel or Google Spreadsheets, but I found that I got to a point where it just felt like a little bit too much friction. Because when I see, say, an email string in an Excel or like in just some cell, I have to do the mental work to view that like an email in my head before I check the vibes. And when you build your own annotation interface, you can do small things like have the email look like an email. You could say it's not email, say it's code or you're generating plots with AI, you could literally just render that and then quickly look at it. I think it is actually worth it to make your own custom interface. It's just so easy right now with these AI tools. Why custom HTML, JS instead of Gradio? You can use Gradio if you want.

Yeah, use whatever you want.

Yeah, use whatever you want.

Doesn't matter. Use whatever you're familiar with, whatever you want. It really doesn't matter. You know, it's helpful if you use what AI is familiar with. So if you use some kind of established, you know, design pattern or, you know, frameworks or whatever, it does help. Go ahead. Okay. So it looks like you already have something.

Something's running. So let's see what it is, folks. So we are here with this custom email annotation interface. And so hopefully already you can kind of see, oh, wow, if I just generate like looking at this like an email is better than looking at this in some kind of CRUD viewer that is pretty generic. But maybe we can start by coding the lives. Yeah?

Yeah.

Let me make sure that— okay, so we got there. Oh, we didn't finish yet. All right, so we're here. So let's start by looking at this. Okay, so Already, already Hugo mentioned a great point here. We don't even know who this role is, so maybe we should have instructed the LLM to fill in the recipient.

You could just say greeting is incorrect, for example.

Greeting is not correct. And I think at least I like to have somewhat more verbose feedbacks in the beginning because it helps me reason about what's going on. The greeting is not correct. The recipient could be the name of the person or blank if the resume does not have the name.

And another error is this salutation, I guess you call it, or is that the best regards, your name, the placeholder is there?

Ah, yeah. Um, yeah, we can say the, uh, is it called salutation?

I don't know.

I don't know what it's called. Closing. The, uh, best regards. That's Hamel Husain Husain.

I have to look that up. I just pulled that out of him.

All right, and then actually reading the content, um, let's take a moment to read it.

Oh no, I think salutation is the beginning. Sorry, but whatever it is, the— yeah, the goodbye, the end.

Um, I think it's okay.

Yeah, it seems fine.

Solves the challenge. Our team is supportive. I'd love to discuss. Please let's schedule a call. I do like the fact that it said please let's schedule a call. Um, sometimes I find that people also mention things that they like, because then you could go back and find common things you like and encode that back into the prompt. But yeah, let's save this. And then the hotkey worked, by the way. I just pressed my arrow, so that's exciting.

That's huge. Hotkeys are huge because you want to make it— you— it really reduces the pain if you're doing this like thousands of times.

Totally. Um, this is great. This is exactly the issue that Hugo mentioned. Right? You're— the LLM just like understood the assignment wrong. It's not that the candidate is trying to apply for this role. It's that we are the recruiter trying to get the candidate to apply for this role. So, um, understood. Which—

this one is hilarious. This one is like—

there's so many things going wrong.

Just, uh, yeah, this really got the assignment wrong. Um, I can't wait to get an email like this from a recruiter.

The recruiter—

be amazing—

like reply with something asking the candidate to apply, not the candidate asking to apply. And then irrelevant background.

So the thing that I want you to pay attention to here, notice that Shreya is adding notes, free-form notes. Okay, we don't know what the errors are yet. The most important thing is like we're going through different examples and we're writing these notes in free form. We're not trying to like apply categories or no, like, late— we're not labeling yet. It's too premature for that. So like we're doing error analysis correctly, especially in the beginning. You want to just observe what you see and just write notes. And then we're going to talk about, okay, like, what do we do with those notes?

Yeah. This was horribly irrelevant. Everything was wrong with it. Quantum physics. I don't even need to mention that. So note that we're also going slowly, and that's like totally okay. We're gonna go slowly. You should go slowly. Like, people spend minutes per output on the first several outputs. Like, think of this as like spending an hour going slowly. It's gonna give you a huge payoff. Okay, so with your extensive interest in data analysis and software development.

Mm-hmm.

Okay, so one of the things that I realized that I would like to have is the ability to search the resume for things that the AI actually said. Um, analysis.

Yeah, I don't think this person looks like an HR director. So it kind of bungled the Yeah, it could have inferred.

Yeah.

You're gonna have to think.

It inferred the wrong experience.

Yeah, and so you have to think, okay, you're not trying to root cause it just yet. You're just writing down what's wrong. Root causing can come later. Right now, we're just, yeah, you're just like writing down what's wrong. Okay, like why did this happen? Why is it thinking that this person has data analysis, software development experience? Maybe there's something wrong with prompt construction or RAG. Or something. We don't know. We don't know yet. We're just recording what's wrong at this point.

Yeah. Oh, interesting. Web page developed. Okay, sorry, I'm doing exactly what you said, which is not— don't try to triage why AI kind of inferred the wrong thing. Just say what happened, which is this. So we keep going. We're excited to introduce you to this customer service.

Okay, what does it say? Okay, clients and manage conflicts. Um, I mean, it's kind of weird vibes, honestly, but establish rapport with clients and manage conflicts. I'll probably say at this point I would change the data annotation. We're not gonna do it now, but I would also add the job post down here for sure.

Yes, yes, that's a good point. So searching the resume. So this is very common actually when I generate an annotation interface, like finding out that there's new things that I wanted after I generated it and then going back and edit it, which now we can vibe code, having search functionality in the resume. As well as the job posting here. I'm not going to put any feedback on this. Honestly, I think it's fine-ish for now. We can keep going. All right. So I was impressed. I was impressed with your background. I believe your skills are helpful, but I do feel like since I've seen Conflict Revolution twice, Like maybe this is just a smell of the language model that I just don't like. Like, yeah, please don't infer skills from their resume that are not listed.

I mean, to be honest, I don't like any of these, but I didn't know that. I mean, I would— now I have a really distinct, like, recommendation I would make, or I have a really good idea. Like, okay, I don't like all this BS, like introductions. I was impressed with your experience. You would make a great contribution to our role. Like when you see that, you just have PTSD from thousands of recruiters, right?

Like, you're freezing, you're okay.

So like, how do you make it good? Okay, like I want something that like, is something very specific to that person, you know? So I would, based upon what I see right now, I would say, okay, like, that's not, maybe we don't want to have that feedback just yet, but that's something I would write down and say, okay, I need to rethink this whole application. You know, I want these like recruiter emails to be like really good.

So hire Hamel Husain as your product engineer, but you know, AI recruiters are writing emails like this. Yeah, you can, you can have your own vibe. The point is you can encode your vibes into your annotation and back into your pipeline. I agree with that.

Yeah, again, this annotation is really important. Like, you know, writing notes. The writing notes part forces you to think about what is happening, and I promise you that these notes are going to turn into something extremely valuable.

Yeah. Um, although your background is mostly in HR and administration, this is exactly what you said. Um, it's not good vibes. Like, you can just say basic things like it's not good vibes.

Yeah, I mean, I would say it feels like a rejection. Um, came across your resume, impressed by experience, like, yeah, impressed by your experience.

Okay, and you don't like impressed by experience?

I don't like it. Yeah, it's just overused.

Don't like. What do you think?

Yeah, okay.

Yeah, again, I know we are being on a soapbox here, but you can write your own feedbacks here. This is not— yeah, you can write whatever you like.

But this is, this is how it looks like when people— you're like, okay, what is looking at your data? What does it mean? This is what it means. Like, this is actually what it looks like. It looks mundane, but if you— I promise you, if you do this on your own data for your AI product, you're going to, in just half an hour or so, you will come away with really significant, you know, takeaways.

I guess it's fine. I don't have as high bar for you as you do. I know you don't like impressed by.

Yeah, it could be. Yeah.

Another thing I would like to do is expand. I don't like—

one question is, why do we, uh, why do we start with a manual audit instead of letting an LLM do an initial feedback and adjusting it as needed?

Don't do that.

Do not use that.

Very good question to do.

Don't jump straight into tools. Like you need to put your own eyes on the data because the LLM is just gonna lead you astray. Honestly, you're gonna be grounded in what that LLM is saying. Um, yeah. And you need to kind of get an idea yourself.

Yeah, I see this a lot. It's people ask the LLM to judge it without having read at least 10 outputs, and then they look at the LLM feedback and they're like, yeah, I totally agree with it. But if they were to do this independently of the LLM, as we are doing, the LLM is not coming up with, maybe it'll come up with this irrelevance thing, but it's not going to get our vibes down. And it's totally worth spending the hour annotating. You're building a whole company, a whole product out of this. It's worth looking at your data, 100%. Now after you do this, and we'll talk about this in our course later of how you can use LLM as judge to kind of scale up the vibes that you're coding. But when you start out, you gotta look at your data. I think this is just too verbose.

Yep, it's very verbose. Am I reading that?

I don't even want to read it. I'm gonna move on. But definitely it's kind of verbose. Yeah, and also I don't like the fact that it says the budget twice.

Yeah, you don't have to nitpick it. Too much, like in the feedback, if there's one thing that is just like the smoking gun of why it's wrong, like if it's too verbose, that's totally fine just to do that. You don't have to like identify every single problem.

Oh, okay. This one is super interesting to me because it, it shows literally one of the phenomena that I study as a researcher, which is how your criteria drifts over time. Previously I said I did not like the fact that it would mention irrelevant things. However, I, I feel like if that were phrased in the right way, like a PS, I'm also interested in blacksmithing, and say I was super into that, I would enjoy the email. I don't know, maybe Hamel Husain, you have a really high bar for anything else, so you wouldn't.

I think it's interesting. Yeah, the PS, I also enjoy one, that's a little bit dangerous for an AI to write. But that's a good, yeah, it's like an interesting direction, honestly. Yeah, I mean, it could maybe pull that from like the profile of the recruiter or the hiring manager. But that's the kind of, yeah, that's the kind of thinking that you might.

Even though it's irrelevant, but the reason, but I only like it mentioned explicitly as something, as a hobby unrelated.

And I highly recommend you don't have to type in the feedback. Me, Shara's going to type the feedback right now, but use like a voice transcription app like Mac Whisper, Whisper Flow, whatever. Over time, it really reduces the strain of, you know, providing the feedback.

This, um, well, okay, so another thing that we would vibe code again on this if we had more time is so our feedback box should not be like one line anyways.

So, so, um, we have a bunch of stuff. Yeah, so like one question is, do you balance affirmative and critical feedback? I usually like in the very beginning, I actually focus on critical feedback. If it's good, I just skip it. Later on, you want to maybe get affirmative feedback if you're trying to fashion an LLM as a judge. But that's, you know, that's beyond this like foundation level.

So yeah, my answer to that is in the very early stages, I would start with critical feedback. Start with whatever is you feel like is the better direction you would tell a human. So if the LLM is like doing a lot of things well that you didn't explicitly detail in your prompt and you wanna preserve that, that's affirmative feedback. But if like most of your feedback when you look at the data is the vibes are off, you can focus on critical stuff. Alright, so I know we don't have that much time. We're probably gonna go 5 minutes over. It's 11:55, but I wanna move on to the next part, which is how we can kind of take this and turn this into eval criteria. So I've done this with like, you can export your annotations if you would like, but for the sake of time, I'm just going to copy paste it into ChatGPT. So a good prompt that might be here is, I've been Coding my data. So coding is a term used in qualitative research, really what we're doing here, kind of systematically taking notes, open-ended notes on outputs. So I'll say coding my data to identify things I don't like in it. Here are some of the feedbacks I've given. Note that this is in order. Later feedbacks represent my updated thinking. And then you could really ask it things like, what are the 3 most common failure modes?

You can do that, or you can just tell it to give you categories and you can apply them, you know. But you can do that too. I mean, whatever. The idea is like, look, you don't like, so you have written all these notes, right? And you can get AI to categorize those notes into, you know, failure modes that then you can analyze and then you can count. Okay. So like, you know, you see, you see these like sort of ones that is writing right now, like redundancy, verbosity. Misrepresenting hobbies or interests, whatever. And then, you know, you have to clean it up a little bit. Now you could, like, what you would do more programmatically is have it generate like a list of categories. And then you would ask the LLM to then go back with this list and then categorize those notes. Okay, and then you would basically classify those notes into these various categories. You know, one note can have multiple categories. Um, okay.

Yeah, no, it's good. Yeah, for issues of time, we won't do that today. I mean, it's a 50-minute lecture, but sign up for our course or go to our future Lightning lessons, and we're going to kind of keep moving on this.

Yeah, so I mean, like, you can categorize these notes, and we don't have time to show that, but basically you can categorize each of those notes and have a pivot table even and just count, like, okay, what errors are occurring the most. And you'll be very surprised, you'll see a lot of patterns in your data. We already saw some patterns in the data we're looking at, and that will help you prioritize, like, what do you work on, you know, like what you should go fix, you know. Should you go— do you have a retrieval problem? Do you have, you know, is it clearly a prompting issue for X, Y, and Z? And then it also helps you focus on what to measure. You know, instead of all these generic metrics, helpfulness score, conciseness score, toxicity score, all this stuff, which is frankly bullshit, and it hurts you if you applied those generic metrics most of the time. It's like now you have concrete things you know are wrong, and then you can sort of go after that. Um, okay, so takeaways from this lesson. One, you should build your own data annotation tool. Shreya did it in like 5 minutes in front of you. Just to drive home the point that you can do it. Number 2, write freeform notes and categorize them. But you know, don't try to categorize too early. Just write your notes, observe the data. Number 3, you're like, okay, how many of these should I do? So one thing that Shreya articulates, which I really like, is keep doing it until you feel like you aren't learning anything new. And number 4, like, this is the foundation of evals. This is at the very ground level before you do anything else, before you think about evals, measurements, all this rich experiments, whatever, you need to do this step. If you don't do this step, then your eval process is not going to work. And number 5 is this is the— you'll get immense value out of this part. You can't skip it.

Yeah. We had a question around what if you had a really large dataset, is there a certain percentage we should annotate, or we just try and improve? Hugo mentioned look at 20 new datasets. Everyone has their own magic number. For me, I just like to do 30. But sometimes, I will stop if I feel like I'm not learning anything new. I think that's the real stopping criteria. If you feel like, you know, when you've looked at 5 more examples, 10 more examples and you didn't find any failure modes that you had already observed before, you can call it and then iterate.

And, Shreya, I just said 20 because a number is forced. And I honestly, you look at 20, you'll probably then look at 30, and then you'll be like, oh, this is interesting, and I want to look at more. And you'll get addicted to it. And if you don't, maybe you want to build other things as well. If you don't enjoy looking at data, maybe you're in the wrong game.

Yeah, yeah, looking at 20 is like a mind— like a Jedi mind trick to like just begin. That's what we want you to do.

And then what's the step after categorizing emails? Go back to prompt engineering? Yeah, we'll talk about that in our course.

You—

if you're really early, if you haven't done much prompt engineering, um, if you've seen so many failure modes already in 20 examples, yes, definitely go back to prompt engineering. You don't want to ship something where like in every example output you see the vibes are off. Go back to prompt engineering, and then, or you could try to write LLM as judge prompts around each eval, which we'll talk about how to do later, and then run that on more examples. Yeah. All right, in the interest of time, we'll stop here. I think thanks everyone for coming. It was super successful for a first lightning lesson. I mean, tell all your friends we're teaching a course. We'll hopefully do a couple of more even before the course. And maybe we'll take a question before, before dropping off. So let's see.