How To Build AI Evals

Welcome everybody. Today we have Lucas. Lucas was a student in the evals course. He then went on to implementing evals in company, in the place that he works, and has some really cool use cases like practical examples and what he did, trade-offs he made, the challenges he faced. And I love bringing these examples back.

Hi Hamel Husain. It's a real pleasure to be here with all. And I wanted to start from the end to the place where we are right now because I think it helps to tell the story backwards and to show where we are and then how we got here. And the main message that I want to bring is that it was really hard at the beginning. Today we can run evals without having to open tickets to extract data or to wait someone to run a script. We use Cloud Cowork, talk to our eval project, and goes with Hamel Husain's skills, and it's really, really helpful. So I wanted to, to open my, my Cloud Code Work and show a little bit of a session that I prepared to give an example of what we've been doing. Prepared this yesterday just to show you a little bit of what I've been doing. And here, let's say I'm from our pedagogical team and I start saying, okay, let's keep working on evals based on your context. What should be our next step? And I'm using like this because it is pointing to my system file where I have all my evals history and my CloudMD saying where I am and all the kind of memory that I need to, to do that. So it looks into my memory files, it says it's a well-organized project, thank you, and it starts to see that there are some criteria that we've been working on, like failure modes, that we've not been able to get to a good judge because we are trying to calibrate these judges using human-labeled data and we measure true positive rates and true negative rates to assess if we got there. And for these criteria, they have been proving very challenging for us because the data that we have been labeled, they've been labeled by different annotators and they disagree between themselves. So it's been really hard to make a judge that can extract this rationale from those labels and have something that is working. So I asked him, okay, what are my next candidates? Like, what are other failure modes that I can start working on? And it gives me this one, which tries to look at the activities that the lesson plan is providing and saying that this activity has a very specific goal, or if like it goes all around and tries to touch in different kinds of part of the curriculum and it's not very focused. So I say, okay, let's start working on this one. That's a great idea. I want to take a look at the proper false positives and false negatives that we get from the last judgment, the last analysis. And see what we can do. So it brings a lot of ideas on and hypothesis on what could be happening. So it says maybe it's because Activity touches the content but treats it shallow, because we are developing the wrong idea, because the object in Activities tries to cover everything and nothing gets depth. So it gives some hypothesis on why this is happening based on the number of false positives it found and gives an idea of how can we create a new prompt for the judge on how to do that, how to make the judgment better. So I say, okay, let's do this, but please follow the skill Write Judge Prompt, which carries the good practices on how to do that. And by loading the skill, it realizes that we've been doing it wrong because we are using training data inside the the split of the dataset that we've been using to assess how good is the judge. So it helps me clean that and it runs it again. And I ask, okay, thank you. And now we use the validate evaluator skill to say, to measure how good this judge is. And then it goes, it runs and it says, okay, this is the fourth version that we are trying. It's deleted because we fixed the training data inside the dev data, but we still didn't get to a good TNR. In fact, we got a worse result than we had. So this new prompt version is not as good as we thought. And by doing that, I finally have like a report based on the prompts that I have and all the things that I tried. And this is what I use, or this is what our education team uses when analyzing the judge and trying to fix the prompt by themselves or trying to induce the Claude to make better choices. This is just where we are. But I think the main reason why I wanted to have this conversation with you is to show how we got here. I would just pause to see if you have any comments, Hamel.

Do you mainly in your workflow, do you work in this like Claude projects? Do you have any custom tools you've created for yourself downstream to like help yourself annotate or fix the prompt or just curious like what your workflow looks like?

So my education team uses Claude Co-Work the way I just showed. Myself, I prefer the CLI Claude Code version of it. Because I think it's more flexible for things I do. By the beginning, and I will talk about that, we created a, a way to access the data at LangFuse, the observability tool that we use, because it didn't have an MCP by that time. So we just wrapped the API into a SKU to do that, but they just released an MCP, I think, 1 or 2 months ago. And for the interface, I learned in your course to build a custom review interface for our annotators. So you were doing that, but then with your SKU, we started to use the /create-annotator-review-interface because it already carries all the good practices that we learned at the course. So we started to use that as well. And even my education team, like when they are just annotating things that were in an Excel file, they create for themselves like a quick review interface. They annotate things, they export to a CSV file, and they continue from there just so they don't use Excel anymore. So I can pass all the conversations here for our solution. And when I'm testing it, I can for example, annotate the data here. So I can give a thumbs up or thumbs down to this answer, and I can annotate here if I like this answer, if I didn't like it. And when I do that, this goes through Inkfuse as well as a note. So we created a custom interface that we can look at all this, all these messages that we got from the chatbot, and it annotates as well. And this is really useful, like, for testing and debugging and things like that. So when someone new gets to Novoskala, they get like our marketplace for SKUs. So we have a marketplace in Novoskola. This marketplace has here, Novoskola Marketplace has 3 different plugins. The plugins they bring together SKUs and connectors, right? So the marketing team has one for like voice and for creating notifications in WhatsApp and things like that. The product team has one, like we have one for governance that helps us across the company. And when someone gets into the team, they just get the marketplace and they can have all the plugins and skills and connectors that everyone is using. And it's well documented, so they know what to do and how to use each one of them. And we also point to Hamel Husain's plugin on evals. So it has all the evals here, the skills here, and we mainly use the ones that are meant to write judge prompts or validate an evaluator or abuse the review interface. We don't actually use error analysis. We wanted to do this ourselves. We don't actually use generate synthetic data. We have plenty of production data to use. So we stick with the other ones.

I like it. So you kind of distribute it through Claude, like your teams, you're like a team plan, sort of like you push it out through that.

Yeah. So they can have access to the plugins that are like common or part of everyone's job. Right. It was not always like that. The first round was a spreadsheet, a very big one and a very messy one. I can just show you how it looked like at the beginning as well. So the beginning, like, this is extracted from our very first attempt on doing evals. So we have like all the lesson plans that were assessed, evaluated, all the failure modes that were not actually failure modes. These were like criteria on a rubric that we wanted to look into. Later on, we started to look at more failure modes than fixed criteria. We started to do like that. And then for each failure mode, we started to do like axial coding. So there are like the most common problems in each one of the criteria, how frequent they were. And, and this happened to us to have conversations with the engineering team. So they say, okay, so we have this problem, how can we fix that? Uh, and then we will go like into a brainstorm session and trying to understand. This is like a title of the lesson plan, is too generic. Are too imprecise and does not correspond to the ID of the lesson. So the title is not connected to the lesson plan itself. Then the other error, like the ones, for example, this one, usually, ideally, a lesson plan can have one learning objective, one learning goal, and it was happening a lot that the AI was giving like two or more learning goals for each lesson plan. And this one is really easy to fix. So this one doesn't require an eval, right? So we understood that this happened, we went back, we fixed that, and it didn't happen again. Some of them, they were still haunting us until today. So for example—

And how automated is this sheet? Like, do you have like a workflow that produces a report for you that you don't have to, you know, collect all these numbers?

So this one was the very first one. So this was not automated at all, but this was like an engineer extracting a CSV file from our Postgres database and we working on it. So the best I could do was to download this as an Excel, not a Google spreadsheet, and open Cloud just side by side of it and working with Cloud on the side of the spreadsheet. But that was the, the best we could do. Like, this was totally manual. And Hamel Husain, one thing that we didn't do as told us to do, having the benevolent dictator, like one expert analyzing the data. I don't know why we chose to have two. That was a real headache because they, they didn't agree between themselves. So we measured how much they agreed between themselves in each one of those criteria, and Many of them, they disagreed more than if I was flipping a coin, right? So if I was flipping a coin, the coin would agree more often than those guys would agree between themselves.

Yeah. This is like, they were like negative. They very negatively correlated in some, some of these. And then how did you resolve that?

So this was a sign for us that the rubric that we were using wasn't clear because they were looking at the same criteria and they were assessing different things. So we went back to our pedagogical team and said, okay, we have to make this more clear because if you want a machine to understand that, I can't have humans disagreeing as often. Like this has to be very specific and very clear. So two people, different people looking at it, they could agree like if it's a pass or fail. So we went there, went back and we would redo the rubric and relabel the dataset. This took a long time. So every time we would run a round like that, it was like a month of work and sometimes 2 months depending on how complex the ideas were. And so it's a good thing now that we could. And to get here, there were 3 main barriers that I think were in the way. The first one was a technical barrier, like actually putting data, joining computer of raw conversations is not something that even myself have fluency on doing that or my pedagogical team. So this was the first barrier that we wanted to overcome. The second one was actually understanding what what does it take to run Evolvs? So there is a process, a systematic process, a disciplined way to do that. And we, we didn't know that, or I knew, but many of the people in my team didn't. So this was the second barrier. And the third one was actually the barrier that matters, was actually the barrier that I wanted to be focused on, was looking at a plan and saying, this is good, this is good, this is high quality, I can trust a machine that's generating that. That's where the judgment of my team lands. And that's where Noviscall is unique when we think of other solutions that is doing that. I wanted to be focused on the third barrier, but the first and the second one were in my way. The first thing to go a little bit deeper, like the first barrier is around data and getting the full traces, understanding multi-turn back and forth conversations, like which messages are calling tools, which are not, what kind of retrievals we are getting from our RAG, what kind of context is being put in. All this was really hard to navigate. This was the first barrier. And the second one was around methodology, like performing error analysis, doing open codes, axial coding, building the judge, selecting the data to label, labeling it, calibrating the judge compared to human analyzing. Like there are many, many steps in how to do that. And there's a trap hiding in each one of them. So like splitting data into 3 different splits. So you can have like training dataset, dev dataset, calibrating judge when they're saying that you can't use unbalanced datasets to calibrate it properly. There are plenty of things that we are falling into, and the course helped us to, to go through those mistakes. I was carrying that knowledge, and I wanted more people to, to have access to that and to finally get to the pedagogical judgment and being able to answer questions like, is this lesson plan actually engaging students? It is motivating enough to pull them in? Does it mobilize the right knowledge, the thinking process? That's the, that's the questions that I wanted to get myself into. So So we came up with ways to overcome the first barriers so we could focus on this last one. And the first solution, the first layer was setting up observability. So everything started with plugging LangFuse and being able to understand the plumbing of the data, like what was happening behind the curtains. Like this was really, really important for us. The second one was accessing this data through a natural language assistant, which is Claude GPT. It helped to make this data accessible for more people, not just our engineers. And the third thing that we did, and it came, I don't know, Hamel Husain, is it 6 months already or a little bit more that you published these skills? When we started using these skills, because then the method was packaged and then I could help my team just telling them how to do, but they had the skills to bootstrap their work and they could have like guidelines on how to work. Yeah, we started with observability. This is a screenshot from Langfield, so I can I'll also show it to you. This is LangFuse. In here you can get traces of like everything that's happening within the, your pipeline, right? So you know agent was called, like which tool was called, what it happened, how long does it took. You can see like what kind of text was brought in, what kind of answer was given. So this is a goldmine for us to understand what's happening behind the scenes. And the very first step was to set this up. And second one was to plug in LangFuse to Claude. At the beginning, as I said, we had to wrap the API. Within SQ, but now there's an MCP for that and it's much simpler to use and it just allows you to, to look at your data through a natural language assistant. And we also use LangFuse to store our test datasets. So all the test datasets that we've been, we've been using, they're here for all those .trainer modes that I mentioned. And here you can see like how many passes or experiments we run on them. This is the dataset itself, the expected output, the metadata. So it's really helpful for us to run the review interface when we are doing labeling, but also through that interface that I showed earlier when we annotate, like when we are debugging within our platform and we say this was a good answer, this is not. And so this goes back to LakeFuse. And this is like very important because when we are trying to hill climb a problem that we're facing, if you don't have this annotation, like if you don't have the reasoning behind the pass or fail, there's not much what the AI can do. Like, the AI needs the reasoning so it can understand, like, this is a false positive because of that, this is a false negative because of that, I can try to change this, or like, this is a borderline, or this is a clear pass, this is a clear fail. The only way that the AI can make those kind of judgments is if it has access to your reasoning through the annotated data. So that's why this is really important for us.

In the next cohort, we're going to be using LangFuse the whole time, and Lucas, you're invited as well as a prior student, because the prior students get access to all future cohorts. But it can be very opinionated on a stack. So to show people like an actual implementation of the whole thing and like the observability layer we're using is LangFuse.

So I have an opinion on that, Emil. The thing is we use it to get observability, like to get our traces and to store our datasets and prompts. I don't actually use that to run the experiments because I find it really hard to measure TPR and TNR and to do all the calibration within LangFuse. I haven't been able to do that. Basically, I'm using it as a database. Exactly. I organized it one, but my impression is that they assume your judge is calibrated, so they go straight to give you a score. And I don't want a score. I want to calibrate this judge before applying it to get a score. I'm having a hard time to find the right tool that I can do the whole process, but I've been very happy to find Claude and to do that within Claude, and it's working now. So I mentioned that there were 3 layers that lower the walls, the first one being observability, the second one being this natural language assistant and plugging Claude to LangFuse. And the third one is the skill itself. So having that, they kind of package the knowledge of how to run evals. So like this, let's say I know I have to split data to train my judge, but maybe I don't remember if I have to split 10, 40, and 50 or 15, 45, and 45. And this kind of knowledge is embedded in the skills. So it makes that easier for everyone. So this skill helps my team to bootstrap their work. So when we do that, like when we have an observability tool, when we have a way to talk to it, and when we package Process itself, the methodology, what remains is exactly what matters. Because now I can spend more time judging the product itself, right? I can judge the lesson plans. I can judge how the product is performing. And there are no tools that are going to abstract that away. Like this is my job, this is the job of my team to do, but all the other stuff made, allowed me to have much more time to do that. So the methodology, the barrier of using the methodology lowers, but it doesn't vanish. And I still have to work on my pedagogical judgment to be able to, to do this kind of, this kind of work. And this is where we are from that set of criteria that I showed before. We've been able to calibrate 4 judges. To production. So these are not perfect TPR and TNRs, but they're good enough for the product and for the failure modes that we had. Those we still have to go back, label more data, work on the criteria so we can actually be able to raise those TPRs and TNRs to a bar that I can trust these judges. So I still haven't found good judges for these failure modes, but I'm happy with these ones. Does it make sense, Hamel Husain?

Yeah, it's really good. I'm glad— I like how you have this, like, you went through the whole process, and like, for any of your judges, like, you know, you've calibrated to human because most people that have LLM as a judge, they just say, oh, the LLM judge is good. Okay. But you actually showed so many examples where the LLM judge is not good and you had to work on it. And then you have these left over here that you still need to work on. Yeah, I think that's really good. And then in this next cohort, what we're going to do is we're going to show you how to use AI to help you hill climbing. You're already doing that as you showed earlier. But yeah, we're going to go into that as well. Like when you have these low scores, like, okay, like how do you get out of that? And again, like, you have the right intuition of like using AI to do it, and it's good that you looked at it first before having AI just do it for you, because if you just had AI do it for you right off the bat, then it wouldn't have done the right thing. Like, you have an intuition, like you're giving it some steering and direction, and that's the right way to go about it.

And those 4 judges that we have, they play 2 roles. The first one is like, they are like a CI gate for us. So every time you have a new PR, we run the judges. And we look at like, is this new version of the chatbot better than the previous one, or does it go back in one of those criteria? So that's kind of a guardian for us. And the second way we use the judges, they are in production. So every day, I think it's 2% of our production data, I have a sample of like what's happening, how many plans were generated. And those lesson plans, they go through all the 4 judges. So I have like every day how my lesson plans are performing against those judges. So you can see here that my judge number 2, the, the one that says that lesson that the teacher asked for is correlated to the curriculum that was brought into the lesson plan, is the one that is giving us the most trouble. So we still haven't been able to generate lesson plans that are very well aligned to what teachers are asking for. And this is huge because if the curriculums are aligned, Everything else is not, so everything else falls apart. And that's my first candidate to try to hill climb using AI when we learn to do it, how to do it. The other judges, they more or less, not just this green one, which is the learning goals, the learning objectives, goes back and forth. So I have to look into that as well. So this gives an idea of how my, my product is performing day over day. It gives me like, am I drifting? Should I take care of some aspects that I wasn't watching before. So it gives me this kind of a safety net to know what I'm looking at.

Have you caught anything with these evals that, you know, have they saved you from any headaches or any kind of, have they informed anything as far as like ongoing development? Like have you opened this dashboard and seen like, oh wow, something went wrong and you had to do something about it?

This one, not really, because they are already in production. So we are familiar with these errors. Like we know they're happening. So At this stage, it wasn't a surprise for us, but in developing, like during development, I think having the evals, looking at PRs every time we go to a new version, it has saved us from going back in one of those, on those criteria. It even helped us in anticipating, because for example, we've been mostly generating lesson plans for common subjects like Portuguese and math and geography, history, and this kind of But Brazilian curriculum also covers, for example, computing, computational thinking. And this is something that rewards generated lesson plans. So the first time try to do that, the evals, they were breaking because you haven't trained any evals to assess computational thinking lesson plans. So, okay, you have to go back and create the evals inside of it again. And right now, before this conversation, I was working on a refactoring of the agent. I'm trying to use Anthropic managed agents. To see how it will perform against the, the one that we have, which uses lane graph and it's a more artisanal one. And having the eval set gave me a lot of speed because every time I developed something, you can have Claude knowing what good looks like. So if Claude knows what, what good looks like, it helps me get there because every time we change something, tests against the evals I already have. So this is really helpful to give me also speed while you're developing new stuff.

That makes a lot of sense. Like when we constantly find is the part that's the most useful about evals is like the journey of making the evals. And when you have the evals at the end, like of course you have the test and that's well and good, but it's like forcing yourself to measure things along the way is like what actually makes it better. So yeah, it's a very common pattern we see. So yeah, not surprised at all.

So I guess it's a good thing that we weren't surprised by the production evals.

If you've been doing evals correctly, you know, you're looking at it so much like along the way, you know, because you're doing error analysis and then you're also looking at the LLM judge performance and making sure it's aligned and you get like super familiar with what's going on and you're fixing everything along the way. You're not just waiting till the very end. And so this is really good. This is what I like to see. This is good.

And I think another lesson learned when we started to do it, we like we had the rubrics, like the pedagogical team, they have like a ruler on how a lesson plan should look like. And then we're labeling a lot of data and assessing a lot of data. On things that weren't a problem at all. So getting the title right, like, was a very small problem that we could fix right away. And I didn't need an eval for that. I didn't even label data for that. And we spent a lot of time labeling data that we didn't need. So after the course, we said, okay, let's just label data on things that are failure modes, prevalent failure modes, and that we wanted to hill climb. And I think that made us saving a lot of resources as well.

I love to hear that. That's the thing we emphasize most When we teach evals, it's like, hey, don't eval everything you can think of. Eval, do error analysis first so you can prioritize what's important because ultimately evals are not free. So it's really great. You're an exemplary, you know, of a student, like doing the right thing.

So, and just to close it here, I think the takeaways for us, first one, like take care of the observability, instrument the system, store what it does. This is the very first beginning. If you don't have that, there's not much you can do. For us to get to bring more people to the process was really helpful to have a cloud cohort. Working with that data. So making sure you can use an MCP or you have the skills to make more people in your team being able to read from the observability tool. I don't know, Hamel, I haven't got the opportunity to see the lectures you're making on the opinions you have on your skills after you created them. I still think they are very useful, at least half of them, but I think you have a different opinion.

Just to give you some context, the problem is that if you use them without any context, You can get lost very fast. And this is the case with lots of skills. And these skills are meant to be customized. You should be really careful about using any skills off the shelf because ultimately it is somebody else's prompt and it's just like my prompt and like, how good is that going to be for your product? So you really need to customize it. So what I have built later on is like an MCP, like it retrieves over all of the course materials and like all the videos and all the office hours and everything. It's a little bit higher fidelity than just like my prompt. And so like, yeah, I was just saying there's another talk that somebody linked where I'm just telling people, be skeptical of skills, be skeptical of these skills. Maybe it's an okay starting point, but Lucas is able to drive it very well because he understands how to use the skill. He has the skills himself and then so that he can then leverage the skills. So that's what we're talking about.

And I agree, like if I just would give them away to my team, they wouldn't know what to do. But after the course, I went back and I redid your course with them. So the 8 classes using your slides and everything. So they knew also like what to expect from the skills. And I think that was really helpful as a starting point. So, because it's really hard to remember all the details and the skills helps you to do that. And least of all, but most important is keep the domain expert at the center of the process. Ultimately, what we're trying to do is about quality and trusting the product that you're building evolves and removing those barriers is a very nice way to approach and to have people that know your product being part of the process. And I think that's the best thing you can do while using the evals.

All right, great. I think it's a good time to open up for questions.

I wanted to ask you, can you maybe share with us the impact of how these evals have impacted the product and the quality of the product and some user experience. Can you see like before and after? At the beginning, the chatbot was really, let's say, crude. It gave you answers that used to mislead teachers into trusting that a specific curriculum skill was right. It didn't even exist. So let's say a teacher asks for a lesson on fractions for 5th grade and it would come up with a curriculum skill, which like a code for the curriculum ability that we were working on. That doesn't even exist. And, and teachers would trust that because they are not familiar as well with the codes and they are not like opening the curriculum to cross-check and do like that and things like that. And that was really disturbing for us because we have a very known brand in Brazil and teachers trust Novo Escola, and we didn't want to mislead them. So having the evals helped us to understand like where we were making mistakes and directing resources to fix it, even because we are a very small team, like we are 290 people in total, only 6 people in technology, and we have to choose very carefully where we bring our energy and our resources. And the evals also helped us to direct that kind of effort to make the product better. So over the year, like this is one year that we've been doing that, the product is totally different, like the new, it's in a new stage, but I can't compare it because since we analyzing failure modes, they change over time, right? So at the beginning, the failure modes that were haunting us, they're not the same that are haunting us right now. How often, like, how often are you checking? Are the judges running on your production data? I think you mentioned daily in one of your examples when you were showing the dashboard, but like what percentage of usage goes through the judges? And someone had mentioned like, how do you manage costs? So like sampling rate is my question, but then also like which model to use and those other things? We started by sampling, I think it was 2% of the production data daily. We do that in batch mode, so we don't charge tokens so highly expensive. And doesn't matter if it runs at midnight or 2 AM, it's the same for me. We do that in batch. And what happened is like 2% was really very few lesson plans. And I didn't have coverage, for example, for an amount of lesson plans in third grade, in fourth grade, or in math, in Portuguese. So what happened was that the evals, they were very erratic. So sometimes it was really good, sometimes really bad. And I was like, okay, so this is not stable enough. And I started to play with the sampling. So every day I was like trying to increase it a little bit and to see when my data was starting to get a little bit more stable, which means for me that I had like a sample that was big enough to extract some information from it. And I think it was around 80% in my case from the God. And the second question about the model. So when we did the calibration for the evals, we were using GPT-5.4, I think. We wanted a very strong model to evaluate. You don't use as strong model to generate lesson plans. You're using 5.2 mini, I think, but we wanted the 5.4 to judge. And since we calibrated the judge with that specific model, we set that. So every time we run again the judge, we use exactly the same model. So we can say that the TPR and TNR still applies.

And one thing I'll add to that is if you have labeled data by virtue of, you know, being able to calculate this TPR, TNR, if you basically, if you have an eval harness, you can try different models to see like how good the alignment is with the labels. And you can always see like, what is the optimal model to use for your trade-offs? Essentially.

I'm still trying to figure out the balance between using two experts trying to take the same trace or the benevolent dictator, just because in education everything is so nuanced and subtle that I think that having more people looking at the same data would be helpful. But then there's another step that is getting these people to agree between themselves and it makes things really hard. So that's one thing that I'm still trying to figure out the trade-off.

It's pretty useful to kind of debug that process actually? Like, okay, if you have two annotators and they're disagreeing a lot, you mentioned earlier that in one of the examples, the rubric wasn't specific enough, for example. But if we zoom out a little bit, okay, like, why isn't the rubric specific enough? Like, how detailed of a rubric does the person need? Do they have good product sense themselves? If it— is it them writing the rubric? If it's someone else writing the rubric, then potentially that is the person that needs to be doing the labeling. And depending on your organization, that might not be feasible. Maybe there's a reason it's set up this way. So it's hard for me to get a blanket answer, but it is worth to try really hard to get to this benevolent dictator type person because like 2 people is a lot of friction, tremendous friction. It's a slow— it's gonna slow it down. So it's worth like digging into that a bit and seeing like, oh, okay, like, like who is the person that actually knows if it's good?

One question I have is that when you're working in a team, and you're fast moving and the— you're revving that agent, you know, maybe daily.

How careful do you need to be in the way you annotate against different versions of that agent? And how do you consider those evals when submitting maybe hourly and the agent's getting better very quickly?

How do you align the evals against the current state of the agent?

So one thing that's important to keep in mind is like, don't get too carried away with evals. And we teach this a lot. It's like, if you see a problem, just go fix it. If it's trivial and you know how to fix it, just go fix it. Like you don't want an eval for everything. Evals are expensive and you should like pick and choose very carefully, like what evals to write. And one dimension is, and like, it sounds like from you is like, if you find an error that's very persistent and is not going away, you don't really know how to fix it. That is a great sign. You need an eval because that eval is going to provide a lot of value. Love to see you in the next one. Thank you so much.

Thank you. Bye-bye.