AI Evals For Engineers: Course Preview (Chapters 1-3 of 8)

Hey guys, how's everyone doing?

Wait, I have a question. How can I check it on the YouTube just to make sure it's actually good?

Yeah, yeah, yeah. Let me give you the link. Um, can I—

can you share a link to book please? Wow, people are already asking for links to the book.

Take our course, take our course. Yeah, I'm still writing it. That was a workshopping session, so this is our first time doing a live workshopping session of this. So I am slightly nervous that Eugene is going to be like, this was horrible, never ever, whatever. Um, but yeah, I'll maybe go over high level we're gonna do. Um, so this is— we know that you all don't have the book, um, but what we did was we told Eugene, please read the first 3 chapters, let us know what sticks, what doesn't stick, and we have a bunch of targeted questions. You know, Eugene is a very senior, possibly even higher title than very senior applied scientist. Oh, did we lose Hamel Husain? Oh no, now he's back. You're back.

All right.

No, you're good. I was introing Eugene. So Eugene is a good friend of ours, very senior applied scientist at Amazon. And what I admire most about Eugene is how willing and open he is to share his thoughts on things that he himself is also learning with the community. He has a very extensive blog. I have learned a lot from it. Everything from— I started following Eugene back when he was writing about recommender systems to now when he's writing about LLM as a judge to evaluations. You can't improve what you can't measure. It's a great journey to follow, so I highly recommend checking out his blog. Today I have a bunch of questions prepared for Eugene around his experience with some of the failure modes and some of the frameworks that we're presenting in the first 3 chapters. Maybe I'll start with a very brief high-level overview of what the 3 chapters are about. First, we talk about what is evals and why evals are so hard, at least in our experience. Why is it different from traditional ML or MLOps, traditional software, software engineering. Then we go into a little bit of thinking about how to work with an LLM. So how does an LLM work as a black box? How do we reason about its strengths and weaknesses? How do we use it effectively? And then kind of some getting our feet wet into thinking about what does an eval mean when we're building applications? What kinds of metrics would we want? And how might we get some of the building blocks to compute those metrics, for example, labels? And the third chapter that Eugene also skimmed over is the very first stage of the evaluation lifecycle, which maybe we'll go through in the diagram. First you do an error analysis, analyze, then you measure things at scale, then you improve based on your measurements. So Eugene and we will discuss kind of the process of error analysis from looking at your data to organizing that into structured failure modes and iterating on those over time. So without further ado, I'm gonna move to my questions and I'm gonna hit Eugene with the first one. Eugene, you ready?

Yeah, go for it.

Okay. So thanks for taking your time to read and review, and you're the first one doing this. So I'm curious from your perspective at Amazon, what are some of the most pressing challenges in LLM evals that you see on a day-to-day basis? And do you feel like these chapters touch on these well so far, and what parts of them might you resonate with?

Yeah, well, thank you, Shreya. It's a huge honor to be one of the first few to read the book. Um, it really resonates with me. I think the main challenge with evals is that people just don't do them. Um, they don't know how to do them. And when you look at a lot of the biggest issues that come up that gets you negative press or has customer impact. Some of the biggest issues is just not having evals to know how to even write the prompt well. Some of the biggest issues is just not having evals to know that the model cannot actually do the thing. Well, the current generation of model can't do the thing, and maybe what you're trying to reach for is way too far ahead. You actually are blind on that. And then the other thing is that maybe you have evals on a very small surface area, like the common stuff. But when you get to the tail, especially on something that has very large scale, you actually don't have insight. This doesn't mean that you have to have very comprehensive evals, but you probably need it to be more comprehensive than you think right now, especially if you have a wide customer impact.

Totally. I also resonate a lot with long tail and inability to specify prompts well in the first place. So why don't we go right into Chapter 1? My first question about that kind of being, you know, this— okay, I'm going to scroll through to kind of where I say where you can use evals for.

I actually— do you mind scrolling up to the gulf? The three gulfs?

Yeah, tell me about that first.

This really resonated with me. Um, I actually didn't think of it this way, and I, I think that understanding the different gulfs helps you understand where you're lacking. And it's also that even if you fully close the Gulf of Specification and the Gulf of Generalization, if you don't close the Gulf of Comprehension, it doesn't make sense.

Let's describe what the gulfs are first, if you don't mind. Somebody—

Shreya.

Shreya.

Oh no.

I know I will have communicated this well if someone else can describe it. So I'll tell you how this figure came about. I thought about what are all of the problems that I face when trying to write LLM pipelines and what are the different hats I have to wear when building good LLM pipelines. One hat is my error analysis hat. I just have to make sense of what's going on, not just in the data, but also not just in the LLM outputs, but also in the data itself. What are the different clusters of data? If you're familiar with data science, you're familiar with this problem. The other hat that I wear is how to be a good communicator because I have to write prompts, I have to specify everything in excruciating detail to the LLM for it to understand, has to be unambiguous, and this is very hard to do. Then the last thing is your traditional ML problem. I've written the pipeline, but does it generalize to my data? Does it generalize to other people's data? If I deploy it out in the wild, will it work? This is what we call the Gulf of Generalization, which is, how do you do task decomposition, fine-tuning, prompt optimization, all of these strategies that you have to invoke book, um, Putting Your ML Hat On. So I would say, yeah, these are— Gulf of Comprehension is your data science hat. Gulf of Specification is your good communicator, good writer hat. And then Generalization is you're good at machine learning, um, and software engineering.

So, and so can you go into a little bit more, if you don't mind? Like, okay, I don't, uh, I haven't memorized it just yet. I really like this articulation also. So like Gulf of Generalization is, for example, you have correctly specified everything in the LLM and, you know, you have a good prompt, but the LLM is still failing to do what you want. And that's like a gulf that you have to cross. Um, you know, okay. And then Gulf of Specification is like, is it just, correct me if I'm wrong, that's when, you know, you haven't refined the prompt enough. You haven't specified your intention correctly, you have to do that iterative cycle. What's, what's comprehension again?

Comprehension is making sense of the data. So also, we could change these names. I'm not tied to the names, but, um, what we say when you say look at the data, uh, the artifact of that is like some knowledge in your head of what your data is about and what your failure modes are about. Um, that I see.

So you have to have like comprehension of what you want to be able to specify it.

Yes. That's what we say of, there's this iterative cycle. You write a prompt, you look at the outputs, then you learn something about what's going on. That learning part is this Gulf of Comprehension.

One thing I really liked about this Gulf, and you might want to spend a little bit more time talking about it, like why you created this mental model. One anecdote that I really liked in the paper is, "Every problem that you encounter, you have to build the bridges in you." I remember that phrase. And I really like that. That helped, like, make it sticky for me, like this, this water and these islands. You talk a little bit about, like, okay, yeah, what made you want to create this mental model of, like, the cycle and these islands and stuff?

Yeah, um, short answer, I'm an HCI researcher, or I'm a CS researcher who likes doing HCI, um, and I like to study how people iterate on pipelines. And I really figured that these are the three hats— this goes back to like, these are the three hats that you wear and you're interchanging between when you're kind of developing pipelines, um, but Yeah, it's nothing more to that. I don't know.

I mean, it's very natural, Shreya, but I guess a lot of us have a lot— I can— I learned a lot from this, and I especially like that it's a tripod. You can't just have two legs without one of them. Um, and it helps me think a little bit better. It's like when someone comes to me with a problem, okay, maybe is it a problem that's not well enough, or do you not fully understand, um, your errors, or are you just not thinking about the long tail? That's how I think about specification, comprehension, and generalization.

Yes.

I also do think that there's another gulf though, which is the gulf of model capability. Um, in the sense that you could be doing all of this right, but the just the model just cannot do this right now. The current generation of models. Uh, you see that sometimes. Yeah.

That's very interesting. I always, I imagine that that is kind of part of this generalization gulf of like, okay, it's not working, but I've specified it so well. So now I have to. I have to wait for a better model or I have to fine-tune models or I need to do RAG or task decomposition or whatever it is. But it's interesting that you say this because I think a lot of people face this problem, myself included, is you don't know when you've hit the ceiling.

Yeah, it's hard.

It's hard. It's like, how do you know that it just— the model won't work for this task. Maybe if you decompose the task better or if you fine-tune the model, maybe it'll work. And I think a lot of people— well, first, very few people get to that stage where they're able to push the ceiling. But when you do get to that stage, and it sounds like you've been there, I think I'm curious if you feel like you have any tips or strategies for one, knowing you're in that, you've pushed the boundary. And then two, what do you do? Do you wait for a new model? Do you just say, okay, I'm going to do something else?

Just to speak more about the Gulf of Generalization, how I think about it is that, okay, your model works probably 50%, 80% of the time, but there may be a few edge cases that in the long tail they just haven't taken care of. And you have a great example in the book that talks about this, about the email name extraction. That's how I think about it. So the Gulf of Generalization is that there are a couple of edge cases that you just haven't taken care of, and in the long tail that's going to happen. The Gulf of Model Capability is you try and do something and it just cannot be done. So maybe 2 years ago, if you try to summarize a video transcript, you could try to do a lot of MapReduce hacky ways, but the quality is just not quite there. The tone is not quite there. It gets, it gets, uh, it hallucinates. And you know, now this is not a problem. You can fit in the entire thing and Gemini will spit out the entire transcript for you. Um, and you can even do all kinds of modalities, right? But in the past—

Yeah, that's a relatively new thing and it's really good now.

Yeah. And, and we take it all for granted, but if, if you had to that if you were asked to solve that problem in the past, you'd be like banging your head against the wall trying all these smart hierarchical map reduce techniques. I know there's a lot of packages around there that do all that, but all of that is like obsolete for, for good or for worse, in the sense that a lot of things are just simpler now. It's just all pushed into the model. Um, but that's the gulf of model specification, whereby it doesn't work for more than 80%. It doesn't work more than 80% of the time. So it's That's how I like that.

I really like that distinction. I might steal that, quote you, put it back. That's what this is useful for. Um, yeah, so— oh, sorry, keep going.

So I just want to say that I love this. No matter what your editors tell you to do, do not cut this out. This is one of the things I highlighted in green. Um, I was, uh, this is novel, new to me, and I will apply it. Um, yeah. When I help teams triage for the errors.

Oh, that's so great. Um, cool. Hamel Husain, you had a question before and we talked over you, or no?

No, I'm fine.

Okay, so it's good.

Cool.

Um, so the question that I wanted to get to, which I already forgot, so let me put up my Google document again. Um, Yeah. Oh, yes. So regarding the Gulf of Specification, I was going to go to the Gulfs. What are some ways, concrete examples of where you've seen people get lost in the specification Gulf, going back to your statement earlier about how people just get lost in prompt writing?

So I think there are two main failure modes. And the first one I'll tell you that is the most common is like people learn about, hear about this thing called few-shot prompts and they write a single example. Um, in older models, and to some extent in the current models as well, when you provide a single example, the model overly focuses on that example. It just gives too much attention to that. And sometimes when you run a lot of samples, you will regurgitate just that sample. And a lot of times it won't return a sample, but you return output that's very swayed by that sample. So for example, imagine if you were recommending clothes or for some reason, and you provide two examples, both female fashion, both high fashion. Now when it gets to a point where we are recommending clothes for males, that's not high fashion, the model just gets swayed too much. It learns too much from the in-context learning. Interesting. And then people come to me, it's like, why is this happening? It's like, have you tried removing the example? Remove the example, it is better. But it's worse on the reason why they added it. So the reason— so therefore, if you want to add instructions, first you try to add criteria that cover the examples. If you do want to add examples, you do need to add quite a number of examples, right? Essentially, your examples need to be representative so that you do— so that when you're trying to solve the Gulf of Specification, you don't accidentally open the Gulf of Generalization.

Oh, that's such an interesting take. Yeah. So, okay, summarizing what you said, one is start out with specific criteria, and then two, it's very similar to traditional machine learning and data science, which is have a representative sample. Like, take your data, actually create a training set out of it, have a representative, have a representative sample of that in your prompt. Um, because if you don't do this data science well right? You're opening other cans of worms that you're going to try to fix that could have been fixed with your correct methodology in the first place.

Yeah. So increasingly, I'm thinking of in-context learning, just like regular machine learning. Your input samples need to be fairly representative. You definitely need a lot less now. You may not actually need entire samples like a DataFrame samples. You can even write higher-level criteria or patterns. It can do pattern matching fairly well, but it needs to be representative. So now it doesn't mean that, okay, you have to cover the entire cover all the edge cases. There will still be edge cases, but you cannot be overly specific. You have to put in care when you're writing this prompt set, future examples.

How do you feel like you can iterate on this and make progress? I know this sounds like a very vague question, but often in the stage, right, people don't have evals in the specification stage, and you kind of have to make gut decisions. On what examples to include or what not to include. Can you talk a little bit about how you approach that?

Yeah, so I think there's, I think there's two main splits. The one split is if you already have a product, okay, just look at the data, try to figure out what kind of queries, what kind of actual input traffic would come in. And then you look at that, you try to find out the main patterns and then you write based on that. Now, if you don't have data, if you're building a completely new feature, let's say a chatbot for a website, You probably have to guess what kind of queries user will come— users will come with. Um, and you know, sometimes it can be very basic, and if you have a good understanding of a product, you— and you've got to understand users, you probably would know what kind of queries will come in. And you may also need to come up with red team queries. So that's how I would do it. And maybe you just need like maybe 5 or 6 big buckets, maybe 10 or 20 questions from each of those. I think that will go very, very long.

Nice. Do you feel like there's ever such thing as too many? Examples?

No, as a data scientist, I definitely don't think that. I always— I definitely do think that more data is always good. Um, and you know, with LLMs, you can very easily analyze it, and it's just standard. I think there's also another example, another problem with specification, is that, you know, you give an example of how you write prompts, right? There's roles, etc. Another problem I see is when users write the system prompt And it's written in such a way that actually leads to a footgun. So for example, you may want your model to be very friendly. You may want your model to be trust— to, um, you might say something like, you are a trusted advisor, right? Now what happens is that the model may become too candid and frank and perhaps even judgmental in the responses. Because you, you have prompted it that you are trusted. I trust you, you should be open-minded with me. Um, what happens because we try too hard to push it in a certain direction, it actually gets pushed in the wrong direction. In this case, the advice I've given and the advice that the labs have given is, you know, just keep it simple, don't add too much of this superfluous.

Yeah, that is also interesting to me because it is very context-dependent and cultural, right? You might say something like, you're trusted, friend or advisor to somebody, and they, if they know you, they might say, okay, this means like I should handle this conversation with care, I shouldn't say things that are mean.

Exactly.

Whereas like in a completely different culture, you could be like, be open, interpret that as being open and honest and direct.

So, so let's, let's go back to our fashion example again, right? You are a trusted fashion advisor. What do you think about this outfit? And then maybe the other might respond, Eugene, come on, how old are you? Why are you thinking of wearing this? You think you're still 20? So something like that, right? You can imagine that, um, in the long tail, things like that do happen. Um, so you just do want to be careful about it. And of course, the final, uh, gulf of specification is that because we keep trying to solve the gulf of specification, we try and make it more and more specific, we actually forget to refactor our prompts. And that is, um, a lot of times when people come to me with prompts that are just not good I just cut off— I just cut it in half or like reduce the space and just works again. So I think like prompts are now like code. You do have to constantly refactor out functions. It's just that sometimes your prompt just returns many, many different scores for different tasks. You maybe want to refactor them out and then you want to simplify it as well. And because if you have evals and it works just as well, it saves you problems downstream.

Totally. I like the saying of prompts as code because that is what it is, but it's a little different because you can't statically make sense of it as well as you can make sense of code. When you read a Python function, you know exactly what the computer does if it's a small Python function, no questions asked. But sometimes you just don't know with LLM.

I agree with that. Recently I've been a little bit skeptical of agentic workflows, and if you follow my account, you probably see this. Um, but recently I've been trying it and trying to push it, and you can write a single prompt, like maybe 2 pages long, and get the LLM to just do the thing. Yeah, for hours at a time, and it will work. Now tell me, is that not an ETL pipeline? Is that not an ETL pipeline for a sample of one? Just doing all the things, doing web search and doing all the summarization and everything. It is. That is a, that is a function where you just give an input and it comes out output. Now it's not the standard code that we see, but actually it works like that.

Maybe the most energy inefficient ETL pipeline in the world, but I do like the framing of that. Anyways, I know we're really banging everyone's head around this 3GOLFS model, but maybe let's go into more things that are more specific to how to go about eval. One of the things that we just—

linger a little bit on the gulf thing.

Um, Hamel loves the gulf.

Why, why is it useful to know what gulf you are on? Like, it's worth, like, just, like, frameless, like, repeating that because it might be wondering why these gulfs. I know, but it's good for you to explain.

I think we'll actually have different answers. My answer here is you— it helps to know what hat to wear, what tools to be having in your tool belt when you're going about solving a problem. So if you're in a situation where you recognize like, oh, the model is not powerful enough, right, like you want to be able to switch modes to this and do research into like, or think about like how to construct fine-tuning datasets. For example, if you're in a situation where you don't know what is in your pipeline, you want to wear your data science hat or open up those tools and figure out how to move there. Um, that's what I think is most powerful. Helpful, but I'm sure others—

yeah, I agree with that. I think it helps you diagnose what the issue is. And if you're diagnosing the issue— so let's say your prompt is great but you still think it's a prompting issue, but it's actually that you haven't looked at the data enough to— so that's how, that's how it helps you focus your energy.

Yeah, I feel like the, uh, data island is ignored too much, no matter how much we repeat it constantly. That's the one that people don't think about.

It's hard.

It's really hard to look at data. People don't quite understand how to look at data. And sometimes when I plot a table, I have all these metrics, people will be like, how do you even read this table? Like, maybe because I've just been looking at data for— I wish I could teach people how to look at data. But I don't know.

We're gonna get to that soon. I'll have my last comment on this. I guess this goes back to Hamel Husain's question of like how this came about. Um, I think that all of these pairs of islands are like of active areas of research and development all the time, but only application developers uniquely actually have to consider all three. Like, I'll, I'll tell you exactly, like developer and data, like that bridge, there's so much data tooling, there's so much data science frameworks, everything to like There are whole industries around data science for this, barring the LLM pipeline, right? Developer to LLM pipeline, like, this is all like HCI and UX and ChatGPT and, you know, how can we collaborate with LLMs. Um, LLM pipeline and data, that gulf of generalization, like, the entire machine learning and AI community is focused on, you know, we have very well-defined tasks and how do we just make models solve them well. But when you're building an application, you got to do all three, and there's no nothing around it. I mean, I think that's kind of what has made people really stuck for a really long time. Um, but anyways, okay, I like that.

That's a really good explanation. Yeah, the Gulf of Generalization, I feel like that's the one that foundation model providers are trying to solve mostly. Um, they're telling you as the application developer, like, don't put your moat around this. 'cause we're gonna steamroll you. You know, but it is, it is still useful for application developer to know 'cause like things like fine-tuning, you know, you might wanna pay, you need to pay attention to that.

Yeah, exactly what Eugene said earlier, like this model capability may not be there. All right, so finally moving onwards, the next thing that we kind of really first introduced, and this, this, you know, is our first attempt at it. So Eugene, welcome all criticism and feedback here.

I just want to say that this is the other thing that I highlighted in green. No matter what, you have to keep this. Yeah, that's Figure 2.

But my question for you was going to be, I'm sure you've done this cycle over and over and over again at Amazon, still probably doing it right now. So two questions here. Which box do you feel like you spend the most time in from analyze, measure, and improve? The other thing is, are there sub-boxes of these that are important to be elevate to their own box, or some boxes to be aware of, you know, in error analysis. Maybe there's some big thing involved in there, but it's still clearly an error analysis.

I think the hardest box is measurement. So I'll talk about it. I think analysis is like you look at the data once. If you know what you're doing, you can sort of figure out what the key failure modes are. You can create a few hundred samples as your eval set. And that's one and done. It's like good capex. I don't mean it's one and done. It's like the big chunk of effort is done. And then improvement is like, once you sort of have a good reward model slash auto-evaluator, you can sort of improve on that. And that's somewhat of a software engineering problem. What is really, really, really hard is how do you translate your evals into an automated black box that measures yourself against it. And a big part of this is like, okay, everyone's using LLM evaluators right now. How do you calibrate your LLM evaluators to perform well on the evals? Um, it's really hard. Uh, and I wish there was an easy way to do this. I haven't figured out how to do this. Like the app I built, Align Eval, was trying to do this, but even then optimizing the prompt, trying to get it to where it is, is still very challenging. And if this measurement is not good enough, directly you won't be able to use it to optimize the generator. I think of the auto-eval— the LMS judge is a validator, right? And then you can use it to optimize the generator. If this validator is just not strong enough or reliable enough, there's no point going to try to optimize the generator.

I have a question.

And the reason why I say that this is the hardest thing and takes me the most time I can get other people to do the analysis. I think PMs and subject domain experts are very good at it. I can get other people to improve the— to do the improvement. Software engineers and other scientists are very good at that. I haven't been able to find— I found it very challenging to try to figure out how to take the qualitative and convert it to quantitative. Um, you had a question?

Yeah. Um, so when Shreya posed that question, I was thinking in my mind, I feel like I spend most of my time with the— in the analyze. Um, I know, um, and my question is, do you find that people can be effective in the analyze stage if they're not data scientists? And how do they learn that skill? Or what, what do you find? I mean, you meant— you mentioned it briefly already. We just want to—

that's a good question. I think they can be taught. I think there are some people if they feel a sense of ownership over the feature and they're able to put themselves in the shoes of the customer, they can try to understand this is bad, this is good. And you know, we have good methodologies to do this. And you know, Shreya mentioned a bit, a few of them, like, uh, later in the chapter, like, you know, binary only or preference selection. I think those become reliable enough. What I find that, okay, I could label some data in a day and I could get a couple hundred samples, but then I'll spend a lot more time than a day trying to calibrate my auto— my LLM evaluator to match your samples.

Well, we gotta send you Chapter 4. I'll do that after this. Yeah, sorry we didn't send you that. I didn't want to make you read like so much. Um, that— yeah, going off of Hamel, I think Hamel, when you ask, like, you spend a lot of time and analyze I, I wonder if, you know, like big companies just like have the infrastructure and also the people to be able to follow these kinds of cycles well. Like you kind of already know who you can go to at your company for analysis. You have like maybe a role in mind for measure. And often, you know, people— a lot of, a lot of the people taking our course are not at big companies, and maybe that is why they get stuck in analysis to some extent. Um, it was very interesting yesterday at the, um, LangChain conference, um, someone asked me what role or like who to hire to help them do evals, like what are the skill sets they should have. And I found that it was like this weird thing of they should be a data scientist so they can do analysis, but also they have to solve the problems that Eugene said And that's not all data scientists.

So it's actually very hard, and I've been through this exercise thousands of times.

I find that like very data science-y though, even the math.

It sounds very data science-y, right? But I find that the analysis part, if you can put it in a spreadsheet and you can just tell people to say 1 or 0, yeah, or left or right, you can actually decompose that. Yes, it takes a bit of time, it takes— it's a muscle. That the, the team needs to learn. And it's a muscle also, writing SOPs, writing evaluation annotation criteria is a muscle. But I think once you build that muscle, um, you can do it. Like, you can spin up a new task within a week. So now the blue boxes—

yeah, one question I have is, okay, I agree with you, you can get people to label, you know, binary labels, go through spreadsheets, things like that. One thing I found that I took for granted is the skills involved in studying that data afterwards, like aggregating it, slicing and dicing that data, whatever. You know, we have a lot of skills as data scientists to just do— we think it's like breathing air or something very simple. But I found that people struggle with, okay, analyzing it.

Um, yeah, I agree. Okay, the second part of categorize failure modes, that's slightly more tricky. That not everyone can do as part of the analysis box.

Um, why don't we go into that? I think that will be skipping Chapter 2, which, based on this conversation, honestly, it's interesting that we immediately went to that. And maybe it makes me wonder what the role of Chapter 2 is, like giving kind of foundations on LLMs to some extent. I'm curious how you felt while you were reading Chapter 2, whether you felt that this was useful. Like, just for folks who've not read it, like, it is basically like—

it's, it's insane. You almost like— it's like almost you read my mind. I actually have Chapter Section 2.2 highlighted in orange. I felt— and, and Section 2.4 highlighted orange. I felt like, um, it was not going as punchy as I would like. I don't know, maybe like for beginners, like some of these basics would be really helpful. But I feel like I just want to get into it. And this was sort of in the way. Maybe it could be an appendix. I don't know. Or maybe it's like I'm not the right audience. But these are the two things I highlighted in orange.

This is great.

And this is why we haven't released the book yet, because we're going to iterate on it.

Yeah.

And that's why Shreya was a student. Asking, I wonder if we should have this. And those are the two things that jumped at me.

Yeah, I think you make a good point here. I think the audience that we had in mind here, like software engineers, literally the title, engineers and technical PMs, probably they don't know. I'm assuming that people don't know about LLMs. But I also agree with you. I think that's a lot of stuff. Especially for somebody like you, you know what LLMs are, you have a lot of experience with them and therefore you know what you should expect and know what not to expect in ways that other people might not. I think we can definitely move parts of it to the appendix, if not a lot, put reader notes to signpost like, get out of here, don't think about this. I agree. I guess we have a little bit of stuff around, and this was, I think we somewhere in here I cited you.

I think I had a sticky note. It's like, hey, look, Mom, that's me. Yeah.

At some point we kind of transition into how to think about evaluations, what types of metrics, and then also how to think about evaluations. What types of evaluations you want, how to think about getting feedback. I feel like this is a little bit ad hoc, and I'm curious what your take was on how you were— how you felt when you were reading it. Did you feel like maybe this would be better with a reordering, or we should just kind of move stuff later?

Let me look at, uh, Section 2.7, right? Nothing actually jumped out at me. Um, I do think that there's one section I highlighted in— I mean, if we are going to details here, um, the third paragraph, I feel like the sentence, other methods focus on, uh, focus instead on relative competence. I think this is great. I really like this emphasis on relative. A lot of Likert scales, a lot of grading, it's just too subjective and it's not consistent. I think this relative approach is actually going to help us solve that. But there's another one, which is that which summary is more factually consistent with the source document. I actually kind of disagree with this. I feel like this is a binary statement. It's either factually consistent or not. In terms of relative, it could be like more things like tone, like which tone do you prefer. So that was the two.

EBay. Oh, I like that. Yeah, I think it's a good point. I think when I was writing factually consistent, I was thinking like which one fits better with the content, but that's, that's not factually consistent.

Like which one is more concise or which one is more comprehensive. There's some things that are just hard to draw a line on.

Yeah, something that's clearly— that makes sense. Thank you for that. Um, okay, so All of this, I'm realizing, let's— we'll change that up. But let's get to error analysis in the last kind of 20-ish minutes. So I'll start off by giving a very brief, high-level overview of error analysis section. This is from Anthropic, by the way. This is not from me. So this is kind of what inspired me in the first place. I would say this is like a year ago where I saw this and I was like, oh, it's nice to think about this workflow. Of like what I should be doing as a human being building LLM applications, but I also did not know what this box meant. So that's the motivation for this. And I'll go to the first figure where I think— where we think about like what are the steps, you know, that you might do when doing error analysis here. And first starts with just coming up with an initial dataset. So I'm really excited to ask you this question because you are at a company in which you probably have a lot of data. And when we wrote this, we wrote this for people who might not actually have data. So for example, generating synthetic data. And here's an approach to coming up with synthetic data that might be better than like simply asking ChatGPT or like asking some random person. Hamel Husain goes on this big rant all the time of how people outsource everything. Don't outsource your data generation. But we come up with a little bit more of a systematic way to do this. So I'm curious what your thoughts were when reading this. Did this— first, is this something at all that you agree with at the high level? Then second is, do you guys do this at Amazon or a version of this? What is that and what are we missing?

So firstly, all opinions my own, I'm not speaking on behalf Sorry, not at Amazon. Actually, it's not there.

So secondly, what do you think people should be doing? Forget Amazon.

I think this is a good idea. I especially like, we have a statement in the next page whereby you share the approach where you do a tuple, right? So you have a statement whereby if we let the LLM to generate full queries in one step, it tends to produce repetitive templates. Happens. Fully agree. What I found is that you kind of need human judgment to say this is the outcome I want. Or let's just say you are generating, again, fashion recommendations. And you know, you— if you ask it to generate fashion recommendations given the same persona and given the same thing, it's going to be generating very same things. But now you may want to push it in a certain direction. Okay, I want fashion recommendations that I want you to generate queries for edgy fashion recommendations, cheap fashion recommendations, for, I don't know, like all the different edge cases that you know because you have worked in the business and you know what kind of queries your customers are asking, and you nudge the LM in that direction, it will be more useful when you generate the queries. Um, before I even read this, I've actually done a bit of this, uh, because I just wasn't getting the kind of queries I want. It was not as rich as I want. And this was the, this was the way to do it. It's almost like, uh, instead of generating chain of thought, you generate rationale where you give it, this is the final output I was supposed to expect, generate the rationale, and then you feed that rationale into your FinShot prompt. Um, I think this is the same thing. This is what I want to achieve. And then you generate. So I think it works. I think it's a good idea.

Interesting. Do you find that it's hard to think about. So I, I told this— I stole this example of the real estate chatbot, or whatever I call it, real estate agent, from Hamel Husain. Because Hamel Husain— also, you should read Hamel's blog post about how he worked on this. Yeah, real actual use case here. But as someone who didn't at all, I, I had to go through this exercise myself, basically, because I'd never worked on it. And I actually found it pretty hard to think about. Like, I'm not even a real estate agent. Like, what the heck? What are the dimensions of a query that could be relevant? Um, and I looked at Hamel Husain's blog post and I saw some example queries there, and I was like, okay, I can break that down into like things that, you know, real estate agents want to do, um, who they're working with. And there's more, but then like this The scenarios, Ferrari is not a good term here, but it was something around query ambiguity. This is just something I've seen on a regular basis. A lot of people try to submit ambiguous things to agents and you always want to know how well the agent handles ambiguity. But yeah, I was surprised at how hard I found this exercise to be. I used ChatGPT a little bit to think about what these dimensions might be and it gave me 15 dimensions. I was like, okay, I don't know if these are the right dimensions, so I'm curious. Both of you actually. How do you go about coming up with dimensions? What parts of it are hard? Is it okay for it to be hard, or am I just weird?

Um, I'm sure it is hard. It's very like specific to each problem. It's like the Gulf, so like you build the bridge each time, um, and these dimensions are different each time. When I was writing the blog post, I was trying to make it concrete and say, oh, these are the dimensions I used that this time, but consider your dimensions, whatever they are. You know, it's not always features, personas, scenarios, but it's more like grounded in, again, the data analysis, like the error analysis. Like, hey, what, what dimensions do I really need to vary the most? And what, what failures do I want to elicit with this synthetic data generation? And I have that in mind, like, okay, I'm going to elicit this kind of error. Because I want this to pop up. So like I make sure like that's in the, you know, this like scenario somehow.

Edge case generation. Fortunately, software engineers are very good at this.

It's not like specific to data science, but yeah, I would echo what Hamel Husain said. It is hard. Nonetheless, a little bit of understanding of the business helps a lot. So again, the fashion example, let's say you're trying to elicit queries, right? And you may know what kind of queries your users send to your fashion advisor. They may be sending query with, I have— I actually don't know what fashion pieces look like, I'm just gonna make it up. I have this Air Jordan 2027 shoe, find me a pair of jeans that matches it. So now the query is very specific to piece of article, or I'm going out for a nice summer day, or I'm going to Hawaii. So that's a scenario, or there may be Hey, find me things of this specific brand. So now you have 3 things, right? Like there's the item name, there's a brand, and then there's the scenario. So based on this, you can generate queries that meet this scenario. I don't know if I'm using the right, the word scenario correctly, but that's how I think about generating queries.

Scenario is not the right term for me to do. The, yeah, the triple or the basically the tuple of dimensions. Well, what you said about the fashion sample reminded me of my footnote here, um, and this is definitely true of search. This is a long footnote, but basically in search there's only very few query types, or the taxonomy is small, um, and I don't know if this is true with agents or not, and I'm curious what your take is there.

I I think that— I think it's the Pareto rule. You'll find the Pareto rule: 20% of queries address 80% of what users want to do. And the same thing for search. So I think that will happen as well. There's gonna be a long tail, definitely. And of course, you may choose to address or not address that long tail. It's up to you. But I think, yeah, you can get very far with Yes, very interesting.

Yeah, especially I'm thinking here with agents, like perhaps even a longer tail, because you could put like essays into these ChatGPTs, um, in ways that it's harder to put it, I say, like Google search. Um, anyways, that was an aside. But so moving on from kind of generating synthetic data and synthetic queries in a more kind of interesting way, So the next part of this kind of talks about how to even like go about qualitatively coming up with the failure modes. And I drew a lot of inspiration from grounded theory from social science here. They do have names here for these kinds of— what's called coding. It's not our engineering coding, but it is like writing on the data. And I tried to cite some stuff for the social science geeks in the room. Um, but I'm curious if this is something that you guys follow to some extent, or you do, um, and what's similar, what's not similar. So for, for the audience, I talk about a two-pass approach here. First, you go through every trace and kind of write open notes and label them, and then the second pass, you try to merge these, um, first-pass annotations together to come up with different failure notes. So I'll hand it off to Eugene. What do you think? And then Hamel Husain.

Yeah, I think it makes sense. But I also want to share how this can go not so well. So let's talk about again our fashion example. You may have fashion experts that may say that, okay, you know what, I actually can't do fashion. I'm going to change a little bit. I wish I could be using— let's talk about translation example. So imagine that you could have errors where the idiom was translated incorrectly, or the text is overly literal, or the text is awkward phrasing. Now, to an expert, this might seem like 3 things, but you may try to get an LLM evaluator to distinguish between these 3 things, and you may never succeed. To a layman, to me, I actually don't— I can't really tell the difference as well, like idiom, over literal translation and awkward phrasing, then why not just merge them? So now this is where you actually have to go against your expertise, your expert judgment, who's saying, you know, these are these, these are three things, like this definition, whatever. Then there are instances where the category is so big, such as the category could just be a category called accuracy. Now, because the category is so big, and then you try to get an LM evaluator to catch this, you actually need a lot of false positives, right? Because the category is so big, you have to write very broad instructions. Now, if you want to reduce your false positives, it may help to split it up. It could be like, hey, you know, you got the subject wrong. It's male, not female. And you know, this happens in certain languages where nouns have subjects, like Spanish. Or the category could be like, oh no, you got the formality wrong. And this happens in German. Where there's a different level of formality and switching the formality in the middle is very jarring for German speakers as I come to learn. Or you may have gotten the nickname wrong.

Interesting.

Now accuracy is not just accuracy in general. And you know, when you just say, hey, evaluate this translation on accuracy, the model will just evaluate grammar and spelling and everything and therefore too many false positives. So I feel it's an iterative process.

Do you feel like this is, um, I mean, you mentioned automated strategies here. Like, do you think that there's a problem that humans have when they are going through and looking at their data? Or tell us more about that.

I think experts don't have this problem. I think laymen have this problem because we're just not trained to the extent— like, I, I can't tell the difference, honestly. When I look at the data myself, sometimes I, I struggle to understand the expert's point of view. And then, then comes the question, right? Are we building this for the experts, or are we doing this for the general customer? So you also— it's hard. You need to put yourself in the shoes of the general customer and how they think about this.

Interesting. So if I were to rephrase what you said, correct me if I'm wrong, it's that if you're not an expert and you're looking at this, you don't realize the nuances of how each trace actually should behave differently, and you try to evaluate them all in the same way, maybe. Translation was the example that you gave with German might expect something different than another language. Then you do that and you actually get bad evals. That was interesting.

I wonder how the LLMs just cannot perform well.

It's not necessarily automated thing. I think in this chapter we can be more explicit about it is like this is not an automated chapter. This is like with failure modes. Do the annotation yourself And I think one, we should be explicit that this is not an automated process right now. And then two, that you need to be an expert or call on an expert to do this, um, for some of the reasons that you mentioned. I think that's good feedback. Um, yeah, interrupt you.

No, no, I was just taking notes. I can also edit this as well, but yeah, I agree. Yeah, I don't have anything else to add.

No, you tell people to look at their data all the time. Um, Eugene, I have a question for you.

I actually bought a hat.

So where do you get this hat? Can we share the link? I need to buy this hat. Yeah, I need to buy this hat.

Oh, um, I made it.

You have a— I made it for online store or something.

I just went on Amazon and searched customized hats and just pick whatever that has the font I like.

There's a thing on Amazon you can customize your own hat.

I'm gonna buy it and I'm gonna ship it to you. Yeah, so I bought a hat just for you.

Okay, I'll buy one and ship it to Shreya then. We'll do a circle of—

and then I'll buy one and ship it to Hamel Husain.

Um, oh wow, you can customize it. Can you customize it in the Amazon interface itself?

You can, you can. Uh, it's I've learned it's quite advanced. You can just type whatever you want in a text box and they'll print it for you. You can upload images. I didn't bother, I just typed in the text box.

Wow.

Sorry, Shreya, we interrupted you.

No, no, I want the hat. I look horrible in baseball caps, but I'll get it anyways. I'll take one for the team. I don't even remember what I was going to say earlier. Uh, oh yeah, yeah, my question was, do you— when you're doing your first pass of error analysis, Eugene, like, it's interesting that you thought of using automated evaluators first because that's something I've literally never thought to do.

And like, maybe that's because Hamel always preaches like, no, I don't, I don't do it. Like, I make the people, like, you know, yeah, it's almost like, you know, I'm forcing you to look at the data.

Yeah, but maybe there's something there to like, Eugene, what kinds of metrics do you think you like are good for like automated evaluators in the first place, if at all? Or like, why do you go to that?

So I want to clarify myself. Um, when I look at the data, the first pass, I do it manually just to get a sense. But then once we come to the failure modes, and then we try it with the LMS judge, LM evaluator. We may find that the LM evaluator is just not able to get it. That's where we may need to revisit our failure modes to either merge them or split them and then test it with the LM evaluator again. So it's a cycle, right? That's why I like the, um, A, uh, analyze, measure, improve. Um, after you analyze it, you need to figure out if you can measure it well. If you can't measure it well, okay, back to analysis.

Yep.

So there's an inner loop there. Um, and that is the, that is the loop that is the hard, the hard loop, at least from where I need to revisit the tools that I—

some of the tools I really enjoy are ones that Treya has come up with in this area. Um, evalgen is one. Have you—

it's now out, it's now out on Chainforge, right? Yeah, yeah.

So it has this like workflow, very innovative human-computer interface workflow where it's like you annotate and you build the judge and you iterate on it all in one integrated flow.

Yep. I think that's a great idea.

Which I think, you know, which I wish was a commercial project. Someone needs to— it's a startup waiting to happen.

It's hard to do. Hard. I mean, it took a year to merge to main, but I think also from like—

how about Australia? UBC or Hamel Husain?

No, no, no. I'm literally teaching a course. I'm a grad student. Um, yeah, I think, I think doing all of them at the same time is the dream. And we— there's actually a chapter in here, Eugene, I can also send to you on like how to build good, um, interfaces for human review. Um, not in the first three, but anyways, I know we only have 5 minutes and I don't want to keep people on much longer, so I might move towards some of the latter questions, um, that I had. And I think there's one more big one.

I'm just gonna wear this. Here we go.

Finish.

Even though I, I don't really look good in this, but okay.

I can get t-shirts and hoodies and stuff. I'll figure it out. I'll up the ante on this situation.

It'll be like a bucket hat or something.

Yeah, the reason why I say this is because Nick quite on the YouTube channel, you should just have a flag or sign in your background. Makes sense. That's why I figured I just keep it on. Yeah.

Okay. The last question I'll ask that's relevant to the book is around— I mentioned we did one pass of just writing your failure modes and annotating, and then another pass of trying to group them together, merge them, or refine them in a bit. One thing that we mentioned here is actually using LLMs to synthesize failure modes out of your open-ended annotations. This is something Hamel Husain and I have discussed about how we like to do this, and it actually works pretty well. But we're curious if you do it, or honestly, if anyone else does it, or how do you do this merging process?

Whereby you say you take all the error, the failure modes, and try to get it faster and do it.

Yeah, basically the LLM is the clusterer.

Yeah, that's what I would do as well. I would take— yeah, I would just copy the errors and the human explanations in Excel and then just paste into some LLM and ask it to synthesize and see.

I mean, it's worked for me. I think there's some clunky parts around like when it does come up with the failure modes, then I have to go back into my spreadsheet and like figure out, you know, what failure mode does it map to, so then I can make Hamel Husain's favorite pivot table. He loves pivot tables, um, or groups.

So I have clients that like this so much that— because I— one of the things that we talk about is make your own annotation tools, and clients like make— you know, when they make their own annotation tools, they actually build this in now to their annotation tool where it auto groups and classifies failure modes.

Well, I—

yeah, I have a note here that says that I think it really makes sense. And like everything I treat— everything I do, I treat AI as a first pass draft. It's like everything I just— yeah, well, not when responding to you guys, like, or like, you know, just talking to my wife. I don't— I don't pass that through a ChatGPT. But you put it through AI, it will come up with a very good draft. It will come out very good clusters. And Now, to be clear, there's always 5 to 10% manual tweaking you have to do at the end, or maybe as much as 50%, but it always saves you time, and it almost always finds me patterns that I may have missed. So definitely, I will always do this.

I love that quote. AI is a very good first pass, and that's it. Yeah, I feel the same way about using ChatGPT to do things. I, I find that my workflow for this error analysis, this chapter 3, is very fragmented across tools. Like, I know we espouse a lot like building good interfaces, manual custom interfaces, but sometimes I just export to spreadsheet and just do stuff in spreadsheet, and then I will copy and put that in ChatGPT. It'll come up with failure modes, and then I like go back and create a new column in my spreadsheet, manually add things.

GPT can output CSV file pretty well.

Yeah, I'm so bad at it. I don't know why. Maybe I don't have the, like, I'm using like, oh, whatever pro. I'm so bad at like prompting it to do anything with CSVs.

But interesting, maybe depends on how many rows. I've gotten it to return like 50 rows or 100 rows. Not ChatGPT, of course, like Claude.

Claude. Okay, maybe it's decent. Maybe it's the wrong LLM. I don't know. I also feel very impatient sometimes when it's like thinking about its code that it's writing. And then it's like doing dataframe.head and then like seeing all these tokens come out. Anyways, all right, so 1 minute remaining. So why don't I conclude?

I want to say that I really enjoy Section 3.6. I wish there was some way I could highlight this. Oh, the thing is, I wish I could put it in the front of the chapter. But most people wouldn't understand it. You had to, you had to pay the effort of the previous sections to understand this very well. But this is a very good section. I especially like paragraphs 2 and 3. Um, actually, I like the entire thing. Skipping open coding, um, use of Likert scales, treating annotation, treating SOP as a fixed criteria, um, those are the most common pitfalls. Those are very common pitfalls.

So yeah, this is a Hamel section. He's like, what are the common pitfalls? I always love working with Hamel. It's like I'll write something, he'll be like, this made no sense. Like, make it clear, communicate better. And when you have him, it turns out that being slightly dumb is helpful. No, it's not slightly dumb. Yes, very good communicator. Anyways, it is 6:31. So maybe any final thoughts? What are you excited for in evals, Eugene? What do you think is up and coming? I feel like the field is stagnating. I'm excited for this to come out. I think there's going to be some new and interesting stuff here. But generally, what's exciting to you?

What I'm thinking about right now is that, you know, you have that loop, right? Analyze, measure, improve loop. I wonder how much we could make that loop automated. I think there are two loops in this. The first loop is the analyze and measure loop, whereby how can you get your measurement to align with your analysis? And that's what Align Eval is. That's what a lot of stuff is. But once you do that, and maybe this LM evaluator is a little bit lossy, how can you get that to move upstream? Instead of catching the defects, improve the upstream generally.

Better.

Um, so you can imagine this like maybe backpropagating from the initial eval set to your measurement, to your improvements. I, I don't know how to think about that yet, but I do know it's very essential. Um, would love to chat with you on that.

I mean, yeah, yeah, you can already see the beginnings of it with, you know, with AlignEval, with, uh, DocEtl, with evalgen. They all have like, are tightening this loop. And those ideas are not— some of them make their way into products. Like we saw some, you know, announcements from LangChain, uh, yesterday or day before. But, you know, it's still kind of hidden away. And, you know, we— none of us here are in the business of trying to create like, uh, SaaS tools or anything, but Yeah, I agree with you. There's a lot of opportunity to, uh, to make this not completely automated but put human in the loop in the right way to make it a lot more, like, a lot more automated.

I think this is the exciting problem of the decade. Like, how do you automate this in a good way, right? Like, not in a silly way that, like, assumes no human involvement, or another silly way that assumes too much manual human involvement.

And also maybe to just play devil's advocate against myself, right? Imagine you have your eval dataset of maybe 400 samples, 100 or 400 samples. Instead of going from the analysis to the measurement and then to improvement, why not just shortcut it? Why not just put your eval dataset in the prompt? With prefix caching and prompt getting cheaper, that's a 400-shot learning. Maybe that's good enough. Maybe I'm overthinking this, right? We have to do this loop, but just do the simple thing. Um, I don't know.

I don't think you're overthinking it. I think you can't get to evals without doing analysis. That's the thing.

Yeah, you do the analysis and just poop it into improvement. You don't even have to measure it.

Maybe. Interesting. Yeah, maybe.

Just to make the analysis so delightful and so frictionless. That you want to do it and you're like, and you see the payoff so fast that you just are motivated to just keep doing analysis because you—

and, and just make every analysis a new few-shot, a new few-shot prompt, and then you fix the improvement. Yeah, maybe. I, I don't know.

Eval V2 is what I'm hearing.

All right, so when are we starting the eval SaaS together?

I mean, can I be an advisor and a silent partner?

I'll invest.

It's like that meme where everyone's pointing at each other.

No, it's like that meme where there's one person digging. Hamel's digging and we're all standing around him.

I'm gonna— no, I'm not doing— I'm gonna— I have to work harder at convincing y'all.

Yeah.

Anyways, we're well over time. Um, so thank you.

Wait, before you go, can you talk a little bit about this book, Shreya? Like Will it come out one day somewhere? Where is it? Where?

How?

All that stuff.

So you want to read it soon, take the course. You'll get like, you'll get each section at each lecture, basically, and you'll have the whole thing by the end of E1. Um, we are— we don't know who we'll publish it with yet. We're working on figuring that out, but it will be a book that you can also buy later, but I wouldn't expect it until like end of summer, maybe.

Okay, and if people want to take the course, there's a 35% discount code in the Luma.

Sorry for not selling.

Yes, of course. So just letting you know in case it's helpful. Um, or just find— just look through Twitter, uh, look through Twitter on any of us three and you'll find discount codes everywhere.

Yeah, it starts next week, so really excited. I think it's gonna be super fun. Um, you learn a lot. Here's— I know now which chapters to send Eugene, so continue with human review.

Just send the whole thing and the whole thing.

Okay, yeah, and the whole thing. Great.

All right, thank you everyone.

See you folks.

See you folks. So Hamel Husain, you stop the live stream and we'll stay here.