AI Agents and LLM Judges at Scale: Processing Millions of Documents (Without Breaking the Bank)

This is a $10 million budget project from the State of California, and they want to have LLMs do this because the director of the School of Journalism said if humans were trying to create this database of officers and any misconduct detailed in the police report, it would take them over 35 years to do so. Now your question becomes how do you have LLMs go through this kind of police reports, extract police officers and instances of misconduct, and then somehow build this big database of all of that information? But the police misconduct example, there's a really, really high cost of doing something incorrectly. So people are still working out like, okay, how can we more automatically verify that things are correct? Because we just can't publish an incorrect database. But for a lot of other applications, you don't really need perfect accuracy. For example, themes from customer support transcripts. As long as it's like a pretty aligned with what you want and the quotes are pretty representative, you're going to learn a lot, right? So that's where I see a bunch of different applications differing that was my.

Friend Shreya Shankar, who's completing her PhD in Human Centered Data Management Systems at UC Berkeley. What makes Shreya's perspective so unique is that she's a true builder with deep industry experience. Before her PhD, she was the first ML engineer at Viaduct, did research engineering at Google Brain and software engineering at Facebook. Shreya's research isn't just theoretical, it's being deployed in production at major tech companies like Meta and startups like LangChain. On top of all of that, she co teaches a hugely popular, nay the most popular and exciting practitioner focused course on LLM evaluation. We start with real world use cases, from building a database of police misconduct reports to analyzing customer support transcripts. Then we dive into Shreya's powerful framework for treating these as structured ETL pipelines. The core of our discussion is about reliability, which is we cover the non negotiable process of manual error analysis. That's right, how to use agents to automatically optimize your pipelines and the practicalities of using an LLM. As a judge, I learned a ton and I know you will too. This conversation is a guest Q and A from a recent cohort of my course Building LLM Powered Applications for Data Scientists and Software Engineers, which I have the great pleasure to co teach several times a year with my friend and colleague Stefan Krachik, who works on Agent Force and AI Agent Infrastructure at Salesforce. Links in the show notes if you're interested. Enough out of me. Let's Jump in. What is up everyone? Hey Shreya, great to see you.

It has been a 4o mini to do that versus a summarization. Maybe you want Claude to do that. So ultimately I see this as a really big search problem. A user has a query and a document collection. First you want to search for the most accurate pipeline possible. And then you want to search for how you can swap in models of those pipelines with cheaper implementations, cheaper models, maybe it's code that's doing the extraction. And you want to do this cost optimization in a way that meets guarantees with respect to your most accurate pipeline. Like I want to achieve 95% accuracy of the most accurate pipeline I've found. So that's the high level overall picture.

Great. And so when we talk about accuracy though, when we ask how do you make things accurate, there's almost an assumption that we have consistency or reliability. Right? So how do we even think about this when LLMs are kind of like flaky? They're like your friend who turns up sometimes and doesn't other times, right?

So in the Doc ETL project we're building, so you can think of it as like a mapreduce like framework where you specify map operations of your data, reduce operations of your data. But these operations are specified by LLM prompts. So you might imagine to do thematic analysis on your customer support transcripts. You have a map operation that extracts themes and quotes from each document. Then you have a reduce operation which groups by the theme and then generates a summary of all the quotes that correspond to a theme. There's one problem of baseline reliability, which is the LLM was just not consistent as you said. Or it just happens to return an empty string or some garbage that if you asked it again it would get the correct answer. So for that we have retries built in, we have code based guardrails. We have something that we call gleaning, which is if you have a cheap validator to check for certain properties of the output. Like let's say I want there to be at least three themes extracted from each document. That's a code based validator. Maybe I want the themes to be sufficiently diverse so I can have a very cheap LLM judge evaluate if they're diverse or not. If they're not diverse, rerun the operation until it was diverse. We have a bunch of validation guardrail like strategies over there to improve output. Now the second thing is overall alignment with user preference. If you didn't define what theme means in your original map operation prompt, there is no world in which you can expect the query engine to extract the right themes that you're having in your mind. But this is where our Doc Wrangler and our more HCI work comes into play. Which is what are the right interfaces and playgrounds that we can build for people to try out different versions of their prompts for their pipelines and then annotate which outputs are good, which themes are good, which themes are bad, and then have LLMs go and analyze that feedback and improve the prompt automatically. Turns out you can do this really well in interactive interface. It's much harder to do in a clique or in the data processing system that you're building.

I love that. And there are actually so many, so many things in there to unpack, which I didn't necessarily intend. I don't want to, you know, go down, but hole too much. Choose. So we're already, you know, getting into validation and guardrails and evaluation and, you know, simple basic LLMs as a judge. So we won't go too deep into this. Go and join Sharon Hamill's course if you, if you want to go, go deeper. But I actually just linked to your Doc ETL project and Doc Wrangler in Discord, and I wonder if you could speak to a particular example. I mean, for example, the US Presidential Debate analysis you did. Or another example just with respect to how you think about when you build out these systems, when to start doing validation evals, LLM as a judge, and how to do it and how the iteration cycle works.

Yeah. So I'll give you a concrete example from the original Doc ETL paper, which is a collaboration with the School of Journalism and the Law School at Berkeley. There's this project, project called the California Police Records Access Project. And what they did was they collected all the police reports that all the police departments had in the state of California. They want to build this big database of officer misconduct. This is a $10 million budget project from the state of California. And they want to have LLMs do this because the director of the School of Journalism said if humans were trying to create this database of officers and any misconduct detailed in the police report, it would take them over 35 years to do so. Now your question becomes, how do you have LLMs go through this kind of police reports, extract police officers and instances of misconduct, and then somehow build this big database of all of that information? Well, the way that we think about it is what does the user need to program? User probably wants to program the logic that is being applied to documents we think of documents as like rows in a table. So presumably what they want to do is apply some functions to each row to extract some attributes. So we call that the map operation. And then they want to be able to do further analysis on those attributes. So, for example, do de duplication on any extracted attributes. Or they want to group by an extracted attribute, group by the extracted officer, and then summarize a bunch of information related to that officer or a case or something like that. So the first layer is just this programming framework of what does the user even want? Well, they want to map over each operation. Then they want to do an entity with resolution step. Then they want to group by an aggregation. These are very familiar data processing concepts. Then the second thing as, what do you, how do you specify these operations? You want to extract police officers from a police record. Turns out that you have to write that down as a prompt. Somehow you have to explain that police officers have this are referred to by sergeant or officer or something, right? Like you have to give some instruction to the LLM and it turns out that you need some way to explain anything that's domain specific. So maybe it is that, like, I only want to focus on the police officer that filed that police report, or I want police officers that are actually in the state of California or affiliated with that department, not like other random police officers. So there's a bunch of specifications and prompts there that you need to write. Then I think there's an output formatting of like, I want to expect consistency from the LLM, so make sure you give me first name and last name of the police officer. If you don't say that, it might just give you Sergeant X or Officer X. And that's going to be inconsistent in all of the LLM outputs. So again, right, it's this like exercise of clearly specifying what you want it to do and what you want it to output. And then I think the guardrails and the validation logic, the manual validation comes in after that. So when people use Doc etl, they will run it on a sample of documents after they've written good specifications for their pipeline. And they will have, I mean, like the journalism school, there's a bunch of interns that sit there and analyze the outputs and will tell you like, this was correct and this is not correct. You have three interns doing this for the span of like a week. And then you go think of this as error analysis, literally. And then you go figure out, okay, what did our prompt not specify that we should have asked for? And Then what is the LLM simply not capable of doing? Maybe this document was way too long and we realized that it was extracting too few instances of misconduct. So the way that we want to do this is to chunk up the documents and apply an LLM operation to each chunk and then you unify the results. So there's so much of that improve process. I think what we see is people will do like three rounds of this error analysis is improvement. So they have human annotators be able to say I got 90% plus accuracy or alignment with my expectations and then they will go and deploy it on the entire database. Now with the police misconduct example, there's a really, really high cost of doing something incorrectly. So people are still working out like, okay, how can we more automatically verify that things are correct? Because we just can't publish an incorrect database. For a lot of other applications you don't really need perfect accuracy. For example, themes from customer support transcripts. As long as it's pretty aligned with what you want and the quotes are pretty representative, you're going to learn a lot. Right? So that's where I see a bunch of different applications differing.

Fascinating. I love that you firstly, I love that you mentioned error analysis because that's a huge focus of what we cover in this course with respect to the, you know, iterative cycle of LLM powered, powered software and the lifecycle. Also we had Jason Liu came and gave a guest talk, guest Q and A this morning and you know, he's heavily on, well, once again part of the look at your data movement. I know, I'm sorry everyone. I know I promised we'd talk about agents and we will. But I also love that you mentioned in terms of error analysis like that the iterative loop of doing it and that you do it like in this case you got people to do it three or so times in order to get a certain level of alignment and accuracy. So I'm just wondering, you know, like 50 plus percent of the people I consult, when I tell them to do error analysis, they say can I get an agent to do that? And I'm like human learning or machine learning before agent learning, right? And I'm just wondering if you can. And people do. I do understand because it's, it's serious lift, right? But also you need to inspect what your software is doing to get a vibe for it and see what's up and understand what's happening with these, you know, quote unquote organic systems we're building. I'm just wondering if you can Speak to how often and how much you encourage people to look at their data and do error analysis and when is enough enough? Because we don't want to tell people that they need to be doing it every day or every week in perpetuity. Right?

Yeah, it's a good question. I mean, in our own course, we tell people like, you have to do error analysis, look at your data. We did a structured process for this. We have people do what we call what's called open coding, basically on each output. So write down, like, any failure modes that you might observe, any theories for, like, why it's bad. Many of the traces might not have failure modes. And that's fine, you don't have to write any notes for them. But the point is, you want to keep labeling, doing this open coding until you have no new failure modes emerge. Like, you've gone and labeled 30 to 50 new traces and there are failure modes, but there are failure modes you've already seen in other labeling. Then you hit the point where you're like, I have my set of failure modes that I would like to more systematically measure. Or I know that I need to fix these failure modes. For example, I wanted the output to have officer name, first name, and last name. And the best way to do that is to go specify that in the prompt. I didn't think to specify that in the prompt, but now error analysis showed me this inconsistency. So now let me go specify that in the prompt. Right. So, like, these are the kinds of realizations that people have when they do error analysis. I find that when people ask if they can have agents or automated error analysis, the agents are not going to find this, because how does the agent know that you expect consistency in outputs? They might have its own definition of what's correct. Like, as long as it didn't hallucinate a new officer, it says it's good, but you want something else. So this is where human judgment needs to be incorporated. In error analysis. The agent could be perfect and it's good. But at the end of the day, you are the one writing the spec for what the LLM pipeline should do, and you are the one who has to do the error analysis to make sure that your spec is exactly what you wanted. And I find that people, once they realize that, then it clicks for them to do error analysis. And they don't, like, ask to use agent to do it.

Yeah, and the other thing I've seen is once people start looking at their data, even if they're hesitant at first, they're like, oh, wow, that's interesting. And they want to dig in more and more, right?

Yeah, yeah, totally.

The other question. And we've got a question from Brad, which we'll get to in a second. The other thing objection I hear is, look, we've got like millions of traces now. Like, where do I even start? And how do you. What do you tell people to do in these types of situations when they're just already overwhelmed?

Yeah, I honestly think going through a random sample of them is totally fine. Again, keep labeling until you haven't observed any new failure modes. If you want to be smarter about sampling, you can, but you don't have to do it your first time doing error analysis. Take a uniform random sample of the data. That is fine. Sometimes, if you want to do a more sophisticated sampling strategy for the next time you're doing error analysis, maybe you can sample based on users. So I don't know, take a few records from a random sample of users, or you can sample based on geographic region, or you could sample based on query modality or I don't even know it's your application. You kind of think of the dimensions of your application that probably differ and then do some sort of stratified sampling around those. But often people are learning from random sampling. So I don't see people saying that they looked at a random sample of their data and learned nothing from it. I do like to do, like in very later rounds of error analysis. Sometimes I kind of already hypothesize my failure modes. So for example, if it's hallucinating a tool or something for those, I can write code or functions that can like, very quickly check for those kinds of traces in my unlabeled set. But again, that requires me to have done some error analysis in the first place to know what filler mode I'm looking for. So then I can go try to approximately find them in the rest of the data set.

Great. So we have a question from Brad. And Brad, if you wouldn't mind just introducing yourself and saying what you're working on at the moment. And Brad also just. He also has a funny accent.

Hey, thanks, Hugo. I'm just tinkering at the moment. I'm taking a lot of courses and I follow a lot of your work, Shreya. So thanks for everything that you guys are doing. It's great. Yeah, I was just hoping you could expand on the open coding part and maybe provide an example. Does this mean you're just logging all traces from the start and manually going through until you can broadly categorize everything. Are we talking about something as simple as a spreadsheet or just whatever tool it is?

Yeah, you should check out chapter three. So we have like the first three chapters of our course reader as like a free preview that's on our AI evals course landing page. So chapter three details this error analysis life cycle. So all of you can like read it right now, but you literally look at your data and you write down freeform notes on what is bad about it and worry about categorizing and turning those into like more quantitative metrics later on. You want to do this open coding for like at least 50 I ballpark 50 to 100 traces. Like do it for an hour, two hours and that like this process gives you so much more lift than pretty much anything else. Any other tool out there. A phone.

Thanks so much Brad for the great question and the great response Shreya. And Brad's actually being really humble. Brad comes from like a non data ML background but is actually building some really wonderful like personal AI knowledge management systems that started in our January cohort of this course. And he's, he's back for more and like building super, super interesting stuff. That's the other thing that this space at the moment in terms of opening up, you know, leveling the ground for everyone to come in and build things is, is just really inspiring for me. So Shreya, what's up with agents? To ask a bit more specific of a question, we're talking about once again programming LLMs to accurately and cheaply process large collections of DOC documents. We've been talking about a lot of the nitty gritty of it, but you're, you're really doing a lot of things but tackling it with agents. So why agents and what makes them a good fit for this kind of problem?

So I think of an agent as an LLM that can kind of choose its own execution plan. So it can choose what tools that it wants to use and it can choose what order it wants to execute them. I'm not asked.

By choose we mean the tool choice is a function of the LLM output.

Yeah, now I'm not super bullish on open ended agents to just do any data processing tasks I think you1 need to write the language which you communicate with the data agent. So like our Doc ETL framework is in a way that kind of language and you use agents for more scoped things. So one of the things that we're doing is writing a query optimizer that is agentic that it takes your pipeline and it runs some evals on it and figures out what is not performing well and it will try to rewrite the pipeline to be more accurate. So maybe it'll take an operator and rewrite it into a more complex pipeline of multiple operators that are much better scope. I think agents are a great fit for like if it has a very good metric to optimize for and it has a very structured set of actions or tools that it can take, for example, structured set of ways to rewrite the user's pipeline, then the agent's going to give you a big lift. But I don't know, I feel like people also want to just use the agent. They want to like upload a folder of PDFs to ChatGPT and ask for something, but there's really no good way of knowing or improving on that accuracy. It's not going to be accurate at all. I mean you have no idea what it did. So that's where I'm not a fan.

Yeah. And I mean what state of the art in function calling or whatever is 90% accurate or whatever it is? And you have a few in a row, I mean, I think maybe six calls or something that like gets you down to a coin flip of whether it's going to, going to work or not. So I am. We talked about what I'd refer to as the flakiness of LLMs before. What I'm hearing is that you, if you have like a small amount of tools that it's capable of using, it will be more flaky. But if you're principled and clever about it and have, you know, robust evaluations then, then you can. That can get you a long way.

Absolutely, yeah. And I think there's a lot of interesting work now in the research community with using reinforcement learning to train these agents to at least get like match 03's performance. So that's quite interesting. Again, it really only works on very small tool sets, like two or three tools. And the number of turns in the conversation is like 5 for some of these models. But it's interesting to see the space explode so much.

Now for a word from our sponsor, which is, well, me. I teach a course called Building LLM Powered Software for data scientists and software engineers with my friend and colleague Stefan Kraucik who works on Agent Force and AI Agent infrastructure at Salesforce. It's cohort based, we run it four times a year and it's designed for people who want to go beyond prototypes and actually ship AI powered Systems, the links in the show notes. I mean, I'm, I'm speculating here, but I mean there's a lot of work in the big labs going into RL and a lot of it's pretty, pretty secretive stuff. So I think this is something that a lot of people are thinking may form. I mean, I, my hot take is that none of these companies have any moat really. Memory is not a moat. Right. And that, look, open AI doesn't even have the network effects of MySpace. Right? And look what happened there. But I, I think RL is something which, you know, especially as, you know, trade secrets could, could help help there. But I, I think I'd love to know in these types of workflows, what kind of tasks are these agents actually doing in your systems and how do they integrate with what the humans and the domain experts are doing?

Yeah, we use agents for rewriting pipelines to be more accurate. So let's say you've given me a MapReduce pipeline. You've just specified your map operator to extract themes from each document. Your reduce operator is going to group by the theme and summarize insights per theme. If your document is very, very long, you might not be able to extract all of the themes. So that map operator is not accurate. So what our agent will do is run an eval to make sure to assess the performance, see that it's not very good, and then it will rewrite that map operator to be like a chain of thought, the decomposition to multiple map operators. Or it might split up the document and run a map operator on each chunk and then aggregate the results of the map operator on each chunk. These are what we call rewrite directives. These are very structured rules of like a. Take an operator that is not performing well and rewrite it into a little bit more complex pipeline still in our domain specific language in our framework, and then see if that performs better. Agents are a really good way to do this because they can kind of match rewrite directives to, and they can implement the rewrite directives. If you say like rewrite this map operation prompt to run on a chunk instead of the full document, the agent could come up with that prompt. So we totally use agents for these kinds of rewrites and as LLM judge to verify whether one rewrite is better than the original or not.

Super cool. And you know, you probably know I wrote a blog post recently called Stop Building AI Agents. Like half tongue in cheek. But like, one place where I think they are really powerful is as coding and coding assistance but also, also doing these types of things in an automated manner. And we're just going to see that get better and better. I am interested how you tell if a rewrite actually makes the pipeline better? And like, do you get an agent to do that and then use an LLM as a judge to figure out whether it makes it better? Like, yeah, how do all these moving parts interact?

Good question. So we use LLM as a judge, but again in a structured way. So imagine you have two pipelines that are logically equivalent. You want to compare which pipeline is more accurate than another. So you'll run each on a sample. So you'll have samples of out two samples of outputs. We have an LLMs judge that has a specific prompt that is specific to that pipeline, that operator. So it's going to judge theme extraction and it's going to have a rubric of criteria that is specific to the operation. So are all of the precision and recall are both assessed in that rubric. So then the LLMs judge takes this custom rubric and does pairwise comparisons on the plans to see which sample output better meets the criteria specified in the rubric. Then we just take the plan that has the most wins in the pairwise comparison and say that this one is the best. So the key points to take away Here are the LLM is Judge 1 has a very structured rubric that is domain specific, can be defined by the user or defined by a rubric generating LLM or something. And then the second is we do pairwise comparison or a ranking against that rubric. So it's not like one to five scale or pass or fail against the rubric. It is truly like, is plan A better than plan B? Given this rubric and the sample outputs, and that tends to align a lot more with accuracy.

How do you figure out? They sound like wonderful processes. How do you figure them out? Like, is there any science to this? Because, I mean, and to give a bit of context for us, we've been working machine learning for a decade plus. You know, we say like a lot of LLM stuff is incantations. So was a lot of the machine learning we were doing, to be honest. So I wonder like, how much is science and how much is art currently?

Yeah, it's a great question. So a lot of the stuff I'm talking about is actually like five different problems that we have individually written papers on. So in our research we have a big vision, which is the system for unstructured data processing. And as we build that or we build components of it, we realize, like, it's exactly your question, you know, is this an art or is this a science? Like, how do I know the technique that I'm doing here is very good? We might not know. So then we decide, okay, let's write a paper on it. Let's figure out what are the baseline, what is like the obvious way of solving the problem. Did that work? Can we measure that performance? If it didn't work, can we build something better? You'll notice that there's a lot of papers I've written in my PhD that are not the DOC ETL paper, but are precursors to DOC ETL. For example, the Spade paper, the Evalgen paper that you might be familiar with. A lot of papers on validation where we just simply tried different ways of doing evaluation for domain specific data processing pipelines. So once we kind of figured that out, turns out you can reliably infer criteria from prompts to. You can reliably elicit human feedback to refine that criteria. Rubrics are a really good way of systematically evaluating accuracy. If you don't have a rubric, if you just ask LLM judge good or bad, we found that that wasn't very good. So a lot of these problems again are solved in these precursors to the Doc ETL project.

Amazing. So Mitch has a great question in the chat. Are you in a position where you can turn your microphone on, Mitch, to ask your question and if, if so, if you could introduce yourself and let us know what you're up to at the moment, that'd be great.

Yeah, sure. So, yeah, I work for the UK civil service at the moment as a data scientist in an AI incubator they have there. And we have a product that's very similar to the Doc ETL stuff that you described. And it was really interesting. I never thought of it as framed as an ETL pipeline before, which is really funny given I've been doing it for a year. I've just been thinking of it as topic modeling. And so my role there specifically is on evaluation and trying to figure out whether the product does what it is supposed to. And we've done several kind of blind evaluations where we'll get, like you said, three. They're not interns, they're civil servants. But we'll get three people to independently undertake the task that we want the AI to do. And then we might even get a supervisor to kind of blindly rank those four sets of outputs and see whether the supervisor can even tell what's the AI and what's the people that are actually in their team. And we're getting reasonably positive results from that kind of stuff. But one of the questions we keep running into is can you reduce that kind of burden of evaluation over time and get to the point where people can just use your product out of the box? Where essentially I think the intuition is that if I go to the supermarket and buy paracetamol, then half of them aren't sugar pills. We've decided at a certain point that paracetamol works and they're all going to be paracetamol. And so I'm kind of intrigued how we get to that point. And I suppose at the moment we've kind of gone down to like do a meta analysis of all these evaluations we've done, try and get some consensus estimates on effect sizes and then try and use that to justify a reduction in sample size going forward. But that's. We're finding that quite tricky conceptually.

Yeah, I agree with you. This would be a great question to solve. So I think there's no world in which there will be zero evaluation for a high stakes application. You're going to have to go through at least one round of ensuring that there's 90% alignment or something with your preferences, otherwise you have no way of knowing. So now the question becomes how do you reduce the number of rounds and then as you said, the number of samples needed per round. I think in these cases it's a domain specific and the other thing to think about is how do you improve specification in the prompt so much to where you can reduce the number of specification failures that emerge in the error analysis? So so many errors emerge in error analysis that like if you just had mentioned them in the prompt you would not have seen that. And I think those are like do you have a good blueprint for how to write your prompt right? Are you using few shot examples? Are you clearly specifying do's and don'ts in the beginning? Are you specifying the output format? I find that that tends to eliminate many different errors that come up. The other thing is using some ensembling strategy here, or not even ensembling but to try multiple models and then see for certain inputs where do the models substantially differ in their outputs and use those in prioritization of human review and that might surface failure modes a lot faster. So to answer your question, short story, I don't think you can get rid of manual validation to some extent, but you can try to reduce samples.

Perfect, thank you. Thanks for the great question, Mitch. And thanks for elucidating all of that, Shreya. That actually answers partially a question that came to mind for me when, when we started talking we talked about how like accuracy and cost are really important and particularly when you have like really large corpora of docs. Right. It can get prohibitively expensive. So I know you're thinking about that on the ETL side of of course. And well the pipeline side of course when you introduce LLMs as a, as a judge, that is another side where costs can, can blow up completely. So I'm wondering how. And there's kind of a juggling you have to do with these kind of different costs and trade offs there. So how do you think about the different types of costs that can be incurred?

Yeah, my recommendation for judge to minimize cost is to try to use the cheapest model possible and limit it to binary evaluation. So whether that be pairwise comparison, output A is better than output B. Basically like any call that has single token or like very short outputs. Because remember you pay the most for output token cost. The other thing with LLM is judge is you can order the requests in a way that leverage caching. So I don't know if you're familiar with the model providers caching pricing model, but for OpenAI I think you can save 50% between all the LLM providers. You've saved between 50 and 90% on subsequent calls that share the same prefix of the prompt. So in this case if you want to judge three different things on some output, the output's going to be cached. Make sure you specify the criteria to judge after the output that you want to judge and then just have binary evaluation criteria. So you're really, really minimizing the cost there. If you find that you know you're using the cheapest model possible and you have really ordered your calls to like take advantage of context caching and they're all binary. So single token outputs, then the only thing you really can do there is like judge fewer outputs. Right. Because you've already minimized on all the fronts you can. But in practice I don't see people minimizing on any of these fronts.

So do you see that that will be something which people do more and more as they incur more and more costs?

Absolutely. I mean people want to save money all the time. I think we will see more specialized systems for LLMSjudge that will do this optimization for you, like reordering of LLM calls and so forth. Like we're already doing this in doc etl. We're trying to reorder all our LLM calls to save as much money as possible. So the more you can have a declarative system maybe for LLM judges to do this the better. However, I feel like we're like quite far out from that because people don't know how to write LLM judges in the first place and the evals vendors don't really know how to do this either. So I wouldn't expect any time soon.

Well also I don't know if things have changed, but last time I looked a lot of the eval vendors, a lot of the eval vendors and framework vendors always talk about judge beforehand labeling in their docs and tutorials as well. Which which is a challenge. I don't want you to give any away any shreyashanker secret sauce, but I am interested in if you're thinking about productizing any of any of the things you're working with.

Oh nothing. I think everything. I welcome open source. I'm a researcher trying to be a professor applying for jobs this year. So nothing.

Yeah, I appreciate we're at time. Do you have time for one more question? Totally cool.

If not sir, one person. Yeah.

One thing I very much appreciate about the workflows we've discussed are how you get humans in the loop. And as I mentioned we had Jason come and give a guest talk today and Innis Montani from Spacey an explosion and one of the reasons, I mean I love the work all of you do and appreciate all of you is how like humans are at the center in a lot of ways and figuring out how to get humans working with the machine. I'm wondering how you think about in these processes and workflows generally how humans fit into it and what does it mean to treat humans as like first class citizens in in these workflows.

Yeah, there's a funny phrase in the HCI community like people don't say human in the loop anymore. It should be like computer in the loop because the loop is around the human and not the human around the computer. I don't even remember but I thought that was funny. I don't know. I think I don't have a great answer to this question. So I feel like I'm trained in a lot of HCI research methodology and I'm in the HCI research community so that informs a large part of it. Every solution that I build. How do you think about designing it from like what is the user experience going to be and then second then what are the technical challenges that need to be solved now, given these user problems. But yeah, sorry, I don't have a great question to that or sorry, don't answer to that other than like, think about users.

Yeah, no, and that comes across in the workflows you've. You've described as well. So I really appreciate that, Shreya, as always, I learned so much from you. There's so much I need to go away and think about it. I just want to go and lie in the sun and cogitate on all of the. Thank you for your time and expertise and wisdom and I know how much you got going on at the moment, so really appreciate you making the time to speak with the course.

Yeah, of course, of course.

Great.

Thanks for having me.

Absolutely. I can't wait to see you later this week in your course as well.

Yeah, sounds good. All right.

All right. Thanks once again, Shreya. Thanks everyone. See ya. Thanks for tuning in, everybody, and thanks for sticking around until the end of the episode. I would honestly love to hear from you about what resonates with you in the show, what doesn't.