How to Process Documents at Scale with LLMs

Welcome everybody. Today we're very excited to have our very own Shreya, Shreya Shankar, host the session. And today she's going to be talking about the culmination of all of her work over the last 5 years and beyond. This is her website, sh-reya.com. Visit it. It's kind of a compilation of a lot of her work. So Shreya has done a lot of things in her career. She started working with MLOps. And more recently working on data processing and how LLMs and AI can apply to that. And also Evald, as she's done some work in academia, she kind of pivoted a little bit to work in a startup for a while. Then she came back to academia and she's a very unique academic. I've always been very surprised by the flavor of what she produces. I'm gonna tell you why you should pay attention. It's not your typical academic body of— Shreya skews really, like, very closely to applied. And let me just show you what this means. So, so Shreya does a lot of interesting work, uh, not only with LLM ops, but also with human-computer interfaces and UX design, and thinks deeply about tools. So all the different pain points that you have when developing AI applications, Shreya tends to focus on that and tries to find ways to make your life easier as a developer and as an AI engineer. What you'll find is not all papers are like this that Shreya is involved with, but a very high percentage is you'll see working software, user interface, a breakdown of an industry problem that people are facing when they're developing. Products with AI, and she'll present software, a user interface, a workflow, and some algorithms, and also all the, you know, work behind it and the studies behind it. And so she'll often also survey lots of users, do lots of interviews. And so what I like to joke with Shreya about is, you look, let's look at another one. This is called RAG without the lag. So this is a This is a really nice paper. It goes into developer workflows of how you can iterate and debug retrieval. Again, we're presented with software, user interfaces, exploring user interfaces. And if you read this, there's like studies on the productivity impacts that developers had while using this. Similarly, there's something called DocEtl. Now this paper. Talks about algorithms and effective ways of processing documents at scale with AI. But then there's also this paper that builds on top of that, which is kind of what I would call the product layer almost, which is again, you have software that helps you interact with these tools in a very applied sense. And this has been, these ideas have been shipped to industry. There's companies that are using these tools. A lot of this stuff has been adopted by various vendors in the AI space because it's, it's all very real. So what I like to joke with Treya about is these are all startup pitch decks in my mind because they have all the sort of ingredients. They have working software. They interview lots of users and their pain points. She does a lot of product discovery. She's solving an applied pain point. And she's, has building a product and she's, she also, she doesn't stop there. She works with companies and uses these products to solve problems. So that's why I'm really excited to have Shreya talk about all of this work in a cohesive way. So that's why you should pay attention to this session. And so I'll just hand it over to Shreya.

Wow. Thanks for the amazing introduction. I really appreciated that. I'm really excited to share my work on AI-powered data processing and document processing. I want to start out by telling people about data systems. I work in data systems. Most people are already very familiar with data systems, but I think it's one of the greatest success stories of computing. They've been around for quite some time. There's been 5+ decades of work, and we've done groundbreaking research in data systems all these decades. We've built new subfields of work that still live on. We've also been able to collaborate with a range of fields in computer science, pretty much every area in computer science. You can see those techniques being infused into data management systems. I think as a result, we've been able to have amazing reach, amazing impact. There's so many companies out there that build data systems. It's a trillion-dollar industry. Every application and every company uses some sort of data management system. So now what's next, right, for the data management community? I'd like to think that we've only seen the tip of the iceberg. A lot of these data systems in the world only focus on relational data, or you might think of this as tabular data, structured data, data that lives in tables. There's a whole lot of data out there in this world that people really want to query, and none of our data systems can really help people do these kinds of queries. And I'll give you an example kind of unstructured data processing task. And this comes from our users, people that we directly work with, with the Doc ETL project, which I'll talk about in a little bit. But these public defender data analysts want to help represent their defendants very well. So there's a lot of data that corresponds to every defendant, and this is very heterogeneous data. So think like court transcripts, police reports, news articles, even images, videos, whatnot. And the user, the data analyst, wants to figure out, are there any mentions of racial bias somewhere in the case? Because maybe if that's true, we can figure out how to make the sentence more just or fair for that defendant. Now, why is this also interesting from a technical perspective? Well, there's a lot of reasoning that's required in order to do this task. Something like implicit or explicit bias is somewhat subjective, difficult to find. I can't really write a program to do it. And to do this at the scale that the public defenders want would cost so much money if they wanted to analyze all of their data all of their cases with the state-of-the-art large language models out there. Clearly, people need better solutions in order to do this analysis. Just having AI doesn't quite solve all of the unstructured data processing problems out there. Now, in the data management community or in the data systems community, there's a new field of data processing emerging called semantic data processing. The idea behind semantic data processing is you have your favorite data processing operators, such as map, reduce, filter, etc. You express each operator in natural language, and the outputs of these operators can be very open-ended. So to give a concrete example of what a pipeline with semantic operators might look like, is say I have a data frame or a collection of court transcripts. Think of that as a data frame with just one column that's text. Maybe I'll run a map operation over every row in that data frame that's going to extract the name of the judge and some statements that are indicating bias. Then I'll want to do a semantic group by or reduce, which groups by the judge and creates a summary per judge. I'll go into this example in a little bit more detail later on, but I just want to give you a flavor for how the data systems community is thinking about using AI and fusing those into existing data processing systems like Spark, like AI SQL and whatnot. And this is not just something that I'm inventing, like, oh, here's, here's this new way of thinking about AI and data processing. Actually, it's been shipped across lots in lots of different industrial databases. For example, Databricks introduced this a few years ago. It's in the Google BigQuery, it's called AI SQL. It's in Cortex AI since it's Snowflake, which recently announced a huge partnership with Anthropic, like yesterday. It's in DuckDB. All your favorite databases support this notion of semantic operators. So my research really looks at two big questions in semantic operators. On the system side, how do we make them scalable? And then on the steerability side and the user-facing side, AI never works off the shelf. AI needs to be controllable. So how can we enable users to control these pipelines much better? I've done work in both areas. I'm definitely not going to talk about all of them today. I'll talk about a few in each area. And I would— what I do want to point out is while my work is quite grounded in data processing, I've been pleasantly surprised by the impact my work has had outside outside of the database community. So not just ideas integrated into industrial databases, but also adopted in bigger software systems and also powering data analysis and actual stuff that society uses. So without further ado, I'll talk a little bit first about the Doc ETL system, the system side of things. Both of these are very important sides. Assume for now that users find utility in authoring these semantic operator pipelines. I'll talk about the usability stuff in the latter part of the talk. But for now, let's talk about how we build these systems to accelerate semantic operator queries. In my PhD, I led the development of DocETL, which is an open-source system for semantic operators over text. Let me illustrate some semantic operators to you. First, the semantic map operator. Now, in DocETL, a dataset is a collection of dictionaries. Think of this as a JSON, so a list of JSON objects. Now, every attribute, think of it as a column in a table. You can map this to a table or a relational database if you would like. Now, if I wanna run an operator on such a dataset, a semantic map operator, first I define the type of operator, map, which means one-to-one, and then I define my natural language description or the prompt. So maybe that's extract all the statements made by the judge that indicate some implicit bias with some explanation. Then I define an output schema, this third part. The schema, think of these as new attributes that are going to be created in each document when I run the semantic map operator. Results from running the semantic operator is a dataset of the same size, but now two new attributes are created perhaps by a large language model or some LLM-powered algorithm under the hood. The same number of documents that comes in is the same number of documents that comes out. So that's a semantic map operator. A semantic filter operator is very similar, except for the operator is some sort of yes or no task. So keep documents that satisfy this condition described in natural language. Did it in— indicate implicit bias? And the resulting dataset, same shape, fewer number of documents. We just drop documents that don't meet this criteria. So think of semantic filter as a semantic map, but with filter semantics. The third reduce— sorry, the third semantic operator I'll describe, the last one, is a semantic reduce operator. Now this is a little bit more complex. The idea here is, say we have our same document collection of court transcripts and we have another attribute that's the judge name. Maybe we want to give a summary for every judge, judge's transcripts. So we specify a reduce key, which is the name of the judge. Then we specify the prompt or the task that is going to be applied to every group of documents. Every document is assigned to its judge group, and then we have a summary that's created per judge. Now, the data input and output looks very different. We have the AI has generated entirely different summary, new summary, and there's fewer number of documents because now every document pertains to a group. So those are three very, very different and powerful types of semantic operators, and what makes the semantic data processing paradigm so powerful is the ability to compose together operators in a pipeline. So for example, if I had my dataset of court transcripts, which is just one single column or one attribute, maybe first what I want to do is extract the judge name from each transcript, and then I wanna group by that judge name, and then come up with the summary per judge. Okay, so this is just an example of a real query that somebody has written. I've simplified it quite a bit, of course, the prompts at least. But now you can see the kind of like powerful data processing that's enabled if I'm able to orchestrate these kinds of semantic operators. So now the question from the data systems perspective is, okay, people have authored this, they find this output's meaningful, Okay, how do we run this at scale and make sure that it works, that people actually get value out of it? And the way that we like to think about it from a data systems perspective, of course, is how do you optimize this query? And optimize is a very broad here. It means how do you make sure it's accurate and how do you make sure it's cheap? Because if you're going to run this on tens of thousands of transcripts with each transcript hundreds of pages long, if you're going to run this with state-of-the-art models, it can cost you lots and lots of money, as I'll show you. Later. So how do we do query optimization? I'll bring in a mindset of traditional database query optimization. There are kind of 3 components to a query optimizer. First is what are the space of query plans that we're going to consider? Second is how do we model the cost of each query plan? Because what we're going to end up doing is searching over these possible query plans and picking the one with the cheapest cost. And I'll go through this exercise of kind of showing you where each of these kind of break down for the semantic data processing pipeline, or at least what we know how to do breaks down and all the new things we had to come up with in research. So first, let's talk about the plan space. Okay, what does that even mean? In traditional database systems or something like Spark, right, what are the different plans that are considered for a Spark query? Well, if you think about the operators that are in a Spark query or a data pipeline, right, we have different algorithms for implementing all of those operators. For example, in a join, you could do different types of joins. For scans, you could do different types of scans. Now, you don't need to know what all of these are, but the point is that there's, there's a variety of different implementations of the pipeline that all could vary depending on the characteristics of the data and the query. So what does that kind of look like in the semantic pipeline setting? Well, there's all sorts of implementations you can have of these semantic operators, right? I just said map and reduce, but You could have different large language models that are running each of these operators. You could use different prompting strategies. You could use ensembles of LLMs, right? Pretty much any strategy that from any LLM-powered algorithm that you can think of to execute a semantic operator is super fair game. So we did that. We tried coming up with some very good implementations of semantic operators, put it in front of real users and journalists and see what happens. And they all said that, man, this is not accurate. For my real-world tasks. And I'll give you an example of this. So for ex— maybe the pipeline first has some sort of map operation to create a very comprehensive summary, right? Most people, when they want a summary, they actually want their summary to in— include many different attributes. So maybe you're summarizing very long transcripts and you want there to be 4 parts to the summary analyzing different aspects of the transcript. You might run this with a state-of-the-art language model like Gemini 3.0, and find that the output is simply not accurate enough. The transcript is hundreds of pages long. Oh no, what can we do? LLM is not good. We've maybe missed two parts here. And you might think, okay, so what, right? Like, we'll just wait. Maybe a better LLM will come out next year and solve the problem. And like, we just kind of table until we get something good enough. Well, what we realize is that even when the operators are accurate, often users will write the semantic operators, specify their favorite large language model to do it, and it's not well scoped to the large language model. So perhaps maybe this operation is very complex and only a model like GPT-5.1 and Gemini 3 Pro are the only ones that can accurately do the semantic operator or get the correct answer. So if only expensive models can do the operation, then any plan that we consider in our query planner is going to be very expensive. We just cannot run this at scale. The interesting research questions become, okay, can we rewrite these pipelines to somehow be more accurate and also somehow be cheaper? What is the right framework to think about rewriting these semantic operator pipelines to be better scoped to large language model capabilities? So we draw inspiration from query rewriting in traditional databases. So it turns out that when you execute a database query, the query optimizer does a lot of rewriting under the hood to make it a good logical form. So some of these you might have heard of, selection pushdown. Some of these you might think make sense, like if you have two numbers that you're adding in your predicate, like you might as well just only add those once so you don't have to keep adding it every single time you look at a candidate, you might reorder your joins, you might rewrite your subqueries, things that will be better suited to optimization. So the question to ask is, okay, can we— what does a rule-based engine look like when optimizing semantic data processing? And this is different for semantic operators because these semantic operators are fuzzy operations, right? They're described in natural language, so you kind of have to understand the semantics of that to understand how you're gonna rewrite it. And you might need to rewrite operations that already, that are defined on atomic units. So in a database, every row is an atomic unit pretty much, or like every cell in a table. In doc ETL or in document processing, a user might think of a document as an atomic unit. But if that document is like tens of thousands of pages, a large language model will never be able to run accurately on that. And we need to somehow be able to break that down even though it was atomic or considered atomic by the user. So these are the challenges, right? I'm, I'm setting up the stage for thinking about rewriting semantic operator pipelines and how we have to think about it differently. So what we realized in the DocEtl project is that you actually can systematically decompose tasks. And I'll give you an example. So in the same kind of implicit bias example, imagine we have a map operator on a very long document to extract all all statements of implicit bias. It's not gonna work if I run it with Gemini 3.0. If there are like hundreds of statements of implicit bias, it just simply will not accurately recall all of them. But one way to improve the recall is to break it up as follows. First, I split this document into smaller chunks, then run the large language model to do the task over each chunk. Now take those results, feed them into a semantic reduce operator, and then get the same answer that I hope will actually be accurate. Now these are all new operators, right? This is an entirely different pipeline. You need to instantiate them in some way. So I need to have some chunk size that's defined. So maybe my split operator says 1,500 words per chunk is pretty well scoped to the LLM. Maybe my map operator, I need to rewrite the prompt a little bit to be specific to processing using that chunk. And then the reduce operator, maybe I need to come up with a good prompt that indicates, oh, I want to combine all the extracted implicit biases into one list. And so you can probably imagine, right, like, oh yes, this kind of rewrite makes sense, right? Like, I can now improve the recall, I can improve the accuracy of the outputs if the LLM is looking at smaller pieces of data. So this is one way to do systematic data decomposition. You don't have to just decompose around the data axis, you can also decompose along the task complexity. Now, the real-world settings, right, nobody writes those simple one-sentence prompts, people actually write really detailed prompts. They define implicit bias in really great detail. Well, what we could do is rewrite this into the following pipeline, each map operation. Does a different component of implicit bias is defined in the original task. And then we have a final operation to unify all of those results together, thereby hopefully improving the accuracy. So we generalized this in our first DocETL rewrite— or sorry, the first DocETL paper into the notion of rewrite directives. And rewrite directives are these kinds of templates that describe how to rewrite subsequences of operators. So here are the two examples that I showed you previously. There are more, like you can imagine inserting a map operation to make any semantic operator easy, not just a semantic map. And the one thing that's interesting that we do is that we use LLM agents to instantiate them. So something, so we use LLM agents to come up with the prompts of the new operators that are substituted into the query plan. The prompts. The output schemas, the chunk sizes, whatever it is, because those decisions are gonna be very specific to the data and specific to the task. Now, as part of these rewrite directives, in our original paper, we came up with 13, and in a follow-up paper, now we have 30+ that's currently in DocEtl. We had to come up with new operator types to enable these rewrites, right? It's not just semantic map, reduce, filter anymore. We also have these non-LLM-powered and some LLM-powered operators. We came up with a bunch of new ones. But before I go into that, I also want to show you that you can use rewrite directives to also improve the cost, not just the accuracy. So for example, I could combine two operators together. Maybe two of them are fairly easy to do. So I can make the entire pipeline cheaper by just fusing two operators together. And then I can eliminate one LLM call, basically one entire LLM pass over the documents. Making it cheaper. My favorite, some of my favorites, one is you can replace different operators, semantic operators with code. So some of these that are simply just adding up the results together or like concatenating results into a list, turns out you can replace those with an agent-synthesized Python function that simply just concatenates, right? That's zero cost compared to the LLM. And then my favorite one is, super, it's a little bit complex, but very inspired by projection pushdown in databases. But the idea here is if that your task doesn't need to read the entire document. So if there are cheap ways of identifying the portions of the document that are relevant to all of the tasks in your pipeline, then you can synthesize an operator to just quickly find those. So maybe it's just a keyword search that'll find those. Maybe it's embedding-based, I don't know, lookup that are gonna find those small portions. And then push that operator down in the query plan. And so now the size of your document is much smaller for all of the downstream semantic operators. So this overall can reduce cost if you're able to synthesize such an operator. So those are the flavors of rewrite directives that we consider. Rewrite directives, I think, are the biggest contribution that the Doc ETL project kind of gave to the database community. And I'd also have to— I also mention as coming up with these rewrite directives, we came up with new operators. I'll talk about two of them. For example, one of them is the gather operator. We realize when you split documents into chunks, when you look at individual chunks, it's very difficult to make sense of it in isolation, right? If you feed that into a large language model, it's just simply not gonna understand who he is. So our solution is to basically augment each chunk with useful context that's surrounding or other context in the document that's needed to help make sense of it. For example, you can think of this windowing idea, like always have the previous chunk appended to a chunk that you're looking at. You can think about having a summary of all previous chunks. You can think about having the metadata or some table of contents included that's always in the beginning of the document. Well, you can also include that in every single chunk so the LLM can make sense of it in this split-map-reduce-rewrite. Another operator we had to do was this resolve operator. We basically elevated entity resolution as a first-class citizen. Why do we need it? Well, if you wanna group by an LLM extracted attribute, often the LLM extracts things inconsistently, so you need to do some sort of entity resolution before you're doing the group by, so you can make sure all of the documents are in the same group that you intended. So if I'm gonna bubble up back to where I was before, what's the plan space looking like for semantic data processing? Well, it's different. It is kind of what you think. You take strategies from the AI/ML community for how to execute these operators. But also, we come up with this notion of rewrite directives. And we also instantiate them with LLM agents. So they can be very, very tailored to the task that the user writes. So next, I want to briefly talk about the cost model. So say I've got a collection of plans. Or a way to enumerate the plan space for a semantic operator pipeline. How do I know which ones are good, right? Which ones are cheap? For example, in traditional data processing, imagine you're writing a Spark pipeline. There are many different Spark plans. Spark wants to know what's the lowest latency. So how do they do this? Well, they have good ways to estimate kind of I/O, CPU cost. They use cardinality estimates of their datasets. And ultimately, like, these kinds of cost models are very quick, right? You run them, you see, you get, you figure out what the cost is of your query plan. Now, in semantic data processing, things are different because cost is not just latency anymore. These semantic operator plans can vary in accuracy a lot, and there's no point doing something with very low accuracy. We also now have to model accuracy. We have to model latency, accuracy, and dollar cost. We're paying OpenAI. People don't want to pay millions of dollars to OpenAI for a query plan. It's hard to estimate accuracy. If you think about these semantic operator pipelines, I have a pipeline like this, how do we know how accurate it's going to be? If we do some rewrite, say I'm going to do a task decomposition and I'm going to use cheaper models, who here can tell me exactly what accuracy this rewritten plan is going to give me? Unfortunately, in order to estimate the accuracy of the rewrite, the only solution I have is to execute it, on samples, which kind of sucks, right? Because that is very expensive and I have to wait a very long time to figure out, okay, after this plan is executed on samples, what's the sample accuracy? So what we thought about is like, okay, is there a— how do we think about coming up with rewrite directives that guarantee that the rewritten plan is within a target accuracy of the old plan? Because now if I can do this rewrite, and by construction I'm able to guarantee that the accuracy is very close to the original one, then I can avoid rerunning or I can avoid running this candidate plan on samples to estimate accuracy. So if that's a little bit to wrap your head around, that's okay. I can get into that at the end if there's Q&A about it. But for now, like, assume what, what if we can do this? Well, it turns out that we actually can. What we can do is we actually look at the related literature around cost optimization from the ML and video processing, video graphics community. So the idea is that you can actually reduce cost while maintaining a target accuracy by routing inputs through a sequence of models of various costs. So I'll concretely explain what that looks like. Say I want to run my map operation over lots of long documents. Well, first what I can do is route that through a proxy or a cheap model. Now if that cheap model is very confident in its prediction or its output, then I can just accept it. And if it's not, well then I can route it to the original plan or the Oracle plan and then accept that prediction. So this kind of cas— model cascade architecture works really well if I'm able to route most of my queries through the proxy only. Like I can bypass going through the oracle model. And this also works only if I'm able to pick some confidence thresholds to meet the oracle accuracy. So that is the proxy model is correlated with the accuracy for many, many documents. So in the literature, in the model cascade literature, the algorithms that people typically do is they run the proxy on samples, they iterate through various thresholds, confidence thresholds in those samples, They come up with the minimum threshold that is going to meet the oracle target accuracy and they ship that off and it works. Like that, that is a good algorithm that has worked many, many times over the last decades. So what we did was we thought, okay, how are we going to apply this idea to our DocEtail style rewrites? So we came up with this generalization of the model cascade idea called task cascades. So it turns out that models are not the only factor of cost. Are not the only thing you can vary in a cascade. You could also vary how much of the data or what slice of the document an LLM sees. And you can also vary how complex the operation is. So there's no, you don't need to send the same prompt to the proxy model. You could send a simpler prompt to a proxy model that simply correlates with your original operation prompt. So this kind of unifies insights from both rewrite directives where we're able to create simpler or cheaper variants of the task that are predictive. As well as ML Sys Insights, where if we run on smaller document samples or we kind of manage our KV cache very well, then perhaps we can reduce overall cost. And so very broadly speaking, like visualizing a task cascade, imagine I have an operation on these court opinions or court documents, and I have segmented my document into pieces. Now the Oracle task or the pre-rewritten plan is as follows, just running that operation as is with the most expensive model. At the time we wrote this paper, it was GPT-4o, but substitute that in for whatever model it's gonna be. And you run this on the full document. So now a bunch of proxy tasks can be as follows, right? Maybe it's varying the operation. A simpler task is, is there any lower court mentioned? If there's no lower court mentioned, then there's no way it can be overturning the lower court, right? So This is a simpler version of the task that correlates a lot with the original. We can run that on maybe a smaller portion of the document. Like that, we can vary different tasks. Does it say— does it have specific keywords? Again, something that a cheap proxy model can do much more accurately and you don't need to do on the entire document. The idea of a task cascade is to then assemble the sequence of tasks in order to minimize cost. So you want to resolve as many documents early on so that very few documents pass to that oracle task. So in our paper, we formalize the idea of this rewrite, the task cascade rewrite, and it's a little bit involved to construct these task cascades, but because we can guarantee the accuracy probabilistically, it's okay. We don't— it's okay to spend money instantiating this directive because we know it will be useful. We show that in our paper, constructing optimal task cascade is NP-hard in the task space, and that motivates a greedy algorithm to do it. I won't get into the details. I also won't get into the details around the accuracy. I think that if you're interested in reading the paper, you absolutely should look into that. But the point is that constructing these task cascades and these rewrites, we're able to reduce cost compared to the pipeline before rewriting to 86% on average, which is awesome, staying within a 90% target accuracy. So bubbling back up to the cost model here, Estimating accuracy of plans is very expensive because you have to run it on samples and figure out. But for some certain rewrites, for task cascade rewrites, we are able to guarantee that accuracy by construction. And so we can help reduce the cost of exploration there. All right, so the last thing I want to touch on before moving to the latter part of the talk is the search algorithm. So we talked about how to enumerate plans for semantic operators. We talked about how to estimate the cost and accuracy and latency of semantic operators. Now how do you search over this plan space to pick the best plan? And in traditional data processing, there's a bunch of different query optimizers and databases, and the idea is to find a single optimal cheap plan. And the way that these query optimizers work is they break down your SQL query or your Spark pipeline or whatever into subexpressions optimize individual subexpressions, and then use a dynamic programming algorithm to figure out the optimal entire query plan. So what's different now in semantic data processing and semantic operators? Well, first is, from the user perspective, there isn't really a concept of a single optimal plan. Like, what does that mean? Plans vary in accuracy, equality, and cost, right? So maybe if somebody has a very big budget, then they're willing to get very high ac— or they're willing to search for very high accuracy. Maybe if somebody doesn't have a very large budget, they just want the highest accuracy within that budget. Well, then we're gonna get something else on that curve, right? So you want to be able to showcase multiple plans that are available and then have the user be able to pick. And then the second thing that's a little bit more subtle is that it's not great to locally optimize parts of these semantic operator pipelines. And I'll Give you an example here. So in traditional databases, we can. We locally optimize, compose optimal subplans. But say we have our semantic operator pipeline. Again, this is just something that was from an earlier slide. We had 4 map operators. Suppose I wanted to just locally optimize this third map operator. Maybe I come up with some great solution to do it, maximize the accuracy of this. Awesome. But now, if I want to consider that in context of the entire end-to-end pipeline, perhaps there are other operators in the pipeline that can correct or augment or change or interpret the results differently of these other operators. So in other words, the way that I execute my third operator actually depends on the implementations that I choose for other operators. So it's not optimal to then just individually optimize different parts of the query plan. We kind of want to consider a very global search. So in a recent preprint, we come up with a global search algorithm to look over these rewrite directives. It's very much inspired by Monte Carlo tree search mechanisms from the AI community, but we treat this as a graph search problem. Every node in the graph is a complete pipeline. Every edge is some sort of rewrite, so instantiation of a rewrite directive, and we keep exploring different query plans until some sort of budget is exhausted. So this is actually a real sample of one of the public defender's workloads. They authored their semantic operator pipeline, used GPT-5, and it gave them— it's actually not accuracy— F1 score of 47%, which is abysmal. Like, it's not usable. So they needed something more accurate, and also they wanted something cheaper because they didn't want to spend that much money. So by searching through our rewrite directives, right, we were able to find something far more accurate, 1.5x. The F1 score, and cut the cost by 95%. So how do we do that? Well, we go through this 3-step search for selecting the pipeline we want to rewrite, then rewriting the pipeline using the LLM agent, and then evaluating on samples when needed to get estimates of cost and accuracy. How do we do selection? Well, we use a UCB-inspired algorithm. Again, I was talking about MCTS. The idea is that we select pipelines that we believe are very good to rewrite. So they have high accuracy themselves and they lead to children that also lie on the Pareto frontier. So we prioritize rewriting those pipelines. And then we use LLM agents. Say we want to rewrite this pipeline circled in red. We use LLM agents to then instantiate the rewrite. So it figures out, given our directive registry of 30+ rewrite directives, which directive applies best here. Because it's— the LLM agent is allowed to read sample data, look at the history of pipelines executed, whatnot, to make intelligent decisions. And then it also does the instantiation. So it comes up with the prompt for the new operators. So say it wants to create a new summary operator to reduce the cost of this pipeline. Well, then it will come up with this prompt to summarize the document, and then that smaller version of the document will be submitted to all the downstream operators. And then we execute that on samples, record the cost and accuracy, update our estimates and so forth. So the final thing was, okay, we did so much, how well does this do? So we take some real workloads, how well, how accurate of plans are we able to find and how cheap of plans are we able to find? So we've got a bunch of workloads inspired by real users of Doc ETL. We also use 3 different baselines. So there's a semantic operator system at Stanford called LOTUS. That they're building. There's a semantic operator system at MIT called Polimpsest and the new optimizer for it called Abacus. And we basically allowed these systems to choose from 11 different models, like all of the GPT variants, all the Gemini variants. And we found that every time we produce the most accurate query plan, which is very exciting, um, and often it can be 2x times the accuracy. Of what the other systems can find. This GPT-5 agent baseline is basically, I gave the Doc ETL query engine to GPT-5 and the documentation for Doc ETL, and GPT-5 is allowed to test pipelines on samples, do whatever it wants, and give us the best pipeline that it thinks it can come up with. So this kind of underscores that, okay, having these carefully designed rewrite directives for data processing can outperform the agent. We're able to find plans that are more accurate. And the other thing is, if you look at the most accurate plan that each one of these baseline systems found, DocEtl's optimizer is able to find a plan with matching accuracy at a small fraction of the cost. So this also underscores, right, if you're able to search the Pareto frontier and search well, search intelligently, maybe you can find very cheap plans that are within budget. So wrapping all of that up, optimizing semantic data pipelines here, we talked about, okay, what are the different types of rewrites that we wanna do in semantic operator pipelines? Well, in Doc ETL, we came up with rewrite directives to improve accuracy and cost. And the second thing we talked about is really search and cost modeling, right? We search over complete plans in Doc ETL. This is very different from traditional data systems. We use learned heuristics to come up with what plans are good to rewrite. We use statistical tools to estimate whether rewrites will be helpful, going back to that task cascades idea. So wrapping that all up together, when you think about how to build scalable semantic data processing systems, I like to say, well, we don't just throw out all the database foundations ever. We definitely build on top of all we know for how to build good data systems. We also are not afraid to adapt them pretty significantly for the realities of LLM workloads. All right. So that was the meat of the talk. I have about 10 minutes left of content on thinking about how to make really good experiences for people to author these kinds of semantic operator pipelines. All right. So I'd mentioned the word steering, and steering is really this idea of trying to get the AI to do what you want to do. So in complex tasks like doc ETL pipelines, steering is very hard, right? Because you're trying to express a very complex idea or concept, something like extract racially charged statements, right? This is not a real prompt. Nobody can write such a prompt and expect to get good results from even the best LLM in the world. Operator descriptions must figure out Users must figure out how do you express the fuzzy task very, very precisely. And often that involves encompassing a lot of edge cases in the data that they didn't know about. So concretely, right, expressing a fuzzy task precisely. If you have ever written a prompt, you know that this is hard, right? You have to think about, okay, what does racially charged mean? What are the different ways things can be racially charged? You have to figure out how to express that in English or some some way to communicate to the LLM. And then the second part is you might not be familiar with all of the inputs that are going to go to your system. And some of them might be edge cases and you kind of do need to explain that behavior or in some way maybe provide examples, whatnot, to the LLM in order to accurately do the task. So we realized that this, the steering problem was hard. So maybe we should build a specialized interface for people to write DocUCL pipelines. And so that's exactly what we did with the DocWrangler project. The DocWrangler project is very much inspired by work in good IDEs. It's two parts. There's a pipeline editor. Think of it as like a very notebook-y pipeline editor. The second half of it is the input and output inspector that allows you to richly look at the semantic operator outputs and make decisions on those. Now, when we built this, we thought, oh man, we can build all sorts of features to help users steer AI, right? I bet if I asked every person here, everyone could come up with some AI, cool AI feature. To help people use, I don't know, to write a semantic operator. But that alone was not interesting because you can come up with anything. What we really wanted to know was, okay, can we come up with a theory of tools to help users steer AI? Like a complete theory of tools. Where are tools needed? What is the complete class of problems users have so that anybody can easily build tools for users now on? So in order to come up with this mental model, we look to challenges in related domains. So for example, in data cleaning, many of you might be familiar with data cleaning or data wrangling. Now, in data cleaning, the user wants to clean their data before they do some analysis or machine learning or whatever on it. One challenge is knowing what is in the data, right? You can't write your data cleaning code if you have no idea what's in the data, what all the errors are in your data. And then the second challenge is, even if you know all the errors in your data, what is all the code that you need to write to clean it, right? You need to write all sorts of pandas expressions, all sorts of regular expressions, all sorts of weird transformations, right? Not everybody can author all of those exhaustively. So our community has come up with a bunch of tools for each, right? Knowing what's in the data. Think about structured data and tables. If you log into your Snowflake database, There's visualizations of your data, right? There's summary statistics. You can find outliers in your data. We have algorithms to help you do that with your databases. For knowing what code to write in data cleaning, we also have tools to help you with this. Folks might be familiar with Flash Fill, for example, in Excel. For, for prediction interaction, Jupyter Notebooks are a great example of a tool where you can quickly prototype different data cleaning or data transformations to figure out what works for your class. For your task, right? So we've come up with a good mental model of data cleaning challenges. Now what we realize is to extend this to the semantic operator setting, all we had to do was add a new island. So now we have challenges between the user writing the pipeline, the data that they're trying to run the pipeline on, as well as the pipeline itself, right? Because those are fuzzy and executed by LLMs in their own world. So you gain names for all of these three gulfs. And if you're familiar with Norman's Gulfs, in HCI, right, you'll know that Norman's Gulf applies to every single interface. So these are not gulfs in the sense of Norman's Gulfs. Not every tool needs to solve all three gulfs. Think of these as three gaps that users have to face, or three sets of challenges that users have to face in semantic data processing entirely. So I'll talk a little bit about them and then how we solved each one, or how we built a solution to solve each one of them, but you could build infinitely many solutions in each gulf. So first is Gulf of Comprehension, right? Unstructured data, we don't have good visualizations. We don't have good ways to, like, identify outliers in documents, right? We don't have algorithm for that. So how do we solve that? The second one is even if users know what's in their data, now they have to write the code or they have to write the semantic operator to process it. How do they express their complex query as some prompts? And then the third is even if they've expressed the pipeline very well and it works on, say, 100 documents, Maybe if you try to generalize that to your entire document corpus, LLMs simply make mistakes. They're imperfect, and so it breaks at scale. In DocWrangler, we contribute this 3 gulf framework, and we also build a feature to assist users in each gulf. For the Gulf of Comprehension, I'm going to ground my demonstration of this in a task from a medical analyst. A medical analyst goes through a corpus of doctor-patient conversations, And maybe they want to extract all the medications from each transcript. And this is just conversations, things like Zoom transcripts, right? So medications are just mentioned in text in there. Now the interface might look something like this. They write their semantic operator in the top part of it. Don't worry about what's in there. And then they wanna inspect the outputs because they, they don't really know what's in their data. So now the challenge with the Gulf of Comprehension is to just figure out what's in the data. So we have a feature so they can expand all of the rows one by one, read the outputs, kind of see how they compare. So on the left is the extracted medications, on the right is the original source transcript. And the key thing that we do here is offer the ability to provide feedback on the outputs. So this is our way of helping people find the outliers, find the patterns in the data. So perhaps they notice something interesting. Like, oh, when I see these medications, I also see the dosages associated with them. I indeed would like my dosages also extracted as part of the analysis. How would they have known this? They didn't know this at the time that they were writing the operation. They just, they discovered it by looking at the data. So they needed this tool to be able to look at the data and arrive at the conclusion. And similarly, they might run into other sort of realizations. They have decided that they also, in this one, they don't want over-the-counter medications extracted. Right, so they don't want Tylenol, they don't want ibuprofen, just prescription medications. They could note that as a note. And all of the notes in DocWrangler kind of persist in the sidebar, so users don't have to forget. And in practice, people can go up to, like, we've seen 25, 30 notes. People just really look at their examples, provide feedback. If you've taken the eval course, that this is open coding. You have to kind of do this process to get a good sense of what's happening in your pipeline, what's happening in your data. So now after this, for the next gulf, bridging the gulf of specification or specifying a good pipeline, we just turn people's notes into better specifications. So we provide an interactive interface. What you'll see here is some green and some red. This is a new prompt that DocWrangler has suggested. Based on your notes. And you can iterate on this as you, if you would like. You can directly edit it. You can add some feedback, have the AI edit it, whatever. We have an interface for that. But the point is here is assisted specification, right? People cannot specify the whole thing themselves. They need tools to assist in that specification. Then the last gulf, Gulf of Generalization, we find that users kind of get to a point where the operation gets very complex. Right? They keep iterating on the data. They keep iterating on their prompt. Prompt sometimes asks to do too much. Maybe we've tried to create 3 attributes in a single LLM call, and one of these attributes, for example, extracting the primary complaint, is not very accurately performed by the LLM. So what we do is every time the user actually runs an operation, we run an expensive LLM judge in the backend to run on a sample. And see maybe are one of these inaccurate. And if so, we'll suggest, hey, maybe you should decompose this operation and propose a decomposition à la doc_etl.rewrite directives. So that kind of brings it all back together. We replaced a pipeline in there. In this case, our rewrite ended up breaking down each individual attribute, I guess, into an operation. Like you could also imagine maybe we'd do a data decomposition. We'd split up the doc— the transcripts into chunks. I don't know, whatever is the most accurate there. Alright. So those are kind of how we went about DocWrangler designing for those three gulfs. Right? New interactions, new mechanisms for users to bridge their challenges. And we actually did an online— we did both an in-person study, as well as an online deployment and evaluation. We learned a lot of stuff from it. We learned that people invented their own strategies to bridge the three gulfs, which I thought was very fascinating. So for example, to bridge the Gulf of Comprehension, just knowing what's in the data, People would write these throwaway pipelines like summarize these documents, extract the key ideas, things that would just teach them more about the data before they actually did the analysis that they intended, which I thought was interesting. People also tried to repurpose their operations for quick validation. So we have histograms for any float-valued or numerical-valued attribute that Doc ETL generates. So people would just add like Boolean attributes numerical things so they could quickly kind of sanity check how did the LLM do. And we have a bunch of different observations that we write about in the paper across all of these pipeline runs. People from a bunch of organizations, at least this is like what DocWrangler was able to identify. We used DocWrangler to analyze the DocWrangler logs. We came up with a sample of companies of people who used it, which is exciting. And what I think is very cool also tying this back to the kind of research arc. Is being able to build such a solution also grew the DocETL community quite a bit. All right. Now, the last 3 minutes, I suppose, I wanna talk about kind of a third problem that we've been seeing that's underlying theme throughout pretty much all of the work, which is now when you have systems like DocETL and DocWrangler, right, how do you run those at scale and then know that the analysis is actually good? And working on this problem actually motivated the evals course, me working together with Hamel Husain Husain Husain. So pretty much people will very much resonate hopefully with these insights. Now when we thought about this question from the research perspective, I was, I was thinking about it from a very Doc ETL perspective, which is people come in to do their data analysis tasks with no-label data and their evaluation criteria is very complex. So when I looked at a lot of like these ML benchmarks and these literatures, a lot of the evaluation criteria or these kinds of RL evals, a lot of them were quote unquote verifiable. They're very easy to measure. For example, did the code run or not? But that's just not the case for a lot of data processing. So we thought, okay, how can we kind of improve the eval experience for people with these complex tasks? So one system that we built called EvalGen, the paper is called Who Validates the Evaluators? It's the most cited paper at UIST 2024. The idea that we had here was anyways, when people are running these kinds of batch LLM pipelines, they spend a lot of time waiting for the results, right? If you wanna run doc ETL pipelines on tens of thousands of documents, you're kind of sitting there and waiting or I don't know, doing your own thing before the results come back. What if we can use the user's input and feedback to figure out what makes for good and bad quality outputs? So that's exactly the premise behind evalGen, which is we built this kind of interface and this algorithmic flow that solicited labels on outputs as they were generated, then created evaluators based on those labels, and then was able to communicate at the end how well the outputs aligned with the user's intentions. And a video kind of looks kind of like this. We built it in the Chainforge AI. We, you suggest some criteria, you label some examples, just going through really fast. You get a report card and then you get a table of basically all of the LLM judges that we've created that align with the labels that you have generated. So we built this interface and again, like, think of this in the context of batch style tasks, right? So this is not just one-off AI task. This is, I have a pipeline that is gonna run on tens of thousands of users' inputs. And I want to figure out how to label, how to evaluate at scale. And when we did our user studies, what we found was very surprising, at least to us from a machine learning perspective, which is that the evaluation criteria drifts across these many outputs. So when users were grading the LLM outputs, they decided they want to change or refine their criteria. So a concrete example in a data processing setting that we did, was we did an entity extraction task and users were supposed to basically extract, or the task was to extract entities from tweets and one of the eval criteria was don't extract hashtags as entities. Now, when users were grading and they saw hashtags, they'd be like, yeah, like that's, that's not good, we don't want that extracted. But when they saw that happen frequently, they changed their mind sometimes. For example, what, Colin Kaepernick, a famous football player, they thought, hmm, yeah, I guess I said no hashtags entities, but this, this one is actually kind of correct. I'm glad this was extracted. And when they saw it again, they realized like, oh man, like the LLM has done this several times. I actually agree with the LLM here. I want to kind of change my eval criteria. I still don't want hashtags extracted as entities unless they're a notable entity in the hashtag. So that was crazy to us because we were like, oh man, like, we need these interfaces, we need these ongoing labeling processes to infer what evaluation criteria we should have. And users wanted to add new criteria, right, as they reviewed outputs, they learned about new failure modes. They also wanted to reinterpret criteria to better fit the LLM's behavior, like as I showed in this example here, you know, better fit the LLM behavior, that's something that you can only figure out after you've observed enough LLM behavior. And this also, if you've taken the evals course, kind of motivated how we go through an iterative process to kind of do eval— to evaluate outputs until the criteria drift kind of stabilizes to some extent. So all of this work kind of around DocWrangler, around evalgen, the papers that we've written have very excitingly been adopted by various tech companies. It's not that people have to adopt tools as is, it's really the ideas, intellectual merit behind them, but very, very popular LLM ops companies kind of adopted these ideas and the ideas have seeded the course kind of that Hamel Husain and I have co-taught. There's a really exciting article in the OpenAI Cookbook about kind of our eval workflow that we created together where we adapt kind of qualitative coding and iterative coding ideas to do evals. And the course has gotten a lot of traction. So with that, that kind of concludes all of the research portion of my talk. I talked about both the scalable side of semantic data processing. I talked about some of the user-facing, the AI alignment side of semantic data processing. And I remain super positive, kind of, we're so well on our way to kind of building this next generation of great data systems, right? You think about many eras of data systems, the next one is coming. We have already done a lot of work to get us there. We've kind of come up with new query optimizers, we've come up with new eval methods. Right? It's not just me. There's a lot of work in the data communities, the AI communities, the HCI communities. And there's a lot more that we can do that's going to get us there. We really deeply optimize on the system side. We cover the entire lifecycle, right? What does EDA look like for unstructured data? What does BI look like for unstructured data? Lots of exciting questions there. And thanks so much. Happy to take any questions.

Yeah, would it be okay if I ask you some questions?

Yeah, please.

The DocWrangler UX and workflow, I really love it. I find that when I saw it, I immediately recognized that it's not, it's not relevant to just processing documents at scale or in like, it's not, it doesn't feel only relevant to, let's say, the Doc ETL kind of work. It feels like I always want this kind of workflow. When iterating on any AI, because like I want to try prompts against a lot of different examples. I wanna see what happens. I wanna iteratively do that. I wanna refine my prompt. If it feels like a good UX, do you, have you thought about that? Like the general, the generalization of this UX, like how do you feel it applies across other problems? Have you thought about that?

Yeah, no, it's a good question. So when we came up with this interface, when we came up with DocWrangler and we came up with these three goals, we did not do this for the purpose of like generalizing to AI systems. I really deeply thought that this was specific to data processing where users just don't know what's, like it's very hard to read documents and know what's in your data. The tasks are complex and so forth. And only after doing the user studies, and people in the user studies who also are generally familiar, like build AI agents, like for their work and stuff. People in my user studies are typically, I always try to do half like engineers, people who are really in the AI space, and half like people who are not at all AI engineers, but do data analysis. So I'm like just trying to see what sticks throughout both. And all the AI engineers were like, oh my God, like I love this ability to give feedback. I didn't think to do, I didn't think that this was possible. And I think this speaks to your point about the generalization. Because when agents, people are programming agents, agents exhibit behavior that people don't really know what to anticipate. Agents make tool calls. Nobody knows what's happening in those tool calls. Are those tool calls going to generalize to all of the inputs? Even if the inputs are small and users have a pretty good sense of what those inputs might be, they still don't know how to anticipate the agent behavior. And I think that was like a key insight for me in terms of like, oh, maybe this model generalizes. And so when you and I taught the course, I was thinking, okay, how do we, like, what do we need to do to adapt this model? It probably will help. And the idea behind open coding on agent traces like looking at traces, looking at all the tool calls, providing feedback on kind of where the LLM behavior was a mistake. It was very key, right, in this evals lifecycle. Whereas in document processing, a lot of it is looking at the original source data. It's the LLM behavior in relation to the source. Like, I also want medication dosages extracted and things that I only see when uncovering the original data. So those are kind of, I think, the comparison and the contrast. I think the model definitely generalizes, but kind of specifically what you look at, like tool calls, might be different.

Another question I have is, and this is a serious question even though I give you a hard time about it sometimes, all right, I'd like to joke, is I really, it is quite fascinating that a lot of your work in papers really look like a product, like, because in, and whenever I show it to someone else, They're like, oh, where can we buy it? Because, and the reason that we feel that way is, hey, there's, of course there's the user studies, there's the theory and the grounding of that theory and all of that regular, like normal stuff. But then what we see is like, okay, we see software, we see user interfaces, we see, uh, all the user studies at like real companies and then kind of like all the sort of trappings of like a SaaS application almost, except like we're not paying. You can, this is all free knowledge and free, freely available stuff. But like, is that an accident? Is that how, why do you think it's, is that, is that like unique? Is there just some reason why a lot of your work?

I think a lot of database work is very app, like it's very applications oriented. Like work in data research, data systems research is all about like building systems that people can use for their data-oriented tasks. And like, as you can see, it's a trillion-dollar industry. So I think it's hard to like decouple the research from like having impact, like, or like being able to see it in a product. I think what's like a double whammy on my work that probably feels this way is like I'm also interested in HCI. And a lot of HCI work is like having good product sense, right? How do you build better experiences for users? And I think I'm more interested in the academic side of things around like, what are the novel interface paradigms and the novel ways that AI and humans have to communicate? And sure, that's also very product-friendly, but I think it doesn't, doesn't mean I personally am interested in all products and product development as a field. I don't know if that answered your question of— No, that— I appreciate the skills. I recognize that product development is a very important thing. I try to use good product mind all the time, but I feel like I'm an academic at heart.

On that point of kind of being academic at heart, so you're currently on like the CS faculty job market. So what kind of research program do you want to build that maybe not, maybe doesn't exist anywhere yet?

Yeah, I really love working in like grounded applications. So I think, you know, as you can see in my dark ETL work, like I build systems that people use and then I learn about things and then those become research papers. So I definitely want to like continue that way of doing research. I want to have an interdisciplinary lab. I want to do data systems work. I want to do HCI work. I want to do AI work and more, I imagine. I will broaden, hopefully will live a long life in career as a CS researcher. But more concretely, like in the short-term future, I really want to see this unstructured data thing happen. Yeah, I just, I feel like the roadmap is there and we just kind of got to build it. Well, I don't know how we're going to build it, but we got to build it.

From my perspective, this data processing, not even at scale, like data processing in general is really an important problem. And I don't hear about it as often as chatbots. Do you have an intuition like why that might be? Because I feel like there is a sort of like what is out there, let's say in terms of education and even, yeah, really like education around AI feels like it centers on a lot of like the chatbot kind of use case. But when I work with companies, it seems like more, there's like a lot more work in this data processing and like not as much as chatbot. So it feels like flipped. What do you think that is? Yeah.

Also, the data management community is actually extremely small compared to other CS communities. Like, the AI community is far larger. And I think there, there's still a startup culture for sure in data management, but there's fewer startups. So I think that the size thing is we can't, we try to do our best. We try to spread the gospel of data systems, but, but there's only so few of us. I think looking forward, I'm also very excited to do more of this kind of outreach and tell more people about DocETL and kind of the way of DocETL thinking once I get a job, kind of building new education around that, how to think in this way of like building scalable semantic data processing.

This is kind of a spicy question. It doesn't have to be, like I say, the perfect answer, but Okay. If you could force every— well, forcing is not— if you could, what is one paper you would recommend reading that is not your own? Like whatever comes, what comes to your mind that you think is maybe underrated?

So recently, I don't read enough AI papers. I should read more. I've been reading a little bit about like these DeepSeq tech reports. I find them very interesting, specifically like tricks that they do to I don't know, train their ML models or their LLMs. Another set of papers that I find interesting is just like how to do like RL training for agents that are specifically trying to do like retrieval. I think the closest thing to this kind of batch data processing that I see in the AI community is retrieval. It's a little bit different. The type of query is different. Like it's inherently like a semantic filter. It's just a semantic filter. Like we also think about other operations, semantic operations. But I find those papers interesting. From an HCI perspective, I think there, there's a lot of work out there, but I'm excited about some of this like human-AI collaboration work. So specifically, what are better methods or ways to benchmark how well AI collaborates with humans? A lot of those are preprints, so take them all with a grain of salt. I try not to read into a paper too much. Until like it's been accepted to publication. Trying to think what else. My colleague and collaborator JD Zamfrescu Pereira has some great papers on kind of how end users interact with AI. There's like a paper he wrote called Why Johnny Can Prompt, which is really fun, um, that I also recommend people read just to like get a taste of what are, how are you gonna end up bridging this gulf of specification? God knows.

No, that, that's, that's really interesting. So the question has come up, so you're a data systems person. And so I think this question is really relevant to you. So it always comes up in almost every conversation I have about AI. People love the idea of a graph database of like, hey, like, can I use a knowledge graph? Now, can you expand, like, what are your— I get those questions a lot, and like, can you comment on like, okay, how do you feel like whether you should reach for these tools or not? And what do you, how do you think about it?

Yeah, I think it's very often misleading. I think there are some cases where, no, there are very few cases where graph databases make sense to use. Um, but I see more often that people use them and make their lives harder. And for most Docketail workloads, actually think of them as ETL style workloads. I have lots of data. I'm gonna transform all of that data in some way and get some output of some shape. Like that shape is not always a graph. And often when it is a graph, I always ask, okay, why do you want a graph? Like, what are you gonna use that for? And people actually are like, oh, that's a good point. Like, I just actually just wanted like entities extracted. And I wanted summaries of the entities, or I wanted groups of the entities. I didn't like necessarily need it to be represented as like all pairs of edges between all entities. Like that's not what I was looking for. I wanted to group by on like a particular attribute. And so I think, okay, like maybe graph is not the setting for you because when you use something like a graph database, then like for any downstream tasks you want, you have to figure out how to wrangle that into the shape that you want. And that will maybe use something like docetl. So you spend all this time putting something into a graph database only to take it out to like run a tool like docetl. I don't know. Yeah. So my advice is always like really think deeply about what you're going to use the graph for.

There are some very specific, let's say technical questions. So there's a question. Okay. How do you handle multi hop reasoning across multiple pages of a document? It was hard to figure out what is the right chunk size. In our experience dealing with large contracts where the data might be very different in different parts of the document. How do you, how do you tackle that kind of problem?

Yeah, it's a good question. So our optimizer searches over different chunk sizes and then tries to pick the chunk size that has the highest accuracy. The other things is like we have that gather operator that augments each chunk with context needed to make sense of that chunk. So to solve this like multi-hot problem, like think about we have an LLM scanning each chunk to do the task on it. All you have to do is kind of like retrieve what information is necessary to make sense of that chunk. So it's kind of a reverse of the multi-hot problem. It's that you're not, you're not going from like, oh, question and like I have to pick the right set of chunks for the answer. It's like I'm actually looking at every chunk in my data and then I'm gonna run an LLM over each chunk to give me an answer and then aggregate those at the end to somehow give me, so multi-hop, I guess is that, that pattern is like inherently captured in these like split map reduce directives.

Okay. Cause the reduce can see across multiple documents. And just to make sure people can follow along, like when we say accuracy and things like that, we are very much saying that like people are labeling, they're looking at their data, doing some labeling. You even presented, I think, some workflows in the DocWrangler interface that like allows this. Is that a correct understanding or?

Yeah. So we, in our optimizer, users specify like an accuracy function to run on their sample. So they can label like a sample of 10 documents or whatever, um, or I think 40 is what we use in our experiments. In an early version of the optimizer, we actually use LLM judges to basically take plan outputs and rank them. Um, the LLM judges are okay. I don't love LLM judges. I think like they're the best solution that we have when we don't have labeled data. But it's definitely interesting work to like figure out how to connect like some of the eval-gen labeling kind of into like an interactive DuckyTail query optimizer. I think like that, I would love someone's interested in working on that. Yeah. Feel free to reach out.

This question has come up 3 times already. So this one is good. Is there any reason this wouldn't work to on multimodal data? Is there any complications or anything? Have you tried that?

Yeah. So the systems at Stanford and MIT, Polymphest and Lotus, those actually do semantic operators on multimodal data. I just feel like we've focused on text because there's like so much immediate workloads in text that I've seen and a lot of problems to solve. So in the future, we definitely hope to look into images and, yeah, video, audio, and so forth. Yeah, I don't know what's— I'm sure you can just like convert that to text or descriptions and like run semantic operators for text. I don't know what's gonna be new and interesting about the other operators, so I'm excited to find that out.

Is there any reason we couldn't overload or create our own operators within the Doc ETL?

Yeah, you could definitely try to do like an image map operator. I just simply don't know like what errors people are gonna run into on real workloads. Like, first of all, like I don't know what the queries are gonna look like. Like what does it mean to join images with text, like, okay, why do people want to do that? Like, maybe to do some sort of labeling on the images. Yeah, I would just want to probe into the workloads a little bit more before we learn more.

Makes sense. There's a question about Task Cascade. And the question was, how is the confidence assessed to determine which path to take?

So we use log probabilities. And there are ways to convert log probabilities to like a probabilistic confidence score. You should read the paper to figure that out specifically. Um, but yeah.

And if I recall correctly, you have the log probabilities and then you kind of tune it, you compare that to hand labels and figure out where you want to set the threshold and things like that.

We compare that to the Oracle pla— so all of this is accuracy with respect to some other like powerful LLM oracle plan. So we compare those proxy labels to the oracle labels and then yeah, we iterate through thresholds and figure out which threshold gives us our target accuracy.

Okay.

Um, thanks everyone. This was really great for me to get a chance to practice the content. So thank you.

It was good.

That was great.

Thank you.

Cool.