How to Build Agents That Answer Data Questions

I'm really excited about this one because we all know about how AI is useful for coding. But I think there's one sort of really big use case that's using AI to answer business questions. Like a lot of office work revolves around this. And one of the main questions out there and that when I run into in practice is how can we make AI agents good at this task? It's a very tricky task. And so Shreya did this really great study where she's dived into this subject.

I'm excited to talk about this work. So this is a paper titled 'Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents,' or DAB for short. And this was a paper and a benchmark that we released during my PhD at UC Berkeley. It was work led by myself and a second-year PhD student. Ruying made these slides, so they're very fantastic. It's all thanks to Ruying. She made my job very easy. So first, let's start with simply what is a data agent. A data agent in our minds is some sort of AI assistant with tool calling capabilities. That's fancy terminology for simply being able to query the internet, maybe read and write to a file system, something like Claude Code, but it's also designed for data-related tasks. And you might be wondering, is it even, does it matter? Do people want data agents? Turns out absolutely yes. There's an increasing demand. And in fact, it's so hard people struggle to use tools like Claude Code or Codex off the shelf for their enterprise or their BI tasks. They make their own data agents. For example, Uber built QueryGPT. At this time, it was 1.2 million queries a month. It's probably multiple million queries a month. OpenAI built their own in-house data agent with an astronomical amount of data that's being queried pretty regularly. And many other companies, Databricks, Snowflake, have their own data agents. Now, a lot of these companies have their own data agents, and also smaller companies, teams like yourself, want to build data agents. However, how do you know if your data agent is good? We need a benchmark for building AI agents for software engineering tasks. We have great benchmarks like TerminalBench, SweepBench. You might have heard of those, but we we don't really have anything for data-oriented tasks. We have the text-to-SQL line of benchmark tasks, which is not really sufficient for a lot of enterprise data use cases. Now SQL is not the only way to answer business questions. Often you need to have some sort of domain knowledge or reason about the outputs of the SQL query. The other issue with text-to-SQL is that assumes that all of the data is clean, it's in a single database like, you know, Postgres or the data warehouse like Snowflake. In practice, enterprises have data fragmented across different databases. I'll talk about in a bit. The other types types of data benchmarks are table QA benchmarks, and the way the table QA benchmark works is you provide a table, like a markdown table or HTML table, like a Wikipedia articles table, for example. You provide that in context to the LLM, then you ask a question. So that's fine, that makes sense. People like to use LLMs for table QA, but it doesn't really scale, right, to the amount of the volume of enterprise data out there. So this is what motivated us to build kind of a new benchmark for data agents. And the other thing is that it's hard to build reproducible benchmarks. So while you might not be building your own data agent benchmark yourself, or you might wanna use an off-the-shelf benchmark like DAB, hopefully this talk, which is gonna tell you how we built DAB, hopefully you can learn from some of the principles that we invoked in building DAB and then apply that to building your own benchmarks. Before I get started, actually, I'll tell you that DAB has actually taken off a little bit, which has been exciting. We've had 17 submissions, actually now 20, over the last couple of months. People really like it. So I'm not just, you know, hoodwinking you into believing DAB is very good. So what will I talk about today? First, I'll talk about how we source the queries and datasets for DAB formative study, then how we actually constructed DAB, some of the challenges that come around, you know, trying to construct a benchmark that's real world, but also validatable in a way. And then the third thing I'll talk about today is we benchmarked a number of different agents on DAB and we figured out what are they doing. First, how successful are they? And then second, what are some failure modes that they exhibit? So let's start with the first thing. To understand what DAB needs to have to reflect real-world complexities of enterprise data tasks, we did a formative study. So what that meant is that we simply talked to people in lots of different enterprises that are trying to build enterprise data tasks. We collaborated with a company called Hasura PromptQL, which also builds data agents. And we interviewed users, and I'll tell you 4 key challenges that we found in enterprise data tasks. First, we found that there's a challenge around integrating data from multiple databases. Okay, so this requires retrieving retrieving data from different databases and integrating them into one answer. And I'll give you an example. So imagine that you are a sales analyst at a company. Maybe you have a question like, what leads from the last quarter should we follow up on? Maybe some of the data is in a Postgres database, other data is in a Mongo database, but to answer the question, you need to issue SQL and Mongo queries to retrieve data from both of the databases, and then maybe write some Python script or figure out some Pandas to join the information together and come up with an answer. So that was the first challenge. The second challenge is ill-formatted join keys. So what that concretely means is that you have to do some sort of cleaning in order to join the data from the different databases. So in this case, the join key might be this ID key here, this lead ID or lead reference. It's fine that they have different names, but importantly, what was even more annoying for some of the people that we talked to is that they had to clean one of the columns, clean both of the columns before doing the join. Like if you try to do an exact match between those joins, you'll end up with no matching records. However, if you try to strip out, you know, the lead from the second table's column, then you can actually do an exact match. Now, why is this very hard? It's really annoying. Humans have to know that they need to clean the data when they're trying to do this task manually. So AI agents also need to know to be able to look at the data and reason about how to clean it before doing that join. As I will show you later on in this talk, AI agents don't do this. But those are the second challenges. The third key challenge is some form of unstructured text transformation. Again, fancy language. What that means is that there's often free text in one of the columns in, in the database. That the agent needs to be able to reason about. So for example, it can't write a SQL query or some code to be able to parse it or make sense of it. It actually has to look at those values row by row and then make a business decision or recommend some business decision to make from it. Now, this is very annoying for agents and also of course humans to do at scale. You can imagine if there are millions of rows in a database, reading and reasoning through all of this text is like simply not possible. And it's also not kind of baked into agents to be able to do this. The last thing that we found, a property of enterprise data workloads is that often queries have domain knowledge. So concretely, maybe in the sales case, that means that there's some domain knowledge or some internal knowledge about what to do based on the intent of that particular potential customer. Maybe we want to issue a call, like a high priority call. Maybe we want to just simply send an email following up. These actions might be internal to the company. However, the AI agent needs to know about it in order to effectively recommend the next thing to do. So these are the 4 challenges that we found in our formative. There's many more. But these are 4 that we found that are not covered by existing benchmarks and very vital for data agents to be able to do. So we created a benchmark DAB. Here's an overview of it. There's 104 different queries. That might seem small, but you should think of each query as like an environment, like a very long-running— these queries take hundreds of turns to be able to answer. Think of each of these as like an environment, kind of the scale of the benchmark is something like TerminalBench. There are 17 datasets. Datasets can have multiple queries, of course, and we have a number of queries for each of the 4 properties that we observed. Lots of different datasets that we drew from, mostly Kaggle datasets, as I'll talk a little bit about and how we corrupted those in meaningful ways. We span a number of databases as listed here. We picked open source databases so people can run this themselves. All right, so that's the formative study. The next thing I'll talk about is how we even constructed the benchmark to include the properties that we found in the formals— formative study. So there are 4 steps to building DAB itself, collecting data from some sort of open source dataset. Second, transforming the data to exhibit the properties that we wanted to induce from our formative study. Third is distributing this data across different databases. And then fourth is kind of engineering components of the hardness that a data agent would be using. For example, descriptions of the database, or if you're really steeped in the data literature, something like a semantic layer. So engineering some of these hint or semantic layers, or what we'll call hints, to really help the agents be able to answer the queries. And you can think of having these files as also ablations for the agent. Like maybe you can remove the hints and see if the agent at the end can answer the question. Adding the hints, what does that do? That I'll get into with the performance. So let's start with the open source dataset. We first collected open source data. I think this part is probably not super interesting to folks. But what was more interesting is we wanted to make sure that the answers to these queries that we came up with are validatable. So what we ended up doing is taking open source data and maybe removing a column from it or embedding columns, numerical columns inside of string columns, changing the column names, corrupting some of the data in deterministic ways to make sure that it exhibited the properties. And you can read the paper for more on the specific strategies that we use there. But the key challenge was making sure that, you know, you can get a validatable ground truth at the end of this. The second thing was transforming the data. The third thing was distributing this data across different databases. So maybe For every single query and every single domain that we looked at, we talked to the people from the formative study and figured out what databases they might use. So it turns out that, you know, for example, Mongo is used a lot for some sort of customer data, for example. Maybe people use DuckDB for something more analytical, DuckDB or ButterDuck or whatever in that family. So based on the semantics of the data and task in the domain, we figured out what databases to put the data in. And then the last thing was that we detailed in a hints file the corruptions that we had made, and we also Wrote a semantic layer, database schema, put it in another file. So in DAB, we actually, with the agents, we test multiple different setups. So we test the agent when it's given the schema and some examples of data content, and we also test the agent when it's given the hints files or the descriptions of the transformations that we induced. So here's an example of a query in DAB. There's a lot going on in this figure, so let's start with the left part of the figure. So the query might be something like, list all books written in English with an average rating of 5.0. So the company here might be some books outlet. Retailer, Amazon, for example, there might be a database of books, there might be multiple databases. So the agent has to first query the database of books, maybe query the database or the table of reviews, might not be in the same database as the database of books, and issue a bunch of SQL queries to learn about what the data looks like. And then there's a lot of steps that are skipped in here, but also then figure out what data to load from each database and then maybe write a Python script to join the data and do some analysis and return the answer. Answer. So that was the benchmark construction. Let's talk a little bit more now about how do agents perform on depth. So we tested 3 different frontier model providers. So we tested Claude Sonnet 4.6 because Opus, you will not believe the amount of money that was spent to use like Opus to even do a sample of queries. So we ended up doing Claude Sonnet, tested GPT-5.5, and we also tested Gemini 3.1 Pro. So at the time we were doing this, these are the frontier models. I guess now that there's like this GPT soul, maybe it'll do a little bit better. And for each of these providers, we use the harness that was recommended. So for example, for the Claude Sonnet model, Claude Code is, or Claude Agent SDK to be more precise, is kind of the recommended harness for that. For Gemini 3.1 Pro, we used a React harness as an open source harness. There is not like an SDK-like harness that we found for the others. So given these three, you know, how do they perform? The spoiler number is that the best frontier model, which is GPT-5.5 and Codex, gets only a 57% pass at 1. So that is given 5 trials, the probability that it passes at least 1 of those 5 trials is 57%. And then the pass at 5 is 70%. So that's also quite low. So let's talk about kind of how we set up the agents and then we will talk a little bit more about their failure modes specifically. So specifically the tools. That were given to all of these agents are basically what you would imagine. Harness like Codex or Claude Code, they have built-in tools. So for example, Bash, which allows it to run any terminal command. And then we isolated the file system. So, you know, it cannot read other data from other folders in your computer and read and write. So it's allowed to read files, write files, do whatever it would like. So if it wants to issue SQL queries, right, it would do that in Bash. Or I guess it could write a script and then use Bash to execute that script. So the network access was disabled. For all of these. Turns out that when you enable network access, searches the internet for the ground truth data because we found it online. So we had to disable network access. Um, and then we also gave it 5 trials per query. So here's a plot, the Pareto frontier kind of of the agents that we tested. I've— there's only 3 that I really want to get into. So Gemini 3 Pro actually very surprisingly is very, very good for the React harness. Another thing we found was kind of as we gave it more attempts, the pass at K kind of increased, but it didn't, it all saturated at 0.7. So with the best one. All right. So what are the failure modes that these agents kind of exhibit? Whenever the query is incorrect, or sorry, the answer is incorrect, what failure modes do they have? So we define 5 failure modes in our paper. So one is that it just simply fails even before it tries to plan an answer to the query. So maybe it like doesn't even try to use a tool or it just says, I'm sorry, I can't answer that question. Sometimes some of these models have like very aggressive safeguards enabled. So we found that it didn't happen that much, but that's one stage it could fail. Then the second stage it can fail is what we call incorrect plan. So it might say, here's the plan that I want to do, but you can just read that plan, which is already in natural language, and it turns out that this is not the correct plan to answer the query. Then the third failure mode is given the correct plan at a very high level, maybe it goes and it selects for example, the wrong column in the database. So the wrong attribute for data extraction. It should have looked at more columns. It should have looked at more data within the table and realized that it was another column that they needed. Failure mode 4, it selected the correct data, the correct columns in their SQL queries or to have in Python. However, it has incorrectly implemented the code to make sense of that data. So for example, if it needs to clean a column in the data by applying some regular expression, maybe it's written the wrong regular expression to do the data cleaning. Like the regular expression might not be comprehensive of pretty much all of the different types of corruptions that were in the data. And then the last failure mode is simply like runtime error. Again, this actually does not happen that often because if it does happen, then the agent simply retries it. But to fail entirely, to like break out of the harness and like return the wrong answer and stop early due to runtime error, we found that did not happen as much. So what we did was we came up with these 5 failure modes ourselves by manually looking at the data. Highly recommend, always, we should manually look at data to really understand what's going Polygon. But then once we came up with this taxonomy of 5 failure modes, we used an LLM as a judge to really scale up our judgment. So concretely what that means is for every failure mode, we use an LLM to run that fail— to basically assess whether that failure mode occurs on hundreds of traces, because we can't look at hundreds of them manually. So I'll show you a histogram of kind of all of the different failure mode quantities. And again, there are some models in here. These are from older versions of DADS, so I think the slides need to be a little bit updated. Still a preprint. But you can kind of see at a high level the failure modes 2, 3, and 4 are the most common ones. Failure mode 2, where it simply just has an incorrect plan, as well as failure mode 4. So what we learned from this is like they do select the right data or the right columns, but maybe they are not able to actually exhaust all of the ways in which the data has been corrupted in their implementation of cleaning the data, or they're not able to transform the data, or something is up with the implementation. All right, so my final conclusion stuff, and we'll get in more into questions, but basically why is DAB useful? Like, should you use DAB? What should you take away from all of this. So, uh, the first high-level thing that I want to talk about is simply how DAB can— is probably the first benchmark that actually really has complexity of real-world data class tasks and can evaluate end-to-end capabilities of data agents. So it's not isolated to simply, you know, the text-to-SQL or table question answering. It's more end-to-end. So if your company is building a data agent, DAB is certainly something that you probably want to evaluate on. And if you're not building a data agent to— as a product, but maybe you want to help your sales analysts. Analysts run queries over the data. I think unfortunately we're at a point where the frontier models as is are not great at solving them. Hence, like, even OpenAI built their own data agent and they actually, they have a set of skills around data agents that they have recently open sourced. So I would really encourage you to take a look at that. Don't assume that frontier models can solve your data questions off the shelf. Absolutely, you should use the skill, build some sort of harness, teach your agents or tail your agents to explore the data and so forth. And you can even use DAB as a benchmark to evaluate your harness. What's next for us? In research, well, we're doing a lot of stuff. One of the biggest things that we found was that a lot of these agents are quite expensive, right? We don't want to spend $30 to answer one query. So we're looking at ways to, you know, match the accuracy of DAB with a fraction of the cost. How can we make DAB or agents solve DAB queries much faster, give them more parallelism, allow them to explore multiple databases at once, use subagents, so forth. So a lot of really interesting stuff there. And please check it out. Here's a QR code to the DAB paper and open source repo.

So one of the things when I was reading this paper, one of the things that surprised me was the amount of failures in the planning stage. I was like, wow, if you can't plan what to do, kind of dead in the water, like not going to be able to get the right answer. And for some reason I thought like the agents would be able to plan better. Like if you asked me to guess, I wouldn't have said, oh, it's going to be in planning, that's going to be the failure. Do you have any intuitions on how that step, like what was going on inside that? Was there anything interesting you found? And then like, is there any intuitions you have like how you would make less of a failure if you're building a data agent?

Why is there a problem with planning? What does that look like? So what we found is that agents like to write the plan before looking at the data, which is very bad in the data setting because you can just assume in enterprise data, data is messy, data needs cleaning, data is corrupted, something is off with it. So when it's writing the plan, the plan often does not— sometimes it doesn't even include data cleaning steps, which is like like not very good. Sometimes it may include a data cleaning step because, for example, our hints file said that data needs to be cleaned. And so it'll say, do data cleaning, but it won't really describe how. And then it will only look at the data after writing the plan, which I don't know how these agents behave, but maybe they're like really overfit to the plan that they wrote before, even if they encounter some mess in the data. This part, you know, beats me. But fortunately, I think there are ways you can get around it. Like in the prompt, right? You can say things like, make sure you look at the data before you make a plan. Now, I don't know how precise you need to be in that prompt, like make sure you look at rows from every table, but maybe you need to tell it exactly how to look at the data. But yeah, the TL;DR is that for data questions, don't write plans before you've looked at the data. Look at the data, write plan. We'll always be looking at data.

And also from reading the paper, so this benchmark is really interesting. The data is really interesting because you did a lot of work upfront to interview companies and ask them what the tricky edge cases are. You infuse that into the data, like the join keys problem and like a bunch of other problems that like it wasn't, it's not like you brainstormed it from scratch. You like looked and saw what was going on. I guess like if people are trying to create data agents in their companies, what's the process they should go through if they're trying to create like a test harness? Should they go through a similar process you did to like figure out what their failure modes, like where the edge cases are in their data and like adapt this? Should they just start from this? How do you think people go about it?

So I think our process was quite good. So if you're trying to build like an in-house data agent, you know, talk to your BI people, talk to your analysts, talk to the people who like really wish they could write SQL queries and like work with data and need to make business decisions, but like for some reason, you know, they're just not data science, they're not like data people, right? So they're bottlenecked. So talk to them and try to figure out what their queries are, like a formative study like ours. The hardest part of DAP actually was not the formative study. It was in creating datasets from open source data, creating a benchmark from open source data because we cannot like release customer data or whatever, right? We're trying to release a public benchmark. Benchmark. So fortunately, if you are at a company, right, you don't— you can actually use your internal data. You can create a benchmark with realistic queries and realistic data and not worry about like leaking your data because you probably won't expose it. The other thing I'll say is at the time that we were creating this, we had like humans come up with the correct answer, and that was very slow and like a really painful process for these like 100-turn-plus queries. So we would do it again as we would have like human plus like Fable. Or we probably cannot spend all of our money on Fable, but if you are using your company's credits, like use able to help you find answers and figure out if your questions are good and your data is good and build your own benchmark that way. You follow our methodology, create your own dataset of queries and, or create your own benchmark of queries and datasets. And yeah, you don't have to use our 4 properties in particular, right? Like you can come up with properties that matter to you, but the high-level process is very generalizable, I think.

Has anybody hill climbed this benchmark already that you're aware of?

Yeah, there's totally people who've submitted PRs with like very, very high scores and that is achieved from hill climbing. However, the problem is that when people hill climb, they often like inadvertently are cheating in some way. So for example, like web search is turned on. So one of our datasets is like the AG News dataset in Hugging Face where we have, it's a news classification task. We augment that with other data. A subtask in the query is like news article classification. So what we have done in our benchmark is drop the ground truth labels, obviously, so that agent has to do classification. But if you pull this dataset from Hugging Face, you get the labels. Like that's cheating and it's really, really hard to prevent the agent from like inadvertently cheating. Like, and the other thing is even if you turn off the web access and somehow the dataset is like downloaded from Hugging Face, like somewhere deep in your Hugging Face cache, the agent will find it, right? So you have to be very careful about like sandboxing and making sure it's not cheating, but you still want to hill climb and be able to, you know, issue safe queries to like learn more about domain knowledge. I think it's quite hard to like successfully hill climb.

Have you found anybody that's made a submission that seems generalizable, that's not trying to overfit or accidentally cheat?

Nothing is perfect. The strategy that people have been doing is like invest in a very good semantic layer. So what I mean by semantic layer, for a lot of folks on the call, is basically have a metadata file that describes all of the tables and all of the columns, describes all of the columns, like in natural language. And this you can populate in advance by having agents crawl over your data. And then you can go even further with these semantic layers and say like, exactly, this is a numerical column, all of the values are between, you know, 1 and 100. You can, you can talk about like actual properties of the data, or like here's a histogram of the data, or here are all the distinct values in the column. So the more that you can kind of put into a semantic layer, we found like the better that the agent performs. However, of course, building a semantic layer is very hard. And I don't think people have like really solved how to do that properly.

Would an ORM make any difference in performance? And I replied with, you should do an eval.

I think the The first thing is an enterprise, right? Like you may not be able to query your data with an ORM. Like, I don't know how, like every enterprise does data query, like interfaces with their database differently. So if you have ORMs and that's the way to work with data, then I suppose you use that. Many people don't, in which case, like I would never ever to anybody on the planet recommend fitting, outfitting your code base with ORMs. Like that sounds very, very painful. Yeah, I don't know if it will be better. It perhaps could be in that, like, you know, you could somehow write like Python or JavaScript or something, some code to be able to interface with your databases kind of in one script. But going back to Hamel Husain, like you should do an eval to see.

How do you get the agent to clean data? Is there any kind of tricks that you found to like make sure it discovers the right steps in like what it needs to do to clean data? Is it able to like be proactive enough? Is there any kind of learning loop or pattern that the agent learns like how to do things? Yeah, there's a lot of edge cases, a lot of tricky aspects about data.

First one is how you get the agent to clean the data. So remember I talked about how we constructed the data? We have a hints file of how we corrupted the data. So it turns out when you add the hints file, the accuracy, the pass at 1, can go up by over 10%, like, per agent. So surely telling it about the issues in the data helps. Now, our hints file wasn't comprehensive, right? It didn't say, like, here are all the different ways that we corrupted the data. The hints file all says about the location, so, like, X column and Y data database is corrupted and then loosely in this way, like maybe there are extra spaces in there. So sometimes there are like trailing spaces or there's like spaces in the middle. It's somehow, you clean that, like it's very possible to look at that hint and still get it incorrect. So being told about the hints helps the agents. However, it doesn't solve it perfectly. So I think what the issue with data cleaning stuff is the way that I see the solution for enterprises is to like have memory or like invest in creating like a memory store for your data agents. So anytime your data agents like kind of run into a specific corruption, like write that to some sort of memory store and then allow the data agents to be able to query that memory store. And you can, you can think of that as a semantic layer as well. In my mind, I think of a semantic layer right now. It's really hard to say semantic layer, the definition is changing, but you can think about all the upfront indexing on the data and like how semantically making sense of it, that can be a semantic layer. And then there's a memory of whenever agents look at the data and try to answer queries and they run into new things that were not in the semantic layer, then there's some memory that needs to be there. And we found that in another concurrent work, we found that having a memory really improves the performance, uh, accuracy of text-to-SQL. But we haven't tried it on DataAgent benchmark yet.

A thing that engineers gravitate towards. Any thoughts on graph databases? How agents—

You can put graphs in relational databases. There's nothing stopping you building a context layer or whatever you wanna call it, semantic layer, and then storing that in Postgres. Or in what, I don't know what relational databases people, ClickHouse, people are using these days. My mind, you wanna have fewer databases. You don't wanna have all the databases in the world. And ideally you want them all kind of the same type of database. I can guarantee you that you have a relational database somewhere in your enterprise, so might as well try to use relational databases wherever possible. That's simply my opinionated take. Thank you, Shreya.