How To Build And Evaluate Search Agents
Nandan Thakur covers building and evaluating search agents where LLMs iteratively call search tools until confident enough to answer. He presents BrowseComp benchmarks, synthetic datasets like ORBIT that use multi-source verification, and Hawkeye, a tool showing agents often waste tokens repeating queries despite high accuracy scores.
Chapters
Welcome everybody. Today we have Nandan Thakur and he is a researcher. Um, he's been focusing on search for a really long time, search and retrieval, especially for LLMs and AI. He gave a talk last year in a similar series where we went pretty deep on retrieval and he did a talk on also the retrieval evals. He's done a lot of work on this topic. He's created several very popular benchmarks. And so today he's going to be talking about search agents and how do you go about evaluating those. He's done a lot of work on that as well. And search agents, as you know, well, search in general in any AI product tends to be the weakest point. And so this is a very important topic if you're building AI products. So I'll go ahead and hand it off to Nandan.
Thanks for the introduction. And without any further ado, let's dive deep into what makes a good search agent. And I'm Nandan Thakur, and I'm currently a postdoctoral researcher which— who is working at Microsoft Research India. So I completed my undergraduate from Pittsburgh Goa campus. And then for a few years, I spent as an NLP research assistant at the UKP lab, TU Darmstadt in Germany. After that, from 2021 till a few months ago, I was doing my PhD from University of Waterloo. And currently, actually last week, I started as a postdoctoral researcher back here in Microsoft Research India. You would probably recognize me from one of these benchmarks or these works where I've been affiliated with. So if you would have either heard about the BEIR benchmark, Miracle, FreshStack, or you would have heard about the TREC RAG efforts, I've been a part of these works. So today we are going to talk about search agents, and this is really a new topic which has emerged since the past year. So this is a very rough talk. So not many things are formalized yet. So many of these things are me trying to understand and my point of view of how I view search agents. We will first start with the motivation and introduction. After that, I've broken my talk into 3 parts. The first part will focus on the search agent evaluation. The second part will focus on the data and the training part. And the third part will focus on the trajectory analysis. Finally, I'll conclude this presentation and give out some thoughts for the future. So let's start with motivation and introduction of search agents. So as you know, Search is evolving. And as I was saying, agents is the new thing, which is pretty hot these days. And it has been there since the, I think so since the start of the year and maybe early or late last year. So last year we did this series and the theme was RAG. And just to revisit what RAG was. So we would have a particular user query, which would first go through a retriever model, which in turn searches over a corpus of documents. So this could be any private corpus which you have, and then that would retrieve you top K documents for your particular user query, which in turn becomes a retrieved context for your LLM to then generate you an answer on this retrieved context. So the main idea here was you would retrieve once in the beginning and then provide this context to your LLM for it to generate an answer. So this was a naive RAG architecture, which was pretty common until last year. Ever since agents have come into the picture, the architecture has sort of changed a bit. So what we now call as agentic search involves the same user query, but then instead of first going to the retriever model, now it actually goes to the LLM model first. In this case, now the LLM model is the main driver of this where it can either reason first and see whether it needs to retrieve. So there is a retrieval method which is present as a tool. So the LLM, if it wants to retrieve some information from the corpus, it would provide a query to this retrieval tool. And in turn, the retrieval model would then return back some top-K context, which is used by the LLM to check some facts and refine its queries. So this is an iterative process. And finally, once the LLM feels like it has almost reached the answer, it would then give you out the final answer. The major difference between RAG versus agentic search is that there was usually retrieval once and then generate in the RAG paradigm. Here you have the LLM, which is interacting with a retrieval tool. It could be interacting with multiple more tools, but for search purposes, we are just considering the retrieval tool here today. And then once it has iterated through these tools and it has done some reasoning, it then tries to give you a final answer. Ever since the beginning of this year and even late last year, there has been an outburst of search agent papers. So if you actually see in the literature, there are just many, many search agents and papers out there. It might feel very new as a field, but the field these days are developing so quickly. There are like many, many papers which are coming out, which are either developing search agents or developing benchmarks and so on and so forth. Let's try to broaden the perspective and try to see what are actually the challenges in search agents. So today's presentation, I'll be focusing on 3 aspects very broadly. So first, I'll be trying to focus on how you would evaluate search agents such that how can we reliably test these search agents on complex reasoning tasks? How can we create benchmarks which can be used for evaluation? The second part, I will be focusing more on the data part. So how you can generate training data, synthetic training data in an efficient way so that you can use this training data to train search agents. And the third part, I'll focus on analysis where we'll focus on why search trajectories or these search agent trajectories are important and crucial. And it can be used to identify particular trends which are there within search agents. So let's now move to part 1, where I'll be talking about search agent evaluation. Evaluation of search agents is hard because since search agents are intermixing retrieval and reasoning, their evaluation needs to be thought And you need to actually put benchmark which can evaluate the progress or the capacity of these search agents. What we find is that existing, you know, retrieval-based benchmarks or QA-based benchmarks such as HotpotQA, Natural Questions, or BEAR, they contain only simple factoid queries and these are not hard enough actually to do search agent evaluation. So the motivation becomes how can we test the complex reasoning capabilities of search agents and what would be like a fair and a reliable setup where you can actually reproduce the scores of an agentic search model and you can faithfully reproduce it, plug and play the retrieval model. So these are the 2 motivations for coming up with a benchmark. So the first style of benchmarks which are out there, so I call them to be riddle-style questions where these are, you know, easy to verify, but actually hard to search and hard to solve them and Here is an example of this question. So this is the BrowseComp benchmark, which was introduced by OpenAI sometime last year. So in this case, you can figure out the question asks the title of a research publication, which was published before June 2023 that mentions so-and-so forth. And it was co-authored by 3 individuals. One of them was an assistant professor in West Bengal and another one holds a PhD. And the answer is, you know, this research publication. So as you can see from this particular example, the question is pretty complex and it has multiple aspects involved within it. How I imagine you would answer or you would tackle this particular problem is you would do something which I have termed to be as a funnel search. So most of BrowseComp queries require or somewhat give you a funnel search in which the first part, if you take, for example, if you want to figure out recent publications which are prior to June 2023, which mentions cultural traditions, so on and so forth. May have more than 100+ papers within that. After that, you actually need to check the number of co-authors in these particular publications. And let's say, as an— just as an example, this reduces down the funnel which originally had 100 papers which were satisfying the first criteria, down to 20 papers which were satisfying both the first and the second criteria, down to only one which will satisfy the last criteria, which is the assistant professor in West Bengal and the other author holding a PhD. So BrowseComp was actually pretty challenging enough. How they initially generated these queries, they would use a human to actually construct the question. And they would give this query to another human, which had access to, you know, search and other tools. And another human will not be able to solve this question within 10 minutes. So that's why these particular queries were then filtered and chosen for the benchmark. Also, they tried GPT-4o, which were not able to solve these queries. So BrowseComp was primarily used a lot for, you know, your search agent evaluation or deep research model evaluation. You would often see this kind of a figure, which was taken from one of these research papers where they would report the accuracy, basically finding whether the answer is present within the search agent or not. However, with this evaluation, there are like certain issues. The first issue is since BrowseComp only provides you queries and answers, a lot of times papers, when they're reporting numbers, they're not reporting what search tool they're using. For their experiments because the search tool which you use can actually impact the EM accuracy a lot. The second question is, are these numbers really fair and reproducible? Since you don't know what search tool was used or what retrieval model was used for searching these documents, you could have actually used a very good retrieval model for one of these, for one of these LLMs, but not a good one for the other one. Who is there to actually confirm this? And the third is, since we are just measuring the EM accuracy, we don't know how much actually the retrieval tool in itself contributes to the end EMI accuracy. So due to these issues, I was a part of this project known as BrowsComp+, which was led by Zijian Chen from University of Waterloo, where we actually constructed a more fair and reproducible BrowsComp benchmark. So what this did was it converted the QA task into actually an open retrieval problem setting. The idea was very simple. So you would take the BrowsComp question and the answer pair. We would then use OpenAI O3 model back then to actually gather the gold web pages which were somewhat relevant or which can contain clues or information which is required to answer the question. So these are coming out as URL or evidence web pages. And then we would actually provide these documents then to a human annotator who would actually annotate like this document is useful to answer this part of the question. For example, this document is useful to answer this part of the question and so on and so forth. So a human verification step would there where a human annotator would actually check these documents are really relevant in answering the question and whether it contains the final answer or not. So this is how we constructed PROSCOMP+. What we really did was originally we had the queries and the answers. Now, in addition, we have 2 things. So first we have a web corpus, so which was constructed using these positive documents, which I just described, which were scraped. In addition, we also generated somewhat known as R-negatives, which is usually put in the corpus to improve the retrieval difficulty. In addition, since we have humans which have judged what relevant documents are there for a particular question, we have relevance judgments which are available for each query. So the Web Corpus, you know, it has almost 100K documents, which is a very comprehensive coverage. It is also very difficult since you have positive and hard negative. So it makes the retrieval task to be difficult and it's of a practical size for evaluation. So it's not too big so that researchers can adopt. This corpus. On the other hand, relevance judgments, so which is pretty great because you have human supervision on this and this typically follows how a retrieval evaluation setup would look like. So to measure the answer accuracy, we usually in the paper, we came up with the exact match accuracy metric. And for measuring the retrieval accuracy, we look at 2 metrics, either the number of search calls which were used during an agent's turn or the recall metric. Which denotes how many positive passages were retrieved in your subset out of the total positive passages available. So some results or key takeaways. What we found from the BROSComm benchmark is this interesting diagram, which is there in the paper, which I absolutely like. So on the x-axis, if you see, so it actually plots the number of search calls and on the y-axis you have the EM accuracy. So here, an interesting finding, which you would find is that actually both The search model and the LLM, both of them contribute to the EM accuracy. So as you can see, GPT-5 is the best LLM out of all because it's the— if you're actually considering vertically, it's on the top. On the other hand, you can actually see that whenever we would do BM25 to QWEN3, which is a dense embedding model. So when we are moving from lexical model to a dense embedding model, you are actually taking fewer search terms. So number of search calls, more going towards zero is actually better for us. And it's also improving the accuracy. So with the same model, using, doing a better retrieval model actually gives you a better accuracy. And it also takes less number of turns to reach the final answer. Key takeaways from here is that both retrieval and reasoning are important for agentic search and a good retriever, A, should increase the accuracy and B, should be able to reach your final answer with hopefully fewer search turns. More recently, there was also subsequent works which was done. So this was one of the, another very interesting plot, which I took from the Reason Modern ColBERT paper, their blog post. So they actually used ColBERT-style model to evaluate a search agent on BrowseComp Plus. And here also an interesting finding is that, you know, the agentic search is not just the LLM problem. It is a very big retrieval problem as well. From this plot, actually, we found that if you just keep your LLM to be GPT-5, depending upon whether you use BM25 or like a Reasoned Modern ColBERT, which is a very good date interaction model, which was also fine-tuned on reasoning-based training data, you can actually go from 60% on BrowseComp Plus to actually 88%, which is a huge jump with the same LLM. So just to show that, you know, if you, the retrieval model you choose is very important here. Finally, we also find that, you know, sometimes you can use BM25 and actually modify BM25 to get and tune it pretty well to actually get a good accuracy. So this is again, another table which has been actually taken from Piserini, which is another paper. And the most interesting part which I found was the DS-Flash, DS-Pro, and the one which is highlighted and in the line. So actually with just BM25, they have achieved a 60 to almost 80% accuracy. So it shows that when you tune BM25, a simple lexical baseline can also perform well. So these are some of the key findings which we had from the BROSComm+ So is the right way to interpret this graph on the lower left-hand corner?
It seems like the multiple vector retrieval, like the ColBERT and friends, like, are doing a lot better, making your friends happy as well, probably with this result. In the series, we talk a lot about this kind of— is that an approach that you think— have you seen it pretty much be Pareto optimal across the board of like using that approach, different models and different things?
I do agree. So I feel like for these sort of experiments on ProSCoM Plus, we definitely see that Reason Modern ColBERT and these models are pretty great. So, you know, you're able to achieve 90% accuracy, which is quite commendable actually with the model. So I do think I find these findings also generalizable to some extent to the other tasks, but then again, I'm not the expert here. So probably the folks here who are behind these amazing, Antoine and others, who are developing these models. I'm sure they can talk more about them. The main key takeaway, I think, so what I find is retrieval model can actually go from 60 to 90% on BrowseComp Plus, which shows that if you substitute a good retrieval model with a bad one, you may actually lose out on the EM accuracy for your search engine. Most of the BrowseComp Plus, as I showed in the beginning, they basically have a short answer, which is verifiable. So you would often do an exact match, whether these exact answer is present in your generated answer or not. It's a score. Of that. So if this answer is present in your evaluation, then it gets an answer of 1. If not, it's a 0. So based on the interest on BROSComp+, there are some upcoming works which I'm personally involved in. So the first one is known as the MAST, which is the Multilingual Agentic Search track. So if you're interested, we are gonna conduct a shared track. We are kind of coming up with a multilingual benchmark on this because we find that there are not many currently available which focus on different languages. Apart from English. So we are, we are having 2 tracks. First one focuses on a multilingual setup. The other one more for an Indic audience or which covers certain Indian languages. So if someone is interested, they can actually, I've linked the project website here. So you can actually visit that and be useful where we'll give call for participation and you can participate and submit your own multilingual agentic search systems for this. Another related benchmarks. Which I just wanted to mention is some long-form questions. So we are also trying this out in actually RAG 2026, which is the current iteration of the TREC RAG. So in these cases, we have actually a question where you would need, there is a long-form answer, which is not a short verifiable one, but which is actually a more of a passage or of a report level where you have certain document-level citations and it's a whole report which is generated for a question. So these sort of benchmarks can also be used to evaluate agentic search. And then this is actually one of the tasks which is ongoing for TREC RAG this year. And some evaluation criteria which would be useful or how you would evaluate these answers would be breaking down it to be as a rubric-based evaluation where you would come up with certain key facts or nuggets. Another metric which is quite important is support or attribution, whether the information which you are citing within these documents, whether they actually support the information present in the document. And third, which is also source credibility. So whether these sources are credible or not. So these are more or less the evaluation criteria. These are some of the other benchmarks you would find which are being used to evaluate agentic search systems.
What's the mechanism in which you would actually execute this evaluation criteria? Is it like LLM as a judge? Well, how do you plan on doing this?
Until last year, we used LLM as a judge. So actually this year we're trying to use an agent itself for these evaluations. So Because till last year we had a very engineered setup, but this year we are actually trying— we soon should have a preprint out there where we are actually trying to do this using Pi, where you can come up with an automatic way to do these rubric support and other criteria automatically for a long-form answer, at least for the TREC questions.
And you're going to do some work in terms of evaluating the evaluator, meaning knowing whether or not you can trust this?
It's an interesting question. So the way we are doing it is since we have human labels from the previous iterations of this task, actually rerunning our whole setup on 2024 and 2025 data to actually see whether our metrics, whether they collaborate or how much do they correlate, sorry, with the human judgments out there. So I personally overview the support one. So for support, we actually check, okay, how well does the agentic system label whether, you know, something is partially supported or fully supported versus a human one. And we do look at Kendall Tau as a correlation, but I feel like there are interesting findings there. Certain things which agents are still not able to do, sometimes it would think something is partially supported, whereas a human is much more decisive in its choices. So I've personally seen that human annotator always likes to either be not relevant or fully relevant, but the agents are biased somewhat towards partial relevance. So this is what I see currently, but yeah, I think so. It's like nice talk in itself. So a nugget is, so this is a terminology which I picked from TREC RAG. So we define a nugget as a key fact, which is just a sentence level information. So it's atomic unit which we use for evaluation. So nuggets, ideally you would want nuggets to be present in your answer. This would mean that your answer is supporting different things and you would want that. And a good answer should support good nuggets. If in case, if you're ever interested with these words, so you can actually check out the TREC RAG doc. And there's been a lot of work which has been happening since past 2 years and papers, we actually defined the nuggets. And support more formally, which can be useful for the audience. Moving on now to part 2, where I'll be talking more about the data and the training of these search agent models. As I talked before, so existing actually thin datasets, which are used to train retrieval models or QA-based models are actually not built for agentic search. HotpotQA only contains up to 2 hops. It was created in a somewhat synthetic way where, you know, annotators were provided 2 concepts, 2 Wikipedia pages, and were asked to come up with a question which talks about both natural questions, more realistic questions, but these are much easier questions. So they don't actually need multiple rounds of search. And the overall issue was actually the training data was too simple for to train agentic search models on these type of queries. So there has been work after that, which have developed agentic datasets, or there have been related papers which have tried to generate these datasets for agentic search. So I'm going to talk about 2, which I find very interesting. First is the WebDancer. So in this case, they tried to generate training data in 2 ways. The first way was crawling through websites. So they actually went through a website where they would click on a particular root node and have hyperlinks to other subsequent nodes. And then they would give this to an LLM, which back then they used, I think, so some ChatGPT, which was then used to generate a question and a particular answer. So this is how they generated some synthetic queries using crawling websites. Another interesting way they actually generated a synthetic training dataset was this E2HQA, where they would take an original question and they would retrieve some particular document And then they would give it, give the question and the retrieved documents back to the LLM, which would further refine the question and make it technically more and more harder. So it's adding more and more constraints at each and each stage. And finally, you would come to a particular stage where you have modified this enough to make it hard. And you have a, again, you actually have an answer for this particular question and maintaining it. So overall, this is the thing. If you're interested, you can look at the paper, but yeah, this was a very interesting way to generate these training datasets in WebDancer. Another one which I'll briefly talk about is InfoSeeker. In this case, they actually come up with an entity linking problem and what they call as a hierarchical condition satisfactory problem. Very simple put, they have an entity map which has mapping some particular entities such as Alan Turing graduated from Cambridge, was born in the city of Europe. So they have these entity links. And they are trying to construct a question which can satisfy some constraint over these props. So they link these and they're trying to blur some nodes and trying to come up with a question. These are all great. I feel like these are very definitely, these are very useful datasets. I wouldn't say they have any issues. The only minor thing which I think is that sometimes they have prerequisites which can be cumbersome and expensive to set up. So for this particular dataset, you need to have entity links and you need to have a good enough graph, which is diverse and covers different entities to be able to generate a good quality training dataset. In our work, we came up with an approach known as ORBIT, which was a more frugal way to generate training datasets. It has zero prerequisites and we didn't imply any expensive web search or LLMs to actually use them to generate the training data. So good part was it actually covers Wikipedia and both web domains. Many of these existing training datasets are just Wikipedia limited. So here you can see a distribution of the verification links. So you can see majority are Wikipedia, but you also have like documents coming from other sources such as ScienceDirect, Britannica, Stack Exchange, which are used for verifying the answer for these training pairs. And also they are more harder questions in general. So I've compared HotPotQA, InfoSeq, which was the previous dataset which I talked about, and the Orbit, which is our dataset. In terms of reasoning step, you require more than HotPotQA. HotPotQA, as you can see, a majority of them is just 2, whereas on the other hand, Orbit has more of a higher distribution, meaning that you need more search terms to be able to answer these queries. This is an example which is there in Orbit. So this was the question which comes. And as you can see, this question is pretty complex in itself. It has multiple different aspects of it. So you need to figure out what's the exact runtime of a particular movie, which was a 2017 animated feature set inside a smartphone, which was directed by a filmmaker, which was previously known for sequel to popular children franchises, featuring a protagonist whose facial expression malfunctions, so on and so forth. So you have multiple criteria which you need to verify. The answer to this was The Emoji Movie, and the exact runtime was 86 minutes for the particular movie. This shows that actually a model needs to search through multiple different aspects to be able to verify each and every part of the question, and then it's able to provide the correct answer to this particular question. We looked at some examples. Let's now look at how we actually constructed the dataset. So at the first stage, what we do is we do a seed creation stage. So we take a domain and we used actually OpenAI's Deep Search tool to give us top 100 Wikipedia categories. And then we filtered a few of them and then we actually got the Wikipedia pages, which are associated to these categories. So you can actually get these through Wikimedia links. So it's not too difficult to get all of the Wikipedia pages. And then we would use a Wikipedia page as a seed in our data generation process. So in this case, for example, concussions in Australian sports. Becomes a seed. In order to get away with LLMs and web search APIs, so I would recommend, again, this comes from a researcher background where I didn't have much access to compute and I didn't have much budget. So a good way which I found was you can actually sort of interact with the DeepSeek chat and where we would interact with them. And we had a Selenium instance which was running one by one with the free version. And then you can actually ask the seed with, along with a prompt. And the DeepSeek interface would do search and reasoning and we would prompt it to give us back a question and an answer. So there's a prompt which contains certain examples, few-shot examples. And then once it has done a whole thinking stage, at the end, it will give us multi-hop question. As you can see, it was this one and it has a particular answer. Since, you know, there can be factual issues with the questions and the answers itself, we do a self-verification stage. Again, with DeepSeek Chat itself. So we would just then give back the question and the answer to the model itself and ask it to generate or verify whether the answer is correct for the given question. Since, you know, then the output would be that it would give you a reasoning along with some sources. And if the DeepSeek reasoning output, which we filtered out using an LLM, if it would say that, okay, this answer is correct for the particular question, we would filter this pair and we would keep it. Since self-verification can also have some issues because you could be biasing as you're using a single model itself. In addition, we also did an external verification stage. So in this case, what we did was we scraped these sources which the DeepSeek model thought to be important. We did a RAG stage in a way. So we have the scraped documents. We would give the question and the scraped documents to QuEN-3 4B, which is an open source model, and as well as GPT-OS-S 120B. And we would do also then an external verification stage where if If these models with the scraped documents and the queries is able to generate the answer, we would keep this training pair. As you can see, the whole process is more or less generation and then 2 stages of verification. The reason for doing this was we actually don't want many false positives in our training examples. It's okay if, if we filter out many pairs, but the training examples which we are using, we want that the answers are actually correct to the particular question and it has been verified by multiple models.
So in this approach, it seems, you know, it's a lot of work, different kinds of verification by models. Was there any human label alignment and checking? So, you know, for sure, like it's acceptable, you know, how did you get to a place where like, you're like, okay, this is good enough. You know, this multiple stages of verification is doing what we want.
Most of it was, to be honest, in the beginning was just eyeballing responses. So I would generate something and then I would actually randomly sample a few training pairs at each time and I would check check myself. Okay. And I would sometimes also use ChatGPT also to get an answer, and I would verify with that. So there was a bit of manual involvement. I didn't have, unfortunately, more annotators to do a good study, but I would say the good way to do this would be to have human annotators who would annotate this, would use a tool, that's fine. Probably they can use some tool to get an answer and then verify whether it's correct or not. But for me, I think so, I eyeballed a few responses and That seems to be correct. At the end, actually, yeah, once the dataset was done, so I figured out at stage 2, if you would take certain data points, the answers are actually incorrect. So I figured we need to do more stages. And then at stage 4, when I eyeballed 50 to 100 responses, I was more or less getting the exact accurate answer. So that's where I realized it's good. Again, it was a 20,000 dataset, so it's not feasible to do a lot of them, but at least with 50 and 100, I'm happy.
Yeah.
That's how I think so eyeballing is The way I did, and I feel like it's probably a good way to start.
Yeah, it makes sense. And how important was stage 4 in this?
Yes, stage 4 was— it's optional, but I feel like we did it just so that, you know, sometimes there might be issues with the self-bias. Like DeepSeek generated the question and the answer itself, so it could be in its own memory that it generated this question and the answer, and the verification which it did was actually incorrect. Probably could have memorized itself. So just to not bias towards DeepSeek, we did an external verification. I would say this removed a few of the hard pairs because since we have scraped all the information and then we are only taking those which can be answered by an open source LLM, we are probably losing out of some harder examples, but I feel like it was a trade-off worth just so that we don't introduce false positives or, you know, train pairs with incorrect answers.
Makes sense.
Okay. So yeah, I'll move ahead now. So in terms of training, what you would find is people usually train with reinforcement learning these days. So common technique which has come up is reinforcement learning with verifiable rewards, where you have a policy model where you would feed in the training data. And this is used for tasks with verifiable outcomes. So in our case, for the search task, since we have a short verifiable answer, we can actually use this. There are also ways you can use the long answers as well. So I think so there are other papers which does this, but for today's presentation, I'm going to focus on datasets with short answers. And usually you have a verifiable reward, which would mean that in our case, we check whether, you know, the answer, if it's correct, we give a reward score of 1. If the final answer is incorrect, we give a reward score of 0. So we chose a very simple reward setup. In the literature, we find that actually for search agent training, sometimes the very simple baseline is actually to not complicate the reward function, just the answer as a simple reward works pretty well for your, you know, stage 0 search agent. Yeah. And after that, usually how, at least from SearchR1, which was, I think, so one of the first search agents out there. So they trained using GRPU and PPO where they would use training data such as natural questions and HotpotQA. And their reward function was, you know, the exact match between the predicted answer and the gold answer, which is available in the training dataset. And they used, trained using GRPU again. Pretty sure probably the audience knows a lot.
Yeah.
If you really want to refer what GRPO is, I do suggest that you can look at either of these papers. These are very good repositories to learn more about reinforcement learning. Moving on. So we actually trained a model using the ORBIT dataset, and which was known as ORBIT-4B. Usually search agents have typically 4 stages or 4 kinds of reasoning things, either think, search information, or answer. And on the left-hand side, I show a small case study which was done using question from a Frames dataset, which is also a good evaluation benchmark. And here you can see initially the model thinks for some time. It then issues some search queries, which is then fed to a particular search model, which will give you back the information as top-K documents. And then there are multiple stages of these. And in the end, it gives you the final answer, which is returned by the agent. This is typically how you would find the trajectory of search agent to be, the reasoning and the information. We evaluated Orbit. We were focusing on very small search agents. So that's why we were evaluating 4 billion search agents when we used PaLM 3 as 4 billion as our base model. And again, in our paper, we don't want to come up with a very complicated training stage as we were really comparing the quality of Orbit versus, again, comparable training datasets. So we kept the training pipeline to be the same. We kept the base model to be the same. The only thing we changed was the training dataset so that it's comparable. And we evaluated across different Wikipedia datasets. As you can see, so we have a, we did a much more cleaner evaluation. So we used a much more latest Wikipedia version, the 2024 dump, and we use the BGE-M3 retriever in our case. So if you would see for most of these datasets, which are Wikipedia-based evaluation datasets, we, Orbit performs really well. The one dataset, which I think is good amongst all of these is the Frames dataset. So it's a much more better benchmark. And I would trust if, you know, Orbit 4B outperforms the other ones means that, okay, there's a useful signal that it's actually doing better and as a search agent.
Why do you like the Frames dataset so much? I also see that MUC-Q is also similarly not as saturated as well. Any intuition about why you like the Frames one so much?
Yeah, that's a good question. So Frames I like because it was much more recent and actually for Frames, it has multiple documents, Wikipedia documents, which are used to verify a particular question or to reach the answer. So you actually sometimes need to read through 5 or more. So I feel like MUC-Q was just a single or maybe 2 hops at max. Whereas Frame is really much more multi-hop reasoning benchmark. So that's the reason why I like Frames more than the remaining ones. Not that they're bad, but yeah, it's just, I have a preference for that. So I think so the key idea here is that, you know, synthetic datasets can actually help improve search agents, help improve the accuracy on these challenging datasets such as Frames. Okay. So that was part 2 where I talked about data. So let's now move on to the final part where I talk about search agent trajectory analysis. This. So this is leaderboard actually, which is there from Broscom Plus. If you actually check as of today, it's there. And the number 1 on the leaderboard is actually this GPT-5 model, which gets an accuracy of 95% and a recall of 93%. Pretty good, right? You would actually think it to be the top 1 because it gets good score. But you know, is it really the best model out there on the leaderboard? And actually, when you check the search calls The first model has a huge number of search calls in their trajectories. So to be able to come up with the answer, it actually searched 210 times the search agent to be able to come up with an answer. So I would think that, you know, this model is actually not really efficient. So it does a lot more turns and it's burning a lot more reasoning tokens to be able to come up with this accuracy. So instead, what I would like is the number of search calls or the number of search turns to be lower as possible. So as you can see from the 2, 3, 4 they have comparable search calls, I would think that, okay, the model which actually gets a 90% accuracy and an 84% recall with 12 search calls is much better in my books versus the one which has a higher accuracy. So my point is like, you know, sometimes just measuring the EM accuracy doesn't tell you the whole story. So, you know, you should focus a lot on what is going on under the hood with your search agent.
Just out of curiosity, what was the impact on latency, you know, of those so many more search calls? Is that part of the metrics at all?
I So that is part of that, but I don't think so. So this is a public leaderboard. I don't think they report it. So these are different vendors, like, you know, this was submitted by AI21. The other one was Sail Research. Third one was from MixBread, LightOn, folks from Winnow. So I don't think we have that, but from my intuition is the more search calls would mean that you are becoming less efficient. So you are doing more search calls. You are repeating search calls as well. So I think so this will look at some trends in the next slide, but also you're wasting more tokens to come up with a final Okay. So there's been one or two interesting papers which have focused on the search agent trajectory. So one of them is this Agentic Search in the Wild. And what it does is it actually took these agentic sessions. They consider their, the whole session as a log, and they would actually check what kind of queries were being generated by these agentic models as search queries. And there are different types which they have defined. So sometimes the model would do an exploration one. So, you know, when was the Eiffel Tower built was the original question. Then it asked, what was the Eiffel Tower built So this was more of exploration question. Similarly, there were more like specialization questions and also generalization-based questions. And lastly, repetition in the query itself. So we are trying to classify search agent queries, which were generated by the LLMs itself and seeing what kind of a question was this. Yeah. And these trajectories and these sessions are very useful because they actually show what exactly your agent is generating when it reaches the final answer for your particular original question. There are some results on GRDIM. I think so the only aspect which I really want to focus here is is they did evaluation on certain type of queries. They evaluated multiple systems, and then they have a broad breakdown of these percentage of queries and how many of them are in which category. So the one which I really want to focus on is that declarative, which is almost the 88% of all queries which were there in the benchmark, has a huge amount of repetition currently. So the blue is much higher, which shows that currently these agents actually repeating a lot of these questions, which is not good, which is wasting tokens. And this is something which I'm seeing actually in these agentic search models. What we find is that, you know, majority of them actually are still repetition, which wastes tokens. But yeah, since we did the trajectory analysis, we are able to understand like, you know, this is where it's going wrong. This is where it's probably not able to find something relevant and it goes in a loop and tries to repeat its own questions.
Can you give me more insight into this example? So question 5, what else was built for 1889 Paris Exposition? Why is that a repetition and not an exploration?
From Q4 to Q5. So—
Oh, I see. Okay. It's the same question over and over again.
No, it was from Q3 to Q5. It's a generalization question, but yeah. So it actually changes in between, at least that's what their paper does is that based on the question, they can actually categorize it to different, you know, different categories of clusters of your—
Graph below, this is the same question happening again.
Yeah. It's the same as in, it could be that there are some words which are different, but the semantic meaning remains the exact same thing, just rewritten in a different way. So yeah, many times we actually see this. A good way to see this or see these trends is currently there's under review this work. We have visualization tool which Christina, which is one of the, my labmates from University of Waterloo. So she created Hawkeye, which is a visualization tool for inspecting these agentic search trajectories. Once the website is out, it's much more clear and nice. Nice to see. But overall, from this image, there are 3 things which I want to focus on. So if you actually see the question panel, which is here, so 1, 2 interesting things what we are doing is we, on the right-hand side, you can actually see what all queries were being asked by the search agent. And these are all BrowseCom+ queries. So you can see sometimes they would actually repeat these queries. In our visualization, we did multiple things. So we are trying to see topic shift. So the queries were clustered into different topics and whether, you know, you are getting different topics with each and every search terms. The second was query provenance. So where does the query originate from? So does it originate from the original question itself? Is the query coming from the context? Is it coming from its own memory or is it coming from previous queries? So we also measure that in query provenance and document novelty, which we are measuring that how much new information this document is bringing. So has this document been already retrieved in the previous sessions, or is this a particular new document which is bringing in some new information? So these are different criteria for which we are able to label agent and their trajectories. Hawkeye is a good visualization tool which we developed so that you can easily see what's going on for each query with an agentic search model. This is another panel which is there. You can see on average some statistics, like on the left-hand side, this is for the correct one. On the right-hand side, this is for the incorrect one or the unknown ones. So what is interesting to see here on the top of the thing is that for most of the correct ones, you are having less search rounds. Whereas the ones which is unable to answer, some of them are like, okay, the model is actually doing a lot of turns, but is not able to reach the answer. So it shows that the model is doing a lot of turns, a lot of rounds, but it's unable to reach the final answer correctly. Also, you can see other stats like, you know, what's the token source ratio as well as the new docs and so on. So the idea is that to give you some insights on what's going wrong and what's going correct. Again, I think these ones were done for GPT-5.5 with the Pen3 model. But you know, you can do cool things like you can actually compare what the feature we're trying to add is. You can compare 2 agents side by side. So you can actually check what's the distribution for this versus the other agent and so on and so forth. This agent likes to bring in new documents, whereas the other agent doesn't like to bring new documents more. It's just a visualization tool to understand what's going on if you don't really want to cluster or give it to an agent, you can actually use the tool to visualize it yourself.
And give me some intuition on when the token source— what would you do with the token source? Like, give me an example of when that is interesting.
So the token source is actually the one thing which I would take is that you— for how much from previous queries it's coming. So it's showing somewhat of a repetition there. And the token source from the original also is interesting. It means that that the agents are trying to take words from the original question and searching those. So something which comes from context, which means that it's okay, it's actually understanding and giving out these queries. I think so, some few things which you can take from these metrics actually. Yeah, so a visualization tool helps us identify the trends, helps us focus on efficiency. And also we have additional things like you can compare 2 agents side by side, which I find would be very interesting if you want to figure out why a particular agent is doing well and why another agent is not doing Or maybe compared to retrieval models by keeping the agent same. So you can actually figure out how well that's also happening based on the retrieval models. So yeah, this is PaCAI. And yeah, with that, let's come to the conclusion of my presentation today and the summary. So very briefly, I wanted to just talk that today we talked a lot about search agents and I focused on 3 aspects more broadly. First was evaluation, where I personally feel that that we should construct realistic and challenging benchmarks to evaluate these search agents on complex queries. So BrowseComp is a good example. Because of BrowseComp, the search agents have been much and much better. And like the BrowseComp performance when it originally came was just 10% or 20%. Now you can reach up to like 90%. So you can see the amount of improvement which has happened and the amount of improvement in search agents. Because of a good benchmark, which is out there. So that is very key to improve these search agents. Second is the synthetic data or how you train these models. So since human-generated training data is nowadays almost very difficult and cumbersome to generate for search agents, because you need to come up with the data which needs to do multiple hops. And often this is very difficult for a human to annotate. It's actually generating synthetic data and even more difficult. So I generated like up to 5, 6 hops with Orbit, but it's an open challenge. How you can generate even more harder questions like PROS-COM+, which requires up to like 40, 50 turns. This is an open problem, like how you can generate good synthetic training data, which can go even like up to 50 turns and use that for training even longer horizon-based agents. And lastly is that trajectory analysis. So don't always just rely on the EM accuracy. So metrics are good. They are a good start, but they are not the only thing which you should measure. So you should also try to look at the trajectories and understand, okay, how many search turns am I taking? Am I repeating questions or not? And even if you just view one or two trajectories by yourself, you can get an idea of where the agent is working well and where the agent is failing. And that can be somewhat used as distilling information for your future training run, or if you're improving the model. So I do think like trajectory analysis is also pretty important, can be an aspect for your training model.
First of all, thank you so much for all your work on benchmarks. I think data Datasets are really underrated. Benchmarks are underrated. Everyone uses them, hill climbs them, makes the entire field better. And a lot of the progress in AI is thanks to people who spend a lot of time on these datasets. So thank you so much. It's like really good work. Working at a company, you have your own search agent that you're building. One of the things that when we work with companies and we help them is like sometimes they're just like they have a cold start problem. They want to know, okay, is their search agent good? We spend some time creating a synthetic dataset. Comparison to what you do here, it seems naive. Like this is a really interesting clever approaches to make that synthetic dataset really difficult. Have you seen people take these approaches into their private datasets to generate corollaries in their enterprise datasets for their own benchmarks? Or do you know, have any intuition how this is being used, like these approaches?
So what I hear is that everyone has a benchmark issue. Everyone wants to have a certain benchmark with which they want to evaluate their models. Be it retrieval, be it search agents. What I've found out is that they have this issue, but they don't know how to create the benchmark. Partly, a lot of times it comes down to enterprise data being private. So you're unable to view it at times. So it's very difficult to come up with a benchmark, which is where you are not able to see the data because of security and privacy reasons. Historically, retrieval benchmarks have been more focused on human-based queries. So we would think that human is the one who's searching. So that's why you want human queries, but nowadays is since LLMs are the one in search agents, at least driving the queries and generating the queries, I want to see more and more LLM-based benchmarks. So where the queries are synthetic, the data corpus can be actually original web documents, but at least the queries can be synthetic. And I think it's a good way to measure certain things. Like name interesting few ones. Like more recently there has been this benchmark known as ObliqueBench by Omar, which is a very interesting benchmark if you actually go and read the paper. So I found it pretty interesting is because they nowadays what I've seen is you take a bunch of documents as your cluster of documents. You don't have queries. You try to map these documents in a certain relation. For example, you know, they can have scientific domain. All of them share the same research problem and you are clustering similar papers around that. And then you're asking a question which was synthetically generated, which links all of these papers. And then you're using that for evaluation of your search agent model. And ideally you would want your search agent model to retrieve all of the papers. I do see in academia, people are taking this approach. Human queries are not really the most important thing nowadays. Nowadays, it's, you won't even be sure if, if on Google search, if you're scraping search queries, whether they're really a human or an LLM one generated. So then again, feel like it's nice to have a mix. So, you know, you have some human queries, you have some LLM queries, and then you're trying to evaluate for 2 separate things. Because I feel like human queries, they can be ambiguous, they can be short, but an LLM can give you really good queries. It actually doesn't take much for the LLM. LLM to give you more and more context. It's probably a few more tokens, but for the retriever, it's a big relief if you actually give it much more context. So yeah, I feel like we need to come up with a benchmark which has both human and LLM, although I feel like LLM ones are less out there. So we need to focus out there. I do think like ROSCOM+ is definitely a more riddle style one. So the queries are more of a synthetic nature, but I do feel like these are improving some aspect of a search agent, which is definitely it has improved the number of search terms which a search agent would do. I think so 6 months or 8 months prior, like not many search agents were out there which were defaulting more than 50 terms. Like the tool use, you had to fine tune, supervise fine tune a model to be actually to learn tool use. But nowadays I feel like it's with even a very basic model, a GPT-3 or even the newer ones, like they do pretty well. Even if you give it a very small amount of training data, they learn tool use very well. And nowadays they are even doing much longer horizon. tasks. So I'm sure there are like multiple amazing use cases out there nowadays with like coding and search is like almost in everyone's workplace. You need to retrieve files from a local system. You need to figure out what's the most relevant file. There is also web search as well. You want to try to figure out something which is not being covered by your own local file system. So I feel like from the coding point of view, also in that domain, there are like many interesting use cases for search engines.
And thanks again for coming.
Thanks a lot.
