Modern Information Retrieval Evaluation In The RAG Era
Nandan, a PhD student at Waterloo, argues that traditional IR metrics like NDCG don't work for RAG systems, which need to retrieve diverse evidence for LLMs, not ranked lists. He introduces FreshStack, a benchmark using real Stack Overflow questions that measures diversity, factual coverage, and recall. Current retrievers like BM25 and dense models all underperform against the oracle.
Questions this video answers
Nandan explains that FreshStack mixes code and text (e.g., Jupyter notebooks, documentation). Text retrieval models perform well on generic text, but niche or private domains (e.g., personal email archives) remain challenging. The framework is generalizable: one can construct a small test collection for any domain by gathering queries, answers, and a corpus, then apply the same nugget‑based evaluation.
Nandan advises mimicking FreshStack’s approach but tailoring the dimensions to the use‑case. For patent search, recall is paramount; for finance or multi‑aspect questions, diversity and grounding matter more. Weighting can be equal by default, but practitioners should adjust based on domain priorities.
Nandan notes surprising findings: OpenAI text embeddings rank low, even below BM25, while newer dense models (QuEN‑3, Voyage, Stella) perform strongly. Closed‑source models once dominated, but open‑source models are now competitive. The leaderboard reflects recent community focus on realistic, code‑text tasks rather than classic text‑only benchmarks.
Nandan says that current FreshStack evaluations use the original query directly. Emerging approaches that decompose complex queries into sub‑queries or employ late‑interaction models could improve performance, especially on long, reasoning‑heavy questions. However, FreshStack has not yet incorporated such pipelines, so future leaderboard updates may show gains for methods that break queries into smaller components and perform multi‑step retrieval.
Nandan is nearing the end of his PhD and is focusing on creating realistic synthetic training data for RAG systems. He is also a maintainer of the TREC‑RAG (Trek‑RAG) competition, which will introduce more reasoning‑intensive queries and adopt the diversity‑based metrics introduced in FreshStack.
Chapters
Today I'll be talking about modern IR evaluation in the RAG era. So this is the guest lecture 2 in the RAG miniseries. So I'm Nandan. I'm currently a 4th year PhD student at the University of Waterloo. So before, before I start, a little bit about myself. So I did my undergraduate at BITS Pilani KK Birla Goa campus in India. After that, for a brief 1-2 years, I was working as a data scientist in India. After that, I joined as a research assistant at the UKP lab in TU Darmstadt. So there I worked with Wills Reimers and Irina Gorobets. Recently, since 2021 onwards, I've been a PhD student at the University of Waterloo where I've done internships at Google Research and Databricks. If not, you'll probably recognize me from, from either of these projects or maybe one of them. Where I've been involved with the BEER benchmark, Miracle benchmark, PressTag, and other such RAG systems such as TrekRAG. Today I'll be talking about IR evaluation, and I've broken my talk down in 3 parts. So first, I'll be focusing on the history where I'll talk a bit about traditional IR evaluation, why it's so rich, and some faults which are there in the modern era. Then I will move on to the part 1 where I would more focus on why IR evaluation needs to change in this RAG era. And in the last part, I'll be talking about a modern-day IR evaluation benchmark, also known as FreshStack. To start with, let's first talk about the history, which is the traditional IR evaluation. So the first slide here is something which I just wanted to deeply emphasize is that RAG is new, but information retrieval has been there since like past 60 years. So if you see the first article on the left-hand side, this is taken from the SMART Retrieval System, which was one of the first document retrieval systems published on like June 1965. So it's been about 60 years since it was— it has been published. And this was one of the first works back in the old days where they created a document retrieval system. Similarly, in the 1990s, the Text Retrieval Conference, or often acronymed as the TREC, is a popular retrieval conference, which kind of also up to date provides us with good IR evaluation measures and like benchmarks. So they're pretty popular in this domain. And on the bottom, you can see the classic Google search interface. So this was in the early 2000s when, when also you would see search based on web page and retrieval there. One such example, like I mentioned before, is TREC. So TREC has been very influential in IR benchmarking and also evaluation. So this is one of the reasons why I wanted to talk about it. So this, brief diagram gives you an overview of all the TREC tasks which has been there since 1992 onwards until 2020. So in the old 1992, you can see that there has been like ad hoc retrieval tasks, which has been there since 1992 itself, to more recent tasks such as human in the loop, where they have humans in the evaluation procedure, or they also focus on different languages such as Spanish, Chinese. In language focus. And yeah, TREC did a lot of different sort of retrieval tasks and they have been doing it since, since the 1990s, basically. The first thing which if someone you would talk about traditional IR evaluation, so you would mention them about the Cranfield paradigm. And if you actually go back into history and look at how the term was coined was in the 1960s, actually this person like Cyril David Dawn and his colleagues ran a series of experiments to actually evaluate the efficiency of indexing systems. This is up to date, still used as the standard for, for which search engines are used for evaluation. And this Cranfield paradigm is like very important or crucial in the history because it gave rise to something which we known as test collections for retrieval evaluation and we still up to date. If you see any IR evaluation benchmark, you would find that it is constructed out of these test collections. So there are 3 vital components in any major test collection in the Cranfield paradigm. So the first part is the topics, which is a fixed set of queries. So these are usually a user would want to ask about some things and then these were manually created using humans themselves, and then once you have collected them, so they become a fixed set of queries. Second is the corpus, which contains a fixed set of documents which comes from either web, which can be web documents, or your bio— or research papers from a collection. So a corpus is just a fixed set of documents which can be used to answer your queries. And the third part is what is known as the relevance judgment. So these are often historically done by a lot of human annotators who would look at the query, look at some documents, and assess whether it can answer the query or not. So if it can, it will be judged as a relevant document. If it cannot, then it will be usually judged as a non-relevant document. So these are the three vital components you will find in any major IR test collection, even up to date. Some examples of test collections. So you would probably would've heard about the BEAR benchmark, which contains a variety of test collections focused across different domains or utilizing different tasks such as fact-checking. It can be argument retrieval when you're trying to retrieve a counter-argument given an argument, and, or it can be question answering. Similarly, we also have test collections as an example across different languages. So Miracle is one such benchmark which has test collections for different languages such as like Thai, which is shown in the example below. And you also have passages and both relevant and irrelevant in the source language itself. And third, you would, if you would see any TREC sort of test collection, you would find that there are usually 3 components of it. So you would find a title, which is a short keyword style query, which has mostly 1 to 2 words, or a description, which is a sentence-long description of the same query, or a narrative, which goes really deep down and explains the query and what you want, what, or what all specific items you need to retrieve for this particular query. Yeah. So moving forward. So we learned about a little bit about test collections, about our history. So now let's focus on the BR Benchmark. BERT benchmark was known for one of the first benchmarks to do zero-shot evaluation. And so I'm sure before I wanted to introduce that, so I wanted to talk about why or what is exactly zero-shot evaluation. So you would probably know, or what we probably call in-domain evaluation, is when you are training a model on a certain domain. So for example, you are training it on Wikipedia. And then you are evaluating it on the same domain itself. So this is known as in-domain evaluation. So you have training pairs for your particular domain or your particular use case, and then you train a model on that, and then you evaluate in the same task or the same domain. On the other hand, a more realistic or a more challenging scenario, which we often find in real-world tasks, is known as the zero-shot evaluation. So in this case, you often might not have training data readily available for your own domain or for your own use case. So in, in typically what companies would do or what researchers would do, they would apply an existing retrieval model which has been trained on, let's say, Wikipedia for which it has the training pairs and further it will be evaluated on these different domains. So this setting is known as zero-shot when you don't have training data for your own particular use case or domain. Why zero-shot is necessary is because high-quality labeled training data is often scarcely available. So in short, we usually have 3 kinds of training of data you would usually see in retrieval settings. So first is unlabeled data. So this is typically available in millions, and this can be like webpages across your, across the web or Common Crawl. So you have a lot of unlabeled data. This is really doesn't need any human annotation. So it's like you can have a lot of million of such pairs. The second part is where you need some training data pairs. So for training data pairs, you actually need human annotations to judge whether a document is relevant to the given query or not. And since you can imagine we would need about 100K training pairs, this would span multiple annotators. So you need to have multiple annotators working for probably 2 to 3 months to be able to get at least somewhat of a scale of 100K training pairs. So this is pretty costly. So that's why you'll not find many such rich training datasets in the community. The last one is the labeled test data. So this is also, since you need only a few training pair or few pairs in this case, such as 100. So this is also readily available in the community where you would need only a few human annotations. So the one of the more, one of the major reasons why BEER was popular was because it was one of the earliest zero-shot evaluation benchmarks in IR. So retrieval evaluation was predominantly in-domain back in 2020, 2021, and that led to two issues. So first was the issue of saturation, and the second issue was overfitting. So on the left-hand side, I've showed a plot which evaluates models on MS MARCO, which is a commonly used IR evaluation dataset back in the days. And here you can see actually the graph since 2020 to 2022, only a few points have increased. So this shows that these benchmarks which were there back in the day were already saturated and models were not improving a lot on them. So that's majorly what BEER is about. And it contains diverse datasets which tested model generalization beyond English. Any single domain. So coming now to more recent observations. So in the BEER benchmark, so the first is it's sadly not zero-shot anymore because you, because what we found, yeah, let me explain that in terms of a timeline. So back in 2021, 2022, people would use a single dataset such as MS MARCO or use some unsupervised training and then they would evaluate on the BEER benchmark. So we would consider this setting to be zero-shot because you actually have not seen seen the BIR training dataset itself. Similarly, in 2022-2023, we found there was some works on domain adaptation. So these were also zero-shot because you're not looking at the BIR training data. But now moving ahead to 2023 and 2024 to present, people have started using the BIR training datasets as well. So there are only a few of them available. We don't encourage people to do that because the evaluation is not zero-shot anymore. And also these days there are many private models, so we have no clue about what the training datasets look like. So we are not sure whether these settings are fairly zero-shot or not.
And are the BIR training sets, are they, is there, do they intersect with the test data as well? Or is it specifically for training?
Yeah, they do. So the domain is similar, although the queries are different, but The distribution is somewhat similar. So if you are training on a particular dataset, not many of them have training pairs, like maybe 5 or 10 out of them have. But if you train on those, you by default will get a better score on those exact same datasets for which you have trained on. So a good use case for that is Fever, for example, has training pairs. A model which includes the Fever training pairs will automatically do much better than the one which is being evaluated zero-shot. So it's an unfair setting if you don't know what's behind the hood, what training data is being used to train these models. And yeah, so coming back, like, it's basically a repeat in history was like when BERT was introduced, people were fitting on the MS MARCO dataset or some other datasets. I found that in recent history, like, this has just repeated itself now in 2023. For, or up to present in 2025. Second is a very practical use case is like when I talk to some, when I talk to someone who is not familiar with the field, they would often ask, which model should I choose for my own use case? So now the BLEADERBOARD is actually now not a great signal to look at models anymore. So the MTB benchmark, if you would know, so the V1 version of that actually has BIA datasets. And if you actually look through the leaderboard, it has over 400 models with scores, and it's like very congested. There are about 10 to 15 models all within just 1 to 2 points of average scores within each other. You would probably face this situation, like you have many models and you're actually not sure which one to pick, um, because all of them are so great. They're only just within a few points away from each other. So in this case or scenario, what I usually like to ask is, all top 20 models, they're really great on the BL leaderboard, but how do they perform on some other benchmark, on some other task or some other niche domain? That's the question which I would really ask before picking a model. So in order to summarize now, so there are some limitations and challenges in test collections. First is existing test collections are static. Now they're old. Since like LLMs are recently being trained on new and new data, there's always a possibility of data contamination. Second is what we have sometimes is incomplete relevance judgment problems in existing text collections. What this means is that since there is a human annotator labeling the queries and the documents, you might miss out— the human annotator doesn't really look at all of them, so it only looks at a few samples and annotates them. So this often creates this incomplete relevance judgments or a shallow labeling problem. Third is these existing test collections lack realistic questions and answer distributions. So this is actually a tweet from today. So the authors of HotpotQA themselves told that I'm asking you to stop evaluating on HotpotQA in, on, in 2025 and beyond. So this really shows that we need new benchmarks in order to evaluate our agent systems, our retrieval systems right now. And lastly, this image has been taken from the MTV authors in their recent papers, which shows that existing benchmarks like BERT are overfitted and saturated. So what they interestingly tell is that the best models actually getting the highest performance may not generalize well to your own use case. So they actually recommend some models which are not so high in performance, but rather generalize well to your own domain.
In the color in that scatterplot or in the— yeah, in the scatterplot, is it the green? Green just is like higher score on a test set?
Yes. Yeah. So I think so. They measure like the generalizability in a way, like measuring whether the in-domain, whether there is a test and train overlap between the models and these datasets. That's how they plot that. So many of these recommended models didn't actually train on these datasets, so they get a lower score, they in general, they recommend, at least in the papers, the authors recommended that they do well on your own use case. Okay.
So this is like classic overfitting. The model is doing really well on some benchmark score, but not generalizing. And they recommended a different set of models that are not necessarily at the top that do generalize better.
Exactly. Yeah. And practitioners don't know about this. You would probably choose the top one model. Best model, but what they really recommend is not doing that, like trying to actually use some other models, which, yeah, they're recommending that paper. Awesome. Yeah, so that was mostly a bit about history and talking about the BEER benchmark. So let's now move ahead on the part 1, uh, where I'll talk about why IR evaluation needs to change in this RAG era. So let's look at like how retrieval was there before and now in the RAG era. More typically in the history of in the history of IR, you would have a user who would ask a search query and you would have a certain knowledge base of documents, which can be PDFs, web pages, text, et cetera. And then you would pass these to a search model, which would then give you a ranked list of documents. So imagine if you're on Google Search, you ask a search query, you are redirected to the Google Search webpage where you're shown the link highlighting the webpage and certain, and then the list of all these rank list of documents. So this was the traditional search setup. How this has changed in the RAG era now is, now instead of asking questions on Google Search, you would find often users are asking questions on web apps such as, sorry, such as ChatGPT or similar AI apps. The knowledge base and the search part remains constant, but the difference here is that instead of showing the web search pages back to the user. So now the retrieval uses these retrieved contexts and prompts along with the original query to a large language model, which in the, in turn then gives you a streamed text response to your answer. So you, as a user, you're not going to another website. You are on the same API, on the same webpage, and then you get back an answer, which is LM generated. And has citations to these particular docs. There is a mismatch between a traditional search and these modern-day systems, as you can see. And to explore or to explain this more visually, let's first look at or meet our traditional search user who was very impatient and wants often Google search results returned very quickly in comparison to our modern-day RAG user who patiently waits until the question has been answered by ChatGPT or a similar web app. So in the traditional search use case scenario, user would ask a very short query on the search bar. Search results would be really quick, often usually takes less than a second, and then the user will look from top to bottom analyzing the 5 to 10 search results it's got. So yeah, in traditional search systems, what we found was like user wants quick results and they scan through these ranked results from top to bottom. According to some recent studies, like by Cresswell et al., they found that the user would most likely click the first relevant search result in this case in traditional search systems. On the other hand, if you now look at the RAG scenario, so now users can ask much more longer queries because the LLMs do allow for much more longer queries and answering takes much more time. So maybe answering a question may take up to 5 to 30 seconds or even more with like Deep Research and other more recent systems. And what you would get is an answer summary with cited documents which will be present. So in the RAG setting, user now waits more patiently. And what is more important is you don't get these ranked list of results anymore unless you click explicitly for it. And also since the LLM is known for hallucinations these days, so the user might actually click on the link to assess the source, whether the LLM is telling the truth or whether it's hallucinating. So these are the two things which we observe. So in the traditional search history, our objective or their objective was, did we rank the relevant page at the top? So that was the objective, to push all the relevant pages as much as possible on the top. And there are two metrics which evaluates this. One is the MRR, which measures the position of the first relevant document. Which the user is likely to click. So if the first relevant document is in the first position, you get an MRR score of 1. If it's in the second, you get a score of 1/2, and so on. And similarly, an NDCG is also a very common metric you will see, which users use to evaluate search systems, which measures the ranking of the relevant documents in top K results. What I wanted to say is that the MRR is actually a very good measure to evaluate the traditional search system because users are like impatient, want to click on the first search result, which they think are relevant. So MRR is a good metric, which actually matches this motivation or objective. Similarly, NDCG is also another good metric because you typically would see probably 5 to 10 search results in the webpage and then a user will choose within those to actually click and check their webpage and their information. On the other hand, now an objective for modern-day RAG user changes. So now our objective is, did we fetch every piece of evidence needed for the LLM to answer this question? So that's now the objective, which has changed from the original search objective. And question is, should we still use MRR here or should we still use NDCG here for evaluation? So the answer for that is no, because users don't click search results anymore. So that's why MRR may not be a useful metric to measure anymore. And similarly, NDCG is also not useful because the users don't see a ranked list of search results from top to bottom. They only focus on the answer and whether it has citations to documents. Okay, so yeah, that was me talking about different setups, and now let's just focus why metrics need to change. So in comparison to traditional search and RAG, the objective of retrieval in traditional search was to give you a very shallow ranked list of documents. The users are very impatient. They would probably view only the top few results. On the other hand, retrieval in RAG is what I think to be more known as evidence collection because you have, you need to provide information to your LLM, which will then use it to answer your question. Second is quality of sources. So first, in traditional search systems, user would provide links or look at them by themselves by clicking at each web page. But now in RAG systems, you now require to verify the document source quality, whether it provides informative, whether it is informative or factually correct. Third is efficiency and relevancy. So in traditional search systems, like I mentioned, relevancy is pretty important. You want to push all the relevant documents in the top K results back to the user. But what I find in RAG systems is important is a minimal spanning document set. So for example, you have probably 3 to 5 documents which can, which can be used to verify all your key facts which should be there in your answer. So if you have 3 key facts and if 5 documents can sufficiently used to answer these key facts or to verify those, So that's like a good enough document set to be provided for the RAG use case. Okay, so talked about brief motivation. So now let's deep dive into a modern-day IR benchmark. Let's deep dive into FreshStack now. So why we build FreshStack, the motivation. So we found that most academic RAG benchmarks often suffered from 3 things. First is they lacked realistic questions. And/or answer distributions. You would find that some benchmarks are artificially easy because they're built as RAG datasets. And third is existing benchmarks can be static and unspecialized. So what we really need is instead of RAG researchers trying to evaluate on these academic benchmarks which are present, we really want them to work on real-world challenges or come up with such benchmarks. So the problem definition in our research work, which we did, was can we build such an automatic framework to construct realistic RAG benchmarks? And in particular, we wanted to focus on 4 things: whether we can utilize real user questions and answers, whether the answers can be grounded from documents at real time because the documents keep on changing, keep on updating. The framework should be scalable. So you want to use this framework for your own use case or domain. And fourth, which is very important, is it should be new so that it should avoid data contamination for a more robust evaluation setting. So how did we start building FreshStack? So first question was where to get our queries. And in our work, we chose Stack Overflow because of 3 reasons. So, well, the benefits of Stack Overflow are first is users ask long and really real-world questions there. And a point to note is they're not really generated for RAG purposes. The community maintains the top voted answers. So the quality of the answer is also really good there. And third is open source and not static. So you keep on asking questions these days with evolving benchmarks and toolkits. So it's not static in time. So for the benchmark, what we did for PressStack was we sourced queries from 5 different recent and niche topics. So here I show all 5 of them. We have LangChain as one topic, which is, which was introduced in 2023. And similarly for other topics, we have more recent versions or recent, yeah, more recent versions for them, which are new in, in new in time. And on the left-hand side, I've shown some timeline of the questions when they were asked on Stack Overflow. So you can see a majority of them were asked in 2023 or after in the first 2 quarters of 2024. Similarly, where to get our documents. So we chose actually GitHub repositories, also because the GitHub repositories are constantly updated and evolved by the maintainer of the repository. And it contains code, text, all forms of technical documentation, which is useful to sufficiently ground or answer the query. And third is open source and not static. So Yeah, tidbit from Freshtag is that you will not find in other benchmarks is that Freshtag queries can be often long and they can be at times even longer than the answers themselves. So if you can see here, the green shows that Stack Overflow queries from two topics and Freshtag are actually much longer than the answers which are present there. So let's break down now the pipeline of what, how we created the Freshtag dataset. So in step 1, what we call, what we do is known as nuggetization. So nuggetization breaks down the Stack Overflow answer into different facts or statements. So you can imagine that these facts are actually crucial to, to be able to answer your query. So this is what it's done in step 1. You would, we used GPT-4o actually to generate these nuggets or these key facts automatically. In step 2, we do oracle retrieval. So this means that given now we have a list of nuggets, we have the original query, we have the answer, and we have documents which is there in the corpus, we want to sample a few documents for each query to be able to come up with this test collection. So in this case, we use different models such as like BM25 model, which is a commonly known lexical model, and similarly other embedding-based models such as E5, BGE, and Voyage. So we use a hybrid or a combination of these to actually come up with our sample documents. Once we have our sample documents, in the third stage, we do support for each nugget. So this means that we actually check whether each document can be grounded or is supported by the retrieved chunk or not. So you take each of the key facts and you check it across these sample documents. And we, if we do this automatically using GPT-4o to say whether it's supported or not. So yeah, let's now go into an example of nuggetization. So this is a Stack Overflow question and an answer example. So this was taken from LangChain. A user had issues from using the chromadb.from_documents function. And the answer was actually just using Hugging Face embeddings instead of Sentence Transformer embeddings and the code started to work. The kind of nuggets we actually generated in this case was first, which kind of summarizes the problem here, was like, the error is due to a mismatch in the fun— function signature. The second part came from the answer. So the second nugget was like, okay, you need to import Hugging Face embeddings. And third, you initialize the Hugging Face embeddings with the Sentence Transformer models. And fourth, you pass this Hugging Face embeddings function into the QoMA from documents function. So this is the list of nuggets which were generated for this particular example. In the second stage and the third stage combined, so first you get a sample doc. So this is an example of a sample doc. We retrieve such sample docs using an Oracle retrieval stage. And yeah, sorry, after that, we actually compute whether it is doc supports this nugget or not. You would actually go through each part in the document and see whether, you know, it actually supports or not. So in this case, this document has been supported by 3 out of the 4 nuggets because it has this information present in the document which supports the information in the nugget. We did evaluation of Freshtext. Like, I wanted to reiterate this point. So for RAG evaluation, I think so, we wanted to have a minimal spanning document set which can cover each nugget or each aspect in the answer which is required to answer the question. So we focused on 3 metrics here. First is a diversity-based metric. So what we used is known as an Alpha-NDCT@10 metric. So simple way to analyze this is that if you, for example, for each key fact, if you have multiple documents citing it, so for example, for key fact A, if you have 2 documents which cites it, you ideally want any one of them which supports the best because adding both of them might be redundant. So this is how diversity plays into our thing.
And again, when you say multiple documents, in this case, it's multiple pieces of code from the codebase?
That's correct. So it can be multiple pieces of code or technical documentation and text. Awesome. Yeah, second after that, looked into grounding. So we calculated a metric known as coverage. So coverage measures how many unique nuggets were you able to support with your list of retrieved documents. So for example, your answer required 4 key aspects to be covered, but only retrieving these set of documents, you're able to cover 3 nuggets out of the 4. So this is how we computed the coverage. Of unique nuggets. And the last one is also focused on relevance, which measures, for example, for a given query, you have a set of all of the relevant documents. How many of them were you actually able to retrieve in your top 50 or top K documents? So that's how you would compute the recall score at 50. Awesome. So yeah, moving on now. So the FreshStack results. So if in case you're interested, I would not go into deep just running a bit on short of time, but you can refer to the paper for them. Just discuss some key highlights and takeaways. What we found on FreshStack was like current retrieval techniques were not able to retrieve really well on the benchmark. Um, and we found that no individual model right now performs really good across all topics. So for example, like right now, if you use maybe a Voyage model or E5 Mistral, so individually they do not perform or so well in the benchmark. Um, third is often you would find people use free rankers on top of retrieval models. They do enhance the retrieval performance, but sometimes they, they do not do well, and you need to explicitly fine-tune these free ranker models. So this is, again, if you want more information, probably best to refer to the paper.
But is there one technique that was like Pareto optimal, like maybe Fusion Voyage 2 or something? Maybe seems best on these categories, or just curious.
Yeah, that's true. So yeah, right now we saw like for the paper, Fusion works well in our case, but I'll show in the next slide. So I recently evaluated more models and they are like outperforming even the Fusion one. Um, so yeah, but what I really wanted to highlight right now is in the last point, which is like Oracle is like much higher. So there is like a lot of gap which shows that we need to build better models for this particular benchmark so that we can bridge this gap. So yeah, coming to that point, so I've been more recently putting effort into adding much more recent models into the leaderboard. So I've been maintaining this FreshStack leaderboard. If you're interested, it's like at this GitHub repository, sorry, at this GitHub webpage. So here you can see I've highlighted a few new models such as like the QuEN-3, 8 billion model outperforms now the Fusion models. So the 4 billion, the 8 billion models I recently evaluated, they're like new and they've been outperforming the Fusion ones. I'm also added some more like Jina V4. And similarly, if you want to be, want to see all these models, you can refer to the GitHub or to this leaderboard, which is hosted there. Lastly, in the bottom, I have an easy collab script, so you can easily evaluate your own retrieval model, or I have some examples there where you can evaluate your model on the FreshSAC dataset and also compute these diversity-based metrics and relevancy-based metrics. So there's an easy script which I link also in my slides below. So yeah, TL;DR, what did we learn today was like, first is the traditional Cranfield paradigm is often useful for traditional IR. Misses out evaluating critical aspects in modern RAG challenges, applications. Benchmarks like BEER encourage zero-shot evaluation, but are scaled now with extensive leaderboard overfitting. So they provide a very low signal in terms of like usability. Second is that the traditional search and the RAG setup has a mismatch. You don't see the rank results anymore and retrieve results are just evidence collection to the LLM right now. And second is you need to evolve these metrics. So right now your metrics should be relevant, be diverse, informative, and correct sources. So you should retrieve such documents and evaluate these criteria. Lastly, we talked about FreshTag. So FreshTag is basically a recent benchmark, which is one of our works, has realistic queries and answers, and is continuously updated. Yeah, and FreshTag, like I mentioned, breaks the answers into certain facts or nuggets and does evaluation based on multiple modern IR metrics which I mentioned, which covers diversity, coverage, and relevance.
Is it okay if I ask you questions, or do you have—
yeah, yeah, some time for that? No, I have time. Yeah, yeah, I have time for that.
Yeah, really fascinating presentation. And yeah, I think it's really— FreshTag is really interesting. What is Your read on if, how well the FreshStack generalizes, let's say outside of domains of coding and the domain that Stack Overflow tends to be in. Do you have any initial reads on that or have you seen anything? What's your feeling on that?
Yeah, so FreshStack has like coding and text mix. So the documents are not just code, but also a lot of these GitHub repositories has like Jupyter Notebooks and also text documentation available in the repository. So I pretty much think of it to be a text and a code together. What I personally find is that text retrieval is— models do pretty well right now on generic text retrieval. What really is the challenge is probably like very niche domains or like a private-based setting. For example, if you want to run search on your own set of emails and you want to find the most relevant information there. So these setups are much more challenging. So I do agree that probably the models which do well on FreshStack would probably generalize well in the generic domain in terms of text. But yeah, I personally don't know whether they might do well on niche domains and things like that. But yeah, FreshStack is an abstraction for code. I feel like the framework is general. We probably can extend it to different domains because What we really need at the end of the day is just a set of queries and answers if possible, and a corpus. So what I would do is construct a small test collection for that particular use case and then use that as judgment whether it does well or not.
Do you mind going back to that slide that you had? It had 3 parts. Yeah, this one. Yeah. Grounding, diversity, grounding, and relevance. Yes. This is a really interesting approach. Do you think that for very domain-specific applications, people should consider creating retrieval evals like this? Let's say you have a financial research AI. Should you consider mimicking this approach if you're trying to optimize your retrieval for RAG? What is your opinion? To what extent should people think about generalizing this?
Yeah, that's a good question. So I would probably— mimicking is the correct answer, but with some careful things there. I feel like as a use case, you would need to first determine whether— what kind of a search system are you building on? For example, if more recently, like, for example, let's take a very old challenge, like patent retrieval is basically an old a very important use case. And when a user gives a new patent, they want to make sure whether the patent is not already duplicated in their search base. So for their use case, actually taking much more time to search and giving, but they need to make sure that there are absolutely no patents which are duplicated. So for those kind of scenarios, I would imagine recall is very important because you want to very heavily find the single relevant document in your set and make sure you find that. So I feel like for those use cases, you want to focus on a recall-based metric. On the other hand, if you're more focused on the generic, fres— or general finance domain, like you mentioned, I do imagine like a diversity and groundings are important. Grounding is important is because your answer may cover like multiple aspects. So an example which I usually take in tech is that, okay, how does Taylor Swift's age affect her relationships? And she has a list of multiple different personalities. So you actually want to credit each source by referring its source information or its document. And, and in finance, it often might be crucial to, if you want to answer a question, you want to cover all its aspects clearly. So I do feel like diversity and grounding is important. It may not be the most, it's not a standard yet. So I've not seen RAG being standardized for other people to use these metrics, but that's something which I personally am more and more interested on to build a standard. Information Retrieval has built these NDCG or other metrics, which has been there in the community since past, like many years. And they came, it's been heavily tested. It's been tested by users. It's backed by statisticians. Those metrics, if you use, you pretty much know you're doing the correct eval. Evals, but right now in the RAG eval space, people are not sure about their evals. I've seen many researchers use MDCG, which may not be the most optimal one for your use case. I do feel like this is a good subset for you to start on, but I don't feel this is the standard yet. So maybe we need more.
I think these are good things to think about. Like it makes sense, for example, you have different aspects of your answer, different facts that you need. If you're going to present. Presenting some search results to a user, the more different documents are supporting that answer, you would trust the answer more if you present it correctly and you're citing them and whatnot. So I think it does resonate with me a lot. From what I gather from your answer, if you're in a domain-specific use case, these different dimensions might be, might have relatively different importance. Maybe there's certain cases where you might not need diversity as much, or you might not need grounding in different things. As much, and you might want to focus on one. Do you think in practice, do you envision people may do some kind of weighted score across these different dimensions?
Yeah, that's a good question. First of all, that's a good summary. I do agree, like, these three are like different dimensions. I personally don't have a good answer. I feel like right now for our benchmark, we did like equal weighted, so you usually compute averages. But I do feel like this depends upon each domain use case. The company or the person who is coming up with the benchmark needs to figure out what diverse dimension they want to focus more on. So if they want to focus more on diverse results, they can tune the weight for diversity to be higher. Or if they want to be covering many aspects, for example, if you want to talk about a comparison question, who would win the next, the next political elections and you want to summarize all the different aspects which are out there. So I feel like a coverage will be very important. Like you want documents which should talk about multiple aspects and you want all of them. So yeah, I really feel like maybe the easiest one, if someone doesn't know, might be an equal weight. But I do feel like this depends upon your test use case and how you individually rate them.
No, I think it makes sense to think carefully about what matters to you and your use case and think about the product. Like, what does diversity mean to you? Does it make sense? It doesn't make sense to just blindly use equal weight, probably for most products. Okay, another question is, do you know the leaderboard for the FreshStack? The things at the top of the leaderboard right now, do they match your intuition or the vibes of what people are saying are the best retrieval models? Does it track with you? Like when you saw these ones on the top, it was like, oh yeah, okay, these are I feel these are the best retrieval models that I've heard.
Yeah, that's again, a good question. Here, what is very interesting is like there are two rows. So if you see the OpenAI text embeddings, so which are like highlighted in blue, they're actually way down in the bottom. OpenAI text embedding 3 small actually even underperforms BM25 here, which is very interesting to see. So I do feel like there's been like. Progress in the community in the recent, I would say, 6 to 8 months where they are actually focusing not just on beer, but extending their evaluations to like different settings. And that's something which is encouraging. So I do these quant3 models, like maybe the 8 billion might be too much, like too, might be too heavy for someone who wants to use them. But if you have the compute or if you have the money to host them, yeah, it's probably does really like it does really well in the code setting. What I found was like, I think so since 2024, before, like last year, like there were more recent closed source models like Stella, or sorry, Voyage were like closed sourced and they were doing really well on like coding based things. But I've recently with GPT-3 and other recent other, other open models, I feel like the competition is now at par. So I do feel like open source models are probably maybe equally as good as closed source ones. But yeah, again, what I call as open source is like very vague at times these days. Like these days when open source, I only mean the model is hosted with its weights, but I have no clue about the training data and other things. But I do think it translates these days. Like I'm finding like the authors have finally chose not to focus on existing just text retrieval tasks. They're like other benchmarks, which I didn't cover in my presentation, but like BRIGHT, is one of them, which focuses on more reasoning-based queries. So I feel like people have now started to focus on either domain-specific, so PressTag is code-text mixed, realistic user questions. So it's one dimension there. There can be other dimensions like BRITE and other benchmarks focused on much more difficult or reasoning-intensive queries. I think so BrowseComp benchmark by OpenAI is also one of the more recent benchmarks, which I saw, which focuses on creating very difficult questions when you need to do a lot of search results and web pages to be able to answer a question. So I like that there are like different kinds of benchmarks which are coming out. And I'm guessing the trend is there, like we are improving when I'm seeing the Quentri models, like they are doing well. The Stella model is also a really good model, which I personally also like, which is there just below the fusion model there and in terms of the leaderboard. And Stella models has been there since the early 2025. That's very impressive for its size. Exactly. Yeah. So I really like the Stella model, like 1.5 billion. It's not bad. And even the 400 million does really well there. What I usually have experience with is that there will be some models which will be always climbing up the leaderboard. So there will be models who would want to do better and better. But you would find that the efficiency and the trade-off, the performance cutoff— oh, sorry, the efficiency-performance trade-off is much more interesting there. So the moment you make better and better bigger models, you'll— they'll start distilling them into smaller models, and then those smaller models are actually much more useful for practitioners. So here in this case, like Stella 400 is an example, but I imagine many more smaller encoders, which I feel like Ben talked Or more others in this series will talk about, like encoder-based models are also really an interesting direction here also, which I'm hoping that they would also reach this level, but they are like much smaller. Yeah. Okay. Last question.
I could keep asking you questions. It's a very fascinating topic. So the people coming after you in this series, so we went over in this talk, like some very interesting. Things to consider when evaluating RAG that I haven't seen talked about that often. Most often people are talking about just using standard retrieval metrics, R and DCG. They're not thinking about these other dimensions. And I think that's really thought-provoking. And so now that we have this refresh of how should we think about evaluating RAG and these other things you talked about. How does this interact with, so there's many ways of doing retrieval, like you touched on, there's like naive vector search, there's it with keyword search. I think we're going to talk about late interaction and we're also going to talk about reasoning models and like how that affects retrieval. How would this leaderboard change with reasoning or late interaction or those kinds of things? Does it? Is it, does it stay stable? What happens in this?
What do you, how would you think about it? Yeah, it's a good question. All of these models right now, so this is a very initial leaderboard. So we evaluated the original query and take that and evaluate on that. More recent works I've seen tend to break down a very complex query into much smaller queries and do multiple retrieval steps. I imagine doing this would be somewhat. The future, which is upcoming. In a very ideal scenario, we want this. We don't want to do this, but right now I feel like an original query, if it's too long and too complex, it's actually helpful for the retrieval model to break it down into smaller components and then do those. So we didn't do such kind of evaluation on FreshStack yet because I feel like these things were being much more recent. FreshStack came out early in 2025. Um, but I imagine these different types of search systems, like late interaction, which I'm sure other Anton and other people would talk about, they're like very good. I think so even in originally in BEER, we saw that they're good in generalization. The reason is because you get to do, get to see the query tokens. You get to see some document tokens, which I'm sure he will explain much better terms, but yeah, because of that, the generalization doesn't get affected much, but embedding models is much more challenging because you embed them. You embed your queries at test time, you have embedded your documents previously, and then you are just doing search. Yeah, multivectors are like very interesting. I personally didn't touch much in terms of retrieval models because I know there are other speakers who are like much better or will talk about this. But I do feel like breaking down the search into multiple subqueries would actually make some systems go on top of these benchmarks. And that's something which I've not done myself, but I think the community, once they start picking up the leaderboard, they'll start doing that. Um, but what I really wanted to focus on is we really need to change the evaluation. We should not just use NDCG@10. So that's has been my effort with this. That makes sense.
So like no matter what kind of retrieval you're doing, whether it's naive or any other kinds of methods, you still should, you can still use that same retrieval methodology like you just went over, it seems like.
If you would tell me this probably 2, 3 years ago, like breaking down search queries as a user, I would say, okay, that's too slow because one of my research agendas is being efficient. The user wants like search results within a second. So doing multiple search queries takes more time. But again, the RAG era or this RAG scenario is, it's really interesting. So if you're doing search for RAG. I'm sure like people can wait a second more, but wait for better results than a rushed retrieval stage. So that's like very curious to me. Like people can now really do multiple systems, put complex pipelines, make sure you get the most correct or the most diverse set of answers out there to be able to assist your LLM or to be able to answer your queries. So yeah, so I, yeah, I think so that summarizes my point there. Yeah. That's great.
This is a very excellent talk. Aside from being fresh stack, it's also a fresh perspective of evals for RAG that I haven't heard, or for retrieval for RAG that I haven't really heard anywhere else. And it makes sense. So thank you so much for showing this to us and presenting it.
Yeah, no worries. It's a bit difficult, like, BEER benchmark was one of my previous works to criticize it, but I feel like it's the time of the year where we should like retire old benchmarks, like HotpotQA authors came today in the tweet saying that you should not use that. I feel similar things will start happening with these existing benchmarks. So I'm hoping that the new research community will build something similar to FREDstack or even better, or continue using these for next-gen evaluation of retrieval in RAG systems, which is important. Yeah. That's great.
Before we go, I'm gonna bother you with one more question. What are you working on next? What is your current research agenda?
Yeah, it's a good question. So I'm mostly like nearing my end of my PhD. So I'm, I would probably defend next year. So it's the time where I start to wrap up things. Personally, I like evaluations, but these days I've been trying to focus on not exactly evaluation, but also like training data. So I found that apart from evaluation. Having training data is like a crucial issue with like systems like RAG-based systems. You want to give realistic training data. So I've been trying to focus on that. So I've been trying to like, trying to better training data, see if we can create really realistic synthetic training data. And those has been some of my recent works. But one thing to mention for the participants, I feel like, so Trek-RAG is basically a Trek task for RAG. Which I was a part of last year. I'm still continuing as a, as one of the, one of the maintainers or the, what do you call, organizers there. It's going to happen this year summer. So hopefully we have much more interesting questions there. So we're building out much more reasoning intensive queries there. And then in Trek RAG, basically we give out the queries, participants participate, give out their RAG results, and then we in the end pool and see how well do participants do and also hopefully come up with a test collection there. So that has been our effort with last year's TREC RAG, but also this year. So that's something if you, if I'm personally looking forward from the evaluation point of view. So I want to introduce my, these diversity-based metrics there. So I want to really introduce Alpha-IndicG and other diversity metrics, which we talked about because this wasn't there last year. I was still like researching, we were still standardizing, but this year hopefully you'll get to see much more of these metrics being used there. Yeah, no worries. So let me just go through this. Oops. Yeah. So if you just go to like trek-rag.github.io, oops, I'm not zoomed in. Yeah. So it has Trek RAG where we talk about some tasks which you have and yeah, you can go through there and I feel like you can. Yeah, you can register from this page. So the Trek page, which I talked about today. So here you can register for all the tracks. So there are multiple Rack tracks, not just limited to ours, which are happening. And yeah, these are like some of the—
to be clear, this is a conference.
Yeah, it's, it's like, it's a conference, but you usually in Trek, like people participate and then they give out their systems, then we evaluate those and at the end we just ask the top presenters to present their work. So it's mostly just for information for others to know what you are working on. Also for us, it's cool to see how people are coming up with new and new RAG systems. And once we have done all of that, we try to combine and create a test collection out of it so that the community can use it afterwards. So that has been the motivation of TREC every year. So, okay. So like a competition. In a sense. Coopetition. Yeah, exactly. I think so. The word they use is cooperation plus competition. So they famously use the word coopetition. Yes. Great.
I'm definitely going to check this out. This looks really cool. Thanks for sharing it.
Thanks a lot for inviting me. It was like a nice challenge for me to summarize all of these things. Not much research has been done in these. I feel like a lot of it is like still in the weeds, you could say. But yeah, I tried my best to like at least encourage people to think about these directions. So hopefully that's it. Sounds great. Thank you. Awesome. Thank you very much. Bye-bye.
