I don't use RAG, I just retrieve documents

Welcome everybody. Really excited about this first part in this 5-part series about optimizing and evaluating RAG. Today we have Ben Clavier, who is an R&D researcher in retrieval at Answer AI. Ben is really talented and gives the best talks on the subject. So it's like a powerful combination. I'm going to go ahead and hand it off to Ben.

Yeah, no pressure. Thanks for the introduction, Hamel Husain. So as usual, if you follow my Twitter, I'm a bit sick today, so sorry if I speak a bit fast, I'm a bit unclear. But thank you all for being here, that's more people than I expected. But, and thanks Hamel Husain for inviting me and talking me into doing this. So right before we start, today's talk is going to be very high level. I'm going to give basically somewhat strongly worded rebuttal to the whole RAG is dead. And then I'll be introducing the next 4 speakers who are like really, really talented information retrieval practitioners and will be going much more in depth about specific aspects of things I'm gonna very, very broadly cover today. And yeah, with that said, Hamel Husain, if you can go to the next slide. So about me, I'm based in Japan, so I think it's quite late for most of you, but for me it's 9:30 AM, which, I mean, I got up early, let's face it. 9:30 AM. I'm based in Musashino City, which you might know because that's where the Ghibli Museum is, hence my brand on Twitter being a Ghibli film. I do machine learning and retrieval R&D at Answer.ai, maybe somewhere else soon, but I won't talk more about that right now. The picture on the right is the picture you've probably seen of me on Twitter because Hamel managed to dig it up, I think from my LinkedIn, and that's like the one picture of me that exists online, so no screenshots or anything. I'm happy with that being me forever. Thank you. Recently, you might have seen my work on things like Ragatou, more Colbert models, and obviously I co-led ModernBERT along with Benjamin Warner and Antoine Jaffin, who's also giving a talk in this series. And yeah, I think that's what all there is to say about me. And yeah, RAG is dead. You've seen that on Twitter, you've read that, everybody's told you RAG is dead, RAG doesn't exist anymore, we don't need RAG, RAG is dead. It's what you're hearing all the time. And you'll see that I don't necessarily disagree with that statement, but I think it only applies for an extremely narrow definition of RAG, which to me is not actually RAG. Like, it's not what RAG is, and it's a bit of a marketing push. So Hamel Husain, if you can go to the next slide. The big point is RAG is dead, as in RAG like you've heard about it in 2023 is dead, but retrieval is not. Retrieval is very, very much alive. And the reason for that is that RAG is actually a really simple concept. And I think we tend to forget what's in the name when we just go on about RAG. And it's actually kind of funny because the paper that coined the name RAG does not actually do what we would consider as RAG. It's very far removed from that, but the name was catchy. So everybody just decided to kind of ignore the paper and keep the name, which, fair enough, I'm gonna say it's a good name. The RAG stands for Retrieval Augmented Generation. It's just 3 words and what it actually means is just like a basic pipeline. You retrieve documents somehow, you just get documents from somewhere, from a knowledge base, from the internet, from Google, and you pass them to a generative model. So most of the time an LLM so that its output is augmented by the content of the documents. Now augmented is a bit of a weird concept because it's a very pretentious way of saying the model didn't necessarily know what's in the document and now it does because it's in the context window. We're augmenting boosting the generation because it's information that was not in the weights or could even conflict with the weights and it's given in the context window. And as a concept, you can't really kill RAG because there's always going to be a need to give model access to information that it didn't see during training. And we'll talk more on that in like a couple slides. I'm not going to go into more details right now, but just remember that like you will always want a way to access the outside world. And so if we want to like visualize what RAG looks like, this is your normal generative pipeline. You have a prompt, the LLM does generation, you get an output. That's it. You're happy. That's every LLM generation without RAG boils down to this. Now, when you generate with RAG, it doesn't actually need to add like a complicated step, as you can see in the next slide. You'll see that when you generate with RAG, all you're doing is you've added a fourth step, which is you add context documents. If your prompt is augmented with documents that come from wherever, then it's no longer just generation, it's augmented generation and it's retrieval augmented generation because you've retrieved those documents and retrieval is very broadly defined. If you copy and paste something into your context window, that's technically RAG. It's just copy and pasting in production can be done like at scale. So it's done through like automated tools, et cetera. But this is basically. This is what RAG is. RAG is add documents into a context window and retrieve them somehow. Now, you've probably heard that RAG is dead because Claude Coté does not use RAG, or so some people claim. And that's not untrue, but that comes from, you know, one of those cases of there's multiple definitions for terms and the marketing people have pushed their own. And the idea that RAG is dead comes from this confusing definition because Basically in late 2023, early 2024, there was this big rage on Twitter, on LinkedIn, on social media of like chat with your PDF, chat with this. And you had like, I won't say it was necessarily done with like bad intent, but people came up with this concept that they had to sell you something. They really had to sell you something. And the simplest product to sell in the retrieval space was basic single vector cosine similarity. There's a lot of reasons for this, but that's because This is a good enough approach to semantic search, and it's an approach that's very, very easy to actually set up. You just need, basically, you need one vector for each document, you need one vector for your query, and all you need to do is a simple distance search. Easy to scale, costs nothing to host. You can sell it for quite a lot of money to people, like this kind of simple hosting. And it was all the rage, and it became like so big to the point where you had a lot of sometimes technical, sometimes non-technical people, like just pushing and pushing about like, this is RAG. RAG is when you do cosine similarity search on a single vector. But that's actually just like one of many, many, many tools. There's a lot of retrieval tools and we've just come to like over-index on this one specific one for a lot of reasons. But the issue, and that's in the next slide, we'll see that simple semantic search, like just using one vector, actually has a lot of limitations. And I think that's screenshot from Twitter in 2014, the good old days, is something I really like because see, like, you read this, you get it, it makes sense to you, it's funny, you get that it's funny because of the words, like the way they're ordered, etc. Well, your model, your single vector retrieval model, is going to have to represent this in a vector. Your model was not trained on this. Your model does not know that James Bond funny should return this, or James Bond strong. Strong is not even a word, it's not going to be a token. And that's a good example of like the limitations of single vector semantic search because what single vector semantic search is, you will embed your whole document or your chunk if you're using chunking into just one vector. Typically that vector will have like 1024 dimensions, so about 1000. And you're asking your model to, if you're doing long context, to compress maybe like 5000 words into 1000 numbers. Mathematically, that's very difficult. We're not going to get an accurate representation. So this is a form of compression. Retrieval done with single vectors and all retrieval in general is a form of compression. And compression leads to information loss. I'm not teaching you anything, hopefully, like we all know this. But this compression loss will have been— this information loss, sorry, will have been informed during training. And during training, the model is seeing a lot of data. And what it's doing when you're training a retrieval model is you're trying to tell it that like document A should be represented in a way that it will match query A, but document B, which is not relevant to query A, will be represented in a way that will not match query A. And what you're effectively doing in this training is that you're telling the model that it should learn to lose information that is not useful for this comparison. And the thing is that that's trained on a lot of publicly available data, like Bing search data is a big one. But that data does not look like your data. And that's a big reason why you often see, oh yeah, semantic search sucks at code. And that's because, okay, every codebase is different. Every language in the codebase is different. And there's not a lot of good, like, retrieval code training data. Hamel will know, given that he did CodeSearchNet a few years ago, and we still use that. Like, it's not something we have a lot of. So when you try to use semantic search for stuff like CoSoar or Claude, you're like, oh man, this index is bad. And that's because we're just catching up and the general-purpose embedding models are not trained for this. So they kind of struggle to find what's relevant to whatever piece of code you're doing. And so, like it says in the subtitle here, it's actually impressive how well one vector manages to keep so much information despite these heavy constraints. But that's also why you're seeing all those RAG is dead takes, because if you use just one tool and a tool that has limitations that everybody in the retrieval space knows about, you're not going to get the best results. And that's pretty much it on the whole RAG's dead from the semantic search angle. And then you've got the other one, which is context window. And you will have heard a lot of people say stuff like, oh, I don't need RAG because, you know, Gemini, 1 million context window. Mamba, there's not even a limit on the context window. Magic, if it ever releases, 100 million. So that's it. That's, we don't need RAG. We just put the whole world into the context always. And so I kind of waved them away in those slides, but like, I think it's important to mention the costs, et cetera, because, okay, technically there's a 1 million context window, but you know, LLM inference is not free. And even if we managed to solve the computational limitation, which means that attention is quadratic, it's kind of hard to scale to very long context windows. We also end up with a situation where inference has a cost. It's got a fixed compute cost. And if you're using APIs, it's got a fixed, like, $3 cost per million tokens. So you could have like 5 million per million context into like every single query that your company is sending, but there's a very realistic chance that after like a week, your boss will get a report and they'll be like, can you, can you tell me why we're paying $3 per query? And you'll be like, oh, it just, RAG is dead. So I'm just using the whole context every time. And yeah, that's probably not going to fly when you have even a thousand queries a day. So RAG's probably not going to die because of cost issues, but let's assume that there's no cost issues. We can just ignore that. Saying that RAG is dead because we have like 1, 10, 100 million context is kind of weird because that's not actually a lot. Like a big book is already a few million tokens. If you go sizable company knowledge base, if you're looking at like financial data for like the SEC, you will need a lot of those 10 million context windows and you don't want to end up with like billions and billions of tokens in your context window for so many reasons. So when people are telling you that RAG's dead because the context window is so big, we never need to use RAG again, it's kind of like someone in 1999 who's got like their 1 gigabyte hard drive telling you that it's fine, I'm going to sell my hard drive because they're going to release 512 megabytes RAM sticks. I'm just going to put everything in RAM, never store anything again. But did not pan out, and that will not pan out for RAG either, just because the volume of information is so big and you would always want cold storage somewhere. And that brings us to the fact that the reason you'll want cold storage is that it's just a good abstraction to provide the model with external information. And we go into more detail on the next slide on this. The reason, like I said, we do still need hard drives, the reason your computer's got a hard drive is because you can store basically arbitrarily large continuously updated volumes of information. One thing is, this is not necessarily information in the traditional sense, but say you're really into films and you've got like 8K movies or 4K movies. Even if you've got a really good PC, that doesn't fit in RAM, that's stored on the hard drive and you read it whenever you want and then you offload it because that's information you want at a given point in time. Like maybe you really like Stranger Things, you probably don't want Stranger Things to be in your life at all times, at all points. And the issue, the other issue is LLM weights are frozen. Like the model has been trained, the model knows what was true when it was trained. And the only way to add new information to them is quite a complex training process, especially if it's just to learn one new fact, because you need to be careful you're not forgetting anything and you need to be careful that your information is actually being learned properly. And one thing that was quite apparent is if you use GPT-2, like the older LLM, the one that they no longer train, or even BERT, And you ask them who is the president of the United States, they'll say Obama or Trump, like Trump won, which is true again, but just, you know, a coincidence. And a lot of them are like, oh yeah, the president of the United States, Barack Obama. I know that because that's what it was trained on. And even worse, they don't know anything about like your own company project. Like if you want an internal chatbot, you need RAG because the model doesn't know what your company policy is or the model doesn't know about your new codebase that you're writing or say you've heard of fast HTML and you need to put FastHTML into context because the model doesn't know FastHTML, it came out 6 months ago. Now Claude does know about FastHTML and you should use it, but when you want to use a new library, there'll be a need to inject that information somehow because space within weights is precious. And what I would want personally is a model that actually doesn't know anything. I would want it to be able to think about what I give it, but I want all the weights to be spent on intelligence, not just knowledge. And that's where RAG and just like retrieval is going to forever be the natural way to provide this external information because that's the simple pipeline of get documents somewhere, e.g., retrieve them, give them to your model and let it generate with it. And so to sum up this part, RAG's not dead. RAG will not die anytime soon. What's dead is, uh, very simple methods, which honestly, they've not outlasted their usefulness. That would be lying because we'll forever be using them. Semantic search is good, but just, you can't just use the one tool in your toolbox and then complain that the whole toolbox is useless because you don't know how to use the 48 other tools that are there. RAG will remain useful because that's how you provide up-to-date information. And long context windows are not going to kill RAG because that's just a waste of resources. And that's, That's just another form of problem, like, just to dwell on this one a bit. Even if you put in 300 pages of context for everything, that means that when your company policy gets updated on page 295, you have to go update all your prompts, etc. Like, there's so many reasons why long context is never going to kill RAG. And yeah, thank you for coming to this part of the talk. But we're not quite done yet because I've basically told you that there's a big toolkit and I've not told you anything about the big toolkit. I've just told you, yeah, this one thing's bad and why else is that out there? And that's why I've invited some really cool people to come and talk to you. I'm a couple slides down. Something, you know, like something did end up killing hard drives because I said you're not going to throw away your hard drives because your RAM sticks and that's never going to happen. But you don't have a hard drive anymore if you're using like a computer you've bought recently and you're not some kind of data hoarder. Probably half the people in here are using hard drives, but that's, that's not the point. The average consumer doesn't have a hard drive in their computer anymore. They are buying new laptops, they're buying a new computer in the shop, and there's no what we used to call like, you know, HDD, the rotating hard drives. Now there's SSDs in the computer, but if you ask that normal person what an SSD is, That's two answers you're going to get. One of them is, I have no idea, please leave me alone, which is very fair. The other one is, um, like, yeah, an SSD is like, it's a better hard drive, you know, it's faster, it's better. And that's exactly the same with RAG. So RAG did not get killed, but what killed 2023 RAG, like the one we've had on Twitter, single vector semantic search, is just that we've got better tools now. We've got better RAG. And we've always had better tools. It's just taken a long time for them to get usable or to like figure out how to put them into pipelines. And so, you know, like, there's lots of facets of retrieval, and I'm going to go really quickly about them because that's not the main topic here, but RAG is the act of retrieving context. It doesn't matter how it's retrieved. This part's really important. So what that means is grep, like Claude Coste does, is a retrieval tool. You're using a hard query. You're doing fuzzy matching with your documents. Grep's a retrieval tool. wget's a retrieval tool. You're retrieving information from the internet. Agentic search is a smarter approach to retrieval. You're just using the model to actually orchestrate your retrieval pipeline instead of doing it yourself. BM25 and keyword search, which you'll have seen a lot of us rant about, is a retrieval tool. Keyword search is very important. ColBERT and late interaction, more on that in a second, are retrieval tools. Searching the internet is a retrieval tool. Even reasoning, and Orion will be telling you about this, is a retrieval tool. And so, yeah, obviously semantic search with single vector is a retrieval tool, but it's one of many, many along these lists. And what's important is finding what's the best pipeline for what you're trying to do. Which is why Claude Card did not kill RAG. They're using a form of RAG that works really well for their method because if you've got access to your machine, the search domain is really limited. I mean, a single folder. Why would I not use grep? Of course I want to use grep. That's just doing smart retrieval. That's not killing retrieval. And yeah, that's a big list and it doesn't cover everything. There's hundreds more tools, but there's no need to discuss all of them. Only thing you need to remember is retrieval is relevant, but what's even more important is it doesn't look like what people are trying to profit from selling you something want it to look. And the way retrieval looks in your project and the way it will have to look in your project is very relevant to what your project needs. Like it's very needs-based. And retrieval really doesn't help itself. Like one of the reasons, you know, about single vector and not everything else is because there's new approaches coming in every day that are done by Researchers, researchers are not good at marketing things. Everyone's telling you that this new thing is so much better using a benchmark that you've never heard of, and you're like, look, it does better on BenchX, and you're like, okay, should I use this? And then another guy is like, no, use my thing, it's on BenchY, and you never really understand why. So like, it's just a hard field to get into. So don't worry though, because in the next few weeks we're going to have like 4 speakers that we've invited to like actually tell you about very specific aspects, so kind of cut the noise down and focus for half an hour on a very important topic so you can better understand why this works or why this doesn't work. And I'll be shortly introducing them. So the first one is Nandan Thakur. Hope I'm pronouncing that right. And he's probably the one that Hamel will enjoy the most because his work is on evaluations. And if you've taken Hamel Husain's course, you know that Evaluations are difficult. There's a lot of pitfalls, but they're also really, really important. And one of the reasons they're important is because in retrieval, like I said, we've got thousands and thousands of approaches. You've got so many models within every approach. There's no way you can try everything. What you need is like trusted public benchmarks, but they're not exactly your use case, but close enough that you can have a decent idea of what are the like top 10 approaches you should try and how to combine them. Historically, that's been academia's role. The problem in the LLM era, and if you follow some retrieval Twitter accounts, you will see people complain about overfitting BEIR or overfitting MTEB, and that's because all of the common benchmarks are now in the training sets of LLMs, and LLMs are increasingly used as backbones for retrieval models. So stuff looks— newer stuff looks better on benchmarks. There's absolutely no guarantee it will actually look better on your stuff. It just now Data contamination is all over the place. And that's a big problem because who do we trust when we've got a thousand tools and we can't even know which ones have decent to begin with? And that's the topic of Nandan's research because obviously academics don't just look at this and be like, well, I quit, I give up. They've, they've been thinking of better ways to do retrieval evaluations. And Nandan in particular has introduced something called FreshStack, which is basically continuously updated evaluations so that it's never going to be in the training set. You'll always have something fresh to test on. The next speaker is going to be Orion Weller, which is someone that I really like. He also worked on ModernBERT, and his work is really interesting because Orion is much younger than I am and he's very LLM peeled. He did his research in the LLM era, a lot of it. And one thing that Orion cares about is reasoning, because reasoning ultimately is a form of increased test time compute, you know. We agree to spend more money, more compute, more everything in exchange for hopefully better performance. But how does that work when you want to retrieve? Like, if models can think, what does that mean for retrieval? Like, can retrievers think? Can we use reasoning to actually do better retrieval? How do we even scale that? Like, how do you reason at scale if you've got 10 million documents? And from a different angle as well, and I'm not going to say more because we need to tease some things, when you've got an LLM thinking, can you use this thinking to actually improve retrieval even when using older methods? Can reasoning traces give us any information as to what we need to retrieve? Well, that's the kind of stuff that Aryan's been researching, and he will be giving a talk focusing on what does reasoning mean for retrieval. The next speaker is Antoine, another ModernBert colleague, and This one's very dear to my heart because Antoine's one of the people that has worked a lot on ColBERT and ColPyRI. If you follow retrieval Twitter, you will sometimes see people telling you to use ColBERT, telling you to use ColPyRI, telling you to use light interaction, telling you to use multivector model. You probably don't know what those mean if you're not spending that much time doing retrieval because a lot of people don't know what it means and it's something that we all need to get better at is to explain what we're talking about. And the short answer is that there are different approaches to semantic search where instead of keeping one vector for the whole document, you're keeping one vector per token and then heavily, heavily compressing those so that they're still storable somewhere. And that allows you to do much, much more fine-grained retrieval, which loses a lot less information. It's its whole universe. It has like different infrastructure requirements, but Antoine is one of the big researchers in this field and someone I really like, and he'll be telling you all about why you should use ColBERT, how ColBERT actually works, and how to actually use it easily, because a big field, like a big area of research Antoine's been doing, is making ColBERT dead simple to use. And he succeeded with this framework that's called PyLite, and he'll be introducing that. And the final speaker, which is Brian Bischoff and Ayosha Azer, they'll be talking about something that I actually like, because data is multimodal, you know, like we always talk about text retrieval, but in the real world, everything's visual. Documents are visual, tables are visual, graphs are visual. So there's an increasing need for like multimodal retrieval. And a lot of people will tell you, oh, it's fine, just use ColPy like in the slide before and do retrieval, but they never actually tell you, but like, what's a multimodal retrieval pipeline look like? Like, what can I reuse from text? What should I not reuse from text? How do I port all the old methods? And so what Brian and Nayush have done, they've actually built a multimodal search engine and they focused on like all those individual components and they've kind of done that thing of, yeah, semantic search is great. If you just use semantic search, guess what? The fact that it's on images don't mean that it doesn't fail the same way. It's like, there's still a lot of ways that it's going to forget information, compress things. And so they'll be telling you all about that journey to like figure out how to reuse all those ancient retrieval tools, you know, all the stuff from before 2020. And make it work with that newer pipeline to do like high-quality multimodal retrieval. And that's it. That's the end of my talk. Thank you all for coming. Sorry, I spoke a bit fast, but I realized it was much denser than I thought it would be.

No, thank you for that. I just want to tell people that, so Ben announced 4 more speakers. You will be automatically enrolled in the other lessons. Uh, they're 30 minutes each as well. So keep an eye out for that and we'll be sending out the recording, slides, notes, so on and so forth. So just keep an eye out. You can find Ben on Twitter or meet myself on Twitter to sign up for those lessons if you haven't seen them. Thank you, Ben, for coming and we will look forward to the next speakers in the series. Really exciting start.

Thank you.