Beyond the Basics of Retrieval for Augmenting Generation (w/ Ben Clavié)

Ben Clavier is, you know, one of the cracked researchers who work at Answer AI. You've heard from several researchers from Answer AI already in this conference. Ben has a background in information retrieval, amongst other things, and has— he has an open source package called RAGatui, which you should check out. He also comes from a deep background in information retrieval and brings that to RAG. And he also is one of the clearest thinkers on the topic. But yeah, I'll hand it over to you, Ben, to kind of give more color to your background, anything that I missed. And yeah, we can just jump into it.

Okay, let's go. So I think that's pretty much the key aspect of my background. You pretty much read this slide out. So I do, yeah, I do R&D at Tensor AI with like Jeremy. You've seen Jono in this course and there's like a lot of other awesome people. We're a distributed R&D lab, so we do AI research and we try to be as open source as possible because we want people to use what we build. Prior to joining Answer, I did a lot of NLP and kind of stumbled upon information retrieval because it's very, very useful and everybody wants information retrieval. It's more of like clarifying what information retrieval is, which I hope today's talk will help. And yeah, my, so my claim to fame or claim to moderate fame at least is the RAGA2 library, which makes it much easier to use a family of models called ColBERT, which we will very briefly mention today but won't have time to go into detail. But hopefully, if you want to know more about that, do feel free to ping me on Discord. I'm generally either very responsive or you need to ping me again. Pretty much how I work. And I also maintain the ReRankers library, which we will discuss in one of the later slides. And yeah, if you know me or want to follow me or want to hear more about what I do, it's pretty much all on Twitter. I'm not on LinkedIn at all. I'm just Everything goes through Twitter, a lot of memes and shitposts, but some very informative stuff once in a while. So yeah. And so let's get started with what we're going to talk about today. And so it's only half an hour, so we're not going to talk about a lot. I'm going to talk about what I think are the core retrieval basics as they should exist in your pipelines, because RAG is a very nebulous term, and that will be the first slide, and Hamel Husain will be very happy about that slide, I think. But RAG is not a silver bullet. RAG is not a new thing from December 2022. RAG is not even an end-to-end system. We'll cover that, but I think it's very important to like ground it a bit when we talk about RAG because it means a lot of different things to different people. Then we will cover basically what we call the compact MVP, which is what most people do when they are starting out with RAG. It's actually an example from Jeremy. And it's like the simplest possible implementation of RAG as in just using vector search. And then the other topics are basically things that I think you should have in your RAG pipeline as part of your MVP. And I'll show that like there's a lot of scary concepts because they're all big words like biencoder, crossencoder, TF-IDF/BM25, filtering. That sounds like a lot, but then I'm going to try and show it that they're very simple concepts and you can have pretty much the same MVP by adding just 10 lines of code, but using like basically state-of-the-art retrieval components in every bit. And the bonus, which I don't think we'll have time to cover when I try this again, was talking about Colbert because I like talking about Colbert. So I might do it at the end if we have some time, but I might not. And yeah, that's it for the agenda. And then I also think it's important to have the counter-agenda, which is what we won't be talking about today because those are just as important for RAG, but they're not what I would put in the very basics. And here we're very much about the basics. So one of them is How to monitor and improve RAG systems, because RAGs are systems and they're living systems and they're very much things you should monitor and continuously improve on. I think Jason covered that quite well in his talk yesterday or last week. Yeah, last week. So I would invite you to watch that and watch Jason and Dan's upcoming course if it does materialize. Evaluations, they're also extremely important, but we won't talk about them at all today, but I know that Joe will talk about them at length in his talk. Benchmarks and paper references. So I'll make a lot of claims that you will just have to trust me on because I don't want to like have too many references or too many like academic looking tables and this, trying to keep it quite lively and airy. I won't give you a rundown of all the best performing models and why you should use them. I won't talk about training, data augmentation, etc. And I won't talk about all the other cool approaches like Splat, ColBERT in detail because they go beyond the basics. But those are all very important topics. So if you're interested to look up, there's a lot of good resources out there. Do feel free to ask me. And with that, let's get started with the rant, which is my favorite part. So this is a thing that Hamel Husain been doing on Twitter recently as part of his flame posting campaign, I'll say, which is basically, there's so much in AI, so much in especially the LLM world that uses words that are like a lot scarier than they need to be. And RAG's probably that because To me, when I hear retrieval augmented generation or RAG, it sounds like that's an end-to-end system. That's a very definite set of components. That's a thing that works on its own. And it's not. It's literally just doing retrieval to put stuff into your prompt context. Like before your prompt or after your prompt, you want to get some context, so you're doing retrieval. But that means that's not an end-to-end system, despite what Jason will have you believe on his Twitter. He's not created it, but he does make a lot of money from it. And It's basically just the act of stitching together retrieval, so the R part of RAG, and generation, so the G part of RAG, to ground the latter. So like you want your generation to be grounded to some context, so you're doing retrieval on whatever documents you have and pass it to your LLM. But there's no magic going on. It's very much like a pipeline that takes the output of model A and gives it to model B. The generation part is what's handled by the large language models, and good RAGs are actually 3 different components. It's your good retrieval pipeline. It's a good generative model and it's a good way of linking them up. So it can be formatting your prompt or whatever. And it's very important to think about it when you're saying my RAG doesn't work. You need to be more specific. Like my RAG doesn't work is the same as saying like car doesn't work. It's like, yeah, but something specific is broken. You need to figure out what is it, the retrieval part, is the LLM struggling to make use of the context, etc. There's a lot of failure cases there. And with that being said, let's look at what like the compact MVP is. So that is basically what you will see, I think, if you've read any Medium blog post about the advent of RAG in early 2023, that's the pipeline that everyone used. And that's also because the easiest pipeline to bring to production is very simple. You have a query, you have an embedding model, you have documents. The documents get embedded and pooled into a single vector. Then you do cosine similarity search between the vectors for your query and for the documents, and that gets you your results, that gets you your score. And this is a bit of a teaser for an upcoming slide when I say this is called the biencoder approach, but just so you get the term in mind, and I'll define it because that's one of those things that is like a scary term that's actually very, very simple when you break it down. But first, let's look at what this actually means in code, this whole pipeline. So the first thing you want to do is load your model. Then you get your data, you encode it, you store your vectors, then you get your query, you encode it, and then here we use NumPy, you do a cosine similarity search, e.g., a dot product between normalized vectors to get the most similar documents and the documents that are similar to, like, the documents whose embeddings are similar to your query embedding is what you'll consider as your relevant documents. And that's pretty much it. That's modified from something that Jeremy did to showcase how simple RAG actually is in his Hacker's Guide to LLMs. So that's what you want to do to retrieve context in the simplest possible way. And you will have noticed that there's no vector DB in this. This is all NumPy arrays. And this is all NumPy arrays because when you use vector DBs, the huge point of using a vector DB is to allow you to efficiently search through a lot of documents because what a vector DB does generally, not all of them, but most of them, wrap stuff like HNSW, IVF-PQ, which are indexing types. And what that allows you to do is to find and retrieve relevant documents without having to compute cosine similarity against every single document. It like tries to do an approximate search rather than an exact search. This is not something that you need if you are embedding like 500 documents, like your CPU can do that in milliseconds. You don't actually need a vector DB if you're trying to go to the simplest possible stage. If you wanted one, it would go right here on the graph, like right after you embed your documents, you would put them in the vector DB. And the second thing I think to discuss about this tiny graph is why am I calling embeddings biencoders? Because that step that I call biencoder, you will have seen a lot of times, but you will always see it generally called embeddings or model. And biencoder is the term that the IR literature uses to refer to that. And it's simply because you encode things separately, like you do two encoding stages. So it's a bi-encoding. And that's used to create single vector representations where you pre-compute all your document representations. So when you're using bi-encoders, you encode your documents whenever you want, like when you're creating your database, when you're adding documents, those get encoded at a time that's completely separate from inference. And then only at inference will you, like in the second aspect of this column, will you embed your query to compare to your pre-computed document representations. So that's really, really computationally efficient because at inference, you're only ever encoding one thing, which is the query, and everything else has been done before. And so that is part of why it's done that quickly. And I did want to take a slight break because I can see there are questions, but they're not showing up on my screen. So if there are any on like this quick MVP, then—

Yeah, let me look through some of the questions. I'm going to give you a few of them and you can decide whether you want to take them now or later. So we got one, it's a 7,000 query, a 7,000 question and answer dataset. Can I optimize RAG to accurately retrieve and quote exact answers? It will also effectively handle queries that are slightly different from the original data. I think there's actually two parts to that. So one is to quote the exact answer, which is something about the, is not the information retrieval part, but it's rather just like what do you tell the LLM to do? But the information retrieval part is probably—

well, you see it. So I will actually cover how to better deal with out-of-context things in like an upcoming slide. It's not—

yeah, why don't you keep going? None of these questions.

Yeah, I'll just say it right now. Yeah, perfect. Yeah. Okay. So the next one is, if that's very computationally efficient, there is an obvious trade-off here. And that is your documents are entirely unaware of your query and your queries are entirely unaware of your documents. Documents, which means that you're very, very subject to how it was trained. It's basically if your queries look a bit different from your training data, or if there's very, very specific information that will be in certain documents and not others. Sometimes you want to know how the query is phrased. You want to know what the query is looking for when you're encoding your documents so that it can kind of paint that representation and represent it more towards information that you're interested in. And that's done with what we call re-ranking. So re-ranking is another one of those scary stages that we'll see in your pipeline. And the most common way to do re-ranking is using something that we call cross-encoder. And cross-encoder is another one of those scary words like bi-encoder that you feel should be like a very advanced concept, but it's actually very simple. This graph here represents the whole difference between them. The bi-encoder is basically this two-column system that we describe where documents get encoded in their corner, queries get encoded in their own corner, and they only meet very, very late, like you only do cosine similarity between vectors, but the documents never seen the query and vice versa. The cross-encoder is different. The cross-encoder is a model that will take your document and your query together. So you're going to give it both your document or like a series of documents depending on the type of model, but to keep it simple, we do it one by one. So you always give it a query document pair and you put it through this cross-encoder model, which is effectively a classifier with a single label and the probability of the label being positive is what your model considers as how similar the documents are or how relevant it is. This is extremely powerful because it means that the model knows everything about what you're looking for when it's encoding the document and can give you a very accurate score or at least a more accurate score. The problem is that you can see how that wouldn't scale because it's not very computationally realistic to compute this query-document score for every single query-document pair every time you want to retrieve a document. Say you've got Wikipedia embedded. You've got, I don't know, like 10 million paragraphs, you're not going to compute 10 million scores through a model for like using 300 million parameters for every single document. Or you, you would eventually return something and it would be a very, very relevant document, but it will also take 15 minutes, which is probably not what you want in production. So you probably also have heard, or you might also have heard if you're really into retrieval, oh, not help at all if you're not into retrieval or other re-ranking approaches like RankGPT or RankLLM. Using LLMs to rank documents has been a big thing lately. For people really into retrieval, you will know of Monotified, etc. So those are not cross-encoders, but that's not really relevant to us because the core idea is the same, and that's basically what we always do with re-ranking in the pipeline. You use a powerful model that is computationally expensive to score only a subset of your documents, and that's why it's re-ranking and not ranking because This can only work if you give it like, I don't know, 10, 50, not more than that documents. So you always have a first stage retrieval, which here is our vector search. And then the re-ranker does the ranking for you. So it creates another list. There's a lot of ways to try those models out. Some of them have an API base. So it's just an API call to Cohere or Jinna. Some of them you run your machine. If you want to try them out, and this is basically the self-promotion moment, I do maintain at answer.ai a library just called re-rankers. With the QR code here, where it's basically a unified API so you can test any re-ranking method in your pipeline and swap them out freely. And that's what your pipeline looks like now. It's the same with just that one extra step at the end where you re-rank things before getting your results. So we've added re-ranking, but there's something else that's missing here, and that's something actually addresses the first question, at least partially, is that Semantic search via embeddings is powerful, and I'm not saying don't choose vectors. Vectors are cool, like models are cool, deep learning is cool. But it's very, very hard if you think about it, because you're asking your model to take, I don't know, 512 tokens, even more if you're doing long context, and you're like, okay, put all of this into this one vector. We are just using a single vector. You've got like, I don't know, 384, 1024 at most floats. And that must represent all the information in this document. That's naturally lossy. There's no way you're going to keep all of the information here. And what you do when you're training on embedding is that you're teaching the embedding to represent information that is useful in their training. So the model doesn't learn to represent all of the document information because that's pretty much impossible since embeddings are essentially a form of compression. What the model actually learns is to represent the information that is useful to the training queries. So your training data is very, very important here. It's like representing the documents in a way that will help you use the queries in the way they're phrased in your training data to retrieve a given document. So when you use that on your own data, it's likely that you're going to be missing some information, or when you go slightly out of distribution. There's another thing, which is humans love to use keywords, especially if you're like going into the legal domain, the biomedical domain, anything specific. We have a lot of acronyms that might not even be in the training data, but we use a lot of acronyms. We use a lot of very advanced medical words. People love jargon. People love to use technical words because those are very, very useful. And that's why you should, and I know it sounds like I'm talking from the '70s because that's actually a method from the '70s, but you should always have keyword search in your pipeline. You should always also have full text search on top of anything that you do with vectors. And keyword search, which you can call full-text search or like TF-IDF, BM25, it's powered by what we call TF-IDF, which is a very basic NLP concept that essentially stands for term frequency inverse document frequency, and it assigns every single word in the document or like group of words, because sometimes we do them 2 by 2 or 3 by 3 even. It gives them a weight based on how rare they are. So like a word that appears everywhere, like a V or A, has a very, very small weight, and a word that's like highly specific to certain documents has a very high weight. Weight. And the main method to use TF-IDF for retrieval is called BM25, which stands for best matching 25. It was invented in the '70s. It's been updated since then, but it's basically just been iterations of it. And you'll often hear IR researchers say that the reason that the field's not taken off like NLP has or computer vision has is because the baseline is just too good. Like, we're still competing with BM25, and it's been 50 years now. My God, it's been 50 years. Yeah, so BM25 existed for like basically my entire lifetime before my birth and it's still used in production pipeline today. That's how good it is. And the good thing is it's just word counting with a mat— with like a weighting formula. So the compute time is virtually unnoticeable. Like you can add that to your pipeline, you will absolutely never feel it. And I know I said I wouldn't add anything from papers, but I feel like because I'm making a very strong claim that this method from the '70s is strong, I should add a table, and add the table from the BEIR paper, which is the retrieval part of MTEB, which is basically the main embeddings benchmark. And they compared it to a lot of models that were very popular for retrieval, like DPR and ANTs, very strong vector retrievers. And basically, you can see that unless you go into very overtrained embeddings like E5-BGE, the M25 is competitive with virtually all deep learning based approaches. At least at the time of the paper, which was only just 3 years ago. We now have embeddings that are better, but we don't have any embeddings that are better to the point where they're not made better by being used in conjunction with BM25. So knowing that, this is how you want your pipeline to look. You'll notice that there's now a whole new pathway for both the query and the documents where on top of being encoded by the embedder, they're also encoded by TF-IDF. To get full text search, and that will help you retrieve keywords, etc. Humans use keywords in queries all the time. It's something you should do. At the end, you will combine the scores. It's— you can do that in a lot of ways. I won't go into too much details, but what a lot of people do is give a weight of 0.7 to the cosine similarity score and 0.3 to the full text search. But I'm pretty sure we could do a whole talk for an hour on different methods of combining that. Okay, I do have 5 more minutes. So the last one that you want to add to a simple pipeline, the thing that I think really completes your MVP++ is using metadata and using metadata filtering because academic benchmarks don't, because in academic benchmarks, documents exist mostly in a vacuum, like they don't exist in the real world. They're not tied to a specific company, etc. When you're using RAG in production, It's very, very rare that someone comes to you and says, these documents came to me in a dream and code them. Like, they came from somewhere. They've been generated by a department. They've been generated for a reason. They might be old Excel sheets or whatever, but they are like, they have business sense or they have like in-context sense. And the metadata is actually sometimes a lot more informative than the document content, especially in RAG contexts. So if you take the query here, which is "Can you get me the Cruise division financial report for Q4 '22?" There's a lot of ways in which this can go wrong if you're just looking at it from the semantic or even using keywords aspect. So when you see this, the model must capture the financial report. So the model must figure out you want a financial report, but also Cruise division, Q4, and 2022. And embedding models are bad at numbers. So you might get a financial report, but maybe for another division, or maybe for the cruise division of 1998. It's very hard to just hope that your vector will capture all of this. But there's another fatal case which will happen, especially with weaker LLMs, is that if you just have like top, top K, as in top 5 documents, and you retrieve the top 5 documents for your query, even if your model is very good, if you just let it retrieve the top 5 documents no matter what, you will end up with financial reports at least 5 of them, and there's most likely only one for Q4 '22. So at that point, you're just passing all 5 to the model and being like, good luck, use the right one, which might confuse it, especially because tables can be hard, etc. And so I'm not saying that your vector search will fail, but statistically it will. Like, in most cases, it will fail. And if it doesn't fail for this query, it will fail for a similar one. But that's actually very, very easy to mitigate. You just have to, like, think outside of the vector and just use more traditional methods. And you can use entity detection models. And one that's very good for this is Gleaner, which is a very recent model that does basically zero-shot entity detection. So you give it arbitrary entity types, so like document type, time period, and the department. And this is like a live thing of Gleaner. You can run the demo on the bottom, but here we just extract financial report, time period, department. And when you generate your database for RAG, all you need to do is basically specify the time period. So when you get an Excel sheet, you will just pass the name for it or pass the date in it. And give metadata 2024 Q2, Q4, sorry, 2022 Q2, Q4, get mixed up there. And then you just need to ensure that this is stored alongside your document. And at query time, you can always pre-filter your document set to only query things that make sense. So you will only query documents for this relevant time period. So you ensure that even if you give your model the wrong thing, it will at least be the right timeframe. So it can maybe try and make sense of it. And with this final component, this is what your pipeline looks like. You can see the new component here, which is metadata filtering, which doesn't apply to queries. Queries go right through it, but documents get filtered by that, and we won't perform search on documents that will not meet the metadata that we want. And okay, I do agree that this looks a lot scarier than the friendly one at the start, which just had your embedder and then cosine similarity search and the results. But it is actually not very scary. This is your full pipeline. This implements everything we've just talked about. It's about 25 lines of code if you remove the comments. It does look a bit more unfriendly because there's a lot more like moving parts. I think there's a lot more steps, but if you want, we can just break it down a bit further. And so we use LensDB for this. And this is not necessarily an endorsement of LensDB as a vector DB, although I do like LensDB because it makes all of these components, which are very important, very, very easy to use. But I try not to take sides in like the vector DB wars because I've used Weaviate, I've used Chroma, I've used LensDB, I've used Spancode. They all have their place. But I think LensDB, if you're trying to build an MVP, is the one you should always use for MVPs right now because it has those components built in. And here you can see just how easy it actually is. So we still load the bioencoder just in a slightly different way, same as earlier. We define our document metadata. Here is just a string category, but it could be a timestamp. It could be just about anything. Then we encode all of our documents just like we did previously. And that's here we've created, so it's not an index, this is still a hard search, this is not an approximate search. Then we create a full-text search index, which is generating those TF-IDF. So what I mentioned before, we give a weight to every single term in the documents. Then we load the re-ranker. Here we're using the Cohere re-ranker because it's simple to use an API. And at the very end, you've just got your query and your search where we restrict it to the category equals films. So we will only ever search into the document that's a about a film, nor about an author, nor about a director. We get the top 10 results and we just have a quick ranking step. And that's pretty much it. We've taken the pipeline at the start, which only had the Bioencoder component, to a pipeline that now has the Bioencoder component, metadata filtering, full-text search, and a re-ranker at the end. So we've added, like, basically the 4 most important components of retrieval into a single pipeline, and it really doesn't take much more space in your code. And yeah, that is pretty much the end of this talk. So there's a lot more to cover in RAG. This is definitely not the full core of RAG, but this is the most important thing. Like, this is what you need to know about how to make a good pipeline very quickly. All the other improvements are very, very valuable, but they have a decreasing cost-effort ratio. This takes virtually no effort to put in place. Definitely worth learning about sparse methods, multivector methods, because they are very adapted to a lot of situations. Colbert, for instance, is very strong out of domain. Sparse is very strong in domain. You should watch Jason's talk about RAG systems and Joe's upcoming talk about retrieval evaluations because those are a clear trifecta of the most important things. And yeah, any questions now?

Hamel Husain and I were just messaging saying we love this talk and like it's just everything is presented so clearly. So we've also got quite a few questions. I'm happy to repeat.

My favorite talk so far. So, you know, not big favorites, but yeah.

Thank you.

No, go ahead. Okay.

Questions. You were going to pick, did you have one that you were looking at already, Dan? I can try it.

Yeah. We've got one that I quite like. Can the way that you fine-tune your bi-encoder model affect how you should approach fine-tuning for your cross-encoder and vice versa?

Yes, I don't think I can give a really comprehensive answer because it will really depend on your domain, but you generally want them to be complementary. So if you're in a situation where you've got the compute and the data to fine-tune both, you always want your bi-encoder to be a bit more loose. You want it to retrieve potential candidates and then you want to trust your re-ranker, like your cross-encoder, to actually do the filtering. So if you're going to use both and have full control over both, you might want to fine-tune it in a way that will basically make sure that your top-K candidates can be a bit more representative and trust the relanker.

Let me ask, this wasn't an audience question, but a related question. You showed us where the, when you choose questions to feed into the re-ranker that's sort of a weighted average of what you get from the, um, TF-IDF or BM25 with what you get from the just simple vector search. What do you think of as the advantage or disadvantage of that over saying we're going to take the top X from one cat— from one of the rankers and the top X from the others? And that way, if you think one of these is, for some questions especially bad, you have a way of short-circuiting its influence on what gets sent to the re-ranker.

Yeah, I think that also makes complete sense. And that's another, that's a cop-out answer I'll use a lot, but that also depends a lot on your data. Like a lot of the time you want to look at what's your actual context and how it's actually being used. Because in some situations that actually works better, like especially if you work with biomedical data, because there's so much like specific documents, it's quite often the embedding won't be that amazing on some questions. So you just want to take the top 5 from both and get the re-ranker to do it correctly. The re-ranker is query aware. So it's a perfectly valid approach to combine them that way. Yeah.

You want to pick a question, Hamel Husain?

Yeah, I've been looking through them. You guys been— okay, Jeremy's asking, can we link, get a link to the code example? Your slides in Maven. We can also, can I share your slides in Discord as well, Ben?

Yes, please. Yeah.

I'll go ahead and share slides in Discord.

Yeah. And I'll share like the GitHub gist for the code examples after the talk.

And I'll embed the, a link to the slides in Maven for people who watch this talk some point deeper into the future and who might lose track of it in Discord. There's a question somewhere in here. I'll find it in a moment, but we got this question for Jason, the speaker, and then the speaker just before you, Paige Bailey said RAG, you know, in the world of million token context lengths is not going to be as important. What's your take on the relative importance of RAG in the future?

So I'm still very hopeful about RAG in the future. And I think I say it as some, some sort of like, so your LLM to me is like your CPU and your context window will be your RAM. And so like, even if you've got 32 gigs of RAM, nobody's ever said, yeah, throw away your hard drive. You don't need that. Like in a lot of contexts, you'll still want to have like some sort of storage where you can retrieve the relevant documents. Having to use a long context window is never going to be a silver bullet, just like RAG is never a silver bullet. But I'm actually really happy because it just means I can retrieve much longer documents and get like more efficient RAG systems. Because to me, it's a bit of a trade-off where if you've got a longer context, it just means you've got a lot more freedom with how quick your retrieval system can be. Because if it If you need to use top 10 or top 15, that's fine. You can fit them in. Whereas when you can only fit the top 3 documents, you need your retrieval system to be really good, which might mean really slow.

We had a question from Wade Gilliam. What are your thoughts on different chunking strategies?

I probably don't think about chunking as much as I should. I am very hopeful for future avenues using LLMs to pre-chunk. I don't think those work very well right now, at least in my test, I've never been impressed. But also I do tend to use ColBERT more often than Bioencoders, and ColBERT is a lot more resistant to chunking. So it's something that I don't care about as much. But generally I would try to, so my go-to is always to chunk based on like around 300 tokens per chunk and try to do it in a way where you never cut off a sentence in the middle and always keep like the last 50 tokens and the next 50 tokens of the previous and next chunk. Because information overlap is very useful to give content. Like, please don't be afraid to duplicate information in your chunks.

I have a question about the BiEncoder. Do you ever try to fine-tune that using some kind of like label data to get that to be really good, or do you usually kind of use that off the shelf and then use a re-ranker? And, you know, how do you usually go about it, or how do you make the trade-off?

So again, context dependent, but if you have data, you should always fine-tune all your encoders, be it the bi-encoder, the cross-encoder. I think ColBERT, because it's single vector, you can get away with not fine-tuning for a bit longer, because multivector, sorry, you can get away with not fine-tuning for a bit longer. But if you have data, it's all about like basically the resources you have. So in this talk, we're doing an MVP. This is something you can put together in an afternoon. If your company says you have $500, Spend $480 of that on OpenAI to generate synthetic questions and fine-tune your encoders. That will always get you better results. Like, always fine-tune if you can. And so, yeah, so a couple questions about fitting ColBERT in, and I'm using present our executive decision to answer those. So ColBERT in this pipeline, some people use it as a re-ranker, but then that's not optimal. That's very much when you don't want to have to change your existing pipeline. If you were to design a pipeline from scratch and wanted to use ColBERT, you would have it instead of the Bioencoder, and it will perform basically the same role as the Bioencoder, which is first-stage retrieval. And if you wanted to use ColBERT and especially if you don't have the budget to fine-tune and need a re-ranking step, sometimes it can actually be better to use ColBERT as your re-ranker still, because the multi-vector approach can be better at capturing keywords, etc., but that's very context dependent. So ideally you would have it as your by-encoder.

For a lot of people here probably aren't familiar with ColBERT, can you give the Quick summary of it.

Yeah, sorry, I got carried away because of the rest of the questions. So ColBERT is an approach which is effectively a bi-encoder, but instead of cramming everything into a single document, into a single vector, you represent each document as a bag of embeddings. So like, if you've got 100 tokens, instead of having one big 124 vector, you will have a lot of small 128 vectors, one for each token, and then you will score that at the end you will do the same for the query. So if your query is 32 tokens, you will have 32 query tokens. And for each query token, you will compare it to every token in the document and keep the highest score. And then you will sum up those highest scores and that will be the score for that given document. That's called max similarity. And the reason that's so powerful is not because it does very well on data it's been trained on. Actually, like, you can beat it with a normal encoder, but it does very well at extrapolating to out-of-domain because You just give the model so much more room to represent each token in its context. So it's much easier if like you're in an unfamiliar setting, you've not compressed as much information. And I do have self-promotion. I do have a pretty cool ColBERT thing coming out later this week to compress the ColBERT space by reducing the tokens it actually needs to save by about 50 to 60% without losing any performance. So that's a bit of a teaser, but look forward to the blog post if you're interested.

And to find the blog post, you suggest people follow you on Twitter or on—

yeah, definitely follow me on Twitter. It's pretty much the only place where you can reliably reach me.

Someone's asking, what are some good tools to fine-tune embeddings for retrieval? Would you recommend Ragatouille or anything else? Like, what's your—

I'd recommend Sentence Transformers, especially with the like 3.0 release recently. It's now much, much friendlier to use. And it's basically, there's no need to reinvent the wheel. They've got all the basics implemented very well there. So sentence transformers.

Question from Divya. Can you give any pointers on how one fine-tunes their embedding model?

Sorry, can you repeat that? I could have slightly—

Yeah. The question is, can you give any pointers or describe the flow for when you fine-tune your embedding model?

Okay, so that's probably a bit more involved than this talk, but essentially when you fine-tune your embedding model, what you'll want is queries. So you need to have queries and you need your documents and you're going to tell the model for this given query, these documents are relevant. And for this given query, these documents are not relevant because sometimes you use a triplet loss and a triplet loss is what you will do when you have one positive document and one negative document. And you'll kind of be teaching the model, this is useful, this is not useful. And I'm not gonna go down too much because this rabbit hole can take you quite far, but sometimes when you have triplets, you also want to use what we call hard negatives, which is you want to actually use retrieval to generate your negative examples because you want them to be quite close to what the positive example is, but not quite the right thing, because that's why you teach the model more, but what's actually useful to answer your query. So the workflow is probably As always, look at your data, figure out what kind of queries your user will actually be doing. If you don't have user queries, go into production, write some, write some queries yourself, then give that to an LLM, generate more queries, and you can have a pretty solid retrieval pipeline like that.

Someone's asking in the Discord, and I get this question all the time, is please share your thoughts on Graph RAG.

I have never actually done GraphRAG. I see this mentioned all the time, but it's not something that has come up for me at all. So I don't have strong thoughts there. I think it's cool, but that's pretty much the full extent of my knowledge.

Someone's asking, okay, when you have long context windows, does that allow you to do something different with RAG, like retrieve longer documents? Or do any other like different kinds of strategies than you were able to before, or does it change anything how you go about this?

Yeah, I think it's a bit what I mentioned before. To me, it changes two main things. One is I can use longer documents, which means I can use longer models, or I can like stitch chunks together, because sometimes, sometimes if your retrieval model isn't very good at retrieving long documents, which is often the case, you might just want, if I get a chunk from this document, give the model the full document. Like if I just get a chunk from it, pass the full context and you just hope the model is able to read it. And if you've got a good long context model, it can. So it changes how you decide to feed the information into the model. And then the other aspect is, like I said, it changes the retrieval overhead because if you need to be very good, like I was saying, if you need only the top 3 documents to be relevant, you're going to spend a lot of time and money on your retrieval pipeline. If you're like, oh, as long as my recall at 10 or my recall at 15 is good, that's fine. You can afford to have like much lighter models. Spend a lot less time and resources on retrieval. Like, there's a lot of diminishing returns in retrieval. We're getting a good recall at 10. So, like, recall at 10 is how likely you are to retrieve the relevant document in the first 10 results. It's generally very easy. Recall at 200 is very, very easy. And then recall at 5 is getting harder, and recall at 3 and recall at 1 are, like, the really tough ones because a lot of the training data is noisy. So it can even be hard to know what a good recall at 1 is. So longer context makes that irrelevant, and that's why it's great for RAG.

Someone's asking, and I don't even know what this means, what's your view on Hyde versus React versus StepBack?

I've only used React out of those, and so those are like agentic systems of function calling. So it's like to give your LLM the ability to call tools. At least React is. I don't have strong thoughts on those in the context of retrieval, so I can't really answer the question. Yeah, I think I would occasionally use React from the model to be able to trigger a search itself, but I think that's still an open area of research. And I think Griffin from AnswerAI is also in the chat and he's very interested in that. It's basically how do you get a model to tell you that it doesn't know? Because sometimes you don't need retrieval, the model already knows. Sometimes you do need retrieval, but that's still a very open question. Like, how do you decide when to search? So no strong thoughts there yet.

There may— you may or may not have a good answer for this one. Is there an end-to-end project, open source project, that someone could look at as a way to see or evaluate the difference in result quality when they do result from just buying code or MVP and compare that to the final compact MVP++ that you showed?

No, actually, that's a very good point. I don't think there is one that systematically goes through every step. And that's probably something that I would like to build at some point or find one because just like most things in retrieval, everything is kind of conventional wisdom. Like you've seen it bits and pieces in a lot of projects and you just know that that's how it is. But unless you dig deep into the papers or like do it yourself, it's quite rare to find very good resources showing that.

Related question, do you have a tutorial that you typically point people to on fine-tuning their encoder?

That would be the Sentence Transformers documentation, but it's not the friendliest tutorial, so that's a half answer. That's what I would point you to, but it's still a bit like hard to gain to, sadly.

Wade is asking if you have GoTo embedding models.

I think my go-to these days when I'm demoing something is the Cohere one because it's nice to be able to work with an API. It works really well. It's cheap. But other than that, no, I would just call bare if I'm using something in my own pipeline. I would just like multi-vectors, but it really depends on the use case because you will often find that like some things work well for you and some things don't. I do have strong opinions on not using— so if you go to the MTB leaderboard, which is the embedding leaderboard right now, you'll see a lot of LLMs as encoders. And I would advise against that because the latency isn't worth it. You don't need 7 billion parameters to encode stuff. And at least some of the early ones actually generalize worse. Like Nils Schremer from Cohere had a really interesting table where the E5 Mistral was worth an E5 large despite being 7 times as big. So probably just stick to like the small ones between like 100 and at most a billion parameters, but that would be my only advice. Other than that, try all the good ones like GTE, BG, E5.

Chris Levy is asking this question about Elasticsearch, which I also get quite a lot. So he asks, anyone here have experience building RAG application with just keyword BM25 as a retriever? At work makes use of Elasticsearch. And he says it's all over the tech stack, like people are already using Elasticsearch. Is there, basically he's asking, is there a way to keep using Elasticsearch with RAG that you know about or that you have encountered? Or do you mainly use like vector databases like LanceDB and things like that? Have you tried seeing people using Elasticsearch and trying to bootstrap off of that?

Yeah, I've used Elasticsearch a bit and it's perfectly possible. You do lose obviously the semantic search aspect, Although I think now Elasticsearch has a vector DB offering, so you could add vectors to it. You could always plug in, you could always just do BM25 and then plug in a re-ranker at the end. That's often, if you read papers on like cross-encoders, generally the way they evaluate them is actually doing just that, like do BM25 to retrieve 50 to 100 documents and then rank them using the re-ranker, which if you can afford to just set up your re-ranking pipeline or call the Cohere API is a really good way to go about it because you don't need to embed your whole documents to sample. How good it would be with deep learning. Because there are domains where you do not need deep learning, BM25 is still good enough in some bits. And you know, like, I think it's become very apparent, like, BM25 has never told anyone they should eat 3 rocks a day, whereas embeddings have, so.

Dimitri is asking, is it worthwhile to weigh the BM25 similarity score during the ring re-ranking step as well?

Probably not. You generally just want to use BM25 to retrieve candidates, but you don't need to give those scores to the embedder, to your cross-encoder.

There's a question. I'm going to change it slightly. So someone asks about retrieving from many documents rather than finding the best one. And maybe the tweak there is if you have a theory that information within any single document is so correlated that you actually want to try and get some diversity, are you familiar with or have you used approaches where you explicitly try and in some loss function somewhere encourage that diversity and encourage pulling from many documents rather than from one?

I have not done that myself. I know that there's different loss methods to optimize for diversity versus clear accuracy, but I don't think I would be able to give you a clear answer without sounding really confident about something I don't know much about.

Have you used hierarchical RAG? Any thoughts on it?

I have not, and I don't think it's very needed for like the current pipelines. I think there's a lot of other steps you can improve.

Since I think we have several Answer AI people here, I don't know if this is a question or a request. I'm eager to learn if Answer AI will come up with any books on LLM applications. In the future?

I don't think so, but never say never. Jeremy, if you want to chime in. Yeah, because I can't make any promises because my boss is watching, so You see anything else too?

Ben, did you say that you can't see the questions?

Yeah, they're all blank for me. I saw one earlier, but they really show up sporadically.

Not sure what's happened with our— and I think people also cannot upvote these. So a couple of quirks today. You see any others here, Hamel Husain, that you think we should pull in?

No, not necessarily. I think like probably going to the Discord is pretty good now. Yep. Tons of activity there as well. I mean, there's infinite number of questions, so we can. I can't keep going. You like? Okay, Lorian is asking, what's the best strategy when chunks, when the documents, when chunks of documents don't fit into the context window? Do you do RAG in a MapReduce style, summarize aggressively? What are the techniques that you've seen work most effectively?

So that's, I think, a very broad question because it's like, Why do they not fit? Is it because every document's really long? Is it because you need a lot of different documents, etc., etc.? So, and how also another important aspect is what's the latency tolerance? Because quite a lot of the time you can make RAG infinitely better, but users won't stay waiting like 20 seconds for an answer. So you need to figure out like, how much time do I have? One way that you can often see What I've done in production actually is retrieve the full documents, but have another database that maps every document to its summary. So you will have like done your LLM summarization at the previous step. You will retrieve the relevant chunks and then you will pass the relevant summaries to the context window. But that kind of depends on like your actual setting. I have another call at 10, which is in 5 minutes for me. So if you've got like another final question.

And really enjoyed this presentation.

Thank you.

Yeah, this is really great. Everything is super clear and well presented. So thanks so much.

Thank you. Thank you. Cheers. Bye.

Thanks everyone.