The Best RAG Technique Yet? Anthropic’s Contextual Retrieval Explained!

Anthropic just released a new retrieval mechanism called contextual retrieval. Based on the results, it's the best performing technique to date. When combined with re ranking, it gives you state of the art performance. I'll call it more of a chunking strategy than a new rack technique, but the results are impressive. To understand this, let's first look at how RAG works. You have your documents, you chunk them into sub documents that for each of the chunk you compute embeddings and those embeddings are stored in a vector store. At runtime or inference time the users ask a question, we compute embedding for those questions and then retrieve the most relevant chunks based on the embedding similarity. Those chunks along with the query are fed into an LLM and you get a response. This technique is great for semantic similarity, but there are still a lot of failure points. That's why a lot of people combine this semantic search with traditional keyword based search mechanisms. One of such techniques is BM25 which plays a critical role if the user query and the database involves keywords. Here is an example from Anthropic. Suppose a user queries error code TS999 in technical support database. An embedding model semantic search might find content about error codes in general, but could miss the exact ts999. However, the keyword based search mechanisms such as bm25 looks for specific text string to identify the relevant documentation and it has a higher probability of detecting this specific error code. The biggest problem with the current string standard RAC systems is that you lose a lot of contextual information when the chunks are written based on the relevance to the user query. It doesn't really have any information regarding the rest of the documents. Here is an example that Anthropic provided. Imagine you had a collection of financial information, say your filings, and those are embedded in your knowledge base. The user query is what was the revenue growth for ME Corporation in Q2 2023? Now the relevant chunk to this might be the company's revenue grew by 3% over the previous quarter. However, this chunk on its own does not specify which company it's referring to or the relevant time period. So it's going to be very difficult for the embedding as well as the keyword research mechanism to retrieve this specific chunk, especially if there are multiple companies mentioned in the same document. That is why Anthropic is recommending to use contextual retrieval. So when you are creating chunks you also include contextual information in each chunk. There are some Issues associated with this approach, which we're going to discuss later in the video. But instead of this chunk, your new chunk is going to look something like this. This chunk is from a filing on Acme's corporation performance in Q2 2023. The previous quarter revenue was this and now this is the actual chunk that your chunking strategy creates. As you can see, they recommend to add a lot of contextual information both to the embedding model as well as to your BM25 index. So how do you implement this? Well, they suggest using LLM to do this automatically. They have provided a prompt for Haiku which is one of the smallest and the most cost effective model. This is a general prompt that you want to customize for your specific application. So in the first part of the prompt you provide the whole document. Let's say you are working with a PDF file and you're chunking that PDF file. You will provide that PDF file here. Then the rest of the prompt is, here is the chunk we want to situate within the whole document. And then you provide the specific chunk that you are working with. After that, you ask Haiku to give you a short context to situate this specific chunk within the overall document for the purpose of improving search retrieval. This specific chunk. So Haiku is supposed to add contextual information to this specific chunk. And as a result you will usually add 50 to 100 tokens to each of your chunk. Now, in practice, this is how it's going to look like. You will take a single PDF file or a single document from your corpus, then convert that into chunks. After that you take one chunk at a time, run that chunk through the prompt that I just showed you along with the original document. This will generate contextual information for each chunk. You combine a given chunk with the relevant contextual information and then you pass it through embedding model. And those embeddings are going to be stored in a standard vector database. On the other hand, we also update the BM25 indexing. In this case we're using TF IDF, which stands for term frequency, inverse document frequency. This is basically the keyword based search mechanism. As a result, in each of the chunk you are adding 50 to 100 tokens. I think you can already see some potential issues with this approach. One of them is that it's going to add a lot of overhead. Not only the number of tokens that you are adding to each chunk, but each chunk goes through an LLM. So that adds up a lot of different tokens. But you can use the prompt caching feature that was introduced by Anthropic a few months ago. Gemini also has a very similar feature. With the help of this, you will be able to reduce the cost. So here are quick calculations. Assuming 800 token chunks, 8,000 token documents, 50 token context instructions and 100 tokens of context per chunk one time cost to generate contextualized chunks is about 1.$02 per million document tokens. So it's a relatively small price. But keep in mind that this can add up if you have millions of documents. The rest of the retrieval pipeline is the same as the standard retrieval pipeline and that's why I think it's more of a new chunking strategy than a completely new retrieval mechanism. So after doing all this, what type of performance improvement you can expect? The Anthropic team performed a scientific study where they ran retrieval benchmarks on a number of different data sets or domain specific benchmarks. The summarizer results are here. Contextual Embedding reduced the top 20 chunk retrieval failure rate by 35% so it went from 5.7 to 3.7, which is a huge improvement. If you combine the contextual embeddings approach which they recommend, plus the contextual BM25 which is the keyword based search mechanism on the improved chunks, you can further reduce the top 20 chunk retrieval failure by 49%. So again you went from 5.7 to 2.9. There are other approaches that you can combine with this. I cover a lot of them in my Advanced Video rag beyond basic scores. In this specific experiment they used a dense embedding model Gemini text 004. According to their experimentations, it's the best performing model on the models they have tested. Based on the results, it's definitely worth checking out now. They also added a re ranker at top of their Rack pipeline and and they saw further improvements. My recommendation is to have a keyword based search mechanism along with some sort of query rewriter plus a re ranker in any RAG implementation. Based on my experience, adding these three different components can give a substantial boost in retrieval accuracy. With the addition of re ranker to the same pipeline, the average retrieval error rate went from 5.7 to 1.9%. Some other things that they recommend to keep in mind when you're building RAC systems or chunking strategy, it's very application dependent so you want to look at your chunk size, chunk boundary, chunking overlap. I have a couple of different videos on this specific topic. I'll put link to those in the video description Another thing to consider is the embedding model. So most of the people use densing embedding models such as OpenAI embedding models or a number of different open source embedding models. In their experiments they found that the Gemini and Voyage embeddings seems to be pretty effective for their particular task. I also highly recommend to look at something like Colbert based multi vector representations. In theory those should be more effective compared to dense embedding models. I have quite a few videos discussing those, so I'll put a link to a couple of those videos in the video description. Another thing is the custom contextualizer prompt. We looked at a very general prompt they included based on Haiku. Depending on the documents you are working with, you will need to customize your contextualizer prompt. And the last thing is the number of chunks that you want to return. They experimented with 5, 10 and 20 chunks and found 20 to be the most performant. I would suspect if they increase the number of chunks beyond 20 they might see some improvement as well. In general, you want to first return a large number of chunks during the retrieval process. Then you want to add the re ranker step in order to reduce the number of most relevant chunks that are going to be fed to the LLM to generate a response. Another thing to keep in mind is how they are measuring their accuracy or error rate. So they're retrieving 20 different chunks and then looking at whether the relevant chunk to a specific query is present in those 20 chunks or not. So they're not really ranking it whether it's the top chunk, but they're basically saying if it's present in the top 20 chunks that are returned, we're going to consider this as a success. Now this is a standard approach, but I wanted to highlight because people can confuse this by assuming that the return chunk is probably the topmost chunk. I'm happy that they tried to address the long context versus Rag question. So they say sometimes the simplest solution is the best. If your knowledge base is smaller than 200,000 tokens, which is about 500 pages of material, you can just include the entire knowledge base in the prompt that you give the model with no need for rag or similar methods. But keep in mind this means that with every query you're going to be sending 500 pages of material to the LLM. The cost will add up pretty soon, and that's why they recommend using their prompt caching feature for claude. And if you use prompt caching, the cost is reduced up to 90% and the latency is going to be reduced by greater than 2% or two times. I have a video on how this prompt caching works, which I think is very important if you are building anything in production. But if your number of tokens are more than 200,000 tokens, which is beyond the context window of the current generation of CLAUDE models, then they recommend to use rag. Now let's look at a code example of the contextual retrievals that CLAUDE has provided. Now here is a notebook in their own repo. I'll put a link to this specific repo. They are comparing basic rank with contextual embedding plus contextual PM25 and adding re ranking on top of it. The repo contains the data that is used. So you can basically replicate their results. So here they are installing all the different packages that we'll need. For this one they're using the VOYAGE embedding model and for RE ranking they are using the Cohere API. Okay, so here's how they're creating the vector db. So in the first step they compute embeddings for the original chunks and then they do basic rag. So they have a code actually runs evaluation on this basic RAG system now based on the results for top five. And that means if you consider the most relevant chunk to be retrieved in the top five chunks, the accuracy is 80%. You, if you extend it to top 10 then 87%. And if you extend it to top 20 then the accuracy of the standard Rack system is about 90%. Now if you use the contextualized embeddings, so they have provided this code snippet which basically takes each of the chunk plus the document from which that chunk is coming. And then run this through the the prompt that I showed you in the beginning of the video. This way they will create new chunks. Again they're using the Haiku model and you can see that the number of tokens are going to be a lot more than what are contained in the original chunks. Now they run the same embedding model for these contextualized chunks. And for top five it sees an improvement of about 6%. For top 10 I think it's again about, for top 20 the improvement is about 3%. Now they are creating a contextualized BM25 index. This is basically adding keyword based search mechanism. And this does I think add some improvements. It's probably about 1% improvement. But you, if you add RE ranking step at the top of it you see about 2% improvement, which is pretty huge going from 92% to 94%. Anyways, this was a quick video on their latest approach, contextual retrieval. It's actually great to see that companies like Anthropic are taking RAG seriously. This shows that RAG is still relevant even in the era of long context LLMs and based on my experience, is one of the only widely used application of the LLMs at the moment. So if you're interested in learning more about RAG systems or agents, make sure to subscribe to the channel. I create a lot of content on RAG and LLM agents. Let me know if you want me to create more detailed tutorials on contextual RAG and how this can be incorporated into in your own systems. I hope you found this video useful. Thanks for watching, as always. See you in the next one.