Advanced RAG with ColBERT in LangChain and LlamaIndex

Welcome to another video on how to do better rag. In a previous video I showed you how to use late interactions as a possible replacement to dense embeddings. In this video we will build an end to end rag pipeline in which we're going to replace the dense embeddings with late interactions using Cold bert. Just like my previous video, we're going to be using the ragatweep package. But this time I'll show you how to use this with LangChain as well as Llama index. So let's get started. Okay, so I put together this Google Colab notebook. Let's look at how we're going to do that using LangChain. So we need to install all the required packages that include the regat to e package LangChain. We're going to be using OpenAI LLM, so we need the LangChain OpenAI package as well. And we're going to be using PyPDF for reading a PDF file. So first and foremost, just like the previous video, we will import the RAG pre trained model from the ragatoi package. If you haven't seen that video, I'll highly recommend to watch that because that will give you a lot more context. Next we are downloading the Codebot v2 model that is going to be used for computing the late interactions. Next, we need a knowledge base. So in this case I'm using the Orca paper and what I have done here is I have uploaded that here to my Google Colab notebook. The way you upload files is just right click, click on upload and then select the file so you can upload it directly to your Google Colab. Now we use the PyPDFloader to load the PDF file. So this will actually give me all the pages. It has a total of 55 pages. But in order to use this with the ragatube package, we need to extract text from each page and put them together. So here I'm basically extracting the text and creating another variable which is concatenating all the text from the file. And we will see the whole file as a string here. So I'm just printing that in here. Okay. Here we're looking at the file type to make sure that we have a string rather than the document objects that LangChain usually creates. Okay, Next we're going to create an index. In order to do that we need to provide the list of documents. So here we're going to be providing that single string that we created. Then we need to provide the name of the index, I'm calling this ORCA paper, you want to also provide the max document length. So think of this as a chunk size. There are definitely strategies to get better chunking, but the beauty of Colbert is that instead of using or instead of computing a single dense embedding for the whole chunk, it's going to compute embeddings for each and every individual token. So that gives us better retrieval accuracy and we want it to split documents based on this chunk size. I'm going to create another video talking about how to do the chunking. Right? But once we set this up, this basically is going to create a plate index. You can use FAIS as well. You will just need to provide FAIS as a parameter in here. Okay. So it created the embeddings for all those chunks and the length is around 31,000, right? So once you create the embeddings, next we need to look at how do you do retrieval? Right? So this part is very similar to what we have seen in the previous video. So we're going to use this RAG object, then call the search function on it, pass on our query. So I'm asking what is instruction tuning, then how many chunks I want the model to retrieve. Now here are the retrieval results. So you get the chunk or text of chunk that is going to be the content. With that you also get a score of relevance, like how close this chunk is to the query that you asked. You also get the rank and then document ID and passage IDs is just to keep track of where this specific chunk is in the documents. Right? So since we asked for three different chunks, here we get three different results or three different passages. Okay. Now in typical rag, you will pass those on as relevant chunks plus the query to your LLM. So you can do exactly the same thing. You can use this as a retriever in LangChain directly. So there is a function called as LangChain retriever that will basically create a LangChain retriever on top of the regatui package, right? And then just like you will use the retriever in LangChain, you can call the invoke function, pass on your query and you will get your resultant documents. So again, since we are asking it to retrieve three different chunks, it will just give us these three different chunks. But now in order to generate a response based on these retrieved chunks, we will need to pass this on to an LLM. And the way you do it is you create a chain, right? So so far, if you think about, we haven't actually used Any embedding model. So it's used, it's an internal embedding model, not like an OpenAI or any other open source embedding model. But for the LLM we will need to use either an open source LLM or a proprietary LLM. In this case, to keep things very simple, I'm going to be using the OpenAI LLM. So here I set up my API key within Google Colab notebook. So I activated that and basically I'm retrieving that and passing it on to this environment variable called OpenAI apikey. Now to create a chain, I will create a prompt template. In this case, the prompt template is answer the following question based on the previous context. And here I'm using these tags to basically tell it what the context is and then it's supposed to give us an answer based on the question that we provide. Right? The LLM is going to be GPT 3.5. So we create a stuff document. So basically whatever context the Colbert model retrieves that is going to be stuffed here as a context. Right? And that's why we are using this Create stuff documents chain. And then we pass this on plus the retriever that we created. So that is based on the regat to E package to this create retrieval chain. So this is going to be our retrieval chain. And now you can ask questions on top of the retrieval chain. So the thing that you get is the input variable that is basically your question. You it will also gives you the whole context and it's going to have another key for answer. So right now I'm just putting everything in the response variable and then just using this answer key to get the answer right. So our question was what is instruction tuning? And the response is Instruction tuning is a technique that allows pre trained language models to learn from input natural language description of task and response pairs. Right. So the response looks pretty good because the context that the Colbert model was able to retrieve is actually pretty close to what we were looking for. Okay, so that was the implementation in LangChain. Now let's look at how we do exactly the same thing within Llama Index. So again we need to install some required packages that include Llama Index. We are going to be downloading the regatui retrieval as a pack from lagamar Hub. This is basically a place where they're putting different think about it as a independent implementations of different algorithms. So it's not part of the actual Llama index package, but people can contribute to Llama Hub. It's very similar to TensorFlow Hub back into the day. Okay. Then we also need to install the Llama index core. And since we're going to be using the OpenAI LLM, so we need the OpenAI package as well. Okay, so very similar to what we did before, here we are just using the simple directory reader to read the PDF file and that will convert it into Documents objects. Next, as I said, we need to download the Ragatui retrieval pack from Llama Hub. So here how you can download it. We're using the Download Llama pack, then pass on the name of the package pack that you want to download and where you're going to store it. So this will basically download and install all the dependencies. Next, we also need to load our OpenAI LLM because that is going to be used to generate responses when it's provided with the context. Okay. The rest is very similar to what we saw before. So we have our regulatory retrieval pack and we pass on the documents. So this is basically the list of documents, what LLM, it's going to use the name of the index and how many relevant chunks we want to retrieve. Right. So previously I said three, now I'm just using the default value of five. Right. So this is basically create a Rack pipeline end to end Rack pipeline. Okay. So once this runs, it will do very similar embedding process. Again, this is going to be using plate because it's using the regular tree package at the backend. But if you want to use FAIS as a DB or index, you can use that as well. It just needs to pass on this specific flag. Now this does the document chunking a little different and that's why we will get a different number of embedding vector or dimensions. Right. The total number of examples that are going to be converted are going to be a little different, but I think you can control that. Now in order to run this, we will use the run command. So basically you use the pack that we created, then call the run command on it and pass on our query. So for the first time, when you run the first query, it has to load the index in memory. That's why it's going to take a bit longer and you will see this message but for subsequent queries is going to be extremely fast. Right now the response contains multiple things. There is an actual response object which is the response of the model. But along with that you also get the information regarding different chunks or documents that are retrieved by the model. Right. So here's the final response if you want to look at it. Instruction tuning as a technique that allows pre trained language models to learn from input and response pairs. So overall the response that it created was pretty accurate because the retrieval step is I think, relatively more accurate compared to smaller, dense symbolics. Okay, so this was a quick video. I just wanted to show you how to use Colbert based Contextualize late Interactions as a possible replacement to dense embeddings. It's a very powerful technique. I highly recommend everybody to check it out. This can definitely help you take your.