Agentic RAG: Make Chatting with Docs Smarter
Learn how to add agent loops to RAG so your chatbot can rethink questions, re-retrieve documents, and check its sources before answering. This tutorial builds a working system with Transformer Agents, FAISS, and GPT-4o that cuts hallucinations and gives better answers than standard RAG.
Chapters
The information retrieval step in retrieval, augmented generation or RAG is highly dependent on how the user ask questions. If the query is not well formulated, the retrieval of information can be difficult even if the information the user is looking for is present in the knowledge base. In traditional RAG we get a single shortage retrieval. But we can fix this with agentic rag. To understand how agentic RAG can help, let's look at traditional RAG setup for retrieval. The user query is run through a semantic based similarity search that looks for the most relevant chunks in the information base. But what happens if the question in itself is not asked in the proper way? Well, in that case your RAG pipeline will most probably hallucinate, that is it will start making up answers or the LLM will tell the user that it couldn't find the information, even though if the information is actually present in the knowledge base. We can fix this by introducing agents in the RAC pipeline and giving them the ability to not only analyze the initial query, but also analyze the responses generated by the RAG pipeline. So here is how it usually looks like you have your initial query that is passed on through an agent. The agent will reformulate the initial query. Then that refined query is passed through the knowledge base and semantic based similarity search is performed. So that will retrieve some chunks that becomes the most relevant documents that the LLM can use to generate responses. But before that, the agent analyzes those relevant documents or chunks again and it will refine the query if it thinks that the documents that were retrieved are not able to answer the question. So based on the refined query, it will repeat this process again. If the agent is happy with both the context as well as the reformulated query, then that is passed on to the LLM as a final context to generate an answer. So as you can see, in this case the agent can plan, analyze and execute to implement the agentic loop. In your RAG pipeline, you have a number of different options. You have frameworks like crewai, Autogen or Lang Graph from LangChain which you can use to build agents. But in this video we're going to be using transformers agents. This, this is a little known features within the transformer package and this gives you the ability to create your own agents. Now personally, I really like it compared to the other packages because it's very clear and modular and does not really add a lot of different abstractions that you come across in the other packages. Let's walk through the code. I'll also put a link to a Google Colab notebook which implements the same code. In my case, I'm just running this locally. First we need to install all the required packages that include Pandas. LangChain the LangChain community package. We will use LangChain for splitting our text into. Then we also need the Sentence Transformer package. We're going to be using local embeddings. We need a vector store for this video. We will use FAIS and we will be using the agents from Transformer. Now if you run into issues, you can also install agents directly using this command. Next we will import all the necessary packages and modules. I'll walk you through the usage step by step. In the rest of the video, a couple of things to pay attention to is this react JSON agent. We're going to be also building some tools which the agent is going to be using. And we have the Hugging Face engine that is going to be used to select a specific LLM. Now in order to build rank we will need a data set. In this case I am using a data set which contains all the documentation available on Hugging Face. So this is basically the Hugging Face documentation, but you can bring your own knowledge base if you want. There are a total of 2,647 different documents in this data set. We run these documents through the document object and split those documents into separate different source documents. Now this step is going to look different for your own documents, but these are very similar to the LangChain documents object. And you can see these are documentation for different tools that are available on Hugging Face. So for example, there is a specific document for Transformers Gradio. I think there are a couple of those. There's a separate document for data set and so on and so forth. Now we need to split these documents into chunks and create embeddings. So for embeddings we're going to be using the GTE small. It's an open source or open weight embedding model. And for splitting the documents we will be using the recursive character text splitter. Each document is split into 200 tokens with 20 tokens of overlap. Now I cover tokenization process or chunking process in a lot more details in my Rank Beyond Basics course. So if you're interested, link is going to be in the video description. Check it out. That goes into a lot more details on how you can choose different chunking strategies and also how to build robust RAC pipelines. That goes beyond basics and there is a ready made code that you can start deploying in your own rag applications. After creating the text splitter. Now we actually need to split Those documents and we need to make sure that we remove duplicates. So this is a manual process that we're doing. So we take each document, split it into different chunks, and then we see that whether that chunk is present in the database or not, if it's present, we just simply discount all the duplicates. So this process will give us unique chunks which are a total of 43181 different unit chunks. So now we need to run those chunks through an embedding process to create our vector store. And for that we will use FAIS as a vector store or vector db. You can use any other open source vector store such as Chroma or Quadrant or any proprietary vector store. For similarity search, we will be using the cosine similarity that's most widely used. But there are other strategies as well. You can just take dot product of the embedding vectors or use something like Euclidean distance. Everything we have covered so far are the different components that belongs to a traditional or standard RAC pipeline. Now let's start introducing an agent into the mix. And as we know, agents have the ability to use different tools. So in order to do rag, we are going to use RAG as a tool. And for that I'm creating a retrieval tool. The name of the tool is Retrieval. The description is using semantic similarity retrieves some documents from the knowledge base that have the closest embeddings to the input. So if you have worked with function callings, the structure is very similar. You will need to define your inputs. The input inputs in this case is going to be a user query which is supposed to be a text or string. And so we need to provide a description of what exactly this input does. So this is a query to perform. This should be semantically close to the target documents use affirmative form rather than a question. And that's what we are telling the tool to do. Right? So the tool is going to get a query from the user and then it's going to do retrieval. And in order to do the retrieval we will provide the vector DB that we created in the forward pass, we will do the actual retrieval, we get the query and we're going to return up to 7 most similar chunks from the vector db. So that is the basic setup of our retrieval tool. So just to give you a quick overview of this is how our agentic RAG is going to look like. So we basically built this retrieval step. Next, we also need to provide the LLM that is going to be both used as a part of the agent as well as generating the final response so let's see how we can set that up. For the LLM part we have two options. We can either use the Hugging Face engine. So these are basically different LLMs that are available through a serverless architecture on the hugging face and you can directly call those API endpoints. So you can use something like llama3.8 billion or 70 billion. I highly recommend to use the bigger models, but usually for those you need the pro subscription of Hugging Face. But you can run these models locally as well. Or you can use something like OpenAI. Now for this video we were we are going to be using OpenAI, but you can use the same process to set up any other LLM. Just to give you a quick overview of how to set this up, we're going to be using the Message rule and glitz get clean message list functions from Transformer Agents LLM engine and then we just need to create a class. So let's call it OpenAI Engine. We're using GPT4O because that's the most capable model. And for the agentic workflow or frameworks you definitely want to use bigger models because those are more capable and they have much better reasoning abilities. So first we create the OpenAI client. Now if you're using something like Odama, you can actually replace those here because they are going to be using the same standard as OpenAI API standard. So we get all the messages that are coming in and clean them through this get clean messages list. So basically this will just get the messages from the user and the responses that are being generated by the agent or LLM. Then we are going to be using the chart completion endpoint. Again, if you're using something like Ollama, this will work. You will just need to provide the base URL in the OpenAI client. Okay, so we have our LLM. Next we need to create the actual agent. So the actual agent is going to have access to tools that is going to be our retrieval tool that we created. Then the LLM that we want to use and then the maximum number of iterations that you want to perform before it stops this agentic loop. And the idea is how many iterations the agent can do to look at both the query that it is reformulating as well as the retrieved context that is going to be passing on to the LLM. So that's going to be controlled by the maximum number of iterations. Now in order to run this agentic loop, we also need to provide a system prompt to the agent. So here's a system prompt and this is based on the example provided by Hugging Face. So using the information contained in your knowledge base, which you can access it with retrieval tool, give a comprehensive answer to the question below. So this is going to be the question, but we also provide some extra instructions. So respond only to the question asked. Response should be concise and relevant to the question. If you cannot find information, do not give up and try calling your retriever again with different arguments. Now this is where we are basically telling us telling it to retry. Make sure to have covered the question completely by calling the retriever tools several times with semantically different queries. So basically we're giving it to the ability to reformulate the user query as well. Your queries should not be questions, but affirmative form sentences. And here we are just providing a few short example. So rather than how do I load a model from hugging face in BF16, the query should be load a model from hugging face from the hub BF16 weights. So this is basically a few short examples that we are providing or in this case a single shot. And then we run this enhanced question or a system prompt through our agent. Now here's a simple question. How can I push a model to the hub? So we can take that question, run it through this agentic loop that we created and here's the step by step process of how it comes up with the answer. So this is the system prompt that we just saw. The question is how can I push a model to hugging face? Then this is the actual system prompt that the agent is using. Now here is kind of the internal thought process. So it says I need to retrieve information from the knowledge base regarding steps or procedure to push a model to hub. And here's the actual flow or these are actually the documents that it is able to retrieve step by step. And it will do multiple iterations if it needs to. And at the end based on the retrieved context is able to formulate a response in a proper JSON. So we can actually look at the the answer and it says to push the model to the hub. You can use push to hub method provided by various libraries and it really goes into the details of other different options that are available on how you can push a model to Hugging face hub. I also wanted to do a quick comparison of how the answers are different from a standard rag. So here's the standard rag that we can do. We take the retrieval question or retrieval tool which is basically implementing the basic RAG pipeline. We can pass that on, get the context and then pass that on to the LLM to Generate a final response if you run that through the standard rag pipeline, here's the answer that it came up with. So pretty short answer. It does have the information but the answer is not as concise as created by the agentic RAG framework when we use as the agent that we just created using the same LLM. Now here's the another here's another question. How can I use the O1 weight embedding models from hugging Face? So first let's look at the standard rack pipeline. So it says to use the open weight embeddings from Hugging face. You can follow these steps. So load the model, use the auto emed auto model class from the transformer library to load pre trained model from the hugging face hub and it provides us with an example model that we can use. But this is not as detailed as we would like. Now if you run the same query through the agentic rag pipeline that we just created, here's the answer that we get. So it says install the transformer library. So it also give us the command how to do that and then it gave us a pretty proper example of how you can actually use that model using the auto model. So very similar to the standard rack but with a lot more details. It also talks about how to pre process the input data and how to finally compute the embeddings. So this is definitely a lot more detailed compared to what we get from a standard RAC pipeline. So this was a quick video on how to create agentic loops for rag applications. I'm going to be creating a lot more content on agents for a lot is coming. If you are interested in that, make sure to subscribe to the channel. Also if you are interested in understanding and learning more about advanced rack techniques, make sure to check out my course link is in the video description. I hope you found this video useful. Thanks for watching and as always see you in the next one.
