Better RAG: Hybrid Search in Chat with Documents | BM25 and Ensemble

Welcome to the first video in our new series on advanced rag. In this series you will learn how to create chat with your document systems. But we're going to go well beyond basics. So here's an architecture of a basic chat with your document system or a RAG pipeline, which stands for Retrieval Augmented Generation. So let me walk you through the basics before we talk about more advanced techniques. So in the beginning we have a document loader which will load a document. So let's say you have a number of PDF files those are going to be loaded. Then you want to split them into chunks because the document might be much larger than the context window of your LLM. For each chunk we compute embeddings. Those embeddings are then stored in a vector store which will create our semantic index that is going to be used for retrieval and that becomes our knowledge base. Now when a user asks a question, we compute same embeddings for the question or query and we do a semantic search on top of the knowledge base that we created using our vector store. The knowledge base will return the relevant documents or chunks those are re ranked. That becomes the context to our LLM and we feed the question along with this newly formed context to our LLM to generate responses. And this is a very basic setup of how RAC pipeline works. If you are an absolute beginner, don't worry. I have a series of videos on creating RAC pipelines. I'll put link to them in the description. So in this new series we going to be looking at ways and techniques on how to improve this basic RAC pipeline. And the first topic that we are going to be looking at is hybrid search. Now there are two additions to the RAC pipeline that we just saw. The first one is going to be keyword based search. So apart from doing semantic search using the embeddings that we just created, we are going to also do the traditional keyword based search whenever user asks a query. So basically in the user query we look for keywords and then we do a keyword research on all the chunks that are available. And the second component that we are adding is this ensemble retriever. So now instead of using just the embedding based retriever, we are going to combine results for both from keyword based search as well as from the embedding based retriever. The great thing about this ensemble retriever is you can assign different weights to the embedding based search as well as the keyword based search, which makes it extremely powerful. So this was a quick overview of the Architecture. Now let's look at a code example. The code I'm about to show you is going to be available in a Google Colab. First we need to install the packages that we will need. We'll use rank BM25 for doing the hybrid search, unstructured IO for reading PDF files, and Chroma DB is going to be used for creating our vector stores. If you have a scanned document, I'll also recommend to install these additional packages. These will help you in interacting with scanned documents and doing optical character recognition on top of your scanned documents. Okay, once we have everything installed, we need to load these packages. So first we are loading the unstructured PDF loader. Again, this is going to be used for loading the PDF file. Then we have our recursive character text splitter. This is going to be used for creating those chunks and we'll compute embeddings of those chunks. And for storing those embeddings we will need Chroma db. Next we are importing some packages from the LangChain core. This is the latest version of LangChain, So we will be importing a chat prompt template. This is essential for creating prompt templates. Then there is output parser, and then we also need the runnable pass through for processing our prompts. Next we will need an embedding model. Next we're going to be importing API endpoints both for our embedding model as well as for our LLM. Now in this case I'm not using a local model, but we are going to be actually accessing both embedding models as well as the LLM through Hugging Face API for absolutely free. So you don't need a computer with a powerful GPU to run this example. And at the end we will need that BM25 retriever for doing hybrid search as well as an ensemble retriever which will combine both the keyword based search as well as the embedding based search to provide our final output. Okay, so next we will need a PDF file. So in this case I'm going to be using the original Orca paper as an example. So on my Google Colab I created a folder called Data and and put this paper in there. Then we're using the unstructured PDF loader, load the file and we're going to be putting everything in this docs variable. Now next we need to convert the file that we just read into chunks. So in this case I'm defining a chunk size of 800. You need to play around with this and we are also going to have an overlap of under tokens. Next we need to process these documents that are uploaded. So for that we are going to be using the recursive character text splitter, which will create chunks from the documents that we upload in. The chunk size that is being used is 800 characters with a hundred characters overlap. I have actually a dedicated video on the chunking process. I'll highly recommend everybody to watch that. So basically when we create this splitter, then we process the document through the splitter to create our chunks. So we are all set here. Next we need to go through the embedding creation part. Now, since I'm running this on Google Colab and I don't want to use a local embedding model in LLM, rather I want to use an API endpoint for free from Hugging Face. That's why I need to provide my hugging face token. Okay, so let me show you how you can add your API tokens to Google Colab. Now here I added my hugging face API token to Google Colab and I'm using this function user data to retrieve that. And that's going to be stored in this hugging face token variable. But let me show you how you can do this process. So now Google Colab gives you the ability to create environment variable inside the Google Colab notebook, which is pretty awesome. So you don't have to put your API keys in the code anymore or you don't even need a new environment variable. So all you need to do is just click on this add new secret, then provide the variable name. So for example, here I have my hugging face API token. That's the name of the variable. Then copy the API key and make sure to activate that so that your notebook can access that. And once you have set that, you just can retrieve that using this function user data.get and provide the name of the environment variable that you just set. Okay, so that's a pretty neat feature from Google. Next we're going to be using the API endpoint for the embedding model and for that we are going to be using this hugging face inference API embeddings. The embedding model that we're using is the BGE base English version. Here you need to provide your hugging face token. Again, everything is secure because nobody can access this outside of your own Google Notebook. So this is pretty amazing. And we create this embedding model. Okay, so next we need to create our vector store using this embedding model. And for that we're going to be using CHROMA So we provide all the chunks that we created along with the embedding model that we just selected and that will create our vector store. After that we need to use this vector store to create a retriever. So this retriever is going to just be using the vector store that is our semantic base search. Now here you also want to set how many chunks you want this retriever to return in order to do semantic search. In this case I'm setting it to three, but if you have more complex documents, you can play around with this hyper parameter. Next we are going to set our keyword base search as well. For that we are going to be using the VM25 retriever. This will get all the chunks. It doesn't do any pre processing on them, but again it will return three different chunks where it finds those keywords that the user is looking for. Now we have two different retrievers. We need to put them together and combine them. And for that we will be using the ensemble retriever. So this ensemble retriever receives a variable or input called retrievers. You need to have a list of different retrievers. First we are providing the vector store base retriever, then the keyword base retriever. Now the great thing about this is that you can assign different weights or different importance to the results of these retrievers. Another thing is that you can actually combine multiple retrievers. So it doesn't have to be just two, you can add a lot more. And this is extremely helpful if you have different types of documents that you are working with and different retrievers work for different types of documents. So you can essentially get the metadata of the chunks and then based on that you can assign different importance to different retrievers. So that's an extremely powerful approach that we are going to be exploring in a future video. Now, the last thing that we want to do is to set up our LLM. We're going to, because that's the one that is going to receive the context as well as the original user query to generate a response. Now in this case we are going to be using the zham7b beta version. Again, I'm using the Hugging Face hub, so I'm not downloading the model locally, but rather we're going to be making an API call to this model that is being served on Hugging Face. And you can access this for absolutely free under certain limits. You can set different hyper parameters. So I'm setting the temperature 2.3. The max number of tokens that it's supposed to generate is set to 1024. And we will need to provide our hugging face token. Okay, so after this we need to set up the prompt template that we are going to be using with our model. Now since we're using the Zephyr model, we need to set up the proper prompt template. So here's the prompt template for this model. We are providing a system prompt which is you are a helpful AI assistant that follows instructions extremely well. Use the following context to answer user question and then I added things step by step before answering the question. You will get a hundred dollar tip if you provide correct answer. Actually tipping LLMs work, which is pretty crazy to think about, then it will get the context from our prompt and our query and we are expecting the assistant to generate a response afterwards. Now once all this is set, so next step we will use that template to create our prompt chat template for this model. And we will also use a output parser and we're going to be using the LangChain expression language. So in this case we need to provide our context that will be basically our ensemble retriever. The query is going to come through this runnable pass through function. We provide a prompt that is the user query. Then that will go through the LLM, the LLM will generate a response and that will be output through this output parser. So pretty straightforward stuff. In order to run this chain we'll just call the invoke function along with our user input or query. So let's quickly look at the Google notebook. So here's what my initial prompt was. What is instruction tuning? Now this is the response that you will get. So basically it shows you the whole prompt template or chart template we that we used. You can actually see that it gets the documents as context. So different chunks are presented as different documents, a part of as a part of the context, which is pretty awesome to see. And then you have your corresponding user input. And here is the model response. All right, so here's what the model response was. Instruction tuning is a technique used to train pre trained language models to learn from inputs natural language description of tasks and response pairs. This techniques allows the model to better understand or follow instructions given to them improving their performance on various benchmarks for both language only and multimodal tasks. And for language only task instruction tuning has been shown to improve zero shot and few shot performance of models such as plan and instruct GPT on various and then I think it ran off the tokens because we set the max output tokens to think around 1024 so probably expanding that will help. Here is another one more specific to the paper how does Orca compare to ChatGPT? And then it here's the response from the model. So it says based on the context provided Orca's performance on various tasks as compared to ChatGPT. Here's the summary. So geometric reasoning capabilities CHAT GPT outperforms Orca by 23% on geometric shape task, indicating that ChatGPT has better geometric reasoning capabilities than ORCA and table understanding and reasoning capabilities. So initially actually thought that it was making up this stuff, but there is actually a paragraph which talks about geometric reasoning abilities with this specific performance difference number. So this is pretty great. Again for table understanding it talks about penguins table task and the model was able to pick this up as well. So for example, it talks about table understanding and reasoning capabilities and it starts talking about this penguins example but it seems like it again ran out of tokens. Okay, so this was a quick example of how to use hybrid search where you combine keyword based search along with the embedding based semantic search and how to implement this using LangChain I hope you found this video useful. Subscribe to the channel if you would like more videos in this series. If you want to support my channel and my work, there are a whole bunch of links in the description. Check them out. And if you also need help with your own projects, I do offer consulting services. Details are in the video description. Thanks for watching and as always, see you in the next one.