Talk to Your Documents, Powered by Llama-Index

Today I'll show you how to implement chat with your document system in just four lines of code. I recently started exploring Llama Index, which is an alternative to LangChain. Just like LangChain, Llama Index gives us the ability to build powerful applications based on large language models. These applications include Document Q and a data augmented chat box knowledge agents. The great part is you can connect different types of data sources, including structured, unstructured and semi structured data sources. So using Llama Index, you can quickly build LLM based applications just like LangChain. The main reason for me of exploring Llama Index was the ability to fine tune embedding models to improve the performance of Document Q and A systems using large language models. So basically, that means you can not only fine tune the large language model on your dataset, but you can also fine tune their betting model and that will improve the performance of your retrieval system. I'll cover this topic in a future video. In this first video, I'll show you how to build a chat with your document system using Llama Index. You probably have seen this diagram before. So there are different components. The first component is basically loading your documents. Then you want to divide your documents into smaller chunks with a predefined chunk size. For each of the chunks, you compute different embeddings. These embeddings are numerical representations of the text contained in a chunk. Then you create a semantic index. So basically this is your vector store. After that, you can actually use this vector store to chat with your documents. Now, in order to chat with your document, the system takes your questions, compute the same embedding as the embedding it computed for the chunks, then performed a semantic search on your knowledge base. So as a result of this search, it will return a certain number of chunks that are relevant to your question. So let's say by default, Llama Index returns only two chunks out of all the chunks available in the knowledge base. And these return chunks will be used as a context to the LLM of your choosing. So the LLM gets the question from the user along with the context that was returned by the embedding model during the semantic search. And the LLM will generate a response and that is going to be shown to the user. As you can see, this whole process can be divided into four different steps. And I'll show you how to implement this whole process using just four lines of codes within Llama Index. Okay, so let's start looking at the code. So I'm using Google Colab in order to show you the notebook and the code. So first we're installing all the required packages. So we are installing llama index, then OpenAI. So for the initial example I will show you how to implement the document Q and a using OpenAI LLM and OpenAI embeddings. But I'll also show you how to replace the OpenAI LLM with an open source LLM from Hugging face. And for that we will need transformers package to access the hugging face based LLMs and the accelerate package in order to accelerate the processing or speed of running local LLMs. Now for the initial example we will need the OpenAI API key in order to access OpenAI's embeddings and LLM. So you need to provide those. Okay, Next we need to import the required packages. So we are importing OpenAI from Llama Index. Then we are importing two auxiliary packages, functions or objects. One is the vector store index and the other one is Simple Directory Reader. The use will become clearer when we look at the code. And then we are also importing some packages just to properly format the output from the model. Now, in order to chat with your documents, you will first need access to those documents. So the way I'm going to do it is I will create a folder called Data. And within this folder I'll upload this file. Now, this specific essay is titled what I worked on. And this is basically an essay written by Paul Graham when he was working with Y Combinators. So we're going to be using this as our example data set. But you can upload any type of documents, for example text, Word documents or PDFs. Now, going back to the architecture, first we will need to load the documents. Now, Llama Index provides a very simple functionality to do that. So there is this class called Simple Directory Reader. You simply need to provide the name of the folder or path of the folder. So in this case our folder is called data and that's why I provided the name Data. Then we need to call the load data function on top of that and that will load our documents. Now the great thing about this is you can have different type of documents within the folder and this class will define which loader to use for those documents. And we can also look at it. This is basically the text file and the whole text file is loaded in here. Now the next step in our architecture is to create a vector store. So this involves dividing your documents into chunks, imputing the embeddings, and then storing both the embeddings as well as the chunks in a vector store. Again, Llama index makes this very easy. We're going to use the vector store index class. Then we are going to be creating the vector store from documents. We'll pass on our documents in here and this will create our vector store index for us. Okay, so we just created our index. Now next we are going to look at how to customize different options. For example, the chunk size, the type of embeddings and everything. But before that we just want to ask questions. So for that we need to create a query engine. Now I find this to be very streamlined it in Llama index compared to LangChain. So here basically you can create a query engine on the index by using the as query engine. Now if you want to chat with your documents and want to have memory, so you will going to use a function called as Chatbot and that will basically enable memory. But as a query engine you don't have the memory component in here. Now in order to use this query engine to get a response based on the user question, so it has to go through this process that it needs to compute embeddings for the question. Then do the semantic search and use the returned chunks as context for the LLM along with a question to get to generate an answer. All of this is implemented using a single line of code. Basically what you do is you use the query engine, then call this function called query and you pass on your question to this function. So for example question that we are asking is what did the author do growing up? And we will get a response in here. Now the response that you get has a lot of different components in here, but we're simply interested in the answer. I'll show you what all this text means in a bit, but first let me show you how to get the answer from the LLM. For that we will use the display function from IPython and put the response in markdown and also bold it. So here's the answer from the model. The author worked on writing and programming outside of the school. Before college they wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. Right. So this is the answer that was generated by the LLM using the context provided by the embedding model. As you can see, apart from the imports, we were able to build a document Q and a system using just four lines of codes. And this implementation is very clean. Next, I want to show you how to customize different parts of this diagram. For example, how do you define the chunk size, chunk olap or how do you change the LLM? And so on and so forth. But before that, let's look at some very important concepts when it comes to the vector stores. So right now the index that we created is just in memory, but you can persist this by calling this function persist on the index that you created. Now, the way it's going to work is if we run this. So by default it's going to create a folder called storage. And within this you will see multiple JSON files. Now, this functionality is critical because it enables you to create the index one, store it on your disk, and then use it in your future runs. Right now, in order to load a vector store that was stored on the disk, we're going to be using the storage context along with the load index from storage class. Now, the way it works is that you will use a storage context to read the content of the index. And then from the storage context that you just created, you are going to recreate the index, right? And then you can use it exactly the way we did it before. Now, I hope this is clear. Another question that comes in mind is what exactly is in this vector store? Now, in order to show you, I'm going to open all these four files. And the two important files that we want to look at is this DOC store and the vector store. So the vector store, as you can see here, is basically the embeddings that were computed for each of the chunk. And depending on the embedding model that you choose, you are going to get different number of embeddings per chunk. Now, similar to the vector store for embeddings, there is a DOC store. So here you will see different chunks from the document. So basically this is text that is divided into different chunks. And the third most important thing is the index store. Basically this has the hash or address for different chunks. And using this index store, it determines which embeddings in here belongs to which chunk. Now, when you create a vector store, so it's going to store both the embeddings that it computed as well as the original chunks from the documents. Now, during the retrieval process, the LLM is going to have access to both the embeddings as well as the chunks that are retrieved based on the embedding model. I hope this is clear. Now let's look at some of the customizations that you can make. The first customization that we are going to be looking at is how to change the LLM. Now, in order to change the default values, we are going to be using the concept of Service context. So this defines all the default values of different parameters within Llama Index. Now, by default, I think the LLM Llama Index uses is the DaVinci 03 model. However, if you want to change it to, let's say GPT 3.5 Turbo. So we basically use the OpenAI function within Llama index, create an LLM based on that. So you can define the temperature, the max number of tokens that it's going to generate, right? And then pass that LLM to the service context to change the defaults. So here we are changing the default LLM to GPT 3.5 and then pass that LLM to the Vector Store index that we just created. And then you can use the Vector Store index exactly the way we used above. Now, if you want to use another LLM such as, let's say Google Palm. So again you can simply import the Palm LLM from Llama Index LLM class, then pass that Palm LLM to the LLM within the service context, that will basically change the default value. And then again you can pass it exactly like here to the Vector Store index and then it will start using the Palm LLM. In this case, again, you will need to set the API key for Palm, but the process is going to be very similar to what we did for OpenAI model. Now, another parameter which is very important when it comes to these chatbots for your documents is the chunk size. So in order to change the chunk size, we can follow the same pattern. So here you can simply pass on and define two other parameters. So the first is the chunk size, the second was one is chunk overlap. So here I'm using thousand tokens, for example, with an overlap of 20 tokens. So I would recommend everybody to look at the documentations and see what the default values are and how you can change them in here. Now, another way of doing this is you can set the global service context, so in that case you don't have to pass it to the Vector store index that you create. You simply set the global service context using the service context that you just created and then you can use the those as defaults in the rest of your code. Now, in this last example, I'll show you how to use an open source LLM from Hugging face. The way you do it is we import, we are importing the Hugging Face LM class from Llama Index LLMs. Then you can also import prompt template from the prompt class. This is very similar to what you probably have seen in LangChain, right? So here's basically the system prompt, because some of the open source LLMs need a system prompt. So this system prompt is specifically for the stable LLM models. And close the user query within some special tokens. Right now in order to use the LLM, it's basically using Llama CPP in the background. So we create an LLM based on different parameters. So for example the context window, you can pass this on maximum new tokens that you want to generate, right? And other parameters such as the temperature that you want to set. Do you want to do sampling or not, right? Then you need to pass the model name as well as the tokenizer name. The system prompt, if the model supports one, then the prompt template with the user query, right? If you have multiple GPUs and you want to use all of them, you can set the device map to auto. Now in some certain cases there are certain specific tokens that are going to be used as stopping IDs. You can pass on those depending on the model that you're using. Because different models are going to be using different tokenizers, right? So these are different parameters you can set. Essentially this will create an LLM class. Now you can simply pass that LLM class to the service context that we created along with let's say the chunk size that we want to use. This will set the default LLM to this hugging face LLM and then you can do query on your vector store and this LLM is going to be used for generating the responses. Now this was a quick overview of how to build a document Q and a system using Llama Index. Now, Llama Index is string powerful and it has some really cool features. For example, one of the applications that I'm currently looking at is how to fine tune embedding models along with fine tuning the LLMs. So this is going to be the first video in this series on Llama Index. I will be creating more advanced tutorials on the on different topics and different features that Llama Index have. If you like what you see on the channel and would like to support it, check out my Patreon. Now, if you want to build a Q and A system and want to use LangChain instead of Llama Index, I would recommend you to check out this video. As always, if you found the content helpful, consider liking the video and subscribe to the channel. Thanks for watching and see you in the next one.