LocalGPT: OFFLINE CHAT FOR YOUR FILES [Installation & Code Walkthrough]

This video is going to be a little different. We're going to be looking at one of my own projects that I am calling Local GPT. Now, this project lets you chat with your own documents on your own device using open source GPT models. Once you download the required embeddings and the LLM, no data leaves your computer and everything is 100% private. Now this project is inspired by this another amazing project called Private GPT which implemented an information retrieval system using local embeddings. There were two things that I wanted to improve on the Private GPT. In the original implementation they were using LLAMA CPP embeddings and for LLM they were using GPT for all J model. These are great choices. However, both of them run on CPU and that's why it is extremely slow. Now, in my Local GPT version we're going to be replacing both those components. First we're going to be using instructor embeddings, which are state of the art embeddings at the moment and it runs on GPU. And then for the LLM, we're going to be replacing it with Vir 7B, which is one of the best open source LLM available. Now the great thing about both of them is that they run on GPU and it's going to be much faster. You can easily replace both LLM as well as the embedding models with anything that you want. So for the rest of the video I am going to walk you through the installation process as well as we will have a detailed walk through the code to see how it's set up and how you can make changes if you want. Now please keep in mind that this project is under active development, so things will change. However, the basic structure of the project is going to remain consistent, so you will be able to run this using the instructions in this video. So let's have a walk through the record. But before that, let me show you the architecture of how this project is set up. Now the main question is going to be why do you want local or private GPT? So apart from the privacy concerns, one great thing about these type of tools is that it lets you use your documents as source of information for LLMs. So basically you are augmenting the knowledge base for the LLMs because they don't have access to your own data. Now, in order to implement this, here's the basic architecture that I'm using. You might have seen this or a similar diagram, so I will quickly walk through this. Now, there are two components. The first component is implemented in this Python file called Ingest Python. In this case, we are ingesting or feeding in information from our local files into the knowledge base of our local GPT. The second component is implemented in Run local GPT PY and this lets the user interact with the information or the knowledge base that was created in the first step. So we are going to be looking at both of these files in a lot more detail. First let me show you how you can set this up on your local machine. And then we will walk through the code base. So for that first go to this code button and click on this copy button. So it will basically copy this GitHub repo location. Next we will open an instance of Visual Code Studio. So I have already created local virtual environment that I'm calling Local GPT, but if you want to create one, so you can do it by simply typing in conduct create dash n and then give it the name of the local environment. So in my case it's local GPT. For this to work, you will need to have CONDA installed on your local machine. So in this case it's asking me a CONDA environment already exists. Remove the existing environment. So I'm going to say no. Now, in order to activate it, I'm going to type in conda activate and the name of my virtual environment. All right? Okay. Next we want to clone the repo. So get clone. You will need to have git installed on your local machine for this and just clone the repo. Right? So now we have everything locally installed. Now the first thing thing that you want to do is you want to install all the requirements. So here, all the requirements. In order to install all the required packages, we're going to be using pip. So pip install R and then requirements txt, right? Okay, so once you run this command, it will start downloading all the required packages and installing them. So once you clone this repo, you're going to have this folder called Source underscore documents and you need to put all your PDF text or CSV files under this folder. So currently there is an example document, the Constitution PDF, which is the Constitution of the usa. But you'll need to replace this with your own files for it to work on your own data. So first let's look at the ingestion part. Now if you are familiar with these concepts, you can skip ahead or if you want a lot more detailed description of these. So watch this video. All right, so currently the local GPT supports PDF text and CSV files. This list is going to expand with the time. But the basic idea is that we want to retrieve information from our own local files. For that we split them into smaller chunks because we cannot feed in, let's say 100 page document into an LLM and then we compute embeddings on top of it. Embeddings are basically converting our documents into a string of numbers or a vector of numbers. Instead of characters, you are just representing a specific chunk or a document by a vector of numbers. And using these embeddings along with the original documents, we create a semantic index, which is basically your database of the information that you have provided in the text or PDF files. So in terms of the code we call the main function. Now let's see what is happening. Right, so it will first simply print the path of the source directory and saying that loading documents from there. Then it makes a call to this function loaddocuments. So let's look at this implementation. So this loads documents, receives the source directory, basically the path to this as an input, and then it goes through the list of all the documents. So you can have multiple different document types. And then it uses this function called load single document to each to load each one of those document. Right, so let's look at here. Right, so currently, as I said, we only have support for three different file types. So for each file type, when it receives a path to a new document, it checks whether it's a text, PDF or CSV file. Right? And depending on the type of the file, it will use a different data loader. So for text file it's using text loader, for PDF it's using PDF minor loader. And for CSV it's using CSV loader. So essentially you are simply going to get a list of document paths or document loaders. We need to split them into smaller chunks so that our LLM can process them. In this specific case, I'm using a chunk size of. This is the chunking process over here, right? And with an overlap of 200. Now I go over these choices in a lot more details in my other videos. So check them out. Then what we do is we take the documents and pass them on to this text splitter. So each of the document is going to be divided into multiple chunks. Now the next part is that we need to compute embeddings for each of the chunks. Now in this case we're using the instructor embeddings. So here you can actually replace it by any type of embeddings that you want. But I have found that the instructor Excel embeddings works pretty good for most of the applications. Now keep in mind that this is going to be not the most efficient. You can actually tweak the instructor embeddings even further for your application. Think of it as a neural network so you can further fine tune it. Okay, so once the embeddings are computed, then we need to create a knowledge base. So for that we are using Chroma db. So this takes your text documents which are chunked, so these are the chunk documents and the corresponding embeddings and put them in a vector store and it persists. So basically that means that you will be able to store them on your local drive so that you can use it for later on. All right, so that's the first part of how you can create your vector store or database based on your own documents. Now, when you run this code, you will see that it will create another index folder and under which it is going to have all the files related to your vector store. Okay, in order to run this, you just need to type in Python, ingest Python. It's going to walk through the step by step process. It will first read your document, convert them into chunks, then compute embeddings and store them in your local vector store. We are going to look at an example later on, but these are the type of messages that you would expect. Now, probably you notice you don't really need to set up anything at all. You don't have to set up env file in here or even the path. Everything is taken care of automatically. Now let's look at the second component. So let's say you have your knowledge base ready and a user wants to interact with your data. So you simply want the user to ask a question in natural language, then compute embeddings for the question that you had based on the type of embedding models that you used. Do a semantic search on your knowledge base, right? And it will simply return the most relevant documents or chunks that are present in your vector store. And then you use the original documents related to the chunks as a context for your large language model along with your question, and you get an answer. Now let's look at how this actually looks like in the code. Now for that we're going to be looking at the run local GPT file. Now again, we start at the main function. So let's go in here. Now the first step is that it has to load the embedding models again just to compute embeddings for your question, right? So that's why we are embedding, we are loading the exactly the same model that we used for computing embeddings for our documents. Next, we also need a knowledge base and because that's where we are going to be asking questions from. So we also load our vector store or knowledge base using the Chroma db. Now this is going to be the one that we created in the ingestion phase and that's going to be created as an index. Next, not using this callback, so I'm going to just comment this out. All right, we need to load the LLM that we are going to be using. Okay, so let's see what's going on in here. So for the load model function we are loading a model, in this case that is vocunia 7B from hugging face. Now I'm trying my best to make it modular. So if you want to try another model, you can just replace this line with the model ID of that specific model. So let me show you an example. So we will look at this user Tom Jovens or the block who is a legend when it comes to LLMs on hugging face. So just look for anything that supports the hugging face format. So for example, this model, right? Then you simply need to copy this model name and then come here and replace it. Now currently this will support the llama based models because we are using the llama tokenizer. But you can easily replace it with other type of models that you want. Next, we load the model using the llama for causal lm. We got our model and then we are creating our pipeline. Now since it's a decoder only model, that's why we're using the text generation. Then pass on our model tokenizer and here's the maximum length. So depending on the model you can actually change this. In this case I think vocunia 7B supports only 2048, right? The temperature I set it to zero. So because I don't want any creative answers, I want for the model to extract information from my document. And then there are a couple of other parameters that you can play around with. So we create the pipeline and then pass it to the Hugging face pipeline function and we get our local LLM. Now when we call this function we will get our LLM and then we're using LangChain for information retrieval where we pass on LLM we are using the stuff type chain, right? Then we are passing out the retriever that we created based on our vector store and we are also asking it to return source documents so basically where it's getting the information from. Okay, so we are basically walking through these steps to get an answer and then the user asks a question. So there is this while loop. If the user types in exit, it will break the loop and you get out of the program. Otherwise there's going to be an infinite loop that is running. Right, so let me quickly walk you through this. This is purely based on the private GPT code. So first the user interact query, right? Then we run that query through our question answer retrieval chain. That will give us an answer in terms of the results that the LLM found as well as the source documents. So next we simply print both the question as well as the answer first and then we go through one. So next we simply print the question as well as the answer and then we go through the source documents. So this was a lot of code. Now let's go and watch this in action. In order to run this, you will need to type Python, then run underscore local GPT PY and then it will give you the user interface. So I'm going to switch to my other machine in order to run this part. Okay, so once you run this, run local GPT. So you're going to be presented with this place where you can type in your prompt. So since we are working with the Constitution of the United States, let's ask it a couple of very simple questions. So I'm going to say how many states are in the USA? So here's the response from the model. There are 50 states in the USA. Then it also will give you some source documents. So you can actually go through the list of documents that it used. For example, it's getting information from these different articles within the Constitution. Now, as I said, it's under active development, so I'm going to be adding a lot of more features. So it's going to have support for more models as well. As I plan to add a graphical user interface. You can simply drag your documents and chat with it. So a lot of cool updates coming in in the future. If you would like to contribute to the project, create a PR and I would integrate your contributions. If you would like to support my work, there are quite a few ways. Check out the description of the video. If you have any questions or comments, put them in the comment section below. If you like the video, consider liking it and subscribing to the channel if you haven't done so. And thanks for watching and see you in the next one.