PrivateGPT: Chat to your FILES OFFLINE and FREE [Installation and Tutorial]

I was planning to create a video on using open source embeddings and Open source LLM to create your custom chatbot for your documents. That's when I came across this amazing project called Private GPT. It's already trending on GitHub with around 15,000 stars. This project combines an open source embedding models with open source large language models in a really nice package way. Now the best part is you can run this locally without an Internet connection. It's 100% private. And now data leaves your environment at any time. This follows all the steps that we have been talking about in the last few videos. Under the hood, it's using LangChain with GPT for all and Llama CPP embeddings. So in this video I'm going to show you how to set it up on your local computer and get it running. One thing to keep in mind, you will need to have Python 3.10 or later installed in order to run this. Ideally you want to have Git also installed, but if you don't have Git, you can still run this by downloading the zip file. So first go to this code button, click on it, and here's a link to download this using git. Or if you don't have Git on your local machine, then you simply download the zip file, unzip it, and it should have exactly the same content. The only problem with downloading the zip file is that you will not be able to get automatic updates every time. You'll have to download a new zip file if there is any changes here. Now another great thing about this project is they have actually provided all the models that you need. So here is the large language model that you need and then the corresponding embeddings. You don't have to go to some other place and download these. So what I would recommend is simply download these two files. If you actually click on them, it will download the files. I have already downloaded these files, so I'm not going to do this step. Now, assuming you have GitHub, then come here, copy the link and let's open a new Visual Code Studio session. So here we have a new Visual Code Studio session. Now, in order to download the files, first I need to use Git to clone the repo. So I'm going to type Git clone repo and then I can actually name a folder where I want to put this in right now. You see I have a couple of folders. If I simply run this, this will download it into Private GPT folder. But if you want to rename it, you can actually provide another name here. So for example, if I say private GPT V2 or something like that, so it will create a folder with this name and download all my files. For this specific case, I'm happy with Private GPT, so I'm going to just click Enter and then you see there is a new folder that is being created and now we have all the files that we saw on the GitHub repo. Now, within this new folder we need to do a couple of things. First, we need to create a new folder inside this folder called Models. And then you need to go ahead and copy the models that you just downloaded. Now you will see I added these two files. Both of them are BEN files. One is the LLM and the second one is the embeddings that we will be using. Now, throughout this stylish process, I'm going to be referring back to the instructions on the repo. With a lot of these open source project there are updates, right? So if I make a video, it might not be up to date. So make sure that you check out the repo. If you encounter any issues. There is an example env file inside the repo and we need to change it to env. So here's the example env file, right click on it, rename it, and then simply delete the example part. So you're going to be left with env. Now, within this env file you have different parameters for your model and project. So this is the directory that is going to be created to store your vector embeddings. Then you need to have a path of the models. We created a model folder. That's where your models are going to be stored. Similarly, the first one is for the embeddings, so that's the embeddings model. The second one is your large language model or LLM. Now if you want to use another model instead of these two, you can simply copy them into the model folder and update the model name here. For example, if you have seen my last video, I was using Instructor embeddings as an open source embeddings. But in this specific case they're using the LAMA CPP embeddings. But you can download a BIN file that is in the GGML format and simply replace this with that specific model. Okay, when it comes to the model path, there is one more thing that we need to do. Because of the way LangChain loads the llama embeddings, you need to Specify the absolute path of your embedding model binary. This means that it will not work if you're using a home directory shortcut, right? So what exactly does this mean? This means you need to actually provide the whole path. Now, in order to run this, I'm going to be using my Ubuntu machine because it has a powerful gpu. In order to run this tool, you need to execute two different files. The first one is ingest PI and the second one is private GPT. This ingest PY is going to be used for creating the vector store using Chroma db. In this case, we we're using State of the Union address txt as our source file. If you want to use your own files, just put them in this Source Documents folder. Now you can put text, PDF or CSV files. This tool has support for these three different file types. But for this specific example, we're going to be using State of the unit txt as our information source, which you run in just py. It will create Vector store based on all the information available in this State of the Union txt file. Now, in order to execute this file or run it, simply come and click on this Run Python file in the terminal button. Start executing the file. Now, this will do the division of the file or multiple files into chunks, then computing embeddings for each of the chunks and then storing them in the vector store. The process is going to take quite a while, so just be patient with that. Now, assuming everything goes well, then you want to run private GPT if you want to interact with the chatbot. So in this case, again, we're going to come here and click on run this file in the terminal. Okay, so now we can ask it a question. So since it's a State of the Union, let's ask it, what did the President say about Russia? Okay, so the query is going to take a few, probably a few minutes to process and depending on the speed of your computer. Okay, so here's the response. The President said that Russia has unleashed violence and chaos and the Putin alone is to blame for the economy and military losses suffered by Russian economy. Right? And it goes on to say some of the things, right? So then it simply gives us the question that we asked and the corresponding answer. So we have the answer and then the corresponding documents where we found a similarity based on the embeddings that were used now. So these documents are going to be used as contacts by the LLM to generate a response or answer in this case. Now, one thing I Noticed is this tokenization unknown token. So seems like the truncation process or the chunking process probably produced some tokens that the LLM is not able to read. Anyways, this is how you can do document retrieval using the Private GPT tool. Now as I said, it's private. Nothing leaves your computer. It's not connected to the Internet. You can do this without Internet connection and it's 100 percently running on your local machine. Now in the next video we're going to look at how we can create a graphical user interface around this or something. Since we looked at it, how it works. I would like to walk you through the whole code. I think this will be helpful to understand how this whole thing is implemented and get a deeper understanding. So first we will look at this in just PY function. So here the main function is the entry point. That's where the execution actually starts. So first it's simply getting the different environment variables that were set as a part of our env file. Next it's loading all the documents. So let's look at the load document file function. Now this function looks at different file types that are available and create a list of paths to those files. So let's say if there are three different text files, so you will have a list of three different paths to those files. Then they're combining all those lists together into a master list. And then they're using this load single document which essentially goes and check the file type. So if it's a text document, it will use the text loader. If it's a PDF file, it uses the PDF loader. And for CSV it uses a CSV loader. At the moment it seems like they're just supporting these three different file types. But you can easily extend this by using a different loader and check the corresponding file type. So now ultimately you are going to have a list of loaders and they're using list comprehension to get that list. So now after running the load documents function, you get a list of all the documents. Then this is a simple thing that we have been looking at. So you recursively split them into smaller chunks. In this case they're using 500 as a chunk size with an overlap of 50, right? And then read those text into this text variable text. They simply create an object for the NAMA CCP embeddings and then they're using CHROMA DB to get the embeddings for the documents that we just loaded. Now these are the things that we have been looking at in the last few videos. If you want to get a better, better and deeper understanding of these topics, I would recommend watching the previous videos. I will put links in the description of this video. So, in terms of a flowchart, I have shown this to you in my previous video. So essentially we are at this step where we create the embeddings and store them in a vector store. In this specific case, we are using Chroma DB for that purpose. Next, let's look at what is happening in this private GPT PY file. Okay, so initially this is llama embedding model. So basically the path of the embedding model that we have stored, then where the vector store is stored. So that's the directory path, right? And some other environment variables that we are using. Again, we are going to be entering this code at the main function. So that's the entry point. So initially we're simply loading our embeddings and this is required because we're going to be running embeddings on a query that we're going to be providing to our language model. Next we are also loading our vector stores and setting it as a retriever. And next it checks the model type. This is coming from the environment variable. If you look at here, the model type is GPT4ALL, but I think it also accepts LLAMA based models. So here it checks whether it's a llama CPP or GPT for all. And depending on the type of model, it will load a different LLM. Now since we have a GPT for all, so we are going to be loading a GPT for all LLM, right? And if the model type is neither of these, then it will throw an error. Next up, we're simply creating LangChain, a retrieval question answer chain. And the type of chain stuff we pass on the LLM, the retriever, right? And we are also asking the chain to return source documents. And then there is an infinite for loop to interact with the user. So it simply asks for a query, then pass that query to the chain that we created and gets the result. So the result has two things. One is the result which is the response, so that's the answer. And then there is the source documents, right? So next we are simply printing the response. First printing the query, then the response. And then since there are multiple documents, so we are taking one document at a time and printing them again. And since this is an infinite while loop, so it will actually go on, so you can ask one question, it will give you a response, then you don't have to run the script again, it will. You can simply ask it another question and it will give you a response again. Okay, I hope this gives you a much better understanding of what is going on under the hood. I have covered all these topics in a lot more detail in my previous videos, so I would recommend to check out the playlist on large language models. So this was a step by step process of how to install this private GPT on your local machine. Make sure to check out the GitHub repo because these are open source projects and things can change for example if they add another model so these instructions may not hold true then if you want to learn more about LLMs and everything that is going on in the generative AI space, check out our Discord server now. If you are working on a project like this or have some other technical questions and need a consulting session so you can reach out to me through my email and if you have any questions or comments, put them in the comment section below. I hope you found this video useful. Thanks for watching. See you in the next one.