Llama-2 with LocalGPT: Chat with YOUR Documents

In this video I'll show you how you can use the newly released Llama 2 within the local GPT project lets you chat with your document on your local device using the GPT models. And no data leaves your device and everything is 100% private and secure. So in this video I will show you a step by step process on how you can use Llama 2 models to chat with your own datasets. I'll also walk you through some of the updates that have been made to this project since the initial release. But before all that, I would like to thank all the contributors who made this a successful project. Thanks to everyone. We have over 10,000 stars on GitHub and it was trending on GitHub for quite some time. Now, if you're interested in how the code works, I would recommend you to watch the original video. But in this video I want to show you how to set this up on your local machine. And then we will use llama 2 model as our LLM as well as some of the changes that I want to highlight in this video. Okay, so first and foremost we need to clone the repo. So simply click on this green button and then copy the link. Now, in order to use this, you need to have GitHub as well as Python installed on your local machine. Okay, so I opened up a new terminal window and now let's go to the GitHub, because that's where I store all my projects. So I'm going to CD into documents, then GitHub. Okay, and now we need to clone the repo. So we're going to say Git cloned and then we'll provide the repo link. And after that I usually create a separate folder. So let me call this Local GPT and underscore Llama. Okay, Llama two. All right, so that's going to clone everything. Now we can CD into this folder. So let's see, type in cd, then Local GPT. So if you see I have a whole bunch of local GPT projects, these are definitely different things that I keep testing. All right, so now let's type in ls, that's going to show us the contents of that folder. Right, so we simply clone the repo so far. Next we want to create a virtual environment. So I'm going to say conda create dash n and then the virtual environment name. So let's call it local GPT Lama 2. Right, and then I want to use specific version of Python. So in this case I'm going to be using Python 3 10.0. Okay, let's hit Enter and then it will start setting up our CONDA environment for us. So you will see something like this. We'll simply proceed with the installation. So if you hit enter, the default value is yes. So it's going to proceed with the installation. Okay, so we are all done. We created our virtual environment. So let me just copy this and I'm going to open the code in a Virtual Code Studio instance. And just to show you how you can start using the Llama 2 models within local GPT project. So here I open a new Visual Code Studio instance. There are a lot more files in here compared to the original codebase that I covered in my initial video. So most of this is related to formatting in Pull request. But there are a couple of major changes. For example, now we have a Local GPT API as well. Apart from that, you will notice this new folder called Local GPT ui. So we actually have a graphical user interface as well. I will cover this either later in this video or we'll have a dedicated video on this. But today I want to focus on the Run Local GPT PY file. Now, first we need to open a new terminal. So you can come over here, this toggle panel, click on it, and this will open a new terminal for you. Since we created a new virtual environment, so I'm going to first activate that here. So let's copy this and we go back to Visual Code Studio. And here you can simply paste that in and hit enter. That will change the virtual environment. So you can see it here. Now we are using the Local GPT LLAMA virtual environment. After that we need to install all the required packages. So for that I'm going to first type in Python dash M just that to make sure that we are using the Python associated with the virtual environment. And after that pip install-our requirements.text file. Now, keep in mind, I'm running this on an M2, so the installation might be a little bit different than if you're running it on a Nvidia gpu. So just wait for it to finish the installation and after that we'll come back. Okay, so we are done with the installation. Okay, so let's first talk about what is different in this version compared to the original version. And then I'll show you how to use Llama 2 models as an LLM. So there are basically two stages in order to run this project. The first one is Ingest py. So the idea behind this file is that if you have your own documents or code base. You want to create embeddings out of it and store those in a vector store. And this is what the ingest PY file does. For the sake of time, I'm not going to go into a lot of details of the implementations. Now, a couple of major changes to the ingest PY file. First is now it supports multi threading. So the document ingestion process is going to be a lot faster than compared to what was it before? Other than that initially it was just supporting Nvidia GPUs, but now you can run this on CPU or you if you have Apple Silicon. So you can use MPS as well. Right? So these are different options that you can pass on and I'm going to show you how to run this updated version. Okay. To explain the working. So first you need to go and put all your documents in this Source underscore Documents folder. So as an example, we have the Constitution PDF in here. Now if you want to ingest that or create embeddings for this document, then you will simply type in Python ingest py. Then, depending on the hardware that you have, if you have an Nvidia GPU and you have Cuda set up on your local machine, you don't need to do anything, just hit Enter and it will create embeddings and a vector store for you. However, if you are running this on a CPU or if you have Apple Silicon, so you need to pass this other flag, which is hyphen, hyphen, device type, and then describe the type of device you have. Right? So let's say in this case I have an MPS or I can use a cpu, right? Both of them haven't seen really big performance differences. But just type this in, hit Enter and now it's going to use the Instructor embeddings, which are the default embeddings for this project to create embeddings for our file in here and then create a vector store on top of it. Now, the project supports multiple different file types and you can have more than one file in here, so that's not an issue at all. Okay, so I'm simply getting this warning installed. Version of Bits and Bytes was compiled with GPU support. I don't have a gpu, so I don't really need to worry about that. Okay, so with the default parameters I have a total of 72 chunks. And you will see here it created this DB folder and within this we have our Vector Store index. Right. Okay, so all good so far. Now let's go to the run local GPT PY file. This is basically where you're going to be running your LLM as well as asking questions from your documents or chatting with your documents. I think it will be helpful if we talk about some of the changes in this file. Just like the ingest PY file. Now you have options to choose which device you want to use. If you have a gpu, you can set it up to cuda or that's actually the default value being used, or you can use a CPU or mps. Originally, when I put together this project, it only supported the hugging face format models, however. So now thanks to the community help, we have extended the support for almost all the models that are available on Hugging face. So if you want to use a quantized model and you have an Nvidia gpu, you can now use the GPTQ format files or models in here. Also, this project now supports the GGML format files thanks to Llama cpp and again, I'm really thankful to all the open source contributors who contributed to this project. Okay, so let me quickly walk you through the process of how you set up a new LLM. So in order to use LLM of your choice, you need to set up two things. First is going to be the model id and the second is the model base name. If you are using a full unquantized model in the hugging face format, you simply need to set up the model ID and make sure the model base name is set to none. However, if you are using let's say the quantized models, for example, if you're using the GPTQ format models, then you will need to define the model ID as well as provide the model base name. Now let me show you where you can get this information from. Okay, so for that let's go to hugging your face and we're going to be using models from the block. Now let's assume you want to use GPTQ quantized model. So in that case you simply need to click on it. For example, this is the model that you want to use, so you will click here. Now the model ID is going to be this. So you can simply copy the username and then the model name. So that's your model id. And for the model base name you need to go to files and versions and then look for the files with dot safe tensor. So that is going to be your model base name. Now keep in mind, if you want to run models in the GPT Q format, you need to have an Nvidia GPU for that. However, if you want to run models on the CPU and still want to use the quantized version, then you want to look for GGML format. I think GGML now also supports GPU inference using the Llama cpp, so you can use those directly as well. Now again, in this case the model ID is going to be this and then the model base name. So here you have a whole bunch of different options because there are different quantization versions that you can use. So for example, this is a 4 bit quantized model. This is 3 bit quantized model, right? And each one of them have different sizes. So depending on your hardware you need to simply copy the name of the quantized version, actually the name of the model with specific quantized version, and that will become your model base name. So once you have both the model name as well as the model base name, then you need to simply come back and populate that in here. So for example, I'm using the block llama27b chat ggml model here and then I'm using the 4 bit quantized version. That's how you are going to set up different models. Now today you are setting this up within the run local GPT file and here's this main main function. Right? But in the near future I think I'm going to remove this and put it somewhere else to make it a little more well organized. Okay, I'm going to go over this load underscore model function just to show you the internal working of how these different types of models are being selected and used. Right. So basically you need to pass the device type. So that's CPU gpu, the for GPU it's CUDA or mps. Then the model ID and the model base name. Let's have a look at this function before running the local GPT file. All right, so as I said, it expects three different inputs. Device type, model ID and the model base name. By default it's set to none. So that's when it expects to run on gpu. Right. So first and foremost it simply checks if the device type is CPU or mps, given that the model base name is not none. Right, so basically if you're using a quantized model now what it does it it simply uses the Llama cpp assuming that you have GGML based model. Okay, so first we check if we are running on an Apple silicon. In that case we simply unload some of the GPU layers. Right. If we are running on a cpu, we don't do that. Now a couple of things that you want to set is the context length. So by by default, the original llama one models, they had a context for window of 2048, but now in the newer ones it's 4096. So I'm going to just change that. And this is how many maximum number of tokens you want to generate. So you can set it to that value as well. Actually this is going to be including your prompt as well as the response. Right, but let's set it to the same value. Okay. And you can update it here if you want, but I'm going to be running it on mps. Okay, so this is the one change that you want to make if you're running the second version of Llama models. All right. Now the second option is if it has the base name provided, but the device type is not CPU or mps, then it means you're using the GPTQ format for quantization. And in that case we are using the auto GPT queue library. Right, so you might want to change some of the settings. For example, if you have multiple GPUs, you can set this up to let's say two of them rather than just the first one, and so on and so forth. Now if the model base name is not provided, then there are two options. First is checks whether you're running it on a GPU or not. Right? So if you're running it on a gpu, then it assumes you're running the full model and it will use the automl for causal lm. If you're running the CPU version, then it will simply use the LLAMA for causal LM for full model. After that we get the corresponding configurations from the pre trained model and we create a pipeline and return the local LLM as a part of the pipeline. Now just to reiterate, if you don't want to know the technical details, all you need to do is set up these two things. Model ID and model base name. Model ID, let's say in this case is the llama 27B chart GGML model, but you can use the 13B or even 70B quantized models. Right. Now let's see how you can actually run this code. So I'm going to clear everything now in order to run the local GPT. So we will type python, then run_local GPT py, then device_type, device_type. And we want to run this on MPS, so let's hit enter. So in this case it's going to go and first Download this model automatically. Okay, so once you run the command, then it will load the model using Llama cpp. So a couple of things to notice in here. First and foremost, where the data is stored. So you have the database path. Then the model that is using. So you notice that it's using the Llama 2.7B chat model and then which device it's running this model on. So we are running it on mps. Okay. And then there are some other parameters associated with Llama cpvm. So in this case, again, we are using the Constitution PDF as our source document. And after that you can actually start chatting with your document. So let's say here's a simple prompt. What is the term limit of the US President? Let's hit enter and we get the answer. So it says the term limit for the US President is two terms or eight years in the office. This limit is set forth in the Article 2, Section 1 of the U.S. you can also ask the model to give you the sources it's using. So if you want to get out of certain chat session, simply type in exit and let's clear this up again. Now, in order to enable the sources, we will need to pass on an extra flag and that is show underscore sources. I believe that's the one. Let me just confirm. Right? Yeah, that's the flag. Okay, so type, hit enter. Again, it will simply load the model. Okay, so same question or similarly, similarly worded question. The answer it came up with this time is maximum of two terms, right? And now here it shows different sections or chunks that it used in order to retrieve that information. Okay, I want to highlight a couple of things in here. Sometimes the responses that you get are not accurate, right? And there are a few reasons for that which you can experiment with. So one of them is the type of chunking process that you are using. So we really need to pay attention to the, for example, the recursive text splitter on how you are splitting your documents. That is very important. Second is in this case we are simply passing on the vanilla prompt that we are using. However, you want to use a prompt template which specifically asks the model to look for answers based on the context that is provided. So I will be covering that in a separate video where I'm going to show you how you can provide a custom prompt template. Because most of these models that we're using, specifically the open source models, each one of them have a different prompt template. So I will be making a dedicated video on how to integrate or use different prompt templates. Based on the model that you choose and that will definitely improve the performance of your information retrieval. Another thing which I'm really excited about is is the Local GPT API as well as the GUI in here. And these were contributions made from the community. So in an upcoming video we will go into a detailed tutorial on how to use the Local GPT API and how to create GUI based on the API. So this is going to be very powerful because this lets you host your Local GPT models anywhere in the cloud or on your premises, your own hardware and then you can make API calls to the model. So that's something I'm really excited about and I'll be covering that in an upcoming video. So watch out for these different videos that are coming up. Another thing I want to mention, a major change is coming to the Local GPT project. There are a few pull requests, specifically this comprehensive refactoring and enhancement of code base which will really change the entire project the way it's set up right now. And I think that will make it a lot more modular as well as powerful. You will be seeing a lot more videos on the Local GPT project in upcoming days, weeks and months. If you would like to contribute to the project, simply look at the current issues that people are having and create pull requests based on those. And I would love to merge. So all the contributions are welcome. And if you like the project, make sure you start it on GitHub. If you're already using local local GPT for your own projects or are thinking about it, I would love to chat. So don't hesitate to reach out. I want to see what people are building on top of this. I already had some really amazing conversations and some great applications that some of the community is building on it. If you run into any issues, you can create an issue in here. We have a very active Discord server where people are sharing their problems, findings and what they are building on top of Local GPT project. So consider joining us on the Discord server. I hope this video was helpful and you found it useful. If you did, consider liking the video and subscribing to the channel. If you're looking for a trusted advisor on your projects, you can reach out to me. Details are in the description of the video. Thanks for watching and see you in the next one.