Llama-3 🦙 with LocalGPT: Chat with YOUR Documents in Private

Let me show you how to get started with llama 3 within local GPT. Local GPT is my own project that lets you chat with your documents locally in a secure environment. Everything is 100% private and nothing leaves your system. The project already has more than 19,000 stars on GitHub. So let's get started. First we need to clone the repo. So click on this code button, then copy the URL to Clipboard. Next we need to clone this repo. So I'm going to open a terminal and then type git clone and then provide the URL. And after that I usually like to have dedicated folders, so I'm going to just call this local gptnama so that I know that I'm working with the llama 3 model. And this will download all the files in the folder. Next we need to simply change directory. So we're going to move to the directory that we just created, and for that we type CD and then the name of the directory that we created. If you type ls, this will list all the files that are available within this folder. Okay, so let's clear this up. Okay, next we need to create a virtual environment. So for that we're going to be using conda. So the type the command is going to be conduct create dash n Then the name of the virtual environment. So let's call it Local 3 just to indicate that we are working with Llama 3 and you also want to provide the Python version that you want to work with. I like to work with Python 3.10. So this is going to ask me whether I want to install all the packages. Say yes and this will start the installation process. Okay, the virtual environment is created. Now we need to activate that virtual environment. So for this I'm going to just copy that command. Now if you see here, we can see that we are now in the Local 3 virtual environment. Alright, so let's type ls again. This will give us a list of all the packages. Now I want to install this requirements txt, because it has all the packages that we need. So we're going to say pip, install dash r and then requirements txt. This will download all the packages that were needed in order to run local GPT, except one package, and that is a Llama cpp. These are the Python binding of Llama cpp. So depending on whether you are using Nvidia GPU or you are on Apple Silicon, you want to use two different commands. So I am running this on an M2 max, which is based on Apple Silicon. So I'm going to copy this section. Okay, so you can see all the installation is complete. So let's just clear this and I'm going to paste the command to install Llama CPP and this will download llama CPP as well. Okay, so if everything goes well, you are going to have Llama CPP installed on your local machine. Okay, so next I open that folder within Visual Studio code we need to open a new terminals. Now we first need to activate the virtual environment that we just created. So we're going to say conda activate and then the virtual environment that I created, I called it Local 3. So you can see that we are in the new virtual environment. Okay, if you are new to local GPT. So the way you change your model is you need to go to constants PY file and within here you're going to see the model underscore ID and model base name. Now if you're using an unquantized model, you just need to provide the model id. And the model ID is basically the address of the hugging face repo in this case. So for example, if you go to hugging face for Lama3.8 billion model, this is the repo ID. So you just copy that and you can keep the base name none in case if it's an unquantized model, if you want to use a quantized model. So for example, if you want to use this quantized model from Quant Factory, then you will also want to provide the ggf file name of specific quantization level that you want to use. Okay, so in my case I'm going to be using the unquantized model from meta. Now if you're using the meta version of llama 3, you will also need to log into your Hugging face Hub account because it's a gated model. So when you go to this page you're going to see that it's going to ask you whether you accept all the terms and conditions because it's a gated model. I have already accepted the terms and conditions. But in order to use this model, I also need to log in to my Hugging Face account on the terminal where I'm using this model. So let me show you how to do that. So for that we're going to be using the Hugging face cli. So this is the command Hugging face CLI login. Then it's going to ask us for the access token. So we need to provide our access token. And in order to get your access token, just go to your hugging Face account, click on Settings. Then you might have to create a new access token. You can see that I have a bunch of access tokens already available here. So I'm just going to copy this one and then I'm going to come back here. Right click hit paste. It's not going to show the token, but it already pasted the token. Just hit Enter. Then it's asking me to add token as get credentials. So just type no hit Enter and if everything goes well, you're going to be logged in to your hugging face account. Okay, so we are all set. Now we need to start ingesting our files. So Local GPT comes with this paper as a example document. Now, in order to ingest files you will need to type in Python ingest py. If you are running this on a system that has an Nvidia gpu, it's going to use CUDA by default. But you can also provide the device type on which you want to run this by adding the flag flag device type. I'm going to be running this on mps, so just hit Enter and this will start the indigestion process. Now, if you run this command for the first time, it will have to download the embedding model that Local GPT uses and you can actually see which embedding models Local GPT uses by go to constant py if you go up. By default we use the Instructor Large model, which is a pretty good embedding model. There are a whole bunch of other options available. You can select one of them if you want. Even if you are looking for multilingual embedding models, we have also listed some of them. Or you can just go to the Hugging face leaderboard offlink models and select one of those. Okay, so the ingestion is complete. It split the document into 193 chunks. So I'm going to just type clear. Okay, Next, if you want to start chatting with the document, then you can just directly go and run Python run_local GPT.PI. again, you can provide device type if you want. Or by default it will first try to use Cuda. If it doesn't find anything, then it will try to use MPS and if not then it will try to use cpu. So we're going to just type that and it will load the model. Right now it's downloading the model because we are running it for the first time. But while this is downloading, let me show you a couple of other things. So if you go to the prompt_ template utils file. Now, I have an added prompt template for Lama 3, so you can actually provide which model you're using. And based on that we're going to select Lama, the prompt template. So for llama 2 or llama 1 I was using this prompt template. But for llama 3 they have changed the prompt template. And I have observed that if you don't really provide the proper prompt template, you're going to run into a lot of issues. So we take care of that for you. Right. Similarly, if you're using Mistral, there is a specific prompt template for that. And if you don't provide any prompt template, then this is the prompt template it's going to use. So you actually need to provide what prompt template to use. There are three options which you can actually find in this local GPT or actually run local GPT file. So let me show you where to see those. Okay, so here are the list of acceptable prompt or model types. So we have Llama 3, which is the new edition, then Llama, Mistral and Nenlama. So this is the default prompt template that is going to be used if you don't provide any of these three. Now another thing that I wanted to mention was that at the moment, based on my experimentations, it seems like the GGUF quantized versions of the models don't really follow the prompt template. I don't know what's actually going on over there. I think there was some issue with the end of the sequence token. So it seems like some of them haven't implemented it correctly. So that's why I am using the unquantized version provided by rather than some of the quantized version that are created by the community. So the model download is complete. Just to confirm, we're using the Instructor Large as our embedding model and it downloaded the Meta Llama 3.8Billion instruct version. And we are running this on MPS. Now we're going to be using ORCA paper as a knowledge base. So let's ask a few questions which are related to this specific paper. Okay, so there's a specific section on instruction tuning. Let's ask the model what instruction tuning is and let's see what type of responses we get. Alright, so my question is going to be what is instruction tuning? And let's see what response local GPT creates. Okay, so here is the response according to the provided context. Instruction tuning is a technique that allows pre trained language models to learn from the input, which is natural language description of task and response pairs. This means that the model learns to perform specific tasks or functions by being trained on examples of inputs and corresponding desired outputs. This seems to be correct. Alright, so let's ask a more specific question. In this case I'm asking How does Orca's performance compare to ChatGPT? There was a small comparison that is provided in the paper. According to the response, ORCA performs at PAR with text DAVINCI003 on an aggregator across all tasks and retains 88% of ChatGPT quality. Now here is the information that it's getting from so the responses seems to be accurate okay, so this was a quick video on how to get started with llama 3 within local GPT. As for the local GPT code base, I'm going to be creating a lot more content in near future. I'm actually rewriting most of the code base and we're going to add a lot more advanced rack techniques such as query expansion, context expansion, re ranking. Those are not the techniques that are currently available within local GPT, but a major update to that is coming. I'm also putting together a course on advanced techniques for rac, so if you're interested in that, make sure to sign up. Link is going to be in the video description. Now in a subsequent video I'll show you how to use the groq version of llama 3 which is available through the Grok API within local GPT. So if you are interested in something like that, make sure to subscribe to the channel so you don't miss that video. I hope you found this video useful. Thanks for watching and as always, see you in the next one.