Mistral-7B with LocalGPT: Chat with YOUR Documents

Can you use Mistral 7B with local GPT? Let's find out. The quick answer is yes and I'll show you how. If you're new here, Local GPT is my own project that lets you chat with your documents securely without compromising your privacy. The beautiful part is that everything is running locally, so you can be assured that no data ever leaves your computer. If you want to learn more about it, I have a series of videos which I linked here in the GitHub repo, so I'll put a link to it in the description. Okay, so by default local GPT uses the Llama 27 billion model in the GGUF format. Now if you have an Nvidia GPU on your computer, I would highly recommend to use the GPTQ format file instead of GGUF format in this video. I'll just show you how to replace this model with the new Mistral 7 billion model. Okay, so in order to download the model we are going to be using a quantized version and for that we are going to go to the Blokes Hugging face repository. Alright, so if you go down you will see that the blog has converted Mistral 7B instruct models into GPTQ format as well as GGUF format. So, so I'm running this on NM2 with Apple Silicon, so that's why I'm going to be using the GGUF format. But if you have an Nvidia gpu, I would highly recommend to simply use the GPTQ format. Now when we go in here we need two things. First is the repo id. So we can select this, this is the model id. Okay, so I go back to my visual code studio here I'm going to be updating the model ID in the constants PY file and let's simply update the model name. Next we need to select a specific version of the model itself. So here you will see a list of different quantization of the model. Now depending on your hardware, select the quantization level that is supported by your system. So in this case I'm going to be using the 8bit quantized model, but you can select either 4bit, 3bit or 2bit. So simply copy this. Let me just copy this. Now here I'm back to my visual code studio and we will replace the model base name. So this is going to be the new model base name. Next we simply need to save the changes. Now if you don't know what I'm doing right now, I would highly recommend to watch the Previous videos. That will give you a lot of context. I will put a link to a playlist on all the videos on local GPT. For testing purposes, we are going to be using the Orca paper. Okay, so the first step is to ingest this file. So I have this Source documents folder and in here I pasted the Orca paper. Now, in order to ingest the file or create a vector database, so we are going to run the following command. So the command is going to be python ingest py. Now if you are running this on an M1 or M2, I would highly recommend to pass MPS as device type. If you're running this on a cpu, you can pass CPU as the device type. For Nvidia gpu, you don't have to do anything. Okay, so the ingestion is complete and we have a total of 193 chunks of text. If you are on MPS, just simply ignore this warning. Now, before running the model and start asking question on our Orca paper, let me highlight a couple of changes that I have made in here. Now, to add support for the Mistral 7B model, I added a new command line option that now you can use. So it's called model type. The default value is llama, but it will also support Mistral and non llama. So basically this will define the prompt template that the model is going to use. Now here if you go to the prompt template utilities file. So I added another option specifically for Mistral 7B model. So we're going to be using this prompt template for Mistral 7B and that's the one that we are using in here. So we have the start and end token basically added to the prompt. Now in the original Mistral prompt there's no system prompt, but I added because this will limit the model to answer questions related to the documents that we have provided. So in this specific applications, we don't want the model to generate answers from its own knowledge, but rather simply stick to the documents that we are providing. So these are the couple of additional changes in local GPT when it comes to the Mistral models. So now let me show you how to run the model. Okay, so the these were the changes related to Mistral 7B in local GPT. Now in order to start a conversation with your documents using this new model. So we are going to run the local GPT file again since I'm running this on mps, so I'll provide NPS as an option. Now by default the model type is llama, but if you want to use the prompt template of the Mistral model. So we are going to simply pass Mistral as an option. Okay, so this is using LLAMA cpp and if we check here. So we are indeed using the Mistral 7B instruct model. Okay, so here's the question that I asked on the ORCA paper. How does ORCA compare to Chat GPT on the BBH benchmark? And it actually gives me a pretty decent answer. Okay, so in fact it's actually generating the answer based on this specific chunk. So if you see here, it says that orca exceeds ChatGPT by 11% and 5.8% on disambiguation QA and snarks respectively. Right, so this is the, I guess the chunk that specifically is being used by the embedding model and then returned to the Mistral 7B to generate this answer. Now it also added some some extra information and that's actually coming from the following I guess paragraph. So it seems like the embedding model put these two together in a single chunk and the model is using this information as well to give us an answer. Now the great thing is that the answers that we are getting are accurate. Okay, so this was a quick video on how to use Mistral 7B with local GPT from my initial experiments. I'm really happy with the results that I'm getting. Keep in mind that the results that you get are really dependent on the type of chunks that are returned by the embedding model. That's why selection of the embedding model as well as how you decide to pre process your model, including how do you split them into different chunks embedded, is going to play a crucial role in the performance of your Retriever augmented generation system. I have a few videos on these topics if you want to learn more. Links are going to be in the description of the video. Overall, it's a great model and I would recommend everybody to check it out. Now, if you're using local GPT in your own projects and are looking for guidance, so I do offer consulting services. Check out the description of the video for more details. As always, thanks for watching and see you in the next one.