LocalGPT & Llama-2: Adding Chat History & Custom Prompt Templates

Today we are going to explore how to add memory to Local GPT so that it can remember the conversation that you're having with your own documents. If you're not familiar with Local GPT that lets you chat with your own documents using any of the open source model that are available on Hugging Face, no data leaves your device and everything is 100% private to your local device. Thanks to all the amazing contributors, we have made it to the trending repositories on GitHub for the second time. So I would like to thank everybody from the bottom of my heart. For all the contributors and the people who are using this today, I would like to highlight yet another amazing feature that was actually added by a community member. So Jeffrey had this PR adding conversation history. This enables you to have conversation with your documents rather than just simply asking it a single question without any context. So with the help of this update, the model will now remember the conversation within a session and I think it adds a lot to the usefulness and usability of the Local GPT project. If you're not familiar with the Local GPT project, don't worry. I'm going to give you a very quick overview of the project in this video, but for more detail coverage I would recommend you to watch previous two videos. So, first and foremost we need to clone the repo. So simply go ahead, copy the link. After that, open a new Visual Code Studio session and create a new virtual environment. So let's say if you have CONDA installed on your local machine, you can use Condo Create N and provide the name of virtual environment. So for example, I am going to call this Local GPT History and then you need to define the Python version that you want to use. I usually like to use Python 3.10.0. I already have a virtual environment, but you simply need to hit enter and that will create a new virtual environment for you. And then you want to clone the repo, so you can use this git clone command and followed by the link to the repo. So again, I already cloned this repo, so I'm not going to do this step. And once you clone the repo, then you want to move to that folder. So for that you will need to use the CD and then whatever the folder name is. So in this case it's going to be Local GPT, right? So that will take you inside that folder. Here I actually created a folder called Local gpthistory, but if you type ls, this will show you everything that is in that folder, right? Okay, after Creating the virtual environment and moving to that corresponding folder, you want to activate your virtual environment. So in this case I have a virtual environment called Local GPT. So Local GPT underscore history, right? And for that I'm using this conda, activate and then the virtual environment name. So you see that I have activated my virtual environment. Now, in order to install the required packages, you're going to say pip install R and then requirements txt. So once you execute this, it will install all the required packages. But I have already done this, so I'm not going to do this again. So let me just clear this. Okay, so as I said, we are going to be talking about two different things in this video. The first one is addition of memory to your model. But before doing that we first need to ingest our PDF files. For that I usually use the US Constitution, but for this specific video I'm actually using this Llama 2 paper, the open foundation and fine tuned chart models. So I wanted to see if we can extract information from this paper. The paper itself is around 77 pages long. So what I did was I actually took that paper and copied that into this Source Documents folder. So you see there is this llama 2 PDF as well as the constitution PDF, right? So now first we need to ingest all the documents. So for that we're going to type in Python ingest PY and then if you're running this on MPS or on a cpu, then you need to provide a dash dash device type. So I'm going to type in CPU, I don't have a GPU, I'm running this on an M2. So then simply click Enter and it will start ingesting the files again. If you want to go into the details of the code base, I would actually recommend you to watch the previous two videos in which I discuss each and every function and that is within the code base in this specific case. For embeddings we are using the Instructor large model. And keep in mind that the Instructor embedding models model actually runs on gpu. So you need to account for that when you are running this on your local machine. But you can replace this with this all mini LM L6v2 version as well that runs directly on your CPU. So you don't need a GPU for that. But based on my experiments I have found that the Instructor embeddings actually works best. So the ingestion process is complete. Now let me show you how you can ask questions and then I'm going to show you the Code changes, which adds memory to this chatbot. Okay, so let's ask some question from the document. So in order to do that, we're going to run this command, Run local GPT PY and then again, depending on the device type you have, you will need to provide that. So I'm going to be using MPS again, so click enter. Now, a couple of things to notice here. The first one is that it shows you that it's running on MPS and It is using llama 213B model and we are using the GGML version. The second thing that you want to notice is what embedding model is this using? So it's using the instructor embedding the large version. Now let's see if we can have a conversation with the model. So let's start with a simple question. What is the length of the context window of llama2? Okay, so you came up with the answer based on the information provided in the passage. The length of the context window of Lemma 2 is 4096 tokens, which is correct. You will notice that I did not ask it to show me the sources of this information, but you can simply do that by enabling this show sources flag when you are running the Run local GPT command. Let's see, we can ask a follow up question and see if it will remember the previous conversation. So my follow up question is what was that for Llama one? So hopefully based on our previous prompt, it will remember that we are talking about the context window. Okay, so here's the answer that it came up with based on the information provided in the passage. The length of the context window of llama1 is not explicitly mentioned. However, we can infer from the text that the context window is at least 77 token long. As this is the length of the model card of llama 2. So it didn't really retrieve the proper information. We need to look at which chunks it's using. However, if you look at actually remembered that we are talking about the context window. So that means that it has memory of the previous conversation that we had and that's exactly what we want. Okay, so now let me show you how exactly is this implemented inside the code. Okay, so here's the implementation. Now we are using conversation buffer memory object and it's receiving question. That's basically the prompt from the user and then the previous history of the conversation. I will show you where this is exactly coming from. But we are creating this memory object and we're using this as a part of our retrieval QA chain. Now keep in mind, in this implementation we are not adding the information sources as part of the memory, but simply the response. Now, there are other options that you can use. Instead of just the conversation memory buffer, I have a different video which covers all different types of memories available in LangChain. So I'm going to put a link to that video in the description of the video. So if you are interested in this topic and want to explore other options, I would recommend watching that video. The second feature that we have added is the ability to use custom prompt template. And we're doing that using the prompt template class from LangChain. Now this is extremely helpful when you are working with open source large language models. So for example, here's the prompt template that the Vicuna expects for you to use. So you have kind of a system message, then the user input and assistant, right? But if you look at llama2, so here is the prompt template that llama2 uses. So having this ability to define your custom prompt can really give you a boost in performance. Okay, so I have a very detailed video on how to work with prompt templates, but here I'm going to give you a very quick overview. So if you're using prompt template class, you can provide multiple input variables, right? So in this case we are providing the question, which is the actual query from the user, then the context, which is the context extracted by the embedding model. So this is basically the chunk of documents that they're being provided. And the last one is the history of the previous conversation, right? And based on this, we are defining a template which the prompt is supposed to follow. So it's going to expect these three inputs and combine everything using this F string, which is defined in here. This is kind of the system message. Use the following pieces of context to answer the question at the end. If you do not know the answer, just say that you do not know. Don't try to make up an answer, right? And then we are providing it, the context, the history, the current question, and then we expect the model to generate an answer. So this is by default a prompt template that we are using. It will work with almost all the models. However, if you want to implement a prompt template specific to a model, let me show you how you would do that. So for this we're going to look at an example. So we're going to be using the prompt template from llama2chat. So in this case it expects a system message, which is your helpful, respectful and honest assistant. And then it goes on to Some of the details. You can replace this system message with your own message and after that it expects the user input, right? So let me show you how you can use this prompt for your information retrieval system. So I have modified my template based on the prompt template provided for llama2 models. So in this case my system message is again the same message I was using above, right? And then we are providing the context history and the corresponding question that we get from the user. Now I think you can move around this. So for example, you can put this at the top right before the context. It's up to you and just experiment with different variations and see which one gives you the best answer. But this is a simple example of how you would set up this template based on the prompt template for a specific model that you are actually using. Now, all of this is implemented inside the main function in the RunLocal GPT file. I'm going to be making some substantial changes, so we'll be moving out most of this stuff from this function to make it more readable in near future. I'm also working on a video on how to use the local GPT API. So that is work in progress. I'm adding some more components to it. So hopefully early next week we are going to have that video out as well. So to everyone who was waiting for that video, sorry for the delay, but I'm making some changes and hopefully it will make it much better. If you like the local GPT project, make sure you give it a star on GitHub. And if you would like to support my work both on Local GPT as well as the content that I'm creating in here, check out my Patreon. Details are in the description of the video. We also have a very active and helpful discord community, so if you run into any issues, or if you just want to learn more about generative AI in general, come join us. And if you're looking for consulting services on your own projects and let's have a chat, just check out the description of the video for how to reach out. Hope you found this video useful. Thanks for watching and see you in the next one.