ChatBOT with Memory for Your Documents - OpenAI & LocalGPT

How to create chatbots for your own documents that have memory. That's the topic that we are going to be covering today. First, we will build a chatbot that you can use to talk to your own documents. Then we will add a nice graphical user interface for you to interact through it. Now, in this start, we're going to be using Chat GPT as our LLM and embedding model. Now, if you have been following my channel, you probably know what that I have my own local GPT project. This project lets you chat with your documents on your local machine and no data leaves your device. So in the next step, I'm going to show you how you can replace OpenAI's models with the local GPT models. For this video, we're going to be borrowing the main code from this new course, LangChain Chat with your data by DeepLearning AI. I recommend everybody to check out this course because it's a really great beginner course. Let's quickly walk through the code. I made some adjustments to it to make it functional. So first and foremost we are simply installing all the required packages using PIP within Jupyter Notebook. Next, we are importing all the required packages and then after that we're importing the operating system module. You need to define your API key as an environment variable. It's actually better to put the API key within a env file and then load it from there. But I'm simply providing it here. And then we're simply defining the model that we want to use. So for this first registration we're going to be using GPT 3.5 Turbo. But later on in the video I will show you how to replace this with an open source model. Okay, let's quickly talk about how you can add memory to your LLM or a conversational chain. So LangChain has different types of memory. One is conversational buffer memory, and this is how you define it. If you're looking for a more detailed treatment of the subject, I would recommend you to watch this video. But basically what you do is you define a memory object, right? Then you define your embeddings as well as the vector store that you want to use. Now after that you need to add memory into your normal conversational chain or any other type of chain that you are working with. Now this is the simplest way of adding memory to your chain or LLM. I have a dedicated video on different types of memories that are available in LangChain. So I would suggest to watch that after this video. Now let's See how you can use these concepts to put together a graphical user interface around it. Here they have defined a function called Load DB that receives three inputs. One is the file name, so this is basically a PDF file. The second one is the type of chain that you want to use for information retrieval. And then the third one is K, which is the number of chunks that you want to use during information retrieval. If you're not familiar with these concepts, I would actually recommend you to watch this specific video in which I go into a lot more detail. Now, within the function, let's see what is happening. First, we load our PDF file and we use a document loader. So for this specific case, whereas you're using PYPDF loader. Okay, so next we define a text splitter, which basically divides a document into smaller chunks or smaller segments. And each segment is supposed to have a size of thousand tokens with an overlap of 150 tokens between the consecutive chunks or segments. And then we use this text splitter to split our documents into those chunks. Right? These should be very familiar concepts. In this case, we're using the OpenAI's embeddings, right? And we are using Vector DB. Now, this specific vector database is DOC array in memory search. So this vector store is basically in memory. You do not persist anything. This is going to be especially helpful if you give users the ability to upload different documents while interacting with your app. Next, we are defining a retriever which accepts the number of chunks that we are going to be looking at in order to extract certain information. And after that we are simply defining our conversational retrieval chain. Now, the chain accepts the LLM as an input. In this specific case, we are using the GPT 3.5 Turbo, then the type of chain that you want to use. So we're going to be using stuff. Then the retriever basically is our vector store that we defined, right? And we do want to return sources as well as return the generated questions. Now, when we look at the generated questions, they're going to be slightly different than your actual prompts because there is a system message on top of your prompt. Now, one thing you will notice that in this specific case, they have not added memory into the conversational chain. They're doing it outside this function. They're using the rest of the code to create the graphical user interface using the panel package. So I'm not going to go into a lot of details for the rest of the code. However, I do want to highlight this specific portion which relates to memory. So within the code they are keeping track of the chat history. And then the chat history is extended with the query and the results that you get for your subsequent prompts and that is being passed on to you, your Q and A chain. Now, in order to run the dashboard, you will need to execute this specific line at the end. Okay, so once you do that. Now once you do that, you're going to see a graphical user interface like this, which is running at the local host with 52743 Port. All right, so let me first quickly walk you through different components. The first one is conversation, which simply keeps track of the user and assistant conversation. Then there is a database which will actually show you the db. Then there's a chat history and configuration. So right now we are working with the US Constitution. However, you can upload your own files if you want. Now if you're curious, so here is where we are defining the default file. So I'm using Source Documents folder and within that we have the Constitution PDF file. But if you want you can basically choose another file and then click on Load db. All right, so let's ask it a simple question. What is the term limit of the President? And then comes up with the term limit of the President in two's term as stated in 22nd Amendment of the US Constitution. Now we can also ask subsequent questions. So I said, what is the age restriction right now since it has memory, so it will know that we are talking about the President. That's why it says the age restriction of the President is that they must be at least 35 years old. Okay, so you can basically upload your own documents and start chatting with them. Now let's look at the database. So this shows four different chunks that were used for the query. Let's look at the end. So it's found some information on page 5, page 13, page 2 and I guess page 17. And so those different chunks that you see are subdocuments that were used for generating the response. They also have a chat history. So you can see the first prompt and then the corresponding response. And here's the second prompt, the corresponding response. And now as I said, the last tab, you can actually upload another document. So let's try to upload something else. Now this time I am going to upload the Orca paper. So I chose the file and then click on Upload DB and it should actually replace this. Yeah, so this is the Orca paper now and if you go to the conversation, we can simply ask a question. Let's say who are the authors and, and let's see what response we get. So here is the response front GPT 3.5 turbo. Now, I wish they would put the more recent interactions here on top, but for some reason they're being put at the bottom. But you can easily change those. Okay, so this is a very nice little GUI that you can use for creating your own chatbots. And as I said before, it's not by code, it's coming from DeepLearning AI. Now let's look at the second part where we're going to replace OpenAI LLM as well as embeddings with our local GPT embeddings in LLM. Okay, so in this case I went ahead and cloned the repo of the Local GPT project if you're not familiar with Local GPT. So I would recommend you watch this video. Now, in summary, the Local GPD is a project which enables you to chat with your documents locally and privately, and nothing leaves your system. So if you have any type of sensitive data or you are concerned about your privacy, I would recommend to check that project out. Okay, so what I have done here is I created this chatbot_local GPT IPython notebook. And this is basically exactly the same code that I showed you, just with a couple of changes. Now you need to make sure that you have installed Local GPT on your machine and after that you can run this notebook. So the first change that we did was we actually importing load underscore model from run underscore local GPT. Now this is basically the function that we are loading from our local GPT file. Now this needs three inputs. One is the device type, whether you want to run it on a CPU GPU or if you have an M1 or M2 MacBook. So you will pass on MPS then the model that you want to run. And the last thing is the model base name. Now we're going to look at these in a little bit and I'll explain some awesome functionalities that Local GPT has. After that, the next change that we are making, instead of OpenAI's embeddings, I'm using hugging face embeddings. You can replace these with the instructor embeddings as well, but they will need to be run on GPU and I think it's around 4 gigabytes file. So if you don't want a big embedding files you, you can just use hugging face embeddings instead. Now let's go to the last change that we have made. So if you remember, we were passing on an LLM, so that is replaced by the load underscore model function. The model ID that I'm actually using the model, this is the Orca Mini V2.7B GGML model. If you notice, you can now work with GGML models within local GPT. Now I'm running this on M2 max. That's why the device type is going to be MPS and then the model base name is basically the model, the quantized version that I want to use. So in this specific case I'm using the four bit quantized model. Okay, I think it's a good point to talk about a couple of changes that have been made to the Local GPT project. I'll be making a detailed video on all the new changes, but just wanted to highlight some of the features here. I released Local GPT project. It only supported the hugging face models. However, since it's released there have been quite a few updates thanks to the open source community. So now it has support for GPTQ format files or models. But keep in mind, if you're running GPT cube based models, then you need to have an Nvidia gpu. Now if you do not have an Nvidia gpu, you can actually run GGML format models. And thanks to the contributors, we're now able to run almost all different types of models. Support for MPT models is coming soon. As I said, I will be making a separate video on all the updates to local GPT. Okay, so we created our LLM chain and the rest of the process is very similar to what we did before. Simply run the rest of the code cells. Now when you run these, you will actually notice something like this because I'm using LLAMA cpp. So this is basically different outputs that LLAMA CPP is going to show you. Okay, so when we run the code, you are going to be presented with exactly the same graphical user interface, but in this case at the back end we are using open source models. Okay, so I asked this model the same question again and it came up with a very detailed answer. Now the reason is that the model that we have chosen, The Arca mini v2.7V model, the output will really depend on the type of system prompt that you provide. In this specific example, I haven't provided any system message or system prompt. But when you are working with these open source models, you need to make sure to pay close attention to both the prompt template as well as the corresponding system message that you want to provide. We are going to look at that in a future video. Okay, so in this video I wanted to show you a few things. First, how to integrate memory into your model. Either you can use the memory buffer object or you can add it as a part of context, the way it's done here. Second, the graphical user interface. And the third one was how to replace OpenAI's models with open source local GPT based models. I hope you found this video useful. Consider liking the video and subscribing to the channel for similar content. We have a very active discord server where folks are helping each other out. If you have any questions or comments, please come join us. You will find a lot of value there. If you're working on LLM based projects and want to discuss those and check out my calendar in the description of the video. As always, thanks for watching and see you in the next one.