LocalGPT API: Serve Multiple Users At the Same time
LARRUZ shows how to set up LocalGPT API so multiple users can query the same knowledge base at once. The tutorial covers ingestion, inference, virtual environment setup, and running the server to handle concurrent requests.
Chapters
Today we are going to look at how to enable multiple clients so that they can talk to a common knowledge base using the Local GPT API. Local GPT is my own project that lets you chat with your documents locally in a secure environment. It has over 18,000 stars on GitHub and you can actually build business applications on top of this. One common use case that I have seen in business setting is to have a common knowledge base and then you enable multiple clients to talk to that common knowledge base simultaneously. And it's possible through the Local GPT API. So in this video I'll show you how to set this up and have multiple clients communicating with this one server simultaneously. There are two different parts of the process. The first one is ingestion. So you provide your documents. Let's assume if you are working in a business setting, these could be HR policies or financial documents. Those documents are chunked and split. Then we create vector embeddings based on those chunks and those are stored in a vector store which will act as a knowledge base. The second part is going to be the inference part in which we are going to be serving multiple clients through an API. In this case, Local GPT uses the Flask API server. The Local GPT server will implement a simple queuing mechanism. So for example, if client 1, 2 and 3 ask different questions simultaneously, it will serve the first client, then the second client and then the third client. And this is going to be assuming that that there is a single GPU that is used for inference. If you have a multi GPU system, then you can serve different instances of local GPT on different GPUs and route client requests through them. Okay, so let me first walk you through a step by step process of how to set up local GPT. If this is the first time you are seeing local GPT. So first we need to clone the repo here. Just click on this green button, click on this link. This will copy the link. There is also a pre configured virtual machine thanks to the folks at Mast Compute that you can use to run local GPT. And if you decide to use this virtual machine, make sure to use the code prompt engineering to get 50% off. So I'm going to be using this virtual machine in the video. You can go here. You will need to create an account. Once you create an account, then click on the creator in category and here just select prompt engineering. And now you can just provide the prompt engineering code that will give you half of this price if you are using this virtual machine. This is the interface you will see when you log in for the first time. So there is a folder called Local GPT. This is a cloned version of the repo. And there is also a welcome document which will walk you through a step by step process of how to use this virtual machine. But let's assume, if you are running this on a local machine, let me walk you through a step by step process of how to set up Local GPT. So first go to your terminal. First I'm going to change directory to desktop because that's where I want to store a copy of the clone repo. Next we're going to copy the GitHub repo address here that we copied before. And I wanted to create another folder called Local GPT API where the repo is going to be cloned. So now you can see that we have another folder called Local GPT API. If we open this folder, it's going to have all the latest files. Next we're going to create a new virtual environment. So we're going to use the conda, Create dash and Local GPT. That's what the virtual environment is going to be called. And then we'll provide the Python version that we want to use. In this case I want to use 3.10. This virtual machine comes with a virtual environment called Local GPT, so I'm not going to create another one. Now, if you're using this pre configured virtual machine, you need to do one additional step. So you want to CD to the Local GPT folder and and after that you want to pull the latest changes. So for that we are going to use the git pull command. I have already pulled the latest changes, but in your case you might need to do that on this virtual machine. There is also an instance of Visual Studio code that comes with pre configured Local GPT. First and foremost you want to install all the requirements and this is a necessary step because we keep making changes to the GitHub repo. So you want to make sure that all the packages that you're using are up to date. I have already installed everything, but you're going to be using the PIP install R requirements TXT file. Another small change that I made is this. So I'm using the llama 27 billion unquantized version of the model. If you want to use the quantized version on Nvidia gpu, you can. You want to make sure to use the GPTQ format. This will be more optimized for Nvidia GPUs compared to the GGUF format. Okay, so we are all set. Now let me walk you through the ingestion part first, and then we will look at how to serve multiple clients using the Local GPT API server. Okay, so you need to copy your files to the source document folder. In this case, Local GPT comes with an example paper. This is the original Orca paper, but you can just replace this with your own documents. So first we need to ingest this paper in order to create a vector store or knowledge base. And in order to do that, we're going to be using the Python ingest PY command. You can also provide the device type. By default it's going to be using Cuda, but if you're running this on Apple Silicon, then you want to provide NPS. Okay, so it divided our paper into 194 different chunks of text and it created this DB folder, which has the vector DB. This is going to act as our knowledge base. Now we can start the API server so that multiple clients can talk to this Vector DB or knowledge base simultaneously. And for that we're going to be using the python run local GPT API.PI file. So just hit enter, this will start loading the LLM and start serving. Okay, so we have our server up and running at localhost port5110. Now, in order to use this API server, local GPT comes with an example UI that you can use. And for that we will need to move to this Local GPT UI folder. So type in CD Local GPT ui and if you type ls, there is a Local GPT UI PY file. Now, the great thing about local GPT is that you can create your own UIs. You're not actually limited to this. All you need to look at is the implementation of Local GPT API and how to call this. I have a separate video in which I go into a lot more details on the implementation part. I'll put a link to that video. In order to run the ui, we're going to type Python Local GPT UI PY and this will start another Flask application. And this time it's running on Port 5111. So we will, we are going to just copy this. Okay, so here we have three different instances of the UI running. You can essentially put this on a cloud server, for example, on AWS EC2 instance, and could be making external call to the API from different remote machines. This is just to simulate that three devices are running and they're going to be accessing the API simultaneously. Okay, so in the first case we're going to say what is instruction tuning? In the second case we're going to say how does ORCA performs compared to ChatGPT? And in the third case, let's ask a completely irrelevant question, which is what is the meaning of life? We are going to click search on them one by one. So essentially we are making three different calls to the API server. Now if you look here, we actually received all three of them. And the way it's going to work is it is going to respond to one of them, wait for the response to be generated, then it will process the second prompt and then the third prompt. And here are the responses that you can see from the API. So it received two of the prompts so far? It will. Yeah, I think it also received the third prompt. So let's look at the responses. Okay, so here we have the responses from the model. So the first one is what is instruction tuning? And this is the response based on the provided context. The second prompt was how does ORCA performs compared to Chat GPT? Here is the response. It looks looked at some numbers from the paper. The third one was what is the meaning of life? And it also gave us a response for that. Although you can play around with the system prompt that we are using so that you can limit the responses only to the provided contacts, rather than creating responses for irrelevant questions. Okay, so this was a quick tutorial on how to serve multiple clients simultaneously using the local GPT API server. Now there are definitely more elegant ways of doing this. You could add a load balancer and queue the requests. If you want to contribute to the local GPT project, check out the GitHub repo. Also check out our really helpful Discord community. I'm also offering consulting and advising services for products and startups if you are interested in that. Check out the video description for more details. There are a lot of cool features of local GPT like this that hasn't been highlighted. I'll be creating a lot more content on highlighting some of the features. So if you are interested in that, make sure that you subscribe to the channel for upcoming videos. Thanks for watching and as always, see you in the next one.
