Ultra Fast LocalGPT inference with Groq API

Somebody recently asked me on the Discord server whether it's possible to use Groq API with Local GPT. So in this video I'll show you how to do that. If you're not familiar with Local gbt, it's my own project that lets you chat with your documents locally on your own machine using GPT models. But in this video I'm going to just show you how to integrate the Groq API so that you can get much faster responses when you're chatting with your documents. A quick side note about Local GPT, One of the biggest feature requests that I have received so far is to have a nice looking ui. And here's a quick sneak peek of what is coming to Local GPT. I think you all will really like what we are cooking. Let me quickly walk you through the step by step process of how to integrate the Groq API within Local GPT. You can run Local GPT on your own machine and for this example you don't need a powerful GPU because we're going to be using the Groq API. But just in case, if you're looking for a pre configured virtual machine, so here is a link on the repo, in category select Creator and in image create prompt engineering and you can use the code prompt engineering to get 50% off of this price that you see here. So I'm going to be using this virtual machine that comes pre configured with Local GPT. But if you're running this locally, you want to clone the repo first. So here we have a virtual environment corner called Local GPT to first activate that virtual environment. Next you want to pull all the latest changes from Local GPT repo because this is not up to date. So if you pull the changes, if there are any new updates that are going to be pulled here, this one is already up to date. Okay, after that you want to install the requirements file. Now we're going to make one change to the requirements file and that is going to be we're going to go to requirements text and we will remove this pinned version of LangChain because you need the latest version of LangChain in order to run the Groq API. So after doing that, let's just install it. I have already installed everything, that's why it's not going to install any packages for me. We will also need to install one extra package and that is going to be Lagchain dash Groq. This will enable interacting with the Groq API. All right, I think I have already installed this as well. Okay, so we are all set here. Now, in order to run the LLM through Groq API, we will just need to make a couple of changes. In Local GPT we have a constants PY where we store all the constants. And this also defines which embedding model to use as well as which LLM to use. But in this case, since we want to interact with the Groq API, we we need to first import the required packages that we will need. And that is going to be the chat Groq from Langcheng Groq and we are making this change in the Run local GPT PY file because there this is where we are using the LLM. The second change that we want to make it is in this function called Retrieval QA pipeline. If you look at the code here, we are actually loading the model that is defined within the constants PY file. So what we need to do is we just need to comment out this line. So instead of downloading the model from hugging face and then loading it, we are going to simply call the Groq API here. After that the next change is going to be to just add this line of code. So now the LLM is Chat Groq. We need to provide our Groq API key and we also need to provide the model name. So by default, local GPT is set up to use Llama 2 models and that's why I'm using Llama 2. But you can also use the Mistral MOE model. The only change that you are going to need to make it is here. So in the prompt template type you will provide Mistral instead of Llama because Mistral and Llama have different prompt templates. But other than that we should be good to go. Now, if you don't know where to get the Groq API key, I covered that in a previous video. I'll put a link to that video in the video description. Okay, Next, to use this we need to first ingest a file. So Local GPT comes with a copy of the original Orca paper and you need to copy your files into this Source documents folder. So first we need to ingest this. Now in order to do that we are going to be using the Python and then ingest PY command. This will basically create chunks from the document that we have provided and we'll create a vector DB. Okay, so here it created a vector DB and it has a total of 193 different chunks of text. Now you will see this warning you can install the LangChain Community package as well if you want to, but you don't need it right now. Okay, so we are all set. Now we can start chatting with our document. And in order to do that, we are going to be using the Run local GPT PY file. First, let me show you how long it's going to actually take to use this Rack pipeline if you were to use a model from Hugging Face. So in this case, we're going to be downloading the Llama 270 build model from Hugging Face. So we set this model up in the constants PY file. Next, we uncommented this section, so we actually want to download the model. Now, in order to run local GPT, we're going to use the python run local GPT file. Now this machine has a 6000 GPU which has 48 gigabytes of vram. So currently it's downloading the model from Hugging Face. This is going to take some time. The model is loaded. We're going to run this prompt. What is instruction tuning? And how was the ORCA model trained? So this again is running on an A6000 GPU and we are loading the full model. So it's not a quantized version of the model. And you can see that even though with this powerful GPU it does take some time. Okay, so here's the response that we got. It took like 10 to 15 seconds. And it says, I'm here to help you understand instruction tuning and how it relates to the ORCA model. Then it talks about instruction tuning and provides a pretty good definition of instruction tuning based on the paper. Then it talks about the model training part as well. So it says the authors of the paper explains that the Orca model was trained on a data set that simulated zero shot settings with standard prompts. Right, so this is a pretty good summary. Now let me show you the speed. When we are using the GROQ API, we commented out this part and now we are using the Groq LLM. And don't worry about the API key. I'm going to revoke this right after recording this video. But let's run the same Python script. Now, in this case it will still load the embedding model because we do need to compute the embeddings, but it's not going to be using the local LLM anymore. And here you can see the message using the Groq API. Now again, we're going to use the same prompt. What is instruction tuning? And how was the ORCA model trained and this is real time. So this is actually how long the model took to generate a response. This was pretty quick because it has to go through the retrieval part using the embedding model and after that it has to make an API call, get a response and show us. This was pretty quick. And the answer actually also has two parts. So first it talks about the instruction tuning itself and then it talks about the training of the RCOM model. So this is pretty accurate. I think we can still prompt it to provide more detailed responses, but overall the responses were pretty fast. So hats off to GROK API. They're doing a really good job. Currently the GROQ API is free to use. There is some pricing information on their website, but we don't know when that is going to go live. Another thing which I wanted to highlight was that while you're running Local GPT you might see these warning messages and that's because we're using a relatively older version of LangChain. We are in the process of updating the code base to the newer version. So if you're interested in updates related to Local GPT or Advanced Flag, there's a link for the mailing list in the video description. My goal with Local GPT was to create a simple framework where you can do a lot of experiments. You can try different models from different sources. You can try models from Hugging Face. If you are running a model on Ollama, you can use that or even you can use some external APIs as well. It also gives you the ability to try different embedding models as well. So there are quite a few new models on the embedding leaderboard. You can switch those around as well here. So I think it's a flexible framework that will let you experiment very quickly and once you like an approach, you can actually use that through the Local GPT API. Anyways, this was a quick video on integration of GROK API within Local GPT. I hope you found this video useful. Thanks for watching and as always, see you in the next one.