Use Ollama with LocalGPT

If you are running local LLMs, Ollama is one of the best options out there. And now you can use Ollama with local GPT. I saw this comment on my discord from Justin. Does anyone know if it's possible to use a model loaded up on LM Studio instead or Ollama? I'm trying to load a model and use it for other functions in addition to having it ready for local gbd. So that got me thinking why not to use Ollama with Local GPT now if you don't know what local GPT is, it's my project that lets you chat with your documents on your local devices or your private cloud using the power of LLMs. No data leaves your device and everything is a hundred percent private. I have a whole playlist on this project which covers the basics and also shows you how this has evolved over time. So in this video the goal is to show you how to use a model running on Ollama with Local GPT just using two extra lines of code. So first you need to set up local GPT. For that just come here, copy this link and clone the repo. In my case I'm running this on a pre configured virtual machine from Mast Compute. Link to this virtual machine is going to be in the video description. It comes pre installed with everything that you need to run local GPT and if you decide to use this make sure to use my code prompt engineering. To get 50% off on your local machine you will need to clone the repo using git clone command. Next you will need to create a virtual environment so you can use the condo create dash n local GPT and then provide the Python version that you want to use. In my case this virtual machine comes with a virtual local GPT so I'll simply activate that and for that we're going to be using the command conda Activate local GPT and you can see that the local GPT virtual environment has been activated. Now if you have a pre existing version of Local GPT running I will recommend to pull everything so this will pull the latest changes. This one is already up to date. Next we need to install all the required packages so just use this command pip install dash r requirements txt this will install all the required packages for local GPT. Next we need some documents to chat with. So in this case the local GPT repo comes with the original Orca paper and that is within the source documents folder. But you will need to provide your own documents so simply copy those in here. This specific Virtual Machine comes with the welcome document and it will describe the process of how you can move your documents to this remote virtual machine. So first we need to ingest this paper to create a vector store, and for that we're going to be using this Python ingest PY file. Once you run this, this will take the documents in the folder, create chunks, compute embeddings, and put them in a vector store. So depending on the number of files you have, this step might take some time. In this case, in this one document we have 193 chunks of text. Okay, so we are all set with document ingestion. Next we need to configure Ollama on our machine so that we can run and LLM using Ollama. For that we're going to go to the Ollama AI website, click on Download and then depending on which operating system you're using, just select the corresponding option. In this case, I'm running this on Linux, so I will just copy this shell command here. I opened another terminal and I'll paste that command and then run that command. This will download Ollama and install it on Local Machine. All right, so everything is set up. Next we need to choose an LLM that we want to run on Ollama. So go to models and here we have a list of all the models that are currently supported in Ollama. For this simple example, we're going to choose Mistral. So the in order to run Mistral, you can just copy this command. Here we're going to go back, we'll paste that command. Now if you run this for the first time, it will download the model. So that is going to take some time. In my case, I have already downloaded the model, that's why it's just ready to start interacting with the model. Now, since we want to use this model with Local GPT, make sure that you keep this terminal running. So next we're going to go back to Local GPT. In Local GPT you can set your models here, here. So for example, in this case it's set to llama 27 billion model by default. And this is the function that is actually used to load the model. Local GPT by default supports different quantization formats, for example gguf, ggml, that's an older version AWQ, gptq and you can also load models in full precision. But, but let's say we want to use the model that we are already running on Ollama. For that we just need to make a couple of changes. The first one is that we need to import Ollama package from LangChain. So we're going to use LangChain.LLMs import ollama. Now keep in mind that with the local GPT I'm using a relatively older version of LangChain which is pinned in here. For the latest versions, Ollama is moved to the community version of LangChain package. So near future I'll probably update the code base to support the latest versions. But for the time being, make sure that you are on this specific version. That's where you will find the Ollama package. So this is one additional line that you'll need to add to the run local GPT PY file. The second change that we're going to make is this where we are loading the model. So this is line 140 after you add that one statement and that is ll m equal to load model. Now in this case it's going to get the model ID and base model name from constants py but we don't want that. So what we're going to do is we're going to just comment this out and we are going to just add this one more line which is LLM equal to Ollama. Then we need to provide the model name. In this case we are running the Mistral model. That's why you'll have to provide Mistral here. If you're running any other model within Ollama you can provide that file, that model name after this. Just make sure that you save the changes and we are all set. Now we can run local GPT py. Now a couple of things are different than when you run a normal local GPT. So here it just loaded the embedding model. It did not load the LLM from constants PY because we are running the LLM from Ollama right now. Let's ask something. So I'm going to ask what is instruction tuning? This is a topic code in the paper and here's the answer. Instruction tuning is a technique used to improve performance of pre trained language models by allowing them to learn from input and response pairs. This technique has been applied to both language only and multi modal tasks. Then it talks about some of the examples that are covered in the paper. So it talks about the Alpaca, Vicuna and Wizardlm as well as Koala models. So this is pretty good. So just with the two lines of changes you are able to use Ollama within local GPT and local GBT is an extremely flexible framework. You can bring in these Lego blocks from different places and they will work out of the box in most of the cases. So do check it out and let me know what you think about it. Another community member was able to integrate local GPT as a tool with our Autogen, so I'll be creating a video on that. Also make sure that you subscribe to the channel in order to get a notification for that video. That's going to be a cool one. And also check out the Discord server. There are amazing people helping each other out. Let me know if there are any other topics related to Local GPT that you would want me to cover. Thanks for watching and as always, see you in the next one.