Local LightRAG: A GraphRAG Alternative but Fully Local with Ollama

In a previous video we looked at Light Rag, which is a simple and fast retrieval augmented generation system that combines knowledge graphs with embedding based retrieval and is a great alternative to Graph Rag from Microsoft. They just added support for local models using Ollama and also added some visualizations of the graph. In this video I'm going to show you how to set this up using Ollama. So everything is going to be running locally. Now I'll highly recommend to watch my previous video which they have kindly linked in their main GitHub repo for better understanding of how Light Rag works. So the first step is going to be to clone this repo. In my case, the repo is already cloned, so I'm going to just use the git pull command to ensure that I have all the latest changes. Next, you need to create a virtual environment. In my case I have created a virtual environment which I'm calling Light Rack. So we'll activate that virtual environment and since there are changes to the GitHub repo, so we need to install all the new packages that are added. So if you want to build it from source, you are going to use the PIP install E dot command. This will recreate the project from the source. Now, to run the local models using Ollama, make sure that you have Ollama downloaded and installed on your local machine. Okay, if you go to the repo under examples there is an example file called literag olama demo. So in here you can provide your model name and also the embedding model that you want to use. By default they're using the the Nomad embed text, which is a text embedding model from Nomic AI, but you can replace this with any embedding model that you want. Similarly, you can replace this LLM with any LLM that can be hosted. If you go to Ollama, you can search for models in here. In this video we're going to be using the QIN2 model because that seems to be a pretty good model. Now for the embedding models I will recommend to search for something like embedding and then you can see that there are a whole bunch of different embeddings that are available and you will need to run both the embedding as well as the LLM through Ollama for this to work. Now one more thing to consider. Ollama limits the context window to 2048 tokens in order to preserve the VRAM that is needed. So we need to expand that context window because during the chunking and vector creation process, we need a much longer context window. Assuming that you have Llama running, we need to start two different servers, one for the LLM and the other one for the embedding model. So we're going to start the server for the embedding model, which is easiest of the two. In that case we will use the default Nomic embedding. So just go to the Onama website and copy this command. Now in here I'm going to paste the command and just run it. So this will download the model and load it. Next we need to set up our LLM. Now you will need to do a couple of additional steps. So assuming that we want to use the Quin model. So first I'm going to go here and copy this command. This will download the Quinto model for you. The same process is going to work with any LLM that is available on Ollama. In my case, the model is already downloaded. That's why it just loads the model. Now the next step that we need to do is to change the context size or context window. You can do it in two different ways. One is you type in Ollama, run Quinto and then once the model loads you can set and then we are going to be setting the context window. You can type number of contexts and provide the new context window. So for example, if I want to set it to 32,000 tokens, that's how I'm going to set it. Now if you type this in, it says that the parameter is set to 32,000 tokens. But I want to follow the official instructions in the Light Rack repo and I'll highly recommend to do it because this probably is the easiest solution. The way we can change this this within the configuration file is that we're going to ask Ollama to show the model file for Quinto. So if you run this command you are going to see that it will create another file within this folder. And this is basically the main configurations of the QWINTU model that are stored through Ollama. So I'll just go to this parameter section and in here we're going to add this new parameter which is set the context window to 32,768 tokens. So just save this and instead of overwriting the configurations for QIN2, we will create another model file. So this command will create a new model file. So this created a new model file for us and now we can run this new model file using this new model that we created. So now I can Do Ollama then run Quinto m, which is the modified model. And if you run this, this will load the new model with the updated context window. Now we need to go back to our example file that we want to run in here. We're going to provide this new model name which is going to be quin2m. That is the model we just created. And here we we're running our text embedding model. Okay, Another thing that you can check out is if you go to any terminal and just type ollama PS that's basically show all the processes that are currently running. So you can see that is currently running the Quint 2M which is the modified model file that we created. And right now it's using the 100% GPU. So now we are all set. Again, we will need some sort of example data set to run this on. So based on the example the repo has provided, I downloaded the same file which is this Textbook Christmas Carol by Charles Deakins and we stored this as a book. Txt. Now in my case I deleted quite a bit of the text from this file just to make sure that we're able to run the demo pretty quickly. But in your case you will need to parse your PDF or HTML files and put them plain text and you can run the indexing process on the plain text. So just a quick recap of how this works. I already created this. I'm going to actually delete the index that was previously created. So first you have to provide the working directory. I am calling it deakinsulama. Then if that directory does not exist, it will create that directory for us. Now in terms of the configuration that we are using, we are saying that look at the working directory. So if there there's no indexed or knowledge graph within that working directory, it will start creating that for us. Then we want to use the Ollama model. So that's why Ollama model complete, that's the function provide the name of the LLM that you want to use, which is quintum that we just created and then the embedding model that you want to use. Now I think you can mix and match. So for example, we can use a local LLM but with the API based embedding model that probably is going to work out of the box, but the configurations are going to be a little different. I'll probably show that in another video. Now once we create this basic light rag object, then we load the text and pass that text to the Rack system to index it. So it will create the knowledge graph plus the embeddings are going to be computed. I highly recommend to watch my previous video if you haven't. Also, if you want to understand how RAG system works, I have a course on advanced RAG systems in RAG Concepts. Will highly recommend to check that out. I'll probably create a more detailed video on light RAG and include that in the course at some point. So do check it out. But then once we create the index, we can just run queries on the index. And the way you do it is you call this query function on your index, you provide the query. In this case I'm asking who is Mr. Scrooge and what is his relationship with other characters. Now it supports four different types of query mechanisms or query modes. I have discussed these in my previous videos. But you can run naive RAG then using the local search on the knowledge graph or using the global mode. Or you can combine both the local and global mode. And depending on the type of the query, in some cases this hybrid approach can be extremely beneficial. Anyways, so we have set this up. Now in order to run this we need to run the light RAG olama demo file which is under the example folder. So we're going to type Python examples and then light RAG and ollama. Okay, so after running this you're going to see a few things happens. So the first one is it will show you all the different parameters. The embedding model that we are using has an output a vector of 768 different dimensions. The max number of tokens are here, so it seems like everything is correctly loaded. We are using the correct model name plus the max number of tokens are set correctly as well. So this is good to know. Now there was a single document that has 20 different chunks. By default it's using 1200 tokens chunk size, so it created or 20 vectors for those chunks. And then right now it's running the entity extraction process. So this can take a while depending on the length of your documents and you can provide multiple documents if you want. So first it ran the embeddings for each of the chunks and now it's using the LLM to create those relationships. One more thing that you can do is you can go back to your terminal and then again run the ollamaps command. So it right now shows that two different processes are running. One is the LM and the second one is the embedded model. The knowledge graph creation process is complete along with the indexing and it created 99 nodes with 31 edges. For those 20 chunks it took close to 10 minutes. And I have seen the indexing process can take variable amount of time for the same documents. So keep that in mind. And then we ran the same query through these four different modes. Now for the naive rag approach it is I think taking the 20 chunks, but they are only using 3 chunks in this approach. And the result is pretty good, not bad at all. So it talks about Mr. Scrooge and it says that he's a central character in the novel. Then he is portrayed as a greedy and miserly businessman. Then the different relationships with different characters, they are highlighted in here. If you look at the local approach, which is basically using the knowledge graph, we see the data is presented in a little different way. So in here the query uses 16 entities, 24 relationships within those three text units. And I think these text units are very different than the text chunks. So again it talks about who Mr. Scrooge is and then the relationships that this specific character has with some of the other characters. Now you see similar patterns for the other modes as well, but the number of entities differ based on whether you are using a global. If you were to add more data to it, it's a non destructive operation. So if you look at something like a graph rag, in that case it will recreate the whole index. But in this case you just need to call the insert function, provide the text, it will update the knowledge graph plus the index rather than re evaluating everything. That is a good thing for Graph Rag. Now let's change one of these questions. So I'm going to just say who is Bob? And if you run the same query or the same file again it should take a lot less time because it's going to be just reusing the index that it already created. So it persists on the on the disk and it's going to reuse that index for us. So I'm going to run this and then I'm going to show you how the index itself looks like. And then we will going to look at some visualization. So this is a real time process of how I'm running it. So it loaded that and it says that the local query uses 60 entities, 26 relationships, 3 text units. Depending on your hardware, it could take some time for running the actual query. Okay, so I would say it took close to 10 or 15 seconds. But here's the response. Bob is a character and he is portrayed as Mr. Scrooge clerk, highlighting him as someone who struggles financially and works diligently for Scrooge. In his office. Right. So this is a quick response. We can see that the number of entities that it used was Quite equals to 60 with 26 different relationships. Now, if you go to the actual folder that it created, there is this graph file which preserve the actual relationships between different entities. And we're going to look at a visualization of this. There are other files. For example, this is the LLM response cache. Then I think there is even like different chunks that it created plus the entities that it found in the document. And then what are the different relationships? So within the example folder, there are two other files that are, I think, interesting. One is the graph visualization with HTML. So you can run this and this will visualize the graph. Now, in this case I had to change the folder. So by default the folder name is Deakins, but I change it to Deakins Olama because we created the index in this updated folder. So let me show you how to run this part. Visualization. So we're going to type Python examples and then graph visualization. So we want to use the simple HTML1. There is also new 4j, but in that case you will need to create an account on the new 4J platform. So I just want to see a representation of the graph on a HTML file. So if you run this, you're going to see that it created a new knowledge_graph.HTML file. We can open this file. Actually, I'll have to go to the main folder and then run that HTML file. So here's the folder. If we run it now, here is how the visualization looks like. We can zoom in. So in the center here we have Scrooge and then the different relationships with other entities. So for example, Christmas Eve Liberty Ghost Clerk. Right. Then there is another one, another character. I'm assuming there are some of these other entities which don't really have any type of relationships. I was using a subset of the data, so it's not the complete data. And that's why probably not all of the relationships are preserved in here. But it's a pretty neat visualization, pretty decent overview of different entities and what their relationships are. This was how to use Olama. Now, there are other demos. So for example, in here, if you look at this file, which gives you a demo of using the models available on Bedrock, you will just need to provide your model name that is served through Amazon Bedrock and then the embedding model that you want to use. Similarly, you can run models available on Hugging Face directly without using O Llama. Now, I ran this example the main issue with this specific part was that it was loading the model for each request. So for example, if it was trying to figure out entities in a chunk for each chunk it would reload the model. That takes forever. Hope that the team is going to look into this specific problem. There is another example for using other models through the API which which are using the OpenAI compatible API endpoint. So potentially you can use something like Groq or Mistral in here. I tried to use Groq for this specific example. I was running into rate limits because it is making a lot of API calls. There is also a demo on VRAM management because some of these models can actually take quite a lot of your VRAM depending on how much context it's using from the LLM. So this is a project which is under active development, but I have seen some really good progress in the last few days. Now this was a quick demo of how you can use Ollama with Lite Rag. I haven't done extensive testing on how good this approach is compared to some of the other approaches, but based on the performance tables that they have shown plus the results in their technical report, it does seems to outperform most of the other Rag approaches. So I think it's definitely worth checking. Especially it has an MIT license, so it's not too restrictive either. Let me know if you want me to make more videos on Light Rag or any other topics that you are interested in. If you want to support my channel, there are quite a few options. Details are in the video description. Also, make sure to subscribe to the channel if you are interested in technical content related to Rag agents and anything in between. If you want to get a better understanding of how Light Rag works, created a previous video, a link is going to be at the top. Or if you are interested in the Graph Rag project, link to that video is going to be here. Thanks for watching and as always, see you in the next one.