Graph RAG with Ollama - Save $$$ with Local LLMs
Run Graph RAG locally with Ollama and llama 3 to cut costs versus GPT4O. The tutorial covers YAML configuration, indexing setup, and compares output quality across llama 3 8B, llama 3 70B on Groq, and GPT4O, showing why Graph RAG demands a capable LLM for entity and relationship extraction.
Chapters
In a previous video we looked at Project Graph Rack from Microsoft. This aims to combine knowledge graphs with retrieval augmented generation to address the limitations of traditional RAC systems. Now in that video we use GPT4O as our LLM, but that was pretty expensive to use. In this video I'll show you how you can use a local model using Ollama as well as the Groq API. And we are going to also talk about why it probably is not a good idea to use local models with Graph Rag. But to get started you will first need to download Ollama on your local machine and then choose the model that you want to use. So in this case we're going to be using llama 3, but I'll recommend to use a much bigger model if your hardware can support it. And I'll explain the reason of using bigger models later in the video. Llama 3 follows the same API standard as OpenAI, so it makes it very easy to just replace the OpenAI API server with this new endpoint and by default it's going to be running at localhost port 11434 and V1. That's basically the API version. We will need this base URL as well as the API key which is going to be Ollama in this case. Now we just need to point the Graph Rag application to start interacting with that API endpoint and for that we will just need to go to the project that we set up for Graph Rag, then go to Settings, YAML, and in here we're going to look at the LLM. Now you will need to set up the project. So again if you're not familiar with it, I'll highly recommend to watch my previous video in that I both cover the theoretical aspect of how a graph rack works as well as like how to set it up. Now in this case we'll need to make a few changes. So initially this was pointing towards the Graph Rag API key for the LLM, which is basically the OpenAI API key. But I'm going to provide the API key as Olama. You can keep the type as OpenAI chat because Ollama follows the same standard as OpenAI for the model we're going to provide llama 3. That's the model currently we are serving. And since Olama also supports JSON mode, so you can set this to trim for the base API, we are going to provide this API point that we are currently running on our local machine. Now there are a couple of other parameters that you want to set if you are working with something like Groq. So if you are serving the model through Groq, in that case you will need to change this API key to the Groq API endpoint. And in that case, instead of the base URL that we're currently using, we're going to use this base URL. I'll show you that later in the video. But this is one of the changes that you need to make, and the second one is going to be the model that you want to use. So, for example, you can select the Llama 370 billion model, so you will have to provide that model as well. Now, Groq does have some rate limits, and that is why you actually need to make sure that you are following those rate limits. So, for example, for most of the models you can make only 30 requests per minute. That's the max number of requests Groq is going to allow you on their free tier. So for Groq, we will need to actually set this number of requests per minute to 30, or probably less than that. That is to ensure that we don't timeout. But keep in mind that if you set that up, it's going to take a while for the process to finish. The second aspect is the embedding model that it's going to be using. Now, in my case, I haven't really found a solution to replace the OpenAI embedding model. And the reason is, I think there is no standard API when it comes to the embedding models that other API providers are following. So when I was experimenting with local embedding models, I couldn't really make it work because I think they're not following the same standard as OpenAI. Now, even if you use the OpenAI embedding model, the cost associated with embedding models is pretty small compared to the LLM. So, for example, so for example, in my previous experiment, we made only 25 requests to the embedding model, compared to 570 requests to GPT4. Oh. So even if we, if you were to use the embedding model from OpenAI, I think it's not going to cost you much. Okay, so once you set this up, the next thing is to just run the local indexing. That's the first part. And for that we're going to be using python m graphrank.index because we want to create the index and we want to provide the file path. So by default it's looking at this folder, and within this folder we have an input folder and you can change that within the settings YAML file. So here is the input folder. You can provide another name if you want. You can also change the chunk size as well as the overlap, but I'm not doing that in this case. Now, the entity extraction part is running. It has taken about 21 minutes and it has only completed about 50% or 58% in this case. I'm running this on an M2 MacBook Pro that has 96 gigabytes of and here's what the GPU usage looks like. Now, this is not the only process that is running on my machine. I have gazillions of Chrome tabs open as well as some other processes running, and we can also look at the output folder. So the output folder basically is going to be creating the embedding vectors as well as it's going to be creating this reports. This is a place where you can see what exactly is going on. So let me give you a quick overview of what is happening. So for the base URL, it's using the API endpoint that we have provided and it is using the Llama 3 model. So that's a good thing. Now it has the other settings that are coming in from the settings YAML file and you can see it's making calls to it API endpoint and sometimes it retries things. So for example, here was an error that it got, but it's able to recover from that error because it was retrying it multiple times and you can select how many retries it's going to have. Now, when I was running GPT4O, the process took much shorter amount of time and the results probably are going to be much better if you're using the bigger GPT4O model compared to when you're using something like llama38bn. But I just wanted to show you how you can set this up. For better results, it might be better to use the Llama 370 billion model from Groq. However, in that case you need to make sure that you set the requests per minute to a lower value and that could result in much longer time that it's going to take to create the entity extraction as well as the corresponding graph for us. Now, it has already taken about 27 minutes. So I'm going to wait for this process to complete and then I'll walk you through how to run this. To test this out, we're going to use the same prompt that we used in the previous video. So we'll use Python dash m graph, rag query and we need to provide where our documents are located or where the index was created along with the graph, then the method that we're going to be using is Global community. I described or explained this in my previous video, so I highly recommend to watch that to understand the difference between global and local. And the prompt is going to be what is the main theme of the book? So let's run this. Okay, so here's the response. Global search response Main theme of the book and the main theme of the book, as highlighted by multiple analysts is the importance of human connection and relationships. This theme is evident throughout the character of Scrooge who continues to answer Marley's name even after his death. Right, so the, the, the summary or the main theme that we get from llama 3 is not as good as GPT 4 and that is kind of self explanatory because when it comes to graph rag, the choice of LLM that you are going to be using is a lot more critical than the choice of LLM in traditional RAG system. And in order to explain this, let's look at just traditional RAC system. So in traditional RAC system the most important aspect is your embedding model because that really determines what type of chunks the LLM is going to receive when it's trying to generate the responses. So you want to make sure that both the chunking strategy as well as the embedding model that you choose is great so that the LLM receives the proper context. So in that case, if even you have a smaller LLM and you provided great context, it will be able to generate good responses. But in case of the graph flag approach, it's very different, right? Because the way you are building these knowledge graphs is that you first extract entities from your text. So you need to have a really great LLM that is able to recognize different entities that are present in, in your documents and extract relationships. So if you use a smaller LLM like llama3.8 billion, then it will not be able to actually extract those relationships accurately. And as a result, as a result the graph that you create is not going to be great. So and then you also need to basically create summaries of the communities that you're creating based on the graph that is created by the LLM. So there are like multiple aspects in which the LLM plays a lot more graph critical role when it comes to graph ride graph rag compared to the traditional RAG system. So a smaller LLM is probably not a great choice here. And you want to look at much bigger LLMs like llama 370 billion model. So here are the results when I tried to use the Lama T70 billion model from Croc. So in this case I replaced the base API, also changed the requests per minute. We're using the Lama 370 billion model and the results is this. Now keep in mind that I didn't embed the whole file in this case because that was taking way too long. So it's just a small portion of the main book, but it says the main theme of the book revolves around Scrooge's transformative journey marked by supernatural events and interaction with various entities. Right? And it talks about the significance of supernatural events, implications of the theme. So it's much better compared to the llama3.8 billion model, but not still as good as the GPT4. And the reason being that it was just looking at a very small portion of the document compared to the whole book. Another thing to consider is these prompts that are being used. So by now we know that different LLMs react differently to the same prompt. So you really need to actually look at your prompt and kind of handcraft it for each and every LLM differently. So a prompt that works great for GPT 4. O may not be a great prompt for llama 38 billion or even llama 3. 70 billion. So if you are going to use Graph Rag in your system, just make sure that you are able to modify these prompts based on the LLM that you are using. That is going to be a very critical component that a lot of people simply ignores and then the system doesn't return good outputs. Now I'm going to be experimenting with Graph Rag a lot more because because I think it's a great framework that needs a lot more exploration and there are some other implementation of Graph Rag as well, so we are going to look at some of them in subsequent videos. So if that's something that interests you, make sure to subscribe to the channel. Thanks for watching and as always, see you in the next one.
