LightRAG: A More Efficient Solution than GraphRAG for RAG Systems?

Today I'm going to show you how to use Light Rag, which is a new retrieval augmented generation method for chatting with your documents. It combines knowledge graphs with embedding based retrieval mechanisms. But you might remember Graph Rag, which was also a knowledge graph technique developed by Microsoft Research. Now, Graph Rag is great, but it is extreme extremely expensive to run. On the other hand, Lite Rag seems to be a lot more performant compared to Graph Rag, at least on the benchmarks the authors have used. And it's very inexpensive to run more on this later in the video. The best part, it's open source, the code is available on GitHub and you can set it up on your local machine. They also released a detailed technical paper titled Light RAG simple and Fast Retrieval Augmented Generation. In this video, I'll walk you through a step by step process of how to set this up on your local machine. We will look at some comparisons between Graph Rag and Lite Rack in terms of performance and cost. And we will run this on an example data set. But before that, let's look at some technical details. If you are just interested in running this and start experimenting with it, skip this section. First question is going to be why do we need Light Rack or yet another RAG technique? So the main problem with the standard Rack systems is that the data itself is represented in flat data structure, which means you simply chunk your documents and each chunk is independent of the rest of the chunks. Hence you lose a lot of contextual information. That's why the Microsoft Research introduced Graph Rag, which uses knowledge graphs to preserve the relationship between different entities and objects present in your data. Graphrag does that by entity and relationship extraction. Then it summarizes that information and generates both local and global communities. I have a very detailed video on Graphrag. Link to that video is going to be in the description. But the problem with Graph Rag is that it's extremely expensive to run because it makes several hundred calls to your API endpoint. By default they use GPT4. Okay, so this is pretty expensive. RAG tries to use Knowledge Base again to extract entities from your data. But then it also combines those entities with traditional or standard RAG system. And in this case I'm talking about embedding based vector vector retrieval systems. Now here is how the algorithm works. I divided this into two parts. The first is the indexing process and the second part is going to be retrieval and generation. So I created this with the help of Cloud 3.5 Sonnet. So you have your input documents those are segmented. So that is basically the chunking process. Then you take all your chunks and pass them through an LLM. In this case they're using GPT4OHM Mini. So it is a pretty inexpensive process. And that LLM determines different entities and objects within your text. Now there are two different parts. One is the entity recognition and then the relationships recognition. So first it will identify different entities and try to identify the relationships between those entities as well. And then it creates something called entity key value pairs and relationship key value pairs. So basically it takes each entity figures out what are the relationship of that entity with other entities. Then it will use that entity as a key and those relationships as text descriptions, which can be very detailed as values. Same happens for relationships as well. And using these two, it creates the knowledge graph. Now this step is very similar to what Graph Rack does, but after that it does a deduplication step. So basically it looks at all the entities that are present, plus the relationships, and it figures out if there are any nodes which are duplicating those entities or relationships. It simply drop those and as a result you, you are left with a very compressed knowledge graph. Now in between, during the indexing process, there is a very interesting step that is happening. So let's closely look at this entity and relationship structure, how these key value pairs are created. So based on the given text, the LLM will generate those relationships. But then for the values we also compute the embeddings. This is basically the part of a standard RAG system. And those vectors are also stored. So not only you have your entities, but the corresponding values, which is detailed descriptions created by the LLM, which describes the relationship between one entity with the rest of the entities that are present in your text. We actually embed those descriptions into a vector store. During the retrieval process, the query goes through keyword extension and it tries to identify the entities. And if potential relationships within your query, then it does retrieval. But in this case the query is also compared with all the values, which is the text description using the embedding vector or vector based similarity search. And that's how you extract the corresponding entities that are present in your knowledge graph. Now the retrieval happens at three different stages. Low, high and hybrid. We're going to talk about that in a bit. But depending on the mode that you choose, it will select certain entities and pass on their description that was generated by the LLM to the final model for the generation step. This is very simplified version of the paper. Just for the general understanding. It's a dual step retrieval process. So it utilizes two different levels. One is Low level retrieval and then high level retrieval. So in the low level retrieval the focus is based on the nearest neighbor of the entity that were selected. So essentially it retrieves specific entities along with their associated attributes or relationship. Queries at this levels are detail oriented and aim to extract precise information of particular node or edge within the graph. So if a certain node is selected it just look at the neighborhood and it will also utilize the closest immediate neighbors. Now at the high level retrieval this addresses broader topics and overarching themes. Think about this as a global information. So if a certain entity is selected during the retrieval process, it will look at its global relationships with other entities and use that information to generate final responses. Now for each of the query, either you can use low level retrieval or high level retrieval, or you can combine the results from both of them which they are calling hybrid which gives you the state of the art results. They call this dual level retrieval paradigm. They also provide a comparison of their light rag technique versus naive rag rag with query expansion hide and graph rag and their results shows that overall it seems to outperform most of these techniques including graph rag which is state of the art. If you look at agriculture, CS and legal in almost all the different categories they are testing Graph rag versus Light rag Light rag seems to be the best model. Now for a mixed data set Graph rag seems to do better but for diverse still Light rag seems to outperform and even in the cases where is outperforming light rag it's pretty close. But in the the other way around I think the margins are very different. Now a couple of other points of comparison between graph rag and light rag before I show you how to set this up. Also this is a new rag technique and the results that I'm showing is based on their paper. I haven't seen external validation so keep that in mind. In most of these cases you really want to test these different rag techniques on your own data set because in some cases you don't need knowledge graphs so you don't really want to go through this extra complexity. And in the cases where you need knowledge graphs you definitely want to test graph rag versus Light rag and some other knowledge graph techniques. Now I think the main reason I would use light rag or graph rag, apart from the performance is the cost. So there is one major issue with graph rag and that is if you want to add more data to your knowledge graph. In case of graph rag you will have to recreate the whole graph. But for light rag you can actually add data to an existing graph without recreating the whole thing. And if you look at the comparison between the number of API tokens and it uses for the same data set to create Graph Rag versus Light Rag, the results are very different. This seems to be extremely cost effective. Even in in my tests when I was running Graph Rag for one data set that I was using, it cost me around $4. But in this case it's I think about 10 or 15 cents. Now, even though I was using different models GPT4 or versus GPT4 or mini in case of Light Rank. But still the number of API calls are substantially lower. So this must be a consideration if you are working with large documents. So hope this gives you a quick overview of how Light Rank works and how it compared to Graph Rank. But now let's quickly look at how you can set this up on your local machine and start interacting with it. A quick note before that, this uses Nano Vector database for vector data management and access. I haven't really used Nano Vector database, so if anybody is familiar let me know. And that way it might be possible to replace some of the functionality with other vector stores or vector databases. In order to install it, you have two different options. One is clone the repo and then run the installation locally. And the second option is you can install this as a PYPI package as well. So this makes it extremely simple. But I'm going to clone the repo, just to make sure that we have the latest updates. In order to clone the repo, we're going to use git clone command and provide the repoid. So if I CD to that repo and then type ls, you'll see these are all the files that are present. Now in order to install this, we can just run the setup file. I'll highly recommend to create your own new virtual environment. You can use something like conda. So I'm going to use conda. The virtual environment name is going to be Light Rag and we are using Python 3.10. So this is going to create the virtual environment for me and then we can install all the required packages. So to activate the virtual environment, so we're going to say condo activate. Then we need to install everything from this repo. So we're going to run the PIP install E command. This will install all the required packages in this new virtual environment for us. Okay, so the installation is complete. Here is an example code. Let me quickly walk you through what is happening in here. So you need to import the west library, Then you need to set your API key for for OpenAI. In this case, by default it's using GPT4OHM mini. I think it probably is going to be possible to use local models instead of GPT4 or mini, but I haven't experimented with it with it enough. I'll probably look at that in a later video. Then we are importing light rag and query parameter. You need to actually provide a folder name. If this folder does not exist, you need to create it, otherwise it's going to throw an error. But in my case I created a folder called Deakins and then created this light rag object in there. We need to provide data. You're going to provide text file. You read those text and then you start appending it into or start inserting it into your rag object. Now in their example they're using A Christmas Carol by Charles Deakins. You can use this command to download that book. I am just following their example. For this initial video we're going to create a couple of more detailed videos on how to use your own data sets. I downloaded that textbook, put it in the same folder, then open the textbook and just reading the contents of that text file and appending it to our rag object. If you want to do the insert in batches, here is what it's going to look like. So you can call this multiple times with the insert function and then provide a list of text chunks. So this is basically how you can update the data or add new data set. This will only update the knowledge graph. It's not going to recreate the knowledge graph from scratch. And that's why it's a lot more cost effective compared to something like Graph Rag, which actually has to recreate the whole graph when you add new data. Now as I mentioned, you can run the queries in multiple different ways. So in order to run a query, you're going to call this query function, pass on your query and then you need to tell it which mode to use. If you use naive mode, it will do standard rag. If you do local mode, then it's going to look at the nearest neighbors of the entity that it finds. If you set it to global, then it's going to only use global information. But if you set it to hybrid, then it uses both the local and the global information to generate the final answers. Now in all of these cases, the query is exactly the same. What are the top themes in this story? And it's a pretty huge document. But let's run this PY file. This is the file that I created. If we run this, then it first going to create the index for us and then it's going to run each of these queries. In my case the index is already created. Now a few things that you will see when you run this. So it's the working directory. By default it uses a chunk size of 1200 characters with hundred characters overlap. By default it's using the GPT4 on mini model with some other information that is in there. I think it's using the OpenAI small model for embedding creation with a max Token size of 8000 tokens. Now based on your needs, you can adjust these parameters if you want. But it created all like our knowledge graph in here. Now it will create multiple different JSON files. So think here are different entities that it was able to extract. So for example, it's talking about Scrooge. These are I think the relationship relationships for this one entity. And you can see that it keeps talking about Scrooge. There are some other characters in here as well. Right. So based on the text, first figure out what these characters are. What are their relationships with other characters. These are the chunks that it used. So with each chunk you have the number of tokens which is limited to 1200 characters plus the actual contents and then token or the chunk index. Now here's how the chunk vocabulary looks like, but let's look at what the actual output looks like. So the first one was naive. I think it used 39 chunks and then it use the re ranking approach to bring it down to only three chunks. So based on that I think the main theme of the story is redemption and importance of compassion and kindness. Now this book might be in the training data set. So if it just looks at the title of the book, it is probably making some of these assumptions. For the local retrieval it used 60 entities with 108 relationships within three text units. So these text units are probably different than the original chunks. And these are probably based on the summarized relationships that are within the text. Now it looks very similar to what we saw in the first case, except I think it added this spirit of Christmas to it. Then if we look at the global retrieval technique. So now it's using 57 entities with 60 relationships, again with three text units. And if you combine them together, I think it boils it down to 50 entries, 60 relationships, three text. Right. And the final one has definitely a lot more detailed response. Now for this whole thing I think I used about 80,000 tokens, which is a lot less than what I was using for when I ran the graph rag. I think that was well over a million tokens for the same book. So this is definitely a lot less expensive compared to running Graph Rag. They have provided some detailed prompting if you want to run evolves on the data sets that are listed in here. Also, there are a lot more details on how both the entity extraction process works, how do you insert more context into it, and then how to generate those queries. Right? So I'll highly recommend to look at their GitHub repo. There is a lot more information in here. It's a completely new project, so I hope people are going to start taking notice of this and hopefully this might be a very good way of creating rags based on knowledge graphs, but time will tell. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.