GraphRAG (beyond the hype)

Foreign.

If you work in artificial intelligence, aspire to, or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at fly IO. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at fly IO. Okay, friends, I'm here with Andy Sexton over at flye Annie. You know we use fly here at changeload. We love fly. It is such an awesome platform and we love building on it. But for those who don't know much about Fly, what's special about building on Fly?

Fly gives you a lot of flexibility, like a lot of flexibility on multiple fronts. And on top of that you get. So I've talked a lot about the networking and that's obviously one thing, but there's various data stores that we partner with that are really easy to use. Actually, one of my favorite partners is tigris. I can't say enough good things about them. When it comes to object storage. I never in my life thought I would have so many opinions about object storage, but I do now. Tigris is a partner of fly and it's S3 compatible object storage that basically seems like it's a CDN, but is not. It's basically object storage that's globally distributed without needing to actually set up a CDN at all. It's like automatically distributed, distributed around the world. And it's also incredibly easy to use and set up. Like creating a bucket is literally one command. So it's partners like that that I think are this sort of extra icing on top of Fly that really makes it sort of the platform that has everything that you need.

So we use tigris here at Changelog. Are they built on top of Fly? Is this one of those examples of being able to build on Fly?

Yeah. So tigris is built on top of Fly's infrastructure and that's what allows it to be globally distributed. I do have a video on this, but basically the way it works is whenever, like, let's say a user uploads an asset to a particular bucket, well, that gets uploaded directly to the region closest to the user. Whereas with a cdn, there's sort of like a centralized place where assets need to get copied to and then eventually they get sort of trickled out to all of the different global locations. Whereas with tigris, the moment you upload something, it's available in that region instantly and then it's eventually cached in all the other regions as well as it's requested. In fact, with tigris, you don't even have to select which regions things are stored in. You just get these regions for free. And then on top of that, it is so much easier to work with. I feel like the way they manage permissions, the way they handle bucket creation, making things public or private is just so much simpler than other solutions. And the good news is that you don't actually need to change your code if you're already using S3, it's S3 compatible. So like whatever SDK you're using is probably just fine. And all you gotta do is update the credentials. So it's super easy.

Very cool. Thanks, Annie. So Fly has everything you need. Over million applications, including ours here at Changelog. Multiple applications have launched on Fly, boosted by global anticast, load balancing, zero configuration, private networking, hardware isolation, instant wireguard, VPN connections, push button deployments that scale to thousands of instances. It's all there for you right now. Deploy your app in five minutes. Go to Fly I.O. again. Fly I.O.

Foreign.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am CEO and founder at Prediction Guard and I'm joined as always by my co host Chris Benson, who is a principal AI research engineer at Lockheed Martin. How you doing Chris?

Doing very well today, Daniel. How's it going?

Uh, it's going, it's going extra wonderful today because if, if I think back over the past sort of couple years and the way that, you know, these shows that we produce together have also impacted my day to day work. One of the ones that has impacted me a lot and also I think been super beneficial. Just Connection is, is one the one that we did about kind of vector databases beyond the hype and super happy to have our guest from that show back who is now an AI engineer at Kuzu Prashanth ro. Great to have you back on the show.

Oh yeah. Hi Daniel. Hi Chris. Very good to be back on the show again.

Yeah, I've taught a lot of workshops over the past couple years. I've done trainings with our customers, I've built things with our customers. And always when it gets down to vector databases, I'm like, hey, you know, anything I say, you can maybe learn some things from me. But if you really want to learn about vector databases and get the right sort of intuition, you really need to go to this series of blog posts on the data query, which is your blog, which was the first thing that I saw which brought you originally on the show. We talked through vector databases and trade offs between different types of vector databases and. And all of those things. Since that time, of course, you've moved on to a company, a data company, working in a different type of data, which we'll talk about today, but also as related to some of those things. So yeah, how has the year been? Lots of updates and fun things it sounds like you're working on.

Oh, absolutely. So I think I can summarize very briefly where I'm coming from since the last time we spoke, which was I guess just about a year ago. So as we spoke in our previous episode on vector databases, in 2023, I was working in the Royal bank of Canada and I'd spent most of that year thinking about vector search and the difference between the various database options there. And during that time I was also simultaneously working on using graphs and graph databases. And I'll talk a little bit more about what they are and how they're useful, but as I was working on them, and I think I mentioned this even in the conversation we had last year, is that I'm very interested in understanding how these two worlds come together because I know knowledge graphs and graphs have been around for a long time in their own space, and there was a huge hype around vector search and vector semantic search around this time last year. So there was a lot of scope, I guess, to understand how these two systems and these two methods can work better together in a way that we can build more advanced retrieval systems. So while I was thinking about those topics, I discovered new open source embedded graph database Kuzu, which is where I now work. And I happened to meet the CEO Semi Saliholu, at a local meetup in Toronto. And that got me really interested in what this tool and technology is all about because I'd been using other graph databases. And when I discovered Kuzu, the fact that it's embedded and open source, which I'll talk a little bit more about, was very attractive to start with. So I spent my spare time outside of work experimenting with it and engaging with the developer team, as well as working with other people in the community in terms of understanding how they're using graph databases and graph tooling. So, long story short, I realized at the end of last year that this was a great and fun opportunity for me to really go deep into the world of graphs and graph databases. And at the beginning of this year I moved over and joined Kuzu as an AI engineer and currently I'm leading the developer relation efforts and I'm engaging with the growing user community of Kuzuk, which is really, really fun in itself. And of course working with use cases that are taking this whole space forward and working with the community to understand what they are using these tools for. And I should also add, I'm still very plugged into the vector database ecosystem. I've not left that behind and I'm actively experimenting with some of those in my spare time as well. And I still very much enjoy using Lance db, which is the tool I mentioned last year in our conversation as well. So again, it's really exciting to see how these different kinds of tools are coming together. And yeah, I'd love to go deeper into some of these topics in our conversation.

Fantastic. Yeah, that the episode that we did last time with Lance and with the, with the other vector database issues, that was a great episode. And if anyone isn't familiar with that, they definitely should go back and listen to that one as well. But just to lay a little bit of foundation here, some groundwork for those who may not be intimately familiar with graphs and how are they useful and applicable and such, could you kind of start us off with just, you know, what is a graph and why should we care about it and what's it going to do for us?

So graphs are sometimes also known as knowledge graphs, and they're a great way to represent structured data via nodes and edges. The nodes are basically entities in the real world, like a person or a company, and the edges are relationships between these entities, like worked with or is CEO of and so on. Now, more formally, I also want to make the point here that the term knowledge graph is actually used to mean something more, slightly more involved than what I just described. Knowledge graph in theory can be used to express data that's quite hard to tabulate and store as records. It supports some logical reasoning over graphs as well. This is kind of what you would do when you store Wikipedia data, for example, but that's outside the scope of what we should go into for the purposes of this discussion. For most people and developer audience, the term graph and knowledge graph can be used interchangeably. So I just use them interchangeably for this discussion.

It's kind of like distinguishing machine learning and AI. To some degree, the edges, pun intended, don't matter that much in terms of their actual use.

I want to direct you for one second before we get past the point is, you did mention about graphs and you briefly mentioned it in relation to a relational database, and just because they've been around so long and it's where a lot of people are starting from on relational databases, you know, and you talked about the edges of the graph, you know, describing those relationships and being able to do that in some ways much better than a relational database can. Could you talk just for a moment so people can kind of make the jump from where they're at into this new tool and maybe, maybe think, next time I'm going to leave whatever database I'm in postgres, you name it, and actually try a graph approach. Could you talk just for a second about what the difference there is and bring people along on that path?

So I think we're going into the description of what a graph database is, right? So I'll just quickly, before I go into the database aspect of things, I just want to point out the benefit of what a graph is even useful for, because most people need to understand what a graph is for, and then you think about where to store it.

Great point.

So I think I want to highlight that the benefits of graphs become obvious when you're looking at data itself that's highly interconnected. So, for example, in medicine, you can have relationships between patients and symptoms of diseases, drugs that treat those diseases and their side effects, and these all have complex interweaving relationships with one another. Similarly, in finance, you can have chains of direct or indirect transactions, let's say, between onshore and offshore accounts, and these ultimately connect to given individuals anywhere in the world. So you could choose to model these using tables in a relational database, or you could choose to model this as a graph in a graph database. And in either case, there are pros and cons. But specifically, when you're analyzing patterns and you want to actually understand these complex relationships in an analytical way, using a graph database can actually be very powerful. It's a lot more intuitive. It's much easier to construct the queries that can answer these questions. And this is where the idea of a graph and how you model the data as a graph becomes very, very powerful. But going back to the idea of databases, so a graph database can be thought of as a specialized database that allows you to scalably manage and query data that's organized as a graph. And the performance aspects of these graph databases come from specialized data structures and operators that allow you to express complex joins and efficiently traverse parts in your data. Now, a lot of listeners may have heard of graph databases already, and the most popular graph database model is called the property graph data model, which was invented and popularized by Neo4J. And I've been a user of Neo4J myself in the past. Today, you have many other graph databases like Kuzu that also implement the property graph data model. But in general, you start off with a data model which is more conceptual, which allows you to express how to store and query your data. And a graph database basically is the underlying system that allows you to express the graph data model of choice. And the reason it's more intuitive than a relational database for certain kinds of queries that involve connected data and the sort of interrelated data is that it allows you to express your queries in a much more concise and intuitive manner using the query languages that graph databases offer, as well as the performance aspects of traversing the parts in the data.

And as just to sort of hone this in a concrete way, are you able to give like a few examples? Like there's sort of the typical personnel related things that you already mentioned, like, you know, Daniel is a node and he's CEO of Prediction Guard and that's another node. And there's organizations maybe for people that can't connect how such a structure would kind of be relevant to data that maybe they have in their enterprise. Could you give just a couple other examples of data that could be represented as a graph in maybe in different verticals or that you've run across in your use cases with Koozoo sort of data that's kind of beyond that kind of social network type of data? I guess.

Absolutely. I think social networks get a little excessive credit for being the original graphs that we know about. Of course they're very powerful and of course they're used. Graphs are heavily used in social networks. But yeah, I think I mentioned an example of the medicine scenario which I described, which is you have not just individual persons or patients in a network, but you also have the drugs and the different symptoms of the diseases that are being treated by those drugs. Each of these can branch out into much more complex relationships themselves. And a lot of this data that you might imagine in a healthcare scenario or a biomedical scenario, this data pre exists in structured form in many different sources. You might actually have medical records of people in a relational database or a data lake somewhere. A lot of existing workflows that work with these kinds of data sets tend to just stick with the database that is already used as a primary store. And many of those happen to be relational databases, which is fine because they're the most, I guess, efficient and convenient way to store this kind of data when you have records of people, let's say, what drugs they've been taking and what symptoms those drugs may cause. And so on. So I guess the idea here is that you can actually think of that very same data that exists in a relational database as a graph and store that in a graph database and apply your graph query logic in a way that allows you to answer specific questions that might have been quite complicated to answer using SQL in the relational database. So that's one example of, let's say, a healthcare scenario. I've already mentioned the financial transaction scenario, where you have a transaction graph between individuals, the merchants they've interacted with, the money transfers they've made between their accounts, how these accounts are connected. So financial institutions make heavy use of graphs to answer these kinds of questions. Traffic networks is another common use case. If you're working with city authorities and you want to understand the flow of traffic and the numbers of people moving between locations. This is something that can actually be well modeled in a graph. And the kinds of questions you can answer can also change based on whether you choose to model this as tables or as a graph. And there are many, many other examples. I will highlight one thing, though. The example of Wikipedia that I gave earlier. The idea of Wikipedia as a graph, I think reinforces the idea that knowledge graphs are a term to be used, I guess, with a bit of caution, in the sense that in general, a graph is a general representation of how nodes and entities like nodes are connected to other nodes in the network. A knowledge graph is something that you can think of as the collection of all knowledge that is available in that domain. In the case of Wikipedia, you can imagine that there are certain scenarios where it's quite hard to tabulate every bit of information. As an example, if you have, I don't know, the current president of the US is Joe Biden. So based on the current political structure and the different parties that Joe Biden represents and all the other people represented in those parties, and then the relationship between a party and a state, the relationship between the state and the country, you can imagine that this becomes a very complicated, branching sort of structure, and not all of it renders very well in a table, because you can't imagine one table that connects to another table when you have this kind of complicated data. So there are certain scenarios where actually the data model and the way you build your graph can actually really have a big impact on the kinds of questions you can ask of your data. So that's why I tend to use the term knowledge graph in a bit more specialized way. In general, I would say that you can think of your tabular data or records as a Graph very conveniently using the property graph model and model things like transaction networks and social networks and drug interaction networks and so on.

Foreign.

What's up friends? I'm here with a new friend of ours over at Assembly, AI founder and CEO Dylan Fox. Dylan, tell me about Universal One. This is the newest, most powerful speech AI model to date. You released this recently. Tell me more. So Universal One is our flagship industry.

Leading model for speech to text and various other speech understanding tasks.

So it's about a year long effort that really is the culmination of like the years that we've spent building infrastructure and tooling ad assembly to even train large scale speech AI models.

It was trained on about 12 and.

A half million hours of voice data, multilingual super wide range of domains and sources of audio data.

So it's super robust model.

We're seeing developers use it for extremely high accuracy, low cost, super fast speech to text and speech understanding tasks within their products, within automations, within workflows that they're building at their companies or within their products. Very cool. So Dylan, one thing I love is this playground you have, you can go there AssemblyAI.com playground and you can just play around with all the things that is assembly. Is this the recommended path? Is this the try before you buy experience? What can people do?

Yeah, so our playground is a GUI experience over the API that's free. You can just go to it on our website AssemblyAI.com playground you drop in.

An audio file, you can talk to.

The playground and it's a way to in a no code environment, interact with our models.

Interact with our API to see what.

Our models and what our API can do without having to write any code. Then once you see what the models can do and you're ready to start building with the API, you can quickly transition to the the API docs, start.

Writing code, start integrating our SDKs into your code to start leveraging our models.

And all our tech via our SDKs instead.

Okay.

Constantly updated speech AI models at your fingertips. Well, at your API fingertips that is a good next step is to go to their playground. You can test out their models for free right there in the browser or you can get started with a $50 credit at AssemblyAI.com PracticalAI again that's AssemblyAI.com/practicalai.

Well Prashanth, before we get into one thing that I'm really excited to talk about on the show, which I've been telling Chris we need to talk about for a while is which is graph rag but before we get there. We sort of talked a bit about graphs. I think it would be useful to kind of just give people a reminder of the kind of what most people would refer to maybe when they're referring to a rag workflow, an AI rag workflow. I was at a CIO dinner last night and one of the speakers is like, we all hear about the rag and the raggedy rag, ragged rag rags. And everybody's hearing about these things. But maybe they, maybe it's worth just a quick 30 second, one minute reminder. So when most people are referring to the rags or the rag workflow, what are they referring to?

Yeah, this is a very fascinating topic. I think about rag a lot. So let me just, I think take a step back and describe the term itself. So the term RAG itself is really interesting to me. As we know it stands for retrieval augmented generation. The key thing to note here is that the term RAG emerged prior to the emergence of the term LLM. And it's come as a result of generative language model improvements that came, I think in the end of 2019, early 2020 time period. And there were two papers that came around in early 2020, one by Google and another by Facebook research that introduced the term retrieval augmented generation. Now note that retrieval itself is not new. We know information retrieval is a field that's been around for decades, so and we've also had systems that can do keyword based information retrieval for decades. So what is new right now is the fact that generative models, generative models have become much better than they used to be. So the generation capability is what's new. So when you look at the term rag, the term retrieval augmented comes before the term generation in that acronym. So I hope that makes it clear as to the fact that the generation part is what we are stressing is the novel aspect of it. Now in 2020, the way Rag was done was to combine a sequence to sequence language model, which was the standard way of doing language modeling back then. And those models could generate text quite convincingly. And you combine those models with the retrieval capabilities of dense embeddings. And the Facebook research paper that introduced the term RAG based their work on dense embeddings of Wikipedia articles. Now you had the initial makings of vector embedding based retrievals that we are taking for granted today. And later in 2020, GPT3 was released. So you had the coinage of the term LLM or Large Language Model. And you could think of GPT3 as one of the first large language models that really extended the capabilities of pre existing models at that time. But what really made RAG take off, in my opinion, from 2021 and beyond, was the arrival of this whole host of new systems that we called vector databases. And they began offering specialized features to make retrieval from these dense embeddings far easier and also more scalable. And this is what led to the explosion of all those vector database companies that we discussed about last year. And I hope this gives some context as to the term RAG itself and how I guess it grew to what it has become today.

So I guess we've been teasing this for a while on the show here. We've talked about graphs, we've talked a little bit about RAG to get people. Everyone's waiting for us to ask you the question about, you know, graph RAG and get into the topic. So without further ado, if you want to dive in and kind of give us an intro to that, would love to hear that.

Absolutely. So let's understand what we were doing with RAG and then go into graph rag.

Yep.

So the early approach to doing RAG is, which we call it naive RAG now. And in that approach you just create chunks of your data. You embed that using an embedding model and you store them in a vector database. So essentially you just store the chunks on the chunk embeddings in a vector database. And when you do a retrieval, you convert your query into an embedding model using the same embedding model that you used to embed the data. And this returns the most similar chunks that are similar to the query vector. So you typically return like the top K, like let's say top 5 or top 10, whatever number you choose. And these top K chunks can then be sent to the LLM as context to synthesize a response in natural language. So in a nutshell, that's kind of what you could say traditional RAG does. Now, on paper, this naive approach to doing RAG is great. But it quickly became obvious that this has limitations. The first limitation is that the dense embeddings are typically done at the sentence level and many user queries use keywords. And keyword based search methods like BM25 can do a fair job at this, and they've been around for a long time. So towards the end of last year, you could see a lot of these vector database vendors starting to offer a combination of hybrid search methods and the term hybrid search itself becoming more popular, where you perform both keyword based search, which is a form of sparse vector search, with dense vector search, which is a search via dense embeddings, and you pass the retrieved chunks from either of these approaches to a re ranker module. So you had specialized modules that do re ranking that give you the most relevant chunks from either of these retrievals. And this is how you combine the sparse and dense vectors into what you call hybrid search. Now, even hybrid search can have its limitations, which is, I guess, where people began exploring further options earlier this year and maybe beyond. Because neither sparse nor dense embeddings can capture explicit relationships between entities very well. And I'll demonstrate this with an example. In certain cases, you can really benefit by modeling some of these entities explicitly. So let's look at an example of a professor and let's say the PhD students the professor is advising. So let's say you had a block of text which is talking about the students and the professor and a bunch of other things related to their work in the university. So in natural language, we understand the relationship between the professor and the student as follows. Student X worked with Professor Y because we know that the act of being a student of a professor means that you worked with them. But in the text itself, you may not have expressed it that way. The text may be written as so and so person X was a student of person Y. Now, if you try to search this using the query who did X work with? This is a very intuitive question. In natural language, we humans immediately can put two and two together that work with and student relationship are more or less semantically similar here. So we are able to piece together this information and know that a person was a student of someone and inherently they worked with that person. However, if you try to search for this using vector search, the dense embedding may not capture the relationship correctly. Where student of isn't close enough to work with in the vector space. So your vector search alone may not retrieve this answer because you didn't model the relationships in that explicit way. However, if you had chosen to model this as a graph, you would explicitly capture this relationship using this concept of a triple, which is person X worked with person Y. So this is the idea of where triples come in. A triple essentially is two nodes that are connected via a relationship. You have a source and a target, and the person X is a source, Person Y is a target, and the worked with is what represents the relationship. So the very powerful idea here is that where graphs come into this whole picture and why it's relevant to RAG is that you can actually provide additional valuable context to an LLM by modeling these relationships explicitly and simultaneously retrieving both from a dense embedding vector search as well as a graph traversal, and then using the retrievals in combination with one another to provide additional context to the generation LLM so that you can actually include this explicit relationship in your answer. And this actually has been proven in practice from some work that's been done recently.

I want to ask a question that may be a bit of a stretch, but could, as you were describing that and that failure of making those explicit, if you're doing that, could that lead to hallucination in terms of your output from the model? Because it's not able to make that explicit. And so it's still trying to provide information and it comes up with whatever it comes up with. Is that, is that a possibility there?

That's a very fair point. And you're absolutely right. Hallucination, I think is definitely a high possibility, or like it's definitely likely in certain scenarios. You obviously cannot predict when hallucination happens, which is a big issue in general with LLMs. Now I think the. I definitely want to expand on this a little bit. The selling point for Graph Rag, by the way, the process I was just talking about before is essentially what you could loosely define as Graph rag, where you bring a graph as part of the retrieval process. Now, the so called, I guess, benefit of Graph Rag, as it was, you could say, marketed in the last several months by various sources, is the fact that it reduces hallucinations. And it's important to note that whenever context is provided to an LLM for the purposes of generation, essentially you provide a prompt. And prompt is what LLM uses to provide a response. In the event of providing such a prompt, it's always possible that at some point of time you will have a hallucination. It doesn't matter whether the information came from a graph or the information came from a vector retrieval. The source of the information is irrelevant. So the very act of using an LLM to generate text means that there is an inherent chance of hallucination. So I wouldn't state that the benefit of Graph Rag is that it eliminates or reduces hallucination. What I would state the benefit is, is that it actually increases the chance of factual accuracy in the sense that a relationship that was not explicitly captured in this vector embedding is now explicitly captured in the graph. And by providing both these pieces of context to the LLM, you're essentially increasing the chances of a factually correct or a more relevant response.

Yeah, so one of the things I'm thinking about is people might have Some thoughts in their mind, like, I know maybe people have built some sort of naive rag system or maybe even implemented some advanced RAG methodologies. But most of the time what they've had is a sort of set of documents like you say, they split up into chunks, they embed those, they retrieve one or more chunks, maybe even in a hybrid way or in some advanced way. But here you're saying you're combining the vector approach and the graph approach. I'm wondering if you could break down very concretely for us, the data side. So if I, if I have like, documents or if I have internal data, what I would need to have in place and how I would construct the data side to be ready to do Graph Rag. And then at the time that I, let's say I receive a user query question in my chatbot or whatever that is, what is actually kind of concretely retrieved? And how is that combined? Or how could that be combined with the prompt into the model? So just walking us through those kind of very concrete things might be helpful for people.

Sure. No, it makes a lot of sense. So the two key stages in any RAG application, not just Graph Rag, is the fact that you divide it into an indexing stage and a retrieval or a serving stage. So to get the data in and store it and index it is what we call as indexing stage. So this is the stage that is upfront or upstream, where you have data that already exists in different structured or unstructured sources. Now, you could apply a variety of techniques, including using LLMs itself for this stage where you could extract the entities, or you could say named entities from the unstructured data or structured data that already exists and store them as entities or nodes in a graph graph database. And simultaneously, you can also extract relationships from this unstructured text. There are many different methods that you could use to extract relationships. Now, what I've noticed in recent times is a lot of these recent papers are using LLMs to help with this information extraction step where you actually use the LLM to extract portions of your chunks. And then from those portions you further refine it and say, okay, from this block of text, tell me what are the triples, which is nodes connected to other nodes by a relationship exist in that block of text. So once all these triples are extracted, they're essentially stored in a graph database. And simultaneously you can also have a parallel pipeline that stores the vector embeddings in a vector database. In some cases, you can have both the vectors as well as a graph, entities sitting in the Same database. Certain databases have those features. In other cases, you may not want to have them both in the same database. You may want to leverage the graph database for its strengths and the vector database for its strengths. And you may also have pre existing workloads that already have the data in those systems. So it's perfectly valid to have your independent sources of data move the respective data into those respective databases. For example, your graph entities would go into your graph database and the vector embeddings would go into a vector database. Downstream of this, you could do additional post processing like linking the chunk reference IDs to individual entities in the graph. Essentially, the node that represents an entity in the graph can have an ID that links it back to which chunk it is a part of, so that when you retrieve a particular chunk, you can actually point to which entities are existing in that chunk. So these are a lot of additional upfront steps that people are doing to construct both the graph as well as the vector store. And once this indexing stage is complete, the retrieval stage can actually begin, which is the stage that we are very familiar with. You have a user query that comes in in natural language, you transform that into an embedding, you find the similarity search on that embedding which returns the most similar vectors from the vector database. And then simultaneously you can also use whatever methods you have to translate that query into a graph query and retrieve the entities and relations from the graph that answer that same question. And then use a re ranker to combine the retrievals in a way that provides additional context to the LLM for generation. So again, just to summarize, the two key stages in the RAG pipeline include indexing and serving, and each of these stages has a suite of tools that you can use to help the user achieve the required outcome.

Well, our friends over at Speakeasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing docs and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with pedantic models for requests and response objects and HTTP X client for async and synchronous method calls and support for server sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API. Learn more at speakeasy.com Python again speakeasy.com Python.

So I am Here we are after break. I am still thinking about what you were telling us going into break and Trying to grock it myself. And I'm kind of thinking about how I can use it in a practical sense to help me get. Get it down and kind of get it from the notional sense into more of a practical thing that I can go do after we stop talking on the podcast. Can you give me like an example, something really hands on that, that folks out there might be doing that, that really puts it into that. Okay, I get it now. I'm going to go do it.

Kind of context.

I actually have a repo that I can share once we're done with this conversation. And I would love for people to pick this up and experiment with it. And obviously, because I work at a company that builds a graph database, I'm very eager to talk to users who are using these kinds of tools. So for my experiment, I've used Kuzu as a graph database and lancdb as a vector database, because each of these, as I mentioned, have their own benefits in their domains. And practical example that I want to demonstrate is this. So the data set I'm thinking about, which I can showcase in the repo, is you have a block of text about the scientist Madame Curie. You know, she discovered radium and polonium, and she won two Nobel Prizes, and she was related to Pierre Curie, they were spouses, and she also was related to other scientists in the whole ecosystem. So I have a text sample that contains Madame Curie's contributions to science and the relationships that existed in her life. So this is unstructured text. So the first step is we can do two things here. We can do the conventional naive rag retrieval, where I try to ask a question, who did Pierre Curie work with? And that's a very simple question to answer. And a vector search will definitely give you an answer if you embed the text into the required steps. So what I notice in this data set is there is an implicit relationship, and this is why I gave that example earlier about the professor and the students. There is one particular person in this data set, Paul Langevin, who was a student of Pierre Curie and who later had a relationship with Marie Curie after Pierre Curie passed away. So it's mentioned in the text that Paul Langevin was a student of Pierre Curie. Now, the question asks, who did Pierre Curie work with? We obviously know that Pierre Curie worked with Madame Curie to find or discover these elements. Now, we also know implicitly that Pierre Curie worked with Paul Langevin, who was his student. The vector search, if you naively chunk These and store them in a vector database, which I do in Lance DB gives me one of the answers. It gives me Marie Curie worked with Pierre Curie. But the graph search, because of the fact that I explicitly insert the relationship and I have some code that shows how the information was extracted from the unstructured data using the Llama Index framework. So it's very intuitive and easy to begin experimenting once you actually install the required packages. So what I'm getting at here is in the graph I was able to retrieve both the answers, like the Paul Langevin as well as Marie Curie, who worked with PI Curie. And rather than just using the result from the graph, there may be other scenarios where my question may have been a little bit more vague or fuzzy and a vector search might have given me a better result. I'm sure if you tinker with the data set and the questions here, you'll find such examples. So what I've done is I've included a re ranker downstream of the vector search and the graph search. And when I retrieve the result and pass it to the AS context to the LLM, I'm adding that re ranker step so that I get the most relevant graph search results as well as the most relevant vector search results. In this case, the vector search missed one of the entities, but the graph search captured it. So the combined context from both these retrievals allowed me to get the generator model to actually give me the correct response. And if you want to experiment more with this, I think it's pretty straightforward to come up with other queries that will show the reverse result to be true. Where the semantic match between the vector search and the query might be closer, you might get a more relevant result from the vector search and the graph search missed the result because of a mismatch. So that's kind of where I'm going at here. There are many ways you can combine vector search and graph traversals to improve the retrieval accuracy. So I'd love for the community to think about these individual parts. I think my biggest takeaway from what I've seen in the last few months of reading the literature and talking to people who are questioning what Graph Rag is, is that people tend to think of Graph RAG as like a graph based solution alone. Whereas the more I think about it, I think that the two approaches of using dense vector retrieval and using graphs kind of go hand in hand for the purposes of rag. All this being said, there are a lot of, I guess, conflicting articles that you may see online and blog posts Claiming that this is the way to do Graph Rag. The key takeaway for everyone should be think of it as a suite of tools and methodologies that come together to enhance retrieval in a way that you can get better generation.

I would love to maybe double click on one of the things that you mentioned which I had in my mind and I think other people might have in their mind mainly because I don't know how many years ago it was five or six years ago, I was doing some graph related data work and I know I looked at a lot of things at that time. There was even a whole area of research called automatic or automated knowledge graph creation. And there's sort of this idea that, you know, if once you have a graph there's so much that is opened up to you in terms of querying and the rich structure and all of that, but sometimes it can be daunting to construct the graph in the first place. And you mentioned kind of this nice tooling around Llama Index and of course we had Jerry on the show in the past and such an amazing project there that's helping many people. But I'm wondering if I could double click on that point. Like what has been your experience kind of current state in terms of how much work and how hard it is to do that graph construction piece in addition to. Because I think one of the things people love about Rag, right, Is you just kind of put a bunch of documents in and hands off, you construct these chunks and oh cool, like you get some, some nifty things out of it. Of course there's the element that you, you highlighted which is sometimes it's hard for people to get that last bit of performance that isn't, you know, captured by naive rag. But yeah, on that graph data construction piece. Yeah. How does that look right now? And what have you found to be useful?

You hit the nail on the head. I think the biggest issue that people I speak to are having in relation to both graph RAG and in general using graphs is how do you create the graph from existing data that you have in other forms? And yeah, I was exactly going to touch upon this anyway as my next point, which is Graph Rag is by no means perfect. And the most significant challenge is indeed around graph construction. Now there's two things regarding graph construction that we can delve into. The first is that the quality of the graph is absolutely paramount because as we know in any RAG system the quality of your retrieval greatly impacts the quality of the generation downstream. Like a poor retrieval with garbage results is going to result in a garbage output from the generation model. So, first of all, we have to stress on the fact that to get the most out of graph frag, we need a high quality graph. But then you go one step further back, which is how do you even get a graph from unstructured text? So, as you mentioned, llamaindex and I think LangChain as well, some of these frameworks offer valuable tooling to help you extract triples or entities and relationships from unstructured text. And they do this primarily through the use of LLMs. But as we know, LLMs themselves have issues with hallucinations, or they just have issues in general with reproducibility. You are not guaranteed to get the same results if you run the same LLM multiple times. And although some APIs provide seeds where you can control the reproducibility, that still doesn't mean that it's not random. The output of an LLM is still more or less unpredictable. So there are a lot of other parallel works going on that are not using LLMs to extract triples from a unstructured text source. And I can talk about a few of those that I found really exciting. And in fact, that's kind of the stuff I've been exploring in this space and I'd love to chat with people who've been doing this as well. So there are custom machine learning models coming out. One of them is called Rebel, which is R E B E L and it's been around for a while, but I think they're upgrading their internals now for a newer version. So that's a model that's trained explicitly for the purposes of extracting triples for the purposes of a graph. And there's another model, I think, called Relik R E L I K that's also an open source model that's been released recently and people have been using it to extract relationships from unstructured text. Now, it's not to say that these are mature models, it still requires a little bit of coercing and people need to understand their outputs and what they're getting from them. But the idea here is that these models are, I guess, more controllable in what you can output from them. It's not like an LLM where you just don't know what you're going to get. And of course, I think because of the fact that they're small models, you can actually use them at scale on very large data. So that's one angle of things I'm sure a lot of users have heard of the NLP library Spacy and I've been using Spacy for many years myself. It's a very like. It's an amazing library on its own. Spacy doesn't have the tools to extract relationships and entities from text. Of course you can extract named entities or named entity recognition ner but recently there have been some add on modules or libraries that plug into Spacy. Two of them which I'll name here, they're called Gliner and Glyrel, G L I N E R and G L I R E L and as their name suggests, one of them is for extracting named entities from text and the other one is for extracting explicit relationships from text. But they plug into the underlying spacy tokenizer and the underlying spacy representation of the data. So I feel like it's a lot more usable. And there are some experimental notebooks that I've been working on and I'm going to be experimenting more on this to see how they compare in relation to using just LLMs alone to extract data. So yeah, I think this is a very active space and there is no right answer in terms of how you can use these tools. But I feel like there's enough options and methods out there that people don't have to feel that I'm relying on completely black box unreliable LLM to do things.

What you've covered so far is really fascinating to me. It's. And I'm learning a lot in this conversation as we are kind of winding up on this, you know, and we've been covering, you know, Graph rag as well as the components that make that up and kind of talking about how to combine vector search, Graph search and it feels like a, you know, a brave new world that we're, we're leaping into. Even, even for the AI topic, which is that way. Anyway, where's this going? Like where, where, where do you envision this going over various timelines in the future, you know, things are happening at light speed so you just go a few months and there's a lot of change. But if you could maybe pick a couple of points out and tell us kind of what you think might happen if you're wrong. No big deal. But I'd love to see what your imagination has in store for us there.

Yeah, I'd love to see this from a year from now because I know half the stuff I'll say will be out of date or wrong, but.

Of course, but that's fine.

That's this topic, the stuff I said last year. I think I was still Spot on in the sense that I did talk about graphs and vectors last year too, and it's still relevant to it today. So yeah, I'll take my chances this time around. So I think yeah, you're absolutely right. Things change way too fast. And my personal take here is that graph RAG is at a point in time and we don't even know in the next three to five year time frame whether RAG is going to be as hot as it is today. Right. I don't think LLMs are going away anywhere. Let's face it, I mean, they're here to stay and they're going to continue to evolve. We've just seen with the O1 model that came out from OpenAI, they're able to do reasoning, they're able to do so much more than what people give them credit for. So as this capability keeps evolving and LLMs keep morphing into whatever else they become, there's no guarantee that the kinds of tasks being done today using custom models, machine learning models, won't be done by an LLM. Now some of the wild takes that I've seen involve the fact that you don't need to index anything in a database. You could potentially have your LLMs parameters store the index in a very fuzzy way and you could retrieve from that index, assuming that research goes in a certain way. But obviously that's a long way away. I don't see that happening in the next few years. What I'm particularly excited about in the next few years is everyone's been talking about agentic systems and if you look at the pivots that all these framework companies have had in the last year, LangChain, llama index, like they're heavily trying to push this field forward in terms of how agents can help build, I guess, more capable systems. Now, RAG is just one small subset of the things agents are orchestrating. They can actually do many other things related to recommendation and could say like search and retrieval and a host of other activities. So the conventional way or the standard way of doing agents has been using this framework called React, which is you could say like it's sort of like a graph based framework, but it's sort of static where you kind of decide the behavior of the system by programming in these paths and the agent acts on it and then you send it back after the reasoning step is over and so on. But where I think actually graphs are going to be very interestingly used in the future and I've seen some examples of frameworks that are already doing this is can you use an underlying graph representation of action spaces to guide your agents. That is, can the agent actually actively update the action space via a graph structure? And once that happens, you can essentially have sort of more powerful agents that are not constrained by a rigid static sort of framework. You kind of have a dynamically updating agentic system, but it's not as open ended as having this recursive agent calling that we have today. Right. You still have a framework. The graph kind of acts as like a base structure for the agent to take its actions. So that's definitely one aspect of where agents are going. But that being said, the field of graph, you could say knowledge graphs and their role in symbolic systems, they've been around, it's been around for so long and in the future they could potentially be symbolic systems that utilize graphs in conjunction with these statistical models that are based on LLMs and a hybrid sort of symbolic statistical system that combines these tools together that may not technically be called an agent, but it's still a hybrid sort of AI system in the sense that you might have leveraged the power of an LLM for the language capabilities, but then use symbolic systems to do the other tasks. All this is to say that I think graphs are way broader of an entity than what we are seeing today in terms of how they're used. There's a lot of other use cases that we couldn't go into in this discussion and Graph Rag obviously is a very, very small part of it. So I would say I'm currently excited about Graph Rag but I don't see this being like a topic that is going to be as talked about in the long term. But graphs as an entity and graph databases, these are systems that are going to actually be a core component of many systems of the future.

Yeah, it's a good perspective to have because this is such a broad topic and of course we'll look forward to you coming back onto the show to give us more of a, more of a deep dive and or reading your blog post. I highly recommend people check out the links to Prashant's blog posts and also we'll make sure and include in the show notes links to the things that we've discussed here. But yeah, thank you so much for joining again. This has been a real pleasure and I feel like I've got a lot of things now that I want to go check out and try hands on and learn more. So thanks for bringing out my curiosity on the topic.

Yeah, no, thanks a lot Daniel. Thanks a lot Chris. And I just want to end by saying Kuzu is a embedded database, so you can just pip install Kuzu and it's super easy to get started. And you can find me on Twitter and LinkedIn. I'm sure Daniel will share the links and we can always chat more with anyone who's interested about graphs.

Yeah, definitely check it out. And we'll also link some of the examples that you mentioned as well so people can get hands on and try some things. I love this element of these embedded tools out there that both let you try things locally and then even have a pathway to getting those things into production. So yeah, also thank you for building great tools. That's a great contribution. So thanks and we'll talk to you soon.

Thank you very much. Yeah, it was a pleasure talking to you both.

Thank you for listening to Practical AI. You know what's cool? Free Stickers. During the month of September, we're mailing out changelog sticker packs to everyone who leaves. Leaves us a thoughtful five star review or blog post about our pods. Simply email proof of your review to stickerschangelog.com alongside your address and we'll mail out the goods anywhere in the world. Once again, that's stickerschangelog.com pics or it didn't happen only in the month of September. Let's do this. Thanks again to our partners at Fly IO, to our beat freak in residence, the one and only Brake Master Cylinder, and to our longtime sponsors at Sentry. Use code Changelog when signing up for a new Sentry team plan and save 100 bucks. That's all for now. We'll talk to you again next time.