Graph Databases: When to Use Them (And When to Run Away)

Really excited to have Joe here with us today. The case, every time I've seen graph databases used in production, 90% of the time I feel like they've been abused or introduced unnecessary complexity. I feel like not many people talk about that. Joe has publicly done so. And so that's what Joe is going to focus on today is to talk about graph databases, when they're overkill, when they actually make sense, and what you should watch out for.

Yeah, thank you for that introduction, Hamel Husain. I have a small deck. So, GraphRAG and graph DBs, and I added a subtitle here on no single silver bullet to solve retrieval. And hello, I'm Joe Christian. I'm the CEO and co-founder of Hornet Dev. A little bit background about me and where I'm coming from. As you can tell, I'm old, I'm grumpy. I've been in the game for quite some time, 25 years. I have a master's degree from university here in Trondheim. I spent one year at Glasgow University studying information retrieval. I've always been fascinated with search. from web-scale search at Yahoo to enterprise search at Fast. I also spent a long time helping developing the Vespa platform for vertical search platforms, also quite popular kind of vector DB, uh, in that space. I did a search for GraphRAG on my Twitter timeline. It's a topic that I'm like often have hot takes and last year there was a paper from these associates from Microsoft that brought up a technique that they called, or called, Graph RAG. So Graph Retrieval Augmented Generation. And at the time, I was also commenting a little bit on this paper. And of course, a lot of the GraphDB representatives, this was like a perfect setting for them. So they want to explain why, why GraphDB is the natural choice for doing retrieval for AI. So I made this little meme. I think when you ask people about what GraphRag is or graph databases, it's a little bit like this. There's entities, there's things going on, there are edges, there are nodes, all the fussiness.

And I really feel like this is what it feels like. Like when you talk a lot of times, you're like, hey, I, they will look at, you will talk to someone and they'll look at you and say, you know what? You need a graph database.

Yeah. I'll touch on some of the stuff, but I will try to focus in this talk on a little bit on the— because I have a background in retrieval and search, and GraphRAG, like, since it uses RAG, is obviously in the retrieval space. So I want to touch a little bit on that, but also give a broader perspective about retrieval in general. And I'll talk a little bit about GraphRAG specifically. And then finally, if you need a graph DB to represent a GraphRAG approach. First, I think like R in RAG, the R is retrieval. Of course, RAG was popularized by post-ChatGPT late 2022. We saw the like really fantastic interest in retrieval and search, renewed interest, I would say. And a lot of engineers were like discovering AI. You had the whole AI engineer thing going. Everyone wanted to build like ChatGPT over your docs, put some vectors together, do some search, and then feed that into an LLM to generate an answer. And I think it's really exciting that more engineers than ever before are actually now working on search problems. But one observation I made in this article is that as a lot of people now moving into from engineering roles where you have everything is like, like unit testing, system testing, choosing a DB into more hypothesis testing, probabilistic methods. When it's not 2 equals 2 all the time, it's more challenging, right? And people start looking for single silver bullet to solve like their retrieval problems. And in the initial wave, like a lot of people grabbed for a vector database, like a specialized data database for storing and searching in vectors, right? That was the like initial movement. But what I've observed is that retrieval is actually a nuanced topic, right? There's lots of different things going on for building actually great search and great retrieval. And I think many engineers that I observe in the field, you wanna find some kind of specific tool, like, or a specific method that kind of solves the problem, that single silver bullet. And that I think is interesting, but it also creates, I think, frustration and you're always looking for the next method or the next thing. And what the heck is actually information retrieval? Sure. This is a field that's been around for decades. It's classic research field. And essentially it's, there's a user that has an information need and this user tries to type this out, right? And then. You have a source of documents and you have some kind of retrieval system, and this retrieval system can essentially be anything, right? Any type of technology. I also consider like grabbing through a file more or less a retrieval system, right? Because it's a way to retrieve information. And then there are obviously ways to accelerate or make things faster and multi cascades of pipelines and whatnot, different type of system characteristics to save cost and so forth. But. Essentially, you could look at this as a black box, right? Where there's something going on, you have some documents and you have a query expressed by a user or an agent, and you get some list of ranked documents back. That's the kind of classic definition of information retrieval. And that is really what the R in RAG is really about. First, I want to cover this actually, because I think now we also right around ChatGPT, we had models or LLMs with context size of 8K or around that, right? So it was really difficult to try to fit some information into them. Now we have Gemini, Flash 2.5, and so forth. They have 1 million tokens, but that's actually only about 3 megabytes of raw text. And, and back in the '80s, you could fit about 3 megabytes of data into two of these kind of legacy floppy disks. So I think still for a lot of enterprises and for a lot of use cases, retrieval as the way to augment a model with information is still valid, right? So I think that context, but of course there are now many cases where you don't need a retrieval component, where you essentially can feed whatever context you have to solve the task into the LLM window. And if you can accept the cost. Then that's perfectly fine. And I had this experience, I talked to an engineer at a small company and they were building like a RAG pipeline and they were struggling a little bit of how to do this. There were a lot of detailed questions about retrieval. At the end of the call, I like, okay, so actually how many do you have? And they had 300 documents and they were not large. They were like a knowledge base, right? And they didn't have a lot of queries per second. So I was like, you might be better off just feeding those documents actually into the context window, not trying to over-engineer and building like a full retrieval stack. So definitely there are a lot of applications and use cases now that where you don't need a retrieval component, right? So I don't want to be the guy that like, oh, always need retrieval, right? So I want to add this to the mix here. So then we get a little bit more on topic. So I think the first mention of GraphRAG was this pipe paper from Microsoft. And I try to summarize in 3 high-level points about this. First is that you essentially feed your entire document corpus into an LLM, right? So it's a quite expensive technique in, in, in the first place because you're essentially putting all your documents with a prompt and trying to analyze your document corpus. And then you try to build a knowledge graph of the key entities. So entities would be things like a person or, or in the, in that direction. and then you will also try to the LLM to actually find the relationships or the edges between these nodes. And then it's like organizing this into some communities, like clustering, and then at query time it uses this to traverse this graph. So I won't go into the details of GraphRAG, but the important thing here is that it's trying to use an LLM to preprocess the document corpus. But what is like a knowledge graph in general, right? This is like a fussy concept. So I'm trying to like bring that down into first principle. And from a first principle perspective, a knowledge graph is essentially consisting of triplets where you have nodes and you have edges. And a triplet is basically like the source entity and the target entity, and potentially a description that is associated with the edge between those, those. So you could represent like a knowledge graph in a text file or in a CSV file. That's the first step. And also you could represent these type of relations in a regular relational database, right? There's no magic of building a knowledge graph. But interesting perspective is that there's ways to traverse this knowledge graph. It is, if it's large, if it doesn't fit into a text file, or if it's really large, and, and how you actually process this. And that's why I will touch also why you might grab for a GraphDB. But I think actually building this knowledge graph or this CSV file to identify these entities and the relationships is actually the complex part of using GraphRag, right? So the classical way, like in Yahoo, they had a whole team, like their own knowledge graph team, trying to build like a comprehensive knowledge graph for the web because this used to be a thing that Yahoo would use in web search for these type of direct displays and It would be used as a part of search, but it's actually quite a large team, lots of data feeds and processing, and maintaining this was actually quite hard. And now with LLMs, okay, maybe it's easier. It's certainly easier to run some nice demos where you take some documents and you ask LLM to identify the graph, knowledge graph, and then dump it into some visualizer and zoom into the nodes. So it's definitely really great demo perspective, especially in enterprises are really excited about this because you find person names and organizations that you can circle around this, these kind of demos. But, and of course, could it be valuable to have on another script? Yes, of course it could be really valuable, but most times I've seen people that are struggling also with updating and maintaining this, right? So I'm not saying that this is the best way to do it. I'm not saying that building a knowledge graph is a bad thing, but it is work either through tokens and accuracy or you have to have domain experts. And then how, how do you actually make the decision if this is useful? And since I'm a retrieval expert, I know you're doing that eval course, Hamel Husain, so you can probably talk a lot about this, but For retrieval, which is my domain, a level of expertise, right, there are different kinds of metrics that we try to use to say if a certain method is better than another method. And you have metrics like mean reciprocal rank, which is a fancy name for just figuring out the rank in the ranked list or the position of the doc, of the first document that is considered relevant. So if your, for a query puts the relevant document at position 1, the score becomes 1. If you put the relevant document at position 2 and irrelevant document at position 1, you get a 0.5 score and so forth, right? So this is like a precision-oriented metric, and, uh, precision obviously is essentially about nothing but the relevant. And recall is essentially about finding all the relevant. If you somehow know that for this particular query, these are the 4 relevant documents in my corpus, then you can actually have meaningful measurement of recall. And how hard is it to build a relevancy dataset that you can actually use to evaluate methods? And I think most people should think about doing this, and that is that you either sample from your production logs, figure out what users are actually building and what actually are— sorry, users are searching for, and sample from that. And then you can think about, if you have knowledge about your document corpus, you can say what should be the relevant document for this type of query. And optionally, you can also assign like a grade level for this type of document, if it's like relevant or super relevant, right? So that's optional, I think. And if you go ahead and create like a list of these, like 10, 15, or hundreds, then you actually have a kind of evaluation dataset that is quite easy for you to actually compare different retrieval methods, right? And then you can actually say, if, if you do it this way or another way, that does it actually improve or not? And I think this step is what most organizations and teams that are playing with retrieval and jumping on. Whatever the latest method that is trending on Twitter, I'm missing out on. So I've been a huge advocate for getting these type of evaluations up before you start jumping on different technology or methods or, and technologies. Then what is then a graph DB? It's a specialized DB. There's 200s or 300s or thousands of databases in the world. That are not like relational databases. But a graph DB is like a really specialized DB and it stores the data more or less as a graph. And often this data is also stored in memory because you're doing random traversals, which is the case for a knowledge graph. You want to traverse it. And as you traverse edges in this graph, you could potentially do that at random. And then it's a really, if you want to have low latency, you essentially need to have this data structure in memory. And this is more of a first class, treating all these kind of links and connections in a graph structure as a first class citizen. So the query language or the types of queries that you can execute on this type of databases is more, what can I say, richer than if you do with a simple or a simpler relational database. So that's the huge benefit of using actually a graph DB, and I think Neo4j is probably the most popular specialized DB in the graph DB segment, and there are also others, but I think that's notably mentioned. And I think the really question is, do you need a graph DB to support searches in a graph? And the answer is no, because a graph is essentially, as I said, triplets. So you could store this in a simple file. You could traverse this data structure writing a small Python script, or you can use a relational database. But the questions I think you need to answer is one is, do you really need really fast and efficient traversal of graphs? And how, of course, there's also the question of how large your graph is. And then, do you— does your use case really require, like, multi-hop? Do you need to go further than, for example, friends of friend in your query? And then you have to balance that if you're going to add that type of specialized DB into your serving stack, right? You have to think about the complexity of adding another specialized DB, and then of course the potential effectiveness in terms of does it increase accuracy, which you essentially can evaluate without adding it to the stack by using the method that I mentioned. And the other thing, of course, if you actually need to have this kind of fast search. In short, in essence, you don't necessarily need a specialized DB to accelerate searches in, in the graph, but it might be that you are— it's perfectly okay to just use. And I know there's an anecdote around Facebook. So in the early Facebook days, they had a pretty substantial graph and they used, they implemented on top of MySQL. That was, I'm not saying that they have also evolved on, on the use cases, but you don't necessarily go need to go specialized in the first place. And I think it comes back to what I touched on at the start is that a lot of software engineers are looking for this kind of mapping. Between some kind of technique and then, uh, database or engine to solve that. So I think GraphRAG is actually the best example where this technique appeared in a paper claiming some accuracy improvements in, uh, some specific datasets. And then people would associate GraphRAG with a graph DB, right? And then of course all the graph DBs really promoted this type of techniques. Then I'm arguing that many software stacks, search engines, relational databases, whatnot, or a text file can actually represent simple graphs. And the fun fact here is actually that HNSW, Hierarchical Navigable Small World graphs that are used inside vector databases to accelerate vector search, is also building a graph data structure. To accelerate the search for nearest neighbor in vector space, where the similarity between the vectors is used as these kind of relationships between the nodes in the graph. And then I say that there is no single method that solves, like, all possible retrieval problems. Of course, it'll— main knowledge about the documents that you're actually going to search on. And if you understand the structure of these documents, if they can contain entities and things like that, can be used to improve the effectiveness of search. A lot of search practitioners out there that's been building search has been building out this suite of different tricks that you can use to boost if something is in the title or if it's, if there's specific entities that are mentioned in the text. And you just append those in that information into the document model. So that there's really nothing new with kind of annotating and expanding documents to make richer and go from essentially unstructured data into a more structured form. And I personally think that's the big thing with GraphRAG, to use LLM to actually analyze and use a lot of processing power intelligence to, to map from unstructured into structured. And we even see that type of use case also for chunking, right? If you can actually afford to take a whole PDF and ask the LLM to, how can I optimize the chunking and identify entities and whatnot, right? That's one technique to more, more intelligently process your documents without having a lot of heuristics. And I think that's actually the main takeaway that people can have from the Graph RAG paper. Okay.

Like on the last slide, it's not really spice. It's just something that you want to screenshot and put a box around. And I really like this bullet point about vector search playing the similarity score nearest neighbors being a, also a graph structure. I don't think people realize that. And I think if you reflect on that deeply, I don't, I'm not sure about that, but I would add another fun fact.

Is that there is an interesting paper that was last year in the— was it last year? Everything moves so fast, I'm sorry. But where essentially in information retrieval, there's something called the cluster hypothesis. And that is something along the lines, if you find for a specific query that a document is relevant, you might want to actually explore if there are other documents that are similar to the document that is relevant. And that's an interesting technique that several researchers from Glasgow University has researched, which can actually improve effectiveness. And that is that you first do a candidate retrieval, you find results, and you identify which of the ones that are relevant. And today we have LLMs that are also quite good at actually assessing this. and then you can do another round of retrieval where you essentially say that, hey, I found these 3 documents that are relevant, and now you change the query to search for vectors or documents that are close to the vectors that represent these relevant documents. And then they actually can improve recall by that. And that's one additional step, and that's essentially using this graph as a part of retrieval. And this is fully possible to represent. On top of all the vector databases and all the search engines out there to do this one additional hop. So you could consider that like also a form of graph RAG in a way.

Yeah, I think it's a powerful thing to think about because like you would hope that if certain documents are semantically similar, are similar in whatever representation you have, you would, if you have a good graph structure, you would hope that it would be close, like, as well. And if there's too much dissonance between those, then maybe that's a smell that like, hey, your graph structure is not really representative, or vice versa. So I think that's a really good bullet point.

I also think that this is the, by the way, the last slide I, I have, and that is that is that now with more powerful LLMs and faster LLMs, and where we see intelligence drop— cost of intelligence dropping to zero— it's not true, but still, for a lot of different use cases that you can use LLM both in the pre-query phase, where you take some task and you map that into using the LLM to a query plan, more or less, to How can I search my data? And leave more of the decision around how we're gonna search the data and do more towards agentic retrieval phase. That's a direction that I'm more generally enthusiastic about, I would say. And of course, same thing there. You need to evaluate if the end results is better than some simple keyword technique like BM25. So it's, it's, Essentially, if you want to try to improve something, the first thing, you know, how can I measure if I'm improving it or not, right?

Can you talk a little bit about how— why is it that engineers get nerd sniped very easily into graph databases? And what is the impact of the technical debt that you swallow if, if you don't need a graph database? Can you talk about it?

Yeah, I can probably touch on that. I think there are several. Let me try to touch on the first thing, on why we're grabbing for our tools. I think this is like a general case, like, where we as engineers like to map some kind of technique into a specific technology. If you want to do this, then you use this technology. If you want to do that, then you like use this technology. I think if you need a relational database, you're probably going to choose Postgres, or if you're going to do a search thing, you're probably going to grab for Elasticsearch, or you will grab for these kinds of tools that you know of. And essentially, I think DevRel marketing to developers works. So that's one aspect. Of this. And in terms of adding specialized databases, it depends on your organization and the level of maturity, how complex it is to add another database to the stack. Of course, then to think about maintaining that, if you're using a hosted service, then you're paying for that. And essentially then you also have to keep that data up to date. You're ingesting new documents. Do you actually need to update your graph? What's the quality of the graph. So of course there are operational constraints before you do that step. I think even if I'm building a retrieval engine, of course there are a lot of different use cases on a smaller scale that where Postgres can really, you know, you can really solve a lot of problems just building on top of Postgres, including GraphRag, including vector search, including text search. So when is the point where you actually grab for something more specialized. I guess it depends on like how valuable the thing that you are doing is. Is it a hobby project? Can you fit everything into a free tier of Supabase or whatnot? Then go ahead. But if you want to grow past that, yeah, you need to start thinking maybe. Yeah, that's another interesting observation in terms of vector databases and specialized databases. And is that— I think vector databases is a good example here in that category is that a lot of people jumped on that because they thought this was the only way to do it. And then essentially, as different technologies caught up, even pgvector in Postgres was actually outperforming a lot of vector databases in even in raw performance. Even for most use cases, you don't need like a raw vector search performance.

So the title of this slide is No Single Bullet, and one of the common questions that are, uh, people are asking are, hey, is Hornet the silver bullet? Can you tell us about what you're doing?

No, definitely Hornet is not, is not going to be a single silver bullet. In Hornet, we believe that gradually, that in terms of workload, the workload changes a little bit. We think that AI and agents specifically they will do more queries than humans, both driven by agents that wants to, for example, search the web, or agents that wants to search your local enterprise knowledge base or values or policies or law documents and things like that. So it, we, that's the, like the fundamental hypothesis we have. And we also think that retrieval, there is no single silver bullet for retrieval. It's hard to actually build great search and that also quite hard to operate the classical search engines that are out there. So we wanna make it a lot easier to build great retrieval and to evaluate so that you actually know you're building great retrieval and that you can have multiple different techniques to choose from and that you can get some help in terms of performance and scaling because also agents are now quite good at both. Using and querying and operating things. So yeah, that's like general what we're doing, but.

Okay. So an engineer watching this talk, they may have gotten turned on to the fact that graph databases sometimes are overkill. What are some smells that you should look for? Let's say you're watching this talk and you were wondering to yourself, maybe what I have is overkill. What do you suggest for those people to do or look at or smell test?

I, I, this is biased, but of course, if they haven't already taken in your eval course, I think that's the first step, right? Because I think everything boils down to measuring if whatever you're doing right now is actually working, right? And this could be like ad hoc, just testing the stack that you have and looking at the failure cases and whatnot. But for retrieval specifically, having these kind of golden dataset for running evaluations, and then you can start answering questions. If I tune this knob, will something improve or not? I think that's the fundamental thing. Of course, if you added like 10 microservices and 4 specialized DBs, of course then, and you're not operating at a Spotify scale or Yahoo scale, or you have an income on the project, then yeah, then already there's a red flag.

And when you've seen like good graph databases in production, or cases when that makes sense? Do you usually see it like backed by evals of some kind? Hey, we're hitting some upper limit here in either performance or latency or whatever. And then they have some kind of justification for that, or how does it usually look?

I, I have to say that I have, I have never encountered actually a graph DB in, in production in, in my experience, but I haven't worked for a lot of different companies, but I talk to a lot of different companies and in most of my conversations with people, I never hear that, yeah, we are using a specialized graph DB for this, but, and even I worked with the Spotify team and they are known for running a lot of different databases, specialized database, but I never can recall like mentioning a graph DB. They might have one, I can't know. So I, unfortunately I don't have a lot of experience with this. What I do know is that—

So out of curiosity, like when you worked at Yahoo, were they, did you have, you, were you doing like the sort of graph modeling just in a, like a relational database or did you have, what did you use there for this structure?

Yeah. So in, in my Yahoo days, I spent, of course, all my time there working on the Vespa platform that was developed and used heavily in, in, in Yahoo. And as far as I recall, in terms of actually the knowledge base that was used for Yahoo Web Search, most of that was represented represented in Vespa as a part of the serving stack. Ah, okay. So not using a specialized, specialized database. I'm not sure if that's a, it's a, that's a good data point, but yeah, we can wrap things up.

But before we do that, you might be hiring people at Hornet. It's a good chance to, if people want to work with you, how can they do that?

Yeah, the easiest way to, to work with me is to DM me on Twitter. It's a lot easier than sending me an email. But I will add the next step. So if you follow me on Twitter, you can DM me there and you can email me and we are hiring and we are hiring in a GTM role in San Francisco and we will have also more roles coming there from product marketing to sales. So we are expanding the team in the US. So if you're based there and want to work in what I think of course is one of the most exciting startups in this space, then I would love to hear from you.

I second that. If you're interested in working in retrieval, there is few, there's few people that are better to work with than Joe. So you should really jump on this opportunity. I really like Joe, his writing and everything shows what he's, he knows what he's talking about. I highly recommend.

So yeah. Yeah. And if you, thank you so much, Hamel Husain. I really appreciate that. And if it, I love a good, good thread on X. Too.

So how he loves graph databases from time to time, you'll get really good reaction if you do that. If you want to troll Joe, it'll be entertaining.

All right, fantastic.

Thank you.

Okay, stay in touch. Bye-bye.

Thanks. Thank you.