Vector databases (beyond the hype)

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@flyio.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder of Prediction Guard and I'm joined as always by my co host Chris Benson who is a tech strategist at Lockheed Martin. How you doing, Chris?

Doing well today, Daniel. How are you?

I am doing so good because a lot of my dreams are coming true in terms of topics to talk about. I've been wanting to talk about vector databases on the show for quite some time. I know that we've mentioned them, but we haven't had a full episode on them. And I was scrolling through LinkedIn and saw a set of amazing posts and very practical posts about vector databases that I quickly shared and also sent a message to Prashanth Rao, who is a senior AI and data engineer at the Royal bank of Canada. Welcome, Prashant.

Hi. Hi. Thanks so much for having me.

Yeah, yeah. Well, now you have a three part series on vector databases, a three part blog series. What makes each one different? Understanding their internals and not all indices are created equal. I hope we can get into a bunch of that. But maybe to start out, could you just let us know what a vector database is and in particular, why are people talking about them now for sure.

So I think the way I want to answer this question is I'd like to break it down into parts and answer each bit sequentially. Great. So to answer what a vector database is, well, let's start with what data is in the first place. Right. The definition in my head is data is an organized collection of structured or semi structured information and it's stored digitally in a computer. Now when you have data, you need somewhere to put it. So that brings us to the question, what is a database? So a database is the system that's built for easy access, management and updating and also querying the data at hand. We also need to talk about what vectors are. Vectors, you could call them a cloud, sort of compressed data representation that contains semantic information about any underlying entity. It could be text, images, audio, anything like that. So now we put all of these things together. What is a vector database? A vector database is a purpose built database that efficiently manages, stores and updates vectors at scale. I think the scalability is a very key factor there. And it also retrieves the most similar vectors to given query in a way that considers the semantics of the query. So I think all of these terms holistically come together to form what you understand as a vector database.

And when you say semantics, what do you mean in terms of semantics and how that maps onto a vector?

So I'm sure everyone, most listeners are familiar with the concept of language models. LLMs are everywhere. So the thing with semantics is typically a query that you have. Like, if you write a query to like a search bar on Google or something, you're thinking in terms of keywords. You're just thinking in terms of, okay, I want this particular thing, this item, whatever you're thinking about, and you type the word in there. Where semantics comes in is, did you type in something along the lines of what the data itself has? So can the query actually translate into something that the database actually understands and produces the result that is most meaningful to the query that you put in? So it's not just about the words or the features of that word, it's also about the meaning of that word and how that comes together in the underlying internals of the data.

Cool. So before we keep going, just because you have developers and data scientists and they've worked with kind of all the other database types that most of us have worked with for decades and we have multiple times over the years had to kind of like understand the new thing that's out and what the, you know, what the value is.

No SQL.

There you go. And so, like, I'm going to jump on that. You know, we started with the SQL query language with these that are for relational databases. And then we went to NoSQL, which there are variants of and things called object databases. I understood your definition of vector, but. But I didn't understand how it related to the utility or lack thereof in some of those other approaches. Could you kind of lay the groundwork or the landscape of what that is?

For sure. So yeah, I mean, I'm a total database junkie. I love thinking about the various kinds of databases out there. So actually, before we go into that, a quick summary in terms of where I'm coming from. Right. So I started off as a data scientist, so I'm fully in your world, Daniel. And it's been a few years down that road for me. And I think for me I've hit that point where I've been lost in the world of Models and hyperparameter tuning and actual data. Scientists would elate with that. But the more I began thinking about it, there are people who have entire PhDs in database theory and their implementation. But then the more I worked with data, I realized that you don't need a PhD to understand enough to build a working application built on top of a database. That's when you began thinking about what exactly are these different flavors that you have out there. Of course, we've all come across SQL databases at some point in our careers if we worked in tech. So to answer that question, I think the general history of how these things panned out is quite interesting. I believe the origins of SQL databases come from way back in the 70s, I think when this field called relational algebra was formalized. It's kind of like, you could say a formalization of the mathematics around what it means to join data, query data, store data in a database in a way that is queryable. I think SQL databases are so mature, so tried and tested, and the reason they've withstood the test of time is because they view the underlying storage or the underlying data as structured. And in many cases you have structured data that is in the form of transactions. And what a transaction basically means is some event happens in the real world and you log that information and you essentially build up a sequential chain of data, which is basically a table. And that's kind of what the relational data model came from. And where relational models get interesting is you have tables that are related to other tables and that kind of maps into real world complexities where not all data is independent. Like some of the data depends on other things. Like a person, person's metadata could depend on what company they work at and things like that. So, so that's how relational data kind of became the norm. People were gathering data from digital systems and then putting them together. And SQL became the sort of standardized query language that you could use to query data. Fast forward to mid 2000s and the NoSQL movement starts to pick up. And where that comes from is there's a point beyond which relational data modeling can become a bit inflexible. Like it becomes a bit rigid. Because in the real world you have data that comes in from various sources. Now some of that data can come in very rapidly. With the advent of big data and streaming and all these rapid ways of gathering data that we have today, it became very obvious that the schema based approach, a schema is basically what kind of data types exist in your table, right? So the way relational models were built was you needed to define a schema and the schema kind of was the ground truth. The data has this data of this type only and that's what you expect in there all the time. I think the NoSQL movement sort of built on top of the limitations of the relational approach of being pre decided by a schema, because to be truly flexible in terms of the massive amounts of data coming in from various systems, you need to have a schema less approach at times. And the schema less approach basically means you store documents, you dump data in semi structured JSON blobs and things like that in a scalable way. And I think horizontally scalable became very, very important in that period. The earlier databases that were relational, I think they were more vertically scalable in the sense that you could just add more and more compute and you essentially scaled up your data that way. But now with NoSQL, the idea of distributing the data as documents across multiple machines and having those machines communicate with one another, that became a new paradigm. But I think the challenge with NoSQL is because of the underlying nature that the data need not necessarily be dependent on itself, like in the sense of relational tables. They didn't adhere to the SQL language standard and they kind of diverged. MongoDB was among the first and there were many others that came after it using JSON based query languages. So there was a big bifurcation, I guess in you could say the database community, when on one hand you have SQL enthusiasts who swear by the declarative nature of SQL and then you have the other community, NoSQL, who uses JSON essentially to query the database. They claim it was developer friendly and JSON is a developer friendly interface language, agnostic and so on. So in some ways it does have its benefits. But then depending on your use case and depending on what you're trying to do, there are people who will argue on both sides that SQL should be the only thing you should use, or NoSQL should be the only thing you should use, and so on. So does that clarify aspects of both those camps before we move into the modern ones?

It does. And then if you could distinguish as you go kind of how vector is different from those others, that would be helpful for me. I know. And maybe some other other folks in the audience.

Yeah. And I think maybe one thing that I loved about your blog post is I see some of the players from the world that we just talked about represented within that landscape, and then also some that I'm not familiar with or at least that I've seen only recently. And, and so you've got these different axes, like postgres, which is a SQL based query language to a relational database, has some part to play in this vector database ecosystem, but then others which seem to have their own query language. So maybe you could also start to break down for us. So we want to store vectors in databases. Now to do these sort of semantic queries, does that need to be stored in one or the other of these types of databases that you've talked about developed over time? Or how has that happened? And what are the sort of major categories of players in the vector database space?

Absolutely. So I think, yeah, before we get into the specifics of databases, I think to answer Chris's point, we definitely do need to talk about the evolution. Right. I see that vector databases are a natural evolution of the NoSQL class of databases. If you imagine a Venn diagram, you have like a circle that represents SQL and the other circle represents NoSQL. You have an intersection, that intersection point. I believe they're called NewsQL now, I'm not sure if you come across that term. It's Quite interesting. But NewsQL, they technically use SQL like languages, but they also claim horizontal scalability and a bunch of other things related to asset compliance and all the other things. So it marries the benefits of both SQL and NoSQL paradigms. I was thinking initially, where do I place vector databases? Does it go in that intersection or does it sit purely in the NoSQL camp? Then I imagine this. As you extend that circle that has NoSQL, it becomes like a blob, like a fuzzy, amorphous blob. NoSQL is huge. And I think in my head vector databases are like an extension to NoSQL and why they came about to understand what vectors are and how they're stored in database. I think it's important to understand what search is and what essentially you're doing when you query a NoSQL database. So where it comes from is in the early days, I guess people were just submitting an exact query using a JSON sort of query language, like how MogoDB has. And that query has to have all the terms or parameters in there that tell you what you want to fetch from the database. Right. In a SQL world, it will be done with a declarative query in SQL, whereas in NoSQL you typically do it in JSON. Over time, I think the idea of full text search became very important because I think everyone wants to be able to retrieve information from massive blobs of data sitting around. And how do you query that? Right, if it's in a NoSQL sort of format. If it's not, you can't write a SQL query to retrieve it. How do you get that information? So the idea of a full text index came about. And what essentially that is, is it uses the concept of inverted indexes. Inverted file indexes, sorry, where you consider the term frequencies of terms that appear in a certain document and obviously the relative frequency of how often those terms exist in a document versus the entire data set. So you combine all those things together, similar to how TF IDF is in data science. There's an algorithm called BM25 which is the most popular inverted file index algorithm. It's the most commonly used one for full text search. So the early days of search involved how do you scale that up because you have massive amounts of data, how do you build that index? Very, very efficiently. And then the querying interface sits on top of that. So you essentially submit a query saying, okay, I want so and so term in the keyword that you put in. And the inverted file index, the BM25 algorithm, it considers the words frequency and it considers subword features and a bunch of other things to intelligently retrieve relevant documents that contain that term while also throwing out useless words, stop words and things like that. So it was more of like a bag of words sort of approach. If you consider an NLP analogy, it's kind of like a bag of words way of approaching text. Now fast forward a few years. I think ever since the transformer revolution happened, people began observing the obvious power of transformers in encoding semantics, right? A transformer is way better at isolating meaningful terms in a document, especially when you're doing things like classification retrieval and so on. So how could you merge those benefits of a transformer with what you have in a database? So I think vector databases, the term got coined, I think much later after transformers came about, it was mostly called search engines. Before that, a more generic term, I think a catch all term for anything that involves search. But nowadays I believe search engine refers to a more like you consider semantics as a key component. So essentially vectors are the only thing that can do that. So to really describe what a vector is, essentially you have a language model, typically a transformer based language model that you use to embed the representation of a sentence into tokens. And the representation is stored as a vector, the vector that you have essentially for a particular sentence. Typically those are done using sentence transformers, which is the most common kind of model you use that essentially embeds the entire semantics of that sentence in the vector. And then the way this scales up is you consider the context of each and every token in that vector in a way that when you submit a query, the semantics of the query are mapped to the vector in your database and you can find a similarity between what you entered as a query and what exists in the data. So a vector is a very powerful way of, you could say compressing the representation of meaning in a sentence or a document in a way that scales up numerically. And you can rapidly query that in a digital.

This is a Changelog News Break We've talked about prompt injection quite a bit since ChatGPT ushered in the LLM era. In brief, that's where you handcraft a prompt that tricks a chatbot into not following its own rules. Well, new research has uncovered some new LLM attacks on the block which aren't exactly that large. Language models like ChatGPT, Bard, or Claude undergo extensive fine tuning to not produce harmful content in their responses to users questions. Although several studies have demonstrated so called jailbreaks, which are special queries that can still induce unintended responses, these require a substantial amount of manual effort to design and can often easily be patched by LLM providers. This work studies the safety of such models in a more systematic fashion. We demonstrate that it is in fact possible to automatically construct adversarial attacks on LLMs, specifically chosen sequences of characters that, when appended to a user query, will cause the system to obey user commands even if it produces harmful content, end quote. The biggest difference here is that they're achieving the jailbreak in an entirely automated fashion, and they make a case for the possibility that such behavior may never be fully patchable by LLM providers.

Game over, man.

It's game over. What are we going to do now? What are we going to do?

You just heard one of our five top stories from Monday's changelog News. Subscribe to the podcast to get all of the week's top stories and podcasts. Pop your email address in@changelog.com News to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com news.

So Prashanth, you kind of alluded to this and I think that explanation was amazing of how this vector based semantic search really exploded around the time that Transformers and large language models did. I think even in this past, let's say year, there's been this huge explosion of interest in Vector databases, could you maybe describe a little bit? So we know that you can search a vector database to find similar statements, let's say, or similar chunks of text where the similarity is based on semantics, or how are people using them with regards to their AI workflows? And how does that kind of correspond to what's sort of popular right now in terms of what people are exploring with AI?

So I think I need to highlight the fact that I'm both fascinated and frustrated by the current state of marketing in vector databases, both at the same time. I'm genuinely interested in the use cases, don't get me wrong. When combined with LLMs large language models like ChatGPT, you could say any sort of language model layered on top of a vector database can be used to build some very, very interesting applications. One of those interesting applications is querying your data by a natural language. I think this has always been a dream of data scientists and people who work with data. Right. Rather than writing my query by hand or constructing the query painstakingly from the ground up, can I just talk in natural language and have the database kind of respond to that query in natural language as well? The application will be built using an NLM at the core, and essentially that would be powering the whole translation of human instruction to machine instruction and back to the human. I could go into the details of specific applications, but one thing I do want to maybe throw back at you is I know this is a practical AI podcast, right? So I guess what I was hoping to get into is I have an idea for a fourth blog post and in the series, basically part of it is the trade offs, right? What really interests me about the various vector databases out there and why I began writing about these at length is when it comes to understanding what tool to use in the real world, when you have a business problem, when you have a particular case you're trying to address, obviously there's tons of information out there. You could go out and read a bunch of blogs and papers and come up with your trade offs. But I think it makes sense to actually walk through some of these trade offs. And my understanding is that as you go through these trade offs, you actually begin formulating the actual, the value of these things much more clearly. And in my head, I think it makes sense to talk about the use cases once we go through some of these key trade offs, because in many ways, using a tool depends on what goes into it and what you thought about the different options that you would use.

You can dive right in because, yeah, I had follow ups which were essentially what I think you're about to cover anyway. So I'll just leave the mic with you, man.

For sure.

Yeah.

So basically it makes a lot of sense to write about this and obviously read it at your own time, but this is a great place for me to begin talking about it and eventually I'll put these down in words as well. So I've broken these down into I think roughly eight categories and the trade offs I'm specifically speaking about. What do you need to think about when you're thinking about a database? And this will answer exactly what you talked about earlier Daniel about the so the first thing I think Daniel mentioned is the idea of deciding between existing databases that have been around document format and things like that versus newly designed databases specifically for vectors. So I'm going to call it purpose built vendors versus incumbent or existing vendors. So I think it's very important to understand in many cases you might just be looking to add semantic search capability or just retrieving information using semantics on top of an existing application, right? And that existing application could very well be built on a well known tried and tested solution like elasticsearch, Postgres and so on. There's many solutions out there and obviously in those cases it makes sense to just say hey, but why can't I just leverage the vector index or the vector storage of that database itself? Like for example you mentioned Postgres. 1 real big concern with this is if you look at some of the material online on the performance of these the methods of PGvector. PGvector is basically the vector plugin add on to Postgres and there's been enough documentation about this. But essentially the way it's been slapped on to Postgres is as like an add on. It's like built by a third party called Supabase and they add a vector functionality to the existing engine that postgres has. So by its very nature, because it's not tightly integrated with the underlying internals of the database itself like the storage layer, the indexing and all of that, you're going to miss out on a lot of optimization. Not you, but the technology is basically not optimized from the ground up to speed of indexing, performance during querying and so on. And this has been well documented. So that is a very big concern. Depending on your use case and how much accuracy and what quality of results you want, are you better off using an existing database that you already have in your stack or actually bringing on a new tried and test purpose built database for that very reason. Right. And from my experience I've been tinkering around with quite a few options out there with purpose built vendors. In my opinion, they are always a better solution in terms of scalability, efficiency and also accessing the latest technology like the latest algorithms out there. What indexing algorithms are out there, how do they get the best bang for buck in terms of your speed of indexing, the quality of query results, the latency of those results and so on. So I feel like in the long term, if you actually are serious about building a vector search or large scale information retrieval system that considers semantics, it makes far more sense to think about a purpose built solution. Many, many database solutions are out there. I've listed some of those in my blog and I think those are going to win out over the incumbent vendors who have kind of built vector offerings, as you can call them.

What we're talking about is exactly what I had hoped we would talk about in this episode because your blog posts were so we're so practical in terms of how you think about the infrastructure that you work with day to day. Would you recommend? Because sometimes you don't know how much you need to optimize at the beginning and you can over optimize. Right. So would it be a valid maybe stepping stone to say if I'm already working with postgres, I could try out the vector capability of that and if it works for my use case and I don't have, you know, 3 million documents that I'm searching over, maybe it's, maybe it's fine. I just have a, you know, I'm doing my personal blog or something and then kind of optimize as you hit a wall. Or is there danger in kind of trying to make that put a square peg in a, in a round hole sort of thing and get yourself in trouble?

You hit the nail on the head. I was going to exactly say put a square peg in a round hole because I face those issues myself. I won't name exact database vendors, but I work with SQL and NoSQL databases which obviously have vector solutions. I think the challenge and the issue with saying that okay, I already have something that works is you got to remember that every single database that has existed for I think more than 10 years, databases come with package and they have their own tech debt that is associated with the underlying programming language they're built on. There's years of decision making and architectural decisions under the hood that they've taken to implement solutions the way they have so they can't just throw all of that away and then build a vector solution that is optimized from day one. Right. It's going to take a fair amount of time before these incumbent vendors are able to optimize their offerings to a point that perform as well as purpose built vendors. Because these purpose built vendors have spent thousands of manners, I guess, per offering in just tuning and building for a very specific goal. So what I've noticed in my experiments is that a lot of features that you take for granted in a purpose built offering are not even available in the existing solution. Like PGvector is a very, very young solution right now. Elasticsearch is vector offering. I worked with that as well. It is also, I mean considering elasticsearch has been around for so long, they only released their first vector like ANN algorithm I think last year, like 2022. So in terms of a database's capabilities, that's very, very young. So I would say like there's a lot of things that you could potentially be missing or lacking and I'll cover some of those in my other trade offs that I missed as we go forward.

Yeah, yeah, let's go on to those. I'm curious what number two is.

For sure the number two is I came across this in my first blog and reading some of the comments on there and one of them brought up this fact that the trade off between using a database that allows you to build your own embedding pipeline or versus using a built in hosted sort of embedding pipeline. And by that I mean how do you generate these embeddings or these vectors? Right. Many people are familiar with sentence transformers. It's available on Hugging Face and a bunch of other open source platforms. So essentially it's quite easy, or you could say it's trivial to put your data into these pipelines and generate sentence embeddings that you can just use to ingest into a database alongside your actual data. So you have your document data that has all the fields and attributes that you have in there alongside the vectors that encode the useful information in that that you want to query on. So that's a relatively trivial thing to do. But there are certain database vendors who offer convenience features on top of that where they embed the API of these models inside their own offering. So if you're just getting started and you don't know much about how vectors work or how LLMs work or any of these things, that might be something to consider. You might be better off using something like weaviate, which has pipelines built in where you can just tell it, okay, connect to hugging face so and so model and it will build the embeddings for you as opposed to you writing your own custom transformer pipeline that actually takes in the vectors, generates the vectors, and so on. Now, if you have experience with transformer models, you might be far better off in doing all of the embedding work upstream, parallelizing and optimizing that portion, generating those at scale and optimizing from a cost perspective, getting those done with the least resources and, you know, most quality that you can, and then just sending the vectors over to your database. So this is an important thing to consider, depending on the level of experience that you, your developers have in your team to actually bring bring the vectors in.

Gotcha. That makes perfect sense. What are some of the other trade offs?

So then the other thing is the two key stages, right? You could say you could break down when you use a vector database as a developer, the first stage is the input, which is essentially building the index, right? I go into the indexing methods in a bit more detail. That's not really a trade off. It's more about knowing what indexing even does under the hood. But what indexing means is you have data that you need to encode into a vector. Right? Now, it's not as simple as just dumping a vector, which is like an array of numbers, onto your database. You have to be able to search through those vectors. So the goal of indexing is to design efficient data structures and store the vectors using those efficient index data structures in a way that they can be queried efficiently and at scale. So that is an upstream process. You do that once up front, you bring all your data in, it's indexed, and now you have a bunch of vectors in there that are searchable. The downstream portion of that is querying, right? It's basically like inference in nlp. The query stage involves you taking the user input, transforming that into a vector, just like you did your raw data. And the vector embedding that you use, there is an embedding model that you use to obviously transform your data so that they're compatible. So that's a downstream step. You're clearly separating the indexing step from the query step, right? So the trade off here is, is your database optimizing for indexing speed or query speed, or is it mature enough that it has optimized for both? And if you look through all the offerings out there, many of the existing vendors have focused more on one end of the pipeline and not so Much on the other. Some of them are faster at indexing and not so much at querying, but some of them are way better at querying and much, much slower during indexing. So generating that index actually can be a very expensive step because it's not only about using a sentence embedding model or a transformer, it's also about the database being able to translate those vectors into an index and it can actually query. So depending on the size of your data, this could take hours or even days. It's not unheard of from here of indexing periods of the order of days. And of course, depending on the amount of money you're throwing at it, you could go use GPUs to speed up the vectorization and use multiple parallel instances of the database to scale that portion up. But that's exactly it. The trade off here is how important is indexing speed if your data is coming in in a stream at a very rapid rate. It's important to consider indexing speed as an important criteria. But then if you're not so interested in dumping large amounts of data very quickly, but more interested in serving results to a very large number of users asynchronously, then query speed becomes very, very important.

I know we don't want to necessarily call out certain players in this space, but I think a lot of people are already familiar with a lot of the names here. So maybe if you could just highlight from your perspective what are maybe some of the ones that are maybe more like you were saying, mature in how they're thinking about both of those phases, whereas maybe certain ones that are optimizing more on one side or the other, which like you said, depending on your use case is going to be a good thing or it might be a bad thing. So it's really about use case. It's not so much about the goodness or how amazing a certain offering is, but more about use case.

Yeah, absolutely. So as you say, I'm not going to call out specific. I mean, to be fair, everyone, every vendor makes trade offs, right? They themselves are obviously juggling a lot of their own trade offs when they build these things. But I obviously haven't used every single one out there. But the ones I have worked with, the most mature ones. I think Milvus is an open source, purpose built database. It's been around the longest, among the longest, I think in the vector database market. It's extremely scalable. I mean, I like to think I've written in my blog, I call it Milvus Throws the Kitchen Sink and The refrigerator at the vector problem. So it can really handle billions of data points. I mean it's designed for that and obviously it has had time, it's been around for about four or five years. I wouldn't say that that would be my go to first choice. That's my own personal preference to be honest. It's more about, I guess, usability, how accessible their Python client is and so on. Then other vendors like Mediat and Quadrant, I think they're also very, very optimized for this reason. So you could say that these are very, very powerful solutions. They scale really well, they ingest data really quickly and they also supply query results very quickly and relatively accurately as well. To be fair, I think the existing database vendors like elasticsearch, Postgres, they're not there yet in terms of the speed and that's partly because they're general purpose databases, they're not specialized vector databases. So it makes sense that they have to deal with other priorities and they cannot optimize for all of these things with the laser focus that, you know, purpose built vendors have.

Thank you so much Prashant for helping us start to pick apart some of these trade offs. And I'm starting to structure things in my mind in a useful way, which is, is really great because I've also been exploring a lot of these and I agree with you, there's, there's a lot of also new entrants into the field that show a lot of promise, even the ones that aren't quite as mature yet. What are some of the other you mentioned eight I think. I don't know if we've been through three or four yet. I wasn't keeping track.

I might have to speed things up a bit.

Just kind of list them off at least.

Yeah, okay, so. So maybe I quickly go through at a high level, then we can go into finer details of which ones you think are the most interesting. Yeah, so okay, let me summarize the first three. It's basically purpose built versus existing solution, right? That's number one. Number two was external embedding pipeline versus built in hosting pipeline. Number three is indexing speed versus querying speed. So I think the others are going more into the actual indexes and generation of those indexes in more detail. So I'll go through them. Number four is recall versus latency. That's more related to how accurate are the results versus how fast am I retrieving those results. Number five is in memory index versus on disk index. I think this is a very big one for the future. So we definitely want to go into. I think some of the details of that. Number six is sparse versus dense vectors, the kind of vectors themselves that are underlying the index. Number seven is the importance of hybrid search where it's full text search combined with vector search. They both have their own trade offs and I think the last one is the importance of filtering. So pre filtering versus post filtering to decide the quality of your search result.

Yeah, I am very curious about this in memory or on disk one? Well, I'm interested in all of them but I know one of the things that has come up in several of the applications that I've worked on has been okay, do we self host one of these things? Do we use the managed service because they're going to be able to scale up and optimize things. There's also the choice of oh well I could just load one in memory on the fly ephemerally I could have an embedded case where I load a bunch of vectors in and then there's some persistent file that I can pass around. And then there's I think more of what you were getting at which is is this index represented on disk or in memory? Could you maybe help us parse through some of those things and go into a little bit more detail of what you mean there?

So yeah, now that you mentioned self hosted versus cloud, I think that's a number nine that I will add eventually. It's a very good point that you've.

Got that up there. Perfect. Yeah, maybe we can find a number 10 to round it out before the end of the episode.

I'm sure there's way more. Yeah, I could go on all day but yeah, so yeah, going back to your in memory I think that's a very important one. So I think this is one of the things that is defining the. You could call it the race towards vector supremacy. I don't think the term is very accurate but anyway I think the challenge with most of the vector indexes out there, I think the most popular one by far is called hnsw Hierarchical Navigable Small World Graphs. And I go into the details of the algorithm in part three of my blog so I'd be happy to discuss more with anyone else outside of this if required. But HMSW index is known for its relatively good trade off between recall and latency. It's fast and it's relatively accurate, but it is also memory hungry. And where this becomes an issue is as datasets get larger and larger and larger this is called the trillion scale vector problem. Now A lot of vendors are talking about it. It's not too far away to imagine that you're going to have to at one day, at one point index a trillion vectors. And that is by no means a mean feat, right? It's a very challenging problem. So the data set in that situation would be way too large to fit in memory. Now, HNSW already does a lot of optimizations under the hood. The algorithm is designed to store a sparse graph in memory and essentially you search through the sparse graph and then through the layers of that graph, you narrow down on the nearest neighbor to the query that you input. But as we go get larger and larger in terms of data, even that sparse graph does not fit in memory. So databases have come up with different solutions as to how to deal with this out of memory issue. One example would be Quadrant. They use this thing called memmap. It's like a sort of static RAM option where you don't actually store the vectors in memory, but you persist it to the page cache and it's still better than directly storing it on solid state drive, which is one level below, right? So in terms of latency hit, it's not as bad. So you don't lose that much performance. And you'll notice that a lot of vendors fight really hard to avoid persisting any vectors or the index to disk, because the moment you go onto a solid state drive, there is a massive performance hit in terms of retrieval because the speed at which you're able to retrieve things from memory is, as you know, much, much, much faster than what you could do on disk. That's a general trend, I think, across the board. Right now most vendors are largely working with storing the HNSW index in memory and then adding some sort of caching layer to avoid having to repeat the queries and waste time in that sense. But this is entirely new index called vamana. So I've written about that in my blog. It's optimized for solid state disk retrievals and the algorithm they use is called Disk ann. Not every database vendor has implemented this. It's still in the early days, but if I look at where the future is going, there are many options that vendors could go down the road of they could choose to implement HNSW on disk, but records suggest that that's not a great idea because its performance would drastically reduce. It would not perform that quickly as it does. Disk KNN seems to be the agreed upon standard across many vendors. But the challenge of Disk ANN is the original research paper that implemented it. The Microsoft Team that implemented it. Their implementation does not directly translate into the database internals. Depending on the language that the database uses, many of these are written in GO or raf. This KNN implementation was written in C. So it's not a direct transplant of the algorithm from the source to the database. It required a lot of rewrites and a custom approach towards optimizing for. NET speed. But that being said, I have to point out one particular vendor that I think really stands out from everyone else on this front. They call Nance db, I believe the youngest database out there. They just come about I think end of 2022, early 2023 and they are the only solution as far as I know who only support on disk indexes. They don't do an in memory index at all. And I was initially very surprised as to how they even do this, how can they go about this. But as I dug into it and I've spoken to some of the team as well, they're really, really open about their research that they're doing and all the models that they're building. But essentially they innovate on multiple fronts. But the biggest innovation is the underlying storage layer, the storage format. They built this format called Lance, which is essentially optimized for on disk reads of data and this, the database itself is built on top of this open source format, Lance. So the whole thing is open source, it's built in rust. So the performance there is already close to bare metal. It's really, really fast. They have already built an experimental disk ANN implementation. So when it comes to this on disk versus in memory trade off Quadrant is going about their own path in terms of how they achieve on disk larger than memory data. VViate is going around its own path. LaunchDB is innovating on a different front. I feel like these are the three vendors who I've interacted with more and used. And I think the future is heading towards one where on disk becomes a requirement and the standard way of implementing an index. But the engineering challenges are still ongoing.

Let me ask you a slightly different question. It's not completely unrelated. The things that you've been kind of addressing there kind of are leading me to the next step on that. So when you're thinking about kind of environments that you want to put in, like I know if you look at the other database types before Vector, you would have some that are, you know, scaled massively in the cloud, you'd have others. As we've moved more and more intelligence and data out onto edge devices and they're either embedded or they're designed to serve in a very constrained environment. What are the options for vector databases in that? I'm assuming that there's obviously the cloud capability because that's kind of always the baseline. Do you also have, you know, as we're moving into an increasingly autonomous world out there and more and more things are being pushed out outside of the data centers in the clouds, or at least the central parts of the clouds, are there options for either embedded or micro serving, if you will, on the vector side?

That's an amazing point. Yeah. And I cover this in my blog post. Number one, in terms of the architectures of these databases and you're absolutely right. I think there is a lot of room for embedded databases to become the norm. I know DuckDB is making waves in the SQL market on this front. I think a lot of vendors are emulating what DuckDB has done in SQL. As you know, DuckDB is an embedded database unlike Postgres which is a client server architecture database. Right. So what happened in the SQL world is now translating into, you could say the vector world. Two databases that are following this embedded approach. LancDB as I mentioned, and Chroma DB. These are the most. Chroma DB is quite well known. People have been talking about it for a while. But between the two of these, I do think that lancdb has, it stands out more in the underlying technology because the Chroma, from what I understand right now is it's still building out its underlying layer. It was kind of wrapped around an existing underlying internal database itself. It was not. It did have its own purpose built offering to begin with, but they're kind of building that out as they speak. So I think between these two vendors it'll be interesting to see how, you know, each of them rolls out their own features and kind of target a specific part of the market. Going back to the point of cloud versus on prem, that's another big thing I think that's going to come up. Honestly, like Pinecone and services like that that are completely on cloud, there are, they could be real potential bottlenecks for companies to be okay with just, you know, sending all their data to some cloud. Even if Pinecone says they would deploy on your infrastructure, at the end of the day it is still a purely cloud based solution. There's a lot of infrastructure related hurdles around that. Self hosted is, I think, as you see, going to become more and more common and certain options like VV8 quadrant, they offer self hosted options in their licensing as well, so the question for me that remains unanswered is which model in vector database vector search will dominate in the longer term. Embedded versus client server. Right. We are so used to the model of client server that's been working for more than a decade. Right now pretty much every database we've used is based on the client server architecture where the server sits remotely and I don't have to have the server running anywhere near where application's running. But I think embedded databases, especially with LLMs in the picture, it makes a lot of sense in terms of data privacy and things like that. And the scalability of these have I guess not truly being tested. Wtb is just 3 or 4 years old, launchdb is less than a year old. Chroma as well. So it'll be interesting to see how embedded databases compete on that front, like how well adopted they are because I think industry generally tends to favor things that are tried and tested at scale, this sort of thing to catch on, it would have to offer real, real business value. And the way these databases monetize their offering, I think that's going to be interesting to see.

And I guess we've already started moving this direction a little bit. But as we draw closer to an end here, I'm curious, you have explored probably more than, well, many people, certainly myself in terms of how all of these offerings compare, what the trade offs are related to vector databases. I'm curious, as you look towards the future, what are you excited to try that you haven't yet tried? And then maybe what, what excites you about this space? I know you mentioned certainly there's things that are hyped or maybe different marketing that plays into this. But what are you actually practically excited about as a practitioner? Practitioner in the future of this vector database space?

I think the low hanging fruit is the immediately obvious one. So I'll start with that. I think in the past when it came to search, we imagine the Google search bar and idea that to build something like that was inconceivable. A few years ago having a scalable, reliable search engine that you could build in house on your own proprietary data was really, really difficult to do at scale. But today I think with the combination of vector databases and LLMs with GPT4 now and all the other models out there, I really think that it's kind of become available to the masses. Right. Like the average company who does not have massive compute is still able to build very, very valuable search solutions, information retrieval solutions on top of their existing data. There are Additional offerings like Haystack and, you know, like the search engines that build on top of vector databases. But I think the foundation layer are actually being enabled by vector databases, which is why I'm so interested in those use cases. So those applications are very interesting at first. The other thing is Retrieval augmented generation. This is a term that came about, I think it was introduced by Meta in one of their recent papers. Essentially the idea behind retrieval augmented generation is typically information retrieval involved. You send a query and you receive a response that retrieves information relevant to your query.

Right.

Where the generation comes in is now LLMs add an additional layer on top of that. You could send a query in natural language and you retrieve the most similar documents to that query. Right. But rather than just retrieving the document itself, you could have the language model go to the document, look at your query, and then retrieve only the part of the document that is relevant to that query and then generate a response that could potentially answer a question that you had, like, what is the birthday of so and so person who runs this company? Right. So these kinds of things were really, really like almost impossible to do before, but now I think it's, it's really actually achievable with the kinds of tools and technologies that are available today. I think retrieval augmented generation is really skyrocketing right now. As a term, I think everyone's talking about it. But what I want to add to that is I want to throw this out here to any of the listeners and maybe potentially I'm going to talk about this to other people in industry as well. Can we add another layer to Retrieval Augmented Generation? And what I'm really interested in is how the two worlds of graph databases and vector databases come together. And I've posted about this a couple of times. But what's really interesting right now is most graph databases, like Neo4j, for example, they use declarative query language interfaces like Cypher cipher is you could say they have SQL equivalent for graphs. The good thing about Knowledge Graph is they encode factual information and in a very human interpretable way. So the things that form nodes and edges in Knowledge Graph, they are something that we as humans put in there and encoded our knowledge of the real world into the data where vector databases sit. Complementary to this is in many cases I might have connected data where unless your person knows other person, like a social network situation, person follows another person, person lives in a city and so on, these are all meaningful connected entities in the real world. But you add some layers of data on top of this, you know data about a city, you know data about a person, you know where they worked, you know what company information that has like there's so much additional unstructured data that attach onto the node in a graph that is actually hard to query using conventional graph algorithms or, you know, graph languages. So I think vector databases are uniquely placed to add new value in that space in terms of, I call it factual knowledge retrieval. Now, the problem with knowledge retrieval is sometimes the queries that you have need to be exact. The ability to submit a fuzzy query that does not exactly match your terms in the graph is something that you didn't have before. It's very difficult to actually generalize your query in a way that retrieves useful information. So I'm very interested to see how the power of natural language querying interfaces enabled by LLMs can be built on top of vector databases that store all the information related to an entity and then encode that entity into a knowledge graph. And then you tie all these things together in a way that you can actually retrieve information and explore and discover aspects about your data that you couldn't otherwise in a way that actually ties all these tools and technologies together. So I call it like an enhanced retrieval, augmented generation sort of model. And this would obviously require tools like LangChain or llamaindex. I mean, these additional frameworks that allow you to compose these different tools together and pass data and instructions back and forth between the human and the different underlying databases themselves. So I'm super excited about those technologies.

Yeah, I think it's great to hear that perspective. Also, you know, usually the answer is not like only this technology and nothing else is the the solution, but a strategic combination of things often is where things end up. And I think those are really interesting topics to explore and I look forward to your many part blog post as you explore those. I'm definitely going to be following your writing now. And yeah, thank you from the community and from us for your work on this topic and sharing that work with the community. It's super practical and we're very privileged and happy to have you on the show. Thank you so much, Prashant.

No, I appreciate it. Thank you so much.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already and if you're a longtime listener of the show. Help us reach more people by sharing practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to at fastly.com and flye IO. And to our Beat Freakin residents, Breakmaster Cylinder for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.

Sa.