#234 High Performance Generative AI Applications with Ram Sriharsha, CTO at Pinecone

One thing we have found and we've written about it and so on is you could take GPT3, GPT 3.5, you could trade all the data that it was trained on, just put that data into a vector database and use GPT 3.5 along with that vector database. And you have better groundedness and better ultimate retrieval capability than GPT4. Which means that a weaker model with a powerful vector database, even just retrieving on data that the model was already trained on, can still do at least as good, if not better than the state of the art models.

Hi Ram, thank you for joining me on the show.

Hey, thanks a lot for having me.

Yeah, great to have you. So, just to begin with, can you talk me through what are the most common use cases for vector databases?

Yeah, so there is a few big use cases. One people might or might not be familiar with is rag retrieval argument generation. So lots of genai, practically every generic application needs access to knowledge. This access to knowledge comes through interacting with vector databases as part of retrieval in a flow that's called rag retrieval augmented generation. That's one large set of use cases. Another one is semantic search. Anytime you're searching and you want to retrieve content based on articles or documents or what have you based on semantic relevance between query and document, you use vector databases. Increasingly we also see it used for labeling and classification use cases where you have something that you want to label and you want to look at neighboring points in this vector space and use the labels dominant label of that neighboring set of documents or neighboring set of what have you to be able to label this thing. So these are the three big set of use cases. But there's broadly, in some sense vectors are the lingua franca of machine learning. And we are discovering new use cases every day.

Okay, that's cool. So we got retrieval augmented generation and we've got semantic search and then there's also classification stuff. So for those, I think maybe retrieval augmented generation is perhaps the most well known. Can you just talk me through how that fits into the idea of a chatbot?

Yes. So imagine that you're building a chatbot that say allows users to just query your knowledge base. A good example is Pinecone has the chatbot that allows people to just search over customer support tickets or search over resolutions to certain questions that may have need knowledge about Pinecone and things like that. Now if you're building a chatbot like this, you probably are using a large language model, maybe GPT4 and so on. Now GPT4 is great. It's a great reasoning engine. It has great general world knowledge. It also understands language, but it doesn't understand your data. It doesn't understand, for example, the customer support tickets, it doesn't understand your prior resolutions, and so on. What vector databases allow you to do is to put all this private knowledge base into a vector database. And then if you have a Cirque query, you go to this vector database, retrieve candidate document chunks or parts of documents that are relevant, compose it into what's called a context, and you give this context to a large language model. The large language model does reasoning over this context and extracts information out of it, and then hopefully answers your question. So that whole workflow is called rag, and that's how it's used underneath chatbots, for example.

Okay, so this is just a way of providing extra information that's specific to your company or whatever, just in order to increase the accuracy of the answers.

Yeah, more generally it's a way of providing relevant knowledge to the language model itself, because the language model doesn't really understand your knowledge.

Suppose your boss comes to you and says, okay, I need you to create a chatbot for our company. I want you to use some rack. What's step one to creating a chatbot?

That's a great question. Step one is actually data. So first of all, presumably you have your data as maybe HTML pages, or it is maybe markdown documents or what have you. So you collect your data together. You also have to decide whether this data is mostly static, which is there's a corpus and you just want to serve it, or is it dynamic, in which case you keep adding and deleting information and then you have to deal with that. And hopefully your data is also at least partially labeled. You have some questions and some answers that are relevant, some answers that are not relevant, and so on. Obviously, if you don't have labeled data, then that's an even more challenging problem. You basically have data that's not labeled, and somehow you need to understand quality of your pipelines. But you start with data, you start with defining some metrics. Do I want to give factually grounded answers? So you have to come up with certain metrics that define that and then you decide on your language model. Now, again, most people are probably rightly so, will just choose the best language model out there, which is what I would advise starting with, which is don't start with optimizing for cost or optimizing for latencies and so on. Just choose the best model out there. Similarly, you want to choose a vector database that allows you to be flexible. For example, we talked a little bit about, is your data static or is your data not static? Now, clearly some databases handle this better than others. You want to choose one that just handles whatever you throw at it so you don't have to worry about that part. And then you start putting a pipeline together that takes your documents and converts them into vectors. Now, this pipeline can be as simple as just chunking your text into certain set of passages, encoding each of these passages using an embedding model, throwing it into a vector database, and then retrieving it from there, and then putting it into your context. So this gives you the most basic pipeline that you could build, and from there you simply start tuning. I would probably tune for quality first, get the best quality that you can get, and then start figuring out, is it cost effective, is it not cost effective? And then what do you need to change in order to make it cost effective?

You mentioned the idea of a difference between static data and dynamic data, where the latter sort of updates more frequently. Can you just give me an example of when you might have a static data set or when you might have a dynamic data set?

Yeah, static data sets are very common in web pages, for example.

Right.

Web pages don't change that often, especially if you have, say, documentation about Pinecone that you want to search it. Obviously, documentation will change as we do new releases and so on. We might add new features and things like that, but it doesn't change that often. So you could imagine a workflow where you just take a set of documents, you just embed it, and then you search over it. There is also use cases where things are very dynamic. For example, if you have a notion page and you're editing notion pages and so on, and you want to search over it, as actually customers of Notion do at Pencone, using Pine Cone. Now, that is obviously far more dynamic. You're changing your pages quite often. These create lots of writes. You also want to be able to be fresh, meaning you want to be able to query the things that you changed. So it's a very dynamic workflow. Similar dynamic workflows exist in sort of product recommendation use cases where your product reviews might change, where your product definitions might themselves change. Some products may be available, some might go out of stock. You need to be able to respond to that. You probably don't want to be searching over things that are not available, stuff like that.

So we'll be getting into details more in a moment, but I'm curious as to how do you know whether your chatbot is successful? What would you track? What sort of metrics are going to say, this is a good chatbot.

This is a great question, by the way. This is not easy. In fact, you have to really think about what you're really looking for. So, for example, you want in some sense factual answers. So you want to retrieve facts. So you want to know whether the language model is actually hallucinating or is it grounded in its retrieval? And it's grounded in its answers. Being grounded is not enough. You actually want it to be relevant. Right? It could be giving factually correct answers which are just simply not relevant to the question you're asking. So in some sense you're looking for groundedness as well as factual relevance, search relevance, and so on. So it's like two or three of these metrics that you can define. Now, the challenge is, of course, in figuring out how to quantify them and how to track them. Right? It's easier to track if you have feedback, if you actually have a way of collecting feedback, and to be able to use that to track whether things are trending in the right direction or not. Sometimes this is easy, sometimes this is hard. Sometimes you're probably building applications that other people are consuming, so the feedback channel is not direct, and so on. So sometimes you can also try to do the following, which is you could try to use more powerful language models to extract questions out of documents. To say that you give the language model documents, it's going to extract questions. Clearly those questions are actually factually related to these documents. Now you want to put them into a flow and decide whether your overall pipeline is actually getting those correct answers or not. Now, sometimes this works, sometimes this doesn't work because language models, it turns out that they're not as good as human beings still in asking questions off documents and so on. So in some sense they are too simple a question. Okay, but still it's better than nothing. So, you know, in these things, I always think about it in terms of start from something, even if it's not ideal, and start kind of improving upon it. So the best thing you could do here is actually collect label data, actually ask people to thumbs up or thumbs down for you relevant or irrelevant examples and so on. But if you can't do that, start with something simple.

Okay, so that's interesting that you've got a mix of AI techniques and also human techniques there. So you can get humans to give you a thumbs up or thumbs down and you get AI to ask questions about documents and Sort of create more data for you.

Okay.

I'm actually curious as to whether humans giving thumbs up or thumbs down works for, say, like a support chatbot, because if people are asking questions, they don't necessarily know what a good answer is, which is why they're asking in the first place. So is that a valid technique?

Yeah. So it can definitely help you compare between two things. So it can help you compare between, say, two language models outputting the answers to the same question. So while a straight thumbs up, thumbs down may not actually work, it can definitely, you can use it to compare between two things. It really depends on how you present it to the user.

Okay, all right. So you've got a range of options and maybe none of them are perfect. So you try a few things and see what's going to work for you. I'd like to go back to the start of this, where you're collecting data. So how do you go about collecting all this data that's going to provide the knowledge for your chatbot?

So even those data collection pipelines can get fairly complex. So we talked about scraping the web, for example. You made up a script aware. You might have PDFs that you want to search over. And then you have to figure out how do you go from PDF to something that the language models can actually understand, the retrieval engines can actually parse and store, your algorithms can actually chunk and embed and so on. That's already a complex pipeline. So a lot of the time I expect people spend in putting together these sort of chatbots and these sort of workflows is even in just the data collection, the data transformation, the quality of parsing clearly affects everything that comes after it. So you could have the best embedding models, you could have the best language models, the best vector databases, and still you won't get the high quality if your parsing is not good. So I think that's where I expect people spend a lot of time.

Okay, so you mentioned this idea of parsing data, and I'm presuming this is rather than just scraping absolutely every bit of data possible and just dumping it to the vector database. You've got to do some sort of pre processing there. Can you explain a bit more about what this pre processing involves?

Yes. So sometimes it involves take documents that are. Let's start with documents that are not complex, just text. So if you have text documents, you have to figure out first of all, how do you break them up? It's called chunking. Now, chunking can either be at a passage level it could be chunking sentences each sentence and then putting it into a vector database and so on. But even that is not sometimes enough because often, depending on the complexity of the document, there is some important knowledge that's in, say, the header or the metadata about this document that is needed for actually understanding the document. Often title of the document, maybe the first paragraph, and so on have a lot of information that subsequent paragraphs and subsequent sentences need for you to actually be able to do good retrieval. So just the art of going from a document to a set of vectors can get fairly complex, but that is the case even with text documents. Now. Think about HTML, think about images, think about tables. So as your data starts getting richer and richer, the complexity of how to go from that to embedded vectors so that retrieval can do its job well becomes far more complex.

Okay. Yeah, I suppose thinking about, like a product web page, you've got pictures of the product, you've got tables of features and things like that.

Yes.

Okay, do you want to talk me through, like, what that involves then, how you go about turning that into something useful?

Yeah, that's actually not easy. So that's why I said you need to come up with really good parsing. Sometimes it is figuring out not just taking passages and embedding them, but having passages be enriched with, say, metadata about the document itself. Sometimes it is. Also, it might even involve giving passages to language models to add context, even before embedding. Sometimes it is embedding the same document in multiple ways. For example, embed sentences separately, embed the document itself separately, and then combine the two. Sometimes it is both sparse and tense embedding, embed the keywords in the document using sparse embedding, embed the rest of the document using tense embeddings, and then be able to jointly retrieve the two. So again, this depends really on the complexity of the document. The domain. For example, legal domains have keywords that matter far more than some of the domains do, and things like this. So it's fairly complex, and in some sense it's still an art right now.

Okay, that's really interesting. It reminds me of it in machine learning, where you're doing feature engineering and it's like you've got all this kind of raw data, and then you want to add sort of layers of sort of structure on top of that.

It is exactly like that. Right now, in some sense, we are missing the. So what was really impressive about the revolution with large language models and even prior to that with deep learning, was a lot of feature engineering became automated and something that could be handled by algorithms. Prior to that, feature engineering was done by teams of people and there was a lot of bespoke feature engineering that we had to do. We don't do any of that today for language model training. Something similar has not happened for RAG and for what we are discussing right now in really knowledge intensive tests. So we are kind of exploring and figuring out what that should look like. So in the interim we have all this R to and a lot of complexity in how this is put together exactly because of that.

And is this something that's sort of coming soon, do you think, some more sort of automation towards this processing?

I think there's a lot of work going on here to figure out how to automate this, how to really go beyond this sort of bespoke feature engineering to something that can be automated. There's a lot of really good, interesting research, but nothing that I think is ready yet. So I think we are still a couple of years away. But there's a lot of hard work that Pinecone and everyone else included is doing in exactly these sort of areas.

One of the other tricky steps beyond sort of pre processing data seems to be around dealing with updates to data. So you mentioned the idea that sometimes data is dynamic, it's going to have to be refreshed a lot. So when your data changes, what your chatbot says has to change, how do you deal with this process of updating?

If you think about what happens when you change data, sometimes you might have to re embed a document, sometimes you might have to re embed portions of a document. So again this comes down to for example, efficiency. So a lot of customers don't re embed the entire document because it's expensive, so they just re embed portions of it. And then you need to be able to know what vectors do you delete from your database, what vectors do you add? How do you make sure that all of this is done in a way that's consistent? So you actually delete something and add something else, but you don't want there to be a partial state in between that gives you wrong answers. This is where the database parts of vector database comes in, which is, this is what they are really good at handling, is how do you do this incremental addition, deletion updates and so on and keep a consistent view of your data. Sometimes you're not even changing the embeddings, but you're just changing information about the embeddings. A good example is when your product goes out of Stock and maybe temporarily, and you just want to say it's unavailable. And unavailability could just be a flag. On a vector in Pinecone, people treat it as metadata. You want to be able to transactionally say that, oh, this metadata has now changed, or the price of a particular product has changed. So when searching for it, you want to now reflect that. Right. So this is where I think the dynamicity and all this comes in is the ability of your database to handle all of this seamlessly and free you up from worrying about that.

Okay, that's interesting. So some of the things you mentioned there, like saying, is this in stock or not? What's the private product? This sounds like very much like standard issue database stuff. This is very much structured data. And I normally think of vector databases as being for unstructured data. So how do you deal with the mix of the two then?

So while it is structured data, sometimes it actually affects search. So a good example is just what we were discussing, which is imagine that I wanted to search over products, but I'm only interested in products that are less than say $50. Right. So immediately the metadata about price is actually something that's important for the index, important for the vector database to know and understand, so that it doesn't give you the wrong answers. Right. So I would treat metadata as two different things. So the ones that actually impact search in search quality versus the ones that are maybe decorating the answer. Okay, so the things that decorate the answer, you still want to treat it as a traditional database problem, and you still want to have a traditional database that kind of handles that for you. Often customers use key value stores, other databases, traditional databases, to do this. But then there's ones that actually affect search quality and that the vector database has to be able to have. In fact, that is what makes vector vector search challenging is it has to handle these database aspects pretty well for it to actually function as a vector database.

I'm curious as to whether things like joins exist in a vector database concept as well.

No, no. So joins don't. And joins traditionally are even. I mean, they're always hard even for traditional databases. Right. For vector databases, I think you kind of do this sort of an implicit join, which is you put the metadata into the vector database, use it during search, and then you retrieve some candidates and then you join outside of the database. So that's like an implicit join, and that's the best thing that we support today. And practically no vector database supports anything else for a good reason, because joins across these two Very different types of data are very, very hard.

Okay, all right, that's good to know that. Don't have to worry about joins. It's all sort of taken care of implicitly. Okay, so I'd like to talk about the LLM side of things as well now, because you mentioned that you're probably going to want to pick a decent like whatever the best sort of LLM available is when you start creating a chatbot, but then you might swap it out for something cheaper to run. Can you think of when you're building an application, can you think of the LLM and the vector database separately, or is there some kind of relationship between them that you need to worry about?

So first of all, depending on the task at hand, you might be able to use cheaper, maybe less powerful LLMs. So that's already something that one should be aware of. So you don't need GPT4 for everything. You don't need the most powerful LLM for everything. It really does depend. But in some sense the quality of your vector database and how scalable your vector database is also has something to say here. So one thing we have found, and we've written about it and so on, is you could take GPT3, GPT 3.5, you could take all the data that it was trained on, just put that data into a vector database and use GPT 3.5 along with that vector database, and you have better groundedness and better ultimate retrieval capability than GPT4. Okay, which means that a weaker model with a powerful vector database, even just retrieving on data that the model was already trained on, can still do at least as good, if not better than the state of the art models. And we found this to be true even for some open source models as well. So clearly this means that if you use vector databases together with LLMs, you actually can. The sum of parts is actually in some sense you can get more bang for the buck by doing that. But even otherwise, it's important for you to know that you don't always need the best LLM for the task. Now, in terms of cost, today the cost is dominated by large language models. So language models really dominate this cost. In fact, the bigger language models provide you bigger context and that actually expensive as well. But sometimes bigger context actually helps. So even understanding exactly what some workflow should cost you is pretty tricky today. So you might go with a bigger context with a bigger language model and end up with a cheaper workflow Sometimes, sometimes you can go with a smaller model with lesser context with more data and a vector view base and end up with a cheaper workflow. But overall, I think what I can say safely is that the more you use vector databases in the loop and more do you kind of leverage it, the cheaper it's going to turn out to be. And in fact, there's a lot of research that shows that you could even take a lot of what the language models already know and put them into the vector database and that's going to lead to a overall more economical workflow. So I think economics is going to drive us in that direction anyway.

Okay, that's very interesting that just if you make more use of vector databases, then you can go with a cheaper language model and your overall cost of creating a chatbot might be less. I'm curious as to when you need like the best sort of LLM and when a lesser LLM or a lower powered sort of a cheaper one will do. Do you have a sense of which tasks you need for which?

Yeah, I think it's mainly reasoning tasks. And so the best LLMs are really good at reasoning and really good at picking out, in some sense, paying attention to the. Even if you give an LLM a large context, it's not necessarily going to be good at picking out the things that matter from the things that don't matter. This is usually the hard problem with giving LLMs the whole context, but we find that some of the, some of the powerful algorithms distinguish themselves in that ability and they distinguish themselves in the general reasoning ability. It's also because they've been trained on a lot of data. But again, this is evolving as we speak. As we speak, there are newer models coming out, particularly open source models that are increasingly challenging language models, proprietary language models, and so on. So whatever I say here is going to be obsolete in three months. Yeah, okay.

Yeah. Actually, just on that note, it just seemed like the best LLM just changes weekly. And I guess you need to build your applications to make sure that it's possible to swap out the LLM in order to optimize it. Is there anything you need to do in order to ensure that's possible?

Yeah. So I think the more you can treat your LLM as in some sense a black box that you can actually optimize away, the better, because that landscape is changing quite a bit. Of course, you really need to understand prompt engineering because the prompt engineering is specific to LLMs and so on, and so you want to be careful about that. But if you can engineer your abstractions in such a way that you can swap out an LLM, that's I think, eventually a good thing to do. That said, while we see that the leaderboard is changing quite often, and there is a lot of competition in some sense at the top for language models, one thing to keep in mind is that there is not a lot of private data, there's not a lot of really good metrics and really good data that the LLMs themselves have not seen. So in some sense, the best indicator of an LLM's performance for your application is actually your data and your queries and trying it out. Because I can tell you from a lot of benchmarks that we do and a lot of data that we have collected and so on, there is still a gap between open source modules and proprietary models and so on. Even though that open, that gap is getting closed as we speak. Because in this entire space we are still missing really good metrics, we're still missing really good data sets that in some sense the models have no idea about. And the ability to generalize that sort of data and those sort of queries and so on is what matters for people at the end of the day.

Okay, so we talked about two of the components of chatbots. So we've got the large language models for the vector database. So there's a third component in this which is gaining a little bit of hype, which is knowledge graphs. And I'm wondering, where do they fit into this architecture alongside the LLM and the vector database?

So knowledge graphs are basically telling you something about the entities that are referenced in the document and the relationships between them and so on. Generally we think of this as information that you use in constructing embeddings or information that you use post retrieval. So usually they either belong in the ingestion pipelines that generate these embeddings in the first place. A good example is that some entity can be referenced by a name at the top of the document, for example, but the rest of the document, it could be called by a proxy. So if you don't understand that the proxy actually refers to that particular name, you're not going to be able to do proper retrieval. That's just one example. In that case, it means that the entity has to be somehow attached to every passage, or there has to be some way to resolve the entity to what's being referenced in the passage, and so on. Likewise, during search, to be able to rewrite a query in such a way that the query can now reference that entity explicitly. So generally, I think knowledge graphs are complementary to vector databases and LLMs and they kind of operate in the ingestion and search layer where you want to reference that knowledge graph in constructing these embeddings. You want to reference the knowledge graph in resolving queries.

Okay, so it's very much a complementary technology to the VegasD database then. It's about understanding relationships between different components. So I presume this is going to be most useful for if you want to do difficult reasoning about things, is that correct?

It's often very useful for domains where entities become important or the ontologies become important. So that's something that the language models themselves would not usually know. An alternative to this by the way, is to try to fine tune your language model to understand those entities and to make those originations and so on. But knowledge graphs are also a good way to deal with this.

So I'd like to talk about some of the other use cases beyond chatbots. So AI agents keep being hyped as like the next big thing. And I'm not quite sure whether it's happened yet, but probably coming soon. How are vector databases used within AI agents?

So I think the best way to think about vector databases is in solving the knowledge problem for AI. Okay, that's really the best way to think about it. So if you think about LLMs today, they do kind of two things. They actually maybe three things. They're really good at reasoning, they're really good at language understanding. Obviously you need language understanding to do reasoning, but they're really good at that. And they're also capturing in their parameters in some sense world knowledge. What they don't have is access to your knowledge or a way to actually use it. The way I think about vector databases is vector databases are great at separating the knowledge part from this whole problem, which is you really want the knowledge to be in a vector database. This could be world knowledge, this could also be private knowledge. Okay, to me that's the logical step forward. What does that leave LLMs with? That leaves LLMs with a really good understanding of language and a really good reasoning ability. Right? So when people talk about AI agents and so on, they're optimizing for the reasoning ability, that ability to reason and the ability to use tools and things like that. What vector databases really do to fit into this whole equation is they solve the knowledge problem. Where does that knowledge needed to act upon something come from? That's the vector database. Likewise, how do you manipulate that knowledge? How do you redact that knowledge? How do you make sure that these language models are not hallucinating and so on? I think the code to solving that problem is vector databases. What vector databases don't solve or don't really affect is the reasoning part, which is the actual agent.

Okay, so in this case it's going to be a very similar use case. The only difference is this time the LLM is calling some external tools.

Exactly, exactly, exactly.

User things. Okay. All right. So the other big thing that's being had a lot is the idea of multimodal AI. So things like using audio and images and video and of course you can store all these things in a vector database. But I'm curious as to how exactly this works. Can you talk me through it?

Yeah. So I think again, there is multiple ways to deal with this. So you would think about storing images embedded into a vector database. You can store text into a vector database already during search, you want to retrieve both. Right. So you want to retrieve. Given a. Given a piece of text, you might want to retrieve images relevant to it. Given an image, you want to retrieve some text that's relevant to it, and so on. So at the end of the day, it is again a retrieval problem, which is there is embeddings. You want to be able to convert your text or images into something that can search over that space and retrieve the relevant embeddings and then map the embeddings back to text or images, what have you. So the nature of the problem is still the same. The only difference is what is getting embedded and what's getting searched.

So it really seems fairly straightforward. You just have to have the right embedding to turn whatever image, video into that vector of numbers. Okay, all right. And do you want to talk me through some examples of how you actually go about using these? Like where would you care about having images or video or audio stored in the vector database?

Yeah. So I mean, today the dominant use cases we see are still text. So most people aren't text, and to some extent images. Often we don't see video as much yet, just as a community in terms of activity, erasers and so on. Generally, I think what people. A good use case for storing images is just same detecting near duplicates. Right. So if you want to detect whether you've seen a near duplicate, one of the best ways to do it is to embed images, put them into a vector interface, take the accredited image, look for the closest neighbors and see whether you're close enough. If the images are close enough, then likely it's a duplicate. Right. And there are people who use this sort of a duplicate detection workflow. Similarly, there Are people who use image search to be able to distinguish between, going back to classification use cases, distinguish between weed and good plant, A plant that's, that's something that should be there in the farm versus something that's not supposed to be there in the farm.

That's really fascinating. Especially the idea of like using it for image classification. I suppose I haven't really thought of that as being like a search problem. If these things are similar enough, then they're probably the same.

Yeah. What's fascinating is that it's being cast into most of these classification and labeling use cases today are being cast into a search problem, which is awesome. Right. It's efficient and fast and we have vector databases that do that really well. Okay.

Yeah, that's really interesting stuff. So I'd like to talk a bit about how you actually go about implementing these things in your organization and I'm curious as to which team would generally be involved in working with day to day spaces. Because it's got data in the name, you think it ought to be the data team, but I'm not totally sure whether that's the case.

First of all, if you look at people who are developing gen AI applications, most of these are application developers, right? So these aren't necessarily data teams or ML engineering teams and so on as they used to be. There was a time when semantic search workflows, for example, were put together by machine learning engineering teams or data teams. These days it's any software developer who can put together a Versaille application or some sort of application that actually can connect the LLMs with vector databases can make a lot of progress even on small amounts of data, moderate amounts of data, building very compelling applications that that leverage, you know. So you're seeing more and more the application developer as the Persona who's actually using vector databases a lot, is using language models a lot. Right. So you don't need data teams per se to be able to do this. That said, your Web3 databases have to be hands off for that to work. Right. So these cannot be the same sort of databases that people used to work with 10 years back where you really need to understand the database, you need to set it up, you need to size it somehow, you need to figure out or work around as rough edges and things like that. So that doesn't work anymore. That's also why people are increasingly turning to these sort of pinecone SAS databases and so on, especially ones that don't require any tuning and this horizontally scale for you. So I think that's what we are seeing more and more, where we see data teams get involved, where we see production engineering teams that get involved, or ML engineering teams get involved, is once people hit massive scale, once you start this way and you're hitting now, you now have to serve thousands or hundreds of thousands, maybe millions of customers, or your data volumes grow to terabytes of data and so on. Now you really need to understand costs, you really need to understand how to scale the system. Because even scalably ingesting elements of data is a hard problem. From a software engineering side, that's where we see people usually put together data teams and kind of really get involved at that point.

That's interesting. So on the sort of application side of things, all software engineering teams probably in an engineering part of the organization, but then the data engineering is needed for all that pre processing and setting up the data pipelines as well.

Yes, yes. Especially scaling them. Right. When we talk about ingesting large volumes of data or having workflows that deal with dynamicity and all of this now suddenly data engineering becomes pretty important. It's less important when you are building a small application and trying to proof of concept or show the value or even for small workloads. So usually we see that there's a rapid adoption of small workloads from Genai developers and people who don't really have teams of software engineers working on this. It's usually one person or a small group of people. But at the same time, once people hit scale, then you have teams involved and now we are talking about really large scale applications.

What sort of skills are needed if you want to get involved in creating a generative AI application?

Main things I think are the ability to put together. You're still putting together a stack. It's putting together language models, vector databases, pipelines and building something of value. So it's really the strong understanding of the different parts of the stack and how they fit together to solve this problem, the understanding of where the cost comes from in this sort of a pipeline. How do you go about optimizing for it without losing quality and a good understanding of the quality you're trying to hit, which is how do you think about metrics, how do you think about data? So it's a new step these days. People need to understand data metrics, how do you optimize, how do you prompt engineer, how do you optimize LLM to do its job well? Likewise, how do you do retrieval better, what do you retrieve, what do you store and so on. So I think it's an emerging stance. You need a little bit of interest in understanding in how embeddings are created, how vector databases work, how language models work, and so on. There's a lot of information out there. There's people typically start with, you know, my suggestion is start with the simplest thing, which is just put. Put LLMs and vector databases together and start with something really simple. Then learn from what, what is work, what's not work. A lot of advice out there on how to put together RAG workflows. Pinecone has a lot of documentation, for example, on how to put them together. There's a lot of starting points, but I would say that this is like an emerging skill set. It's like people have to figure out how to make all of this work together.

Yeah. So really just understanding what the different tools are evolved, how they fit together, and then just what the use cases are. Get some of the basics. How do you measure stuff? All right, excellent.

There's also an area where we are in some sense trying to lower the barrier, which is the fact that you have to put all of this together and you have to get some expertise in all of these things is something that we are trying to figure out. How do you solve for? How do you lower the barrier to entry so that people don't have to think about all this and it's not just us who are doing it. I think you'll see this happening over the coming years, that this whole workflow gets simplified and presented in such a way that people can just use it like they would use a model.

Can you tell me what sort of innovations are happening in the world of vector databases at the moment?

I think the main innovations that are happening are a around cost and scalability. So a lot of vector search research prior to a couple of years back was all about squeezing performance, but it wasn't worried so much about cost. Most vector database algorithms out there, most vector database out there, until Pinecone came along and even pycon, serverless and so on, were all in memory databases for the most part. And the cost associated with them is pretty dramatic. So there's a lot of research that we've done, there's a lot of research that the community is doing in terms of reducing cost, in terms of improving scale, in terms of dealing with the data management problems we talked about. How do you actually effectively update and delete and manage data when you have data sitting in vector databases? Likewise, we talked about the fact that you cannot actually join with structured data, but you still have to deal with structured data that leads to very unique challenges for vector databases. So there's a lot of research happening around how to solve that effectively. So on the vector database front, innovation is happening really around new types of indexing algorithms that can work on block storage, new types of indexing algorithms that dramatically reduce the cost of maintaining this sort of data and the cost for a given quality and so on. At the same time, dealing with the data management challenges. Now that's the effort focused around the vector database itself. But even around rag and even around how do you combine language models with vector databases, there's new innovation happening constantly. So the things that I'm particularly excited about are the ones that fundamentally rethink the language model architecture in terms of how do you really leverage vector databases the best way with the. But in some sense rethinking how language models should work. There's a lot of research papers that try to attack this and there's some successfully, some partial, I mean, some partially successfully, some not, and so on. But what I'm particularly excited about is the fact that everybody's paying attention to it. And I feel like that's very productive way to think about how vector this is fit into the language model architecture in the future.

Okay, so some of the research is around sort of very, sort of targeted stuff. So I guess, yeah, yeah, it's quite, quite a nerdy sort of algorithm thing. And then it also goes to the broader sort of technical solution as well.

Exactly, yeah. Some very foundational ways in which language models are being fused together with web pair databases. So that started. That's an interesting area of research as well.

And with large language models you mentioned, there are some very long context window large language models at the moment. So these can store an awful lot of text in a single prompt or response. How does that change the use of vector databases?

To me, it doesn't actually change how. It doesn't obviate the need for vector databases or it doesn't fundamentally change how vector databases are used, but it does change some economics around how they are used. For example, you no longer need to stress a lot about how many candidates you retrieve from vector database. Like you don't have to optimize to the nth degree what you do there because you could retrieve more candidates and you could actually give it to a lavish model and hope that it can actually pick the relevant portions of your data. Right. Now again, you cannot quite do this because language models are not great at even now, they're not great at sifting through context, which is you can provide them a really large context. They aren't necessarily great at sifting through context, especially things that appear towards the middle of the context window and things like that, but theoretically you could. So theoretically it reduces in some sense the over indexing on exactly what you retrieve from a vector. You could retrieve a little bit more than you would normally retrieve. Right. Other than that, it doesn't fundamentally change anything. So you still have to rely on better databases for retrieval because there is no hope of feeding the entire corpus to two language models. Even if you fed the entire corpus language models, the fact that they have even infinite context window is not going to mean that they're going to produce a better result. So to me, context windows are useful when used correctly, but they're not a solution for not having it anymore.

Okay, so it sounds like the large language models still need to solve a problem of can you extract useful information out of a long context window before you can actually really make use of them?

Yes. And there are fundamental reasons to believe that you can't just solve it within the elevator. You have to really solve it in a different workflow.

So do you have any final advice for organizations who are wanting to create generative AI applications?

I think the most important thing is to figure out use cases where you can actually get started and start simple, right? Because it's going to take time for language models to get really good hallucination. Like we discussed drag to become like a package workflow that people don't even have to think about. Right. So today it's still an art, it's still a lot of moving parts. You kind of need to worry about it, but I think nothing should prevent you from getting started. And there are a lot of use cases where you get tremendous amount of value even from simple pipelines. So even from simply putting language models with vector DBs together in a very simple workflow, you can already get substantially far. And we've seen customers do that a lot. So to me, it's really important that you start getting into this gen space and start really leveraging it and build your kind of in some sense, your ability to iterate and the ability to think about your data differently, think about labeling, think about actually measuring quality and so on. Because all of this is really only developed by doing, right? So the faster you can get to doing that, the better.

Yeah, definitely a big fan of learning by doing. And I like the idea. Just try something. You're going to have to iterate before you get something perfect. So yeah, get started.

Yes. All right. Wonderful.

Yeah. Thank you for your time, Rob.

Thanks so much. Thanks for having me.