Systematically improving RAG applications

Today's little topic is systematically improving RAG applications. And in particular, the interesting thing here is this is really a system, right? So this is a system that you should be able to apply regardless of the applications you're trying to build, because the idea is that we're going to, you know, use this to do a data analysis, come up with hypotheses, test them, and eventually sort of get an understanding of where we need to improve the systems that we're building. Um, before we get started, I just want to roughly introduce myself. I imagine most people here might be a little bit familiar with me from Twitter, but my actual background is that I am currently doing a lot of AI consulting for companies that do work in the LLM space and the RAG space. And so, you know, you might notice some of these companies here, for example, Trunk Tools, um, Narrow, like Limitless. We— they're all RAG tools And then on top of that, we also have a bunch of AI hardware companies that again really care about improving things like memory. But the issue is once you deploy this application, when things are failing, when you are losing users because there's maybe like, you know, not enough confidence in the responses, maybe we are unable to answer certain questions. The idea that we have to go and figure out what kind of questions are actually causing the issues and we can't really blanket and say, I want to improve RAG. Instead, we need to be able to say things like, Hey, um, questions that are retrieving documents by file name are, uh, 40% of the questions that are being asked, but you know, there's a 30% user satisfaction. And now we can actually go in and identify exactly what segments of the population are asking questions that we cannot serve well, and then come up with ideas in the future to make some of these improvements. So I'll talk about that process, um, in a bit, but if you you know, maybe you had to leave early or anything like that, all the stuff that I talk about today I've actually written down before. And so if you're ever interested or have some time to spare, you can always take a look at some of my existing writing on the topic. And, uh, cool. This is, this is basically the table of contents for today's conversation. The idea is I want to cover the RAG playbook that I run for many companies, whether it's small or big. The playbook really is the same. You know, I'll cover how we should think about the feedback mechanisms, understanding that, you know, feedback isn't something that every single query will have. And so we need to also measure things like the cosine and the re-ranker scores, how we can cluster these things, and then use that information to really figure out what is the issue. And then near the end, I'll talk about what kind of low-hanging fruit that we have in terms of systems that every RAG app should— may as well include. In order to get the most performance out of it. But the first part really is the fact that, you know, we need to have a feedback mechanism. We need to actually have an objective that we care about. When a company comes to me and they say, hey Jason, like, we want to improve our RAG application, we actually don't know what that means. Are you losing users because we don't have faith in the system? Are we unable to give good citations? Is it the fact that the things that we're searching for aren't available in the vector databases, or is the fact that, you know, we're using things like file names, but because we don't use full-text search and only use vectors, we can't actually— can't actually map against that. So the first thing we have to ask is, what is the actual objective? And the easiest way to capture that feedback is, you know, having a thumbs up and thumbs down score. But even there, the copy is actually very important. I was working with a client recently where the original copy was, how did we do? Thumbs up and thumbs down. And what this actually resulted in was the fact that we did not actually know what we were trying to capture. It turns out if the answer was correct but the answer was too long, people would thumbs down. If the answer was correct and it was concise but it took 3 or 4 attempts to get the right answer, people would thumbs down. But that information isn't actually helpful in understanding what kind of business outcome we're trying to drive. What they really cared about is whether the answer was correct. And just by changing that copy, just by saying, "Did we answer your question today?" versus "How did we do?" we were actually able to 5x the volume of ratings and improve the ratings for the data that we had. And what this meant is, you know, a couple weeks from now, we'll be able to look at the things that people thumbs up and thumbs down and have a heuristic on understanding what exactly are people happy with and what are we doing poorly on. But, you know, even if only like 5% of the population actually uses these mechanisms, we still want relevancy scores across all your search queries. And so not only do you want to have a feedback mechanism that measures something like satisfaction, you also want a relevancy measurement that is easy to measure, right? Like you could use things like having an LLM, but in practice it's much more practical and much more cost effective to use something like the cosine distance of your embedding score. And if you're using a re-ranker, the re-ranking scores that might come out of something like Cohere. And so now your logging data looks something like this. There's a query, there's a cosine score, and there's a feedback score. And that feedback score might be null for 95% of cases, but by changing the copy, we can improve that quality and improve the signal that we have is something we're trying to do better on. So now you have all this data. You have basically a piece of text, a couple of numbers, and your goal is to figure out how to prioritize my team to systematically improve the Rack application. And the most practical thing you can do initially is to do some form of unsupervised learning. If you use something like BERT Topic or LDA, we can actually use the query strings and generate topics of clu— questions and topics of clusters that we can use to really focus on the segments that we care about. And so if you, if you run this topic modeling, and we'll, we'll cover that later, um, if people are interested, uh, you can basically have a topic, the number of state, the number of questions within a topic, and now you have both a mean cosine score and a mean feedback user satisfaction score. And just by looking at this, we're going to give ourselves a lot of ammunition in figuring out how exactly we're failing the user and whether or not it's actually relevant to even improve. For example, maybe we find a topic that is very low count, very low relevancy, and very low satisfaction. You know, we can at least recognize that that topic is something we might not want to prioritize in the near term in terms of improving. And we might even realize that these kinds of questions are so irrelevant, we wanna put it into the prompt of our language model to say, hey, If you're going to ask me questions about this topic, this is not something I'm going to try to answer today. Whereas you might be in another domain where you find topics with very high counts, very high cosine similarity, but very low feedback, right? Maybe what we can say is this is the top cluster that we've discovered. Our search engine says that we're finding relevant data, but the users are extremely unhappy with how we're answering the question. Now we know we need to be thinking about what kind of answer we're generating, whether that's the question prompt, is there something else that's going on that we're not understanding for the users? And we can go figure that out. Other times we might get topics with high counts, with low relevancy and high satisfaction. And again, by looking at this sort of grid of options, right? High or low count, high or low relevancy, high or low feedback, we can pretty much identify the regions that we can we can dive into and sort of brainstorm on how we can improve these systems. And when you build these topics, there's going to be really two kinds of topics. There's going to be the topic where the content that you're searching is the interesting piece. So I'm going to just call that a content topic. There's another kind of topic which is a capability topic. So imagine I'm doing a marketing, uh, marketing RAG app. A topic cluster could be questions regarding pricing versus questions regarding, um, like different kinds of case studies, right? If those kinds of questions have low relevancy and low feedback, usually the solution is to improve the inventory that you have, right? It might be the case that our customers are coming in and asking about pricing and we're not doing well because we don't have enough pricing documents. That would be a topic cluster. The other topic we talked about was a capabilities cluster. Here, instead of asking a question like, you know, um, what was the pricing that we have, instead it could have been who is the last person who updated the pricing document. There, the capabilities topic really is around like modified dates, and if that information isn't available to the language model, if it isn't available in a text chunk, you'll never be able to answer that question. But if that does show up, we can look at the counts, we can look at the cosine distance, and we can look at the feedback to identify whether or not it's worth fixing today or tomorrow based on how you prioritize things, right? Other simple examples of a capability might be comparing and contrasting across multiple documents. If that's the case, you might want to have parallel search. If you encounter things where we ask about recency or latency a lot, recency or latest, we might want to have date ranges. And if we have date ranges, now we can recognize that we need to have the current date available in the prompt for the language model to prepare a search query like that. A really funny example that we noticed recently was in different industries, FY24 and FY25 don't end on January. And so you, you might discover that, uh, financial year is being referenced a lot, and because of that, we're matching across the wrong year token. These are all capabilities of a model that we need to improve by changing the search system compared to the topic clusters that we talked about before, where the way that we can fix that is to identify, like, inventory issues and making sure the documents that we have are actually available. And then if we have issues there, we might build out rules that say you know, if there are less than 10 documents that match this question, it might be useful to report back to the user or report back to the, you know, content management team and say, hey, lots of people are asking about pricing and this is not available in our search tool, right? So now there's different ways of solving the kind of RAG problem by basically looking at this simple topic count, score, and satisfaction. So once you develop these topics, then you have to do a little bit of like, you know, reading the tea leaves. Once you identify these topics, you're going to have an insight as to what kind of topic clusters and capability clusters actually make sense for your use case. And today we ran this project process offline, and so when something changes in the future, we might not be able to detect that unless we rerun this process. So once you identify the topics and the capabilities, one thing that is really useful to do is to actually build classifiers that can classify these questions in real time. If we have these question type labels, whether it's a type called, you know, is privacy question or it's, you know, a capability like compare and contrast or date range filtering or ownership, we can do this classification task sort of asynchronously as data is being processed and then send that to something like Amplitude or Datadog where we can monitor over time how these things change. And what you might notice is today we have some seasonality where, you know, maybe like summary type questions are being asked like Monday morning at the beginning of a work week. But more importantly, when you get new users or new kinds of inventory, it's really valuable to understand how the distribution of these questions and how you prioritize improving these questions change over time. A simple example could be, um, you onboard a new enterprise client and you discover that after the onboarding process, the amount of questions that we were ignoring skyrocketed because it turns out this company really cares about compare and contrast questions, whereas your previous user base did not. And being able to have this real-time monitoring of saying, okay, 20% of our questions are document search, and all of a sudden it went to 100%. Maybe we just got like a really big company that, you know, is using the application differently than it was before, and there would be no way to detect that unless you had this real-time observability that told you that something was going on. Otherwise, you might just run this clustering job, you know, a month from now, 2 months from now, and then realize that, you know, this new customer was very unsatisfied because we deprioritized the type of questions that they really, really care about. And generally what happens is when you actually build up these clusters, there's usually going to be like 3 or 4 things that stand out. And instead of having a goal that is like improving the RAG application, we can set very simple goals like improving daytime filtering, right? Or improving our meta— the amount of metadata we include in text clusters. And now you go from this very ambiguous improving this AI system problem to basically figuring out how much like time does your team have and be able to run these very small experiments. On top of that, because you have these specific topics, when you want to evaluate new datasets, you can actually use things like synthetic data generation to generate more questions of the topics that you care about and then be able to really quickly test whether or not your experimental changes has improved some system, right? Um, and so this is something we're going to talk about right now around from the lower-hanging fruit. The general idea really is the fact that once you have this topic information, you know what kind of questions people are asking. And now you can prompt a language model and generate more questions like this in order to maybe help you generate better baselines in order to evaluate your search tools. A simple way you can do this is by saying, you know what, I'm gonna randomly sample a text chunk. I'm gonna use AI to generate a question using that text chunk. And my evaluation metric is simply, if I search this question in my RAG app, that text chunk must show up in the top 3, top 10, top 25 results of the search system. And I know that whether that— whether or not that shows up is really relevant because I know they come from topics that I care about versus the topics that I don't, right? And if you, you know, include better metadata or add better daytime filters, you'll immediately be able to recognize whether or not this will improve your baseline. So instead of trying to guess whether something's going to improve your system, instead of trying to guess if Daytimes or if new embedding models or new text chunking models are going to improve, you can now just test these by having synthetic data even before you have a lot of, you know, user feedback data on something like user satisfaction. And that's basically it, right? Like you want to make sure you, you do things like having a full text search involved, right? Usually when you build these re-ranker, when you use these re-ranker models, we've always found improvements in how we return our, our baselines, right? Because people ask questions like, who edited the last document? What was the title of that document? When was this file last updated? Show me the latest version of the pricing doc we showed to Fortune 500 companies. You know, that query isn't going to be just an embedding, that's going to be a date filter. With a category filter, with a, you know, document type filter. And by looking at these topics, figuring out that these are the things that are important, and then building specific nuanced search tools, you'll be able to actually identify and report back to, you know, your executive, your manager, even yourself. And we can say, hey, for this topic where we have 40% of the questions be in this topic, we've been able to meaningfully impact and improve cosine distance or user satisfaction, right? And this will give you sort of the— to me, at least, the data and the evidence you need to invest more time in improving a system and knowing that improvements in the system is actually going to go leave a better outcome, right? You can imagine we have things like the count, the mean cosine distance, the mean feedback, we might have, you know, the average character length or the average latency, and we can now go try and correlate these variables to other business outcomes. You know, is my RAG app actually being able to convert users better? Are those who have high satisfaction or low satisfaction using my RAG or my chatbot once a week, twice a week, 3 times a week, right? Those are the things that you can do once you actually have these metrics and once you have these clusters to identify what exactly is going on. And obviously on top of metadata, you want to do things like adding type filters and document filters and date filters. I see we have 8 minutes left, so I'm going to just go jump to the next slide as well. The goal of this really isn't to like necessarily show you exactly how to do these things, right? The goal really is to show you that one of the simple ways of going about this is to do this kind of data analysis to identify and prioritize where you want to include your systems. And the main goal for me really is to also hear some questions that you guys have on, um, how you guys are running into these kind of issues. And the goal is that after this course is done, um, Dan and I will be spinning up a course on really breaking down this process and understanding how we can improve these applications. And so if any of you guys are interested as well, I'll be available on Discord to answer any kind of questions. If you guys are interested in, uh, learning more about the course or, you know, telling us what you guys are interested about the course. This is a QR code you can go check out, and there's a Typeform survey. Should take like 2 minutes, and it'll give, it'll give us a good understanding of what are the concerns and questions that, you know, your teams have around improving these systems and what kind of outcomes you're trying to drive. Dan, do you have anything else?

Uh, yeah, we'll drop a link to the survey also, just a direct link in Discord, and then we've got, uh some Q&A, so we should use this time to go through some of the, uh, the questions.

Yeah, do you want to pick the questions?

Yeah, I'll take— I'll, um, I've ranked them from most votes to least. Uh, top question is from Simon Willison: how many real user queries do you need to have before these techniques start to be worthwhile? Presumably if you have only a few hundred logged queries, you're better off eyeballing the data directly.

I would say so. I think it's still useful to find these clusters because, you know, like, even if you just have 100 questions and the difference was between, like, regular questions versus questions that involve, like, time, right? An embedding model should be able to at least discern, like, oh, latest, recent, those are things that show up. Um, it's really just useful to have some kind of mechanism that you can group by and then focus on those things independently. Because really, this topic clustering is just summarization. So if you copy-pasted all the 100 questions and put them them in the opus, you're kind of doing the same thing. But once you have these discrete variables, you can then group by them and say, hey, are they— are the cosine distances the same? Are the thumbs up and thumbs down the same? Oh, it turns out, you know, uh, every question that is in Topic 1 was asked by this customer of ours. Let's go, let's go reach out and ask and say, hey, you guys had really low satisfaction. What were you looking for? What kind of outcomes were you looking for? Right?

Let me pull up the next question. How to— okay, I'm going to actually rephrase it as I talk, but someone's asking how to have a standardized data ingestion pipeline when you've got complex data like PDFs, PowerPoint, flowcharts, visual charts, and so on, which is Yeah, I think one of the challenges I've seen most frequently, uh, is, you know, especially charts. Um, do you have any, uh, quick reactions to this, Jason?

Yeah, so if we have an understanding of what kind of questions people are asking, what I would— I usually prefer to do is I would prefer to commit much more compute up front try to pre-answer these questions, right? Like, um, if I have a bunch of images in a PDF, is the goal to really pull questions out? Like, do I just want to be finding images, or is it usually the case that the paragraphs above images are also referencing these images themselves and I want to do some kind of question answering system, right? Like, for example, um, uh, one of them is like pictures of clothing, right? So if I have bounding boxes that I run on, onto these images, maybe what I'm actually doing is I just want to find, like, find all images where it contains at least one shirt. So understanding what questions people are asking is usually the first step. The thing that is more general would be around generating summary indices, right? Where, like, for all images, I can generate a text summary of what these images look like. For charts, I can generate text summaries of those charts. And again, the goal here really is to create synthetic text chunks that I can use to look up alongside this like other multimodal data. And if I know what kind of questions people are asking, I can always say, given a chart, given this chart and questions people ask about charts, you know, what are some questions that would treat this chart? And again, building a baseline of what exactly is going on.

I would just add to that, um, So I've worked on PDFs that had many tables, um, and there are, uh, Python libraries that will help you extract those tables into CSV. Uh, they're really finicky, and so if your tables have, for instance, uh, common problem that we saw is nested headers. So you could have like in the header you'd be like revenue and then subheader would be individual columns would be revenue broken down by country. We never got that to work very well. But if your tables are pretty standard and simple, we were able to extract those to CSV and then it passed the CSV to the LLM. And we've had not fantastic results, but reasonable results from dealing with— Tables like that, visual charts, I imagine that you could use models that are, that have a vision component. Again, something I haven't worked on a lot myself, but yeah.

Like one specific thing I like to call out in this tables example is, you know, understand what kind of questions are being asked about these tables is a huge lift, right? And sometimes when I have a table, I might be trying to do aggregate statistics of that table, in which case I might want to then be able to do something like taking— given a CSV file, write some kind of Texas SQL engine to answer questions on aggregates. That's not always the kind of, you know, data, data masses we want, right? Other times we might be processing specifications and like large tables where we're trying to find like, you know, certain rows, right? If we knew that ahead of time that we knew that we want to do summary statistics versus a scan of the table, we can build separate systems that can do that, right? And that really comes from, again, looking at the questions and making sure that this is what you're asking for, right? Like, I think before we were processing like tables with like 20,000 rows, and my solution was, oh, let's put this into SQLite. But it turns out they just wanted to figure out like, when did we order like these kind of hex screws for this kind of, you know, construction site? And we just needed to chunk them, uh, like 20 lines each, we'll be able to find the answer because we only looked for figuring out like which row something existed in.

Yep. Another question, I, I really like this one, from Sai Ji, uh, any suggestions on capturing feedback when the RAG element is hidden away from the user? For instance, I'm using agents to build a report. My RAG tool— and my RAG is a tool the agent uses to build elements of the report. Since the user doesn't direct— since the user does not directly use or query the RAG, how do you suggest to capture feedback?

Yeah, like in the report context, you, you kind of still have like fields and inputs. And so one piece of direct feedback is, you know, do we, do we ask the user to ever correct the inputs? If there was a correction, that's a piece of feedback, right? Now we know that like, oh, the fields of this type require a lot of user corrections. What fields are those? Turns out when the fields are enums of a high cardinality, we tend to fail. Like, that could be like a discovery. So on a field level, there's, there's edits that we can do. Without the feedback, we still have this idea of the re-ranker and the cosine distance, right? Even if we don't touch anything, for every internal RAG API call that we make, we are getting n text chunks. And for those n text chunks, we are getting average cosine distances and the max re-ranker relevancy, the mean re-ranker relevancy, right? And, you know, whether it's like 0.7 or 0.8, that's not really going to tell us much. But if certain fields are getting us like 0.3 and other fields are getting us 0.8, it still guides us to where to look to come up with a hypothesis to then come up with an experiment.

If you have a report that you're generating with the RAG and you're not able to edit it, it seems like really bad UX, really bad product design, I think.

Yeah.

Yeah.

And, and it really depends, like if your report is generated using like a markdown output, it's really hard to like assign back what is going on. Whereas if you have a report that's like a JSON object where there's keys and values, we can figure out, okay, like this key is having lots of edits and we can fix that. So there's a little bit of like structured reasoning in how we even design the report experience.

Dan, I was gonna say, this also seems to me to be the case, and it's probably the one that I have the most experience with. This seems to be a case that might be the easiest case if you're extracting data for, for a report. There might just be a ground truth, or the, the case that I've dealt with, there's a ground truth. Now we can literally just write a set of assertions and say instead of relying on the feedback being— excuse me— thumbs up or thumbs down, we need to extract a certain fact. We're going to write a test. And the benefit of this compared to one that uses user feedback is user feedback, you actually have to deploy a system to figure out whether it's an improvement or not. But if we want to experiment, there's changing the model or changing something about how we do chunking. Improve the system if I can write a bunch of tests because I know I'm just pulling out facts to feed into a report. Now I can iterate very quickly because I can test that in seconds rather than days, which you would typically need for a deployed system. The next most upvoted one I think is a comment slash question. Someone says cosine usually isn't an exact measure of relevancy. It's relative and to some degree arbitrary. Why would you recommend including it as a measure anyways?

Yeah, the quick answer is it is at least a number to look at, or like, right? What happens is if I wanted to generate, like, RAG to do emails for a sales project, the best metric would be like sales conversion. If I could capture that data, I would do it and I would try my very hard, but usually what's gonna happen is I'm gonna have maybe hundreds of emails before I get a sale. And so I don't have enough signal in low volume to really make anything actionable. If I did a thumbs up button, maybe with 1,000 emails I get 50 interactions. And so I might need to have like tens of thousands of emails to even be able to figure out satisfaction. Whereas with cosine distance, it is at least a metric. That I can have access to for every single request. And the idea is like, as I increase my volume, uh, I should be able to rely less and less on that, right? And so you— in this case, it was sum of ups, sums down is the best we have. Cosine is available for everything. But really, you know, we're just trying to have some direction to go, and we got to recognize that like, yeah, like not all metrics are useful, but they could be useful in, in sort of a moment in time.

Also, the, the way we're—

you—

the fact that it's relative is fine because we're actually using— trying to figure out, like, we're using it in a relative way. Um, all right, how about this one? I'm gonna reorder by current upvotes. Can you speak to the use of graph embeddings in RAG and how to evaluate that?

I mean, I don't really work too much with knowledge graphs or graphs in general. Usually if there was a graph Like, I'm going to be like, talk a little bit more technical than maybe— I don't, I don't know how technical the audience is, but if there was a graph and I was sort of factoring the adjacency matrix, all I'm really saying is, you know, I want to train, I want to fine-tune my embedding model beyond something that's like a Sentence Transformer or a, um, or a, uh, like OpenAI call in order to have some outcome, right? And the question is You know, the question I would really be asking is, if I switched out OpenAI embeddings with Graph embeddings in the search tool, uh, how much would my relevancy metrics show up? And the answer is, you know, depending on the question type, it doesn't really matter. If the question was last modified date, you know, maybe, maybe the Graph won't be able to do that, whereas the Graph embeddings would be able to improve, you know, the, the content topic type clusters, but not the capability clusters, right? And so these kind of questions are really hard to evaluate. Unless we have these sort of segments and scores that we care about in order to help us prioritize.

Got a question from Peter, uh, one that quite a few people hypothesize about. How much, uh, of what you're talking about today, which is a RAG, um, will still be relevant when a hypothetical GPT-5 is released?

Yep, I think something like this will be even easier when GPT-5 is released because part of it is just understanding how to write queries and recognizing that the embedding isn't the only thing that matters. So like if we embed a text chunk and the question was like, can you compare and contrast the pros and cons of these 3 services, right? GPT-5 will just be able to write, you know, maybe 6 queries: the pros of this service, the cons of this service, the pros of that service, and then be able to synthesize that information. And so I don't think we ever get into a place where this is going to be something that fails, especially if you recognize that even as long context grows larger, it does not mean that search becomes less important, right? Because if you're actually a business, it's not just accuracy or user satisfaction. You also have to trade off things like latency for business outcomes, right? Like, if GPT-5 is really good and you can put 10 million tokens in it, and I'm building an application that's going to sell products, if the response time is 5 seconds, I might— someone might click the back button. I might not, not actually buy the product. And so there's always going to be trade-offs we have to make. That said, understanding what the— what your customer is actually asking, that is a skill that you develop as, like, the data scientist and as the business. And those things will be, you know, here forever. If anything, I would just love to be able to copy paste all the questions in the GPT-5 and ask it for the clusters and ask it for the prioritizations. But, you know, building that thumbs up, thumbs down button, like that's something that is sort of separate from the analysis that GPT-5 could do.

All right. Um, I've got another one, which I think is again about knowledge graphs. I think we've covered that. Um, can you talk a bit more about date and doc type filters? Uh, do you embed something about that in the LM? So yeah, uh, I think there's someone basically just asking, can you talk a little bit about metadata filtering, even just like what is the idea of it and how does it work?

Yeah, if you see my screen, do you see the slides or do you see a blog post?

See a blog post.

Okay, perfect. Um, one simple way of doing this is actually using something like Instructure to generate structured outputs, right? So in this example, I have an output model that is a, like, metaphor query that has a rewritten query, a published date range, and an allowed list of domains. And my prompt effectively says, given a request that comes in, I want you to structure it in a way that I can use in a subsequent search system. And so now, you know, for a question that is like, what are some recent developments in AI? I actually get a question. I get a rewritten query that looks like this. It says, okay, the query should be like novel developments, advancements, AI, Arthur, blah, blah, blah. I create date ranges because this is trying to capture what recency looks like. And in this specific instance, it also recognizes that there are good domains that I want to search against. So in this case, the category or type is going to be, I only want to return documents created from this time range from archive, right? And this is something that you kind of have to customize because you are the one that has to build out the start and end date range, right? But because you understand that this is something that's important, it is something that you can go out and build and deterministically know that this will improve how your search application runs. And so if you have different document types, you know, maybe this is something that is, you know, the content in your CRM that is tagged by your marketer. And so you know there is a pricing tag, and if you knew that ahead of time, you can just make sure you only search documents across certain prices— sorry, about pricing.

Great.

Actually, let me ask a follow-up. When you do this, Jason, so we're getting a query, you're using Instructor to map it to the right metadata to query on. Do you also, when you are putting documents into the vector database, use Instructor to create metadata? A lot of metadata, it just is what it is. You know when the data, document was created. But historically, have you used Instructure to create metadata that you add on for each document in the vector database?

Yep. So one of the things I do— this is again based on the use case. So one really funny example was that for different industries, the financial year does not end in the— the fiscal year does not end in the calendar year. And so we just had some instructions that said, okay, if you're in mining, FY24 is actually in FY23. And we just had a small prompt that just says, for any financial document, convert it to the actual date versus a, like, the effective fiscal year. That's a really small prompt, a really small extraction. A more interesting example actually was around pulling complex diagrams from complex tables, right? In this situation, we knew that this table could be, like, just shoved into the prompt and generate a response. But what we did was we said, given this table, generate for me 6 or 7 questions that would retrieve this table. And then when we actually planted them, we wrote this table in the database, we embedded the questions where the table was metadata rather than embedding the table itself, right? But these are very specific decisions we made because we knew that what people were looking for was we need to just pre-compute an FAQ. Right, and like, that was a decision we made because we realized that what we really needed was FAQs across all documents and couldn't include a table.

We got a question here: what's your favorite platform for building RAG systems? LangChain, LlamaIndex, something else?

I think for the most part I've kept things very simple, right? And so really most of the applications we've built is going to use something like, uh, LanceDB. And the reason we choose LanceDB is primarily because full-text search, SQL, and vector search, uh, can be stored in a single database. Whereas historically, if you use like early versions of Pinecone and OpenSearch, you would have to have an OpenSearch index, a Pinecone index, and a Postgres table in order to be able to do like a WHERE clause and a HAVING clause with a full-text search, with a vector search. You have to like get the indices intersect them and then do something else crazy. And so things like that, like Lancedb, make that very easy. In terms of the query building, you know, like basically what I end up doing is I have a Lancedb instance, I have a processor that converts the user's question into a Pydantic object, and then I just have that Pydantic object have a method called search. And that makes it a very explicit call because I have all this knowledge about what my data looks like. I know that this is a CRM, I know that it's going to have tags. I know that this is going to have a company tag, and I want to embed all that knowledge into the query engine. I think something like LangChain Llama index is very easy when, when we just have sort of like very deterministic general data, but as data becomes much more specific, it's, it's just simpler to have the control you need to, um, like turn those knobs, right?

Uh, how many more questions you want to handle?

Uh, I got time.

Cool, let's keep going then. Um, how do you develop an intuition for what questions you have at the start of a project? How does this change if you're building the RAG from scratch? Do you wait for feedback to come in before you add metrics?

Um, yes and no. So one of the things I do for every project that is at this point in my opinion, a must-have is once you've chunked the data, you should have— the first thing you should do is like use something like a language model to generate 2 or 3 questions per text chunk. And you might say, well, how do you know the LLM is going to ask questions that my user is going to ask? We don't, but it's going to be something. And oftentimes you'll still be surprised at how hard the search problem is. For example, I had been using Paul Graham essays for a long time to demo And so I ran this synthetic data generation process and we found that our search relevancy was like 97%, right? And then we did BM25 and it was also 97%. It turns out for Paul Graham essays, you don't need semantic search, you just need text search. And then I thought, okay, this is probably too easy. Let's just move, let's just move this to GitHub issues. We do the same thing for any GitHub issue, generate a fake query for me. And then this time we got 60%. Full-text search got 61, semantic search got 62, semantic search plus full-text search plus re-ranker like 65. We're like, oh wow, like we actually can't even search— even if we cheated, we can't do well on this. And now you go and begin the process of understanding what the heck is going on, right? It turns out for GitHub, uh, a question might be, how can I get started, right? It turns out if you don't condition on repository you'll never do well. Okay, this means I need to build UI to, you know, condition on this kind of variables, because if I don't have these filters, I'll never be able to do well. And that kind of— that's just what you learn as you're spelunking across the datasets and spelunking across the kinds of questions, right? And the idea is like, today you have 1,000 synthetic questions and no users. Tomorrow you have 1,000 questions and 1,000 users. And eventually you can wean yourself off of this off of the synthetic data. Um, but that's kind of a transition as you deploy to a production system.

What's the term that you— I've been kind of making up a term, um, when I try to explain it to my clients. Like, so there's, you know, there's like query expansion. I call it target expansion. I just kind of make it up, like, to, you know, make each document like into questions and basically make it more likely like the recall happens.

Yeah.

What is the right word for that? I don't, I just feel like I made it up.

I mean, yeah, I basically tried to, so I think the research is like you are building a synthetic text chunk, right? But I just see it as like I am eating additional compute cost on insert in order to improve my retrieval, right? It's like if I'm writing a bunch of, like if I have a primary key and it's an integer, To a database, you're just inserting a row, but behind the scenes, it's like building some kind of tree, right? And like that insert is expensive because I've specified like primary key, bigint, unique, or something, right? And you know, the insert is slower, but I get better read, right? And that's how I think about it. I'm just like doing this pre-computing.

I'm just inserting to make up new terminology. I'm just curious if there was.

Yeah. Okay. Yeah, I think, I think the general consensus is just like synthetic text chunks. Yeah. Coming back to the opposite would be hide. Yeah.

Coming back to this question, it says, how do you develop an intuition for what questions will be asked at the start of a project? I think that if you don't have at least a theory for that, I'd ask like, why are you doing this project? Like most projects, they have some goal and you say like, oh, we want to help users. Find, answer their questions about product reviews. And then, like, it seems to me that I can imagine a hobby project where you're just like, I just want to build a RAG system. But if you've got a business problem you're trying to solve, you should have an intuition about, like, what are the questions people will come in with.

Um, Oh, if you do the BM25 one. Yep, I'll do that one. Um, when you do something like using synthetic data to generate your, uh, question-answer pairs to evaluate, you'll usually find that BM25 and VectorSearch are not that far off in practice. Like, it might be a 67, 68, you know, 92, 95, but there are specific examples where BM25 is much, much more superior. The first one is if the author of the document is also the searcher, we're much more likely to have exact word matches on how people search for things. Because I wrote the, I wrote the thing. I just, I just like, I had 15 hours of interviews. I just want to search across my transcripts. I, I know how I talk, basically. So that's one thing. The second one is very specific, which is oftentimes, uh, if you are building like an enterprise RAG application Like, I would not be surprised if 30% of the users, uh, use it as document search. Like, they don't really care about the answer, they just got to figure out which document it was. And it turns out, if again, like, if you are the author of these documents, you know the file name, and BM25 will match the file name and outperform semantic search by, you know, multiples, right? Because if I know the file name is just like company name 24 report, I, I do want to just be able to say like company name wrote like 24 report, I'd be able to find that document. And just matching on file name alone will get me way better results than, and then vector search. And these are just small examples of why it's effectively, you know, I think a default to be able to include BM25 in conjunction with vectors.

Uh, we got a question with 4 upvotes, but I think we had to get it at just the right moment in time. Uh, someone says, wait, can we deep dive on that idea? What's a good UX for, uh, for report editing? Uh, and I think this was related to— Hamel made a comment about, um, if you're generating reports, they should have editing. I don't know that if that's so topical to where we are at this point. Do you have anything else you want to say about that, Hamel?

Um, the only thing, I mean, I guess like you have to make your— make sure your product is good if you're gonna— it's not just AI. Narrowly focus on AI. So you have to think about it holistically. I mean, yeah, you can focus on AI all you want, but if the— yeah, you have to think about the product too. It's kind of like being a data scientist too. This is the case in ML as well. A lot of times, as you become more senior, it often—

sorry, guys— about the product as well.

Yeah, people can get my—

sorry. All right, so my Wi-Fi got knocked off. Uh, where are we?

You're back now. Yeah, cool, I got it. Um, go ahead.

Yeah, so the question was like, how do we improve the UX of report editing? I think the dangerous thing that I would love to avoid is having the report be generated as a Markdown file. Because if you generate as a Markdown file, you might say, okay, well, if I have headers and action and bullets, it's structured output and this is going to be easy to edit. But what's going to be hard is it's hard to attribute edits for certain regions of a document. And so if I can be much more opinionated and do structured outputs, right, where I have keys and values, there's inputs and input fields, then I can much more easily edit specific parts of a problem, right? Like instead of having action items and summary as Markdown, if I said action items are a list of strings, summary is a text field, then I can delete an action item with a delete action. I can, instead of going from 3, I can just add a new one and create a new one. Those actions are a little bit more straightforward to measure. Right, just if you create the action item, if you add another one, that means you have bad recall. If you delete one, it means you had bad precision, right? You can again sort of come up with like better—

As structured data either, that's just like implementation behind the scenes. So yeah.

Okay, how do you— we actually Okay, how do you do RAG where we want the LM to pinpoint which documents they got the answer from after getting the top K documents? Uh, for example, after it gets 10 documents from RAG and then the answer for the top 10 documents, basically, how do you do citations?

Yeah, so the easiest way to cite is to cite entire text chunks, right? So what this hap— what happens is if you have 10 text chunks, you can just format the prompt in a way where you can say here's 10 text chunks, each one is an ID, and as you generate your answer, include, uh, like a citation, right? And maybe the answer is just square bracket chunk text chunk ID. Um, and usually that works, right? If these text chunks are very small, then you can still build some UI that says if I mouse over like the square brackets 2, I want you to render text chunk number 2. That's like a very straightforward way of doing it. A much more complex way of doing it could be tucked like citing by spans and whatnot. But generally the idea is that because most of these language models are trained to output Markdown, if you make the citation look like a Markdown URL, it works pretty well. And so you can say not only does every text chunk have a user ID, but to make it look like a URL, it actually performs so much better in practice.

What do you think about, uh, high-dimension embedding models as re-rankers versus cross-encoder models?

Yeah, I almost always want to use both, right? Like, again, it's mostly like a latency trade-off. Uh, vector databases are pretty fast in terms of search relative to, like, you know, cross-encoders. And so the— again, the idea is like, I might have a million documents, vector search gives me 100 documents, and the re-ranker gets me, uh, you know, 20 documents. And really, I'm just making trade-offs between precision, recall, and the latency of these systems, right? It's going to be very hard for a vector database to realize whether or not "I love coffee" and "I hate coffee in bed" to be similar or different, but it is something that your cross-encoder is going to be much, uh, more comfortable handling.

Next question is the same about just adding citations. We talked about using Markdown for that. Can you comment on hierarchical retrievers from Llama Index, fine-tuning embeddings to improve RAG, how effective— this is actually two questions. Can you comment on hierarchical retrievers and then also separately, Um, key comment on fine-tuning your embedding model to get better, um, measures of, uh, relevance for RAG.

Yeah, so with the hierarchical retriever stuff, yeah, like funny enough, I was like talking with Jerry on that kind of stuff. I think generally just like as language models get better, we'll, we'll definitely need to reason about this like hierarchy less and less because like these hierarchical changes are probably going to be like small percentage point improvements. Right? In which case, like, a longer context just means we don't have to be as clever to solve these kind of problems. Like, the reason hierarchical matters might be if we had 4 text chunks in a row, 1, 2, and 3 were retrieved, the idea is like maybe 4 should— like, 3 should have been retrieved too, so there's no like missing context. I think that can be solved by just having bigger chunks and like better retrieval. But, um, this is like an opinion I hold strongly actually, mostly because I've been doing like fine-tuning embedding models for search since like 2015. 2015. Um, I think any company that is making money is leaving money on the table by not fine-tuning embedding models. If you think about large e-commerce websites, you know, Netflix trains an embedding model because they know what you watch and what you don't watch. And, you know, we can like have that information be used to build the recommendation system that's better. Amazon has their own custom embedding models because they have checkout data, and so they know that a user embedding and a product embedding is, is very, very good. And it's just unclear whether or not I believe that OpenAI has data with question-answer pairs for your domain in their training data, right? So it's like, chances are OpenAI does not have that data available, and you have that data available. On top of that, because of how smart these embedding models are in general, uh, if you go and search like modal embedding fine-tuning, I actually worked on a project with modal where we found that Even with 2,000 question-answer pairs, we will outperform OpenAI and Cohere, right? And so in these relevancy tests, we got maybe like 84% from OpenAI by default. But if you fine-tune a 200-megabyte BERT model, you're going to get 86% with 2,000 examples, and you're going to get like 89% with 20,000 examples. And if you're a company with like real revenue and real users, you're going to get 20,000 questions in in weeks, right? And then you can do much more interesting things and recognize that, hey, you know, before I had to like synthetically generate text chunks or synthetically generate text chunks from the questions because I know that the question and the text chunk don't look the same, and I'm hoping the embedding model can capture this. If you fine-tune your embedding model, it does exactly what you want it to do.

Um, coming back to a topic from earlier, having a really tough time extracting data from tables and PDFs in RAG pipelines. Any tips to improve that?

Yeah, um, a blanket answer I feel pretty comfortable saying now is actually try looking at things like LlamaParse. They've put in a lot of good work in blending language models and existing text extract tools to basically generate tables that are better. A simple answer in terms of prompting actually is to just switch from CSVs to Markdown tables. I found that in practice, giving language models— asking language models to output Markdown tables is actually higher performance than generating CSVs. And I have like some recall metrics of like, can you find the row in the— in, in like a 600-logline column, it does look like Markdown is a better tool. So for example, if you had a PDF and a table that you have a hard time parsing, it might be worth giving this image to GPT-4o or to Opus and asking for it to generate Markdown tables back out. And then it'll be able to handle things like multiple headers or multiple rows and indices because it can just manage the whitespace.

But All right, thanks for the informative session. Do you have any specific guidelines, metrics, telemetry hooks where knowledge extraction, and it involves extraction? Oh, uh, you want to read this one? I can also read it, but yeah, I can try.

Um, basically the question was like, are there any guidelines on, you know, how metrics, telemetry hooks to figure out where we should instrument our system? In order to sort of figure out how our system is working. Luckily enough, I spent like 3 years instrumenting recommendation systems. If you go back in my writing, there's a blog post called N Levels of Complexity of RAG Applications, and I kind of just outline where I like to log and what kind of data I try to log. Right. Yeah. Because it basically looks like a recommendation system, right? Which is a user signed up, like signs up, I'm going to show them 10 products, they're going to scroll and look at the products, they're going to click a product, they're going to buy the product, right? Um, it's very much the same thing as like a question came in, we use a bunch of text chunks, right? One simple thing that we can do is if I give it 10 text chunks and it generates an answer and cites 2 of them, you could imagine those 2 are more relevant than the other 8. That is still feedback that you can put into an embedding fine-tuning model. You can say, okay, for, for any n text chunks I present to the language model, uh, I want to fine-tune a model that ranks the ones that were cited higher than the ones that were not cited, right? And so you can again, like, use these little micro games in your infrastructure to generate the data you need to fine-tune a model in the future. But this is all kind of like in, in the end levels of RAG posts, and as I build more, I'll have better information.

Uh, what's your recommended chunking strategy? How big should the chunks be? Should you have overlap between chunks?

Yep, so this is a recommendation that I'm kind of stealing from OpenAI and Anthropic. And interestingly enough, uh, as they are increasing their context lengths, the recommended chunk sizes have also increased. So back in the GPT-4 32K world, I think everyone was recommending like, like 500 tokens with a 50% overlap. I think now both Opus and Claude are like 100K context, and now both of them roughly recommend like 800 overlap, 50%. What I've really found is when you actually generate these synthetic datasets, right, with question-answer pairs and you play around with the chunk size, I have not seen too much of a performance change, right? And so what I, yeah, what I basically do is I just follow their advice, 800 tokens, 50% overlap. And if I want to improve my systems, usually I'll improve them by generating synthetic text chunks of whole documents rather than trying to augment individual text chunks, if that makes any sense.

Is the answer like have evals and test, at least test some things.

Yeah, but I also find that like I've never had an eval where when I went from 500 to 800, like something amazing happened, right? Um, because it's so specific, like, like certain kinds of data have longer, like, longer token, uh, references, right? It's like, it, like, it really depends what kind of data you have. But that is something you would basically be able to find out if you generated synthetic datasets, right? Like, it turns out that, you know, in legal documents, it turns out on page 7, it references a date that is a reference to a reference of the signed date, which is on page 28. Like, that might not be detectable if you use like 50 tokens, like 500 tokens, right? Because it'll just say like, well, this is due 35 days after signing, but signing date is like a different document somewhere else. Right? But that's, that's very specific, and you'll still be able to uncover that with synthetic data. Let me, let me refresh this.

Yeah, refresh. That looks good. There's a question.

Yeah, um, what kind of model do you use? Because I mostly work with enterprises, the model choice ends up going down to either Opus, Haiku, or, uh, 4o, primarily because we can afford to be less worried about context length limitations. And so, but just by using one of these models, we know it's going to do a good job. Um, the only— yeah, I think that's, that's mostly the, the thing, because we have that context window, we have— we get to be a little bit less clever with how we do our search, and we can have much higher resolution instructions on how to generate outputs that we can use where we can render citations and we can render different interesting UXs, right? So for example, when for Limitless, not only do we generate the action items, we generate action items with assignees, and Opisk is able to actually generate a URL that we can parse and render like a profile head or something like that. Those are the things that we matter. And for large-scale applications, it's really hard to get around the rate limits that we have on some smaller systems. Another question is, any picks on picking embedding models? Ultimately, again, this is about having those evals, right? Like, if you have that synthetic set of 1,000 questions against 10,000 text chunks and you have your eval that is just, you know, when I search this, the top 3 must contain the original text chunk, we just gotta try them all and figure out what's what. And luckily enough, if you, if you were in a world where you had, you know, 2,000 questions and 10,000 relevant text chunks, um, any model you fine-tune will just outperform whatever base model you have. And so you kind of need the evals to make a decision in the beginning, but once you actually have production-level data, like even 3,000 question-answer pairs, uh, you'll just fine-tune, in which case you're making a different set of constraints, right? How big is the model? How much throughput do I have on the model? Does it fit on an A10 or A100? Those end up being the decisions you have to make. Let me think. I want to talk about this one just because it's interesting. How much do you look at the data and inspect the data up front to understand the questions versus adapting over time? In a bigger company, like maybe not a company like 10 people, a company that's like 50 people, I imagine there's going to be someone that's just constantly looking at this data, like maybe like once a week. Um, like one small thing that I do kind of like the stand-up process is when we start this initiative, it's going to be like, you know, give me a CSV file of like 40,000 question-answer pairs. I'm gonna run some clustering algorithm, we'll look at the clusters each Have some ideas, rerun it, have some ideas. And then what I do is I create like very explicit labels. I say, okay, like this is a marketing, like pricing, da da da da da da, capabilities, compare, contrast, needle in a haystack, et cetera. And I log that. And basically what I do is I just make sure I have an other key and I monitor the percentage of questions that are other over real time. And so I'll just like check once a week and hopefully other doesn't go from like 10% to like 50% randomly. And if it does, I can now just come back in and say, show me the 10,000 questions that were other, rerun the cluster. What did I learn? So it's, it's gonna be a very much iterative process. Uh, you know, it's, it is gonna be iterative for the rest of your life as long as you're getting new clients and new customers and new users and new inventory to search against and new types of questions. And, you know, we just do that kind of like during standup. We're just like, hey, you know, 10% of the questions were other. Uh, for some reason, for questions around categories and date times, it has like 30% satisfaction. Let's go double-click and figure out like what the heck is going on.

How do you, um, teach your clients to look at data? I feel like it can be hard too.

Oh man, if I did I would, uh, I would have a job and thank myself a job. No, I think it's really hard. Like, it really is just like the science, because it's basically about teaching like engineers a scientific method, right? I think I've been thinking about more as like, what is the report you can create? What is the data that you can collect? And then what are some hypotheses? And then once you have these hypotheses, uh, for example, like, I think that we are not doing well because It looks like all these questions have recent, recency filters and we don't have recency filters. That's a hypothesis. The experiment is to add recency filters and the observable is, you know, that satisfaction score should go up. Like a simple example was people had feedback on like the transcript summaries we generated and it kind of felt like the issue was just, it was too short. Like a 30-minute meeting and a 2-hour meeting had the same length in transcript. So we had that hypothesis. So I just plotted like transcript length versus summary length. And like the color was satisfaction. I said, oh, it looks like they're correlated. And then what we did in practice, we just said, we just added a line that's like, make sure the summary's length is proportional to the length of the transcript. We ran an eval that just said, for these 50 transcripts, Without this prompt, it was like 1,100 characters on average. And then when we added the prompt, it was 2,000 on average. Okay, that's my experiment. I'm gonna deploy this and see if satisfaction went up. And it went up, right? But it's hard to figure out what the system there is. But the really, the ideas are like, yeah, it's like hypothesis, data, intervention, experiment, hypothesis, data, intervention, experiment.

Seems like, uh, yeah, seems like data scientist.

Yeah. Um, the most popular question right now is just vector databases versus cross-encoders. Oh, I think we answered that. Um, yeah, I think it just comes back down to really it's going to be your evals and your latency constraints as a function of your evals and your business outcomes. In recommendation systems, 100 milliseconds of latency could make you 1% higher revenue. And so that might just be the difference of I want to top K 1,000 using embeddings and top 100 using re-ranker versus, you know, top K 10,000, then top 100 re-rankers, right? Like, that just might be slower even if it might be 1% better. I make a size less money in the long run. Yeah. So let me, let's do this one. I had a client that wanted to ask questions of lease contracts such as, what is every red flag in terms of late fees? This statement would appear 50 times in a document. How would you search for this? If I knew that upfront, I would just write a synthetic text chunk that it's like, what are the red flags around late fees? And just have that in explicitly, right? Like if it turns out that, you know, 20% of the questions are about late fees, I might just have Opus look at a single PDF and just say, give me all the statements on late fees in a single chunk, and then have that as something that I can augment my, my dataset with, right? You are creating the inventory you need to kind of sell to the customer that you have. Like if you were an e-commerce website and everyone wanted to search shorts but you didn't buy shorts, you just gotta go and like, you know, get some shorts. And I think this would be the same example. I would just take Opus or Haiku and apply Haiku on every document. And in this situation, what you might want to do in practice, you can say, okay, first task is for every document that comes in, can I predict whether or not it is a lease contract? If it is a lease contract, I know that my customers care about red flags on late fees. For every document that I predict as a lease contract, extract out clauses around late fees. And then you put that back into the vector database. Yeah, let's do like 2 more and then wrap this up. Uh, any recommendation for vector stores and how you decide metadata for different cases? Again, it's just like, it's very much like looking at the data. Like, I think the answer is kind of look at the data and like have real examples to work with. The only thing I would call out around vector stores is you don't really need a vector store, you need a search engine, and that includes more than just vectors, right? You have BM25, you have SQL, right? Like if I want to do an aggregate, a vector database is not going to do an aggregate for me, right? And so vector store is only one part of the problem, right? You might— like some companies might just want to use LensDB, some might want to have DynamoDB and OpenSearch and Pinecone in separate places, some might be like S3 files. Those are kind of like implementation details of your business. I think all these have 1-star ratings, so— Okay, I'll do this one as just like a closing statement, which is like, how do you find your first customers in need of RAG for your consulting services? My background is in recommendation systems, and one of my core beliefs is that recommendation systems and RAG systems are pretty much identical. So the business I worked on before was a business called Stitch Fix, where a human customer sends an email to a stylist, and that email is like a paragraph long, and the stylist uses the AI to turn that paragraph into a bunch of search queries. And we, we basically show the stylist like 100 pieces of clothing, and the stylist picks the clothing and then writes a note for our customer. It turns out that process is the same as an LLM. Instead of a stylist, a human sends a paragraph to the style— a human, uh, LLM. We produce like 100 text chunks versus 100 pieces of clothes, and then the, the LLM generates a response, whereas, you know, the style generates response. And so I think I just had a lot of experience instrumenting systems that look like that, monitoring and measuring systems that look like that, and improving systems that look like that. And I've just been sort of converting all this REXIS experience into improving RAG applications. And, uh, yeah, that's all I got.

All right, this is awesome. Uh, thanks, Jason.

Yeah, no problem.

Thank you for going so much over as well, but the questions are—

no worries, no worries. Yeah, I think a lot of these ones just have like, uh, no thumbs up, so I should be good to, uh, I'll save this and put them somewhere else.