Building Production-Ready RAG Applications: Jerry Liu

Hey everyone. My name is Jerry Liu, co-founder and CEO of Llama Index, and today we'll be talking about how to build production-ready RAG applications. I think there's still time for a raffle for the bucket hat. So if you guys stop by our booth, please fill out the Google form. OK, let's get started. So everybody knows that there's been a ton of amazing use cases in GenAI recently. Knowledge search and QA, conversational agents, workflow automation, document processing. These are all things that you can build, especially using the reasoning capabilities of LLMs over your data. So if we just do a quick refresher in terms of paradigms for how do you actually get language models to understand data that hasn't been trained over, there's really like two main paradigms. One is retrieval augmentation, where you like fix the model and you basically create a data pipeline to put context into the prompt from some data source into the input prompt of the language model. So like a vector database, you know, like unstructured text, SQL database, et cetera. The next paradigm here is fine-tuning. How can we bake knowledge into the weights of the network by actually updating the weights of the model itself some adapter on top of the model, but basically some sort of training process over some new data to actually incorporate knowledge. We'll probably talk a little bit more about retrieval augmentation, but this is just like to help you get started and really understanding the mission statement of the company. Okay, let's talk about RAG, Retrieval Augmented Generation. It's become kind of a buzzword recently, but we'll first walk through the current RAG stack for building a QA system. This really consists of two main components, data ingestion as well as data querying, which contains retrieval and synthesis. If you're just getting started in LlamaIndex, you can basically do this in around like 5-ish lines of code. So you don't really need to think about it. But if you do want to learn some of the lower-level components, and I do encourage like every engineer, AI engineer to basically just like learn how these components work under the hood, I would encourage you to check out some of our docs to really understand how do you actually do data ingestion. And data querying, like how do you actually retrieve from a vector database and how do you synthesize that with an LLM? So that's basically the key stack that's kind of emerging these days. Like for every sort of chatbot, like, you know, chat over your PDF, like over your unstructured data, a lot of these things are basically using the same principles of like how do you actually load data from some data source and actually, you know, retrieve and query over it. But I think as developers are actually developing these applications, they're realizing that this isn't quite enough. Like, there's certain issues that you're running into that are blockers for actually being able to productionize these applications. And so what are these challenges with Naive RAG? One aspect here is just like the response. And this is the key thing that we're focused on. Like, the response quality is not very good. You run into, for instance, like bad retrieval issues, like during the retrieval stage from your vector database, if you're not actually returning the relevant chunks from your vector database, you're not going to be able to have the correct context actually put into the LLM. So this includes certain issues like low precision. Not all chunks in the retrieve set are relevant. This leads to like hallucination, like lost-in-the-middle problems. You have a lot of fluff in the return response. This could mean low recall, like your top-k isn't high enough, or basically like the set of like information that you need to actually answer the question is just not there. And of course, there's other issues too, like outdated information. And many of you who are building apps these days might be familiar with some key concepts of just why the LLM isn't always guaranteed to give you a correct answer. There's hallucination, irrelevance, toxicity, bias. There's a lot of issues on the LLM side as well. So what can we do? What can we actually do to try to improve the performance of a retrieval augmented generation application? And for many of you, like, you might be running into certain issues and it really runs the gamut across like the entire pipeline. There's stuff you can do on the data, like, can we store additional information beyond just like the raw text chunks, right, that you're putting in the vector database? Can you optimize that data pipeline somehow, play around with chunk sizes, that type of thing? Can you optimize the embedding representation itself? A lot of times when you're using a pretrained embedding model, it's not really optimal for giving you the best performance. There is the retrieval algorithm. The default thing you do is just look up the top-k most similar elements from your vector database to return to the LLM. Many times that's not enough. And what are both simple things you can do as well as hard things? And there's also synthesis. Why is there— yeah, there's a V in the— anyway, so can we use LLMs for more than generation? And so basically, you can use the LLM to actually help you with reasoning as opposed to just pure, pure, just like, just pure generation, right? You can actually use it to try to reason over, given a question, can you break it down into simpler questions, route to different data sources, and kind of like have a more sophisticated way of like querying your data. Of course, like if you kind of been around some of my recent talks, like I always say, before you actually try any of these techniques, you need to be pretty task-specific and make sure that you need a way to— that you actually have a way to measure performance. So I'll probably spend like 2 minutes talking about evaluation. Simon, my co-founder, just ran a workshop yesterday on really just like how do you evaluate, build a dataset, evaluate RAG systems and help iterate on that. If you missed the workshop, don't worry, we'll have the slides and materials available online so that you can take a look. At a very high level in terms of evaluation, it's important because you basically need to define a benchmark for your system to understand how are you going to iterate on and improve it. And there's like a few different ways you can try to do evaluation, right? I think Anton from, from Chroma was, was just saying some of this, but like you basically need a way to evaluate both the end-to-end solution, like you have your input query as well as the output response. You also want to probably be able to evaluate like specific components. Like if you've diagnosed that the retrieval is, is like the portion that needs improving, you need like retrieval metrics to really understand how can you improve your retrieval system. So there's retrieval and there's synthesis. Let's talk a little bit, just like 30 seconds on each one. Evaluation on retrieval. What does this look like? You basically want to make sure that the stuff that's returned actually answers the query and that you're kind of, you know, not returning a bunch of fluff and that the stuff that you return is relevant to the question. So first you need an evaluation dataset. A lot of people have like human-labeled datasets. If you're in building stuff in prod, you might have like user feedback as well. If not, you can synthetically generate it. Generate a dataset. This dataset is input, like query, and output, the IDs of like the returned documents are relevant to the query. So you need that somehow. Once you have that, you can measure stuff with ranking metrics, right? You can measure stuff like success rate, hit rate, MRR, NDCG, a variety of these things. And so like once you are able to evaluate this, like this really isn't kind of like an LLM problem. This is like an IR problem. And this has been around for at least like a decade or two. But a lot of this is becoming like, you know, it's still very relevant in the face of actually building these LLM apps. The next piece here is there's a retrieve portion, right? But then you generate a response from it, and then how do you actually evaluate the whole thing end to end? So evaluation of the final response given the input, you still want to generate some sort of dataset. So you could do that through like human annotations, user feedback. You could have like ground truth reference answers given the query that really indicates like, hey, this is the proper answer to this question. And you can also just like, you know, synthetically generate it with like GPT-4. You run this through the full RAG pipeline that you built, the retrieval and synthesis, and you can run like LLM-based evals. So label-free evals, with-label evals. There's a lot of projects these days going on about how do you actually properly evaluate the outputs, predicted outputs of a language model. Once you've defined your eval benchmark, now you want to think about how do you actually optimize your RAG systems. So I sent a teaser on the slide, uh, a few, like yesterday, but the way I think about this is that when do you want to actually improve your system? There's like a million things that you can do to try to actually improve your RAG system. Uh, and like you probably don't want to start with the hard stuff first, uh, just because like, you know, part of the value of language models is how it's kind of democratized access to every developer. It's really just made it easy for people to get up and running. And so if for instance you're running into some performance issues with RAG, I'd probably start with the basics. Like, I call it like table stakes RAG techniques, better parsing so that you don't just split by even chunks, like adjusting your chunk sizes, trying out stuff that's already integrated with a vector database like hybrid search, as well as like metadata filters. There's also like advanced retrieval methods that you could try. This is like a little bit more advanced. Some of it pulls from like traditional IR, some of it's more like kind of really like new in this age of like LLM-based apps. There's like re-ranking. That's a traditional concept. There's also concepts in LlamaIndex like recursive retrieval, like dealing with embedded tables, like small-to-big retrieval, and a lot of other stuff that we have that help you potentially improve the performance of your application. And then the last bit, like this kind of gets into more expressive stuff that might be harder to implement, might incur a higher latency and cost, but is potentially more powerful and forward-looking is like agents. Like how do you incorporate agents towards better like RAG pipelines to better answer different types of questions and synthesize information? And how do you actually fine-tune stuff? Let's talk a little bit about the table stakes first. So chunk sizes. Tuning your chunk size can have outsized impacts on performance, right? If you've kind of like played around with RAG systems, this may or may not be obvious to you. What's interesting though is that like more retrieved tokens does not always equate to higher performance. And that if you do like re-ranking of your retrieved tokens, it doesn't necessarily mean that your final generation response is going to be better. And this is This is, again, due to stuff like loss-in-the-middle problems where stuff in the middle of the LLM context window tends to get lost, whereas stuff at the end tends to be a little bit more well-remembered by the LLM. And so I think we did a workshop with like Arise a few— a week ago where basically we showed, you know, there is kind of like an optimal chunk size given your dataset. And a lot of times when you try out stuff like re-ranking, it actually increases your error metrics. Metadata filtering. This is another like very table stakes thing that I think everybody should look into. And I think vector databases like, you know, Chroma, Pinecone, Weaviate, like these vector databases are all implementing these capabilities under the hood. Metadata filtering is basically just like how can you add structured context to your chunks, like your text chunks. And you can use this for both like embeddings as well as synthesis, but it also integrates with like the metadata filter capabilities of a vector database. So metadata is just like, again, structured JSON dictionary. It could be like page number, it could be the document title, it could be the summary of adjacent chunks. You can get creative with it too. You could hallucinate like questions that the chunk answers and it can help retrieval. It can help augment your response quality. It also integrates with the vector database filters. So as an example, let's say the question is over like the SEC, like 10-Q document and like, can you tell me the risk factors in 2024? If you just do raw semantic search, typically it's very low precision. You're going to return a bunch of stuff that may or may not match this. You might even return stuff from like other years if you have a bunch of documents from different years in the same vector collection. And so like you're kind of like rolling the dice a little bit. But one idea here is basically, you know, if you have access to the metadata of the documents and you ask a question like this, you basically combine structured query capabilities by inferring the metadata filters like a WHERE clause in a SQL statement, like a year equals 2021. And you combine that with semantic search to return the most relevant candidates given your query. And this improves the precision of your, of your results. Moving on to stuff that's maybe a bit more advanced, like advanced retrieval is one thing that we found generally helps is this idea of like small to big retrieval. So what does that mean? Basically, right now when you embed a big text chunk, you also synthesize over that text chunk. And so it's a little suboptimal because what if like the embedding representation is like biased because, you know, there's a bunch of fluff in that text chunk that contains a bunch of irrelevant information. You're not actually optimizing your retrieval quality. So embedding a big text chunk sometimes feels a little suboptimal. One thing that you could do is basically embed text at the sentence level or on a smaller level. And then expand that window during synthesis time. And so this is contained in a variety of like LlamaIndex abstractions. But the idea is that you return, you retrieve on more granular pieces of information. So smaller chunks. This makes it so that these chunks are more likely to be retrieved when you actually ask a query over these specific pieces of context. But then you want to make sure that the LLM actually has access to more information to actually synthesize a proper result. So this leads to like more precise retrieval. So we tried this out. It helps avoid some loss-in-the-middle problems. You can set a smaller top-k value, like k equals 2, whereas over this dataset, if you set k equals 5 for naive retrieval over big text chunks, you basically start returning a lot of context. And that kind of leads into issues where maybe the relevant context is in the middle, but you're not able to find out, or the LLM is not able to synthesize over that information. A very related idea here is just embedding a reference to the parent chunk as opposed to the actual text chunk itself. So for instance, if you want to embed not just the raw text chunk or not the text chunk, but actually a smaller chunk or a summary or questions that answer the chunk, we have found that that actually helps to improve retrieval performance a decent amount. And it kind of, again, goes along with this idea. A lot of times you want to embed something that's more more amenable for embedding-based retrieval, but then you want to return enough context so that the LLM can actually synthesize over that information. The next bit here is actually kind of even more advanced stuff, right? This goes on into agents, and this goes on into that last pillar that I, I mentioned, which is how can you use LLMs for, for reasoning as opposed to just synthesis? The intuition here is that like, for a lot of RAG, if you're just using the LLM at the end, you're one, constrained by the quality of your retriever, and you're really only able to do stuff like question answering. And there's certain types of questions and more advanced analysis that you might want to launch that like top-k RAG can't really answer. It's not necessarily just a one-off question. You might need to have like an entire sequence of reasoning steps to actually pull together a piece of information, or you might want to like summarize the document and compare it with like other documents. So one kind of architecture we're exploring right now is this idea of like multi-document agents. What if like instead of just like RAG, we moved a little bit more into agent territory? We modeled each document not just as a sequence of text chunks, but actually as a set of tools that contains the ability to both like summarize that document as well as to do QA over that document over specific facts. And of course, if you want to scale to like, you know, hundreds or thousands or millions of documents, Typically, an agent can only have access to a limited window of tools. So you probably want to do some sort of retrieval on these tools similar to how you want to retrieve like text chunks from a document. The main difference is that because these are tools, you actually want to act upon them. You want to use them as opposed to just like taking the raw text and plugging it into the context window. So blending this combination of like kind of embedding-based retrieval or any sort of retrieval as well as like agent tool use It's a very interesting paradigm that I think is really only possible with this age of LLMs and hasn't really existed before this. Another kind of advanced concept is this idea of fine-tuning. And so fine-tuning, you know, some other presenters have talked about this as well, but the idea of like fine-tuning in a RAG system is that it really optimizes specific pieces of this RAG pipeline for you to kind of better like improve the performance of either retriever or synthesis capabilities. So one thing you can do is fine-tune your embeddings. I think Anton was talking about this as well. Like if you just use a pre-trained model, the embedding representations are not going to be optimized over your specific data. So sometimes you're just going to retrieve the wrong, wrong information. If you can somehow tune these embeddings so that given any sort of like relevant question that the user might ask, that you're actually returning the relevant response, then you're going to have better performance. So an idea here is to generate synthetic query dataset from raw text chunks using LLMs and use this to fine-tune an embedding model. And you can do this, like, if we go back really quick, actually, you can do this by basically fine-tuning the base model itself. You can also fine-tune an adapter on top of the model. And fine-tuning an adapter on top of the model has a few advantages in that you don't require the base model's weights to actually fine-tune stuff. And if you just fine-tune the query, you don't have to re-index your entire document corpus. There's also fine-tuning LLMs, which of course, like, a lot of people are very interested in doing these days. Um, an intuition here specifically for RAG is that if you have a weaker LLM like 3.5 Turbo, like Llama 2, 7B, like, these weaker LLMs are bad— are, are, are not bad at like, um, uh, Wait, yeah, weaker LLMs are maybe a little bit worse at like response synthesis, reasoning, structured outputs, et cetera, compared to like bigger models. So a solution here is what if you can generate a synthetic dataset using a bigger model like GPT-4? That's something we're exploring. And you actually distill that into 3.5 Turbo. So it gets better at chain of thought, longer response quality, better structured outputs, and a lot of other possibilities as well. So all these things are in our docs. There's production RAG, there's fine-tuning, and I have 2 seconds left. So thank you very much.