Turbocharge Your RAG Applications with Powerful RAG Analytics

Sam.

Hey everyone, my name is Diana Chan Morgan and I run all things community here at DeepLearning AI. Today we have a very special workshop where we're going to be focusing on RAG retrieval Augmented Generation as it has emerged as a leading approach in developing generative AI applications. However, building error free RAG systems comes from significant challenges including maintaining chunk and document quality, refining prompts and addressing output hallucinations. In this session we'll distill our learnings from conversations with hundreds of AI teams utilizing rag. Our focus will include overcoming the five major roadblocks in RAG application development, effective strategies for mitigating these five challenges, introducing four robust evaluation metrics to identify and rectify issues in your RAG system, and how to build production ready rag powered applications 10 times faster. Today's event will be recorded and the slides will go out afterwards. Please ask questions for the speakers. We have dropped a link in the YouTube live chat on Slido where you'll be able to vote upon what are the best questions for us to answer. For our first event partner, I want to introduce 4th Brain. 4th Brain is an AI and ML education company where they offer workshops for engineers and data scientists and generative AI and other ML topics. They also have offerings for executives and leaders on how to adopt AI in your organization. You can then learn more at fourthbrain AI. Now for our presenting speaker we have Galileo. Galileo is a generative AI evaluation observability solution that helps enterprise teams bring production ready applications to market faster. Galileo instantly integrates into any AI environment and enhances critical workflows with collaborative tools and research backed evaluation metrics. Today, Galileo is trusted by AI builders from cutting edge startups to Fortune 500 companies. I'd love to introduce our first speaker, Vikram Chatterjeet. Hey Vikram.

Hey Diana. Thanks for having me.

Absolutely. So prior to Galileo, Vikram led product management at Google AI where his team leveraged language models to build models for the Fortune 2000 across retail, financial services, healthcare and contact centers. He also led product for Google Pay in India, taking it from 0 to 100 million monthly users, the most downloaded fintech app globally. Vikram was one of the early members of the Android OS team and believes Genai is ushering in a similar technology technological wave to mobile. For a second speaker, I'd love to introduce Atin. Atin is was an engineering leader at Uber AI responsible for various machine learning initiatives at the company. He's one of the architects of the world's first feature store. Michelangelo and early engineers on Siri to Apple building their foundational technology and infrastructure that democratized ML at Apple. Hey Aten.

Hey Dana. Thanks for having me.

Absolutely. And we also have some of their the Galileo team in the chat. We have Kike and Jonathan who will also be answering questions in the live chat. Other than that, I'd love for you guys to take it away.

Awesome. Thanks, Diana. Good morning everyone. Super, super excited to have you here for the session today. Thanks so much for making the time, Aten. And I promise that we'll try to make this as educational, as productive for you as possible. And just in general, over the past year we've spoken with hundreds of different AI teams building applications that are leveraging large language models. And honestly, RAG has bubbled up to being one of the most predominant parts of the stack. It keeps coming up and a lot of you here today are probably here because you're either leveraging RAG already in production or in your applications pre production or are super curious about what that looks like and what the pitfalls might be. To effectively use rag, it becomes critical to know its potential failure points and how we can qualitatively as well as quantitatively evaluate the health of our RAG systems. And that's exactly what we aim to cover today. So here's what we will cover today. First, we'll provide an overview of RAG and the common but super critical roadblocks that AI teams face. From there, we'll provide a practical RRG evaluation guide with some qualitative methods as well as some really powerful quantitative methods and metrics for evaluation. We'll then practice what we preach by actually diving into a live demo where we showcase how you can evaluate RG systems using everything that we mentioned in the guide before that, including the qualitative and quantitative metrics. We'll also be sharing an exciting announcement at the end pertaining to ROG evaluation from our research. So do stay tuned for that. We'd love to hear your thoughts and feedback and as we go and speak more and talk more during the session, we'd love to hear from you. So do drop in any questions that you might have in the chat and we'll keep addressing them as we go. So please feel free to drop in your questions as we go. All right, so with that to kick things off and begin, let's take a step back really quick and look at what the three big trends have been in Genai for 2024. First, while we saw AI teams experiment and create demo apps in 2023, with a fledgling AI stack and some really clever prompt engineering, we're now starting to see that that AI stack is getting more firmed up with a multitude of powerful closed and open source LLMs coming about, many many different embedding models getting presented, vector databases becoming really powerful part of the stack, and inference optimization techniques that have emerged to make productionization cheaper and faster. And given this, we are starting to see more and more Genai applications getting introduced and productionized, and this trend will likely only increase in 2024. It's already begun. The second trend we're seeing is that RAG is proving to be quite the winning strategy to productionize LLM applications. And the reasons for this are kind of obvious because RAG allows users to leverage their external data as well as their large language models on this proprietary data that which is pretty much their boat without having to fine tune your own models, which is also a viable strategy. But RAG tends to be tends to short circuit that so you can get to a use case delineation sooner. So it's a quick and cheap way to build really good and powerful applications. And we're just seeing that that Trend increase in 2024. However, most users still are lacking a way of building trust and confidence in the quality of their systems, given how non deterministic, especially generative models and sequence to sequence models can be. Products are getting stuck in that prototype phase because teams and companies are lacking this way of proving to themselves and others that this application can work and actually establishing trust in that application. Whether it's from a hallucination perspective, privacy, security perspective, etc. Evaluation continues to be a big challenge on the path to productionization, but there is a solve for that. Therefore, in today's talk, what we'll focus on are specifically the last two pieces here, RAG and evaluation, and we'll go deeper into both. And since 2024 is the year where we as a community productionize more Genai models and applications do join us on March 13th for a free virtual session where AI leaders from LinkedIn, Comcast, Fidelity, Procter and Gamble and more, as well as founders from AI from Llama Index, VV8 Fireworks Ready Base, which are emerging as very critical parts of the AI stack. We'll talk about the challenges and practical solutions for productionizing Gen AI applications. So truly, truly hearing from people in the trenches. Practical advice? No exact talk. It'll be truly practical advice about the pitfalls and challenges and solutions for productionizing general AI. You can register for free at Run Galileo IO Genai Productionized 2024. Look forward to seeing you there. All right, let's now dive into the first part here, which is a quick overview of RAG and some common but critical roadblocks that teams typically face. Now, just to rewind and take us back to maybe mid 2023 or early 2023, if you will. Right. We started with vanilla prompt engineering where everyone was all about taking a user query and adding a prompt, a system prompt, etc. From there having a model in the loop which would then go about and provide a response that would be generated. Now this prompt as well as this model in the loop was basically the combination that everyone was using. Now this model then generates a response which mostly what we noticed was could tend to be hallucinatory. You could see here in the user query that it says the question was what are the common side effects of metformin? Now this actually happened with an AI team at a healthcare company that we work with where the response from the model said the side effects for this drug, which is a pretty popular drug especially for type 2 diabetes, the side effects for this drug include nausea as well as trouble breathing. Now mostly the LLMs responses tend to be believable for the person without information and knowledge. In this case, when you actually look under the surface, breathing trouble is actually not a common side effect for Metformin. We're not experts, but we extensively Google searched this before just to make sure that was the case and that's where this came to the fore for us. But however, it is a common side effect for other similar kinds of drugs. Right. And so somewhere in the embedding space the LLM basically got confused. And so these kinds of hallucinatory responses are number one common. Right? Because LLMs by nature are basically like these dream engines where they're constantly hallucinating responses. But the whole our job as an AI community is to mitigate any kind of non deterministic response which can harm the user. So the second issue with prompt engineering starts to become the fact that you're always restricted to the data the LLM was pre trained with. Right. So and you're not 100% sure about whether what data exactly OpenAI was pre trained with or maybe even llama, and the llama models have been pre trained with. There's some level of opacity around that. There's also the issue of the fact that they were not necessarily pre trained with the most current information. And so there's a huge dependency on that. So if you're working in a domain where you need very specific kind of information, it may or may not exist in the pre trained data that the model had. The third issue becomes the fact that your proprietary data cannot be leveraged at all to provide context. So in this case, as an example, in the case of this healthcare company, the proprietary knowledge from their medical database is completely locked away from their, from their, from their gen AI systems. And so those are the three big issues, right? The fact that there's a huge tendency to be unable to contain the hallucinations, there's a restriction around the fact that there is opacity around what the data model was pre trained with, as well as the lack of introducing a proprietary data. All three of these kind of led us all to start to figure out are there different ways by which we can just bring our data into the mix and have more control over the outcome of the model. That's where specifically RAG started to become more popular. Now I will preface this by saying that of course you can also fine tune your language model as well. It's an either or, you can do either one of them, you can do both, it's completely fine. However, what we've also noticed is when you start to fine tune your model, you need to labeled data, you need, that is it can be expensive to label the data or create that synthetically. It is more involved. Whereas RAG has emerged as a faster way of getting to that point where you can actually create a pretty well functioning application quickly. And that's a big reason why it's become very popular. However, do keep in mind that fine tuning RAG can happen both in tandem and Most likely in 2024 they are going to start happening together again. Going back to the RAG piece here, let's look at the same user query again where the question was what are the common side effects of Metformin? While coming up with this response, remember all of that data that was locked before from being available to the model, let's take that data back, let's take all of that domain data back and let's chunk them. That's typically what we do in Rag. And there are many, many different chunking strategies by the way, which we can get into. And once you've chunked all of this data into smaller bits and pieces, you can push them into an embedding model. Now this embedding model, what this does is basically it's taking all of these chunks of data and placing them in an N dimensional space. This is just a screen grab from, from Galileo where you can see the embeddings, you can see the specific data in the N dimensional space projected in a two dimensional space. But this is what an embedding model typically does. It just projects that and organizes the chunks in an N dimensional space. There are many different embedding models for you to choose from. If you simply go to the MTEB leaderboard on Hugging Face as an example, there are lots of open source options there, many closed source options available as well. A very powerful One is from OpenAI, the ADA and embedding embedding model. So the many of these embedding models to choose from. However, when you are choosing embedding models right, you have to be careful about the different parameters like what are the max tokens you're looking at, how do you create, what about the re ranking averages for these embedding models? So you have to be mindful about those pieces as well. Once you've actually using one of these embedding models to keep all your chunks in those embedding models, that's when a vector database comes in to be able to retrieve the specific kind of chunks that are useful for responding to that particular user query and that's stored inside of this vector database. From here it's then passed into. Also, there are many popular vector databases by the way. There's VV8 pinecone, which is closed source, Milvis, which is open source, and many others that are powerful and come up often when we speak with people and practitioners out there who are actually building these applications with rag. But with this vector database, what it then does is once it's retrieving all of these chunks, it then takes these specific chunks which are most apt for responding to that question. In this case you can see the first one is about vitamin B12 deficiency. The next one is about metformin being used in certain use cases. All of these chunks are then sent into the prompt, the prompt that you have created. Now, if you remember from the prompt engineering workflow, it's simply the prompt, but in this case it's the prompt plus a very specific context that is useful for that particular question that was asked. From there, once you've augmented the prompt with the right kind of context, it goes into your model and from the model the response is generated instead of it now saying previously it was saying that there's trouble breathing versus now you can actually get a better response where it talks about the side effects include gastrointestinal issues including nausea and abdominal pain, which from our deep Google research we figured out that this is actually accurate very quickly. Using RAG instead of prompt engineering, at bare minimum you can get to more precise answers and unlock all of your data. Needless to say, it's very powerful and a lot of you probably already know about all of this with rog, but hopefully this was helpful as a quick primer. Now, given all of these different pieces in the mix, a lot of things can go wrong. So many, many pitfalls along the way, which I wanted to call out. Number one, when you actually have all of this domain data, you have to make sure that you have sufficient, sufficient data as well as the quality of the data is high, right? And so there should be a good distribution of data. You know your data best, but we've often seen people just pull in a couple of PDFs and expect magic to happen that doesn't, doesn't work that way. You have to know what data to add. Also, when it comes to the chunking strategies, you have to make sure that you're writing using the right chunk sizes. There are, there are downsides of having very large chunks and very tiny chunks as well, which we'll talk about in the guide in the next step in the next section. Apart from this, when it comes to the embedding model itself, there's many different parameters that you would need to optimize. As an example, you can see in different communities that people have been talking about how there's a bunch of different parameters to optimize and sometimes it can come across as a black art. And this dark art concept of using rag. Have we've heard of this multiple, multiple times, which is specifically why evaluation analytics becomes very important. Apart from this, when you actually go and start to retrieve the different chunks, the question becomes given the user query as well as the context that is being provided through all of the knowledge and the chunks, is the retriever really retrieving the right chunks or not? What's the quality of the retriever? It's a very important question to figure out. Are the chunks actually high quality? A good measure of this is to see are these chunks actually attributing to the response or not? It's possible that the retriever is spitting out five different chunks, but zero out of five of them are actually attributing to the response, maybe just one of them, which is again very wasteful when you productionize different models. Apart from that, when you actually go to the next step and you have your own prompt, the question is this the best prompt for this task or not? When you actually then pass all of the prompts and the context to the large language model and generates a response, the immediate question that comes to mind is great, but is this model's response actually adhering to that context that I provided with in the domain data or is it going out of syllabus, going back to the water was pre trained with and maybe hallucinating a lot in its responses. How much is it actually adhering to the context I provided? Especially when you're building out these Genai systems at scale, it's really hard on a query by query basis to figure out whether this adherence was actually happening or not. We will talk about how you can measure this for your own Genai application. This is super, super important to figure out because this is where hallucinations come in, even with reg use cases. And apart from that, the other piece here is given the question that was asked by the user, did the model actually completely answer the user's question or not? So a bunch of different questions across the entire flow and therefore we feel like careful evaluations and experiments across the entire reg stack are very critical for high quality application development. You have to be able to answer most of these questions such that your ROG applications can actually be high quality. It's super, super important. Otherwise you'll use rag, but the models will hallucinate in production. Your entire applications can be super expensive. You can mitigate costs, improve your model quality by taking some very quick steps. So let's talk about that. I'll pass it over to Aten now to talk about a practical ROG evaluation guide. Awesome.

Thank you, Vikram. Yeah, good morning everyone, or good evening depending on where you are. I'm gonna just sort of give you a high level sort of evaluation guide, provide you with that more generally for all kinds of RAG systems and sort of walk you through the key components. We already got like the high level workflow, but the key sort of different parts of the data that sort of go into your RAC system and the key questions that you want to ask to assert the quality of your Rack system, I'm going to go deeper into that and also give you a nice little demo. So to start with, I'm going to present like a simplified view of the inputs and outputs to any RAG system. So on the input side you have the prompt, the instructions, the query as well as the context. And on the input front, the goal of a practitioner is to essentially try to curate the most relevant query, the most relevant prompt, with minimal, accurate and precise context. By minimal and accurate, what I essentially mean is something that can be, you know, that's dense in information, that can be packed with as much information, just enough for the LLM to produce a high quality output. Because there's Also the issue of cost and the number of tokens, which you want to be precise with. The inputs, of course, go into your RAC system, which produces your output. And on the output side, the user's intention is to essentially construct an accurate and a correct response that completely and comprehensively answers the query that was provided. Now, for us, in order to sort of accomplish these goals, there's four key questions that any RAG practitioner would ask related to both the input and the output data, but also related to your RAG parameters, whether it's the, the chunks or anything to do with the RAG workflow. The first question is essentially around the output of the model. How much of it is actually sticking to the context that was provided? I think we already gave an example of a hallucination that we saw when the model went back to its training syllabus and stepped out of the context. The second question is again around the output quality, which is whether the model is answering all parts of the query that's been provided. The third and the fourth are more parameters or questions around the retrieval strategy. The first one being whether the chunks which are retrieved are truly attributing to the model's generation, whether the model is actually using the chunks in its reasoning process. And then the fourth one is within a chunk that's attributed to how much of the chunk is really being utilized to generate the response. Now, you will see how all these four questions sort of interrelate with each other, but we're going to go one by one and see specific examples of the importance of each of these questions. So let's start with the first one. The first question is essentially around the output of the model, how much it is sticking to the syllabus. That's that the context is providing, which is essentially the main principle of rag.

Right?

You're trying to, you know, you build a RAG system to prevent an LLM from spitting out a general response, which is why you provide the context that the model should use. So one of the key things to measure or to look at is the ability to which the model is actually sticking to the response while answering the question. Here's an example question that we just saw. What are the common side effects of metmorphine? The first response from the LLM, the side effects include nausea and trouble breathing at the onset. This looks like a fairly coherent response. It could be correct. But if you look at the context that's provided here, it's very easy to see that the breathing difficulties is not part of the highlighted section, which is what the model picked up in its reasoning. And this is a real customer example that we're sharing with you. So what does this suggest? It essentially suggests it's basically a hallucination where the breathing difficulty is not in the context, and here the adherence to the context is low. And it matters because of the main reason where you want to mitigate the LLM from hallucinating and saying things you're not wanting it to say. The second question is again around the quality of the output. Now, we spoke about the importance of the LLMs adhering to the syllabus that the context is providing, but is that really enough to assert quality? Not really, because there's the other aspect of output quality, which is the comprehensiveness of the output. Now, let's take a simple example here, the same question. What are the side effects of Metmorphine? You did another query to the LLM. This time it gave a slightly better response and it stuck to the context. There's little hallucinations here. If you look at the context, you could see that it's sticking to what's provided and it's answering based on it. But there's much more information in the chunk itself, which talks about the effects of lactic acid, etc. Which also are the common side effects of Metamorphine, which the model did not pick up in its response. So what this suggests is that the response is adherent, but the completeness is low because it did not answer, it did not pick up enough information and the comprehensiveness of the answer was low and that suggests poor quality output. Now, the third and the fourth one are more related to the vector store retrieval strategy. The first one, this one is mostly around the quality of the retrieval part. So you're checking here. Basically you're trying to see whether the LLMs output is attributed by the chunks that are provided. An example here is this is a query for recommendations for blood pressure management. And let's look at the answer. It says recommendations for managing high blood pressure includes weight loss, lifestyle changes and a balanced diet. Seems like a pretty decent generic sort of an answer. But it's interesting when we sort of dive a little bit deeper into the documents or the context that was provided. So here you see that we provided sort of three categories of chunks. One was more around blood pressure docs, BMI docs, and then dietary management docs. And just looking at the response, it's clear that the model sort of attributed the BMI chunks to this response, which is why the output is more around weight loss and lifestyle changes, which is correct, but there's clearly a lot more information that's potentially present in the other docs. Like there's a blood pressure chunk which likely has a lot of useful information which the model or the LLMs generation did not pick up. So here you're seeing that the chunk, the number of chunks being attributed are used in the response is low. And there's two aspects to this. Either the LLM did not leverage the chunks because they are irrelevant, which essentially implies that the amount of context that you're there's a lot of wasteful tokens and wasteful information in the context that you supplied, or the model essentially did not pick up the relevant information in other chunks, which means that the output was incomplete and low quality. The fourth and the final question is essentially around attribution within the chunks. This is like a course measure of intra chunk utilization and this one is related to the third question. But let's take an example to clarify same question recommendations for managing blood pressure. Let's look at the output here. This time the model's response was more around sodium intake, maintaining a healthy weight, which is again a fairly cohesive correct response. You can see that it clearly this time chose the dietary management chunk. And here it picked up in the first section of the chunk to answer this question, which rendered the utilization low. And as a result the response was fairly high level. Correct, but high level. And it it only focused on a small part of the chunk. This is why measuring utilization is important, because again it attributes to two things. Either the model is generating based on a low utilization because there's little information in the rest of the tokens, or the model simply missing out on useful information that it should include in its output. Now, all these four questions. To solve them we need to be able to quantify them for every single input output request that we get here, I'll be talking about four metrics that we've created to help quantify these four questions that we just discussed. Just to quickly walk you through and introduce these metrics to you. The first is context adherence, which is essentially a measure of how much the output is adhering to the context that's provided completeness. As the name suggests, it's measuring the comprehensiveness of the output, whether it's able to answer all aspects of the query or did it leave behind key information or not. Attribution is a coarse measure of context quality, the number of chunks involved in the decision making process of the LLM and and finally there's utilization, which is measuring intra chunk precision. So within the chunk, the extent to which, how much of the sentences and the tokens were utilized in the reasoning process. And you can see that the comprehensiveness of these metrics sort of go from very top level, holistic semantic understanding and completeness, comprehensiveness of the output all the way token level, sort of sentence level utilization in the reasoning process. And all these four questions interrelate and correlate with each other as you'll see from the demo. But before that, I want to highlight how these metrics came to be. To create these metrics, we essentially built a framework for quantifying a lot of these issues we see with modern LLMs. Of course the main starting point for creating this framework was to mitigate or quantify hallucinations. To create these metrics, we essentially built a framework for quantifying a lot of these issues we see with modern LLMs. Of course the main starting point for creating this framework was to mitigate or quantify hallucinations. The method is called chain poll, which is a combination of chain of thought prompting and ensembling which helps us create some of these metrics. And we have a paper here, the chainpole. There's a paper references link here and you'll have access to these slides after. And if I talk about the results of chainpole, you can see that the chainpole methodology for hallucinations, it beats any existing technique, any modern technique to detect rag hallucinations and for the other metrics measuring output quality and chunk retrieval quality for them as well, Number one, there's no metrics today in the industry to surface you this level of insights. So we created the best possible technique that practitioners would use after conversations with many rag practitioners like yourselves, and we created a very strong prompt to GPT ChatGPT 3.5 and we compared that technique with the chainpole technique and we saw that it significantly outperforms across all the four metrics by over 160%. Just to dive a little bit deeper into the results, we of course do our research experiments pretty comprehensively. We have over two dozen benchmark data sets that we run our experiments on. This is one example data set and the results for all the four metrics, if you keep an eye on the blue lines and the blue bars, you'll notice that the chain pool technique that created the four rag metrics that we just spoke about, it significantly outperformed some of the standard techniques which are used, like Raga's faithfulness as well as the OpenAI GPT baseline that we that we tested it against. So all this is to say these metrics of course are extremely precise, they're backed by a lot of research, many months and years at this point of research that we've done. But also they're built for low latency and high throughput because we've baked it into the Galileo platform. And we do a lot of very interesting things like batch ensembling and batch inferencing to make sure that the these results are quantified in the minimal amount of time and with minimal payloads and essentially can handle a lot of scale. One last thing before we get into the demo is to highlight the importance of actionability of all these metrics. So as part of this evaluation guide, I do want to leave you with this almost table, if you will, of if you detect low or high values for each of these metrics, what do you really do? What steps do you take? So in the first case, let's talk about adherence. If you detect low adherence in your system, what all can you do? You can essentially tune your prompt to tell the model to maybe look at certain parts of your context and you can tune the chunk size, maybe increase it, incorporate more data so that the LLM has more data to work with. Or it might be just suggestive of low quality data in your vector store, which means that you add new data. Then there's low completeness. Here is of course, this is an output quality metric. So how do you increase completeness in your response? You add more data to it.

So.

So either you tune your chunk size to add more data or more relevant data, you can even explore a new embedding model because sometimes in real life applications we've seen that the quality of the embeddings, like was mentioned before, plays a significant role in the quality of the chunks that are retrieved from the vector store. The vector store is simply going to store your vectors and give you similar data. But it's really the quality of the embeddings which decides the semantic quality of the data, that context that you're adding. Then there's low utilization. This is a very interesting one because number one, it's a rag parameter or it's a rag metric vector store metric, but it correlates very, very tightly with output quality. For example, if there's low utilization and the LLM seems to have answered, the completeness is high. So all aspects of the query are answered. What that essentially means you want to reduce the chunk size because you are essentially wasting tokens, because the model is answering fully, but not really utilizing much of the context that you're providing it. On the other hand, if the completeness is low, that's a quality issue and you want to increase the chunk size and add more data and inspect your data in the vector store. So all these metrics correlate with each other and play well with one another. And finally there's low attribution, very similar to low utilization. You want to again tune your vector store parameters to increase attribution. That might include removing data from the vector store so that you're supplying clean data and the LLM can attribute more for every request. So, enough talk, we're going to see all of this in action. Just to preface the demo that I'm going to do, just to let you know what the use case is, it's essentially think of a Q and a chatbot. That's a useful tool for a data analyst. And the tool, what it does is it looks at earning call transcripts and it gives you a sort of a knowledge base and a querying system on top of that. And it's. This is a very common use case in the industry. Like a lot of people are building Q and A chatbot systems on top of their data to increase productivity. In this case, if the user is an analyst, I mean they usually have to sift through hours and hours of content and it takes them so long to get to some kind of a conclusion around the queries they're trying to answer. A system like this is extremely beneficial. It's a very popular, very practical system. A lot of open source tools also have these demo applications as part of their documentation. The stack we are using, we're using Pinecone for the vector store LangChain to create the chain itself, OpenAI and Galileo for evaluations. The goal here is to basically do a bunch of requests and, and really evaluate and improve my not only the output quality, which is of course what I care about a lot, but also see how I can get to a very high quality output with minimal chunks, minimum vector store data and really tune these at a request level. So let me dive right into the demo. So I'll start the demo with a notebook that we've created that essentially tests out that qnabot. I'm going to run it right in front of you. So here we just install the libraries that we need. I'm going to set a few environment variables here and here I instantiate Pinecone. I'm using GPT 3.5 as my model. Here I instantiate the VectorDB index. Now these are the questions I'm going to Ask. I'm testing my system with the usual questions which I'm going to ask and this is mostly about asking for the various companies revenues in different quarters and doing a quarter by quarter comparison. That's the use case here. Here I'm going to just create a list of all the prompts that I want to test out with. Here is where I install or start using Galileo. Our Python client is called Prompt Quality and within Prompt Quality we have a bunch of scorers like there's a whole host of metrics that we offer as part of our Guardrails metric store. And these metrics are, most of them are very proprietary to Galileo and they have a lot of research baked in. For example, the first three sort of talk about the metrics that we just spoke about. So the brand new metrics and along with that you can also there's other out of the box metrics we provide to you as a user. I'm going to just instantiate all these metrics here. Gallery also allows you to add a custom metric. If there's a metric that we don't offer in our roster and there's a specific metric that you've created, you can just upload that as a Python function and we host that for you and manage it for you. It's a very flexible customizable platform. If I run that, then the first thing I do is I'm going to do a PQ login which is basically just starting Galileo. It takes me to this Galileo screen here. I'll just zoom in a bit. I copy this API key, paste it in here. This creates a Galileo session. Now my notebook is ready to be used by Galileo with Galileo here. This is where I'm creating the chain. Nothing out of the ordinary here. It's a fairly standard chain that I've created. It's a rag chain. Here I've using Pinecone as the vector retriever. The models, the prompts and I stitch them together into a chain, instantiate the chain. Now the question is, how do you incorporate Galileo into your workflow? So this is a very lang chain specific thing that we've shown. It's a one line callback that you add to your LangChain input before you actually call, invoke your chain. But we have similar hooks for most of the popular RAC based frameworks. Whether you're using Llama Index or any other custom framework, we have low code hooks to hook into any of them. In this case we add this one liner prompt callback going to execute the chain. So this has invoked the chain. It's running on all the data that I provided and at the end of it you should see a processing the model will Galileo will process the chain and it'll give you this link and you go to this link, you basically come here and I'm just going to walk you through. This is the Galileo console, the Galileo UI which allows you to do rapid experimentation and evaluations. At the top we give you a lot of these metrics which are bucketed into different categories. My interest is in the RAG metrics here. You can see that the adherence and attribution, they're all fairly on the lower side. So immediately you get a high level signal of something's going wrong here below this you see that there's this chain experimentation list which basically entails or lists down all the inputs and outputs that were invoked as part of the experiment. So let's go into one of them. So the first one was the question around the prompt was query was the Costco's revenue in Q1 and how it compared to the previous quarter? Let's go into that. I'll just let it load here. You can see that it's interesting here because the completeness of this particular request is high, but the utilization and adherence is low. What does this suggest? If you really think a little bit about it for a second, Completeness is high means that the output had all aspects of the query which was answered. And that's evident in the response. You can see here I have the revenue for Q1. I also have a quarter by quarter figure here of 6.1. So completeness is high, but utilization and adherence is low. A low adherence symbolizes hallucination. So I'd be very curious to go into why the hallucination is happening and what part of the output has hallucinations. So I'm going to dive a little bit deeper into the. Let's look at the chat OpenAI here. So here I can see the entire in this node I can see the full request, but this is a lot to sort of sift through manually. But the final output, I can see that it has the 56.7 billion which was the quarter revenue value and then the 6.1 is the quarter by quarter increase. Now, because the adherence is low here as the signal suggests, I'm going to go into the vector stored retrieval and the first thing I want to see is that in the retriever did the output get attributed to any of the chunks? And the first signal I have here is one of the four chunks attributed to The LLM response. So that means the LLM found the answer and it produced the answer that we saw the 6.1 and the the 56 billion. But clearly something's wrong. So let's see which chunk was attributed here. I found the fourth chunk which said attribution less equal to yes, there's 11% utilization. So part of the chunk was utilized. And if you really look that I found my answer here, it says the 56 billion figure is there and the 6.1 figure is also there. But the 6.1 increase is essentially a year over year increase instead of a quarter over quarter increase, which is why the completeness was low. And sorry, completeness was high because the figure was there but the adherence was low, which means the quality of the output was bad. So you can see how we used a combination of these metrics to figure out a very, very hard to point out hallucination which your standard metrics and basic things will. It'll take you days to read through all this context and really get down to that level of detail. One other interesting example I'll show you where all the rag quality metrics are low. So there's low utilization, low completeness and low adherence. And what that suspects to me is that probably the output is completely wrong or low quality or was not answered. So let's take a look. Let's go inside and see what the output was. So in this case the question was around gaps Q3 revenue and its comparison with the previous quarter. And my hunch was right. The model said that this information is not present. It's a complete ignorance of the question. And it said that sorry, I cannot. It refused to answer it, which is why all three metrics are low, but because adherence and utilization is low. I am interested in going into the vector store to see what happened. The first thing I notice is that again, attribution is present. So the model found the answer but refused to answer it. So that's getting into very interesting territory. So let's go into the chunk that was actually attributed to while making the answer. I'll just zoom in a little bit more here. If you read through this particular chunk, you can see that there's certain numbers which tend to be signaling to the answer. Like net sales of 3.8 billion decreased 7% versus last year sales were down 2%. So the answer is here and the model reasoning did attribute it to this bit, but it did not pick it up. And the reason why it did not pick it up is because the phrase net sales and the phrase in the question, which is revenue, was a bit confusing to the model. Now you would wonder how I found that information out. You can either assert it yourself by reading it, but the magic of Galileo is that all of this that we just did, you get it out of the box in a single shot. How you do it is by hovering over the low adherence. Let me show you this example here.

Sorry.

So over here you can see this is the first example of the which company was this? This is for Broadcom. And you can see the adherence was low because Broadcom's revenue was 9.3 billion. However, the claim there was no information provided how it compared to the previous quarter, which is incorrect. So you can see that the chainpole magic gave you this whole reasoning that we already like. Without Galileo, this would have taken you hours and days. With Galileo manual inspection, it took you a couple of minutes. And with this explanation, it would literally take you a second to debug this. So imagine the power of automating all of this and you get all these insights and explanations out of the box. That is a blitzkrieg demo of the capabilities of Galileo. Of course you can take a lot of actions in the tool itself, which I had mentioned previously. But coming back to just talking about the one last thing here, which is you saw the power of chainpole and you saw how we give you these metrics and give you explanations for all these very hard to find issues in your rag system. And chainpool was designed to figure out very nuanced errors in your rag workflows. But one of the limitations of chainpool was that while it's extremely state of the art, it can get expensive because it's employing the chainpool workflow employees LLMs behind the scenes to do chain of thought prompting or ensembling, which tended to in some cases be more expensive than what people would want. Although we did tons of optimizations on badge inferencing and making sure the scale is not exponential as you do more requests. It was essentially just linear and sublinear, but it was still of course making third party calls, so. So there was the issue with privacy as well. What we realized though is that we wanted to evolve chainpool to the point where you get this kind of production grade evaluation metrics, but at low cost, possibly free in a private manner without sacrificing for accuracy. So over the last few months we've evolved chainpoll to come up with the next version of Chainpool, which is chainpool local. We call it local because we employ essentially run chainpool in A private manner, so there's zero costs associated with it. We work very hard to make sure the accuracy of chainpole does not suffer. And it's built the models, the LLMs, we've replaced them with in house Galileo fine tuned models which are very lightweight models. They're 400 million parameter models which can essentially run on CPUs. Just to show you some very fresh off the oven results here, this is a comparison of the chainpole local technique with the original chainpool technique which used OpenAI models in some cases, in some data sets you can see that the power of the fine tuned model, it beats out OpenAI. We've built these chainpool local so that you get drag metrics without depending on LLMs. It is private and very cost effective and the performance and accuracy of the results is not sacrificed. And with that, that's the end of the presentation. We're happy to take any questions.

Absolutely. Thank you so much. I know our community really enjoyed this presentation. Let's dive into a few questions. Questions. I'm just going to remove the slides.

All right.

What is the best approach of data security is of highest importance. For example, no usage of external APIs and both model and RAG knowledge sources, all on premise.

Yeah, I can take a stab at this really quick. So that, that is actually quite a common request because especially when you're using your own proprietary data, there is no way you can just send it externally to most services. There are lots of options there. So for instance, when it comes to vector databases in rag, there's a lot of open source options as well that we see people using like VV8, Chrome, IDB, many others. Milvis is a very popular one in the enterprise which are available offline or within your own cloud environment. So if you manage your own instance, you can just host them locally and none of the data needs to necessarily leave.

Yeah, Just to add to that, I think one of the challenges in that case would be evaluation is this other area where how do you do private evaluations without losing the power of these third party LLMs and which we've been working very extensively on how to bring the power of OpenAI into your private environments. It's all these research backed metrics which aims to essentially give you the power to do rapid experimentation and figure out what's wrong with your RAC system without sacrificing on privacy.

Absolutely.

Great.

Well, our next question as calculating the embeddings is an expensive task and needs a lot of computational resource for large datasets, what is the solution if Context is really large.

Yeah. I think the reason why chunking was created was to solve this problem where you essentially size your data up to the point where the embeddings are generated in a less expensive way. But again, chunking also has. There's many strategies to chunking and it also is very subjective. It really depends on how you chunk would depend on the kind of data you have, because at the end of the day, what you're trying to do is you want to measure holistically the kind of use case that you're solving and the kind of queries you would get. And your goal should be to chunk in a way where for every category of requests that you get, data that's fetched from the vector DB is fully sufficient and fully complete, like the metrics that we showed you. So these metrics play a key role in even deciding the chunking strategy, because at the end of the day, you want high quality outputs from your LLMs with minimal context and minimal data in the chunks. So the chunking strategy, these metrics play a key role in formulating the strategy for chunking.

Absolutely. Well, that definitely loops into our next question. How do you determine and balance between the cost and size of the context window? Any guidance on chunk size?

Maybe I can just take a stab. There's no silver bullet, honestly, with having one sort of chunk size. It again depends on what your appetite is for. Of course, everyone wants cost. The lesser the cost, the better, but the more context you give it. There is, of course, a balance of you don't want to send too much context and too many chunks. There's a famous paper called Lost in the Middle, which essentially entails the model's reasoning process gets diluted the more data you send it in general, think the lesser the better. And how do you achieve that? There's no silver bullet to achieving lesser. It really depends on the categories of requests that you get. But again, highlighting the importance of these metrics, like measuring chunk attribution, chunk utilization, we go to the token level to show you how to how many tokens were used in the chunk. That should be the guiding light for figuring out the chunk size, which is usually dynamic. There's no one chunk size which will solve all your problems. But it's important to use these metrics to determine, say, for future requests of that category, whether the chunk size should be high or low.

Absolutely. I think we have one or two more questions left. This Is there a specific technique to deal with documents like PDFs provided to RAG, but instead of text, it Contains images of text. Is the performance compromised?

Yeah, there are many different techniques to do this and it's again very pretty common for PDFs to have text, tables, images, et cetera and things. Typically the first step is to just quickly split the file into its raw elements and then, and then take it from there towards chunking. You can also use a lot of open source tools which help you do that. Unstructured is a popular one, which is open source, which helps you parse these images and text and tables from documents and PDFs into digestible JSON which can then be easily consumed by different vector databases.

Absolutely. And I think our last question. Is there a way to untrain a model that was trained with RAG with wrong knowledge, like wrong documents or text?

That's interesting. How RAG is essentially an inference technique. So there's no training per se. That's happening like you're not modifying model weights. But the whole purpose of RAG is to make sure you untrain a model in the sense that you bias it less towards the training data of the model in the first place. As far as, you know, preventing wrong knowledge and wrong documents. I think again, like, I don't want to make this a rhetoric, but coming back to the metrics, again, the whole point of these metrics, the four metrics we spoke about, is to prevent two things, right? Prevent bad quality, which can often happen with the right knowledge provided. But that's. That's indicative of just a bad model request or model response. But also there's. Those metrics are indicative of bad documents or bad knowledge that you're supplying the text with many of. In the slide where I showed the four metrics and the actions you take, a lot of these actions were pointing towards getting new data or replacing your old data to supply better the right knowledge, if you will, for your requests. It also comes down to the embeddings. The quality of the embeddings plays a significant role in the quality of your responses. So it's good to experiment with those as well, because embeddings can. They are basically the knowledge. So if you use the wrong embeddings, it's wrong knowledge in the first place. So there's all these different techniques, but these metrics, whether through Galileo or otherwise, these questions that we showed, they are the guiding light towards tuning your RAG applications.

Absolutely. Well, I think that's all that we have time for today. Thank you so much to Galileo with Vikram and Etin for providing presenting such a wonderful workshop on rag. I hope everyone learned something new today. And the slides will be available after this event. So we hope to see you next time. Thank you everyone.

Bye. Thank you. Thank you.

Knowledge, if you will for your requests. It also comes down to the embeddings. The quality of the embeddings plays a significant role in the quality of your responses. So it's good to experiment with those as well because embeddings can. They are basically the knowledge. So if you use the wrong embeddings, it's wrong knowledge in the first place. So there's all these different techniques but these metrics, whether through Galileo or otherwise, these questions that we showed, they are the guiding light towards tuning your RAG applications.

Absolutely. Well, I think that's all that we have time for today. Thank you so much to Galilea with Vikram and Atin for presenting such a wonderful workshop on rag. I hope everyone learned something new today and the slides will be available after this event. So we hope to see you next time. Thank you everyone.

Bye. Thank you.

Thank you.