Deep Dive into LLM Evaluation with Weights & Biases

Sam Hi everyone, my name is Diana.

Chen Morgan and welcome to the next DeepLearning AI event. I bring together all things AI community for everyone and today we are very lucky to have a workshop with some special speakers from Weights and Biases. In this workshop we're going to dive into how we can effectively evaluate LLM systems with a particular focus on retrieval augmented generation, also known as RAG systems. We'll start by discussing the eyeballing technique and why Weights and Biases Prompts stands out as the first great tool in this area. On to supervised evaluation, highlighting why it's worth considering and pointing out some limitations. To wrap things up, we'll also take a look at how LLMs can be used to evaluate themselves, from generating their own evaluation data sets to using standard metrics and even evaluating retrieval systems. Lastly, we'll also touch on how Weights and Biases Sweeps, an excellence tool for hyperparameter optimization, can be utilized to find the ideal balance to maximize accuracy and minimize costs. The session today will end with a Q and A with the presenters, so make sure to answer some or ask some questions in the chat. We'll be presenting a link that you'll be able to vote on all the questions. This workshop is also based off the foundational learnings of Deep Learning's course on evaluating and debugging LLMs built in collaboration with the Weights and Biases teams. Everything covered in the workshop is presented as continued education from the course and we will be dropping the link in the chat. I want to introduce our first speaker, Kerry Phelps. Hey, Kerry. Kerri is a founding product manager at the Weights and Biases teams. She studied computer science at Stanford and also went on to found Carta Healthcare before joining Weights and Biases.

Our next speaker.

Oh, sorry, do you want to say something, Gary?

Thank you, Diana. Excited to be here?

Absolutely. Our next speaker is Ayush. Ayush is also part of the Weights and Biases team as a machine learning engineer and Google developer. Expert in machine learning and TensorFlow. He's interested in everything computer vision and representation learning and for the past seven months he's been working with all things LLMs and RLHF.

Hey Ayush. Hello.

Perfect. Our last speaker for today is Morgan McGuire. Morgan leads the Growth ML team and is an ML engineer at Weights and Biases. He has a background in NLP and previously worked at Facebook on the Safety team where he helped classify and flag potentially high severity content for removal. We're so excited to have them here today. So without further ado, let's definitely get started. So, Carrie, take it away.

Awesome. Thanks, Diana. So I'll zoom out for a moment and talk about the larger product vision of weights and biases and where we're coming from. And then we'll get into more of the details with Ayush and Morgan. So what are our tools for iterating and experimenting on machine learning at a high level? We have a platform across models, which is building your own models, training from scratch, prompts, which is iterating on LLMs, looking at the trace Viewer, which is the focus of the course, as well as monitoring LLMs in production. And then we have our platform components. So shared tools across the system. Let's talk about models for a moment. We started with experiment tracking. OpenAI had this problem where they wanted to see if as they trained models over time they were doing better or worse. It ended up being frustrating to track those metrics in one place. We built an experiment tracking tool. Then they needed to be able to visualize how different data sets compared. We got into this data visualization field and then a collaboration tool, so being able to share those insights centrally, and that's where reports came in. And finally, we need the central system of record for different teams across the organization. So how can everyone working on a project see the results in one place? And critically, and Morgan has been deeply involved in this work, is integrating and making W and B accessible across the different tools available to someone working on a model. So we started out TensorFlow was the really hot thing back in 2018, and PyTorch was kind of the new kid on the scene. And it was clear to us that building integrations with something like Pytorch, with Pytorch Lightning would make W and B more accessible to, say, the academic audience training models. And so this is a critical area of focus for us is how can W and B play really nicely with the other tools in the ecosystem? We also integrate with whatever infrastructure folks are using. So say you're a student just getting into machine learning and you're training on a GPU under your desk or even your laptop itself, we can track there, say you're using the OpenAI API for iterating on a prompt. We can track the results there. Wherever it is that you're doing your machine learning work, your prompt engineering, our goal is to make it really simple to capture those results. Let's talk about prompts, which is the focus of our discussion here. First, with traces, we have a core tool to understand and debug what's happening during the process of Taking an input and producing an output. And the thing that's interesting here is it's not as straightforward as you might imagine. You don't just ask someone a question and directly get an answer. There are steps like, for example, using a tool like a calculator to go calculate an answer and then incorporate that into the result that the LLM produces. And so this debugging can be really complicated, especially if you're trying to look at it on a terminal output. It's hard to parse everything that's going on. So this visual trace timeline gives you a way to understand what's happening as you're interacting with the LLM. And you can also drill into your model's architecture so you can see what the steps are that it's doing. And then under the hood, you can see even more detail the metadata that we're capturing and each of the inputs and outputs that you've fed into the system and received. The goal with prompts at a high level, and this trace tool in particular is making it really easy to iterate on the prompts that you're writing and get a better result from the LLMs that you're working with. That's really the focus of the course. I also want to talk about another exciting thing that we're developing here that's monitoring what happens once you have an LLM that you're monitoring in production. Well, there's a lot of people involved here. The few problems that we see across our different customers we're working with is that it's hard to systematically evaluate what poor performance looks like. Say I have a chatbot helping my customers and I've built this LLM to answer their questions. It's hard to really evaluate if they give it a thumbs down, if that's because the answer was wrong or because they don't like the answer, like, maybe I can't change my flight. So it's hard to systematically evaluate. Is it doing a good job? Is it providing the answer that the user wants to hear, but it's actually incorrect? This is a real challenge for monitoring elements in production. We've also got high inference costs, so it's an expensive process. And it's hard to run traditional patterns like what we've seen in traditional machine learning. It's also there's this really fresh set of tools that people are using. And so it's really hard to just make one decision, like write the prompt once and then leave it and stop iterating. It's really important to be able to keep Tweaking as you get more information about how users are interacting with your system. And it's also really hard to capture and combine data from different points of feedback in your LLM system. And so we've tried to address each of these key points with our production monitoring tools. So being able to explore within the context of the results that you've logged in W and B via evaluation boards, being able to monitor in real time and see the cost of the work that you're doing, be able to integrate with other tools in the WMB ecosystem and beyond, and then streaming those results in live so that you can react quickly. And what that looks like is something like sort of a datadog dashboard.

Right.

But for LLMs in particular. And so this observability tool is this new component of our prompts platform that we're really excited to have now in private preview. And so this is the next step beyond iterating on your prompts and experimenting with what would work well is monitoring the the performance of those models in production. And we'll get into more of the details of what's interesting in our LLM tooling here with Ayush up next. So I'm excited to pass it over to Ayush.

Yeah, thank you, Carrie. Hello, I'm Ayush and I'm a machine learning engineer here at whitsunbasses. I'm really excited for this talk and I think I have good amount of content to cover. So if something is not clear, please feel free to drop in the comment and I will try to answer that or clarify that as well as feel free to reach out to me after this call, after this talk and then I can try to answer that question as well. Having said that, all those who are joining from India, a very happy Independence Day to all of you. And now let's get into the presentation. Okay, so in the past one and a half year we have been interact. Okay, let me see if this is working or not. Yeah, Yeah. In the past one and a half year, the way we are interacting with our world is kind of quickly changing. Right? We envision interacting with software and that's changing the way interact with the tools, that's changing. For instance, as developers, we have relied on Stack Overflow for over a decade now, right? It was heavily used for finding code snippets and solutions to bugs. We either got a working answer because someone already asked for it on Stack Overflow, or we post it on the site and wait for a response. But over the past one and a half year so has lost around 35% of its traffic. And Charge is considered to be a huge reason for the downfall of Stack Overflow. It was reported In March of 2023 I guess that stack overflow alone lost 14% of its traffic. And I mean the reason is that a lot of developers have started asking questions directly to ChatGPT and there's no significant wait time. Right. And you can literally kind of peer program with Charge PT. You can throwing the execution error code back to ChatGPT and maybe it can try to resolve the code for you. It's a more natural way of development. So is GitHub Copilot. It was launched back in October of 2021 and you see the decline of Stack over right after that. For GitHub Copilot, we also have VS code extensions, the place where you're writing your code exactly at the same place you can do code completion and fancy stuffs. And that's a great way of development. And thus the point is that the way we are interacting with our tools and the way we are developing software has changed over the past year and a half. Right. So the question is then what was StackOverfall's response? Well, they released their own generative AI powered overflow AI tool. With this tool you can ask questions like you can ask questions as well as ask follow up questions in a conversational style like we are used to doing ChatGPT. The answers are summarized with citations and that, I mean you can use that to draft your question and later post on SAC Overflow as well. Just like Copilot, they also have their own Visual Studio plugin. So now you can prepare program using this tool. The point is businesses, companies, institutions, etc. They've all started leveraging generative AI capabilities into their own product and services. Awesome. To give a few more example, Khan Academy released their Kanmingo tutor bot. Then there's Notion AI and then Zapier also released some gen AI power interfaces. Great. And I mean the number is just increasing. There's like 250 plus generative AI startups that has been mapped as of August if I remember correctly, August 8th. But again this whole field is exploding now. The reason this is happening is because it has been powered by things like GPT4, Palm 2, Falcon, Llama to Claude models, etc. These are all transformer based architectures which has been scaled to billions of parameters and trained with billions and trillions of tokens at this point with really large context lens as well. The talk, the focus of this talk is on Language language on large language models. But do remember that large image generation models are also playing a role in this landslide shift that we are seeing. Right. Personally and also everyone at VSN Bioses act to be very genuinely proud of our contribution to this whole shift. Because almost every major lab and companies in this space is kind of using weights and biases in some form. Using weights and biases to track experiments, do model versioning and whatnot. And we are instrumented to be honest, in almost every LLM fine tuning workflow that you can find out there. If there's not any, we are working hard to make sure that we can provide that capabilities to dot com to those repositories and workflows as well. So why scaling billion of parameters is a thing? Well, the LLMs are powerful because with scale they are literally getting better on tasks for which they are trained on as well as they're unlocking new abilities. So you train a large language model for say language understanding or summarization, but it turns out that it can do some common sense reasoning, it can do some arithmetic, can answer some physics based questions which is like literally unlocking new abilities. However, under the hood or under the blanket, it's just nest token prediction, right? You provide some prior token or some prior text to those large language models and what they do is if you're creating a summary or if you're doing translation, whatever, you are giving some prior text and based on that prior text is generating the next token based on some probability distribution, right? Special focus on the term prioritize right now because that has like it has adapted into its own whole field right now. And it's like formally called prompts, right? Prompts involve instructions and context passed to language model to achieve desired tasks. Say in a text we can specify the instruction we can pass in the context, we can give in some input data or even some few short examples and we can then have an output indicator and just throw in this text to a sequence model or in this case a large language model and it will just generate the next sequence or next token based on the prior that we have given. And yeah like because of that the whole prompt engineering field has evolved and a lot of prompting techniques has been developed over the past few months. For example. And to be honest, like all these prompting techniques are somewhat unlocking even newer capabilities of these large language models that we didn't think was possible. For example chain of thought or three of thought. These are like really interesting ways of interacting with LLM and or making LLM do things and the beauty is that you're just specifying think step by step in your prompt and then model starts thinking step by step. It's really magical that way. So I mean we talked about language models, we talked about prompting, but to be honest, everything that we are saying right now is kind of possible if we have really flexible frameworks which can encapsulate all this prompting techniques. And like to be honest, a prompting technique might be, might be released a week back and we need to access those techniques. Right? And these frameworks like LangChain and Llama Index has made it really easy to do so they, I mean they have access to data connectors, off the shelf stains. We can do memory management using this framework as well. So the point is with these frameworks it has become really easy to build AI powered solutions. Right now back in 2019, Gartner Research, if I remember correctly, said that only 15% of all the AI solutions deployed in 2022 is going to be successful. I'm not sure about the percentage, but I'm sure that it's really easy now to build such AI powered products because you don't have to kind of start from scratch and go through the whole ML life cycle for your particular use case. You can leverage a large vision models or large language models for that. So like yeah, if you want to build your own QA internal document bot, you might find a LangChain prompt or a Llama index prompt template out there. So hurrah. The point is we have this powerful tool, let's go ahead and solve every problem under the sky. But that's not going to happen. Why is that? Because LLMs are a great primitive, but we still not sure if it can do everything, if it can solve all the problems. And the biggest challenge is if it can, if we can actually properly test the LLM based systems that we want to build for our use cases. And this brings to the main topic of today, evaluation. And going forward I will just be talking about evaluation. If you have any question, feel free to drop in and I can take a look at that as well or maybe at the end of the presentation now before I go in in some depth about how we can evaluate these LLM based systems. For those who are new to evaluation, we first need to understand the difference between the traditional evaluation and LLM system based evaluation. So in traditional ML. Yeah, and to be honest, this ML that I'm talking about is what we used to do few months back. Now LLM has taken over everything. I mean there are still a lot of Folks out there doing proper deep learning. But by traditional might just mean proper, like doing classification, doing like object detection, those stuff. So in a traditional setting we have a training distribution, right. From which we split two sets, the train set and the validation set. Then we train our model on the train set and get some metric. So in this case accuracy of sales, say 90%. Right. We also evaluate the model on the validation set. We don't use the validation set to train the model, we just evaluate the model on that and by doing so we can basically estimate if the model is overfitting or if it's underfitting. Right. We then evaluate on the test set, which hopefully is coming from the production distribution. This evaluation then gives us a domain shift. So what was the shift in the training and the production distribution? And then if you're satisfied with the performance of the model on the test set, we deploy it and evaluate on the production data to get a drift. It's like pretty standard stuff, but. Well, but while we are building LLM based system, we literally don't have access to the training distribution. We might know which data set was used to train the model, but it's kind of hard. We don't exactly know what data was thrown into GPT4 to train it. Right. So most of the time, and most of the folks will not even care about that. And because we are mostly making API calls at this point. Right. So, and the models that were trained, were not trained for your particular use case. So there's no alignment of those LLMs with your production distribution. So these are two big issues that we have in LLM system evaluation today. Right. So before I go ahead, I just want to make a quick distinction between building LLMs and building LLM powered applications. So if you're building LLM, so if you plan to build GPT5 or LLAMA4 or LLAMA3, you can still use traditional evaluations. It's still a very active area of research wherein we are trying to build better benchmarks. But if you're building LLM based applications, we have to be thoughtful about it. You have to come up with like interesting solutions for your particular use case. Whatever. I'm going to talk in the, in, in this presentation. It might not even cater to your use case. So you have to come up with your solution, your own, your own genuinity about this evaluation strategies. Right. So how about we, so how do we think about evaluating an LLM based system? So I was inspired by Joss Tobin stock that he gave a few weeks back and I created this 2 by 2 chart with data with the evaluation data on the y axis and metric on the x axis. So the evaluation data will need to be correlated with your use case. That makes sense. Right? And the data that you want to evaluate on should also be close to your production distribution. So I mean when you're evaluating your LLM based system you have that control but you have to be really thoughtful about how you want to create those data sets. So one thing, and this is maybe the like, probably the first thing you might want to try out is pick up a LLM that's performing really well on public benchmarks. So if an LLM is performing good on public benchmark, it might perform good on your use case as well. But, but there's a like caveat. If your use case is like medical heavy or it's like legal documents, those stuff, you might want to be thoughtful about what public benchmark you are using to pick that model. Right. The model might not be, might not be performing great on that public benchmark, but might not have much intuition about legal documents or, or like your medical use cases. And I mean to solve that problem we have fine tuning that's like whole separate thing at this point which I'm not going to touch upon. But this is where you should start. Right. So if you pick a model based on the public benchmark, then the next, I mean this is not the next recommended way, but this is like one of the most extreme and the recommended way of evaluating your pipeline wherein you hire annotators to build gold standard eval set for each component of your pipeline. So your pipeline might have document parsing with different chunk sizes, might have a retrieval etc. And you need to evaluate every component. So hire annotators to create data set for that. You can imagine that it's both time consuming and it's also very expensive. So if you have like a business or a startup and you have some user base already a dedicated user base who is interacting with your product, you can then do some sort of user user testing. You can do some sort of AB testing with a LLM based logic that's serving your product and custom, I mean heuristical based logic that you were using previously. So you can do the user testing and yeah make incremental changes and see how that's working. Again it's time taking. It might be expensive if like there's some incentive that you need to provide to the users. So it depends on your use case to be honest. But if you do so, it's going to be really correlated with your use case. And it's also going to be really. It's also going to mimic the production distribution really well because literally your users. Somewhere in the fuzzy area is the law. Is the idea of like a crazy idea of using large language models to evaluate large language models? Right. It works, to be honest, and the quickest and this is the quickest way to evaluate like any incremental changes that you make to your pipeline. But having said that, you really need to think about a carefully. Okay. Right. Maybe I can take a pause and see if there's any question I can take. And then we're going to some really meaty stuff. Diana, should I start taking questions or should I carry on with the slide? And maybe I can take that at the end.

We can, we can just continue and take questions at the end, but we'll send out a link to vote on questions.

Awesome. Awesome. Great. Yeah, let's carry on then. So we talked about the changing world. We talk about this LLM space. We talked about how LLM created this whole field of prompt engineering and why prompting is important. We covered traditional evaluation and how the LLM system evaluation is different than the traditional evaluation. We also thought about evaluating a system with that two by two axis. And we thought about few like top level ideas of evaluating. But let's talk about evaluating in practice. Right? Yeah. So in my opinion in practice we have three ways of evaluating an LLM system. Or in my or one can say that this is a more practical workflow for evaluating any LM system. So number one is eyeballing, right. We start building LLM based system. We usually start our evaluation by eyeballing on few samples or few running examples. So we have my three or four examples. We build the system on like around those three, four examples in order to solve those examples. And while doing so we are literally using our eyes to see if the responses have been generated or the chunks or the documents or chunks that's been retrieved by the retriever. They are good or not. Right. And we try different components and we try different prompt templates. So the fact that we are trying a lot of prompt templates and we might be using frameworks like LangChain and Llama Index at this point I will just say that and like biased opinion that you should try Van DB Traces, it's a great tool and carry touched upon that as well. So if you're using LangChain Alarm index, you can literally adding like two or three lines of extra code to your script and you can start Tracking the trace timeline, the input responses, relevant calls, etc. And actually let me quickly show that, like show tracers in action. If you can see the screen, this is a trace timeline or a trace table. So I can see the timestamp when this particular input was added and then there were no response. So the trace. So the chain basically threw a value error. In this case, the question is what is the sine of 0.47 radian? And we got some output. But what's even interesting is that we can look through the sequence of events that happened to come to that decision. So there were some tools used as in like a calculator was used. Let's see this example here, which is working. So a lot of LLM chains were called, calculators were called and calls are made to the Open Chat OpenAI model. We can literally look at the prompt. So what was going in the LLM. And the prompt has both the user prompt instructions, the context. We can see the generation that was done. We can also look at the parameters like for this particular generation, GPT 3.5 turbo was used and temperature zero was used. So everything has been tracked and literally it was just two lines of code that was added to the script. You can start, you can literally debug your pipeline with this because you have access to so much more information that you might miss out if you're just using the framework as it is. Right. So going back to the presentation, the next way of evaluating, again, this is not the next exact step that we should do after eyeballing, but we'll just wanna. I just wanna cover this before I jump into LLM evaluating LLM. Right. So this is a recommended way because this is going to give you the most trustworthy evaluation of your pipeline. But it's expensive and slow, as I mentioned earlier. Yeah, but here's a very simple example that I tried. So the idea was that I will build an LLM based calculator. So the calculator. So you give an expression to the calculator and the calculator is going to use the bodmas rule, it's also called pendmark tool and it will generate the answer for that mathematical expression. So for your problem statement, you might have a problem statement wherein you are generating code. So you can throw the generated code to a computer to execute or compile the code, right? So that's also a form of a supervision that you can provide to the generation that you are doing and you can use that to evaluate. So again, like if you might have, you Might be generic structure, structured SQL. You can run those SQL and see if the. If. If it's giving good correct results or not. So supervision can be achieved in a. You know, in clever ways. In some scenarios you don't necessarily need to hire annotators. But yeah, now if you have such an example, what I did was I just to simulate hiring an annotator and building a data set, I came up with the expressions myself and then computed the values for all the expressions and then asked GPT4 with the given prompt template. So the prompt template was the following neural mathematical expression Find the answer using botmaster right and provided the question and the and the formatted instructions. Right. So the model was evaluated on 106 samples that I generated. The evaluation result along with prompt usage and the cost were all locked to vandebut tables as you can see in the screen. And with VAN DB tables I was able to find out all the examples for which the model was failing. So in this case the example 19th the true value should be 35, but the model GPT4 is giving 25%. To my surprise, the model gave only 59% accuracy. So GPT4 model which was not able to do basic bodmas arithmetic. But like doing the evaluation on this, on those 106 examples costed just $1. So it's very cheap, which is a point I mentioned earlier as well. Also the evaluation cost is going to be really negligible compared to the data separation cost. Even if I created those 106 expressions and evaluated them. It took me maybe 30 minutes to an hour to build that while the evaluation took just few seconds. Great. Just want to take a quick detour to emphasize the importance of tracking with Pandabi. Because when I was preparing this slide deck, I was not able to recall which model was used for that particular evaluation job which gave that 59% accuracy in the previous slide. So if, if I switch to the tab, this is basically the workspace where I log everything. So there, there are like 134 experiments that I did and this was calculator I guess. So I can search that particular run, go to that run page and go to the overview page and in the configuration because I logged the config parameters I can see that yeah, GPT4 was used and again I didn't recall while I was building a slide. So I actually did this step to find. Also I demoed one DB tracer previously. So let's go to a generation here. So for this example 5 +8 +2 to power 3 it gave some answer. So let's see which model was used. So if I go to the LLM here I can see that SGPT4 that was used under the hood. So again I was not logging this information but this information was tracked automatically by using those two lines of code. Wait, let's come back to the presentation. Great. So at this point for the supervised setting that I'm talking about, we have both the data set and we have the metric so we can optimize the prompt parameter prompt parameters to find the best set of prompt parameters like which LLM model is going to be best, what's the temperature of the LLM model, the prompt element, etc. In traditional deep learning we used to do hyper parameter optimization with a tool like 1 DB sweeps. If you have tried that well we can do the same SUI for prompt parameter optimization just set up a YAML file throwing define your parameter space as in the prompt template is the temperature the model and then once you have that run the job for say 50 or 100 experiments and once you have that you are going to get parallel coordinate plot like this and let me actually run through, actually show this live here to all of you if you can see the screen. So this is so they were like 50 experiments that I did. Different prompt templates, different models. So the text davinci003gpt4turbo and if I wanna pick the system like this QA, I mean this calculator system to give the best accuracy I will have to use the GPT4 model with this prompt template 3 so I can look at the config here and but this is gonna cost me what some $2. So if I'm evaluating 106 samples is costing me $2. It's like negligible but still is costing more. The point is if you want to optimize for the cost and the accuracy, there's always going to be a trade off, right? So if you're okay for your system, maybe you're okay with an accuracy in the range of 60 plus you can literally get away with like a dollar, right? So again you're using GPT4 but you're only spending dollar with maybe not so heavy prompting technique and yeah you're okay with this accuracy. So if you run prompt parameter optimization suite like this, you can do this multi objective comparison and like find out the trade off and pick a, pick a model that or system that works best. So in this case if I want to pick this, yeah I will look at the configuration for need sweep Maintain experiment and just pick that and use that for my. And deploy that maybe if I want to deploy that. So that's optimization. Great. Now one. So that was everything about supervision. We talked about eyeballing and how it was important and how it is like how it is important and we talked about supervision. Now the fun part is how we gonna use LLMs to evaluate LLMs. Given is the fastest way to evaluate your pipeline sensitivity. Right? So just to make things a bit more complicated, let's take an example of a retrieval augmented generation or a RAC pipeline. So for, for those of you who is not familiar with this, it's basically a way to bring your own private data to an LLM. So you take the your document, chunk it, embed it, and create a vector store. When you are querying. You can then embed the query, retrieve the most relevant chunks from the vector store and pass the query as well as those retrieve chunks to LLM to synthesize the response. Right. So there are so many elements that's going on and probably in your use case it might be even more complicated than this. Right? You might have multiple retrievers and you might be routing the queries from multiple to multiple retrievers. You might have the concept of agents as well. So now if we imagine evaluating a RAG pipeline for a question answering bot over a document, we can maybe evaluate three important bits here. So number one is going to be how factually correct the response is. The two is how good the retrieve chunks are. So if the chunks that's been retrieved by the retriever is not good, we are not providing good context to the LLM. So the response is not going to be good. So we need to evaluate that as well. When you are actually taking the document and chunking it into smaller text, you also need to evaluate which chunk size is going to work well for you. But before that we need to think about the evaluation data. Again, you can hire annotators and build the evaluation data. Great. But again that's going to take time. So we can leverage large language model here to generate the data for us. In this particular example, I use GPT 3.5 Turbo to generate the question answer pairs. How did I do that? Well, if you're using blank chain, you can literally generate with three lines of code using this QA generation chain. But under the hood this chain is nothing. This chain is just calling LLM with this particular prompt. So in this prompt it has some instructions like you are a smart assistant to come up with meaningful, meaningful QA pairs and the pair should Be the point and the answer as detailed as possible. Right? So the prompt takes in the chunk of text that you provide from which you want to retrieve that question answer pair and yeah, the LLM will do that for you. Great. Now let's think about the metrics. So you have the question or the query, you pass it to the bot, you get the answer and you also have a ground truth answer which was generated using a, which was generated using an LLM itself. Right? So we can either use LLM as a metric itself. So you ask, ask an LLM, provide both the ground trip answer and the predicted answer and ask if it's correct or not. Or you can grade in the scale of 1 to 5 or 1 to 10, or you can throw in a standard metric like F1, score, blue, etc. So the standard benchmarks that we use, we can also throw it here because we have access to both the ground truth answer and the generative answer. I mean, now one can say how can you be sure that the ground truth or the LLM generated ground truth answer is accurate? I mean now that's where the iterative cycle comes in. You need to get in the weeds, maybe look through some examples, have some more interesting way of generating the answer questions appear. But this is me just giving one idea. So if I so why, why LLM can be used as a metric, so it has good understanding of semantics of the test. That's something we all can agree on, right? So we can leverage it to compare the ground truth with the, with the generative response. And it's like, and in many cases it's going to be better than the traditional metrics like F1 or blue. Right? And this is like a 1dB table that I wherein I logged both the question answer and the prediction and the evaluation. But if you look at one highlighted box here, you can see that for that given particular question about what is the broader four broad families of self supervised learning, the model response was kind of fuzzy while the ground truth answer was really solid and was coming from the bullet points that was in the PDF, the archive PDF file. And thus the GPT4 said that no, this is not the correct answer. So you can do this for an evaluation. So if you are using LangChain, we can again do this in freelance of code you have something called QA eval chain. So with the QA eval chain you can just pass in the pair, the prediction, everything and it will just give you a response correct or incorrect. You can also change the prompt template to give it like a grade 1 to 5 or 1 to 10. Awesome. Again, under the hood is just a simple prompt and you can modify the prompt as per your use case. Awesome. Let's talk about some standard metrics. Since we have the ground driven prediction, we can use something like a squad metric which computes the exact match and F1 score for those who are not familiar. Exact matches if the tokens of the model's prediction exactly matches that of the true answer. While F1 metric is basically precision and recall of the system. So precision in this case would be ratio of the tokens that's been shared between the ground truck and the prediction divided by the total number of tokens in the prediction. Likewise, recall is shared tokens by number of tokens on the ground truth. Again, fancy metric, you can use the metric as per your use case. And again, if you're using something like one DB tables and logging into it, you can actually see for all the data points the individual metric as well. And not just a min metric. By the way, I also for this particular use case I use hugging us evaluate because it has a lot of traditional metrics which you can literally load with just two or three lines of code. Break. Again, I will not go into the hyper parameter optimization job that I did for this, but the reason I'm throwing it here is to share a very important insight that I found from this experiment. So basically, if the evaluation data set was created with GPT 3.5 Turbo model, the GPT 3.5 Turbo model give better responses as per dise valuation metric. Right? So you can see that for GPT 3.5 the mean F1 score is 51, while for GPT 4, which is a superior model on public benchmarks is giving a score of 47, mean F1 score of 47. So and also I will come upon this point later on because this was something which was also proved in the Geval paper by the authors of the paper. But yeah, let's move in. Moving forward so far we were able to evaluate the responses, evaluate the responses using the LLM as a metric or using some standard metric. Now how about evaluating retrievers? So we can then like we can just save the chunks that were used to create the QA pairs. And while we are retrieving, we can basically do some sort of re ranking wherein you take the chunk, you rank it with the retrieve chunks and see what's the rank you're getting. In this case say if the three retrieve chunks are cat and catty cats and the source chunk or the truth chunk Was cats, the rank is one by third and you can create like do this reciprocal ranking and then get a mean rankings. And the objective would be to improve the mean reciprocal ranking. And again you can use a prompt template like this running. Yeah, please tell if this given piece of information is relevant. And yeah, you can provide some few short examples and then you can use an LLM to evaluate a retrieval as well. Great. So that was some ideas on how you can evaluate a retriever. How about evaluating the chunk size? So and this is an idea that was proposed by Jerry, who is founder of llamaindex in a tweet. So the idea is that you build separate retrievers for each chunk size. So you can have chunk size 128 and you can build a retriever for that. For chunk size 512, you build on a retriever and during query time you route all the, you route the same queries to all the retrievers parallel. So you don't take like the extra overhead time. And then you pull all the retrieve chunks from all the retrievers, pass it to a rev anchor and allow the revancher to rank all the chunks. So once you have all the chunks ranked, you can then compute the mean reciprocal ranking which I showed previously. And since every node that you are ranking is subset of that like is of a retriever, you can basically say that. Great. For retrieval one with a chunk size of 128, I am getting a better mean reciprocal ranking. So let's use the chunk size 128 and that way you might be able to evaluate your chunk size or if not evaluate, you can use a better chunk size. That's improving the performance of your model. Great. Yeah, I guess just have two or three more slides and then I will be done. So I talked about LLM as a metric. So LLM can be used as a metric for evaluating chance, evaluating retrievals responses, whatnot. But we need good metrics as well. So one such method is this semantic answering answer similarity score or sas. So if you look at the table here, this was evaluated on squad benchmarks. So a lot of metrics were used to evaluate the squad benchmark. And also humans were used to. Humans were also hired to evaluate the benchmark. So there's a two numbers, R and tau. So these values are basically Pearson and Kendall. They are basically correlations. And the correlation was done between the gold standard true answer that's in the benchmark with the answer that was generated by a system. And you can see that metrics like blue and F1 score or rouge. They are not even close to the correlation that a human annotator is getting on that benchmark. So I mean a SAS based model is able to get such a system, such like a better or a closer correlation compared to a human annotator. Thus we need to have metrics that are more close to human annotators. And in some cases you might be able to find metrics that even surpass human annotators. And this class is very simple. It's just a cross encoder model. You pass in two sentences with a separation token and you get a score from a classification head. So if the score is 0.9, the both sentences are very similar and this is 0.1. They are not that similar. And you can get an idea of how good the generation is based on that. Yeah. Finally, we'll touch upon this Gwal paper. So the idea here is that you specify some task introduction. So introduce the task to in a prompt or as a text. Then you also define the evaluation criteria. So how you want to evaluate, do you want to grade in a scale of 1 to 5? Specify that and you pass. Provide both things to an LLM and ask an LLM to come up with chain of thought, basically asking LLM how, if given this task and given this criteria, what are the steps that I should take to evaluate? So it's like auto chain of thoughts, right? And you take all of these things along with the input context and the target generation, give it to an LLM and then generate a score for you. And hold on, you have a better metric here again you can read the paper, it's really good. And they also have an open source repository which you can try. But I really want to stress upon few of the points that the authors of the paper came up with. The number one point is that they said that GPT4 is a very strong evaluator, so we can use it. But if you use some chain of thinking, chain of thoughts and some fancy output tricks, we can literally use this as a good evaluator. If you are asking an LLM to evaluate or grade in a scale of 1 to 5, make sure that or rather like sometimes the LLM will have a dominant generation. So in a scale of 1 to 5 it will dominantly generate 3. In a scale of 1 to 10 it will dominate to generate 7, which is not good. So maybe you can ask the LLM to generate text. So correct or incorrect like that. Also if you're asking LLMs to generate decimal value, so 2.3, 1.7 like this LLMs have the tendency to generate integers, which is again, it just kills the granularity of your evaluation. So you want decimal values, but so you have to be thoughtful of these things. And when you're evaluating, you just cannot blindly throw in an LLM. You need to do this analysis as well. And finally, the point that they concluded with was that GPT based evaluator has a bias towards GPT output. And this is exactly the same thing that I showed previously from my Hyper Parameter prompt parameter optimization job that I showed where the GPT 3.5 Turbo was performing better on the question generated by GPT 3.5. So again, you need to be thoughtful of what model you are using to build a pipeline and what model you are using to evaluate your pipeline. Right. Again, I mean, I think I am, yeah. I mean it's almost time and if anything was not clear, feel free to reach out to me. But these are the main takeaway points that you should consider. And finally, it was a lot that I kind of tried to cover. But I highly recommend trying out this evaluating and Debugging Generative AI course launched on DeepLearning AI in partnership with Plan DB Carrie was the amazing instructor who walked us through that course. And yeah, again, feel free to reach out to me on any platform that you feel like to ask more questions. Having said that, I am mostly done. Yeah, we can. Then I guess we can take questions.

Amazing. Thank you so much. Ayush. I'm also going to bring back up Morgan so we can answer some questions from the audience. I hope this was very helpful for a lot of people watching. We only have time to go through a few questions. We will drop the resources in the chat and the slides and we'll also drop a survey as well so that we can improve the event experience and all the education we want to bring to everyone. But to start off with, for our first question, why don't we start with how do you come up with good data sets to assess the LLM performance?

Yeah, I mean the answer, to be honest, depends. Again, if you're building an LLM based system, the good data set is something that you need to come up yourself or you have to be clever about that. If you're building an LLM or a large language model, there are really good benchmarks out there that you can try. Helm is a good one, if I recall correctly. There are a few more as well. Big Bench is a good one, so you can use those to assess your LLM performance. But for your own application you need to Be thoughtful about how you're building those data sets.

Absolutely perfect. Let's move on to do you know any domain specific LLMs such as legal Wondering what are the best practices to train these types of LMS? Build one from scratch or adapt existing LLMs.

You want to take that?

Sure, yeah. I haven't heard any domain specific legal ones recently. Not in this like current crop of yeah like multi billion parameter. There's been like other NLP models before like on the smaller scale that have been fine tuned on particular legal data sets. I'd say if you go to the hugging face hub, maybe just like search legal or law or lawyer you might find something in terms of like best practices. Definitely adapt an existing one. You know you want to rely on like the like incredible amount of like capability in like a llama 2 model. So you'd yeah just fine tune that. You know if there's a particular set of like law in your like country or just jurisdiction or type of law, you know, just try gather all that data, you know, parsley, clean it and then fine tune an existing. Yeah powerful foundation model.

Awesome. Perfect. Why don't we dive into what are the drawbacks to allowing the LLM to evaluate completions.

The drawbacks can be multifold. Again as I said that say if you are building an LLM system and using cloud model or a cohere model and you are trying a GPT4 to evaluate that might be. GPT4 will not do a good job at evaluating those systems. You might have to try out a bigger cloud model for that also many times for example I showed in a slide that as you scale LLMs unlock abilities. But that doesn't mean LLMs will unlock abilities to perform really well on legal use cases or a medical use case. Maybe you need to fine tune an LLM for your use case and then try to use that as an evaluator. So there are drawbacks. You need to be thought about it. To be honest. Morgan, you want to. Do you want to add anything to that? No.

Yeah, it's still, it's still early days. The jury's cutting out on how effective they can be. They can, with a bit of work they can be directionally correct. So it mightn't like get every you know, one match a human annotation but you can maybe broadly know if you're you know your generations are going in the right direction.

Yeah, personally I tried cohere and saw GPT for not evaluating it well and vice versa. So yeah, we need to be thoughtful about it.

Absolutely perfect. Well, I think this ends our time for our workshop and event today. Thank you everyone for attending and thank you the Weights and Biases team, not only for working with us on our course that you're able to take, but also for this amazing workshop today. I'm sure we'll do many more in the future. We have dropped a link in the for the survey in the chat, so if you could please fill that out. Definitely helps us decide what will our next topic be for the next event. And thank you Ayush, Morgan and Carrie for coming out today. We'll see you guys next time.

Thanks everyone.

Thanks everyone.

Thank you.