Mitigating LLM Hallucinations with a Metrics-First Evaluation Framework

Sam.

Hey everyone, my name is Diana Chan Morgan and I run all things community here at DeepLearning AI. Today we are so lucky to have some special guests from Galileo to walk us through LLM Hallucinations with the Metrics first evaluation framework for everyone that is watching. The session will be recorded and available for replay after the fact, but if you have any questions please check the link in the chat in our slido which you'll be able to vote on questions this feature speakers will answer at the end. We're also very fortunate to have some of their developer evangelists, Setu and Jonathan in the chat as well. So if you have any additional questions feel free to ask any questions there as well. So for today's event and workshop what you can definitely expect is a deep dive into research backed metrics to evaluate the quality of the inputs, whether that's data quality, RAG context quality and outputs, whether that's hallucinations within the context of LLM powered applications. We'll also be talking about an evaluation and experimentation framework while prompt engineering with RAG as well as fine tuning with your own data. And of course we'll have a really well led demo by our speakers today that will definitely help you be able to implement this on your own. This event is also inspired by DeepLearning AI's generative AI short courses created in collaboration with AI companies across the globe. Our courses will help you learn new skills, tools and concepts and efficiently within one hour. And now to introduce some of our speakers. Our first speaker is Vikram Chatterjee. Vikram is the co founder and CEO at Galileo, an evaluation, experimentation and observability platform for large language models. Prior to Galileo, Vikram led product management at Google AI where his team leveraged language models to build models for the Fortune 2000 across retail, financial services, healthcare and contact centers. He also led product for Google Pay in India, taking it from 0 to 100 million monthly users, the most downloaded fintech app globally. He's also one of the early members of the Android OS team and believes Generative AI is ushering in a similar technological wave to mobile. Hey Vikram.

Hi Diana. Thank you so much for having us.

Of course. To introduce our second speaker, this is Atendreo Sanyal Atendreo is the co founder and CTO at Galileo and prior to Galileo, Atendrea was an engineering leader at Uber AI responsible for various machine learning initiatives at the company. He was one of the architects of the world's first feature store, Michelangelo and early engineers on Siri at Apple, building their foundational technology and infrastructure that democratized machine learning at Apple. We are so lucky to have them all here today. And without further ado, let's definitely get started. So, Vikram, do you want to take it away?

Sounds good. Diana. Thank you so much for that introduction and also welcome everybody. Really excited to have everyone here for the workshop. It's going to be hopefully an exciting one, but also something that you can take a lot of really interesting nuggets away from. The topic of today's workshop is something which is super close to our hearts and something which is very top of mind. I feel like for the industry as well as we all grappled with large language models, it's about how do you mitigate LLM hallucinations, but how do you do that with the metrics first evaluation framework so to kick things off, I'm going to start with something with an agenda about what we'll be covering today. The first thing we'll talk about is the need for LLM experimentation and evaluation frameworks. I feel like we're all hitting these brick walls, but and quickly realizing that there's a need for this. We'll go over exactly why and what that means. Second, we'll talk about different kinds of evaluation methods and when to use them. And then we'll further, number three, dive into emerging hallucination detection metrics that are coming out in the ecosystem, but also how we can push the envelope further and build them up specifically for the LLM era. And last, we'll go through a demo to walk through how you can bring this all together and have this experimentation be done in the right way, but also with the right kind of metrics in place and walk you through a simple app that we've built out and how you could mitigate hallucinations, whether you're using RAG or whether you're fine tuning with your own data. To start with, I'm going to start with 100,000 foot view. When you think about LLMs, you have a lot of inputs that go into the LLMs and then you have the outputs. This is classic machine learning. You also have a feedback loop from the outputs which go back into the inputs. That's basically it. It should be simple, but it's not. Because as soon as you start dabbling with large language models, you immediately realize that there's an explosion on both sides, the inputs and the outputs also on the input side. For instance, as soon as you start working with large language models, you realize that, wait, I have a bunch of different large language model parameters that I need to tweak whether it's the temperature or any such thing, there's a whole host of them that you need to tweak. There's the prompt template that we all know. If you tweak it just the slightest bit leads to all sorts of different results. On the other side, constant experimentation happening there. There's chains that you can bring together, create agents and have multi agents. It becomes super complex super quickly. You have rag in the mix with different rag parameters, all the context data that you need to add into the vector store, whether it's good, is it bad, then start thinking about fine tuning. You start thinking about the quality of the data that you're putting in, do you have sufficient data in the first place? And so much more. So it just blows up. The inputs themselves are mind boggling when it comes to the outputs. It depends on your use case. Now you might have a text summarization use case or text generation or chat, et cetera, or question and answering or code generation and so many more. And for each one of these different kinds of outputs that you're expecting from large language model, right, the way you evaluate them has to be different. You can't be evaluating turn by turn chat the same way you're evaluating text generation. It just doesn't work that way. So the complexities just exponentially increase. And so in order to solve for this, what we really need to think a lot more about is on the input side, right? How do you iteratively experiment? And today when you look at how people experimenting amongst in our community, it's highly, highly manual. It's mostly still being done in notebooks or we've seen people use Excel sheets and notion documents, all sorts of stuff. And when you think about it, that's not how data science should work. You should not be experimenting with all of these different ingredients which are essentially your IP in notion documents or Google Docs. On the output side, the output evaluation has now reduced to essentially eyeballing with erstwhile natural language processing. We at least had our friendly neighborhood F1 score which could help us be the help, be the North Star for are you moving the right direction or not? It's debatable how accurate and good it is, but at least it was an evaluation metric that we all gravitated around with. Lms, you have that. So we're seeing companies at the highest level of sophistication still just eyeballing the outputs with large number of humans in the loop, which obviously increases the amount of bias and is far from accurate. Right. So when we think about this and we thought about this at Galileo. We built out what we call the Galileo LLM Studio, which is essentially an experimentation, evaluation and real time observability bench for any data science team or developer that's working with large language models. The way we think of it as from a framework perspective is there are three parts to this, right, and three modules as a part of the product as well. The first one is called prompt, which is all about accelerating prompt engineering. So when it comes to the inputs from your vector store or your prompt evaluate, your prompt template or your LLM parameters, how do you experiment across all of that to make sure you're working towards the right set of inputs? The second one becomes fine tune, where you're fine tuning your large language model. You have to make sure that you're doing that with the highest quality data and you're constantly moving in the right direction. It's super important to experiment and be iterative in this process. And the third one is monitoring because you can create all the guardrails you want before production. We all know when you put a model in production, given its probabilistic nature, things can go wrong. You have to immediately know whether you need to tweak your prompt or you need to add additional data for fine tuning. And all of this together is essentially how we think about experimentation and observability. And so we strongly urge anybody, whether you're using Galileo or not, to be thinking about experimentation. Observability constantly across all three of these big pieces, apart from this experimentation is meaningless without metrics, without good metrics. That's how all fields of science work. And so for us, since the very beginning of day one of Galileo two and a half years ago, we were all about language models and all about building metrics for evaluating large language models. We built out what we call the Gardrail's metric store, which has a bunch of research backed metrics which we have designed a lot of them within Galileo, but a lot of them have been built on top of open source metrics as well so that you can make sure that you can reduce hallucinations and build trustworthy LLM applications. But these guardrail metric store metrics are super important from an output evaluation perspective. So we can, let's dive deeper into that for a second. And if you double click on output evaluation metrics and in general, as you think about building large language models based applications, when you think about these evaluation metrics, I'd be urge you to think of it in three different ways. There's the output quality metrics. There's customized metrics for your specific product and business use case. And then there's the output hallucination metrics. So what do I mean by that? First of all, for the quality metrics, there's a whole host of them. For instance, tone of weiss is your output of your chat product getting? Is the tone angry? Is it happy? Is it sad? What do you want it to be? Is there any PII presence? You know, for a lot of regulated industries, that's super important to be completely on top of. Is that toxicity? Is there any kind of bias that's being spewed? Is there hatefulness or sexism or many such other kinds of quality attributes of the output that you need to be very on top of? The second kind of metric is we call it the custom metric. Like for instance, AI policy adherence. There's the EU AI act coming up and some other data act in the US itself. You have to be adherent for that if you want to launch credible models. Compliance adherence or business metrics adherence as well, and many other such customized metrics that you need to think about. The last one is called the. We call it the output hallucination metrics. Right? This is basically how do you know whether the LLMs output is correct or not? And this is where there's a huge bottleneck today. How can you consistently and accurately measure the correctness of the LLM output? When we think of this, we divide that into two large buckets. One is for any team that's working with REG workflows or non RAG workflows and non ROG includes that includes fine tuning or you can combine fine tuning with rog, but essentially think of it as, do you have context involved or do you not have context involved? And go deeper into this a little bit more because this warrants its own discussion. So what we'll do next is go deeper on the LLM output hallucination metrics and the different categories and some proposals around what you could use for detecting hallucinations with your LLMs. Do you want to take it from here?

Yeah. Thank you. So, yeah, we'll dive a little bit deeper into the third category of output metrics which Vikram was talking about, essentially hallucinations and the existing methods that help quantify some of them. Before we dive a bit deeper, I just want to set context on what we mean when we say hallucinations. Essentially it refers to instances where the LLM output might sound coherent or plausible, but has like logical fallacies in them. Or there's Factual validity issues as well as if there's context, the LLM does not adhere to the context or the instructions which are provided in the input. And then there's many reasons why hallucinations occur, which is a bit beyond the scope of this talk. There's limitations in training, data model quality, et cetera, and there's a few methods which have been explored in the recent literature to quantify modern hallucinations and. But they kind of broadly fall into three techniques, three categories of techniques. The first one is essentially based on measuring the amount of overlap between the output of the LLM and a reference ground truth. Now of course, this is only possible if there is a ground truth or a reference data point in existence. And a lot of the use cases around LLMs typically work without a reference these days. In those cases, this sort of n gram matching category does not apply. The second technique is based on asking an LLM if it thinks whether an input completion pair has hallucinated or not. Now this is the basis for a lot of the more widely adopted modern techniques to detect hallucinations. And of course it has certain merits, but there's many limitations around that which I'll get into. And the third category is essentially most LLMs are basically token spewing machines and each token has a probability distribution amongst a vocabulary. So when you gauge into the statistical properties of these probability distributions, you can get some intuitive sense around whether the model got confused and whether it hallucinated. So those are the high level intuitions and I'll get into each of them one by one. So the first one, a popular suite of metrics which leverage the first technique, include metrics like blue and rouge scores, which are essentially measuring n gram overlap. So they look at phrases and they essentially try to see whether how much overlap is there between the output of the LLM and a reference ground truth. And that gets converted into a 0 to 1 score. There's also Meteor, which is kind of a different version of the same idea, which takes into account things like word order and synonyms. So it bolsters the blue and rouge scores a little bit. But they're all based on this underlying principle that measurements of similarity to existing ground truth can give you some semblance of hallucinations. Now of course from the definition there's decent amount of applicability, especially in historic use cases. These kind of metrics were typically used in translation quality, not just in language technologies, but also in speech based systems as well. But you can see that there's a lot of limitations in this kind of an approach. For one, it requires the presence of a ground truth and reference data, which is typically not common as language tech has been adopted more and more. These kind of metrics, they fail to adapt to stylistic changes where words might be different but the overall meaning might be the same. So it can lead to a lot of false positives and false negatives. And essentially to summarize, comparing words and tokens to measure hallucinations is overall not a great idea for most part. The second category of technique is it revolves around the idea of asking a GPT model or some state of the art LLM if it thinks whether the pair of the input and the output hallucinated. And there's many ways to go about this. Of course you can directly ask an LLM, hey, do you think this is hallucinating? Or you can do multiple reruns and get multiple generations and check consistency across them. This kind of category broadly falls under the self check method, which was recently published a few months ago. And it certainly has its merits, the foremost being it's not restricted by a ground truth and it's kind of designed to capture some of these higher order policies, which means that the applicability is fairly broad. And this kind of a technique can detect more higher order hallucinations. But again, it is very restricted. Vikram, do you want to move to the next? Yeah, yeah. So it's fairly restricted. Number one being that it's a black box technique, right? You're asking a magic ball if something's hallucinated, it will fail to capture reasons behind false positives and false negatives. It can be prohibitively expensive if you use more state of the art models like GPT4 to give you whether something's hallucinated or not, and more importantly, it lacks the explainability which is critical for users if you want to take corrective action. As you'll see, there's many different kinds of hallucinations. Open domain, closed domain. And taking the right action relies heavily on being able to explain what the hallucination is about. The third and final category of existing metrics to measure hallucinations is based on doing math and probability statistics on top of the probabilities. You essentially look at the token level confidence of these models as it's constructing the output. And the intuition here is that this kind of measure gives you a sense of the model's uncertainty in its own response and may suggest some semblance of hallucinations. But from our various in depth experiments over the last year, it has shown that while techniques like These are great secondary signals of model confusion. They are essentially a bit too granular. Vikram, if you want to move to the a bit too granular in its ability to capture higher order signals of hallucination. Another restriction here is that a lot of LLM APIs typically don't return you log probes. In those cases, techniques like these become a little restrictive. We've talked about the three broad categories of metrics which are today used for health nations and, and there's many experiments which we've done internally to identify issues in these existing metrics. And we essentially figured that there is a need for a new technique which not only takes some of the better principles of the techniques which I spoke about, but also leverages these distinctive powers of these new LLMs, like their ability to respond to chain of thought prompting and use that to create a new category of metrics which number one is more accurate in its results and number two, it's applicable to real world scenarios, it scales to different kinds of tasks and finally it is cheap to compute and it should work at low latency because we are all building practical metric systems here. With that we created a new technique to identify hallucinations. And this technique is called chain poll. And chainpoll essentially is a high efficacy method to detect hallucinations. Move to the next one. Here in chainpoll we break down detecting hallucinations into two phases. The first phase is called chaining. In chaining we essentially use a very detailed chain of thought prompt to make a secondary LLM request and gauge if it can reasonably sort of break down the reasoning behind the completion that it just made. And then we collect the chain of thought reasoning. Think of it as we structure it down into a specific schema and then we do what we call is polling. And polling is essentially this ensembling technique where we try to gather evidence of policies in the. In the chain of thought through voting. And voting is a common technique used in machine learning. Whether it's like classic models like random forests, ensembling is a general way to get really good results and think of it as though it's, you know, there's domain experts in the room and you're kind of getting structured schematic responses from each of them and then you're trying to make a claim of whether the model hallucinated or not. So here's a high level flow of chainpole. The user makes a query to the LLM and the LLM responds with a completion. Then what we do is we check if the LLM also provided a Probability distribution of response tokens. This goes back to the third category where these log probes act as a good secondary signal to hallucination. If the output has no log probes, we employ a special generation prompt to match the original log probes. In theory. Then we send the newly generated log probes along with the original prompt and response to a chain of thought module. We call it the detailed chain of thought module. And that's basically our core ensembling algorithm. Behind the scenes we are using batch inferencing to be able to essentially reduce the number of model calls we make to just one. The key learning behind this has been that the depth of the chain of thought prompt makes a significant difference in the quality of of the hallucination detection. And the more detailed and more organized the prompt is, the smaller the model that you can use to generate accurate scores. In fact, you'll see in a couple of slides ahead that we used models like Da Vinci, which are at this point fairly dated to be able to achieve over 23% improvements compared to the next best hallucination technique around cell checks. And then we take the output of the ensembling module and aggregate create like this aggregate score along with an explanation using the chain of thought process, pass it to this cleaning and optimization module and then finally give a user a normalized score of hallucination along with a clear step by step explanation of what may have gone wrong. So just to highlight a super high level result on all our experiments on all data sets, Chainpole here significantly performed better than the next best which is a self check BERT technique which was launched this year in July in the uk. And you can see how much better it is compared to some of the more metrics, common metrics which are used today to detect modern hallucinations. There's many advantages to chainpole. Number one is that it is designed to be as accurate as possible. The whole motivation of the ensembling and the chaining is to increase the accuracy across a breadth of possible real world tasks. Now the whole motivation behind this is that our metrics should apply to real world scenarios and not just stick to academic data sets or textbookish examples. The second is that explainability is key. Like I mentioned, hallucinations in modern LLMs are very very nuanced. Sometimes it spews opinions. So the devil is in the detail of what's right or wrong. And these LLMs essentially act very similar to humans sometimes. So it's very important to give you feedback which is understandable by a human being to be able to take action and next steps. There's a a need for low latency as well. There's a significant amount of engineering our team has done to make sure that the algorithm itself works in parallel and the different computations are asynchronous. We want to minimize API calls as well and do batch inferencing to finally reduce cost, because cost is a very big factor here. So this is chainpoll. In summary, this is the method, but the outcome of the chainpool method are two metrics which we apply to the two broad workflows we observe in LLMs. The first metric is correctness, which applies to these open ended open domain use cases where the user essentially passes a query with or without context, and the correctness metric looks at the consistency in the reasoning behind the LLMs completion and how correct the output model can be. The other is the fact that the chain pool is also correctness in particular is kind of agnostic to whether the context was passed in or not. The second metric is context adherence, which essentially measures how much grounded the LLM output was in the context which was provided. It is very helpful in rag workflows where you essentially have context in the form of documents which are fetched from the Vector store. I'll show you a couple of simple examples of correctness and context adherence.