A Deep Dive on LLM Evaluation

So I'll be giving sort of a very brief, about 30 minutes. There's much more than can be covered in that time, but as much as we can, or sort of an opinionated summary. I guess an alternate title for this talk could have been basically everything you didn't realize that you needed to ask about LME vals, which is sort of a taste of what you should be looking out for. Yeah, here we go. So just, yeah, a little bit about me. I'm a research scientist at EleutherAI. We're a nonprofit research lab. We, you might know us from sort of a number of language models that Eleuther released open source over the years, including GPT-J and GPT-NeoX 20B. There were some of the best of the time a few years ago. Things have certainly changed, and there are way more options now. But we also do other research on things like interpretability of models, datasets, distributed training, evaluation, which is this talk, and we also build and maintain a couple different repositories for sort of tooling for the open source AI ecosystem, especially tailored toward researchers, but just useful for practitioners in general. Yeah, so I, in particular, I'm a maintainer on the LM Evaluation Harness. Here's a link. It's a library that was originally started by some of the founders of EleutherAI in 2021, sort of originally just with the well-scoped goal of sort of one-to-one reproducing the evaluations described in GPT-3's few-shot prompting in the paper on the GPT-Neo model, just to sort of track progress and reproduce these results. And since it's grown a lot with the community, and there's, yeah, a lot more people working on LLMs and evaluation, and we've been lucky to have it widely used by a lot of people. You might know it from the OpenLLM leaderboard. It's used as the backend there. Yeah, so in this talk, like I said, there's way more than can be covered in just like 30 minutes, but I'll try and give like a couple deep dives into specific topics. I'll briefly give some background on why LM evaluation is so hard and why we need to think so much about solving many problems here. I'll give sort of like a very, very brief crash course in how evals are commonly done under the hood, some of these gritty implementation details that often aren't talked about, and also give some takeaways as a result of this and some sort of minimal best practices. And so as a disclaimer for context here, I am a researcher. I don't put LLMs into production in my work, so my recommendations are going to be somewhat based on my experience in research, but at the end I'll sort of touch on which of these points are most applicable still if you're doing work in putting LLMs into production and which sort of don't directly transfer. So yeah, so there are many reasons that evaluation is hard and also meaningful, but two that I'll sort of try to focus on here is first that scoring is very difficult and later reproducibility is tough. So what do I mean that scoring is difficult for LLM evaluation? So there's a very difficult problem, which is basically how do we evaluate the responses of a language model in natural language? Like, how the heck do we do this? So like in this, in the figure, sort of there's a couple of different clouds, one both showing different like potential responses from a language model to sort of a very simple factual question, which is who is the current US president? And so on the left, the model answers the correct answer, just saying Joe Biden. On the right, the model says, oh, well, it's definitely not Joe Biden and doesn't really go on to elaborate. And so one of these is correct, the one on the left, and the other isn't. But, and so, like, you or I, if we just look at this question and we have the necessary context, we can sort of eyeball it and say, well, we know that one of these is correct, the other isn't. It's fairly obvious to me. But how would we actually— when we want to do the evaluation, we want to sort of be able to reliably get a measure of our model's performance and do this without much effort and in a way that we can sort of get the same result repeatedly if we want to rerun this evaluation many, many times. And so we'll want something more reliable than just sort of a human eyeballing every data point. And so we've got a problem, which is, well, so one solution we might think of is, okay, well, if the model's answer to this question includes the term Joe Biden, then we'll mark it as correct and sort of treat that as an approximation and go on with our day. But the problem is, in this example on the right, the phrase Joe Biden appears in the model's answer, and yet it's actually saying, oh, it's definitely not Joe Biden. So, like, the meaning of these two responses is entirely different. But if we use this sort of naive approach of just checking for the right term in the, in the output, then we would end up sort of scoring both of these as correct, even though one of them is not. And so this, this problem as sort of a motivating, a motivating challenge of sort of how do we check a language model's work is going to shape a lot of the approaches to evaluation. Because if we could sort of, no matter what, give the correct answer on whether a model's free-form response were correct, then we'd sort of have solved the hallucination problem entirely and have the perfect language model already able to evaluate these answers. And so, yeah, so people sort of take many different approaches in different situations to resolve this issue of actually being able to score outputs for correctness. But there's no sort of perfect solution. Everything is either too costly or maybe unreliable. or, has different sort of failure modes. And yeah, so, so with this in mind as sort of like the, the thing, the constraint that we have to work under, there are a couple different ways that we can try to evaluate a model, um, but in particular how we can sort of probe these models to inspect their capabilities or characteristics. Uh, and so there are 3 different ways that people can often interact with these models or sort of try to measure things from them that I'll talk through. Um, the first is log likelihoods. So, yeah, so just as sort of a refresher on language models and the necessary background here, when we feed a piece of input into our language model, what we get out is logits over a vocabulary. So for each of our language models, like for example, for GPT-2, we've got sort of a space of 50,000 different possible tokens that it knows. And when we run some input, x0 through xn of words or tokens through GPT-2, what we get out is this, uh, sort of for each different possible, uh, next word, uh, GPT-2 will give us like a numerical score, uh, representing like a logit on this, uh, token. And if we apply softmax, then we can turn this into a probability distribution, uh, which will basically tell us that GPT-2 thinks that there's a 50% chance that the next word is the, or, uh, like a 25% chance that the next word is dog, or 10% of cat. Et cetera. And so, so sort of this is going to be the basic output from our language model, this probability distribution over its sort of guess for what the next word or next token will be. And when we want to most often use language models, we'll sample text from them. And the way that we generate text with this is that we just, we feed in the input, we get out sort of its probability distribution over what its guess for the next word is. We pick, we sample from this distribution and pick a possible next word, and then we feed in that next word to the model and we get out sort of what the guess for the next, next word is, and so on until we're satisfied and done. And so in this way, sort of a language model maps an input to a probability distribution. But a key detail in how we train most autoregressive large language models is that when, when we train GPT-2, we'll feed in many, many paragraphs all at once as sort of the same sample. And at each next word, we can have the model guess sort of what it thinks the next word would be. So at the sort of first sentence of this paragraph, we can guess what it thinks is going to end that first sentence. At sort of the very end of the paragraph, we can guess what the very end word is going to be and do this all at once. So the model is going to give us many, many probability distributions, both not just predicting this x sub n plus 1, but also x sub n and x sub n minus 1 and also x sub 1 and so on. So we basically have information about what the model's guesses would have been, even though we also know the correct next token throughout this input that we feed in. So what we'll get out of a language model is these logits, but it's not just sort of a vector of shape vocab size, but it's sort of for each sequence position along the sequence length that we feed in, we get a logit for each elements in the vocab. And yeah, feel free to pop in with questions if there are questions, but otherwise I'll keep going.

So there is firstly, just quickly, this is, this is great. And there is, there's not a question, it's a comment, just a massive, and that a lot of people have upvoted, just a massive thank you to Hailey for maintaining LM eval and for all the amazing work coming out of EleutherAI. And we did just have a question. Is it better to measure or score evals per sample generated or over a corpus? Does this vary between modalities and use cases of capability?

Okay. I think I could try and answer that one later on. I guess I see this.

And there's one around models as well. That may be.

Oh, yeah. Yeah. Okay. Yeah. So, so there's a couple of details I think I see in the questions here. So I guess one thing is that By simultaneously, I do literally mean simultaneously, sort of. So when, when GPT-2 processes an input, many of the operations, like these sort of MLP layers, operate on each token individually. So you can do this sort of all at once. And then in the attention layers, you have something called a causal mask, which is basically like a triangle of sort of visibility across the inputs. So for each, like, for token 10 in the input, it can see all of the previous tokens. For token 20, it can see token 19, token 18, all the way back to the beginning. But for token 2, it can only see the previous one token. And so using this, basically we feed in the entire sequence all at once, and it sort of only can see at each point in the sequence, it can only see the previous parts, but it can sort of compute everything in parallel. This is at both inference and training time. Yeah, and oh, and then, yeah, one other question is, so closed source models do produce a logits matrix, but they're often not fully available, which is a caveat I'll mention later in the talk, just due to how the APIs are implemented and what information is made public. So, so, so now we've, yeah, we've talked about logits and specifically that you can turn these into probabilities. But, like, why is this, why is this useful other than just sampling text? So one way that it's useful and one sort of measurement that you can take from a model is if you have some input string x and some output string y, you might want to know sort of how probable it is for your model to output, say, Y, where Y is the correct answer to some question. And so, like, so, like, for a simple sentence, the cow jumped over the moon, if we say feed in the cow jumped over into our language model, maybe we wonder how likely is it that our model gives the correct next two words versus some other words. And so this can be used basically for, yeah, we'll get into a number of things that can be used for, but like you could use this to sort of measure how likely the, like, ground truth target is. And so the thing that I mentioned earlier is that you can get these logits for every sequence position in parallel. And so what this means is that for any length of output string y, we're going to be able to not just sort of generate these words one by one and check if they match up, but we can do this in just sort of one pass through the language model, feed in just one input, and figure out the whole probability of a string y. So this is a bit of a dense slide, but basically, if we want to compute the log probability of any string y, assuming that we're sort of conditioning the model with an input x, we can do this just with one pass through the model by first taking the concatenation of our tokenized x and our tokenized y. So just like x followed by y, and then pass it through the model and we get these logits. And these logits are available to us not just at every position, within this sequence X, but also within sort of the sequence Y coming after X. And so if we just sum over sort of the proper indices of the logits, what we can do is we can go check basically, okay, assuming our model has seen all of the tokens in X, how likely is it that for the specific like token ID that we know the first token of Y is going to be? How much probability does the model assign to that token? And so we can check basically how much probability the model assigns to token 0 of y, token 1 of y, and so on, and just sort of combine those probabilities. So if we're using log probabilities, just sum the log probabilities. If you're not using log probabilities, then you multiply, but this gets you sort of like very, very small numbers for a long number of tokens in y. And so what this basically means is that it's very easy to check sort of the probability of some output string conditioned on an input string from a model. It's just as simple as one pass through the model. And you can do other things, like you can also check if y is sort of the most probable next n tokens for your model to produce, still in one call, just by checking if each of these, like, true tokens in y are the most probable for your model to output. As opposed to just how probable they are, and so on. So this is sort of like a primitive algorithm we can use to get the probability of some output sequence. But then why is this useful? It's useful in a common use case for evaluation that people use very frequently. This is used in the majority of the OpenLLM leaderboard tasks, for example, and in MMLU, a popular dataset, to evaluate your model on a multiple choice question. If you have some sort of set of closed, a closed set of answer strings, y_i, which in this case, in the case of MMLU, where it's sort of a 4-choice multiple choice question standardized test, and the answer choices are A, B, C, or D, what we can do is for each of A, B, C, and D, we can use the previous algorithm we discussed to calculate the probability of producing A conditioned on the input question X, the probability of producing B, of producing C, and producing D, and so on. And we get basically comparative log probabilities of each of these potential choices. And then we can see, like in this example, A is the most likely. So what we're going to say is we're going to pretend that A is the answer produced by our model. And so in this case, for this specific sample question, the model's answer is incorrect. But maybe if it had put more weight on the answer D and had a higher chance of outputting D when we prompted with the question, then it would get the answer correct. But so basically, multiple choice is a very common way to do LLM evaluations. The first, the first reason just being because it's way, way cheaper than doing generation for with your language model. If you're trying to generate many, many tokens, for, say, for like a long chain of thought to answer each question. This is like a lot of calls to your language model and many steps that you can't really parallelize versus just sort of passing 4 different inputs through your model. And so in practice, like multiple choice question answering is a cheap way to do evaluation. Another huge benefit of doing multiple choice is that because we've not only said that there's only these 4 possible answer choices for our model to choose from, but also we're only comparing the probabilities of these 4 choices, there's no way for our model to give an invalid response or say abstain from answering and pick a 5th incorrect choice. It'll always pick one, and its guess might be wrong, but it is going to guess. And in my opinion, I think that multiple choice question answering implemented this way is pretty nice for base language models, especially small ones that you're training, say, from scratch. Because it's sort of a nice way to not have to deal with these, like, finicky parsing failures and still get sort of like a nice measure of your model capability, even when it might not be able to coherently generate long-form text. But by the same token, vice versa, if your small model can generate multiple choice question answering well for just sort of like a single token ranking, like we described in the previous slide, but can't really produce long-form chains of thought, then this means that your evaluation isn't really matching up well with a real-world use case of, say, like, using the model as a chatbot. And so, in this sense, like, it's definitely a step away from sort of downstream usage. It's also a disadvantage for some evaluation types that chain-of-thought can't be used, especially since models are commonly trained with chain-of-thought. And it's also somewhat misleading as to sort of real-world scenarios in that you don't necessarily want your model to be just solving standardized tests all day. You want to sort of have it handle open-ended questions where it's not given 4 choices to choose from. It's actually supposed to generate and come up with the choice itself. I guess, yeah, are there any questions?

There are a few questions, comments, but we can leave them to the end as well. I don't think mission critical currently.

Cool. Yeah, that sounds good. Great. Um, yeah, so, so basically, like, multiple choice using these log likelihoods is a pretty common way to evaluate language models, but it definitely has its downsides, especially when you're sort of worried more about generation performance or sort of chat usage. Um, yeah, and so the second common way to evaluate language models and sort of take a measurement that you could, uh, sort of assess a model's capability or behavior with is perplexity. So perplexity, here's the formula, but sort of to describe it in intuitive detail, we're trying to measure how well a model fits a given data distribution. And the way that we do this is we have some dataset, which is a collection of documents that are just sequences of words. And the thing that we're going to measure is pretty similar to the sort of loss we use to train models, but Here we take the log probability that the model assigns to sort of the true next token for every possible next token that exists in the dataset. So for every token in this dataset, we check basically how likely the model is to output it correctly, and we average this. We just— this is just average over all of the documents in our dataset. And over all of the tokens in each document. So basically per token, how well does the model fit that token and how well does it predict this dataset or how likely is it to produce this data? And so this can be done basically very, very trivially because it's this self-supervision where we know the correct label just because we've got this input document and we know what the next word is going to be for any sort of prefix in it. So we can take any dataset like say Wikipedia or sort of like some Wikipedia page of our choice and just convert it into something we can measure perplexity on just by checking sort of how well the model fits these sort of true next tokens in the dataset. And so perplexity is a useful tool, especially since it's just basically like using a different validation set during training, and you can use it for sort of any data distribution to see how close you're getting. But it's also not super important, especially in sort of downstream use cases for language models, because for an instruction-tuned model or a chatbot, like, sort of just evaluating how well it fits Wikipedia might be misleading because actually the model is sort of editorializing, outputting like a text, maybe perspectives, etc., that wouldn't match with Wikipedia style, or like the prompt format might not match, for example. However, it can still be a useful diagnostic tool, especially if you did have sort of a dataset or data distribution that you want your model to be fitting better for downstream use. Yeah, so basically perplexity is a useful tool to have in the toolbox. It won't be used too, too frequently, except for sort of like training models from scratch. And so perplexity, it seems like a pretty simple approach, and it is, but there's definitely sort of pitfalls and footguns that can occur for both perplexity and the log likelihood approach that we discussed before. One, one complication is that both these log likelihood and perplexity approaches, because they're taking sort of either the sum over a number of tokens or averaging over the number of tokens in a dataset, it matters what tokenizer you use for your models. So if two different models have a different tokenizer, the numbers that you're producing might not be directly comparable. So a perplexity of a certain value might be easier for a model with a larger tokenizer to achieve because there are simply fewer tokens to predict over. And so there, there are ways to sort of remedy this that, that can be implemented to sort of use the tokenizer as part of the system that you're evaluating and then have a metric that's normalized with respect to the tokenizer. But yeah, so this is like a lot of text, but the important part is basically that there are ways to control for this, but they're all sort of like small implementation details that change what you're measuring and how you're calculating it. And then, of course, the final way that one can evaluate a language model is by generating text from it. This is basically crucially important if we're going to use the model to generate text, such as like a chatbot like ChatGPT. It's what we care the most about. And chain of thought, of course, is something realistic and important for models to use, especially in sort of multi-step problems. But there are downsides to doing this sort of generation-based evaluation, which is that Again, we don't know how to, like, always correctly score free-form natural language responses. So in the case of multiple choice evaluation, we sidestep this by basically saying, okay, there are only 4 strings our model is ever allowed to output in this, for this document. And so that way, like, if there's a string that's not in those 4, we just disregard it, and we only ask the model to predict one of those 4 strings, and we know one of them is correct and the other 3 are not by construction. But for text generation, the model could output any sort of string, and we want to be able to say whether it's correct or not, or as a matter of degree, how correct it is. Is it, is it half correct? Is it 3/4 correct? Not at all correct? So one way that you can do this very simply is just sort of do a very, very rough heuristic and just say, okay, I'm going to look for the part in my model's generation where it says the answer is X. For some phrase X, and then we'll just basically grab what that X is and then check if it matches sort of the gold standard phrase that we had. So like, as an example, like going back to the previous sort of comparison of like answering who the president is and the answer being Joe Biden, we could basically say, like, tell me what— tell me who the president is, answer with the answer is X, and then hope that our model follows that format and that it tells us Either the answer is Joe Biden, or it tells us something else, in which case we know it's incorrect. However, this is not great because this just means that we'll be penalizing models that don't comply with our expected format. So if we implement a format of the answer is X, models that are trained to produce this format always when it asks the question will do better on our evaluation than models that don't. And so there's sort of these confounding variables that we have to deal with. And another way people do this is by using an LLM to basically check if an answer is correct, but these are definitely fallible. And of course, there are also other, in this case, this is another sort of pain point caused by tokenization. There are other reasons that sort of generation, as with other evals, can be very finicky. So here's sort of like a prompt to the audience. Here's 2 different prompts. These are sort of the first document in human eval fed into a model. Hypothetically, is there a difference between these two? And if so, which do you think is going to give better performance, or which one do you prefer? So maybe just think about this for like 10 or 15 seconds, and then I'll, I'll give the answer. Yeah, so, so these, these two prompts look very, very similar, but there's one key difference, which is these, uh, two lines at the bottom. Uh, this second prompt ends with one new line, a second new line, and then a tab. And so what this means is that if our code generation model has tokens that look like, for example, a tab and then the return keyword, and then go on to generate the rest of a solution, it means that if our model sort of— if the best solution to this function is just a, like, one-liner return some expression, our, our model that's prompted with a tab, if it tries to generate a new tab, it's going to create a syntax error, or, so it's forced to generate a token that's like return, but without a tab in front of it, which might not be a token that it has available, or it might be a token that it hasn't really seen during training. And so as a result, if you evaluate models with, like one using this prompt with no trailing whitespace at the end, and another model with this trailing whitespace, human eval performance will be something like 5% worse. For a number of models that are tested. And so basically this means that, like, just basically going down to, like, the minutest of details in your code that you're using to implement your evaluation, it could drastically affect performance. And so, like, if you implement your prompts using this trailing whitespace and someone else implements it where they trim off any trailing whitespace in the prompts, then you'll get different results, but it'll be very hard to tell sort of what went wrong. How things changed. So basically, in short, there are a couple of different sort of measurement options we have available to us, all which are trying to overcome this issue of not being able to easily score reliably freeform language model outputs or just natural language in general. And these are sort of— amongst these measurement options, there are a number of things that can go wrong or ways you can implement them subtly not necessarily incorrectly, but just differently to how is standard, and so on. And these implementation details are even not often sort of discussed or mentioned in papers or tech reports as things you should care about. So it's difficult to sort of be aware of them a priori. And so all of these challenges we've discussed are only scratching the surface. They're sort of only accounting for, did I run my language model correctly? Like, am I getting sort of the correct output from my model. They aren't even sort of accounting for external factors like dataset quality of your evaluation, like if it's measuring something you truly care about, if you're overfitting to it, and so on. And so basically, scoring models is hard and sort of influencing evaluations is difficult. And as a result, it's very difficult to achieve reproducibility in evaluations, where reproducibility here means basically If someone publishes their results, they should ideally publish enough details that you could go ahead and sort of write your own code and reproduce the same results they're getting within sort of a very small area of margin. And so this is key for research or just for sort of iteratively developing better models for production, because comparisons have to be fair in that advantages aren't given to a new model. For example, if you sort of spent much more effort prompt engineering your new language model you've just pre-trained or your new sort of prototype for production, if it does better, you don't know if this is just because you spent more effort trying to make it good or if the other one, if you just prompt engineered for 5 minutes, it would also be as good or better. And so there's definitely a question of sort of what is the correct way to set up a fair evaluation comparison and if it is actually holding. Everything constant, or maybe using a prompt format that, you know, the model was trained with, or so on. But at minimum, these evaluation details should be known and should be accounted for. So, like, for example, in the table on the right, the Llama 3 number, this is from the Llama 3 release, and these Llama 3 70 billion numbers are the ones run by Meta because they trained the model, and this is evaluations they ran themselves while developing it. But the, Claude and Gemini results are just results that are self-reported by the model developers, in this case Google and Anthropic. And they are not, as you can see these sort of subtext here, they're not necessarily using the same settings across models. And in some cases, it might not even be super clear what the developers did if they didn't release the code that they used for these evaluations. And so as a result, sort of like we see how Llama 3 measures up against these models, but we're not really sure if if sort of the developers were preferential towards their own model, or if just there were like easier prompts that had more information provided to the model, and so on. It's difficult to draw like a clean conclusion with these numbers. So basically, strong reporting standards for evaluation are important, but also actually reporting enough details in like the tech report that you report is very difficult. There are many, many things you might forget to account for or just assume that they're standard because it's standard to how your organization does things internally. But it turns out that other people don't have that insight or wouldn't implement it in the same way or don't think that's as natural. And so basically, in conclusion, like, I'm going to claim that if you don't see the evaluation code, then things are not going to be fully reproducible, or at least you're going to have a bad time trying to perfectly reproduce them. So I would argue that you should always share your evaluation code if you're doing something like developing a new model a new method in research, etc. And so, yeah, so basically reproducibility is important, but very hard, and sharing evaluation code, especially that's like clean enough for people to read and use, can be tricky. And that's where libraries like Evaluation Harness and other options like Helm or OpenCompass come in by having sort of a easy to reference, like gold standard, or just like, like frequently vetted code base for evaluation, it's at least possible to sort of not have to worry yourself about all of these tokenization intricacies and normalization of log likelihoods, etc., and more worry about, am I doing the evaluation I want to do? And then it's easier to sort of just say, instead of implementing all of these evaluations from scratch and saying, and sharing your codebase, you can instead sort of use these shared codebases. And so, yeah, we've been lucky to see that, like, the eval harness at least has been used for a bunch of research on evaluations, has also been used when new architectures like Mamba are proposed to sort of run on a couple tasks that the community has decided are canonical as, like, log— as log-likelihood-based evaluations of, like, base language model performance, and so on. And then I guess in contrast to other libraries like Helm, the eval harness, we, we sort of more intend to just put tools in the hands of practitioners and researchers in that sort of many tasks are supported in our library, and it's easy to define new ones or edit the prompts for your own purposes. But it should be up to you which of those tasks you'd like to evaluate on and which are best for you. Yeah, so, so I guess, yeah, so that was sort of more research inflected, but as a side note for production, there's a distinction that's been made by a number of people, I think, between model evals and downstream evals, where a model eval is more something like the MMLU benchmark, which is meant to sort of measure how generally capable or generally intelligent, with heavy scare quotes, your your language model is to sort of measure it against actual, like, base models versus a downstream eval, which is more, I have this concrete use case, like, maybe as a chatbot for this specific closed domain of answering questions about my company's documentation, and I'm going to evaluate my model and see how well it does on this specific task that I want to use it for and nothing else. And so basically, whenever possible, downstream evaluations should be used if you have a concrete use case in mind. The best eval is one that you've run the code for yourself instead of sort of pulling it from a tech report and just assuming that's how good a model is. Always try to test it yourself. But then even better is an evaluation that you've designed yourself that matches up with your own use case. And if you're trying to measure things that are more subjective, like, say, like sort of how, how high quality or how preferred your chatbot is. Potentially hiring human evaluators, although expensive, is worthwhile. And the incentives and sort of trade-offs for downstream evals are very different because for model evaluations, the thing that we care about is like model quality, and we, we think of a high MMLU score as implying that the model is a very good language model in general. And so doing things like training on the test, on the train set or the test set or sort of overfitting repeatedly by trying to maximize your MMLU score over the course of fine-tuning, we think of those as a bad thing because they're sort of ruining the ability of practitioners to draw conclusions based on the evaluation score about how good your model is. But for a downstream evaluation, if the evaluation is literally just how well does the model perform on sort of real-world chat transcripts for your actual use case, it might be beneficial to overfit and try to get to 100% 100% on that downstream evaluation, because doing better on that evaluation task just means you're doing better on exactly the things you care about and nothing else. So yeah, so basically some like high-level takeaways in this talk are basically that like implementation details matter a lot for LLM evaluation. Models are very, very finicky to specific prompts, details, and formatting, to specific sort of implementations of how you generate text or sort of normalize these measurements. And these can skew not just your numerical scores that you get out, but also the conclusions you draw about which models are better than others. And so this matters for research if you're trying to draw fair comparisons between a new method and old ones. And it also matters for production because, like the sort of tokenization bug that I showed in this coding example, If you were feeding your model a prompt in actual production with this trailing whitespace, then all of a sudden your performance would be tanking. And even though the evals looked good on paper, if they didn't suffer from the same bug, you might sort of have a worse model in production and not know why. And yeah, so, so this is a very, very non-exhaustive list of evaluation woes and things that can go wrong. There are many more, including data contamination, overfitting to your evaluation, or having it saturate and no longer be sort of useful for telling models apart. The actual measurement validity of your evaluation, like, is it measuring something that is a real thing that you should care about? And is it actually sort of a good match for that if that thing is like capability or something? And so on. Yeah, so, so in conclusion, some of the material in this talk and some of the, in particular, like implementation details are documented in a recent paper we put out from EleutherAI called Lessons from the Trenches on Reproducible evaluation of language models. And then also, if you're interested in trying to abstract some of this evaluation, uh, gory detail away, then check out the evaluation harness and other tools. Um, we recently launched the ability to wrap model prompts in chat templating thanks to a contribution from Hugging Face. Uh, we're definitely interested in extending to further evaluations beyond just the sort of text-only ones we support. Uh, and yeah, we'd love to sort of hear from you if you're interested in how we can make the library better, or if you're interested in helping out, even since we're very constrained on sort of ability to do all the things that need doing in evaluation. Yeah. In conclusion, thanks so much for attending, and love to take questions or anything else.

Thanks so much, Hayley. That was, that was awesome. I'm— we do have some questions that I'd love to get to. I am interested if people just wanted to start playing around now? I know, like, historically, I've gone straight to, like, Hugging Face and explored the MMLU dataset. It's really easy there with the dataset view of that, and that type of stuff, and you can, you know, get started locally with notebooks and that type of stuff. But if people wanted to, wanted to get started now, what's the best way to, to do that?

Yeah. So we have, like, an examples folder in the evaluation harness repository that has, like, a Colab notebook where you can just run the scripts in the harness. And then, yeah, I'd highly recommend checking out things like the OpenLLM leaderboard and just sort of looking at those datasets and what's in them and thinking about, yeah.

Very cool. So we do have a bunch of questions. One is some benchmark datasets have some existing errors in them, like grammar, et cetera. For example, HellaSwag has 26% of samples with error of some sort. Does this actually matter in practice?

Yeah, I would say definitely. I think, yeah, the Hello Swag example is particularly egregious, and I like to bring that one up. So the place that that's going to come most into effect is, say, if 10% of your dataset samples have errors in them or are mislabeled, and you're trying to train models that are performing better than 88%, like better than 92%, even though there's 10% errors or something, and you're sort of fighting over 89, 90, 91% accuracy, you're very, very close to sort of the point at which the benchmark has outlived its usefulness, if it hasn't already. Sort of any improvement getting toward 100% accuracy is only going to be a product of sort of overfitting to that evaluation and not a product of the model actually getting better. So there's definitely a point at which evals stop becoming useful for telling models apart in capability, which is the thing we might care about from them. It's not always what we care about. But if that's what we care about, like ranking models, then yeah.

That makes sense. We have a question from Kit. Blows my mind we can get all of these. So this is back to the getting all the logs and all of that, get all of these simultaneously. But so Kit's wondering, in practice, how do we get all of these probabilities for one given inference task?

Yeah, so, or maybe if it's possible to elaborate on the question, so there's two components, I guess, like, when you evaluate on a given task, obviously you're going to have to run inference on, like, each document separately. You can definitely batch those documents together, but you're going to have to sort of figure out what the model's output on each different input would be separately. But in terms of like getting the predictions at each timestep in parallel, this is sort of a characteristic of Transformers as well as other sequence models like FFMs. But it's the reason that when you train a GPT-2 model with 20,000, 2,048 tokens in context, you don't have to run 2,048 steps. To get the model's prediction at timestep 1, timestep 2, timestep 3, you can just sort of run your batch through the model a single time. And so by construction, our models are sort of able to do this parallel processing, and this is why you can only, like, you can process your entire prompt up to a certain point, you know, only one step, or sort of pre-filling before generation, and it's why you can train without doing sort of a number of steps equal to the number of tokens you're feeding in.

Awesome. Thanks for that extra color. We'll get to a few more questions, and if people who are here, if you could upvote the ones that, that you'd like to hear answered as well. Shamik has a question, and you have talked about this a bit. What are your thoughts on LLM as a judge? In that case, how to ensure that the judge model is correct? I, I want to add a bit more flavor to this question just from You know, the way I think about it, and you, you really spoke to this with your Joe Biden example. We're actually in a weird situation, right, where all our old tools— now we have LLMs generating a lot of natural language. All our old tools of, you know, pattern matching and NER and all these NLU tools aren't quite enough given that essentially when LLMs respond to questions or try to answer things, We do actually want to know something about the semantic meaning of the text they produce, not necessarily the string matching or regular expressions or anything, anything like that. So we are in a situation where kind of even if we didn't want to use LLMs as judges, we're kind of— there's a forcing function of some sort. So given that context, what are your general thoughts on LLMs as judge, Hailey?

Yeah, so I think there's an interesting tension between sort of the fact that we want to use LLMs as a judge or sort of human evaluations and annotations as like scorers for tasks that are inherently more difficult or complex because it's harder to sort of come up with a heuristic that's going to closely match this performance. Like, if it's sort of a more subjective or just like multi-step reasoning process to decide whether a task has been done correctly. It's more desirable for us to use an LLM as a judge because we want to just sort of be able to have something that spits out a number that says whether it's correct or not. But at the same time, this is exactly where LLM as a judge is going to be less potent because just these models are going to be better at the simpler task of, say, extracting, like, the multiple choice, uh, sort of answer that was produced, uh, from the model's output. So, like, I think LM as a judge is, like, a very valuable tool, but I'd like to see more work done on sort of where they fail and what their limits of capability are, because, like, if you're trying to use GPT-3 to evaluate GPT-4 on a task that GPT-3 can't do, you're likely going to have a bad time. Yeah, so in short, I think it's a useful tool, but more attention has to be paid to sort of what is the performance of the judge model before you use it willy-nilly.

Great. Someone has asked about the ARC benchmark and that it's been offered on Kaggle, a $1 million prize for the winner. Why is ARC a harder benchmark?

Yeah, yeah. So ARC is much more focused to, like, generalization. Many of the evals that we use for language models are ones that, while it does sort of measure, like, if a model can perform the task, that means it's capable and useful. At its core, many of these, like MMLU, are sort of a combination of, okay, can the model do in-context learning, but then just like, does it know enough facts? And so these are things that if a model has just seen it enough in its pre-training corpus because it's listed as a fact on the internet, then it's going to be able to answer this question. And so like, I think for ARC and for things that require many, many hops of reasoning, it requires at minimum a lot more scaffolding around the language model to perform these multiple leaps. And so like current benchmarks are often memorization tests, or at least sort of can be heavily improved on by just increasing memorization, whereas sort of performing tasks that require many, many leaps of reasoning, again, reasoning with scare quotes, is a much more difficult challenge. Right.

We have a couple of really nice practical questions around, and one is One's from May, the other's from Simon Willison. Simon's speaking later today, everyone, if you want to check out a talk that we're all excited about. But they're asking around just how to get a model to reliably give a multiple choice answer without, like, a whole bunch of fluff or not giving the answer as well.

Yeah. So I guess one component here which we have not integrated or explored in the LME evaluation harness is structured generation tools that can actually enforce, say, we'll do freeform generation from a model, but it won't be freeform because we constrain it to only output the appropriate tokens. So things can be done to mask out other tokens' probabilities and sample from the model but prevent it. I guess another component here is, for the most capable models, I guess, just prompting it with the system prompt. To directly and only return the answer. But I would say, like, if you don't have the ability to do structured generation because you don't have that sort of access with your API, asking nicely and, like, I think, yeah, it's tricky because I think a lot of models are trained nowadays to sort of always produce large chains of thought, either because it helps them or just because people like the conversational style. So I think while like system prompts can help on this, if you don't have access to some way of like sort of more reliably ensuring with structured generation or log likelihoods, it's going to be tricky. Cool.

I think asking nicely is a nice note to end on as well. And particularly given, you know, the breadth of everything you covered here, super grateful for you sharing all of your wisdom from the the bleeding edge of all of this work, Hayley. And thank you all for joining as well. We didn't get to all the questions, as is usually the case, but we can get to the rest in Discord as well. So feel free to continue the conversation there. And definitely we'll share more of the links that Hayley has shared. Are we able to share your slides as well, Hayley, or—

Yeah, I can send a link. And I don't believe I have access to the Discord channels, but I can— I'm happy to answer questions after the fact. Amazing.

I'll I'll send the link to you on as a Twitter DM as soon as we jump off the call. But thank you once again, and thank you all for joining.

Yeah, thank you so much for having me, and thanks everyone for the questions. Awesome.

Alright. See you soon, everyone. Ciao.