The Evals That Made GitHub Copilot

Alright, then let's kick it off. This talk is the evals that made GitHub Copilot. And just to get things started, we'll go through quick intro to— so that you guys can know who you're talking to. My name is John Berryman. Very nice to make all your acquaintance. I worked at GitHub starting in 2019. But that was, that was well before Copilot was a thing. And so my, my first thing at GitHub was search. At, at that time, search was the world to me. It's, it's, it searches my hammer and the, the world is a nail. But over time, after I worked on the Copilot, I worked on the code search team. I moved on to data science and then worked with Hamel there. Actually, it's the first time we got to meet. And then from there moved into Copilot because I had kind of a data background and an engineering background. So it was a good fit for me. My projects while I was there was code completions. And then after about 6, 7 months of that, I moved into the chat, but it was web chat. So it's not the IDE chat. Since then, About a year ago, I stepped away finally from GitHub to become an independent consultant, following again in Hamel's footsteps. And I wrote a book that was published last November called Prompt Engineering for LLMs. Also, since he's given me the bully pulpit, and Hamel, if you wouldn't mind, like, just posting a link from this, that would be awesome. I do have a Maven course coming up soon about building LLM applications. So The idea is like we're getting lots of buzzwords and there's like a million different ways that you can build all this stuff. But when you get right down to it, there's a few underlying principles and if you know them, you can rearrange them to make, to make an amazing diversity of things. So that's kind of the, the, that. Sean, do you wanna come in here?

Yeah.

Hi. I posted John's course in the chat.

Thanks, Hamel Husain.

Hi, I'm Sean Simister. I worked at GitHub for 3 years, starting just after Copilot had come out in beta. Before that, I was at Google working on some machine learning stuff, some search engine stuff. And yeah, so at GitHub, I took over some of the offline evaluation work. Building out— we'll talk about the frameworks that we're building out for offline evaluation. And then I prototyped the first version of Copilot Chat. And that led to a whole bunch of new evaluation techniques as well. And so I'll talk about that later. And then, yeah, just recently, I've been working on some personal research projects, ExplainPrompt, Ground Rules, just really enjoying building with LLMs and building developer tools. And thank you, John, for inviting me to reminisce about our times working on GitHub Copilot.

Yeah, this is really exciting. I mean, from my perspective, just, you know, I was around when GitHub Copilot first started. And when it first— that effort first kicked off, I was pretty skeptical. You know, they just partnered with OpenAI, you know, and it seemed like you know, the first project they jumped into right away, there's a whole like menu of different things. One was, you know, code search, which I thought, okay, that one seems like you could tackle it. You know, no one really ever has seen like a language model like that before, the one that OpenAI privately shared with GitHub. And, you know, I was really skeptical, like the first version did not work well at all. It was like barely functional. If that. But it's really like because of the evals that I saw the product improve. Like you could see like every other day all the engineers would make progress against evals. And by the time, you know, first it was like vibe-based evals, like people on the GitHub team, we would be using it and be like, okay, this is cool. And after a certain point, like the evals you know, like we would be moving the metrics on evals and after, you know, that provided a lot of signal into what wasn't and was working and seemed like it was really unlocked the product. So I think from my perspective is like very critical. So I'm in, but I don't know the whole story. Y'all know the whole story because y'all have been there. Y'all worked on it for quite a long time.

So we know the whole story up until the point where we left and it's a really interesting story. So, uh, I guess let's just, let's dive in. First off, obviously quick slide, but make sure we have a shared foundation. Copilot, you guys all know what it is, right? Yeah, you type in some code and it automatically completes stuff. I think this is probably one of the most common completions that I got while writing comments. It was always insulting my work. This was the first part that grabbed everyone's attention, but it wasn't the only part. Both Sean and I worked on the code completions, but Sean was actually one of the very first people to work on the chat component to it. And here you can see kind of a screenshot of how that's evolved into what it is today. And that's not the only part that there was. There's— it's probably you're less familiar with it. But this is the part that I moved to after code completions. There is an online component that you still have access today. I think it's free to everyone. So you can ask questions about your repo and stuff like that. But as Hamel Husain, evaluations were really a core piece of what made Copilot. And so let's dig into that. Now, This is— this kind of gets into how my own thinking has shaped since working at Copilot. And I'm going to attend Hamel Husain's course. So I'm excited to be corrected. But this is the way I kind of mentally cut up the evaluations that we were doing at GitHub. On the easy end of the spectrum, you have something that's like algorithmic, things that you get the code completion, and it's easy to just like check, like an equalities check. For whatever you're testing. So does, you know, if I'm extracting structured content from a particular website, does this website lead to exactly this output? And you can have, you know, a bunch of those as, like, tests that you use in an eval. Does the response structure adhere to JSON? And in particular, the schema that I'm looking at, Is the length of the response within limit, or does, you know, maybe it goes, it's too verbose or something, or too pithy? Is the code all in brackets? Any of these things are things that are just really easy to write a normal programming check for. At the other end of the spectrum, we have LLM as judge, which is hugely important because so much of the work that we're doing is subjective in nature, and you kind of need a human to do it. But humans don't scale. So LLM is judge is anything that is subjective. You ask, you know, what is a preference for, you know, this completion versus like the canned response? Am I hallucinating or is my response, you know, actually accurately using the context provided? Those type things, more subjective things that you can't write an algorithm for. Now Copilot was at this kind of interesting in-between that I've come to call verifiable evaluations. There are things that it's like, it's algorithmic in some respect, but it's not just so easy as looking at the output. It's things like I'm generating code and I wanna make sure the code compiles. I wanna make sure that the code passes unit test. It maybe it's SQL and I wanna make sure that the— I don't care what query it writes, but it needs to have the right answers come back. So it can have some degree of freedom with that. So all these things, the code completion, you can't immediately look at it, but you can run it, evaluate it somehow in a way that's more complicated than just the algorithm. And finally, underlying everything is A/B testing. There's nothing— particularly special about this for Copilot, but it was just an important part of our history. All right, so we're going to walk through each of these and tell you the piece of the puzzle as it lived itself out at GitHub. So the first thing was exactly what Hamel Husain was referring to a moment ago. We called it Harnesslib. Verifiable Code Completion Evaluation. And the idea was, you know, we've got this magic box— certainly it was magic at the time— that will look at your code and write, hopefully, working code. So as we were working on making the completions better, pulling in more context data that was, you know, we hoped would be helpful, we had to test the changes before we go live with them. So this is an offline tool that, that we did for this, and it was one of the first things I was introduced to when I started with Copilot. The idea is simple, but you got to admit it's a really cool idea. First off, we would gather test samples, and here's the rundown of how that works. Basically, you grab, you know, we We've got all the repos. So you grab a large set of open source repos, specifically in our case, Python and JavaScript. We had to narrow it down. You run their test. That's basically why we had to narrow it down. We could run Python and JavaScript tests pretty easily. You run their test and you keep the repos where all the tests pass. So it's kind of a funnel. Of those, you use code coverage tools. To associate the functions with your unit tests. You had to make sure that you're actually testing something. And then of those functions that had some coverage, we've kind of forgotten the exact criteria, but basically the candidate functions had to have one unit test with at least some percentage of its lines covered. It had to have no more than blah, number of lines. Otherwise, it was too hard of a task for early days Copilot. And it needed a docstring. So we're looking for something that is the function, function name, the signature, the docstring, and the context around this piece of code. All those became candidate tests for evaluation. And then running it, you kind of see what we would do. We would grab some of these, one of these candidate functions, kind of, you know, melon ball or scoop out the implementation of it, generate the contents again, and run the unit test again. And if we do this across lots and lots of functions across lots and lots of repos, then we can start to get an idea of directionality. Like, as a baseline, we know what version X of Copilot did. It wouldn't succeed at all of them. And I forgot what it was up to. It was like 40% or 50%. It wasn't even a huge number back then. But we know that—

Can you talk a little bit about, if I recall, y'all selected these repos to be like, you had a way of selecting or filtering ones to like higher quality and stuff like that. Can you talk a little bit about that?

Maybe, maybe I touch on to it a little bit in this lessons learned thing right here. But Sean's going to cut in here in a second. See, we got these cute little pictures, and he can probably address that a little bit too if I miss any points. But, you know, as far as quality is concerned, that was one of the things that was kind of a lesson learned. We, we had to make sure that the content itself wasn't trained into the model. We would get these models from OpenAI, and guess what they were training the models on? Long, all, all the code at GitHub. So when we had closer relations early on with OpenAI, because we were one of their first main customers, we could get a list of what went into the training and make sure that we had something that was trained after that date. It's also important to make sure that your repos are consistent with the traffic that you expect in production. And I I think we never quite— I mean, it was always very subjective of this. But I think we always knew that we wished it was a little bit more consistent. Some of the models, because they weren't in the training set, were the newer models. Sorry, some of the code bases that weren't in the training data were newer code bases. And guess what? Newer code bases tend to be smaller, tend to be certain type of thing. And so a lot of our code bases were a little bit smaller. Than what, you know, you see work done in production.

That's really interesting about the relationship with OpenAI. You mentioned, like, in the beginning it seemed really close, like every week there was calls with Greg Brockman and Ilya. I remember I would get on conference calls with just them and a few other people. It seemed like super close. Was it— so it didn't stay that way? It wasn't like that? Like, necessarily throughout the entire time.

Yeah, it's interesting. And by the way, Sean, you feel free to cut in here at any point that you want. Um, I— you, you had an earlier window than I did, even though we were gonna get out at the same time, but you were in the middle of it at that point. Uh, Sean, uh, got to see about a year before I, I did, and then I think both me and Sean left, uh, at about a similar time, uh, last middle of last year. So yeah, at some point, um, we, we were the, we were the big one, the big kid in the kiddie pool. But eventually, you know, ChatGPT landed, and it was very important for them, uh, to do the best thing for, uh, their own product. Uh, maybe a couple— we've got a lot of ground to cover, so maybe just a couple more tidbits here, um, and, and then we've got, uh, the 3 other types of vowels to go through. But a lesson that I think I learned, and the reason I've got our pictures here is because Sean almost learned the opposite lesson, and he makes a really great point. But I always felt like our test harness should have been more, should have basically been a headless VS Code so we could absolutely test every single component. Of our code base as it changed. And I still agree with myself. That ends up being really good for integration tests to make sure that you're keeping the evaluations high. And now, maybe Sean, you can pop in here, but I liked the points that you're making about the flexibility of the harness.

Yeah, yeah, just there was this kind of— these different phases of product development where, you know, sometimes you've got a new model with new capabilities coming out and you're trying to imagine like, oh, how would people use this? And how can we evaluate something that people have not even had a chance to try yet? And so in that sense, you can't just wrap the existing product in an evaluation harness and have that be the only way to test things. You do need this more experimental system where you can kind of prototype out new ways of using the AI, new ways of using the product. And, but then those are always less reliable. And so once you launch that, you kind of have to go back into making it more, more reliable, more, more easy to evaluate.

So that covers Harnesslib. The next big chunk is, so Harnesslib was verifiable test, and it was probably the biggest chunk of our evaluations and also definitely one of the coolest. A/B testing is what we did whenever it was time to ship our changes online. The offline test that we had would, would, you know, hopefully, you know, limit, limit the exposure to, you know, shipping something that was surprisingly bad. And I think it did a good job of that and helped us really flesh out what was going to be a good product to ship. But once we finally did ship it, we had to make sure that the changes are appropriate, that, you know, we'll start them— start our users on like 10% of the traffic and kind of wrap it up, keep, keep an eye on how the product is doing and see if anything is changing. And a really important lesson for us is it came from basically how these things were set up. We had just a very few key metrics. And to the best of our memory, it's these 3 items on the screen here. Complete— completion acceptance rate. So, you know, a user gets the ghost text on the screen. Do they accept it or not? So how often do they accept it? Characters retained. A lot of times, especially for users like me, I would like mindlessly accept everything and then edit. That was kind of the pattern that I use. So we also had to incorporate that type of user, that type of use case into it and make sure that, you know, people weren't zombie-like accepting everything and then, you know, being, not liking any of it and having to, you know, go back and change it. And the last thing is kind of an obvious one is latency. If we come out with a recommended piece of code that is too late, then it's, it's just bad for everything. Now, beyond the key metrics, we had several guardrail metrics. And by several, I mean, high tens, maybe, maybe even into low, low in the, the hundred. But it, there were a few of these were like variations of these things above, like characters retained after a certain number of minutes. But, you know, anything that would give us a way of kind of measuring how much it changed to see if there's anything weird going on in the side.

So I have a question about guardrails. Like a lot of times guardrails, people have guardrails that are like binary pass-fails, like really catastrophic errors or something they want to like prevent against. Was there some of that too, or something like that too? Or this, was it mostly like these metrics you keep an eye on that are like drifting wildly?

Yeah, mostly we were looking at the key metrics that made the conversations really easy. Just so we would focus on just those few simple things. Guardrails, we didn't have like just a hard fail if something went wrong with them. They were, they were part of the conversation if something was really going wrong. Or if we were trying to understand the metrics, the guardrails were just useful side conversation to diagnose and figure out what was going on with the key metrics. So they all kind of fed back into that.

When in the cases where, let's say, those key metrics like acceptance rate, latency, and everything were improving, let's say, but then the other metrics were sort of— going astray. But did that ever happen? And like, did you do a bunch of analysis, like data science-y type of work to like figure out why? And how was that— how was that conversation like surfaced, let's say?

Uh, so stuff like that would happen occasionally. And, uh, I, you know, I'm gonna be super fuzzy on this because I wasn't really in the middle of this discussion, but we did have, uh, like weird coupling in several of the metrics. It's the second bullet here, weird trade-offs. So like sometimes you would maximize one of the metrics and it would just tank another metric and you'd have to think really hard about why those two were trading off. Like, you know, if you really optimize for acceptance rates, well, one silly thing that would happen is like you have a bunch of little tiny completions. And so the experience was garbage when that happened. But you'd have to really dig around in some of the other metrics to figure out why a sudden departure happened from whatever your baseline was.

How did that process, that detective work happen? Was it like a lot of data science-y type of work? Were you kind of doing a lot of data exploration or— You know, like, yeah, how did it— who did that kind of ex— like, detective work? How did that— how did that work?

Yeah, so as part of, like, these, these A/B experiments, you know, we, we would ship out new, new models, new prompt crafting techniques, and then, you know, we would hope to see, you know, the key metrics going up, uh, but then sometimes we would see something like a guardrail metric, like number of lines generated would, you know, average number of lines generated would jump in some, some weird direction, either up or down, could be kind of alarming. And so then there were different scales of that is like, you know, we'd have the prompt engineering folks in the meeting, and they might say, oh yeah, we, you know, we expect that because we improve the prompt in this way, and it's giving more targeted suggestions. So it's actually okay if the number of lines goes down. Because they're higher quality lines. And then like you said, there are other times where everyone's like, we don't know why the number of lines have gone up. This is concerning enough that it's worth delaying the launch for a week while someone digs into the logs and actually figures out what's going on. And then it becomes a data science project. And so luckily, yeah, we had a whole bunch of people on the Copilot of Alice team that had that skill set who could kind of go off and dig through the data and figure out what was going on, come back the next week for Shiproom and say, all right, we looked at it, it's okay, here's why, let's, let's move forward.

Hey Sean, you're, uh, you're doing the talking right now. I think this is one of the most interesting parts, and you were, uh, right in the middle of this LLM as judge stuff. Do you want to take this next section? Just tell me when to progress the slides.

Yeah, yeah. So, um, So yeah, we felt we had a really, really good system with this offline harness lib eval where we could run different prompt crafting experiments or fine-tune models and see how that affects the product in an offline setting. And then we could also kind of ground that in online A/B experiments and make sure that the changes actually expected the way— actually played out the way that we expected with our real user base. So we're feeling pretty good about that. And then ChatGPT came out and it kind of changed how we thought about evals. And I think it was obvious from the beginning that these chat models were gonna be a big deal, but we didn't quite realize at the time, you know, how much it was gonna change our evals. And so yeah, this is how we kind of backed our way into LLM as judge, I guess. And started looking more at subjective quality judgments instead of these purely execution-based, does the code work or not, which was one of the key ingredients to getting the Copilot completion product to that point. So at first— yeah, thank you, John. At first, I think— when we looked at what people were doing with ChatGPT and like kind of conversational programming, I think we were calling it at the time, it kind of looked like what we were already doing with Copilot. You know, people would ask it, you know, make this code cleaner or, you know, write some unit tests for me or help me fix this bug. And then it would, you know, go through multiple steps of conversation, but there'd still be some code output. And so I started building out the first version of Copilot Chat and thinking about how I would evaluate that so we could, we could launch something eventually. And I started to realize, like, even though there was code coming out of chat, what we needed to evaluate wasn't just the code that was coming out of chat. What we needed to evaluate was the actual conversation, right? And people obviously cared about getting good code that came out, but they also wanted to feel like the conversation was productive, like the, the assistant was listening to them and giving them good feedback and that sort of thing. And then, you know, we also had use cases in chat where there wasn't code coming out, right? So someone would say like, explain to me how this code works, or like, you know, why, why is it using this API and not that API? And those are really helpful answers, but there's no code that we can run. And so I started to think like, maybe the only way to evaluate these is just to have like side-by-sides, right? Like you have your baseline, like, here's how ChatGPT handles it, and then here's how our agent handles it. Which one is better? You just put it in front of a human and you ask them to rate which one is better, which one is worse. But obviously, that's slow and it's expensive to hire people and get a lot of those kind of judgments. And you have to come up with a lot of guidelines too to help people You can't just say which one is better. You have to explain what better means. And they kind of read through and judge which parts of it.

And then—

so I set up a basic version of that in Label Studio so I could just kind of test the flow. And I found it difficult just to say which one was better. And so I tried to make my life easier by passing it into GPT-4 and saying, OK, here's my grading. Criteria, tell me which one you think is better. And then now I can just click through in Label Studio and kind of judge the judge and say, did GPT-4 make the right call on which one is better or worse? But even that was challenging. This was years ago when models weren't as good at following the system instructions. And so sometimes it would get distracted. And it would say, you know, this— the code on this conversation is more maintainable because, you know, they put their braces on a new line, whereas this one, the braces are on the same line. So someone would have to go in and adjust that. And I was like, no, that's not what I'm evaluating at all. You know, I'm trying to ask, like, did it write better unit tests? And so I started going in and actually just writing out in, like, bullet points explicitly what I wanted it to evaluate. Like, does it have a test for passing in null values? Does it have a test for passing in an empty list? And finally, that got it to the point where I could use this LLM as Judge. And it would stick to my very granular list of things that I wanted it to evaluate on and give me good results. And I could click through quickly and double-check these. And then once I verified that it worked for one model, then that could kind of become the baseline. And I would only have to compare the new conversation to see if it matched the baseline. So that was kind of how we backed our way into this LLM as judge. It wasn't something that we decided that we needed at the beginning. It was more just kind of realizing that Harnesslib wasn't going to be able to give us what we needed. Because Copilot Chat was, in fact, like a whole new product. It wasn't just the obvious next step for code completion.

Now, were you able to measure, like, somehow the alignment between the judge and yourself or other people? Like, how did you— how did you go about instilling trust in the judge?

I would've loved to have done a much more extensive study proving that our judgments were being aligned with the judge. It was a crazy time to be working on AI and everyone was shipping chat products as quickly as possible. And so, yeah, the first version of Copilot Chat went out with, basically our gut feelings of, you know, here's what we think users will do with this product, and here is an offline dataset that kind of models that gut instinct. And then, you know, once we launch it, then we can, we can actually start to collect telemetry and see how users are using it and then adjust our offline benchmarks. But the first version was really just like, a bet on how we thought this conversational programming would evolve.

So on balance, like, would you say the judge was helpful, like more helpful than, than wrong? Like you were able to see the outputs of the judge and find, you know, use case, find examples you needed to work on or correct things in, you know, in the product?

Yeah, absolutely. So once Once we kind of made the adjustment to micromanaging the judge, where we're saying, is there a test for this? Is there a test for that? Is there a test for that? Or does the answer mention that you have to do this? Does the answer refer to this specific part of the API? Then it was much more reliable. And I think that helped people get over also this weirdness with LLM as judge, where it's like, OK, you're asking the LLM to write some code for you. And then you're asking the same LLM or a slightly different LLM whether it did a good job. Like, isn't it always just gonna say, yeah, I did a great job? But the intuition here is that like the generative task of starting from, you know, the user's question and actually solving it is a much harder task than giving it this really like detailed list of requirements and then just getting it to check each one off. So once we got to that point, yeah, we were pretty confident that it was giving us good results. I mean, obviously there's still, Sometimes would be hallucinations where, you know, you ask it to check out off on the list and it adds 3 more items to the list.

You know, people always like to ask, okay, when you're doing all this work on the LM as a judge prompt, do anything from that prompt make it over into the, that the code completion model? Or, you know, did you end up prompting another model with elements from the rubric of the judge? Did you end up, you know, making the main prompt more specific based upon, like, you know, yeah, like, just curious, like, did it kind of feed into each other at all? I don't remember any prompt of the model. Maybe it's just like a code completion model that was just, you know, but I'm just curious if there's anything like that going on.

I don't remember any specific cases where it did. I mean, Like I said, we were much more familiar and confident with how we were evaluating the code completion product at that point. And so much of the Copilot Autocomplete was like, can we get the right context in the prompt? And so— and we were also fine-tuning models. But it wasn't like crafting the right system prompt the same way we were doing with Copilot Chat. I know that we had done a number of experiments with writing unit tests in Copilot completion. And then we did other experiments in chat with writing unit tests. But I don't think one ended up feeding into the other. It ended up just mostly happening in chat. But one of the interesting lessons that we learned coming out of this was— or that I learned personally building this was that there ended up being these two very different personas on the team for evaluation. There were folks who, you know, were trying to ship things, we're trying to keep things from breaking, we're trying to prevent regressions, and they would come in and they would want to run via eval. And so we would have to have, you know, really clean, reliable ways to run via eval, get a number out, compare it to the previous number and make a shipping decision. And At the same time, there are a whole bunch of folks who wanted to pull everything apart into LEGO blocks and reconfigure them and say, what if the product worked this way? What if we use this model? What if we did that? And so it was much more experimental and trying new things that hadn't been built before. And so, yeah, it wasn't always clear that that should be all one eval pipeline. And I think over time, we developed more and more evaluation tools, evaluation pipelines, and it felt a little bit overwhelming to some people to have more than one way to do eval. But I believe it was necessary because there's just more than one type of eval. There's these kind of unit testing experimental things and the more regression testing, you know, shift decision things.

What's like one of the more experimental things where you kind of took it apart with modular pieces and whatever? That sounds really interesting.

Um, actually, John, John was doing some of this stuff, um, with like trying to— yeah, originally we would just kind of chop up code by line numbers and shove it into the prompt, uh, and John was trying to be a little bit smarter about that and find ways to, to treat code as code.

Yeah, it's, it's hard because it's, it's so, um, I've forgotten a lot of the details, but I remember we used to have when we were running the evals, it was basically a notebook. And in the notebook, we had a pattern for like a typical Harnesslib, a typical Harnesslib eval with, you know, where to get the data from and how to set up these individual tests. But, and it was like Lego blocks. You could just use the piece that you needed. But it's kind of nice because if you were to make something that fit in the same, place that the old Lego block went, you could, you know, kind of mix and match. And like Sean is saying, you could prepare for evaluation of a product that doesn't even exist yet, rather than do like a regression test against something that's more rigid and frozen. And so I've forgotten the details. It's going to be uninteresting, but some of the work I did was, whereas initially we were like melon balls scooping out the full contents of our candidate functions, a lot of our important completions were the one-line completions. And so I just made another way of harvesting candidates as the Lego block where the candidates were the function with a piece of the code missing or like the bottom half of the code missing, something like that, where it looked like you were editing a block instead of editing the, you know, here's the new function, which is a little bit different than what our users typically did. So Yeah, it was really neat, the Harness lib, just how modular everything was.

You mentioned that some of these evals run in notebooks. Did those note— do you run the notebooks programmatically? Was it more like ad hoc, kind of open a notebook and run each cell? How did— that's really interesting, I think, to a lot of people.

You could have done it either way. And so, like, for the more regression testing, like, flow, I guess these things were pretty pretty canned. It was in a Python script, and you could just, like, run the thing.

The—

but especially as you're getting, you know, Sean had been working on this for a year by the time that I was introduced to it. And to get an understanding of how these things work, it was just so easy to go to a notebook, which was the same stuff as in this, you know, canned script. But you could see how each bit of it worked, and you could, you know, it's a notebook. You can muck around with it and and play with the Lego blocks, change out the evals and, and kind of get an idea for what you could do in the future. It was kind of neat.

Yeah. Yeah. A lot of like quick and easy idea or experiment ideas started off in a notebook and then would kind of move into just a regular Python script. So you could have an eval that, that people could run easily. And then some of those would move actually into like, you know, Azure pipelines that would just run automatically. But notebook was mostly just for experimentation.

Yep.

A quick note on time. We're 15 minutes past time. There's a couple of slides left.

No, we can keep going. This should have been an hour. I don't know why I made it 30 minutes. This is my mistake. So, whatever.

Yeah, it's because we just had me in the beginning, and then we have a whole year's worth of knowledge prior to me. That joined after. Well, so let's keep going, and then— but we can obviously stick around for a bit. The conversation's been fantastic so far. All right, so the last little bit— remember in the first slide, we had the 4 things we used? This is just kind of a footnote, to be perfectly honest. It's a much smaller type of evaluation. As far as I know, it might have grown and become a really important thing after Lyft. It was coming into existence about the time that I left, and I probably used it a solid twice. But it is a demonstration of the third— the fourth type of evaluation in my type hierarchy, algorithmic evaluation. And what we were doing with this is, as the chat models, you know, gave way to Chat with Functions, you know, these more interesting capabilities, it was important to double-check that the function was making the correct tool calls. Or sorry, that the LLM was making the correct tool calls. And the way you could do that is you provide the context, user message, you know, all the tools that the the model is going to have access to.

How many tools approximately did Copilot have roughly at the time?

I could probably speak to the web interface, and Sean could probably speak to the VS Code. But it's kind of funny. This is a hack. You can go to the web interface in GitHub right now— homework assignment— and you can ask it what its tools are, and it'll tell you. So we don't, we don't like hide that very well if we're supposed to, but it didn't matter. And I think there was probably, when I was there, just a handful, like 5, 6. I think since then it's kind of moved up to about double that, but I haven't checked in a bit. So not just a whole ton, but more than, more than a couple. Yeah, but the, the cool, the cool thing you could do is, you know, given given that expected kind of trial input, you know what the output should be. And you can, besides just doing like, you know, checking for accuracy by seeing if this is an exact match for the tool call, you could do cool stuff. Like you could do confusion matrices like this. I mean, it's a toy problem I showed you on the right. But, you know, you'd say, here's all the, the tools that my model has access to. And we attempted to call them this number of times. Here's what actually got called instead. And so that provides you with a lot of good information there. You can say, if you got any tools that get called, that get mistook a lot, you can step back and say, oh, I need to figure out how to make these things more distinct. That your tools ideally need to partition the space that they're at, they're operating in, so that there isn't, you know, silly overlaps that will confuse the model. So that's just a, just kind of a tidbit.

I'm glad to see the confusion matrix. Actually, in an earlier Lightning lesson, Brian went over very similar confusion matrix for evaluating agents, like agent handoffs. So Yeah, this is pretty— yeah, right, it's pretty cool. It makes sense.

Yeah, you can see how often they— yeah, it's really important, right? Because if, if you're— that's just a very similar task. Uh, if you're handing it off to the wrong agent, then probably what it means is, is there's just too much overlap with your agents and the orchestrator model doesn't know how to decide between them. So yeah, confusion matrices are, are great for that type of thing. All right, so we're back to this slide again. And we have our bases covered. Verifiable evals was Harness code with code completion. LLM as judge was that really interesting conversation with Sean then that revolved around chat completion. Shiproom was our A/B testing. And that's whenever we change the model or tokenization or the prompt or anything like that, we made sure that we put that through A/B test.

And then finally, Just a footnote, but tool use evaluations was, was an example of where we're also using algorithmic evaluations. So that is it for the content. Thank you guys for sticking around for 22 minutes longer than we'd set this up for, but I guess we can still—

Yeah, there's some questions if you don't mind. One from Jesse is asking, what are your approaches with respect to caching? Like, if the inputs the prompts, the model is the same. Do you somehow leverage caching with all of these tests, like you're running them in mass? How did that—

I remember— I'll punt this over to Sean shortly, but I remember wishing that we'd done something more intelligent with caching for Harnesslib. Some of those tests took a while to run, and I felt like there was still room to do more caching. Sean, do you remember some of the specifics? It's a horrible thing I'm doing to you.

Yeah.

No, no, no. I don't remember any specific prompt response caching. I mean, obviously, the model hosting had some optimizations to speed up reusing the same context, that sort of thing. But most of the caching approaches we had in Harness were just sort of the preparing the code. So John talked about how we would select these repositories and then we would go through and run the tests and then we would parse out the function bodies and that sort of thing. You don't have to do that every time you run the eval harness. And so actually touching on another question we have here from Pastwar, we could run all of that analysis once for this fixed set of repos store the list of functions that we had and the offsets into those files because they weren't changing in a SQL database and then just refer to that SQL database every time we had to rerun the harness.

And then you build like your own dashboards and some kind of other alerting. Like what kind of thing did you build on top of that eval data that every time you run tests you get all these, all this telemetry? I'm thinking. Did you have something on top of that that you use?

So for the offline eval, you know, this was just something that we were mostly kicking off manually and then we would, you know, build a pretty simple little, well, I don't even know if you could call it a scorecard, but just a list of metrics, you know, showing pass rates. So there wasn't a lot of dashboards there. But for the Shiproom decisions, there was a lot more. And so the teams that we worked with at Azure had already built a lot more tooling around displaying the results of the A/B experiments and kind of making sound decisions based on how much data we'd collected and whether they were representative samples and that sort of thing. And so that was really helpful to have that expertise brought in into the product.

Um, did you find that, you know, when it came to troubleshooting errors that you saw or kind of digging into the results of the evals, was it, you know, was it necessary to hand things off to a separate data science team, or what were like engineers like able to figure things out, or what's your intuition on like can this be done by one set of skills, or do you need a distinct set of skills to do it? Um, you know, that's one of the questions that always comes up with evals, because a lot of people are doing evals, you know, there's people of different stripes doing evals. There's like, you know, a lot of software engineers with no data science experience. So I'm just curious, like, how did it unfold?

Like, uh, one thing that I've often heard you say Hamel Husain is, is that, uh, and I started parroting myself, that whenever you get someone to evaluate these things, uh, you get the people that are actually good at this stuff. You get, you know, the content curators that know what the heck they're, they're talking about when they're reviewing RAG or, you know, whatever it is. In this case, it was great because I was an engineer, I was using this stuff, and when I— and, and so, and the, the harness lib was really engineering-heavy testbed. I was using, you know, Python notebook a lot of times when I was just running, kicking these things off. And I was the best, and I was the one working on the product that was running through the valves. So maybe it's, you know, there's so many fortunate things about this very specific niche of CodeGen that doesn't exist in other places.

Where you really do have to have the other person that's not the engineer.

But a fortunate thing that we had is like we could do it all, and it made it, I think, a lot easier in that circumstance to say like, this eval went sideways, but I think I know what's happening. And you didn't have to have that communication and the breakdown.

What about, what about like for situations where code completions for programming languages that you are not familiar with. I don't know which programming language you're not familiar with. Like, maybe you're not comfortable with Haskell, maybe you're not comfortable with— maybe Haskell is like a little bit lower resource, but I like, I don't know, maybe not comfortable with C or something. Maybe you are, but pick a language. Like, how would you feel okay with like trying to debug? Like, hey, what's going on with this code completion? Like, why is it not, you know, working the way I want it to?

You can take this, Sean, or me.

I mean, I was surprised when we looked at the internal numbers and saw, you know, hey, we're running Harnesslib on JavaScript and Python and we're getting, you know, roughly similar acceptance rates for, you know, Haskell and, you know, Visual Basic or whatever people are writing in this thing. And it's like, it was really, encouraging to see that the evals that we were doing translated over to other languages that we weren't necessarily evaluating on or comfortable with. I'm not familiar with examples where we, we had to do a deep dive into, you know, one of those, those less familiar languages. Um, and I think a big part of that is like anytime you're trying to diagnose one of these problems, especially in production, you want to kind of go back and be like, okay, what's our baseline? You know, where is the point in the past where this worked? And so you're always kind of building on the eval runs that you've done in the past. And so if you don't have those evals for like, hey, how did Haskell work in Copilot a year ago, then it's kind of hard to diagnose that in the present. And so—

Okay, yeah, no, that makes sense. It sounds like the high-resource languages, Python, JavaScript, are a good proxy, sounds like, for even the low-resource ones.

Surprisingly so, yeah.

Um, okay, I'm gonna put you all a little bit on this, on the spot here with the question, but maybe it'll work out. What is like a surprising thing that y'all learned? Like, if you can think of any example that you learned from doing the evals, like, you know, that turned into an improvement of some kind, like, you know, that like something counterintuitive or something that you remember that sticks out in your mind?

I, well, just kind of go back to, uh, to Sean's part, honestly, with the Ellen judge. I really— this is well after the fact, but, uh, uh, in our conversation to, to put this talk together, I thought it was just neat to hear him walk through the learning process that necessitated LLM Judge. We, I mean, this process has been done at several places at this point, but what you kind of saw right then was the birth of LLM Judge. Like, this needs humans. Humans don't scale. Here's how, like, we break this down into something that, that is easier for humans. We, like, have it list its criteria so that we can digest it. Why not just turn that over to an LLM? It's easier to do. Like, true or false, that's going to be a lot easier to deal with than something that's subjective as a human. So I thought that was kind of a neat lesson that came out of everything.

Yeah, I think one of the things I remember as being kind of interesting is we started off with this harness. That was ripping the bodies out of functions and trying to synthesize them again. And also this intuition, like John was saying, of the more characters that are accepted and retained in the code, that that's a signal that we did something good. And I remember we kind of discovered over time that users were getting frustrated when Copilot was trying to write too much code for them at one time. And they would, you know, reluctantly accept it and then delete, you know, the last half of it and then get a better completion for the second half of it. And so I remember we did a number of experiments and changes to the prompt to actually generate shorter completions, or even just, you know, chop off the completion and let the user kind of naturally tab their way through the solution instead of trying to one-shot it. And so that was something that was kind of counter to our initial intuition of like Copilot should write as much code as possible for you, and then it should all be good quality code on the first try. And this was more like, no, people actually want to incrementally build up a good solution.

Makes sense. If y'all were going to build a product today, like some other kind of AI products, where would you still do evals? And like, where would you do it? Like, at what point would you start thinking about it? Let's say, if, if at all. I don't want to lead the answer.

Well, it is, it is interesting because I— the good, the good student answer is, well, you should have evals always from, from day one. And I bet Sean comes closer to that than I would. I think it's super important to have a valve if you have any type of ears product, but there is a point, there is a point in time to introduce them. A lot of, a lot of times when you're just really early on trying to figure out what the product does, good old vibe, vibe check is, is, is will get you a long way. You can say, is this working at all? Is, you know, what do I need to do with this? And I would be interested, honestly, Hamel Husain, in hearing your opinion as I go through your course here in a few weeks about like, what are the limits of, you know, how much evaluation you do? Because, you know, we've all seen companies that are like, we're supposed to have a VAL, then they go a little bit whole hog into that, and now they're in a VAL company, but they're not really measuring the right thing. So I feel like there's— it's fairly early on, but it's not at the very beginning. At some point in the beginning, you need to be just making prototypes and seeing what works.

Yeah, yeah, totally agree. Yeah, like I say, like, the purpose of building a VALS is not to have a VALS, it's to build products, right? And so that should still be the number one priority. Shouldn't not build the product because you can't think of the eval yet. And so you should be listening to your users and doing these A/B experiments to understand what your users want and then going out and building those products. But then thinking in the back of your mind, okay, eventually this product is going to be successful. How are we going to keep it from regressing? How are we going to evaluate it? But it shouldn't, shouldn't block you from doing something wild and crazy and different from your competitors.

That makes sense. I agree with that. And what we'll teach in the course is you should, you know, you should build evals after you've built your prototype. You don't need to start it too early. You need to have something first. And even while you're doing evals, like, if you notice an error in the process, like, you don't— there's a tension on, like, when do you write an eval and when you don't. It's a judgment call on if there's an ROI to writing that eval. Some errors you'll find like, okay, you know exactly how to fix it, just go fix it. Like, you know, don't necessarily need to write an eval for every single thing. But yeah, that makes sense. But kind of my experience.

I mean, on that Harness or on the A/B experiments, so many of our guardrail metrics, I believe, came out of, you know, things that broke in the past, right? It was like this accumulation of like, Oh, we tried something, it broke, someone spent a week digging into it and then found this metric after the fact that would've, you know, caught it earlier. And so a lot of that stuff you kind of only learn by building it.

I have a really fun question for you before we wrap up. That LLM as a judge prompt, do you think like that would be, that would make for good cursor rule like the one that you have? Uh, have you— it seems interesting, right? Like, if you have all these bulleted points of like, hey, this is good code, this is bad code, whatever, uh, it seems like the perfect prompt for yourself in a way.

Yeah, maybe. Um, I mean, it's— you always get to cheat a little bit when you're building your own eval set and kind of say like here's what I think programming is, and here's, you know, the tasks that are representative of what our users will be doing, and I'll only evaluate those tasks. That the scary thing whenever you're doing kind of especially like a chat-based interface, it's like it's just wide open and people can ask anything and do anything with it. And so, yeah, I, I would hope that the prompt that I did for LMS Judge would generalize to more just kind of general code quality. but I also got to kind of pick and choose which use cases I covered and tailor it to those.

Makes sense. I kind of want to see this. It's pretty obvious I want to see the prompt.

Nice try, Hamel.

Nice try.

Yeah.

Um, no, thank you so much. This is a really great conversation. It's, uh, really interesting. So thanks for coming on and teaching everybody about how y'all build evals.

Yeah, thank you for having us. This has been great.

Yeah.