LLM Eval Office Hours #2: LLM Observability

So I read your articles and you brought up some points about building a UI to make addressing these problems more tangible. You're able to access and explore what's wrong with your service with a lot more ease. I actually haven't built any of that and what I've basically been doing over the course of a couple projects, including the Sales Development Representative project, which I might have alluded to, is I just utilize, for example, I use FastAPI's API to determine, to explore whether or not my agent is performing, is doing what it's supposed to be doing. And it is like, it's definitely better than not having any visibility, but it's still a bit cumbersome to navigate. So I was curious, like, how much time do you— how much time do you spend into making these, uh, these UIs to have a better visual of, of how the performance of your system?

Yeah, um, it keeps getting shorter and shorter in the amount of time I need to spend making it. I think the first time it may have taken me like 3 or 4 days. Just to understand even if this is a good idea or what I'm— what am I doing, and like kind of figuring out what I even want. Um, but then nowadays it takes me, you know, maybe like 2 or 3 hours to, to do something like that. Um, and especially if you, if you know what— where the data is. I guess the, the time-consuming part is like querying the data from wherever you're storing it. And then materializing it in, in some front end. And you have to have some understanding of like the structure of your data and how you want to show it. Once you have some workflow that you like, like someplace you like storing your data, maybe it's, you know, LangSmith, Braintrust, uh, Horize, or maybe it's a SQLite database that you have, right? I don't know what you want, but You know, as long as you have a workflow that you like and you know that data, then it shouldn't take too long, especially— and you know, obviously you should be using AI-assisted tools to help you build these frameworks.

Yeah, right, um, fair. Yeah, it's just that I never really considered using a frontend, but I do recognize that it would make things a lot easier. It just seems like a, um, pretty massive It just appears like a pretty big undertaking, but if you just keep it simple, I guess it's not too bad.

No, just keep it really simple. So basically the idea is like you want to see all the information you need in one screen. You don't want to be jumping around to like this database here, some other screen here, your traces in one place. You need all the context in one place. You know, you know, bring it all together, put it on the screen. Doesn't have to be pretty or anything. It just— so you can read it and, and you can like flip through data really fast and maybe like label something or take some notes, do whatever. I mean, maybe you don't even need to take notes. Like, you kind of have some idea of what you need probably, and that you can just like quickly look at data to where the— like, the idea is looking at data shouldn't stress you out at all. You shouldn't think in your mind, oh, I need to look at data, it's going to be kind of— yeah, like, it's going to be painful. Should be like, oh, look at data, oh, that's really fast, I'm just going to open this app and I'm just going to like check out some stuff. And then, okay, I will say this, you— so what the first place to begin is not even building an app, is like, first, can you use a spreadsheet? That is the simplest the way to view data, you know, that exists.

Um, it's— I would rather actually build a frontend than use a spreadsheet. But, um, yeah, I mean, I can create a CSV file, I can create an Excel doc, but not my strength.

Sure. Um, so like, yeah, Excel can be super easy depending on your data. If your data is like a lot of multi-turn conversations or something, then like, then the spreadsheet might become unwieldy because you have a lot of stuff going on in each row, um, and you might want something else. Um, but yeah, and like their spreadsheet, okay, like you can walk up towards UI. There's, you know, so you can do spreadsheet, you can even do notebooks. So Jupyter Notebooks or whatever.

I don't know, what do you program in, like TypeScript or Python or TypeScript, TypeScript, Python. I don't do a lot of work on Colab and Jupyter Notebooks. Yeah, yeah.

So you can kind of like with Gradio and, you know, in notebooks, you can kind of make your own little like sort of UI thingy looking thing. And yeah, that's also good. It depends what you're comfortable with. Okay, what are we looking at? Open plugin.

So this is, this is one of the projects that I worked on. And one of the major problems that I ran into, and it was, um, very hard to, very hard to address, was that this, um, this entire project had a lot of dependencies, like a lot of chaining dependencies. One was you have an LLM that can fail at like inference time and failing to produce like the function calls, for example. Um, and then on top of that, like, the, the— so this is— have you ever used ChatGPT plugins?

Uh, a long time ago.

Yeah, yeah, yeah. So this is— that's when this basically existed. That's when this was relevant. But I think that the concept that I'm gonna get at, it's, it's still somewhat relevant. So the function call might fail, for example. Um, then after that the, the plugin that it's trying to like reach out to, which is kind of an unreliable third-party service, might fail as well. And then presenting that to the user might have failed as well. So there was like several chaining, um, areas of potential, several chaining pitfalls.

Got it.

Um, being able to like address that, I— what I— how I addressed it was just print logging as much as I could, and particularly like logging the errors whenever they could happen.

Let me tell you something that will help you. Uh, so if you have— are you using any framework, like, I don't know, like LangChain or something? Are you, are you rolling your own? I was just curious, like, um, no, um, okay, no problem. So what I would do is I would use some observability tools. There's a lot of them out there. It depends what you like. There's paid stuff. I mentioned some of this before like Braintrust, there's Langsmith, Arise has a paid one. There's an open source thing that I quite like, it's called Phoenix Arise. It's like the open source thing. It has a Python SDK. So if you don't like Python, then maybe Braintrust, you know, could be good as well. You know, like if you're into TypeScript. But basically you want to log the traces, you can instrument your code and it will— there's like very convenient ways to instrument your code such that it will follow the whole code path. Or the whole like LLM path, like from, you know, you'll be able to see the entire trace of what happened, um, like what the prompt was, what the function call was, what the response from the function was, everything. It's a nice way that it's kind of like standard in the industry, becoming a standard industry to like log your data this way. And once you log your data this way, now you will have a whole host of tools that will allow you to go further. So, um, if you log your data into one of these platforms, number one, you can easily pull it out and you can build your own UIs on top of it. You may have read in the blog post where I talk about building your own UIs. Like, I built a UI on top of LangSmith, for example, because I was working on LangSmith at the time I wrote that post.

Oh, isn't LangSmith itself a UI?

It is, but sometimes you want So the point of that post was like, you want to customize the way you look at your data for yourself. So like sometimes the data you push into here may not be in the format you want to see it, you know, because ultimately like this is not necessarily a data viewer. This is like more like a, it's like a Datadog for LLMs. This is a very bad analogy, but it's kind of the closest I can get. Okay, to help you to understand, it's like, so essentially, like, uh, you, you can push your data in here and it'll help you navigate and filter your data and, like, query that, those logs in a, in a very, like, structured way, if that makes sense. Um, and Langsmith, Langsmith provides, like, does Langsmith host the, um Yeah, all these tools like Braintrust and Langsmith, the, the paid ones are all hosted as far as I know. And but, uh, yeah, like, and it stores the data for you. Yeah, it stores the data for you. So if you may or may not like that, uh, the open source one, uh, Phoenix, which you were just looking at, that's like self-hosted if you want. That's it. Actually, like, host one for you too if you want, but you can host it yourself. Yeah, um, they're all like pretty— they're pretty easy to use. Depending on what you want, you can see already how you get an idea, how it makes it easier for you to look at your data. Yeah.

If I was able to configure this, that would have been a huge time saver.

Yeah. It's pretty easy to configure. A lot of these tools have auto-instrumentation. So if you're using something like the OpenAI SDK, which, you know, or whatever, or like the Cloud SDK or whatever, they will have like a wrapper around that that basically intercepts all the calls you're making and tries to instrument everything for you without you having to change that many lines of code.

Interesting. Okay. Um, yeah, so, so basically creating, creating a system that already like handles much of the like abstractions and just tell them.

So my advice is like you don't want to build a logging tool yourself. I think you can build your front end to look at data, that is worthwhile, but you don't want to build like the, like the logging and the observability yourself. I don't think, you know, that's kind of too, probably too far away. You probably want to focus on building your AI apps.

So yeah, that's basically what I was doing with OpenPlugin. Like, for example, whenever this happens, I, I just get— I receive a log, um, that is tagged with like failing. I don't remember specific details at this point, but yeah, not everything was basically just a Mongo— it was a MongoDB collection that just like logged everything. Um, and so I needed to build like additional, additional endpoints to like parse through this, which I think that, that could, could have been abstracted by tools like LangChain, LangSmith, and Phoenix, right?

Yeah, take a look at those. Um, you know, use one of them, um, that, you know, like whichever one looks like it kind of fits your needs the most based upon like the languages you use and your preferences around like open source and closed source or hosted and not hosted. I don't, you know, it's all a lot of stuff to navigate through, but like, you know, you get an idea. You can probably make a decision really fast, like, taking a look at these things. And, uh, yeah, yeah. And it's, it, it really helps. And it helps you not only like look at your data, you can build a lot of things on top of it because they have APIs and stuff. Um, but also help you when it comes time to write evals, because all of these tools have facilities for you to like write tests and execute those tests against data that are in those, in those databases. Like, you can curate those traces, you can like save them off as like test cases in a way, and you can do things like iterate on the prompt and like see what the outcome of that is. Um, you know, you can write tests, and there's— it's, it's, it kind of is very helpful, right?

And when you're saying iterating on the— iterating on the prompts of these test cases, you— do you mean like, does it actually perform inference after you submit the, uh—

yeah, yeah. So what you can do is like, you can— yeah, you can basically, um, use data from your traces as inputs to, uh, like inference. So like you can have like your prompt and you can say, okay, like take this dataset of these traces where the, you know, and use like the inputs that you supplied in those traces to this like other prompt, and you can test it out and you can see what the output is. And you could do that in parallel and you can like keep iterating really fast, just like kind of do They all have the concept of this like prompt playground thing. This basically has string templating that in the string templating pulls from the data that you save.

Are you describing a tool that you built or something?

No, no, that's in all of these things. If you adopt any of these tools that I'm talking about, they'll have that. So I'm just telling you that's a—

that's a—

yeah, all of these are specifically built for LLMs. Yeah, yeah, they're all about LLMs. So like they have all these other tools. Once you put your data in there, they try to help you with like a bunch of other things. So it's not only about a place to store, it's not just a glorified database is what I'm trying to tell you. It's like a little bit more.

Yeah, yeah, because that was one of the main issues that I was running into. Like when I was, whenever I was running this agent action, I would have it go from like zero to the end. And that just led to a lot of wasted, wasted time and money. So being able to just directly pinpoint a specific area is good. Yeah, so the last thing that I wanted to mention is something that I'm going to be working on now, which is this concept of basically generating your own actions and saving them in a database that can be accessed by any other agent. It's based on this paper, Dinosaur Beyond predefined actions. And one of the big things that I see, one of the big problems that I expect to run into here is that it is very— it should be like really unopinionated on how the users will end up using these actions. So I just, I don't know how I'm going to be able to effectively evaluate this when it's like so open-ended as to how our users can use this. Um, because it's basically— we're building an internal tool that's going to serve basically other agents.

Okay, so you're asking, you have this component in your LLM-based automation and you're wondering, like, because the scope of it is so large, you're wondering how you might evaluate it? Yes. Okay. The best way to evaluate something like that is to do something called error analysis. So error analysis, um, I can show you one of my posts. I can share my screen. Let me see if I can do that quick.

Do I need to stop sharing?

Uh, I don't— I'm not sure. I don't— uh, yeah, go ahead and stop sharing. I'll help with the recording. Um, so there's this post, Creating LMS to Judge That Drives Results. It's quite a long post, um, but there's a section in here called Perform Error Analysis. And so the main point is, hey, like, when you're doing evals, there's so many things you can eval, right? There's like so many components of your system, there's so many things that your system does Like, how do you even start? How do you even know what to eval?

And feature scenario persona.

Yeah. And so basically, um, it's— so what do you want to do is ground what you're testing in the errors that you see. So you kind of want to go a little bit in reverse and say like, okay, I can eval anything in the world under the sun, but you don't have time for that. So what are you gonna do is you're gonna go through and start looking at data and classify any problems that you see. So like, basically, so this is like a very, there's a video on it. This is a very old concept from machine learning. There's one, this video from Andrew Ng in the blog post.

It's in production that you're logging these errors or during, yeah, during development, I, I guess, obviously.

Okay. So there's two approaches. One is if you don't have production data, then you can generate synthetic data to try to simulate what a user might do. And this blog post will kind of give you an idea of how you can try to simulate that data. But you need some data to push it through, either like a— you need either need like fake humans or fake users or real users or whatever, or real, you know, real interactions or fake. It doesn't matter. Of course, real is better, but during development you may not have a choice. Mm-hmm. Um, but basically, like, what you want to do is say, okay, like, you want to go through— and it's very simple. Error analysis sounds like a very fancy term. It's really not fancy. It's like kind of dumb, which is good because it's very accessible. What you do is, like, you go through your data and you start categorizing, like, what are the problems you see. And you just— and you, like, maybe you look at, I don't know, 50 examples and you start like making a note. Okay, this is the problem I see, this is the problem I see. And then when you're done with the 50 examples, you just add it up and you kind of see, okay, like what is the— what are the largest problems I see? And you can like just do something like this. Like this is just a very toy example, like, okay, most of my problems are like around user education, or you know what, my internal tool has some authentication or access issue. And then you can just, just ground it in whatever the errors are. That's, that's what I would do.

Okay. And this is like, um, potentially something you could determine after running the, like, feature persona, uh, scenario ablations.

Yeah. And like, you know, don't, don't, uh, feel like you have to go take, you know, to follow this blog post so closely. I mean, do what makes sense for you. This is just like an example of like to get you started thinking about how you can simulate like your data. Maybe there's, maybe there's something that is not a feature scenario persona. Maybe there's like some other dimension that makes sense, right? But I was just trying to give people an example like, okay, this is how you can think about it, you know. When you—

yeah, the, the feature scenario persona is something that really stick with like just getting yourself thinking about the different use cases of how a program might be used. Yeah, it seems pretty useful. I'm definitely going to be just playing around with that.

Yeah, I mean, but certainly start with your— start— I mean, you should be using it too, like this thing. You should be trying to use it yourself, and hopefully you are generating some data yourself as you use your own tools. Yeah, you should certainly start there. As some intuition. So, um, like these blog posts that I have, like they say, okay, you can't— you, you shouldn't, uh, rely on vibe checks, but you should certainly— you should start with some vibe checks. You just can't— that will only take you so far, and at some point you will need to do this kind of simulation and generate data, whatever, if you're trying to make it robust before you, let's say, release it to the outside world, you know. But before that, like, it is good, you should be using yourself and seeing what it's doing. And then, you know, that's really important.

Yeah, yeah. What I was— the experience I had with the sales development representative agent was, um, that it— everything I was doing was just basically vibe checks. That's how I was evaluating everything. And just over time, gradually, I ended up building out some sort of just like internal API, internal endpoints to better be able to understand things and to save myself money and time. But I feel like maybe if I used a tool that was, you know, designed for that, it might have saved me a good amount of time. But at that point, I don't know, like, when, when should, when should I like switch over to this tool as opposed to just building my own tooling? Like, what's the threshold for—

yeah, it's hard to know that. It's hard to know that. I would say like the logging stuff, you definitely want to go to your tools. Uh, you don't definitely want to use tools, but then you definitely want to build tools to look at data. It's actually, um, a presentation that you may find helpful. Um, let me share my screen again. It's like, uh, Let me see if I can find it. So yeah, if you Google Hamel AI engineer evals, there's this short presentation. It's like 15 minutes or so that kind of like I talk a lot about, okay, like what tools to use, like what to buy, what to build, how to think about evals. It sounds like, I don't know, I have an intuition that you might find it useful, like this specific video. Um, you know, I'll talk about like different kinds of tests. Um, you know, like I talk about logging your traces, kind of like what we just talked about. And I'll say like, you know, like look at your traces but build your tools, like, you know, in this case if you need it.

Um, yeah, as I'm like— as I, as I read through your article and as you're like sharing these, these ideas, I just constantly keep thinking like There's so many opportunities to just automate this work and create workflows between, for example, yourself and your expert, domain expert, to just better analyze LLMs. It's so tempting to just go immerse yourself in building these additional tools as opposed to focusing on the product itself.

Yeah, it is tempting. I will say there's a lot of people building tools in that space. We are not the only— it is an extremely crowded area. Basically everyone. Yeah, it has occurred to a lot of people. Let's put it that way.

Yeah. Okay. Yeah, that's a good amount of resources to work with and ideas to play around with. And, um, again, I appreciate, I appreciate your time. And if it's okay, I'll, I'll let you know how things are going.

Yeah, please send me an email and let me know, or get in touch and let me know how it goes. Really appreciate it.

Yeah, I appreciate it, Hamel.