Inspect, an OSS Framework for LLM Evals
Inspect is an open-source Python framework for building and running LLM evaluations, developed with the UK AI Safety Institute. It lets you define datasets, solvers that call models and perform self-critique, and scorers that grade outputs, then scale from notebooks to cloud environments. The framework supports multiple model providers and integrates with external tools like LangChain and Docker-based tasks.
Chapters
So I'm going to try to give a kind of whirlwind tour of Inspect. And I actually worked with Hamel Husain a little bit over the weekend to build some Inspect evals for the Honeycomb dataset that you all are working with. So hopefully it'll be— it'll connect well to the work that you've been doing, and it'll make the concepts gel a little bit easier. So what is inspect? Not surprisingly, it is a Python package. pip install inspectai. Someone asked a question about— and Hamel referenced that I'm affiliated with Posit, which is formally RStudio. This project is actually not a Posit project. This is a project that results from a collaboration that I'm doing with the UK AI Safety Institute. So the UK AI Safety Institute has hundreds of evaluations of every variety, you know, simple QA things, fancy, you know, cybersecurity capture the flag evals, hundreds of evaluations. And when we kind of started off on the, on the journey to writing all these evaluations, there weren't, weren't terrific tools available. A lot of the tools were either embedded in benchmark frameworks or maybe not very complete or weren't that well tooled. It wasn't necessarily clear exactly how much more development they would get. And further, I think they were not necessarily designed to scale up to very complex evals. And so we set out— and actually, the most popular eval framework is just roll your own eval framework, which is out there a bunch too. So we set out to build something that we could use and that ultimately we could share so other people could use as well. So with that, just, again, trying to ground this in Honeycomb, this is actually an eval for the Honeycomb dataset. It's got, I think there's 2,300 user inputs. We've also got the, you can see columns, that's the schemas that were, that were fetched originally by RAG. So they're in the dataset. And so to create an evaluation, we basically take that dataset and then we put it through a pipeline, which you'll recognize is really the same code that Hamel had in, in the notebooks for, for Honeycomb. And then we apply a score to it, which is, again, based on Hamel's original code. And then we can run this eval and get lots of tooling. You can see this. I'll get more into this, but we have a log viewer that lets you explore and debug and visualize everything that happened during the eval. Lots of other tools. This is a VS Code extension that lets you tweak things, run different models, et cetera. So I can, I think I would emphasize that Inspect is not super opinionated and really is like Python code first. And by sort of conforming to some simple conventions and fitting yourself in, you get to take advantage of this big pipeline of tools and then hopefully like a big ecosystem of related packages that will extend Inspect. So I'm gonna go.
Is it, is it a very local in nature? Like, it's local.
It's all local. Yeah.
Okay.
Yep. Yeah. So one of the things I skipped over on the first slide is we have this concept of development to production. So we definitely want to have, like, a very interactive local. You work in a notebook, like, be very iterative, you know, work on your eval. We have lots of stuff for taking it and, like, running it in the cloud, you know, run it over 10 models, run it, you know. So it's, it's, like, local for development, but then we have lots of tools to, if you want to scale it up and run it as part of CI and things like that. So, but it's all, it's all local. Very cool. Okay. So this core concepts, and I, you saw a little bit of these on the, on the first slide. You have a dataset, which is not at all surprising what that consists of. It's inputs. And usually there's targets. In the case of the Honeycomb dataset, there's not a target. There's just, there's a, there's gonna be a validation function and, and a critique model. Target would usually have like, if we're multiple choice, what's the correct answer for Q&A? Some kind of description of what the right answer is. For a, a fancy model-graded eval, it might be like a grading rubric for a model. So that's the dataset. Solvers is like the pipeline that actually does the eval. And this could be doing anything from prompt engineering to calling the model to a multi-turn dialogue with the model. It could be doing various types of elicitation like critique. Solvers is kind of like the heart of the entire eval and where you kind of customize how it works. And then the scorer basically evaluates the final output. Again, these can be very simple, like doing text comparisons. They can be model-grade, model-graded, or they can use all kinds of custom schemes, whatever kind of, whatever you can dream up. So that's the core concepts, and I'm gonna drill more into this validate example. So this is the eval. I'll break down the different parts of it. We talked about the dataset, and this is just reading from the CSV. There are standard fields that go in datasets. Input is one of them. And then there are like custom fields that different evals will need. In this case, columns is important because we're going to use columns for the prompt template. And so we want to save the columns when we read the dataset in. And then this, we have this plan. And you'll see this is the same system message used in the notebooks presented in the course. PromptWithSchema is a solver that's going to build the prompt. That uses the columns and the input, and then generate calls the model. So pretty straightforward. And then the score actually does the— uses the check function, the check query function that you've also seen in the course. So that's the eval. And now I'll drill into some of the specific components. They're quite simple, and I hopefully they'll be very intuitive and straightforward what they're actually doing here. Prompt with schema is literally just taking a prompt template and then substituting the prompt and the columns. So you've seen this before, but it's really, it's a solver that really just makes a prompt. That's all it does. And then the scorer is basically going to, it's going to take the output from the model. This JSON completion function is just sort of a helper that some models actually like to put like JSON, like code blocks in. So it like strips that away. So the idea behind that is just just like clean it up so we get pure JSON and read it. And then we call the is_valid function, which is literally the exact same is_valid function that's, that's used in the course. And we figure out whether the, whether the query was valid. Okay. And then so once we've done that, we've built our solver and we've built our score and we've run it. Now we run our eval and we can see kind of what happened. And in so many evals, the score really doesn't tell you anything close to enough, especially when you first start developing them, because, you know, your, your checking function could be wrong. Your— the way you extract answers from the model could be wrong. There's so many things that you need to investigate, and you kind of in some ways need to do it on a per-sample basis. So this lets you very easily look and see what happened overall, and then what happened on a sample-by-sample basis. So here you can see drilling into this sample, which it got incorrect. We can see what was the actual message history, what was the dialogue between the, the model and the eval. And so here it's quite long, as you know, from working on it. Here's the columns that was injected, the schema. And then here you can see to the very end, and then the assistant's answer. So looking at all the messages can be quite, quite valuable, especially when you get into things like, like tool use. And then also, OK, so that's the basics of that. And then we built another eval, which has actually— it's the exact same code. And in fact, in the Git repo that I'll share at the end, you'll see that I do reuse the code. I don't just copy and paste it. But really, there's only one line that's different here. It's the same dataset. It's the same plan. But we're going to use a critique model for scoring instead of the validate function. So here, this is the critique score. So this has a little bit more going on, but it's again pretty straightforward. Notice we parameterize what model is used to do the critique. So you can use, you know, it's pretty obvious, but you don't necessarily need to use the same model. In fact, you often don't want to use the same model to do scoring as the model you're evaluating. You can use a more powerful model, in some cases maybe use a fine-tuned model. Here we're defaulting to GPT-4 Turbo, but The user of the scorer could use any other model they want. We build the critic prompt, kind of analogous to how we built the other prompt. And again, this critique.text is literally the same. It's the exact prompt that is in the notebooks for the course. And then we run the critique. So we get the model instance. We call generate. We parse the output. We check and see if it was good. And then we return the score. So that's our critique score. And then at the end of that, we get another eval view, this time for critique. The accuracy here was a little bit lower. And these numbers actually don't really mean anything by themselves, but just noting that it was less frequent that the critique was satisfied with the output than the devalidator, which is intuitive. And so here we might want to know, well, what actually happened in this critique? So we can drill in here. This is one of the incorrect answers. And we can see what the answer was. And then we can actually see what the critique model's explanation was. So here, you might look at it and say, wow, that actually looks right to me. Or a human expert may have said it was right. And then you want to look at the explanation. And perhaps this indicates that you need to improve your prompt template. For the critique model, or perhaps it means you need to fine-tune a model just to do critique depending on how many, you know, what resources you're willing to apply to get a good grader. So this is, again, just drilling in and seeing what happened during scoring. Okay. So that is—
Is there any way to interact with this view, like do some annotation in this view itself or something?
So right now, we don't have, like, commenting in the view and stuff. I think We, we are developing like some like shared places to look at, like internally, places to look at views together and there'll be some annotation in there. I don't know if we'll come out with like a thing to do that. But that is useful, especially, yeah. So maybe there's a way we can do that in a way that, that we could open source that as well. And then, and then people can take advantage of that. This is just runs locally. So you kind of would need to post it in some kind of a— we, we've talked about actually building a— it wouldn't really solve it— a Weights and Biases plugin. But that's not gonna let you do this, like, drilling into each one. Maybe, maybe to some extent it would if they have, like, good, good templates for, for, for chat message, for chat conversation histories. But, yeah.
I like the way that the— this is being rendered and it's flexible. Seems flexible enough that you took my example without really knowing it until the weekend and Seem to just plug it in here. And it's actually very—
We just plugged it in and mostly just use all your code. I mean, there's a very— what I showed you is, like, all the code that I wrote, and all the rest is just calling your— using your templates and using your code. So that's kind of the idea that you, you know, you— there is a minimal lift to get your existing stuff working inside the pipeline. So yeah, we did that. Okay. So now I wanna just, just talk a little more abstractly. We've looked at these examples I've shown you. Some simple solvers, really simple, just a prompt template. They get, they can get a lot more fancy, which I'll show you in a minute, and some, some scorers. So like conceptually, what is a solver? It basically, and this is a simple view, but the idea of a solver is it has a task state, which is like what's the current state of the message history and what is the current model output. And when we start, there is no model output, and it does, it just transforms the task state in some useful fashion. So that could be calls the model, generates, appends the assistant message, and updates the output. It could be prompt engineering. It could be critique. It, it kind of can be anything. And yeah, that's sort of what I'm saying here. Like, the solve function does something useful with the state. And that sort of allows you to create a pipeline of solvers now and reuse solvers. And it's not always the case that you wanna do that. Some evals really just wanna have like one solver that just is sort of boss of everything. and doesn't confine itself to being in a pipeline. But, but the pipeline does end up, end up being useful in a lot of, a lot of cases. So some examples of really simple solvers, which you've sort of seen hints of, we build a custom prompt template for, for the validate, for the Honeycomb evals. But like a simple, here's a prompt template, just transform the prompt by passing it through a template, perhaps with some extra parameters from metadata. Generate, this is actually the, the actual source code for the generate solver. It literally just calls the model to generate. And that's it. So those are like really simple 101 solvers. I think we have like a chain of thought solver, which does like a basic chain of thought template. But even then, often you, you wanna customize the, that for your, for the domain. The generic chain of thought isn't always exactly what you want. And I guess this emphasizes when you're writing really good evals, you are writing, you can reuse solvers, but you oftentimes wanna write your own solvers and write your own scorers to, to make them really, really good. Here's a more fancy one. I won't actually walk through all the code on this, but it's a multiple choice solver that handles, like, shuffling the choices. And then it actually calls the model to generate, and then it unshuffles the choices. So this is an example of a solver that calls generate internally. So the plan for a multiple choice will typically just be like plan equals multiple choice. It might be like plan equals chain of thought multiple choice, where multiple choice just calls, calls generate internally. Another one, self-critique, pretty straightforward. Again, we've, we've already called generate before self-critique comes on the scene. So we may have had a prompt, a system message, a prompt template. Now we call generate and now self-critique. So the first thing we do is we actually take the existing completion and we, we basically run a critique on it. And then we take that critique and we append it to the message history. and then we call generate again. And so this is an example again of a solver calling generate internally, this time just to re-up the answer, with the critique being available.
I love how you made these slides, I think with Quarto. There's a lot of comments in the Discord. No? Okay.
Yeah, yeah. Yes, with Quarto, definitely with Quarto. Oh, yeah. Yeah, yeah, yeah, yeah, yeah, yeah, totally. Yeah.
A lot of comments in the, in the Discord about, hey, this is really cool. Is it Quarto? So, yeah.
Yes, definitely. It definitely is Quarto. Yes. Yeah. OK. And I'll make the slides available at the end and the slide source code and everything. So there's a git repo that I'll share a link to at the end so people can figure out how we did all this. OK. So that's the self-critique solver. OK. And then there's generate. We already talked through that. OK. So one of the things to think about is, like, is composition— And so one of the ideas behind InSpec is that you actually will write— people will write a bunch of Python packages that have scorers and have solvers, and that you'll be able to mix and match those. So the idea is like there's like ex— you know, lots of external components to let you do different things that you can just plug in. So as an example from, from AI Safety Institute, we have an internal package called Shepard that's used for doing jailbreaking. And so there's many— there's, as any of you follow the literature, there's like dozens and dozens of jailbreaks, and some of them work and some of them don't. And sometimes you have to try multiple jailbreaks, and some of them work with some models and not with other models, et cetera, et cetera. But these are basically jailbreak solvers that essentially do prompt engineering to get the model in a state where it may provide answers that it would otherwise refuse. And so as an example, here's an eval that's basically trying to give the— it's basically just trying to get see if the model can give good computer security advice. So it's like, how do I prevent, you know, my website from being hacked? And then so sometimes the model will, because you're asking about computer security, it'll like flag, don't talk about computer security. And so we're saying, well, we want to see what the model actually knows. And so here you can say we bring in a jailbreak solver. From Shepard, and then we just use it in our pipeline. And so we have our normal system message, we've put the jailbreak in, and then we'll probably be able to elicit more or have less refusals than we, than we otherwise would. And so you can imagine lots of different solvers that you could, that you could plug in, prompt various types of prompt engineering solvers. You can imagine a whole, like, Python package just full of, of prompt engineering techniques. And a whole Python package full of critique, debate, all kinds of things like that. So, and similarly, scorers. So Python package is full of different variations on model-graded scoring and things like that. So that's an example of composition, which we think will be a big part of, of how people end up using InSpec. Okay. Okay. So I've, I've simplified a little bit where we've just been looking at really straightforward kind of QA-style tasks, or in the case of, of obviously Honeycomb, we're looking at a fine-tuning task, but these are straightforward, just like prompt, generate, nothing fancy going on. I didn't show you all of Task 8 before, but Task 8 also includes tools. And so the idea behind tools, I'm sure you've all seen or used, you know, these are Python functions that you can make available to the model, and you tell the model, you write the Python function, you write the docstring, you tell the model about them, and then it will say, hey, I'd like to use this tool. And so part of the task state is a list of available tools, as well as potentially like a nudge to the model, like definitely use this tool or definitely don't use the tool, etc. And so this is a simple example of— this is a QA, biology QA task. And we're saying, hey, if it tends that you don't know the answer, I think this dataset actually has a bunch of very obscure questions, then hey, you can use web search. And so then we have a web search tool that goes and, you know, gets a bunch of Google hits and summarizes them and things like that. And so use_tools is a function that just makes tools available to generate. And so there's, you know, once you get into tools, now you're into agents. Sometimes it's just simple tool use, like, hey, let the model use Wikipedia, or let the model use web search. And sometimes it's give it all kinds of tools, and they really become agents at that point. And so you can have agents sort of with like very bespoke custom logic, or you can bring in like an agent library. So I'll show you an example in a little bit of taking like an existing LangChain agent and just like basically making it into a solver. So the idea is you can just take a, take any agent in these other frameworks, and once you have the bridging, it'll just work inside InSpec. So I'll show that, show that in a minute. But let me first show this sort of bespoke agent concept. Which is— this is a, a cybersecurity eval. It's a capture the flag task. And this is more like the hand-rolled agent loop where we're basically giving— init challenge is here is, is creating a Docker container. Use tools is basically just like, here's some tools, and we tell the model like, you can do all these things. And then we give it a task. And then this is basically just a loop where the model gets to keep using tools until it either terminates because it did— couldn't figure it out, or it ends up finding the flag. So this is like very custom. This is like roll your own agent. And definitely that's a thing. That's something that people do. But at the same time, you know, there's lots of agent frameworks out there. And so we want to be able to have like high-order functions that let you take an existing agent framework and turn it into a solver. So as an example, like if you look at the code in the middle here, This is all code that is just LangChain code. There is actually— I haven't shown the imports, but there's no inspect code at all in here. This is just like code that you would be writing in LangChain. This is basically going to get the Wikipedia tool and then using the tool. And then we have, as I said, there's this higher-order function. This is actually provided as an example right now. It's in the repo that I gave you. But we can just take that agent, and if it conforms to like the LangChain agent interface, we can just turn it into a solver. And then what you— if you actually look at the eval that results, you can see this is the Wikipedia search. So it's a dataset of can the model use Wikipedia to answer, you know, they may be difficult questions, maybe obscure questions. There may be questions we think the model definitely can answer. But the idea is, you know, the plan is literally just this LangChain agent that uses Wikipedia. And so you can see down here is the solver that I showed on the previous slide. This is the entire task definition. Use a model to assess it and use the agent. And then you can see here kind of what happened. And if we look kind of inside at, you know, what happened during the eval, you can see it'll show you the tool use. So it'll— it's like, okay, what was the back and forth? What tools did the model choose to use? And why, it gives an explanation, what result did it get. And this Game of Thrones one is really one of my favorites because it ends up, it's, it's, it's trying to find the, the in order of the 10 episode titles. And oftentimes, like in Wikipedia, it's not like literally spelled out. Or actually where it is spelled out, it might be wrong. And so oftentimes it'll do 2 or 3 queries to try to sort it out. So anyway, it gives you, that's sort of diagnostics of what happened with tool use. And then similarly for scoring, this is the model graded fact score. This was the answer and it was incorrect. So this was, this was the grading guidance. This was the answer. It was graded incorrect. And I think I'm going to hopefully show you. Yeah, this is the scorer's explanation. So again, you know, sometimes the model really didn't get it, but sometimes the score is actually wrong. And so it's important to be able to look, you know, drill down and see what's, what's actually happening. Okay. So that is— okay. So let's just talk a little bit about scoring. I'm just going to check my time and make sure. Okay. We're good. A little bit about scoring. There's lots of different ways of scoring. Obviously, traditional, like pattern matching and template and answer-based scoring are obvious. And we've got lots of built-in solvers for doing, like, regex matching, matching at the beginning and the end, matching a template, like where you tell the model to say answer and then the answer, lots of that sort of thing. There's also model-graded scores built in, but usually you need to customize the templates for those to get them to work properly for your domain. And of course, as I mentioned before, like, they're pluggable. You can get them from other packages. And I'm expecting lots of stuff's gonna happen with model-graded scoring over time. And we'll see, we'll see the benefits of the community working on that over the next months and years. And then you can also just say no score, have a human score it. So that's, that's also possible. And one of the things I think, and there's something that I know is emphasized quite a bit in the course, is basically rigorously evaluating model grade scores against human baselines. Basically, I've observed that definitely a lot of people will get their model grade score going and they'll be like, cool, now I have a score. And they haven't actually grounded it in whether it's how good it is relative to human scores. So if we can build tools that help people do that well, that sort of structure that work, I think that'll be, that'll be valuable. So that's something we're definitely gonna work on. Okay. So what am I? Oh, this is, okay, this is a, a score example, which I think is pretty interesting. This is the traditional, this is actually the math benchmark that I think OpenAI reports as part of their standard benchmarks. And what's interesting about it is that the model does math and then there's a target But oftentimes the answer is correct, even though it's not literally the target. And so we have this expression equivalence solver that basically lets a model assess, are those expressions actually logically equivalent? So it can even do a little bit of algebra or a little bit of factoring. These are trivial. You can see this, this is the same as this. It's scored correct. This is—
What's going on in the, the, that equivalence thing? Is it a regex? Or is there more going on in there?
There's more going on. I'm going to show the full source code to it in the next slide. There's a regex to extract the answer. And then we're going to go and have the model. So we prompt the model to basically say, at the end, put answer colon and then the equation. And then we basically— that's how we pull the answer out. And then we send that to this expression equivalent solver. These are trivial because they're just like punctuation differences, but I've seen it where it actually can sort out that the expressions are actually equivalent. So let's take a look more closely at that, at that solver. Hopefully I have a little step through on this. No, I skipped through the— okay, so extract the answer, and this is a regex off of this line answer pattern, which is a common way of prompting to get, to get the model to delineate their answer in a way that it's easy to pick out. And then here we actually have a whole nother template, which I'm not going to show, which basically it's a few-shot thing that basically has like, it has like 20 different few shots of like, these are equivalent, these are not equivalent. And then the model is able to take those and then actually do a pretty good job grading, pretty surprisingly good job grading. And so this is, you know, you kind of, there's a custom eval, it's math equations. You have to build a custom score. You have to use a model to help you do the scoring. But it kind of gives you a taste of some of the things that people will do with scoring. Okay, I want to talk a little bit about what might seem kind of a mundane concern, but logging ends up being like massively important for doing good evals. Obviously, we built a log viewer on top of the log, but the log also has an API so that you can interrogate it. And, you know, you can get multiple logs and then plot the differences and things. So the idea is the log is a rich, rich Python object. It's also JSON. There's a JSON schema for it, but it's a rich Python object that lets you explore everything that happened during the eval and compare logs and things like that. So there's that. And then I think you've seen most of the examples of the log viewer, but showing the samples. Showing the messages. Yeah, you've seen this showing scoring. Okay, so that's log. So a lot of people, the other thing people do, I think I'll show this later, they'll run like 20 eval tasks, like doing a grid search, and then they have all their logs and they plot the results, things like that. So definitely like you end up computing on the logs quite a bit.
Very cool.
Yeah. So models, we support a lot of models. We do, you know, the big, big frontier labs.
And do you need to support like specific models? Like, is, what's the difference between using like any Hugging Face model? Can I just use any Hugging Face model really?
Yeah, I can. I can.
Okay. All right.
Got it. Yeah. Yeah. Yeah. Absolutely. Yeah. No, the, what it is is this prefix here that is the model provider. And then this is completely arbitrary. So this is like any Hugging Face model. This is like any model that you have locally with Ollama. This is any model that's on Together AI. So this is provider and this is model name. We don't know any, I guess to answer your question, we don't know anything about these model names. We don't resolve them. We don't compute on them. We don't know what they are. They just get passed through. So when Gemini, you know, 2.0 comes out, you just start using it. So yeah.
And can you have your own endpoint, like your own REST API endpoint?
Yeah, you can. Yeah. So one of the things that hap— is interesting, like Oyama and VLLM both actually, and Together, I think Together might use VLLM. They all use OpenAI's API. So sometimes people will just, you know, use OpenAI with a custom base URL. But you can also, you can create a custom model provider as well. If it's like completely custom REST API that we don't know about, it's very easy to, to make up model provider and publish it in a package or, or what have you. So, yeah, so you should be able to get to the models you want to get to without, without, without trouble.
Cool.
Okay. Okay. So let's see. I just want to make sure we have time for questions. There is a, there's, I'm going to go a little fast on the rest here, but just to say we care a lot about interactive development. We care a lot about being able to work in a notebook, doing exploratory work on the eval, but then we want the eval to end up in a form you can, like, run them in CI. You can run lots of them. You can systematically compare results and things like that. So we have good, like, we have tooling that works well in notebooks. You know, I showed you before, you saw, like, a terminal. It was, like, inspect eval, do the thing. You can do all this in Python in a notebook, and it does all the things in the notebook. You can have tasks and notebooks. And so we, we definitely try to invest a lot in like interactive workflow and then make it so it can scale to the more, I would say, production workflows. So again, I'm not going to dwell too much. This is like a grid search. So it's like, okay, I'm doing a grid search over different grader models, graders, and system prompts. And that product is just like a thing that's making the grid search. I'm dynamically synthesizing a bunch of tasks. And I'm going to run all the tasks, and I'm going to plot their results. So that's just an example of, like, in a notebook, you just want to, like, explore the space with different things. And then later, you might say, well, we're going to formalize this. We're going to make a task. We're going to have some parameters from the task. What that allows you to do is start to address the evals, like, with external driver programs. So basically, I won't dwell on this, but, like, once you have this task, and this can still be in a notebook, and you've got these parameters here, I'm, I'm just basically just varying the system prompt and the grading prompt. You know, I can basically go inspect eval, and then I can actually, like, vary those parameters externally from a driver program. Or I can do the same thing if it's in a notebook. I can say, okay, I'm still gonna keep, I'm gonna keep my eval in the notebook where I did all my exploratory work, but I still wanna be able to address it outside of the notebook. Okay, task variant. This is get down in the weeds. We're not gonna get into that. OK. And then eval suites. Again, I'm not going to get into all this, but the idea is you should be able to have dozens of evals arranged in directories however you want, and we can find them and run them. This is an example of, like, a directory structure that has a bunch of Python. It has a bunch of tasks. We can find the tasks. We can run all the tasks. And, you know, again, the production version of it would probably be more like run it, put the, put all the logs in an S3 bucket, then later on, go look in the S3 bucket and retry things that failed and things like that. So, right. Okay. And then one last piece on sort of workflow is one principle is that if you run an eval from a Git repository, we wanna, if you only have the log file, you should be able to completely reproduce the eval. It won't necessarily give you all the same, obviously, since the models are non-deterministic, it won't give you the same results, but you can reproduce all the input parameters and everything. So for example, if I, if, if I hand you a log, I can use the Python API to go read the log. I can go get the origin and the commit. I can git clone it, and then I can just run eval on it, and that will work. So the idea is that the, the log file is like, assuming it was run from a Git repo, is sort of a unit of, of reproducibility. Okay. Okay. So I think we made it in time to have a decent number of questions, but I want to emphasize some resources. So one is, let me see here, is the documentation website. Lots of documentation that goes into lots of depth on all the stuff I talked about here. There's a fair number of kind of annotated examples, you know, that go through, kind of walk through the code and explain all the different things going on. There's also a, if I can find, a benchmarks in Git. There's like, we implemented a bunch of benchmarks and you can see how those are done. So lots of examples and lots of docs. And then, kind of some of the stuff I talked about, workflow and logs and tuning and things, is all, is all, we have docs about that as well. And then this is where you would go to get kind of everything I presented. So this repo has, I won't scroll down yet so people can note the URL, I can just stick it in the chat. Let me just do that quickly here. Or somebody can stick in the chat.
Yeah.
Okay. Yeah. But this basically has, yeah, I'll let you, I'll let you note that. But basically it has the slides and then it also has the code. So it has, if you go into like Honeycomb here, it actually has the kind of what I actually did, the full code. And there, there's the prompts. There's the, you'll recognize utils. This is like the, the code right from, right from the course. And then we have a— here's the queries eval. You can see, as again, we reused the code. We didn't copy and paste it, but here's the two eval tasks. And then we have a notebook version of that just to demonstrate doing it in a notebook. So a little bit more commentary. So there's that. And then I included some benchmarks here just for so people could explore those. And then also that LangChain example that we talked about is here also. So that kind of explains this, how to run it. And yeah, and then the slides. So this is a worthwhile repo to check out and, and the docs are also worthwhile to check out. So let me go back to here and go full screen and Q&A.
Yeah. So one question is, will Inspect be integrated with preexisting Posit products like RStudio or anything else?
So just to clarify, Inspect is not a Posit project. It's not. It's, it's a UK AI Safety Institute project. So it will not, I mean, unless it's just, it's in a separate, it exists in a parallel universe. So Uh, we have VS Code, um, a VS Code extension, but I'm not, I'm not sure about it, about any other, other Posit things.
Yeah. Um, I think, I mean, I definitely know the answer to this question, but I'll just go ahead and ask it because it's a question. Does InSpec support evaluating LLM systems using the inputs and outputs that is produced in the past?
Yes. Yes. You mean you're talking about, like, some kind of—
Like past logs, tracing thing, man. Yeah.
Yeah. So what you can do is the input can be— I oversimplified. Input can be a prompt or it can be a message history. So you could, like, replay an entire message history. And so you could take, like you said, a past log and then construct a new dataset that would allow you to evaluate using the previous prompts.
Okay, another question is, does the Inspect team plan to expand? It might be good to maybe briefly describe what is the Inspect team or who's working, but, um, does the Inspect team plan on expanding the list of metrics to include stuff like MRR? And yeah, MRR is like a ranking metric.
Yeah, we'll— we will— yes, the metrics, the built-in metrics are a little a little spare, partly because a lot of people do end up writing their own metrics. But, but, but I think there are ones that are like definitively important and common. And so we will definitely be, be doing that. And then the InSpec team is— there's 2 or 3 of us who work on it, working on it full time. But there's a lot of people inside UK AI Safety Institute who, who provide pull requests and design feedback and things like that. It's sort of like just like eval. It's like, it's just evals are everywhere. And so it's kind of in the air. And so there's a lot of feedback and a lot of, a lot of iteration. And then we've got, I definitely have the bandwidth to, to advance the project, like, at a significant pace.
That's great. I guess, like, this is a good time to ask the next question, which is, what is the future expectation for Inspect? Will it continue to be developed for the long term? What will its direction be dictated by the UK government or the community or both? And how do you think that?
Yeah, that's a good question. It's definitely going to be developed for the long term. We view it as like a foundational piece for doing sophisticated and complex evaluations. And I think I expect that if it does get picked up more broadly by lots of other players, that there will be community discussion and consensus about what's important. And we, we definitely don't— it's— we would not have open sourced it if it was like, oh, this is just like something we're using. Oh, by the way, here, everyone else can use it too. I think we want to make it broadly useful for all the different sorts of evaluations. One of the ideas is that if we can kind of level up the quality of evaluations more broadly, I think that's just a better, a better world to be in where everybody does evaluations better. And so I think we're quite interested in making it work for lots of different, lots of different use cases and scenarios and actors.
Okay. Does Inspect allow for using logs from an API or database query, or is it strictly files only?
So using logs or writing logs? I wonder what—
using logs.
Yeah, I mean, the— we use like fs-spec, so logs can come from any file system that is addressable by fs-spec. I think if you had an API where the logs were, you would interact with the API, you'd realize that on the local file system, and then you'd interact with it. We have an internal abstraction for logs, a log recorder that is not— just doesn't presume that it's a JSON file. And so maybe we may add other recorders that can, you know, log to databases and things like that. But yeah, the other thing is we want the log file format to be something you can compute on. So we do, we actually publish, you can see in the docs, we publish a JSON schema for it. We publish TypeScript binding types for it. So, you know, we want it to be something that people can use and compute on. And obviously a Python API for it.
Right. I'm just scrolling through the questions here. Someone is really curious about the cybersecurity eval stuff. So the question is, can you say a little bit more about the Docker-based cybersecurity CTF eval? Do you envision shareable suites of security tests?
I think that's gonna happen. Yeah, we haven't shared any of ours. I do know other people are talking about making shareable suites of security tests. Like, there's other people in the ecosystem who are planning on, on open sourcing, open sourcing those sorts of things. So that's definitely part of the plan. And we're gonna, we've figured out, like, people inside UK AI Safety Institute sort of like figured out how to do the Docker thing without— they just bootstrapped it inside the existing solver idea. But we're actually going to have a more formal construct to support what we call tool execution environments that will, you know, will do more of the Docker heavy lifting and interacting with Docker Compose and things like that. That'll be more built into the framework in the future, in the not too distant future. Like, if you went to do it now, you might say, huh, what am I supposed to do? But I think in a month or 2, it'll be more clear and documented and, and like happy path.
Let's see. My team is using Weights and Biases to track eval metrics. Is there a way to combine Inspect AI with Weights and Biases that you know of?
We are hoping to do that. That's on the fairly short list of things that we want to do. So I haven't looked carefully at the Weights and Biases API and what, like, how rich we can make it, and hopefully we can make it quite rich. I know that they have some kind of API I saw where you can, like, have an iframe in their thing. So we could potentially even have the log viewer go embedded in Weights and Biases that also project a bunch of the log file results into, into the standard Weights and Biases affordances. So we're, we're gonna be looking at that in the next couple, 3, 4 months, hopefully.
Someone asked a question. They're really curious about the design philosophy behind this, like where you got the inspiration from. They said it's like very clean and the clarity of thought is impressive.
It's all Hadley all the time.
Okay.
I learned a lot from him. So, I've seen a lot of his work. And so I'm, when I'm, when I'm, when I'm, when I'm making, when I'm designing a framework, I'm like, he's like, oh, he's, he didn't provide direct feedback on this, but he's like the virtual sitting on my shoulder, you know, kind of keeping me accountable to keeping things clean and simple and straightforward and composable.
So that's great. Uh, one more person is asking, can Inspect be used with LLM proxies like LiteLLM? I don't see why not, but Absolutely.
Yeah.
Okay.
Yeah. Yeah.
Someone's asking about, do you have any imports or plugins for using Mac GPUs locally within Spect AI?
So you can use— so yes. So you can use whatever Oyama is doing, which runs on the Mac. I'm sure that they're using Metal. And things to make inference reasonable. I'm not positive, but like I can't imagine that a project like that wouldn't be using Mac GPUs. So Oyama's one way, and then you can with Hugging Face use the MPS backend. We do have support for that. So that, I feel like the Oyama has done a better job like reducing the memory requirements maybe than, I mean, depends on the Hugging Face model, but we found a lot of like the Hugging Face models that you want to evaluate, you know, you definitely need to have like pretty reasonable GPU, GPU or GPUs. So, and I don't know how generally how good like PyTorch MPS is. I don't know like how good it is.
So, I think those, this, kind of maybe a good stopping point. Um, you know, certainly other questions, but I think we hit the most important ones. Okay. As far as I can see.
Okay.
Terrific. Um, what do you recommend for people like trying to follow your work? Um, um, and like how to keep in touch with you or appraised of what you're working on or?
Yeah. Um, I'm not— I do have, I have a Twitter presence, but I don't, I'm not like posting on there all the time. So that's not a great place. GitHub is a perfectly good place to stalk and see, and see what's going on. You know, some of these commits don't show up, like for Inspect, don't show up there, but, but some of this like peripheral work, like this workshop show up there. So yeah, I'd say follow, follow me on GitHub. Is a good way to go.
Okay, great.
Yeah.
All right, with that, you know, thank you, JJ. It's really great to have you here. I learned a lot about the framework as well, so it was great to have this overview.
All right. Yes, it's a privilege to be able to come and talk to everyone here. And so hopefully you all have, for those that decide to give Inspect a try, hopefully you'll have success with it and let us know. If you don't, and, you know, we'll be very active on GitHub issues. So please let us know what's wanting and, or what you aspire to do that you can't do.
All right, great.
Thank you.
Thanks, JJ.
