Inspect - A LLM Eval Framework Used by Anthropic, DeepMind, Grok and More.

Welcome everybody. Today we have JJ Allaire. It's kind of difficult to properly introduce JJ Allaire, but I'm going to try. JJ Allaire is a legendary developer, engineer, and founder of several companies. Most people might know JJ Allaire either indirectly or directly if you've ever used the R programming language. You know, JJ Allaire founded RStudio. Which is currently Posit. And, you know, you should check out Posit. They actually have an excellent Python IDE a lot of people may not know about yet. More recently, JJ Allaire started working on evals for LLMs. And, you know, specifically, like, you know, he's kind of took a sabbatical from Posit to work on these evals. And Inspect AI, which is a great evals framework that's very unique. And if you took my course last year, you might remember JJ Allaire. There's actually been a lot of progress since then. So yeah, I'll just hand it over to JJ and let him get into it.

Thanks very much, Hamel Husain. I'm excited to share with you guys a little bit of what we've done in Inspect. Let me tell you a little bit about the framework, what it's for, what it does well, how it may map to the things that you're trying to do. I developed this framework working at the UK AI Security Institute, which does a huge number of evaluations, actually like thousands of evaluations of frontier models, usually pre-deployment. And so they probably run more evals than any org in the world, maybe save for one or both of Frontier Labs, but they just do tons and tons of evals. So I got there and they had 20 people writing evals all day long. And I kind of said, okay, let me see how I can make this better for you. Developed a Python package, Inspect AI, that's a framework for writing evals. I think we're trying to both create a great research experience, so like a great interactive experience for when you're developing your eval and the tooling around it, and a great production experience, because again, you know, we're running thousands of evals like every day. So we try to have a system that actually can stand up to that. It's used at the AI Security Institute for almost all of our evals. And one of the biggest things I wanted to do was make this a community effort and something that everyone in the LLM community could benefit from. And so we open sourced it. This was in last May. And since then, we've seen huge adoption across the ecosystem. We see it in academia, but we also have Anthropic, DeepMind, and Groq all are using it internally. Orgs that you might have heard of, like Epoch, Meteor, Apollo, are all using it. So it's very widely used. And as a result, we get a ton of feedback on it. And are able to improve it quite rapidly. So I think one of the things I want to— I know you guys have spent all this time learning about evals, and in some cases you're like, well, this is the first time we're doing evals, or we just want to like get our first, you know, half a dozen evals running so we can make our models better. And there are definitely— I'm sure you've both learned nice workflows for doing evals and scripting them. And also you probably maybe have seen some products that do evals, or I don't know, but I would say the one thing about Inspect is it's like a very sort of advanced evals platform. It is, everything is programmed in Python. It's extremely flexible. It has all kinds of things for doing like fancy scoring and parallel operations and agents. You might see some of this and say, why do I, like, I'm just at first base or second base on evals. I don't need all this fancy stuff. You might at some point. So I would say some of you, this might resonate and go, wow, I could really use a framework like that. And some of you might say, well, I'm happy with just doing some straightforward evals and not getting into all of that. Just keep it in mind. Because I think in terms of depth of features and flexibility, it's— I don't think there's anything equivalent out there. So the concepts are probably— I don't need to go too slow on this because you guys have learned all these concepts. Essentially, you always start with a dataset, and there's lots of flexible ways for getting datasets into the system, which has some kind of grading guidance or ground truth and some kind of input, which can be end messages. Of any kind. And on the other side, you have a scorer, which essentially does whatever you guys have sort of gotten into this, like LLM as a judge and other types of scoring schemes based on, you know, JSON schema or text comparison. And then in the middle of the system, there's this— the heart of the system is kind of this concept of a solver, which does something to help identify the final output to be scored. So this could be like traditional prompt engineering. It could be techniques like critique. It could be an agent scaffold. It could have tool use. The solver is sort of this big general-purpose thing in the middle of the system that lets you elicit the model to see how well it performs for the benefit of scores. And so, like, a really simple example, hello world, is going to exhibit all these elements. At the center, again, is the solver chain. Here we're saying, like, prompt to use chain of thought, do a normal generation, do a round of self-critique, and then generate the final answer. So that's like the solver in the middle. And then you have a score at the end, which is going to use a model. And all these, like, this is like the simple, like, unparameterized use of these things. Obviously, every single thing in here has templates that you can customize and replace. And, you know, and you'll frankly need to do that in order to get like the best score and the best solver for your given domain. But this demonstrates the basic concept. Okay, so I'd say I can probably go through this. Very easy to create simple benchmark-style evals. And I'll show you later, we've got implementations of like 70 benchmarks, source code for those that are open source. But, you know, as I said, most rigorous real evals will not just use defaults. You'll be customizing all the templates and all the prompts. Sometimes you do chains of solvers. Sometimes you create like a big fancy Python script that does everything itself internally. And that task decorator is just like how things get registered to run. Okay.

So the question about that, yes. You know, things like generate, chain of thought. Yeah. Are those things that you might use outside the kind of scope of evals, like as a general LLM framework, or how do you feel about that?

Yeah. So we have really steered towards like, I mean, the interesting thing, I think I was meeting with someone who has an LLM product. I forget what it was called, but like, how do you convince people to do evals? Cause they're kind of selling into the corporate market. It's like, how do you do that? I'm like, oh, my customers only do evals. That's all they do. They're like labs and these like safety orgs and stuff. So I really leaned hard into that. We don't do production. And there are a bunch of features that we would do in order to like facilitate production use that we're just not doing because all of our users just do evals. So you could use those things in production, but we don't promote that, document that, lean into that. So my guess is if you're putting, if you're looking for a framework for production LLM, pipelines, like, this is not a good way, direction to go. But yes, you, you could, in theory, use those things in a normal, in a normal pipeline.

Thank you.

So I think you can look at InSpec like 2, roughly 2 high-level views of it. One is like this sort of snap-together evals from prebuilt parts. That's a little bit of the view that I showed you there. And some of the benchmarks that do like simple QA multiple choice are that way. But like, another way you can look at it is like a bunch of useful facilities that like all evals require, and then you can just sort of code evals however you want on top of that. So that's things like the universal LLM interface that supports all the, you know, different model providers, all the different ways to call tools, internal tools like web search, you know, just sort of very deep universal LLM interface, a bunch of stuff around scoring and metrics and aggregating scores and getting the statistics right on things when you run multiple epochs and reduce them, a bunch of things around tools and sandboxes and things like that. And then a bunch of stuff around, like, parallel execution so you can run, like, 50 evals at once. So, like, it's sort of like there's high level and then there's, like, I just want to use all these facilities and kind of, like, write a bunch of Python code to do things however I want, and both are supported. So the high-level API, I've shown you a couple examples. This is, like, the super simple, like, okay, it's GPQA diamond. It's multiple choice. We score the choices based on the correct answer. Like that's all the source code. That's like a complete eval.

Like GPQA Diamond is a benchmark that model providers tend to report on.

Although it's getting saturated, it's probably up to 90%, but you know, 85 to 90% of most frontier models. But like, you know, 2 years ago it was like 50% and everybody's like, how are we doing on GPQA? So this is an example of just like. Snap together parts, dataset, multiple choice, do the thing, run for 4 epochs, et cetera. Okay. Yeah. So I won't explain this in depth, but you can see, you know, we've got this use multiple choice, shuffle the things. Another one, this is a, this is a, a capture the flag challenge. So it's like an Agentic eval where you're basically telling the model, hey, you've got Bash and Python and you're in a Docker container. See if you can capture the flag, right? And it goes and it's sort of like you can capture the flag as cybersecurity, but it's sort of largely also kind of measures like soft proficiency and software engineering type tasks. And there's other evals that we've implemented like SWE Bench and things like that, or SWE Lancer, things that are more software engineering, but here it's a cybersecurity agent that's trying to do capture the flag challenges. So here again, it's very simple. You give it a system message, you say use these tools, and you say go. Again, very high level. And it can be very useful, by the way, when you're starting an eval just to start with, a high-level expression at least just like gets you, okay, great. Now we're generating scores, the dataset's being ingested properly, and then you start going into depth. Okay, now let's really work on our prompts. Let's really work on our scores. Let's, let's like make this better. The low-level API is more like I'm in a solver and I'm doing the kind of things you typically, you know, do when you're writing a production LLM application. Like, cool, I'm adding, I'm actually adding the system message explicitly. I'm basically making tools available and then I'm running in a loop looking to see whether tool calls are still happening. The people who make more advanced agents or want to do really tailored processing end up writing some of these lower-level solvers that do more into the mechanics of generation and tool calling and things like that.

It might be worth a pause for a moment.

Yeah.

Because I think some people are confused. So to set the stage, you know, in this class so far we've been talking about very domain-specific evals, like evals you would want to write for your product. What JJ shared earlier is like that benchmark he showed was something that you wouldn't use more for if you're a foundation model lab.

Yeah.

And so, so that's very different. And then when JJ was talking about production, you know, don't use this in production, he was talking about that you could still use this framework to analyze a production system. Yes, you just don't want to put this in your— you might not want to use like some of these utilities in your LLM calls.

Yeah, you don't like deploy a production system. Like it's not a substitute for LangChain or some system that's like, so it's more strictly for evaluation, although you certainly could use it for that. It's just not something we emphasize. So yes, it's a very production-ready tool and it can be used to analyze data from production workflows, but it's not for deployment. It's not the LLM you deploy. It's more like the testing. Ring that you use. Okay. Yeah, so we have this concept I want to focus on higher level, but this idea of a solver, which I've given you a few examples of, is really just like manipulate messages and yield output. You can imagine you basically do something— prompt engineering, generation, critique— you do something in the solver to advance the evaluation. Basically something either make the model perform better or give the model feedback. Or run a tool use loop or whatever it is you need to do. So some examples of solvers, like a prompt— this is probably the actual source code or, or if not very close to it, to a prompt template solver, which is really just like basically just, uh, do a string format on a bunch of parameters to customize a prompt. That's a really simple solver. Uh, or even simpler solver is generate, which literally just says like, just generate, append the message, call tools if they're requested, et cetera. So there's like really simple solvers that correspond very much to the kind of like workaday stuff you would do when working with an LLM. A fancier solver is like multiple choice, which is doing like shuffling and it's unshuffling and it's doing some more fancy stuff. And so you can still kind of package this up into a simple like multiple choice solver, but it's like doing a little bit more under the hood. I'm not going to dwell on the source code for this. It's more the idea that you can implement these kind of workflows and solvers. And certainly—

I'm just going to linger on this for a second because there might be still some confusion. Is, okay, I know you mentioned that InSpec is very popular by the foundation model providers. Now it is the case that you can still use InSpec for your domain-specific things. Just, you know, you have all these primitives and all these things. You might not use all of them, like the multiple choice one, maybe you use it, maybe you don't, but they're there for you when you implement your own eval.

That's right. That's right. It's really like a toolkit for doing all this stuff. It's more just like a lot of possibilities. And you might go very deep into it or you might be doing things at a very high level. But as you'll see, it really helps you structure all your evals, run them in a systematic way, get a huge amount of observability to what's happening in evals, handle scoring very flexibly with lots of different ways to score. So, but the dataset is whatever. The dataset can totally be, you're just like, okay, this is our, we have 2,000 samples that we always make sure we run every time we change our prompt. We wanna run these 2,000 samples and we wanna see how things have changed. and so it's a very good tool for that. So it's a, it's a very good tool for the things you've been talking about in this course. It's sort of this flexible programming framework. It probably plays a little more to once you become a little more advanced in doing evals. But I think honestly, even for simple, straightforward evals, it's a very good tool. Here's a self-critique, which I won't do all in the code, but the idea is the model creates an answer and then we play it back to the model to say, can you critique yourself? You could also have another model critique. It, and then once the critique is generated, we have it generate again. And we find that in some cases, this improves performance of the model. And there's lots of variations of this, but I'm just giving you a flavor for the sort of things that go on in solvers. There's sort of these arbitrary little mini programs that do something to advance your evaluation or do something to elicit performance from the model. Again, I won't dwell on the details of this. And I think one of the ideas is composition, where you basically, like, create Python packages that have solvers and scorers. And then you can reuse them in lots of different evaluations. So, for example, we have at UKAC, we built a bunch of jailbreaking solvers. Again, you probably don't need these for your— but in the case of, like, trying to figure out what the model might know about some unsafe domain, it's useful to have jailbreaks. And so these jailbreaks are basically doing— each of these solvers is doing some prompt engineering. And so then you can write an eval and say, okay, this is a normal eval, like testing knowledge of computer security, and you basically grab the solver from this Shepard package and then put the solver in your normal generation chain. And now all of a sudden, hopefully, if that jailbreak works well, we're actually able to see more about what the model is actually able to do because we've gotten past its, like, initial layer of safeguards. Again, how many of you are, you know, the evals you're writing, involve like tool use and how many of it's just like, just kind of QA and reasoning and, and things like that. I don't know if I can get a sense.

Yeah. Okay. So what we're doing evals for is real world production systems that are domain specific and those things all have RAG. They all have tool calls.

They have calls and everything.

Yeah. To do the eval.

So you have to reproduce this. So that we have a, and I, I won't go dwell on this too much, but basically you can write a tool as a Python function or a tool as a model context protocol server. And so there's lots of ways to easily, if you've got Python code that does the thing, it's very easy to register that as a tool. Pretty much if you just have the function and it has doc comments, you can just register as a tool with InSpec. And then you prime the model and say, here's the tools you have, and then you run it in a tool use loop. So we have a very comprehensive system for even validating the tool input, going back to the model and saying, hey, you got this wrong. That was supposed to be a string, not a Boolean. And the model fixes it. And so we're a pretty deep implementation of tool use.

That is pretty important because What you're saying is, okay, if you have like your production code, you know, you have some code that does like, that basically defines an agent and has like tool calls and RAG and stuff. You don't wanna, you're not re-implementing all of that.

Yeah. So one is you, you can just call the Python functions and those are the tool calls. And most frameworks, production frameworks that you're using will have some similar thing where you can just like, here's a Python function, use it as a tool. So I think that code doesn't need to change. We also do have an agent API that both lets you build your own, you know, there's lots of primitives for building fancy multi-agent systems, but you can also basically, we have a thing called the agent bridge, which essentially lets you like bridge to an agent that knows nothing at all about Inspect. So it's essentially just like messages in, messages out, and you can still get all the benefit of our observability and scoring and all that stuff. But just use your agent unmodified. I think a lot of the people here, if they're building a production system, will basically have their agent and then they, and you can just plug it into Inspect using this agent bridge. So it's really like you're basically saying my agent that I run in production is the solver and I want to plug that in, but then on the front end, I want to put a dataset and I want to put scores and I want to inspect like observability and things like that. So the agent bridge is the kind of the, I can, it's in the docs. But like, that would be the thing that would like, if you want to take your code and like use it sort of unmodified with no knowledge of InSpec. It's pretty cool.

How do you pull that off?

We, what we do is we serve you the messages and then we sort of monkey patch the OpenAI API so that you call the OpenAI API and then we receive it and use it to interact with models and things like that. The idea is instead of calling InSpec's model API, you call the OpenAI model API. but then we capture all those interactions and put them into our system.

Okay.

It's like a proxy. Yeah, I think there's another version of it that will— we currently do it by monkey patching, but I think we're actually going to change to do it through proxy. But yeah, it's essentially like a proxy. So yeah, so that's, I think that would be the best path for like directly testing production code. You know, in our agent docs, there's a thing called agent bridge, which sort of describes, there's an example of like an agent. There is no inspect code in here.

Right?

It's just using OpenAI and doing a thing. And then this actually will work as an agent inside Inspect. There's docs on how to do that. And okay. And people have used like, we have an example, a LangChain example, which is like how to write a LangChain agent. Again, this has no Inspect dependencies at all. It's all just LangChain. And then it's very easy to run that inside Inspect. So I'd say that if you guys are doing things with agents and tools, this would be the most logical place you'd want to bridge into Inspect. Okay. Let me get— where is it?

So is it like when you specify model equals Inspect, it knows what is happening?

It knows exactly when you say model equals Inspect. Uh, where did I do that? Somewhere I did that. Here. Then we intercept it and say, great. Okay. Because in Inspect, you can say, I want to evaluate size models. I want to evaluate YAMM 4.1. I want to evaluate Anthropic Haiku. And so you just say model equals Inspect, and then we just route it to all the models. Models.

Okay. And you have a somewhere else that you specified the model?

Yeah, somewhere else externally in your eval, you've said what the model is. Yeah.

Okay.

So this is pretty magical way to take your existing production agents and run them through the eval pipeline.

All right.

So that's that. I'm glad we covered AgentBridge. I think that's super applicable to everyone here. If you're not using the AgentBridge, we have a whole bunch of built-in tools. Again, this is if you're using our native agent API. They're useful for the types of evals people run on software engineering evals and software engineering agents and cyber agents. Probably not as useful for you, although web search might be useful if you know you're going to be using Anthropic web search or OpenAI web search as part of your production system. We know how to call the model and tell it it can use web search.

It's really interesting. It's like you have almost like a parity with, with like a LangChain, except you're not trying to do that.

That's exactly right, Hamel Husain. Like, it is sort of like all the same features as LangChain, but you'll, but heavily tilted toward a focus of just evaluations. Again, because our main users only write evaluations. You know, they, that's all they do.

Like, if you're writing evaluations all day and you're using these tools and you love these tools, I wonder if you just use them in production. Just be like, hey, let me just keep using them.

Some people might do that.

Okay, I hear what you're saying.

Exactly. Well, it's probably would work great. It's just like, it's what features we emphasize and prioritize are more in the evaluation domain. And therefore I don't want to steer people toward doing that. And then they're like, well, what about this? I'm like, well, that's not our most important thing anyway. So we have a whole approval system, which actually becomes very important with model context protocol and various computer use, like approvals of big things. We have a whole approval system. This is just showing one part of it where like you can say, I want certain tool calls to be approved and then a human operator has to approve them.

Is that computer use or something?

That is web browser use. Yep. So it's basically saying that the model wants to click a button and it's showing you what button. It wants to show you. I know, man.

This looks really good, really cool. Use this outside of evals now.

Yeah, we can. You could. You could. We've done a lot. But yes, I'm just— yes. I don't disagree with you. Anyway, we've implemented a bunch of evals. I'll show you. Actually, I think I can quickly, at the risk of— so inspect evals. Yeah, inspect evals is like this website we have that we have 70 evals across. These could be useful if you're fine-tuning models or doing you're like bending the model and you want to see, well, how does it, how have I affected its ability to do reasoning negatively by my fine tuning, for example, or have I, you know, have I somehow changed its core behavior on like important benchmarks? So this is just, and the other thing that inspect_evals does, it's like there's 70 evals. So they're like good examples, good examples for you to learn, learn what, what good evals, how they're composed. So again, I won't dwell on that, but you can check that out.

So this is interesting. If you want to reproduce it for whatever reason.

That's right.

You have like a relay.

They're all here. It's very easy. Yeah. Like here, like you literally just go like pip install and then inspect eval. That's it. Like super easy to reproduce all these evals. So yeah. And so there's some, yeah, there's a, we, we tried to get all the evals that the frontier models report. And then some of the frontier model vendors are using, I know Groq, but all of Groq, DeepMind, Anthropic run evals from here because it's a very convenient repository of standard evals. That they, you know, the source code's available for them.

No, like, this class is not about foundation model evals, and it's kind of very different than the evals we teach. But I think, like, this is really interesting. Just, you may want to take a look at foundation model evals. Just, yeah, yeah, just see what they do. And also, yeah, see what they do. And also, it's, like, actually really difficult to run some of these benchmarks. They have to create an environment. You have to, like, get the data together and all this stuff. And, you know, you can spend, like, entire days just trying to reproduce the benchmark. But this is like, allows you to do it with just like one command.

With one command. That's exactly right. Yeah. And you might learn something because some of your systems might be like, if they are knowledge-oriented systems, you can see how some of these evals are written and how the scoring is done and things like that. Okay. Scoring.

Okay.

I will not go in. There's a lot of things on scoring. There's model-graded scoring. There's running multiple models and voting. There's ways of combining the scores. Different ways of reducing multiple scores into a single score. It just goes on and on. So this is a math— are two math equations, two expressions equivalent? So we prompt the model and you tell us that these are equivalent expressions, which gives you a much better idea for a math eval of whether it got the answer right because a lot of times it'll say 4/12 and the answer is 1/3. You want it to score correctly. So that's just an example of a fancier custom scorer. Yeah, and there's a bunch of stuff I'm not gonna get into. We can run 50 evals at a time. You know, we have all kinds of concurrency stuff and we spin up Docker containers and spin down Docker containers and run on Kubernetes. And there's all this kind of production stuff, which I'm not gonna get into. You can imagine a lot of stuff for production, a lot of stuff for logging. There's a whole API for interacting with the logs. There's a whole front-end tool called InspectView for basically seeing all the samples, what happened. Transcripts for the samples, all the messages, what the assistant did, and all the model API calls. This is sort of the observability. We can kind of look at everything that's been, that's happened. And then we have a VS Code plugin that basically like run, you can run inspect view inside VS Code.

Can you host?

Yes, you can. So what you can do is if you have some evals, there's a command called inspect view bundle and just makes a website and the website basically just publishes all your evals to a website and then And that's it. Standalone website, just a static website you can publish anywhere. So yes, you can publish them very easily. You publish them to GitHub Pages, whatever. And one of the things that I'm working on now is a thing called InspectViz, which is going to be a data visualization system for data that's produced by evaluations. And one of the things we're going to have is provenance where if there's anything in the visualizations, you can find the exact log or message or model interaction that it corresponds to. Using that web publishing capability that we just talked about. So the VS Code plugin does lots of fancy stuff and that's worth definitely if you start playing around with Inspect, the VS Code plugin. Yeah, sandboxing.

If you're familiar with JJ's work, you know that he cares a lot about the developer experience and especially like the IDE and making it like magical feeling.

Yep.

This VS Code plugin is fantastic. It's very nice to use and it has all kinds of nice navigation buttons. You can see visualization here of the logs and the scores and all kinds of stuff. So it's definitely worth checking out.

You definitely need to, yeah, we've tried to spare no effort at making everything visible and transparent and productive. That's a major focus. I won't dwell on this as sandboxes, which is basically like the model is going to give me some Python code or wants to run some bash commands. I really don't wanna let them do that on my system. You know, so you put them in a sandbox, which is, could be a Docker container, could be a Kubernetes pod. It could be the same file system that you're running locally, maybe because you've already put the eval in some kind of jail. And so this is an extendable part of the system where people can create like sandbox environments for different things. The Docker is built in, Kubernetes is a plugin, and people have done some custom things with.

So how do you, how do you interact with these environments? Is it like also Python API?

Yeah, so what happens is that the tools essentially call exec, say they call sandbox exec. So like the Python tool says sandbox exec. And so anytime any Python code is executed by the model, it goes into the sandbox. And so the idea is you basically have like exec calls, read file, write file, exec that are available to you that run in the sandbox. And then your samples can have files that are copied into the sandbox and you can run a setup script in the sandbox, et cetera. Use it in production. You can fork Inspect and say, this is the production-friendly version of Inspect. So there's a lot of good production stuff in here. And we have people who are running like, why can't I run 200 Docker containers at once? So we try to be as scalable as we can. Although eventually you blow out to Kubernetes, like one node with more than 100 Docker containers is like, starts to get sketchy. Some tips for learning more, like the, that's the wrong URL. It's changed to, we used to be Safety Institute, now we're Security Institute, but whatever, that'll redirect to the right place. You can just search for Inspect AI and you'll get the website. There's lots of documentation on the website. There's a tutorial that kind of walks you through the basic mechanics and lots of stuff about all the more advanced things. That eval site, you can get to it from the main website, and you can also just Google for Inspect evals. That's got all these benchmarks. And then we have, we've had over 100 external contributors. So it's a very open project. We get pull requests all the time from all over the place. We definitely welcome contributors and we'll work to help ramp you up. If you're, if someone, for me, if someone puts 3 hours into a pull request, I'm gonna make sure we merge it, even if I have to work with them to get it in shape. You put effort in to try to help the project. I will make sure your work lands. So that's my prepared remarks, I guess. And we can do questions if people have them.

The same theme of all my comments. The one thing you showed me that really blew my mind is that browser use CLI. It's like, I don't have any tool like that. I haven't seen any. That's very nice. Are people using, they must be using this like for all kinds of stuff because I don't see it.

Yeah. So the web search APIs that are built into the models. So now Anthropic, OpenAI, and Gemini all have built-in web search. So, so you can basically, they just do the web search inside on the model server. And the real challenge with web search and web browsers is bots being blocked. And so when we first put our web browser tool out, it was very, like you said, it was sort of unique. No one else had done that really well. And that was actually contributed by DeepMind, actually, that was their pull request. And it worked really well, but then we've seen increasingly over the last year, I think with more LLM agents running, that bot, like bot detection has really ramped up. And so you see more sites saying like, you can't talk to me because I can tell you're in the headless. I can somehow tell that you're in a headless browser, or I can somehow tell that you're not. So there are like services now, I think Browserify, there's like, you, there are model context protocol servers that do this. That sort of, they're a service, you have to pay for it, but they will sort of try to circumvent the bot detection. And so, but I think the future of this is probably these built-in web search tools because they're presumably also doing some kind of bot detection. They can be a little expensive, especially OpenAI's is quite expensive. But, and there's also startups like Tavali and Exa. We support them as well that basically do it for open.

It's not new, but they just raised a really big round, Browserbase.

Okay, Browserbase. That's another one we should probably be integrating with. So like, I think web search is kind of the key and I think probably built-in and third-party services that can make the investment in circumventing bot countermeasures is probably the future there. The web browser is really cool and it's really cool to watch it. You can like watch it browse the web and figure out how to do things and it's super cool, but then you get these disappointing failure mode where the site just says like, eh, so anyway.

We can open it up to questions. I think it's funny, like some other students were really impressed by that specific tool too, that browser use one.

It's a fun one. When I first saw it working, I was like, holy crap, this is so cool. And then one of the first things somebody did was at NAC where I worked, they're like, okay, here's my prompt. Make money. Here's a web browser. Make money. So the model ends up on some like sketchy crypto site, you know, it's trying to figure out how to make money. So it's like, I need a Bitcoin wallet and I can't get the Bitcoin wallet.

We can take the live questions, I guess. Raymond.

Yeah, I just want to double-click on that, the agent bridge thing, because I think even though Hamel's trying to convince me to use your thing in production, I'm kind of particularly curious about like just using as an eval set. And like one extreme example that I thought of just because I'm starting to use it a lot more in my daily work for a lot of things is like Claude Code, like it's like a CLI. So like what—

great example—

evaluate it as an agent.

So we are actually just about to implement a Cloud Code Agent on top of the Agent Bridge and make it available. We're going to do it. But yes, that's what you would use the Agent Bridge for. I've got an agent, it exists, it has an API, and I just want to bring it in. So we're going to do that. We're going to make a new package called Inspect SWE and have Cloud Code and maybe some other software engineering agents. But yeah, perfect example of when you would want to do that. Cool.

And if it's okay, I have a second, just like anecdotally. I was also fixated on your built-in jailbreaks. I'm curious, like, have you seen just adding a jailbreak make a model exceed on one of the benchmarks, like more than the publicly—

Well, the benchmarks tend to be things like, does it refuse or not? Right. And so yes, you can go from always refuses to never refuses off of a single jailbreak. That definitely—

Have you seen it get better at a useful skill? Like, is it better at SWE-Bench if you jailbreak it?

No, no.

Okay.

I don't see a lot of refusals in SuiBench. There can be refusals in SciBench. You may not— SciBench is another sort of similar to SuiBench, but it's cyber. And sometimes the model goes like, oh, I, you know, like, whoa, like read the password file. Like, I can't do that. So yeah.

Got it.

One thing I'll mention, the question of cloud codes, if anybody's interested in Inspect, any project that JJ works on, if you engage with that project and open source thoughtfully. He puts a lot of attention. He makes sure PRs land, issues get answered. The speed of which that happens is insane. It outpaces paid tools somehow.

We're just into it.

Yeah. So it's like actually surprising, like when you, like, I would encourage you to get involved if you have the inclination and you'll be quickly shocked on like how, how much support you get.

I'll have Claude—

we have a PR. Okay, so one of the things I should say is that I think for people in this class, um, we have an InSpec community Slack server. We don't advertise it because it's very high signal-to-noise, and so it's, it's, there's probably about 800 people on there, and there are people, you know, from labs and safety orgs. So If you, you know, I can send an invite to Hamel Husain Husain Husain that the people in this class can join it. We add them, but we don't have it like on our website. We may do that soon, but it's been pretty high signal to noise. And so we like, and the people that we work closely with are in there, but if you want to be added to the Slack server, we can do that.

Steven has a question.

One of the, I've two questions, but if I'm only able to do one, I'm prioritizing.

I can do two.

Go for it.

One of the themes in this course. Was about going through your data and annotating it. And the proposed solution in the course was just like, it's really easy to vibe code your own annotation tool. I'm curious, what are your thoughts regarding annotations?

Yeah, I think we've definitely felt like that helping people with building datasets and helping people with create LLM judges and building front-end tools to help with that is something we've talked about wanting to do as part of a roadmap. We haven't, and I think Hamel does make a good point. Like, it is easier now than it ever was just to make a bespoke tool quickly. So maybe that takes a little bit of pressure off, but it's definitely like, I don't think people are as productive as they should be, nor do they follow good process as much as they should when they're creating like LLM judges. So like tools that help people do that better is something that's interesting.

What access have you thought about helping people along that? Front?

Well, for example, just first of all, you have to build an interface for the human grader that essentially uses the same dataset. And then you have to give the user tools to see if they have a model grader or multiple model graders, and then they have 100 human graders analyzing that to see, is my model grader equivalent to, or as good as, or what weaknesses or strengths does it have relative to humans, right? So just understanding What do we have here? Is this a thing? Oh, look, it actually proxies for a human grader. Oh, sorry.

My question was like, were you thinking of creating developer tools to help people make annotations?

Yeah, it could be a combination of like some data-driven way, like we'll build your annotation UI for you based on you just telling us a few things. But the vibe code is sort of, you know, to the extent that the domain makes that possible. Now it may be that there's too much custom annotation. And that's not actually a reasonable thing to do.

And like lovable for annotation apps or—

Yeah. Or it could be that like vibe coding is like fine. We haven't looked hard at it, but I don't want to presume that we can just build a universal annotation UI and it'll just work, but maybe. So I think it's a combination of the UI and then also then the, the, some Python functions to help you analyze the data that's generated from human annotations and, and model grading and, and understand the strengths and weaknesses of your LLM judge. That's something we've talked about, and there are people, there are a couple orgs that we work with that are very interested in this. RAND Corporation is one of them, and the UKAC is another that are super into science of evals and how do we make really good LLM judges. And so they're doing work there. So we may get to that sooner rather than later, but it's definitely on our roadmap.

That's really exciting. Please let me know. Would that intersect with the PyShiny at all, you think?

Yes, we could, we could do it based on PyShiny. We might choose to do it based on Python. I'd have to look at it, you know, from first principles.

But gotcha. Yeah, makes sense.

A second part to the question.

Go ahead. Oh, you have another— sorry.

Earlier in the presentation, you mentioned something about, like, evals and tools, and I think I may have it flipped in my head, but, like, were you referring to using evals to evaluate agent and tool performance, or were you referring to using agents and tools to duct your LLM evals.

No, I was talking about, well, I was talking about two things. One, I think in that case I may have been talking about like actual pro— there are products that help you do evals that are on the market.

Okay.

And they have different strengths and weaknesses and they do different things. And like, I was just saying, you might be familiar with some of these. I think those will tend to be a little more like point and clicky. Not that they don't have Python APIs, a little more point and click. And a little less flexible. So I was just trying to situate Inspect to say we're on the high flexibility, high capability, high programmability, not on the like point-click, I have an eval. But it's not that much effort to have an eval on Inspect. So I don't think the point-and-click is buying you that much. It demos well and it's easy to sell, but I don't think it's really getting you that much.

Mike, I think you asked some questions in Discord. You want to I don't know if they all got answered, but if not, you can ask it here.

Mike, me?

Yeah.

So I want to— I'm just trying to map my use case into some features or availabilities here. I have a labeled set of query-response pairs, and the response is actually— it's not just a binary, it's a list of tags, and then there's going to be precision and recall involved. But the nice thing is I have this set and there's hundreds or a couple thousand if I want to go that many. What my task is, I'd like to swap in different models, swap in different prompts and so forth. The question is, what features are available for the experiment tracking aspect of that or companion tools?

There, what you do is, for example, you write a task, And I skipped over this, but like tasks can have parameters. So all the things that you might want to vary on, right? You make parameters of your task. And then essentially when you run all your tasks, then those hyperparameters that you've created all get recorded. And then there's a bunch of functions that basically let you take the eval logs and read them into data frames. And then you can use that data. Then you basically have a data frame to do whatever analysis you need to do. We're also going to be soon like creating a Weights and Biases plugin. So if you use Weights and Biases, and you've done all these type of parameters, like, and it can just push it up to Weights and Biases. But I'd say the path that most people take is defining hyperparameters and reading them into data frames and then plotting and summarizing and analyzing the data frames. So there's a function, there's a namespace called inspectai.analysis that basically has all the, like, read eval logs into data frames. Great.

Thank you. So that's built in. And then as a detailed question on that, can you track cost and latencies because I want to, you know, yeah, basically, so latency information is all there, including—

so one of the things we do, a lot of retrying model calls, but we actually track the actual time taken for the actual request that succeeded. And so we have a concept of working time, which is basically time actually spent working, because latencies, like if you're— the model API is being overloaded, you can wait for 30 minutes, but then like Tomorrow it's 1 second. So we try to track that. So we have very good time and latency tracking all the way down to the model call and the sample. And then we have a PR open for cost, which essentially we track tokens very carefully. So you have tokens per sample and then you have tokens per model tokens and we track all the tokens very, very— so then the PR for cost is basically like just getting a database of token costs and then multiplying. So yeah, I think tokens is a good proxy for cost in the meantime. But yes, we'll also have cost. People use this in agents actually. They're like, hey, like run this subagent, but like limit it to 20,000 tokens, you know, and if it doesn't succeed in 20,000 tokens, just like give up. So there's like, you can put cost limits or you can put token limits and time limits and things like that on all the things that happen.

All right. I think that's There, there might be some lingering ones, but I think we have to wrap it up, and then I'll triage those questions in the Discord. But yeah, thank you so much for, for coming.

Good. I'm glad. It's a pleasure to talk to y'all. And just let Hamel know, I'll give you info on joining the Slack server if you, uh, if you want to.

Thank you.

All right. All right.

Thanks, man.