This is a YouTube video titled "Epistemic Evals - SDK Playground Demo Update (Build In Public)" containing a technical discussion between two speakers, including Jonathan, about a framework for evaluating AI prompts using falsifiable hypotheses and failure mode analysis. The conversation focuses on structuring evaluation hierarchies, testing constraints, and improving AI agent performance, particularly in sensitive, hyper-personalized domains like health and finance. The dialogue is detailed and technical but does not include a speaker named Robert Ta.
Okay, do you want to explain epistemic evals at the intersection of evals?
Yeah, I think this was, like, that kind of experimental view, and I thought it related to that conversation with Teresa Torres about opportunity solution trees. Because in both cases, what you're trying to do is use a tree to kind of narrow down what the hypothesis the test is. And in this case we're really testing is this hypothesis about the prompt, which, if you look at this, it's inheriting the context from the role in the context of the conversation, and it generates a prompt instruction. And then we can use that and do the open coding and failure analysis in that mode. So we can, like, create data sets. Like, you know, basically this tree gives us, like, context to test a data set and whether it's working.
Okay.
Like, this one is, like, just for characterizing meta analysis from exercise. So we can look at, like, the retrievers that are pulling the documents and analytics that are only meta analysis. And we've got these constraints. And so the idea here is that the failure modes can be, like, really concrete, and the constraints tell you ways that it fails in a particular way. Like, if you define a failure mode as a constraint going out of bounds and make those two things equivalent, you can see what I'm trying to do here is draw really tight boundaries around what each prompt is doing.
And that way there's just scientific rigor down to, like, adding more constraints. Just really helps us get to a more concrete falsifiable hypothesis of the interaction. Right. It's. Is that the philosophy behind that?
Yeah, that's falsifiable hypothesis. Yeah. That's the same thing we try to do with assumption testing and opportunity solution trees. We're just trying to use language to make it a really testable, falsifiable thing. And yeah, my hypothesis is that, like, if. If you really constrain the height, if you constrain what you're testing, then you don't need 150 samples of data. You can probably get like a tenth of that and know that you have some good samples that are working, just as long as you test each of the constraint scenarios and the failure modes that way.
Okay, yeah, that makes a lot of sense.
Yeah. So that was like meta analysis review for exercise. This is RCT analysis for exercise, either biohacker experiments for exercise. And you see, in every case, what it does is kind of like it has a pattern that it can generate the temp, the prompt template, and then we can use, like, we can basically find, you know, we have our three data Sets that Hamill taught us about, you know, that's the training data set, the test data set, and the development data set. You know, you can take. You can kind of like use your training data set right here with some examples. You just need a few of those. And then run the failure mode analysis, right. With open coding or axial coding to identify examples and where they fail. And so this is all happening at the bottom of the tree. And what we do in the middle of the tree is more just like routing. And if you remember, the Superpower bot just told us that it had identified your intent. Right? And.
Yeah, that's right.
So we can have a semantic router. That's all we have to test there is like, you know, given that we have a health enthusiast, what's their intent right now? And we would come up with examples and just identify and train the router to have an intent. And so we're decomposing what could be like one big long prompt that does a lot of stuff. And we're just breaking it down into different components of the system that can be easily tested. And then we want a UI here. Like you see it surfaces the failure modes. We want a UI that can kind of like trickle up what's failing and what you need to pay attention to. I've been playing this game that has like a little red dot every. Every time I need your attention. It's like a really great kind of mechanism to follow around. I want to experiment with something like that, you know, just so you can kind of gamify the improvement of the agent and fix problems from the bottom up and you can get the whole thing working altogether.
Yeah.
And then when you're at the couch.
Real quick, could I reflect something? So like, if an AI team were using this, they would get a bunch of traces, they would run it through the system and they could see the evaluation hierarchy is what it's currently called in this constraint based, falsifiable hypothesis way of doing evals.
Right? Yeah. Like this is.
Go there and see. Oh, okay. Yeah, go ahead.
Yeah. This is example.
Can you talk through this as if, like I didn't know about it?
Yeah. So once we configure this hierarchy, you know, we've kind of isolated where our different failure modes are. We can load a data set in and it can test the whole thing from like the top down as if like, you know, and so we imagine the data set is like the input and the responses. And it would tell you, here's the failure modes that are identified. Mapping. This is Like a health inside looking for evidence and the failure modes that we saw there. So it kind of breaks down the failure modes by the pathway that it's following, which is like super helpful because that's what we're doing. Like in the class they had the failure analysis like heat maps for this purpose. Right. So we could even have a heat map. Visualize something like that.
Very cool.
Yeah.
So now you're clicking integrate. What does that do?
It just, it's going to take that. I think it was showing us like the last time of the failure modes and then it imported this and now we can view the hierarchy as a whole, you know and say we try to improve our success and our coverage. Those are like the gulfs of specification and generalization. You know, you can, you can kind of like improve those by going down the ladder and like fixing all the pieces that are broken at one at a time.
Yeah, yeah, sweet. Thanks for walking through that. And yeah, so this is just a very different way to approach evals and our belief is that for hyper personalization sensitive spaces like health longevity, mental health, education, personal finance, that type of thing, this would help you identify the fairly modes better systematically and continuously improve the agents evals over time. So that's the current hypothesis. Awesome. We got to put some more revs into this with some of our agent builder friends to see, you know, is this helpful? How could we make it better for these hyper personalization cases. Cool, thanks for walking through that Jonathan.
