Why Do Smart Engineers Struggle With This In AI?

Why is this not a solved problem? Why is it not a solved problem to create a model of a user?

If your agent doesn't give a good recommendation, then you might not retain the user. They might not refer the agent or product to somebody else. When I talk to VCs or builders in the Valley, startup founders, building AI, the hardest part is if you're an engineer building this stuff is you're used to getting a prd, then you could extract ground truth and then build it and then cover for ground truth. This is a whole different ball game. We got to figure out a scientific way to systematically make this better.

AI alignment is like kind of an overlaid term and it means a lot of things to a lot of people.

Let's define that. Let's define that. What's your definition of AI alignment?

Well, I think it means that.

Welcome to the ABCs for building the Future podcast. My name is Robert. I'm joined with my co host and co founder on Epistemic me, our open source SDK for AI alignment solving for belief systems and self models. On this show we bring on researchers, scientists, entrepreneurs, builders of the future, people doing big things with big stories to tell. On this podcast today we're going to go through a true built in public update. Jonathan and I haven't talked really in like five days. We usually talk kind of every day and I was on a work trip to Toronto. So we have basically a year's worth of startup knowledge to exchange in this meeting today. Because we've been five days behind on exchanging info with each other. We're going to do that live and that's going to inform some of the customer journey and product roadmap that we're getting to with our new offer and our tooling. The other thing we're going to talk about is epistemic evals. So that's the term that we've come up for the market position that we want to create and dominate. And it is, if you think about it, there's an evals hierarchy of needs for LLMs, there's the foundational model layer, you know, OpenAI, anthropic, deep seq, and then on top of that there's application evals. So for my AI health coach, for a certain problem space or domain, for an icp, that's application level evals. We just took a great course on that by Hamil and Shreya. We learned a lot about how to systematically make a better AI experience by making your application evals better. Today we're going to introduce the term of where epistemic me plays. And we're saying it's epistemic evals. We don't compete with application eval tools or developer tools. We actually are adjacent and help further with personalization in personalization sensitive spaces such as mental health, health, longevity, personal finance, education, et cetera. So we're going to talk about that a little bit and why we think that's key to AI alignment. And then we'll go into build in public update, product roadmap stuff, customer journey stuff. So cool. Before maybe all of that, we're going to do a little bit of a brain dump. Jonathan, do you want to give some thoughts about what you've delivered recently for epistemic Me and then maybe ground some of that delivery against the customer journey stuff you were talking about and I can kind of introduce the epistemic me evals concept intersecting that.

Yeah, that sounds great. It feels like there's a lot to go over.

There's too much also. It's good to just talk to you again, man. I feel like, I feel like we talk every fucking day. So then if we don't for a little bit, it's like, whoa, this is a different experience.

So yeah, there's like several different layers of stuff to catch up on. Yeah. So I guess if you want to start with this idea of epistemic evolves as, as like the bridge from normal evaluation and making sure that like a prompt does what it's supposed to do. And then there's a jump then to like evaluating your system, evaluating your system so that you know that the system can understand the user and can personalize and create recommendations for that user. And I think that that layer is what we're going to call epistemic evals. And of course our approach to that has to do with like modeling the user and creating belief system structures and like really using epistemology to like model where the user is at. So that's, I think, why this term epistemic evals is like really appropriate.

Yeah. And we'll link in the show notes a paper that Jonathan had found that was released about a year ago on what they called user centric evals. The point of it being that I'll ground it in a very human subjective experience. Any person that goes, you know, and talks to ChatGPT, if you use it enough, it'll ask you things like, do you like response 1 or response 2? Response 1 or response 2 has a different set of heuristics. Like, if you really think about it, it's like generally one might have emojis and packaging of the information might be markdown header style, whatever. The other side might have no emojis, it might be more raw text. So that might be one heuristic is even how do I package the response for the user? So one user might like it packaged with emojis and markdown headers or whatever. One user might just want raw text and that also might be based on their previous interactions that the software system learns that preference. So there's this idea of user perception of qualities, quality of the response or efficaciousness of the response. You can think about it that way. And every user is different. Right? I you see this best, I think in just human communication where I think A, I try to say B you receive it as C and like you hear it as C and then it absorbs in the brain as D it's just these little telephone hops. So epistemic evals are what we say are at the user and belief level of evaluations. So. And there's alliteration there. So that's why I like the term. I think it's going to be really catchy.

Yeah, good stuff. I was wondering if you had the picture handy of the diagram from that because it's pretty illustrated.

Yeah, let me show that. Let me show that that's a good idea. So we're going to link this paper in in the pod. But here's one diagram which kind of shows up. I go back, okay, response feedback. There's. So there's for typical conversational recommender systems or recommendation systems. Conversation component. Recommendation component. And so basically the user interaction is I give some level feedback, the recommendation and the response comes back and this should be optimized for what the intent of the user is. And this paper kind of argues that, you know, we've looked at application evals, we've looked at these other layer of evals. So there's this great paper and I'm showing you a diagram of the paper which talks about user centric evals and and gets to user beliefs. We really like this paper and what you are seeing in front of you. We'll link it in the show notes. What you're seeing in front of you is a diagram that shows what I just talked about. These very subjective qualities that are N1 user perception level like my perception of these qualities and how well the agent meets these heuristics or qualities is going to be probably different than many other humans or users. One example is explanation. How well did the agent explain Things in interaction, adequacy, recommendation accuracy, recommendation novelty, the diversity of the recommendation, information sufficiency, interface, adequacy. These are pretty subjective heuristics, right? And mapping this, you know, these subjective heuristics to user beliefs, that's in the middle here. So you'll see these boxes. Transparency, control, perceived usefulness, perceived ease of use, and then user attitudes and then tying into behavioral intent. And this is kind of the product portion where understanding everything to the left here for your conversational recommendation system, your recommendation engine, whatever you want to call it, gets you to purchase intent, conversion intent, user intent. And that's why you would want to care about hyper personalization and what we're calling epistemic evals for the problem spaces that are sensitive to personalization, mental health, health longevity, personal finance, education, et cetera. Jonathan, do you want to add anything on this? I'll also say like, there's more to the paper. This diagram doesn't say everything, but it does show. Oh wait, if I were an application developer trying to build an agent in one of these type of personalized sensitive spaces, I would have to go build all this. Well, real quick, plug Epistemic me is open source and we've done a lot of the heavy lift here for a belief model engine and a self model engine. Jonathan, would you like to add more on, on this?

Yeah, I thought this is a cool paper. You know, inherent in what they're doing is kind of like a model of the user and I think they're having to do some philosophy here in terms of like, well, how do we, like how do we represent what the user is doing, how we relate these components together. There's attitudes, there's beliefs, there's qualities, there's intentions. And you know, like, like we've come across all of those different aspects too and we just kind of have like a different way of composing the epistemology about it. And it's like, it's really interesting to see this approach. And I think this is coming from you know, like machine learning community and machine learning academics. And this is an active area of discussion and conferences and papers. And so it's an unsolved problem to sort of take a chatbot, have it, understand the user in a way. They can, you can personalize responses to it and then make good recommendations that the user can accept. And so this is like a, like a step in that direction. So yeah, epistemic evals, I think you can kind of get the idea right.

Yeah, epistemic evals. I love the name as well. One other plug for the audience. Me and Jonathan are always trying to test our market position. Obviously we talk about our project to anybody that would listen. At least I'll say I do. And I just recently talked to another builder who works as an embedded engineer for a VC that handles mission driven startups. So what he does, he works for 5th year, his name's Kia. What he does is he basically goes in and ninja fixes things that their portfolio of startups and founders need on the technical side. So he might go in and sort of ninja fix things or be a technical advisor. So he's trying to stay up to date on what is AI, native development, what is AI, anything. And I talked to him a couple days ago and I showed him what we're doing and he immediately clicked with the term epistemic evals. And that was such a positive sign because I see him as big overlap for our icp. So not only is it a nice marketing buzzword because it has alliteration that's great, it could go on a T shirt, but it also groks with the kind of philosophical agentic builder that we would say we are and want to attract more of for our open source community. So I thought that was really cool. I'm pretty stoked about epistemic evals as a phrase and a term. It reminds me of Nvidia saying, hey, we're making GPUs, we don't mess with CPUs, we just are in a different category. So I like that. Where we're not trying to compete with application eval tools were also helpful on top of those use cases too. Is there anything you want to say about that, Jonathan? Like market positioning or us at the intersection of these other application eval tools of which you've researched a lot and have used yourself?

Yeah, well, I think it just all comes down to like the utility usually getting value out of something that's like, you know, they see something on the screen that is different than the evals that they do today. And you know, for us there's going to be like personalization evals and recommendation evaluations. And I think it just all comes down to like seeing like, oh, I evaluate these differently, I get more value and I can do different things with them. And you know, I think that just comes down to like the product management and making it a unique value that deserves a unique name.

Yeah, unique name. With that said, because we're talking about beliefs and we're basically making the statement a strong hypothesis, I should say that we don't think you could solve the hyper personalization problem. And furthermore we don't think you could solve AI alignment without modeling users belief systems and having a flexible protocol for that. So one question I wanted to ask you Jonathan, is can an AI ever truly understand a user without understanding the beliefs? Could we really solve the personalization recommendation problem without the belief modeling? What do you think about that?

It's a philosophical question. I personally don't think so. And I think that there's a lot of overweighting on maybe axiology and when it comes to like normal scientific method, sort of like how do we understand somebody, you know, and what beliefs do is they give context the axiology and they tell you like the use the value that the users perceive in the context of their beliefs. And that's really the only reasonable way to manage the complexity space and sort of like get to a paradigmatic way of looking at the user where the more data you collect about them, the more you can learn about them. And you know, like, you just have to step back and ask yourself like there are so many people that work in this field, why is this not a solved problem? Why is it not a solved problem to create a model of a user? And honestly this was like a shock to me when I got done this path and I was like maybe 26 years old, so almost 20 years ago. I thought like, you know, certainly we have like I study electrical engineering and I was used to these like ideas of communicating through a channel and seeing it. And I thought like certainly we have ways to model users and understand whether the user has received and understood a message through a channel. And it, it, there's just not like there's not a semantic theory of communication and there's not, there's not like models that represent users in ways that we understand them. And you just have to ask like why is that? And that. Then you get back, you get to this like long history of philosophy specifically where analytic philosophy has given us gains, where we can take, you can take like axiology values, error, error codes or failure modes. And we can understand those things in a way that like we can use tools like gradient descent and we can do machine learning, but they don't approximate the reality of like how a user is represented in the world. And so it's sort of like you can't, if your model is not good enough to like learn and train and improve over time, you're not going to get a better understanding of the user. And that all comes back to your question, which is like what do you need? What do you need to do? And I think that beliefs are what give context to that, to a training model that you can incrementally understand better. User.

Yeah, that makes a lot of sense. And it was a softball question, audience. Obviously, if you've been listening along, we totally believe it starts at the belief as the fundamental unit building block. Like from a first principle standpoint, a user's perceived qualities of an experience will depend on their belief system at that point in time. Right. So if you don't solve for that, then I think the efficacy of the recommendation just can't, cannot be proven to be the best. So anyway, that's our take on that. So real quick, extending on that, let's ground epistemic evals then on AI alignment. Now that we've kind of formed epistemic me team's position on it starts with a belief modeling for a user model that evolves as they evolve and that can help create a recommendation engine that has that hyper personalization component. So let's talk about AI alignment with epistemic evals. Can we solve AI alignment without epistemic evals?

Well, I think first AI alignment is like kind of an overlaid term and that means a lot of things to a lot of people.

Let's define that. Let's define that. What's your definition of AI alignment?

Well, I think it means that like the machine understands human and the human understands the machine and that they're like there's interpretability, measurable interpretability between both parties. That would be like a baseline AI alignment. And then I think if you layer onto that interpretability understanding that I do understand the user's values and I'm not going to do anything as a machine that's outside of the user's values. That's really where I think that AI alignment takes on meaning in the way that it's used, say by OpenAI or anthropic or something like that.

Yeah. So interpretability we did learn from Ham Altreya's course there's lots of metrics, right. One that really stood out was just the simple AI alignment metrics. They had around true positive rate and true negative rate. Where if a human were to judge the conversation turn or dialectic, like what did the user query for? And then what was the response to that from the agent and what was the evaluation? How let's say you have 100 of these traces or turns or dialectics and then you have a domain expert, like a health and fitness longevity coach, and you have an AI Coach like AI Brian, how many times out of that hundred is that coach going to agree with this was a bad response or this was a positive response? Bad response would be, yeah, I agree that this is bad. True negative rate. So you would kind of want to include that in your goldens once you get to it. True positive would be, hey, yes, for this thing, Robert's asking for vegan recipe and yes. Oh, you looked at where he lives, you looked at his macros, you looked at whatever to then make the recommendation. So you looked at all the traces and toolchain calls and the rag and you evaluated. Okay, yeah, that is a great response that would work for a human client. So that's what we mean by true positive rate. True negative rate. That's a very grounded example of, of an AI alignment metric. Do you want to add anything to that, Jonathan, on AI alignment metrics or interpretability?

Yeah, so you're talking about the calibration of LLM judges as alignment characteristic. Yeah, where there's an LLM judge and they're, they're aligned with the humans that are also human evaluators. And so you have a consistent like judgment of whether you know a response is good or not based on. Yeah, cool.

Yeah, thanks for the clarification on that. Yes. And for the audience right there, just so you're not confused, this is like if you have an AI health coach live, you have thousands of traces, real live production data and you want to automate your eval lifecycle better, you would introduce a solution like Elemis judge where you build that into your eval pipeline. But you also want that thing to be really aligned with a domain expert like that health or longevity or fitness coach. Thanks for the clarification, Jonathan. Cool. So anything more that we should say about beliefs? I suppose we should define belief system because I was just talking to somebody about belief systems and we got really into a rabbit hole where they would say they would describe a deeply rooted belief, like a hey, from childhood level belief versus like I learned something new about AI tooling that evolved a previous belief I had around the limits of AI tooling. So they define those differently. I define those both as belief systems, but maybe with differences in weight and how deeply rooted they are. How would you define belief systems for epistemic me, Jonathan? That might be useful for the audience to understand.

Yeah, I was actually just looking for that paper that Victor shared about the paper where they took survey of beliefs of people on the Internet and they bring to like, how do you your beliefs about like abortion or about God or, you know, about these things. And I think that's people. What people normally think about when you say, what are your beliefs? And I mean beliefs in a, in a more general sense than that. And like, you know, I think. Well, one way we formalize that is look at like Carl Frischen's free energy principle, where every model of self is an abstraction, like a metaphysical abstraction, that you have a model of self, that there's, there's a self and there's a world, and beliefs are what predict what the state of the world is going to be. And so it's very general and it can apply to like, you know, how, how the, the self of a, A cell sort of like, interacts with the world, or it can predict how the self of a human or a monkey or a dog or anything that has like a self model and a world model interacts. And so, you know, like, there's, there's, like, there's a belief system that there. That is really just a causality instrument. And I think that's where I think that, like, beliefs come across as like a way of, say, understanding causality in the world in a philosophical sense. And I heard a, I heard a philosopher on, I think it was Sean Carroll's Mindscape podcast recently. He's, he's a professor of philosophy, and he said a lot of people get into philosophy and they think like, oh, this is something that, like Plato does and Aristotle and Kant and like, you know, those are philosophers. And. But they don't understand that anytime that they are trying to make a causal inference in the world, that they're using a conceptual framework and they're using philosophy. And so I think, like, we're kind of like live in this time where all of us use philosophy all the time and we don't like, label it as such. And when just. So if you're making a decision about, like, what time do I go to bed tonight to get a good night of sleep, There's a causal implication there, right? That, like, if I go to bed and get eight hours of sleep, that, that, that, that I'll feel better the next day. That's a conceptual framework. There's causality. There's all kinds of beliefs baked into that decision that you're making. And that's just something you do every day. That's philosophy. And there's an invented set of belief systems around it. I think you just want to model the self, model the way it interacts with the world model the way it, it, it constructs the causal patterns through a conceptual framework. And then that's where you have a belief system structure within the self that predicts that.

Yeah. And just to tie it. So I really appreciate that grounding in the very generic definition of belief system from Carl Frischer's free energy principle, that how we define beliefs for the audience in our SDK, like that's the definition that we align on. And going back to that user centric evals item, I just want to show that again, Jonathan was saying the beliefs help us inform the causal structures around the behaviors right in the real world. And so everything like kind of on the left here, you could kind of look at this as a belief system that evolves as we get more interactions from the user. Just look at this as like a model of the user that evolves. Each of these things evolve as we get more data from the user, as it talks to the agent more, as it navigates in your software more or application more. And then we can map to, well, what if we had that belief system evolution, like timestamped based on all their interactions and. And then we see when do they purchase, when do they share this AI with a friend, when do they have some sort of interaction that we can map to one of the pirate metrics, like, you know, revenue retention, all that. So anyway, I just wanted to really just tie the loop on that is we can get pretty deep on the neuroscience and philosophical end, but at the end of the day, like, it should provide value to the user and the developer. And that's what we're really focused on is as a developer, you shouldn't need to know this level of detail about belief system mapping and blah, blah, blah to get to the outcome, which is, oh, I could really understand the causal structures to help my users achieve their transformation. Whether it's, I'm going to an AI health coach, I'm trying to run a marathon for the first time, trying to lose ten pounds, or I'm going to a customer support bot for a tool that I use and I need help right now. Or I'm going to a bot to, you know, it's a private tutor and I'm trying to learn something that's difficult for me. So just wanted to ground that in the very real, like product or functional example. Cool.

Yeah. You can say it all comes back to recommendations, right? Yes, if, if you're recommending something to a user and it could be correct as far as they're concerned, but if it doesn't meet their worldview and it doesn't meet their. The way that they see causality and change, then they're less likely to adhere to the recommendation. And yeah, you know, like, so it's all about, you know, you model the user's causal view of the world, you meet them there, or you help them evolve it, you know, through an updated belief system. That's. That's what's going to make a great recommendation system.

One human example I just thought of as you're saying that everybody could relate to this. Have you ever been with somebody upset and you're like, maybe calm down? Like, would that. Do you think that calms people down? No. Fuck no. Anybody that's ever tried that? Hell no. So that's what a conversational recommendation system would be like. How do I meet this person right there, right now? If you're going to say that, you're probably not going to have a good time. The mirror sort of thing on the product AI business landscape is if your agent doesn't give a good recommendation, then you might not retain the user, they might not refer the agent or product to somebody else, they might not convert to buy whatever it is to up the value ladder for the products and services that you serve to that user. So I just wanted to ground that in a very real thing, especially if. Oh, another one. If somebody's being defensive. I've done this before, like a fucking idiot. Stop being defensive. You think that's going to work? No, it's not going to work. So there's a level of that human subjectivity of what is the right thing to say right now. The recommendation engine in our own heads, we learned this in interactions in the real world, hopefully most people. And then just mirroring that, that's. That's sort of. Yeah, just to say that's all. It's all down to the recommendations. Completely agree, Jonathan. Okay, so we've talked about belief systems. We talked about why creating representations of the self, the user and belief systems and an engine to evolve that over time is very useful to get to the causal structures and to approximate reality as perceived by that user and in the context of your AI experience or agent or LMM application or chatbot. And we also talked about epistemic evals, which I'll add a graphic to the show notes for the podcast. I kind of think of this eval hierarchy of needs or hierarchy of eval needs for an agent to really understand A user starts at the foundation models, then application evals and then user centric and belief centric. And that's epistemic evals at the top. So that's what we do. And towards that we make agents better and bigger picture. We help with AI alignment for that interpretability component that Jonathan mentioned. I am a user. I understand the machine, the machine understands me. And it all starts with the beliefs. So now let's get into our building public updates. Jonathan sent me pretty refined thinking on our last update to the product roadmap because he's been chugging away at the engineering portion of this. So I think where I might start, Jonathan is just asking you for a quick update on where we're at with everything. I don't know if you want to pull up the GitHub repo or if you got the demo available to kind of show where we're at, but. Or even just a brain dump might help. And then we can get into some of the brainstorming that you wanted to get into and just discuss that life. Does that sound good, Jonathan?

Yeah. And two kind of like areas of thought I wanted to share as an update and one is that bridge we talked about where we talked about a developer ICP to use epistemic me as a tool for personalization recommendations. And so I had, I built our self management agent into something that could like, you know, I guess the trick is to say, hey, if you can do evals like in the best practice way like they're taught in this course, then we can like drop some little breadcrumbs from that and then have them using personalization recommendation tools on top of it. And so I like, you know, how do we make that easy? Right? And so I, I sent you this, this roadmap. And I guess one thing I realized was that the course that we took, in the coursework and the homework there's only examples of evaluating single turn evaluations. And then of course there's examples of doing retrieval evaluations, retrieval augmented generation, but there wasn't examples of doing multi turn evaluations. And Hamill did include this as a chapter in the textbook. It's like maybe just three pages long. It's pretty brief. And if we step back a second and we talk about the personalization use case, it almost always is a multi, multi turn evaluation. So that was one thing I thought, how can we sort of take where the recipe chatbot homework was, which is a single turn solution, and then introduce that layer of multi turn evaluations. And I've been working with multi turn evaluations for several months now and there's a lot of tools for that, but it's still, I would say, an unsolved problem. And in terms of ways to do it like fluidly and elegantly in a production pipeline. Yeah. So in the single turn example you take like a sample.

Jonathan, just for the audience because I realized they might not have taken the class and they might be hearing single turn, multi turn and some of them, I think some of the audience might be deep into agentic building. Could you define that real quick just with a quick example with an AI health coach. What's a single term?

Yeah, this is familiar to everybody. Which a single turn is like ask a question to the LLM, I get a response. So there's, that's one turn. A multi turn is like I'm asking the LLM multiple questions or follow on having a conversation with it or it's asking me questions and there's, there's like turn after turn. There's a whole thread, you know, and so like a thread chatgpt is a multi turn conversation.

Great. Or a therapist or a doctor asking you multiple questions to get to a level of I'm ready to make you a recommendation. That would be an example of a multi turn. Cool, awesome. Thanks Jonathan. Yeah, please carry on. You were saying kind of a general unsolved problem in an elegant way in the space right now. And it's, I think you were going to connect to that idea.

Yeah. So when you do a single turn evaluation, you're trying to like create some inputs that are going to sort of stress test the boundaries of like, you know, does it work or not? Like in the recipe Chatbot example, you might give some examples of like dietary restrictions and make sure that it's adherent to that. You might get some examples of like, you know, food that's under a certain cost, make sure it's adherent to that. Right. And so you're trying to come up with sample input queries and then say the output and then say that it like and that's the gulf of generalization is like how, how broadly of the input queries can you test to make sure that the LLM responds correctly? Now with the multi turn conversation it's a little trickier because there's so like, you know, you can basically what do you consider the input? You know, you consider the entire conversation to be the input and then you know, like you want to make sure that the user sort of gets what they want to out of the conversation. So it's a little bit different than this, like sort of exact. Like my prompt's doing the right thing. It's like my prompt does the right thing over the session and over to the thread. And so in the textbook, Hamill talked about that, like how, how they do that on a team. And so it's the same. You have SMEs, you know, subject matter experts that review it with you. You're still looking for like, is this acceptable or unacceptable? But in order to vary the input parameters, they'll sort of like take short conversations and long conversations and sort of like try to approximate. You know, it's like a net, not a scoop type of testing approach. Right?

I like that it's a net, not a scoop. Or it's like fishing with a net versus like spear phishing. Right? Really, really quick. You, you actually shared this screenshot from the course reader. Let me just read it to the audience. This is a little snippet that Jonathan pulled out about this multi turn sort of problem or dilemma that we're going to talk about in a sec. So we pay special attention to failure modes that correlate with position of the conversation. Many systems behave well in the first two or three turns, but then begin to degrade either by forgetting context, ignoring instructions, or contradicting earlier responses. Our test sets include conversations of varying lengths to detect these issues. So that's one solution to that. We also balance holistic and granular evaluation. Session level scores help us track overall system performance. So that's that session net level that Jonathan was kind of just saying. Granular level or span level analyses help us debug specific weaknesses. So that would be the spear phishing sort of analysis. In practice, we often combine both. We identify failing sessions first, then we perform deep dives on a handful of representative traces. And I also wanted to share so Hamill, by the way, audience, we're going to throw all these awesome user resources into the show. Notes. Hamill has this question from the FAQ that he created from the course. How do I debug multi turn conversation traces? So a little bit of this here and I suppose since I'm already here, I might as well just read this out from the experts. So first of all, start simple. Check that the whole conversation met the user's goal with the pass or fail judgment. Look at the entire trace and focus on the first upstream failure. Read the user visible parts first to understand if something went wrong. Only then dig into the technical details like tool calls and intermediate steps. When you find a failure, reproduce it with the simplest possible test case. Here's an example. Suppose a shopping bot gives the wrong return policy on turn four of conversation before diving into the full multi turn complexity. Simplify it into A single turn. What is the return window for product x1000 if it still fails? You've proven the error is not about the conversation context. It's likely a basic retrieval or knowledge issue you can debug more easily. For generating test cases, you have two main approaches. First, you can simulate users with another LLM to create realistic multi turn conversations. Second, use n +1 testing where you provide the first n1 terms of a real conversation and test what happens next. The N1 approach often works better since it uses actual conversation prefixes rather than fully synthetic interactions. But it is less flexible and doesn't test the full conversation. User assimilation is getting better as models improve. Keep an eye on the space. The key is balancing thoroughness with efficiency. Not every multi turn failure requires multi turn analysis. I just wanted to share that because I know we're basically just talking about this now that I've kind of provided some of that context. Jonathan, do you want to layer in some more of your thoughts on the question you kind of posed me in terms of how might we solve for this multi turn thing for developers?

Yeah, and I think, you know, one of the things you see in that answer when you read it is that there's kind of this hidden complexity that there may be retrieval calls, there may be tool chain calls that are necessary for the conversation itself to be acceptable. And you know, like, so you have a lot of variables that you're dealing with when you're just reviewing an entire thread and you're saying is this acceptable or not? How do you decompose that? How do you analyze it? And they gave a number of like failure analysis sort of toolkits to be able to do that in the class. But the, that, that managing that complexity was definitely top of mind. And that's where like for one I found some sort of actual intellectual sort of like affordance I guess to approach the problem and thinking about how the working memory of the, the conversation is being used. And so like in, in the example where we have like a single turn, you're just. The working memory is just the input of the query, right? So the input from the. And then when you have a multi turn conversation you have this, the context window that, that starts to fill up with the previous parts of the conversation. So you have the, the query which is the new the in, you know, and then you have the N minus 1 context window or like you know, all the other terms in the conversation that fill in the context window in working memory. And then from there you're supposed to, you know, do your retrievals, do your tool chain calls and you have input output from those retrievals and tool change calls also filling up the context. And so, you know, like, for me it helps to think about like, okay, as we get to this complexity, if we look at the provenance of the AI agent as we're using it, like, as in like what is the exact ledger of everything that is done and how does it fill up the context window at each point starts to become a more important part of like how we stress test the input to expect a certain output, if that makes sense.

Yeah. So you're saying, I think, let me take a step back to say this at a macro level and then at a micro level for this use case to reflect my understanding. And I might be totally off base, but this is what I'm hearing, correct me if I'm wrong, is input complexity into the eval will help with efficacy of the response. And input complexity could be the query itself, it could be all the tool chain calls, et cetera. It could be the, sorry, I should say the previous conversations, it could be the entire conversation terms as it is now, like the n minus one stuff we talked about. It could be, here is the timestamped user's belief system and their model of self at this point in time. And so you're saying there's these layers of memory. I think you use the term working memory. So am I right that it's really the complexity of the response? And for this multi turn use case, it's like the complexity of the input for this eval would be what did they say so far? What have they said before? And then for us, I think what we summarize is everything that's happened before as your belief system that feeds into this for the eval, does that kind of make sense or am I grokking that the right way?

Yeah, that's the, that's totally general idea. Like we're doing a bit of context engineering here now, like shaping the context of like what, what is the input now that the ll needs to respond to it in plus one in the next message. Yeah.

Oh, I like that. N plus one. Yeah. So real quick, just because I love grounding things in human relatable things, there's, there's actually a metric out there that I'll reference right now where I forget who it was that found this and who the researcher was, but I'll add it to the show notes. But they discovered that without defining too much what happiness is or positive interactions are or negative interactions, they found that for a person to be happy they need 3 to 1, positive to negative interactions, generally speaking. And then what they found is in a relationship where they're in love with a partner in the partnership itself to be happy, there's actually a biased weight where it's now five to one. I now need five positive interactions in this relationship for every one negative interaction in this relationship to be happy. I think anybody in a relationship or a long term marriage, whatever probably understands that. Just the feeling that your partner's there. Maybe you take them for granted, but that's something that they found in research. I thought that was pretty cool. So I'll also say for me it does get annoying when I've used ChatGPT for a long time now. And then when it starts crapping out in a multi turn conversation, I'm like a little miffed. I'm like this isn't the best user experience. And then I angrily smash the new chat button. I'm like this shit forgot me. Whatever. New chat. Okay, so now think about another real world example. Like I don't know my therapist, we've had eight and a half years of a working relationship. She has lots of notes of who I am and she has good mapping from her professional lens on my value system, my belief system and my goals in life. So when she's asked me questions or mirroring me to help me get to my target self, which is a less upset self or remove limiting beliefs, she's using all that context from all the years that we've been working together. And so there's different layers of memory there in that interaction. There's the session level memory, I'll just say from lack of a better word right now, before we started getting rigorous definition of what was Robert saying in the last session and then in this session. And then there's like maybe the relationship level memory of every single interaction that's ever happened to all of our therapy sessions and everything I've ever told her from the eight and a half years of our history of the relationship. So I just wanted to say I feel this in real life and when we're talking about these things technically from like a, let's quantify subjectivity, let's figure out the belief system modeling all these really just map to. We want to make that agent experience like that, like these really great human personalization experiences. Right. So just wanted to add that that kind of human context or use case example.

Jonathan. Yeah, that's a great example. So because we were talking about like the recent history like your thread in this conversation. And like in the chatbot, you need to be accountable for that as like as a developer of the chatbot you need to be accountable for like what, what the users told you just recently. But then there's this like what did I. I've met from my therapist for eight years and I have all this, this record. And so whereas that recent history was in work is in working memory, you need to remember it. The what you talk about in your history of your experiences is going to be episodic memory, meaning that we're going to store like so in order to personalize, you know, we need to sort of like how has this user like acted in these contexts before and what's relevant to like what they're doing right now? The more we can know about them, like the better the personalization is going to be. So now like that kind of touches on working memory and epistemic episodic memory.

Totally. So I think here is probably a good part. Where should I share my screen and show that Google Doc and then we can define these different memory layers to map to the functional use case we just talked about.

Yeah, sure.

I think that would be useful for the audience because we're introducing some new terms. We also talked about context engineering which I'm going to poke at you to define in a sec. But let's talk about the memory stuff. So what everybody is seeing is. Can you see this? Well Jonathan?

Yeah, it looks good.

Okay, so this is, Jonathan's been working through the roadmap and he put together this Google Doc for me to look at a couple days ago of some evolved philosophy for our design and implementation for our mvp. So high level memory at the intersection of provenance at the intersection of the self model or the model of the user equals the bridge from basic rag to personalized recommendations. That's like the tagline. Let me just skip ahead into input output. Given an input query was an acceptable response and we were just talking about the input complexity which can inform the efficacy of the response. So do you want to just one by one define these working memory, episodic semantic and procedural memory with that use case of like the therapist Jonathan?

Sure, yeah. And, and this is, this is like the customer journey. Right. Like each of these L0L1 we're trying to move to incremental complexity, to bridge, to create that bridge to personalization. And the first example is like, you know, say in a therapist, you might ask a therapist like hey, I'm feeling sad. What are some things I can do? And that's all it knows about you is that you wrote this query.

That's it.

And that's all happens in working memory. Like what your agent knows is just what the query is. And then it tried the prompt is then sort of tuned to respond to that and give an answer. So the next level was with the input output with retrieval. And here you have the query plus you have the tool input output which in this case is a retriever which is going to like, you know, maybe it's, it's like a therapy list of like common reasons that people feel sad, right? They're going to, this is like the encyclopedia of sad sadness. And it's going to reference this and then bring back chunks and input, insert those into the working memory into the context window so that you have a query plus you have these, these like chunks of relevant semantic information that are pulled. And so there's in the semantic memory there's like a server with say it's an MCP server with documents that, that would be the encyclopedia of sadness. Is this in the semantic memory? And the procedural memory is like the tool definitions for like how we, how we like sort of pull and process that information in the context window. So yeah, it gets a little bit more complex with the rag example. And now, now we have a conversation. So this is the, this is the multiple turn conversation. And I think the only difference here is not now the conversation itself is part of the context of the working memory. So every time you submit the prompt like you have to know everything that was previously said in the conversation. And so that's where it's going to show up in working memory. The working memory gets a little bit more complex. And then like just like we just read off some of the practices that Hamill had for multi turn conversations. We'd use those here. But also there's, there's this variability of like the retrieval and making sure it's retrieving everything in context on each turn to like create the right sort of conversation. And so okay, then we get to the personalized conversation with retrieval. So we have given a conversation reference documents and tools, personal details about the user. What's an acceptable next response? And I give some context here. The self management agent initiates a user profile from existing database model and using epistemic me to create a self model, a related self model with belief systems. The personalized conversations start with the checklist of information. The agent must collect it from the user if it's not already correct collected. And we talked about those checklists in the past about personalization readiness. That should be pretty familiar. And this is where we start to introduce the. Into the working memory, the cohort of the user, the intent of the user, something we call the observation context. And that filters the possible leasing context. And so we're trying to determine in memory, like from a vast set of beliefs this user might hold, which beliefs are they applying to this moment. And we're trying to include those in the context window so that we know like what they're actively trying to like learn what they're trying to predict and how that works towards the intent that they have. The cohort is like a useful construct to sort of say, like we have a lot of users like this and they're generally trying to do this kind of thing. So we pair cohort with intent as a way to, you know, just like say this is who this user is and this is what they're trying to do. And that's the less the working memory. I'll stop there. Does that sort of reconcile with everything we've been talking about so far?

Yeah, and let me just ground that in my therapist example, because we're talking about it. But so for example, my therapist, if you think about her like an agent, let's say she has a hundred clients, right? So let's say she, you know, every day has a client list she's working through for therapy. And so imagine that these are all the different types of use cases she gets, right? Like here it's given one input query. What's an acceptable response? I mean, it gets a little bit more complicated because she's human. But let's say like somebody is just. I just email her a question like, hey, do you have any recommended books? You mentioned Carl Jung last time. What's your favorite book by him? That would help me with where I'm at. And so that would be like, okay, I know, like here's Robert's query and he's just looking for a book, right? Very simple. And then basically it just gets more complicated is what Jonathan is saying. And, and for that use case, how do we treat the complexity in such a subjective domain to objectively, systematically improve that subjective experience? That's how I think about it. And the way to think about this in the therapist example is I'll cross reference everything I know about Robert. I kind of know what his personality type is like. Cause we talked about that Myers Briggs personality type. We've talked about enneagrams. We've talked about lots of things. Oh, I know Robert's extroverted. I know he's Asian. I know he's from an immigrant family. I know that he's a slight perfectionist. I know that, you know, all these things that give a cohort, that give that context, that contextualizes my beliefs in that moment, so that I, as Robert's therapist, could give him the next best response or recommendation or question to help him towards the target self he wants to be at, which is maybe less stressed or whatever I'm facing at that given moment or conversation turn. And the other plug I'll say is this is where we start getting to cool stuff like epistemic evals. So just wanted to make a call back to epistemic evals. This is what we can help with, is that belief system context providing. Okay, sorry, Jonathan, just wanted to add that. Go ahead.

Yeah. So if you want to talk about the evals here that are, that are the epistemic ones, you know, first we need to make sure that while the user is in the session that we can actively understand their cohort intent, observation context and possible leasing context. So we're trying to build evaluation tools to make sure that we understand where the user's at and what they're trying to learn, what the information is that they're trying to get to. And that happens in working memory. We're also trying to build an evaluation system to make sure that in the episodic memory we have user experiences and that relevant user experiences are being pulled into the context window as part of the experience. So if, if a, if you're asking, if you're asking a therapist like, hey, I'm feeling sad today, this would go back to all the times that you've been sad before and all the times that the therapist has talked to you about like anxiety or depression or whatever. And it would pull those experiences in context and understand like what has worked for you in the past, you know, so it's a richer experience based, you know, so we need an evaluation for that. So these are, you know, new layers of evaluations. And just like we do retrieval evaluations on like reference documents. And so in the semantic memory we have reference documents, belief evidence. So this is evidence that the user is going to use for, to justify their beliefs. We have the user profile and the self model, and this is all stored in semantic memory. And we need to make evaluations there so that we have like a clean holistic profile of the user and that we can like retrieve information about them as it suits, like us answering the question and like facility the conversation. So those are, that's that's it. Those are the epistemic emails, the three layers. There's the working memory ones, there's the episodic memory ones and the semantic memory ones that all have sort of epistemic structure to them that helps us personalize for the user. And as we'll get to the next step, you'll see how we do it for recommendations.

Yeah, so L4, let's talk through that.

Yeah, we have a similar scenario here. And the agent recommends a plan that aligns with the user's beliefs about change and about their ability to adapt their habits. Right. We have the same, we have the same set of information and we're, we're really going to have. I think, I don't know that I indicated it here, but in semantic memory there's going to be like a log of protocols that say like for users with this cohort who have this intent, who have this sort of like habit and belief structure, what are the recommendations that have been shown to be effective in the past? You know, and that's where you're meeting the model of causality for the user is you're trying to say like, okay, we think we understand who this user is and how they're going to see the causal outcome they're trying to achieve. Let's recommend them a protocol that's consistent with that, it's belief, adherent to that and build a recommendation for it. And again, we need epistemic evals at all three layers, working memory, epistemic memory and semantic memory to make sure that we kind of like promote the right recommendation in this context of the user's end.

Yeah, and that's really important to get to true causality. Right. And based on what we were saying before with it all starts with the belief as the fundamental unit or building block. So yeah, hopefully that really helped folks grok our philosophy of agent evals. And then where do we intersect with epistemic me and epistemic evals? I know there's more to this. Do you want to, are there sections of this you want to go through? Should we go into the multi turn kind of philosophical question we have or problem we have?

Yeah, I think, you know, I think that's good to show, you know, how do we bridge from like sort of basic evaluation practices into personalization recommendation. And this is, this is our sort of approach and how to use epistemic evals to get there. I think that's a good overview.

Yeah, this might be cool to talk through.

The auto rendered mermaid diagrams are not the prettiest but yeah, this is so as in the evaluation cycle, then you can think about, you know, at the highest level you're, you're doing functional dialog tests and working memory that can be your whole conversation in level turns acceptable or unacceptable. When something doesn't work, you've got to drill down and say like okay, what's failing? And that's when you're going to do your retrieval evaluations. That's when you're going to do your evaluations on your episodic memory. You're going to evaluate your procedural memory, make sure that tool calls are happening when they're supposed to. And then on the left hand side I have this evaluate with ledger where that's where I thought that surfacing provenance in the working memory. And again provenance is like when you use ChatGPT's reasoning models or like when it tells you what it's doing as it's figuring out your answer. That's just all like it's partial reasoning, populating the input of the working memory. And so we just want to use that as a tool to like evaluate what's going on in this complex kind of LLM tool chain. We can either hide that or show it to the user, but we can use it. And when we're doing dialog tests. So this is. Yeah, this is the plan. Sweet.

I like this. The ledger acts as the GPS coordinate of any regression. If metric drops, the ledger tells us which memory call or tool called it. So it's like an audit trail, right? Makes a lot of.

Yeah, and it's like, I think normally provenance in these evaluation tools is part of the trace and it's part of a diagnostic tool. And I'm saying like just put it in working memory. Put it. You know, like let's just evaluate it like as the user would see it.

Yeah, no that, that makes a lot of sense. Okay, where should we go next?

Here.

Cool. Okay, so show me what you have so far from the last week or so of building. Now that we've level set on working memory, semantic memory, episodic memory and also grounded it in that mental health therapy use case and sharing how epistemic evals and epistemic me intersects with all this to provide value to the developer.

Yeah, so I've been building on the self management agent interface and this is in the GitHub repo where the self management lives. Still a private repo, but we'll open it up soon enough. There's a lot of pull requests happening actively here. And let's See this is the interface for the self manager agent. It starts out like we said, kind of like a client portal idea that you can come in here and use the self management agent to connect the users in your database to epistemic me and then use evaluations to train through the personalization recommendation steps. So we have a quick, we'll call it our AI therapist maybe. Yeah, gives the therapeutic of you. Let's see.

And for people, as we're going through this, we're right now taking on three fast follower customers and canvassing for customers after bringing AI Brian to beta. And what you're seeing in front of you is our current take on our vision for what a client engagement would look like if we were going to help you make your agent experience or LM applications or a applications or chatbot applications systematically better with a focus on application evals and then the epistemic evals we talked about. So this is the wizard where we would set you up and then get domain experts in and do open coding, that type of stuff.

Yeah, and so I'm in the project setup wizard now and this is where we can add team members if we want to invite subject matter experts or other developers on our team. And I'm just going to skip the project objectives right now. Those are going to be used in the agent later on, you know, and so we have this kind of like standard template. This is the customer journey that we're trying to fulfill, you know, from. And these are all the steps of evaluation that are needed to like get from basic evaluation through, through the personalization recommendation here. So yeah, and I think we invite stakeholders in, then they can, they can also annotate through open coding and leave annotations or assess failure modes. So this is just a way to like kind of practically manage the data model for that. We have different phases here that like explain what we do in each phase and who's involved. So that's kind of like the customer journey from the project perspective. And then we have some different tools here. Like we can, let's see, what do we define the prompt actually? Well, this is one, this is one test interface where you can like test out. Oh, there's the prompt configuration. Cool, cool. So I have some here already for like customer support. There's not one for AI therapist, but we could have one. But the idea is just get going quickly here, like the code reviewer prompt. And I'm trying to make it easy by generating some sample queries around these prompts so that you can quickly test it and just like know that it's what's working. And. Yeah, so this should kind of like. It's funny. This is there. This one's for engineering related questions. And we asked it like trouble logging into my account, but you get the idea. Then we go into the evaluation.

Is it mixed up on the. We picked a templated software engineer system prompt. But on the right. I don't know if it understood that we picked that and maybe just defaulted to customer support queries.

Yeah, yeah. There's a bug here where it's loading the sample queries from the customer support. Yeah, yeah, yeah.

Okay. Gotta add that to a ticket or a backlog. We're doing it live. Yeah, Cool, cool.

And then we go into the evaluation step from there. And so like once we have. Is open coding analysis. And so like once we have our prompt and we have some sample queries, we can start open coding analysis. And this will like, this will take the sample queries and generate three traces from it that we can then use to annotate and review. And so this meant you can just run experiment, experiment and sort of like collect the data set. So what's happening here is that there is a query and then there's a response. And we as the evaluators are reviewing this. So we can review and say like this is too wordy or something like that. And here we have the existing failure modes and if there's one of these, we can sort of like select it, save the annotation, go to the next one and do the same thing kind of over and over. This is incorrect. Yeah.

So could I give a quick comment on this? So sure. One of the biggest things that we learned in investigating and researching this space is the whole goal, the moat for AI applications are application evals. Like what we're kind of showing you here. We believe the deeper moat for hyper personalization sensitive areas like health, longevity, mental health, personal finance, education, that type of thing is going to be the epistemic evals. So you need both. And what we're kind of showing you here is open coding or error analysis or annotations or labeling, whatever term you might use. We'll continue with the term open coding. That's what we learned from Hamill and Treya from this AI evos course. The goal is if it's a verticalized moat, meaning I have a problem space, it's longevity protocol focused, or I have a problem space, it's customer support focused for, you know, telecom chatbots, or it's customer support focused for, you know, like hardware devices Like, I don't know, gaming laptops from Asus or something like that. So this is what I mean by verticalize motes or eval moats. Okay, so back to this. What we learned from the experts and thought leaders is that the moat, how big your moat is and how good the moat is, is a function of the throughput of getting your domain experts expertise into the agent or the software system into your eval lifecycle. Open coding is the way to get that throughput. So what we kind of started with, hey, you can invite members of the team, domain experts like other AI health coaches or mental health therapists or in this particular case a customer support support specialists for a given vertical and you would want them to increase throughput. So the KPI here, the leading indicator of success here is how much throughput am I getting? Like how many of these traces can I get through as a domain expert and how well am I annotating these traces? Those are sort of the two levers we have. The reason why I bring this up is an important thing is because here, even here, like spending a lot of time on the UX and thinking about oh, how do I increase that throughput? I think that's actually going to be where we could provide a lot of value. Imagine, you know, you're the expert, you're not super with it with AI tools or software, you're coming in a newest system and then you're asked to annotate these things. And so we want to streamline that as much as possible where maybe on your first interaction you don't even need a human to walk through what you got to do to transfer your expertise. So that's like how do we get the self service? The other thing would be, well, how do we get them really productive where first 10 minutes, maybe they've done 10 annotations, next 10 minutes they do 20. Right. So it's really that throughput I'm thinking about and how do we get them as productive as possible to get that throughput? With that said, one of the things I was going to ask is I love the gaming gamification like progress bar. I think that's good. We learned that from the course. I like the suggestion, the failure mode. So it's easy to just click. Maybe we should critically think about like how do we make this even more understandable for one of these domain experts or if that's not a direction you want to take it. That's something I was kind of interested in.

Yeah, well actually like I was hoping we'd have time to get to like, I have a whole solution for that and I have a whole interface for it, actually.

Let's fast forward then. Go ahead.

Yeah, cool. And so, like, where we left off, we, we did some annotations here on the open coding, and then we go to the axial coding where we've, we're, we're analyzing like, what failure modes do we find in our data set and then, you know, what key issues address. And this starts the actual coding. And then we can take the, the manual annotations that we did here and create new failure labels. If the one that's existing failure labels don't like, sort of modify what's actually going on. Like, I can call this new failure mode. Great. And so that's going to create a new failure mode and then we can apply that to our data sets going forward. And so. Yeah, and that's the basic kind of flow. And I think you can picture like, how, you know, somebody in our course that's doing like the recipe homework could sort of like jump into an interface like this and use it that way. Very cool. So then I started to think about, and, and this is, this might take us longer than six minutes. I don't know how much time you have.

Yeah, we can keep going. I got time.

Okay. You know, like, where, how do we, you know, how do we quickly show, like, improvement and like, get to the rapid, the iteration, right? Because you want, you want this kind of like analyze, measure and improve loop to be like, really fluid in the user experience. And I think this is a good start. And you can, like, you can go back to your data sets, you know, and like, create data sets for each experiment and compare them with one another. And I thought that would be one way to surface like, that the failure modes are reducing, the quality of the prompt is improving and you know, kind of update the prompt that way. But what I ended up, like, thinking about, like, how can we really constrain the problem so that we can do open coding and axial coding in a more effective way? Because, like, at the end of the day, we just need to get to the failure modes labeled on the data set. Right. And I thought, well, can we constrain the way that each of the prompts is written so that like, we can, we can do this more effectively. And I ended up coming up with a system here that I'll show you the, the output, but then I'll, I'll backtrack to like, you know, how I kind of got here, if that's cool.

Yeah, let's do it. Cool.

So this is, this is, this is like a hierarchical constraint approach to like managing each of the evals. And the idea here is that you're looking at a complete set of everything that you need to evaluate for the system. And here you can see like the number of queries and users that each one has. And we've, we've used the context of like, you know, here's the agent that we're billing, here's the cohort of the user. This is a health enthusiast. Here's the intent class that they're trying to evidence research. Here's the category which is exercise and here's the sub intent which is going to be meta, meta analysis. And what, what we're doing here is we're like creating a meta or a system prompt and the prompt is like you are exercise evidence analyst for health user, health enthusiast users. The user is seeking evidence research specifically about meta analysis review. And then we put in some specific constraints here and these constraints can be used to like generate the failure mode. So you can think of the constraint as like a tool that if it's out of bounds then that's a failure. Right. So it gives you explicit failure modes and so it lists out the constraints and it lists out the evaluation metrics and lists out the failure modes here. And this is an approach that I thought we could sort of gamify that you just like as you build the chatbot, you can just have each of these evals that you need to make complete and then it trickles up the success percentage and the amount of coverage. So the coverage is like the gulf of generalization, the success is the gulf of specification. And so you have kind of like a complete interface here so that you can tune the evaluations to make sure they work and then you can go and test it at the highest level, which was like kind of the loop we just looked at in the Word document.

I love the kind of the hierarchy audit view. Like it's just so pleasing to the eye. I don't know, like it just, it helps me understand the almost the reasoning. Right. Which is very helpful. Like I like the visual, I like these numbers and I think overall like that's my gut reaction is oh cool, this is a highly subjective space. I'm some sort of SME that needs to go into this highly subjective space to understand how the chatbot performed in this multi turn thing. How do I make any of this better? Yeah, that's really cool. Could you explain these top numbers nodes? Is that the number of nodes in this hierarchy and then constraints. Could you explain that a little bit better?

Yeah, this is really just a proxy for the number of nodes that you have to evaluate. So in this case like we've broken down, this is specific. Like if we do randomized control trial analysis as a way to answer a question about exercise for our health enthusiast. Yeah, like we have to get this passing and then they're saying like there's two failure modes on this one right now. So like we can, we can like look at the prompt and then we can go and do open coding on it or do failure analysis for it. I haven't built that part out yet. And yeah, like you just want to get all these passing and then you want to go and test at the highest level and then do like the whole multi turn conversation about the health coach level.

Got it. Could I easily do N minus 1 and N plus 1 test from this once I do a little bit of science, so to speak.

That's the idea is like this kind of like frames the test that you're going to do through the open and then you can, you can do open coding results. There's kind of like this. Yeah, we can, we can pull in a data set here, the health coach evaluation, you know, and it like this is a little bit, it's not really doing this processing but it looks like it is. So it, it's evaluating the data set against the known failure modes and it's bringing this back. And this kind of assumes that you're doing LLM as a judge. Right. And then you'd come back and you look at the hierarchy and you'd see where you need to make further improvements. So the idea was to like see if we can start with this interface and get people in that loop of improvement through failure analysis, review and updating the prompt and like just like get the whole thing working all as one component. Sweet.

Awesome. This is so cool. I want to test this with some of the agentic builders that I've met and just see like, hey, would this help you with your work in systematically making these application evals and now these epistemic evals better for the agent experience. So I'd be hyped to get this demo working on my laptop. So once I go into calls with leads I could share with them this and get you know, really good feedback from those from our icp.

Yeah, I wanted to show like how, how I got here from. This is my chat with Claude about the three Golfs and LLM development.

Cool.

And you know, I Was basically just asking like, you know, I've been thinking about approaches to the chatbot and starting with constraints to the specification. And like I was thinking could we just use a semantic router to like, route to like real more simple prompts so that we don't have to like review 150 traces maybe you only have to review 10 traces, right? And then you get the prompt working, but you have a bunch of these little prompts, you know. And so the, the client did a really good job of saying like this is where the textbook sort of supports this idea for constraint based design and how it's using dimensions and tuples, spectrum of agency in different, you know, sort of. And then, then it gave like a plan. But it said at the end the user like users don't follow constraints and constraints feel artificial. I was like, I agree with that. But then I talked about like our experience building this functional health coach and the constraint that the idea of the coach is to help each user become a devolved version of themselves. That's one constraint. And we have a cohort, we have an intent. I pulled out the intents that we learned from the don't die users. Yeah, right. Plan the task automation, opinion research and evidence research. And then, and like basically Claude really like thought these, these set of constraints kind of filled in the, the issues that it had and it created this hierarchy structure here. And so I think this is a, this is like really kind of promising approach. And the whole thing revolves around like instead of doing open coding to create your failure modes that you, you do open coding to define your constraints and you define your constraints thoroughly so that your failure modes are like, you know, something that has a very verifiable, acceptable or unacceptable basis to it.

So is the idea being get the experts first in a room and have them critically look at the constraints. Is that the idea? And then that leads to better failure mode analysis and therefore a better agent experience downstream. Like we kind of flip instead of like get the agent in the wild, get users, get them to bask a bunch of shit. Now it's more like domain experts that have been health code and longevity coaches. They probably know the patterns. So let's look at constraints first and then we can get to failure modes. Is that kind of how the audience should kind of grok this?

Yeah, I think it's a good overview. Like for instance, if we're putting constraints around our cohorts and we say we have a health enthusiast here, you know, we can generate constraints like they have a regular activity baseline they have intermediate knowledge and they have a goal oriented mindset. You know, whereas the sedentary beginner has constraints of it. It needs to have simple language as low barrier to entry, focus on habit formation. You can see how like, like, as we move down to evidence research, like doing evidence research for the health enthusiasts is gonna be different than doing evidence research for the sedentary beginner. And we just carry that, that constraint along with us, you know, in that process.

Yeah, yeah. And you know, another thought is like, day one of my first therapy session with my therapist of eight and a half years, it was, I was a different cohort of a person, right. I had a different set of beliefs. In fact, when I first went to therapy, I didn't believe in it. I had a very limited belief, I had a very toxic masculine belief. And then that got shed over time as a limiting belief. So yeah, this is, I really like that. Which turns the whole paradigm on its head, but it should almost right. Size the input complexity philosophically. Right? That's ultimately what we're getting to here.

That's the nature of the experiment. It's like, yeah, kind of reduce scope and use constraints to like reduce the input complexity so that each time we're doing evaluations, we don't have to collect as many samples and we can get to something that performs reliably and then we can combine them together and then all we have to do, like when, you know, when we were a scene of something, like we use like a semantic router here to say, is this a health enthusiast or is this a sedentary beginner? And then we route them to the right query, right prompt. And I thought Claude did a really good job of constructing the system prompt given the context that we have here. So this is, this is the context engineering approach where.

Wait, real, what is context engineering? You sent me an article on context engineering and I was like, shit, there's a new word I gotta learn. There's so many new words. Six months. So could you explain to the audience context engineering real quick? I didn't get a chance to dive into the article.

Yeah, I think it's the sort of intuitive aspect of it. If you have like a long thread on OpenAI or Claude and you want to start a new thread, it's going to lose context, right. And like, you know, you know that the new thread that you ask is like not going to respond the same way that the previous one does. So that problem is the same in like agentic design and like, you know, prompt engineering. Is that how do you make sure that the context is available to the agent that needs to make the response? You know, and in this case, like we're sort of making some design decisions here so that we can carry across the prompt context of like the role and the context for the prompt that's needed. And like, so the type of AI coach that we are is passed on, the cohort is passed on, the intent is passed on, the intent class and so on. So that like by the time we're doing like a meta analysis review, we have all of this context available to us in a formulaic way. Yeah. And. And then I feel like the constraints is even further context engineering where like there's a formulaic way to, to like create the constraints at each level so that you can, you can test this prompt, you know, and you have each of these constraints then carries its own failure modes that like are testable against. So you can collect less examples over here in order to like create an LLM judge that can evaluate this effectively.

That's really cool. I like the idea of. It's sort of, I don't know, the word modularizing evals comes to mind like where we're really feeling this. Right. And I love it because in us introducing our concepts around belief system modeling, user centric and belief centric evals with these epistemic evals, it does afford us to challenge the paradigm and rethink. Like, is this the best way to increase throughput of domain expert expertise to the agent? Right. And the main goal or sorry, the goal of that is first principles. We want to make sure every response works for every user. Like there's high efficacy of response. So I like this idea that you're pulling this thread with, with can we turn this on its head and start modularizing and providing right sized context? So that's really cool. I'm excited to share this with some users to see what they think of it and how like whether or not it might actually help them. The user, the ICP agentic builder that is tasked with either building from scratch or making an AI agent or chatbot systematically better. So really stoked to do that. We'll find time where we maybe get together for 10, 15 minutes and you could help me get this set up as a demo on my end when you have it at a good milestone for yourself. Anything else you want to say before we wrap that conversation on epistemic me? Agent builder roadmap, the multi turn conversation complexity problem. Some of the dynamics in which you're thinking about this paradigm for solving for minimizing the amount of traces we need to maximize our evals process. Essentially.

Yeah. I think, you know, what I just showed is kind of experimental, but it follows this philosophical principle that the axiology should be constrained or contextualized.

Right, Yeah, I mean, well, I was about to say something kind of far out. Wait, do you want to finish your thought and then I'll add that in?

Sure, yeah. Well, yeah, and I think what I mean by the axiology in the case of evals is getting to the failure modes. And so, you know, like, yes, one approach is to like do 150 traces of an unbound prompt and then create the failure modes from it through like axial coding. But like, so the experiment is like, well, what if we have a prompt that's like greatly reduced in scope and can we take like five to 10 or three different examples and then identify all the failure modes? Because the constraints are defined up front. Yeah, so it's the same kind of philosophical tool that we use. We're talking about beliefs in oxology. Yeah, we'll see how, we'll see how it all comes together. But I feel like it really makes the jump from like the simple case with like, here's a simple LLM and here's a simple prompt in rag to the more complicated, you know, approach of like multi turn conversations and multiple tool chain calls. Managing Providence. Like, it, like it makes each of the components of it very manageable. And so, yeah, I mean, you know, we've been working on this for a while. So this is synthesizing a lot of like things I've tried in the past and trying to just make it like a manageable, gamified experience. We'll see how it comes together.

Yeah, yeah. The thing that I was going to say is because we talk about philosophy a lot, we read it a lot, we're, we're nerds about it. I mean it's called epistemic me for a reason. We're talking about epistemic evals. Like, shit, we're philosophy nerds. So the reason I really liked what you said is about axiology and subjectivity. So the hard part here, like when I talk to VCs or builders in the Valley, like startup founders, like building AI, the hardest part is if you're an engineer building this stuff, is you're used to getting a prd, then you could extract ground truth and then build it and then cover for ground truth. This is a whole different ballgame. And so our ICP is in this new world where it's like Shit, we got to figure out a scientific way to systematically make this better. But it's subjective. And the subjectivity, that's the axiology piece, right? So what is good or bad for what my therapist should have said? Fuck, that's really complicated. So what we're trying to do here, hopefully the audience you really see, we're putting a lot of thought reps into that particular problem where a developer is used to ground truth and now they're in a space where that is no longer truth. It's all subjective. And we kind of want to give them a systematic way to get through that uncertainty to get to ultimately, I'm still improving the agent in a scientific manner. The reason I like this is because. So the far out there thing I was going to say to relate to the audience is the categorical imperative by Immanuel Kant. So think about that. So he talks about how just the thought experiment of just think about, well, if everybody did this thing, would it be good for the world or bad for the world? So would it still exist or not? And maybe Jonathan could add some refined thoughts to the philosophy here by Kant. But what I recall of reading from Kant, from the categorical comparative is, well, is lying bad? And it's like, well, everybody in the world lied. There would be no truth. So, yes, don't do that. That's the categorical imperative. So you have that thought experiment. So another one is like, well, if I had a choice between saving my mom or I don't know, I'm in this terrible dilemma where I've gotten to my head, prisoner's dilemma or something like that, save my mom versus save like a thousand strangers or something. It's like, well, that's very hard to answer. And then it gets to, well, what if everybody chose their mom and then everybody had that experience going on at once and then we all die? Well, that's probably not good. The categorical comparative is a useful, I think, tool in philosophy. And it was a very useful concept when Kant came up with it. But it really speaks to, I think, the need for axiology, which is it's not so binary. At least my philosophy is that some of these thought experiments are like, can't call myself a good person or bad person if I want to save my mom. At the intersection of the categorical comparative. Right. It's hard for me to make sense of that. And so that's when in postmodern philosophy, there's this concept of axiology, there's a spectrum. Good, bad is a spectrum. Evil, whatever. It's a spectrum of things. So subjectivity is a spectrum. It is no longer deterministic ground truth for engineering. It's not zero or one binary. Yes, no, true, false. It's. Everything is at the center of axiology or subjectivity or the product. Folks will call it taste nowadays. I just wanted to add that when you were talking about axiology and how we're trying to basically share that philosophically through our design for the agentic builder to get them from old world to new world, Kant came up. Yeah, I don't know if you have thoughts to add to that, Jonathan, but definitely philosophy nerd fucking moment right there.

Yeah, it's pretty funny to bring categorical imperative into it. You know, I, I don't. Philosophically, I don't spend a lot of time on, on morality or axiology as morality. I think if Kant had like the metaphysics of Carl Friston and Michael Levin, he'd be like, oh, the categorical imperative is just like metabolic homeostasis and staying alive. Yeah. Like, it's kind of like what Brian Johnson's saying is like the categorical imperative is don't die. Right.

Yeah, yeah.

So, but I think like, when I say if Kant had the metaphysics of, of Michael Levin, I think that's. Then we have rich subjective context behind like, every single, like, value judgment that, that we make and we can. And like, you know, metabolic homeostasis is a rich value judgment and like, so for the most part, like, that's what we're doing when, when we're like, choosing to act in any particular context in the context of like, what is actually going on here. We say this is, this is an actor that has like, limited cognition. They have an ancestry on, on this planet. You know, every time you include the details of phenomenology and being into the categorical imperative, like, you know, it puts a whole new meaning onto like the subjective experience.

Definitely. And this is where I will link that story of philosophy in 30 minutes, whatever, into our show Notes. It's a great video. Me and Jonathan go back to it from time to time. We've talked about it on the POD a lot. But yeah, I totally agree with that. And I also agree that we need a good integration of all these things. It's not just like categorical comparative is the way. And I like what you put with balancing that with Christian or Levin's metaphysical point of view is. I don't want to say better, more balanced, I suppose.

I think it's more evidence of what the world is. We just had was it 300 more years than Kant had to collect that and create new models. Yeah.

Yeah. Okay, cool. We've been going for a while. Anything else you want to say before we wrap the pod on two of the things that we went over today? One is epistemic evals at the intersection of AI alignment. The second one is that stuff feeding into our desired user experience for our ICP agent or AI builder to help you with that subjectivity problem to systematically make your AI better application evals and user centric belief evals or epistemic evals. Anything else you want to add to that for wrap?

Jonathan I think we covered it. It's just building from here, getting a great user experience together.

And by the next time we probably do the pod, I'll have some feedback from real users. Our ICP because we sourced a bunch of leads so now we could really show them our prototypes and stuff like that. So awesome. Looking forward to seeing how our philosophy on this design evolves and how the design itself evolves. Jonathan, great work.

Great.

All right, I'll end it there. Thank you everybody for listening to the pod. If you're watching this, listening to this, I love getting comments from everybody. I started receiving some emails from folks if you want to join. We have a slightly private Slack community. It's invite only. If you want to join, drop us a comment or shoot me an email. Robert systemicme AI if you're a builder that is interested in being on the ground floor of a net new market defining categorical tool, epistemic evals. Epistemic me. We're open source. We're wanting to get builders in at the ground floor to help you build the best agents ever. And also if you're part of this, you're really chipping away at AI alignment with us. Hope you had a great time listening to this and I hope everybody has a wonderful day. Take care. Bye bye.