Can an AI Doctor REALLY Understand You?

Why do AI evals matter for health and longevity apps?

With AI in health and longevity, we're getting closer to an understanding of the user that a doctor could never have had in the past. And what I mean by that is you think about all the information, all the data that you carry around as a patient. You have these biomarkers, you have physical capabilities, you have everything that your Apple watch or Garmin is tracking. You have just all of this information that a single person couldn't really look at and understand all these aspects about you in a way to like personalize an experience. You're talking about a team of doctors, maybe a specialist working together. I think what AI can do is help synthesize that, create a model of where you are with your health today and where you want to be. And a lot of the medicine and things that have worked in the past, like RCT trials, randomized controlled trials and vaccines, things that have been proven to give population level health benefits. We can now start using some of the same tools on an n equals one basis. Sort of like a biohacking view of who am I coming into this health questionnaire? What are my longevity goals? How personally can I adopt new protocols in order to advance those goals and get answers to questions right away that don't require a doctor or a specialist. You have a team of doctors available to you. And what evals do is they make sure that all that domain expertise that would come across the specialty specialists and the doctors is being presented to you correctly. But they also understand that we have an opportunity here to really take all this big pile of data and create a model of you that's going to be predictive and reproducible from patient to patient or user to user. And that's, that's the future of personalized AI healthcare experience in my mind. Hey everybody.

We're doing a new video series. Pretty excited about it. It's about testing AI products, maybe even doing lives with the community where we could test AI products live. You could let us know ahead of time which ones you want us to test. And we really focus on the personalization aspect of the AI and talking through things like evals and error analysis as we go through the subjective user experience and critiquing it from a product taste lens as if we were users using it. So this very first episode is about Superpower AI. It is a health concierge progress on that job to be done. For me, coming into the Superpower app and talking to the AI concierge, this ask Superpower AI, one of My jobs to be done might be give me the next best protocol recommendation for me at this point in time based on my data. One question I might ask, what are the obvious things I should be doing to improve my health? And let's go ahead and ask it. And Jonathan and I, from epistemic me are going to critique the agent response and experience. So already what I'm kind of seeing in real time is this is a lot. The animation is very fast. There's no way I could have read this. And I see that they have like a simple markdown packaging for the information. You can see that like 1, 2, 3. It also looks like within the 1, 2, 3, they have chosen for numeric lists only. And no bullets. Just make it a little hard to read. Let's start at the top. Based on your biomarker data, here are some key areas to focus on for improving your health. I'll explain each recommendation and its connection to your data. First off, I like that it's explaining, hey, everything past this sentence tells you that we are trying to get your job done, which is, I wanted a recommendation around my N equals one use case, my biomarker data. So I like that it did that. Let me go to Jonathan for initial impressions. Any initial impressions, Jonathan, we're going to slowly scroll through.

Yeah, I'd agree with you said, you know, didn't consolidate a brief set of things that, you know, is easy to walk away with and implement. But more than anything, what I'm drawn here to is that when you say me and my health. Right. It's looking at you and it's seeing biomarkers like it doesn't really see somebody that has experiences and, you know, like, so it's not talking about your energy levels throughout the day. It's not talking about the food that you eat or the sleep that you get, you know, and some of the things that are just like foundational and critical. It chose not to answer the question in terms of behaviors. And it's not looking at you in terms of behaviors. It's looking at you in terms of these specific biomarkers. And it considers all of health to be changing these biomarkers. And, you know, that's just kind of like maybe like a neurodivergent way to look at a human.

Right.

Like you wouldn't expect. So I guess there are some general lifestyle recommendations, but those are general. Right. Those aren't personalized.

Well, and then also what's interesting is they didn't lead with this. Right?

Yeah.

And in the longevity space, what you learn is pretty quickly the Pareto principle applies, which is how do I improve my longevity strategy? Sleep well, exercise well, diet well. And so you get 80% of the value doing the sleep, exercise, diet thing and optimizing the depth of that. And so I actually, for me as a person that understands that, that's my belief based on all the information I've consumed throughout longevity science is I would have liked this personally and we are in my account. So I wish the system had a way of recognizing that's where I'm at in my longevity journey and really recommend this first. But not only that, like, I don't know, I mean, for example, it's telling me to aim for seven to nine hours of sleep a night. But what if I'm already doing that and then it just gave me a piece of text that now I have to read and I have decision making fatigue. It's like if I'm already sleeping well, I don't really care about that in this response.

Right.

I'm looking for again my next obvious things for my health. So that's, that's another commentary I'll add there.

Yeah, I think like the amount of sleep you got when you were taking these tests would be some of the most interesting context that you could have. Right.

Um, yeah.

But if you're getting six hours of sleep a night and this, these were your biomarkers. Well, maybe an easy test to do is to get seven or eight hours of sleep a night and take the same test again and see how they're different. And it doesn't even ask you where you're at currently. Yeah. Just like the, the model of self and even what the biomarkers mean are like very analytical way of characterizing this subject.

Yeah. It's like it saw a stat sheet and it was like, here's what you do based on a stat sheet, not who you are. Yeah. What do you think about this structuring of observation recommendation?

Yeah, I mean, I think this is close to how doctors are trained to evaluate. You know, that's just. That is like the epistemology of observing that you have a symptom. And in this case the symptom is that like your uric acid is like below the recommended range. And then it's got to recommend like a therapy or a protocol for you. If it had a pharmaceutical, it might recommend that. So I think it is, it is tuned to like the way that doctors are trained to see patients as humans. Which, you know, serves us to a certain extent, but not fully.

Yeah. And you know what's interesting? It's just like, well, I like that they start with observation and recommendation, but visually this is a little hard to read. Observation. For example, there's a key data point and I'm basically looking for as a user, is this good or bad? What is the data and is it good or bad? And from here I have to read a bunch of texts to then say, oh, it's above the typical reference range. So this could have easily been from the ELEM output. You could have evaluated, oh, the user wanted to make sense of data, or hey, we should give them the observation, help them make sense of the data, then make the protocol recommendation. So the UX could have been a couple cards that show me, is it a red card, green card with the number in the middle to just even say, is this. As I'm flipping through this, there's like 10 things to read. This might take me an hour to really, you know, really grok this and dive into it and maybe dive into literature if I want, or my particular recommendations to contextualize. But I would have liked to be able to just go through here and be like, what are the red things? And I'm going to zone in on those because you're representing data here. And that's what my job to be done here is, is I'm looking for the next best thing for me. So that's another UX critique. Another UX critique. This obviously doesn't know me because it's saying limit intake of red meat, organ meats, shellfish. I'm vegan. I've been vegan for years, so doesn't know me. This is a waste of me reading this sentence even like this could have just not been there. And my life would have been better using this, right? Reduce alcohol consumption if it had known me too. I haven't drank alcohol in about five years now, so this is also just more fatigue for me to read, like text fatigue. This is interesting. Stay well hydrated. Cool. Consider increasing your intake of cherries or cherry juice. That's interesting. I've never heard of that. But here I would even question why does this matter? And I would have loved like an evidence point, like maybe if you're going to give a recommendation, depending on the user, like maybe for me, for example, if you're going to recommend to me a more engineering or scientific oriented person as an icp, I'm more likely to evolve my belief system if there's scientific rigor around the evidence for a protocol recommendation. So here, if the AI knew That I am likely to take a recommendation based on knowing that I like that scientific river as a Persona. It would've maybe decorated this with where it got the information from. Like assuming they have some sort of rag behind this to evaluate, retrieve for this, address elevated uric acid levels, context, retrieve some information from that and then distill it into a singular recommendation bullet point. I would've loved to be able to click into this. And in fact, I guess while we're here, I'm just gonna ask it real quick for this recommendation, what's the scientific events that this will help my problem. And I'll just put in quotes and add that and see if the agent kind of contextualizes it.

Yeah, it's a great example actually because the epistemic evals, you know, one of them is do we provide evidence that support the beliefs of the user? And so this is where you're pointing out like, hey, you're. I'm open to understanding why cherries and cherry juice are able to decrease uric acid, but what's the evidence, you know? And you may be looking for scientific information. Somebody else might be looking for testimonials from a friend who did this.

Yeah, somebody else might be like, I don't even have to worry about that. I believe you because of all our previous interactions. You're my AI health coach. That's always done me. Right.

So, yeah.

So yeah, tell me where I could buy the best cherry juice. Can you link me to a store near me? And that's a conversion opportunity. Right. Depending on if you know where the user is at in their journey and what they see as progress on their job to be done and you have that context at the evaluation point for this AI response, then this could have even been a conversion opportunity. And right now, every opportunity that I see when somebody's interacting with the agent, it's either an opportunity to increase loyalty, retention, engagement, referral and obviously revenue conversion of the user inactive agent, but it's also an opportunity to lose them forever. Right. So you could see how that personalization aspect with the intersection of the episode of evals that Jonathan just mentioned could really affect the user experience for this AI application. So another thing I note is like I asked a follow up question and it is not giving me a signal that maybe the server's down or that it's thinking. So I'm going to go ahead and refresh this and I'll see if it remember. Okay, it didn't. I don't know what that was about. That might have just Been like a server error on the back end. So I'm going to go ahead and just try that out again. Can you give me scientific evidence for this response, this recommendation? Cool. Okay, now it's. It's still going at the speed of light, so. All right, so I couldn't. One thing there. I've noticed that other AI tools have adopted for more of a human to human sort of interaction paradigm where a human interacts with the agent as if they're a coach. Right. It the animation or the agent thinking having some sort of like I'm thinking reasoning. I've seen other apps have this. Something I just wanted to point out on the human interaction design on this is if a human is really coming to this as a coach, an AI health coach grounding in that coach experience. When a human asks a coach, probably want to do some thinking before. You just say whatever the next thing is on your mind. Right. You're thinking about who is this person? What should I say to them? How might I help them reach their goals with what I am saying to them? So I find that animations really help. So you could see here it just spat this out and the animation was really fast. I could have never read this. So there's that sort of feeling of like, you know, when you're talking to a person. It's almost like here's a bunch of stuff, stuff. And I'm just saying a bunch of stuff. An interaction design that I've seen other apps have is thinking and then a time increment like 0 seconds, 1 seconds, 2 second, 3 seconds or reasoning 0 second, 1 second, 2 second, 3 second. And that gives that sort of interaction where it feels a bit more human and it gives me time. Like it makes me feel like this thing thought about it before it gave me a response, but it just gave me an instant response. That feeling that it really thought about My use case is not quite there.

I think the design pattern you're talking about is called explicit prominence.

Oh yes. Okay, thanks for having that.

I totally agree. That's so helpful to understand because if it was telling you I'm searching these documents now and now I'm going to come back with an answer in a second. You're following along with yeah, yeah. Where it's coming from.

Totally. So let's go ahead and just fact check it for real. But let's just go ahead and look for this. All right, cool. This looks real. Cherry consumption, decreased risk of recurrent gout attacks, arthritis and rheumatism. Okay, cool. So that's Cool. It gave me some scientific evidence, which is what I asked for. It knew that I was asking about the cherry thing. I just quoted it and gave it to it. So it knew that.

Yeah. I think these type of searches, you know, like, perplexity is really strong at this kind of stuff. There's, there's a lot of places you can go. I think the opportunity when Superpower has your health data, like if it knew your current like diet for the day, like maybe it could squeeze in cherries in a very complimentary way in one of your meals where it doesn't like spike your glucose too much or something like that or.

That's true.

It's got a whole person picture here to work with. It can do a lot more than what Propel 60 is doing because of that. And I think that's the opportunity with the chat experience here.

Yeah. And that's really for AI health coach applications or teams that have that as a product offering or feature. That's really the moat part. Right. The personalization part is the moat. And that's why everybody in the AI space talks about how AI application evals are the moat in hyper personalization sensitive areas such as health, longevity, mental health, personal finance, education, those very emotional domains. It's just a bit more personal. This is where getting obsessed with the personalization of the response matters a lot and can affect your retention, engagement, conversion stories. So yeah, we just broke down a few ways that this happened, even on that first interaction. But we have like six more questions that we prepped. Let's, let's move to the next one. So let's ask the one that I think a lot of people will come to these apps like function health, don't die, Mito Health, Superpower. Like anybody going an AI health coach, longevity coach type thing, they'll probably be concerned with their biological age and how to reduce that. So I'm going to ask how can I reduce my biological age? Let's see what it says.

And the superpower, do they measure biological age through epigenetic test or do they do.

Oh, they approximate some sort of bio age based on. At least right now all I did was a blood test. So I just got a blood draw. So they approximated that based on the.

Blood draw might be what function health does The DNA and vino age, which is able to take from a blood draw and calculate. Yeah, it's. It's a population correlation.

Yeah, yeah. So I asked how can I reduce my biological age? It says reducing your bio, lifestyle and health strategies that slow down the Aging process at the cellular level based on your biomarker data here. Our targeted recommendation is to help you optimize your biological age and overall health. Cool. So just kind of scrolling through this. Improve your pace of aging. Your pace of aging is 109% indicating you're aging faster than average. That's really bad. Recommendations prioritize sleep, exercise, stress management. I like this one cuz sleep honestly is my worst. I know that myself. So I do like that this came first. I don't know if it really knew that about me though. I don't think it could have because I don't connect my wearable data to this. But me personally, I just know that's a problem. So I like that it came with sleep first. Let's go through this. Optimize nutrition. So it still has that observation recommendation flow with the numeric list. I don't know why this bothers me so much but it's like two one, two. It's just, it's visually just adds visual fatigue of where I should be a bit. Address key biomarkers. Okay, so it's talking about uric acid. Again follow the recommendations I stated earlier. Again here is where it represents data. But the data could have been packaged in more of a UX analytics kind of way rather than just text. Anything you're seeing here.

Yeah, I do like how it kind of goes through different factors and it looks as we understand this is the data that we have for you and then some recommendations to improve it. Yeah, I think like you're saying, you know the prompt engineering could do some work where there's just like a TLDR at the top. The biggest impact based on evidence that can move the needle and yeah make it actionable for the. So these are recommendations and I think the best test of recommendation is when the user comes back to the system and they accepted the recommendation, implemented their life. It's not really bubbled up to that level where it's easy to take away, put into your protocol, come back later, say I did these things. Here's my new test and kind of like work along with the AI.

Yeah that's where like we don't have the behavioral data. Right. Like so for here it's recommending that I regularly track my biological age and biomarkers to assess the impact of these interventions. Adjust your approach as needed based on results. But this is the part I want help on. Right. Like I want help on this.

That's why you hire Superpower for this job.

Yeah, I want to adjust my approach as we go. And I want you to be adjusting me and being proactive, preventative. Right. So ideally it's like this thing we, we go through this experience of you help me make sense of things, you help me create some protocol recommendations and then I like you said, accept some of them because you can't like I can't just change 10 habits at once, for example. So it's almost like I need a habit roadmap and then I need to check in with the coach for account and then I want to see how I'm making progress on that and how does that make progress on my job to be done reducing my bio age. Without those I don't have the feedback loop of I'm actually improving my bio age. So ideally here what happens is I do stuff and then I measure again in three to six months the same test and then I see did doing the stuff that this AI coach recommended, did that result in progress on my job be done reducing my bio age? So right now what's going to happen is I have to write all these things down and I have to make sense of which ones I'll do in what order and then I gotta backtrack and kind of map that scientifically myself to baseline data to an updated lab result in a few months after trying some stuff. So there's a lot of gaps to fill there because so I, I don't get my full job to be done complete here. That, that's what my takeaway is. Okay, should we move on to the next one? Let's do the here's a menu one. This is our thinking for the health Optimizer Biohacker, longevity enthusiast Persona. And there's many Personas. Just, just be clear, those can all be different Personas and within them they could have different psychographics and stuff like that. But let's just say overall that type of person that would come here, there's the overall higher job to be done which is I want to reduce my bio agency progress on that. But day to day I want a coach to help me so I want to reduce the decision making fatigue so that I'm getting the bigger job to be done in the micro moments. So what's a micro job to be done? Here's a use case. I'm Robert, I'm out at a restaurant with some friends for lunch and I still want to make sure every single action, including my diet, everything I eat, even if I'm out for lunch, maps to my bio age or at the very least I want to know at this restaurant, what's the best thing I could do so I don't hurt my progress on the job to be done? So I might ask it something like, here's a menu. What should I eat for lunch based on this menu? With the strategy to improve my biomarkers, I'm going to go ahead and just attach that. I have an example menu as a jpeg. I don't know, I clicked add, but it's not actually adding it. It's gonna be a little hard. All right, so I'm gonna try to just copy paste it in now. So I guess first off, it's not doing the job because I can't actually conveniently give the information a way that I would like, which is I'm at a restaurant, take a picture, upload, but now I gotta like do some copy copy pasting.

Try refreshing the page and attaching the first.

Let's try that.

Yeah, maybe like the. The first chat in the thread. That's a good OR two chat.

Yeah, I do like that. It persisted. What I said. That's cool.

Yeah, that was good.

All right, let's just try to tap.

It sounds like a little red flash on the top of the screen.

Yeah, I saw that too. Is that a. Here, maybe. That's brave. Let me try to. Okay, so the red flash is obviously saying something as an error, but I don't know what it is as a user and I'm just at the lunch place trying to get.

What's that? There's something. One of your extensions there, the red one next to the fox.

Oh, this is an AD block one. I don't think this is going to do anything, but let's try it out.

It's blocking something.

I suspect it's a pop up that says format not accepted.

Okay.

And then it just throws an HTTP status code that says it's not the right call or we don't accept that data. So I'm just gonna copy paste. Yeah, it's okay. So I'll just. How about the AI and I'll copy paste this stuff. So I just took a text copy paste of the raw text of this menu on the right. All right, cool. Let's. Okay, let's take a look. So based on your biomarkers, here's a strategy for selecting a lunch option to support your health. Talks about my uric acid again. Limit the red meats, focus on plant based. Okay. Kidney function. So, okay, so now, now it's giving me a overview of my biomarkers so that it contextualizes the Recommendations later. Next thing, recommended lunch option. Falafel plate. Why these are plant based. Avoiding red meat. Okay. Yep. Well, first of all, if it knew I was vegan, it doesn't need to explain this stuff. But the second thing is the yogurt dip might have dairy. So it's not. Didn't ask me about my dietary preferences or you know, even cultural. Like what, what if I only eat halal food? Even like that could be something it knows about me. I, I don't personally, but that would feed into the personalization story at hand. Okay. It's saying some things. If you choose a wrap, opt for whole grain. Oh, I like that. That's a good. That's like here's a piece of advice so that you still maintain autonomy and decision making, but you could optimize your autonomy decision making foods to avoid. I like that. Okay. One, one thing I'll say is if I'm coming to this and I'm asking for this, I just want the result at the top. I want like a do this and don't do this. And that's basically what it gave me here, which is do this and why and then here don't do this and why. And instead of parsing through all this text imagining I'm on mobile, I'm at the lunch place, I just want a big red don't eat and then a big green do this and then no more decision making fatigue. That's how I probably would have packaged this from a UX perspective. What do you think, Jonathan?

Yeah, I think you're. You're right on it should have dietary preferences from you already so that it could filter the menu. And I think it does a pretty good job of saying this is what's good, this is what's bad. Based on the biomarkers you have. I think it's still, we're still pretty generic, not too personalized and I think that there's something to be said about what, what types of food do I like to eat and what type of taste do I have that it could. No, because getting somebody to be adherent with a recommendation means meeting them at their preferences for today. Right.

I'm going to ask it why it didn't ask me about my dietary preferences. Dietary preferences are an important part of tailoring any recommendations? I apologize not addressing that earlier. If you have specific dietary preferences, blah, blah, blah, please let me know. My goal is to ensure the advice aligns with both your health goals and personal choices. Would you like to share any preferences or restrictions? Could you tell me about the AI evaluation you went through to make. How would you classify my intent and.

My original jailbreaking superpower?

I don't know. Let's see if we can. Oh, shit. It's doing it. All right, let's take a look. Intent classification. Interesting. Okay. I consider the kiosk. Your intent appeared to be seeking personalized dietary advice based on the menu provided, with a focus on improving your biomarkers. This falls under the category of health opposition actionable guidance. Okay, yeah, that makes sense. You were asking for specific recommendation while implicitly requesting that the advice align with your biomarker data and health strategy. Okay. AI eval process. Understanding context. Reviewed the biomarker data. I also consider the menu provided. Prioritizing biomarkers. I focus on biomarkers that could be influenced by dietary choices. Yeah, that's cool. I like that. I prioritize addressing elevated or borderline values to guide you towards choices that could improve or stabilize these markers. I like that too. Like heat map it. Right. Matching menu options. I analyze the menu to identify items that align with your health goals. Okay. Excluded options. Personalization. I tailored the response to your biomarker data, but I missed explicitly asking about your dietary preferences, which would have had added another layer of personalization. This is something I'll aim to address in future responses. Oh, thank you, Superpower AI Bot. Yeah, that's funny. It's pretty interesting. Okay, yeah. So let me ask you something else. Now that we're here. What else would you like to know about me to best personalize my experiences?

Yeah. This is what we call this personalization readiness, right?

Yeah. Here's all these things. Preferences, restrictions, health goals, lifestyle activity levels, daily routine, meal patterns, stress and sleep, supplement use, medical history. Yeah, this makes a lot of sense. So none of these were in the input before, but it gave me a ton of responses. So I think what would have worked well, similar to a real health coach, if I went to a real health coach and I said, hey, look at this menu. First time I interact with them, look at this menu. What should I do? And then the health coach was like, here you go. They didn't ask me questions. Well, they. They weren't ready to give me the recommendation like that. You can call that personalization readiness or recommendation readiness. And so I think what would have been nice is if I asked this. And then the superpower concierge AI asked me a bunch of these questions before. It gave me the very tactical advice so that it would align with what I want for my own health and longevity goals.

Yeah. For a preface, the responses saying, we can give you a better response if. But here's the response I can give you with the information I have.

I really like that. That's a good point. Like instead of just saying, hey, I won't give you anything until you give me more, you might say, hey, here's some generic advice. But if you really want hyper personalized advice or like just for you, could you answer some of these questions? I like that because then that's maybe what a coach might say is like, hey, you know, here's some generic stuff. But to really get into the weeds of this, I'd really have to know your daily routines, I'd have to know your dietary preferences and that kind of thing. That's.

Have you used OpenAI as a deep research question?

Yeah, yeah.

Where it won't even start the research until you've answered a preliminary set of questions. And I think that's like a nice prompting technique for serious recommendations. You know, where you actually have to go and gather a lot of information and you actually want the user to be responsive to the recommendation. Treat it like a deep research query. Right where we want to get this right for you. Here's the things we need to know up front.

Yeah, totally. So I guess anything else to. And I do like that I think most people would grok that like people listening, you know, use. If you're listening to this, you're probably already buying ChatGPT subscription and you probably already use deep research. But that recommendation readiness in that interaction where it clarifies more, that totally is in line with what we're talking about here for the health use case. Before we leave this anything else you want to talk about with the AI concierge experience, epistemic evals, what we do and personalization what we do. Anything else that builders might consider?

Yeah, I think this last query you did is really interesting because I think it's sort of like there's a lot of low hanging fruit that the system already knows that it could improve responses given this information. And so I think, yeah, it's really illustrative. I think we went through some of the key things, knowing who the user is, identifying their intent and getting that information up front so you can really provide a personalized plan that is as good as the information that you have can allow you to do. Yeah, those are all good touch points.

Yeah, yeah, definitely. Yeah. So we went over think about the user's job to be done entering in this domain for this as an AI health and longevity coach. So we did a deep dive into Superpower Beta, their AI concierge wishes in the domain or vertical of health, longevity, wellness, that space. And so we talked about the jobs to be done as a user or icp coming into here, I want to make sense of my data. I want to reduce my biological age. I want to know the next best step to do that. I want to know in this micro moment for this menu from this restaurant, what is the best thing I could eat given my biomarker data to meet my biological age goals. And so really understanding the user and their jobs to be done. You could see in our critique of the UX and the experience here and what the LLM generates as responses or even how it packages the different information for the user. It really should be grounded in that user centricity, right? If it knew I was vegan, it would have given a different response. Or if it didn't know my dietary restrictions or preferences, it should have recognized that and evaluated, hey, the next best thing is to ask for more information and you see like there's a lot of opportunity for improvement from a UX AI interaction perspective to get those jobs to be done progressing for the user. So this is a fun a case study Superpower team love y'. All. And this is just meant to be constructive and productive. And if you're an AI builder, especially working in the health longevity space, a mental health space where personalization is much more sensitive, hopefully this is helpful to you. This is fun to do. Give us suggestions for other AI tools that you'd love to do a case study on and we'll definitely take those recommendations and get those done. Cool. Let me stop sharing screen and then I'll close out. Thanks everybody for watching the ABCs for building feature. We did an awesome case study today where we double clicked, triple click, quadruple clicked and even Jailbroke Superpowers AI concierge in Beta. That was really fun. We talked about how you know AI evals and the best product experience tied to how well do you understand user, how well can you model them and their belief systems and that leads to better personalization in these agent interactions. We also talked a lot about just general thoughts about epistemic evals versus AI evals. What makes for personalization? Why is it a hard problem? Jonathan, anything else you want to leave on before we end it?

No, I thought it's a great episode. I think maybe in our future I see us creating like a grading sheet for the personalization readiness of these different apps that we that could be a fun like scorecard to give.

That's a good idea. That's a really good idea. I'll add that to my to do list. Make a little epistemic personalization scorecard rubric. I think that's a great idea. Cool. Awesome. Thank a lot. Thanks a lot for listening. And if you have any comments, please drop them on YouTube and go to our website if you want to get in touch. And if you have an AI application, LLM application and you're curious to get us to do a case study on it and give it some design UX critiques and epistemic evals lensed critiques, let us know. Awesome. Everybody have a great day. Take care. Bye Bye.