How To Build AI That Earns User Trust

Hi, you probably know that on this channel I talk a lot about evals, but there's one problem that's a lot more important than evals, and it's, are you building the right thing? And this has come up so many times that I think it's worth doing a special episode about. So let's jump right into it. What I'm showing on my screen right now is the kind of AI product a lot of companies are building, and it's a user asking some kind of question like, what was the revenue for a product last quarter, or any kind of other business question, and the AI goes off, it queries a database of some kind, it does some thinking, and it comes back and it gives you an answer. Seems really great, right? You don't need data scientists anymore. You can just get the answers to questions like this you're looking for, and maybe your business moves faster. But actually, this is not great. So you have to think about the user. If they get an answer like this, how are they going to know whether it's correct? How are they going to validate that this 4.21 million is the right answer? It would be a really dumb idea to take this and put it on a board deck and say revenue for product A last quarter is $4.21 million. You can get in a lot of trouble if you're wrong. So how can you gain more confidence in this answer? And to check this answer, you would have to redo all the work. You would have to query the database yourself, do some checks. And really, when you think about it that way, this doesn't really save that much time. We should take a look at this tweet from Lenny, which also highlights the same issue. Not enough people are talking about how AI is impacting the role of data science. So Lenny was chatting with his data science friend and he said most of this team's work is now reviewing half-assed AI data analysis from PMs and data engineers, and that 50% of the time the analysis is wrong. And so what we're doing a lot of times with these AI applications is just creating more work. Now we're in a place where creating the work output is super easy. The bottleneck is now verification. We have to design products with that in mind. And so if we ever you're looking at an AI application like this one and you're wondering how will you eval it, meaning how are you going to check that if it's right, and it seems like it would be very difficult to check if this is right, that's a product smell because your users are going to have to check if this is right as well. And so your product should be collecting evidence to support this fact and showing it to the user. So let's take a look at what that might look like. So if we were to reimagine the same application, it might look something like this.. I'm gonna show you this in two parts. So first of all, the user would ask the same question, what was net revenue for product A in the last quarter? But then in addition to the answer, which we had before, you would have maybe some supporting information. So let's take a look at it carefully. One is you would want to maybe base the analysis off of other analysis other people are doing. So in this situation, the AI is referencing a notebook., which we're gonna talk about in a second. And a notebook is just a detailed analysis that's already vetted, done by somebody else, or some other analysis that someone has already signed off on. This is optional and this is just an idea. Next thing after the notebook is stating the assumptions after the analysis. So if we look here in this section, we see that the AI is confirming the definition of the metric that we asked for. And this definition can come from a number of places. A lot of times companies have a semantic layer or some kind of knowledge layer that has these definitions. And you want to have this stuff be linkable so that a user can click on it and see what this governed definition is. And usually when there's some kind of calculation like this, a lot of times there's intermediate calculations. So you could see this chat interface is showing us intermediate calculations of things like returns, number of customers, etc. And it's raising issues that we might care about. And this is all illustrative. Your application might be different. The whole idea is that we should show supporting evidence on how this number is created by backing this up with some additional information so that you have something to base it off of and you don't have to start from scratch. So you might be wondering, what is this open as notebook thing? So that's just a UI idea. When it comes to data analysis, we should put yourself in the shoes of a data scientist. How would a data scientist check this number here. Number one, see if anyone else has done that same kind of analysis before. But you would also do things like check the definition, or you would check the intermediate calculations. And a lot of times you might want to do that in a notebook. How would that look? So let's just pretend like I clicked on this button here, open a notebook. What would happen? It might take you to a view that looks like this. This is kind of a very common interface for doing data analysis called a notebook. And there's different variants of notebooks. There's Jupyter Notebooks, there's Marimo Notebooks. But you would have a narrative of some kind, such as some kind of prose, and then you might have some queries, and you might have the same kinds of things like retrieving the definitions and so on and so forth. And you could scroll down in the notebook as well and see, okay, what are the more details in terms of what is the analysis the AI went through? By the way, this notebook can be completely AI-generated. And the whole point is, is you could step through the analysis in more detail You could edit it. Perhaps you could ask AI questions about the notebook if you wanted to. You could put AI in the notebook as well, and you could make sure that it checks out and it looks good according to what you want it to look like. And anything the AI couldn't verify, you can have it specify what it couldn't verify. So this is now a rich conversation that you have with the AI, and it's a lot easier to check. A data scientist doesn't have to start from scratch if they want to validate this number. They can go into this notebook and kind of read through this notebook and check the cells to see if it's good. And so now all of a sudden you're reducing the burden on the user. So this is a design pattern that you should pay attention to, which is how do you make things more verifiable? You want to provide intermediate calculations, you want to provide provenance. Where is the information coming from? If there's any kind of social proof. So for example, in this case, Somebody has done an analysis here, and this helps us gain more confidence that this information is based on or similar to something someone else has done. Maybe you can click on this and view it. These are all good design patterns that you wanna pay attention to. You might be thinking that this sounds complicated and it's not that realistic, and that this sounds like science fiction, but it's not. There are real products out there that do this kind of thing. One of my favorite examples of this data AI agent is Hex. Hex is exactly this. It is a chat interface that goes to a notebook. So I did not make this up. This is a well-established pattern of UI, UX, and progressive disclosure. So there's an element of progressive disclosure here where you get the user asks a question, you get an answer, but then you can explore the answer and then you can go over here and get more information. And you can keep iterating here with the AI. So this is an established pattern, but this is just something that we have forgot somehow when designing AI applications is, hey, we need to provide the user with enough information they need to gain confidence in what they're looking at. So let's move on to a different example. This is an example I help with, which is a physical education assistant. And this is used by kindergarten all the way up through high school of helping teachers that teach PE come up with lesson plans. And so that's what you see here. You put in some information here on the left-hand side, like what your grade is, what the class length is, where the location is, what kind of equipment you have, so on and so forth. You generate this lesson plan and it spits out this nice lesson plan here. So it looks great, right? The problem is, how do you as a developer evaluate whether this is any good? But similarly, if you're a user, how do you know if this is any good? You don't. Sure, if you're a teacher, you can kind of take a look at it. Oh, seems reasonable, seems good, but it's not as valuable as it could be. So a better example of this would be something that looks maybe like this. You put in your information and instead of getting just that curriculum back, you see if you can base it on some kind of existing curriculum. So for example, when it comes to physical education curriculum, there's a lot of established curriculum out there and What you wanna see as a teacher is what is everyone else doing? You would have more confidence in a plan if you could see trusted colleagues in other places that are using this and you could get an idea of what other people are doing. That would be a lot more valuable than just having AI spit out a lesson plan because it would allow you to gain more confidence and also allow you to learn what is going on. Like what are your colleagues doing? And so kind of constraining the problem a little bit and saying, Hey, this is a lesson plan exactly like yours, gives people a lot more confidence in what they're seeing. And then if you do use a lesson plan that is adapted from someone else, then you can now present the edits to that plan. You can say, hey, we're making some kind of edits to the plan based upon your inputs. Like we shortened it to a 45-minute class, we matched it to your equipment, so on and so forth. You can accept it, you can reject it. Again, this is a design sketch. You can go a lot of different directions with this. And you might be wondering, well, what do you do if you don't have any lessons planned? Is there a cold start problem here? What if you don't have this? And the idea is, okay, if you don't have this, there's a lot of different ways to get this. You can have experts seed a couple of plans in your system. You can try to do some work upfront to get vetted lesson plans. But if you do that, it's gonna make your application a lot easier to evaluate and a lot easier for users to gain confidence in what they're seeing. And I would argue this is just a lot better for the user. It's a lot more valuable as a teacher to quickly see what other people are doing. See, like, this is used at 14 schools, has been run 30 times. You can see, like, where it's being used. You can even go and view the original lesson plan if you wanted to. That's all really useful information. Even if you're not trying to generate AI, you just get to learn so much and it's really valuable. Sure, you may not be able to do all of these things., and you may not be able to operate in this perfect environment, but what you want to do is when you're presented with something like this that is just trying to generate something one shot and present to the user as an answer, you want to instead think about how you can present users with signals so that they can be more confident in using the AI-generated output. And the last example here is this other product I help with, which is a workers' comp report generator. What it does is it takes a patient's entire chart in their claim, which can be thousands of documents. And what it does is it creates an entire report that a doctor is supposed to use to either support or deny a patient's claim. And again, the question becomes, how do you evaluate this? These reports are often as long as 52 pages. So if you have an AI go through entire patient's chart, generate a 52-page document. How do you evaluate it? How do you know if it's any good? The doctor is also going to struggle to know if it's any good too. They're going to have to redo all of the work from scratch. They're going to have to go through each one of these documents here, and they're going to have to validate if those documents are represented correctly in this report. Honestly, that's not helpful at all. So if you think about it really carefully, what is the work that needs to be done here? The doctor first needs to start with research. They need to understand all of the documents in the patient's chart and all of the other documents outside the chart and get a view on how they relate to each other, what the key facts are, what the contradictions are, what the gaps are, so on and so forth. And you need like a research assistant to help you get a lay of the land before diving into the report, because as a doctor, you need to understand what you're submitting. You need to have high confidence. You need to think about the user's workflow a bit. And this is what it might look like. Instead of just spitting out a report, you might want to act more like a research assistant. What you could do is you could present more atomic elements to the doctor, such as any contradictions that occur, or what are some key facts, or what are some open questions. And you could allow the doctor to interrogate each of those and get understanding. Like when it finds a contradiction, you could open both pages or you could resolve the contradiction. If you see key fact, you could validate that the key fact, you could agree with it and include it, or you could dismiss it. If there's an open question, well, that's not something an AI was able to find, it's just a gap that they identified, and then you can add a note. And you kind of get the idea here is you have an application that instead of Spitting out a report helps you do the research first, en route to the report. And so again, this is gonna be way more helpful than just spitting out the work product. You always wanna think about what is the work that a user would normally do and help guide them along the path of that work so they have an understanding of what is happening rather than just providing a solution, especially when the solution is super hard to evaluate. Like there's no way a doctor just given this 52-page report can quickly eyeball and see if it looks right. And so you need to give the doctor more information and need to build up understanding alongside them. This itself is very helpful. Even if this system doesn't spit out a report at all, all of this understanding and information gathering, these facts is gonna be helpful, arguably even more helpful than the final report. So that's all the examples I wanted to show. So let's recap what we learned today. So there's a couple of things worth keeping in mind. Number one is provenance. What is provenance? That is showing where the information has come from. That can be either an analysis someone has already done, that can be documents that you're reading from, that can be another lesson plan that someone has created. But you want to show where the information comes from, and the more trustworthy that information is, The more curated the information is, the better it is because it'll allow your user to gain trust in what you're doing. A second principle is progressive disclosure. And what progressive disclosure means is basically show the details at the right time. What that means is you don't want to overwhelm the user. You want to show the details behind the calculation. You want to show what is informing a particular piece of text, what is informing a particular change. If you recall, when we had the diff on the lesson plan, we showed like why we're changing it. When we had the data analysis, we showed more details in a second step where you could open the notebook. So we didn't try to throw the user into notebook right away. So that's progressive disclosure. And by the way, progressive disclosure will change over time, meaning as your product matures or user matures, you will need to think about what is the right information to show that user given what they know and what your product is good or bad at. Another thing that you want to keep in mind is signals and heuristics. So when it comes to checking information, experts in various fields often rely on signals and heuristics to quickly check if an answer is right or wrong. So if you're doing data analysis, you might check like another report someone has done. You might do a smell test to see if a related metric looks correct. When you're talking about the lesson plans, you wanna know, is a teacher that you already trust, are they using it? So these are things like social proof, or it can be supporting evidence, et cetera. So you wanna think about these things, and again, you wanna show this to the user. And the last thing you should keep in mind is how can you make this more modular and break up the work to be done into smaller pieces that can be verified? So I'll just call this keeping things modular. Small steps. What I mean by small steps is, look, you can still do a lot of the work for the user. You don't have to have them check every single step, but just show your work along the way. Show all the steps, have it be available to the user. In some cases, you may want to force the user to be in the loop. For example, in the medical case, you might want to force the doctor to step through the information before generating the final report because there's just so much information. You The goal is to make the human understand, not necessarily to generate a report. If you think about the product really carefully, that's the whole entire goal. So I hope this helps. I wrote an entire blog post that goes through everything in a lot of detail with more interactive examples. You can actually click on these examples, you can scroll through them, etc. I'll put a link to this blog post in the description. Highly recommend checking it out. Again, I think this is a really important topic because I think a A lot of times people are building the wrong thing and they come to me for advice about what to eval and I end up telling them you shouldn't eval it because I don't think this is the right thing. Let me know if any of this resonates with you. Looking forward to your feedback.