Stop Managing AI Projects Like Traditional Software

So I've been talking to Hamel Husain for the past couple of months, and I'm Brian. I work at Theory Ventures. I'm the head of AI. I previously was the head of AI at Hex, and before that I was the head of data at Weights and Biases, and so on and so forth, and other companies and logos and blah, blah, blah. Okay, so I've been talking to Hamel Husain for quite a while about evals. And one of the things—

it's like all we talk about. Yeah.

Yeah. One of the things that we really sort of like agree on, I mean, besides the obvious, like you should have them, they're valuable, you should dig into them and stuff like that, is kind of like how to operationalize them. I, I sometimes worry that amidst the sort of like drum banging that like we need evals and evals are important and evals are all you need, that we actually kind of lose some of the message on like what do you do with them? And like kind of like how do you, how do you do it? And one of the ways that you can operationalize evals from my perspective is in road mapping and communication, in particular in sort of like determining what, how, and sort of when you're going to build things. So in talking to Hamel Husain Husain Husain, especially over the past couple months, one topic that kept coming up is sort of like roadmaps. And I thought that was a worthwhile investment to spend some time talking about and how it intersects with evals. I did some deep research and I couldn't find anything that kind of talked about the intersection between like evals and planning, evals and road mapping, evals and sort of like, you know, operations. And so that's what I'm going to talk about today. Now my talk is called Burn Your Roadmap: Why AI Needs Explorers. The reason for that is because I want to try to push you outside of the traditional product engineering, like, operations flow. I think road mapping and, like, traditional product development has a lot of really effective mechanisms and frameworks, but today I'm going to try to communicate why some of those are a little bit tricky for AI. So we'll get started. Uh, I want to talk about why product roadmaps fail. I want to talk about experimentation. We're going to talk about evals-driven development, and finally we're going to talk about how to analyze evals. Um, I think all of these should feel pretty natural, and you know, see, we'll see which ones make sense to you initially and which ones you feel uncomfortable with. As I mentioned earlier, you should feel encouraged to shitpost in the chat. I'm really interested to see your reactions and hear what you think. So let's talk about why product roadmaps fail. Traditional software roadmapping is really focused on time estimates for when specific features are ready and for users' hands. This is great. This is really valuable. Sometimes you'll hear things like the roadmap is like a, like a ledger. It's something that you can kind of transact against. It's something that you think of as sort of like a list of tickets that you can kind of bank on. And roadmaps actually aren't fungible and items on roadmaps aren't fungible. And just as a reminder, this means that they're not all equivalent and they're not all equivalent in value and they're not even really sort of like— you can't really exchange them for value in that more traditional sense. What's the way that people tend to do this? They often think of this as a sort of like, we're going to use this roadmap to sort of transact when we can do a marketing launch. We're gonna use this roadmap to plan some other releases. We're gonna use this roadmap to manage sort of hiring and staffing. And I think that is really great. It's really effective. It's really powerful. It's really important. However, it does sort of run into some big challenges in AI. And so like, before I move on, I just want you to pause for a second and I want you to think about like roadmaps done well. You're going to ship your new features, you're going to talk about which engineer is in which swim lane and how many, you know, sort of tickets we're assigning to each engineer this sprint. We're going to use that information because we're going to sort of check in at least twice a week to make sure we're on track. If we're not on track after a sprint or two, we're going to take a step back and reassess and sort of ask the question, should we cut scope? Are there things that we should abandon? Maybe we were overly ambitious. Maybe we weren't ambitious enough. And importantly, we've got some marketing dollars that we're going to go to waste if we aren't careful. And so you can, you know, you can think, you can put yourself in these situations. Maybe you've been in these situations many, many times. I know I have. And I think you can probably agree that, like, there's value in that. There's a lot of power in that. But I think there's some danger. Because when you're doing AI, and I'm gonna use the term doing AI, your goal is to discover new territory. Why would you expect the routes on your map to get you there? Remember, we're living in the age of intelligence. This is, this is all new. If you're building yet another, then maybe this is fine. If you were simply implementing the GraphRag paper, Cool. Then, you know, maybe you can put some, some roadmap around it. If you are sort of very aligned that sort of like the agentic flow that you read about is going to map to your problem pretty directly and you're gonna, you know, set this up as a very simple React agent and you kind of just need to fill in the prompts. Then actually you're probably fine. You could probably roadmap that normally. But I have a hunch that a lot of you are trying to do things that are new, and you are trying to do things that are not on the map. I know I have been. Now don't get me wrong, I've been doing my fair share of like good old-fashioned product engineering with some sparkling API calls under the hood. And that's great, like that's valuable and that's really amazing. But I do want to point out that some of the most exciting work that I've done lately and some of my colleagues and peers and friends have done lately have been pretty damn far off the map.

So, and by the sparkling API calls, you're talking about the little sparkly icon, right?

Oh, that's exactly, yeah, exactly. Exactly. If you don't have that, that's not it. So roadmaps are just very rigid, almost intentionally so. They're rigid to kind of incentivize engineers and sort of benchmark engineers. They're rigid to keep people aligned and in lockstep. If you've ever heard sort of like, I wanna make sure this team is delivering, like, what does that mean? Like delivering? It means like kind of like delivering on like a cadence that everyone's really excited about so that they can build other things around it, build structure around it. And so here you go. Here is your roadmap. You're going to deliver this amazing and novel AI feature. It's due on September 21st. And the reason is because I want us to commit to this launch deadline for these new features that I just came up with. It's been 9 seconds since they came into my brain and we are ready to commit. Maybe you've had the experience of being told when they're going to launch. Before you've had a single experiment or a customer conversation. And it reminds me of going to school. It reminds me of seeing that we're doing Lagrange multipliers on test 3. It's in the syllabus. I don't know what Lagrange multipliers are, but like, I guess I'll be learning that this quarter. Who in chat knows what Lagrange multipliers are? This is, this is your bonus quiz. Let's see who can win, who can win points for knowing Lagrange multipliers. And yes, you can win points in this lecture. Nobody.

This feels like a real situation, Brian. It's very— I don't know, it just feels real.

Yeah, yeah, you know, it's best when you speak from the heart, as they say. Chris, you have a math PhD, you don't remember Lagrange multipliers? Come on, come on. If you've ever taken a qual, you know what it is. Okay, okay. Can we use AI? No, this is not open book. Um, what if I told you That Lagrange multipliers are very similar to gradient descent. Get it? It's a joke. So you may say right now you're feeling, no, no, no, these are just bad product roadmaps. This is not what great product managers think like. You may say, are the great product managers in the room with you right now? I'm very fortunate. I actually do live with a great product manager, so I do get to say that they do exist with confidence. But despite that, and despite your rejoinder that I'm fixated on bad product roadmaps, I actually think this is more common, this way of thinking and this approach is more common than you think. There's a popular O'Reilly book on how to product manage. You may know it. Called Product Management in Practice. What does it have to say? You will likely have more success trying to make those Gantt charts better than you will trying to convince people to abandon them. And this is a really This is a really like popular and honestly kind of beloved book on product management. And so it's, it's a pretty endemic perspective, I would say, that like these product roadmaps that are strict and rigid and hard, they're very, very important. And it is definitely difficult to pry them out of people's hands. No matter how many engineers push back on them. I'm not sure I've experienced that, Louis. My experience is that in product-led orgs, sort of the roadmap is still quite rigid, but you know, I think in theory I kind of agree with you, and in practice maybe not as true as I'd like. And again, the problem is you can't roadmap what hasn't yet been discovered. That is the point. And so if you believe that we're doing novel things, if you believe that we are sort of going off into the, like, the proverbial, like, horizon, then you should be anxious. So what do we do? We experiment. And experiments start with a question. Great experiments are driven by hypotheses, and hypotheses are questions. Hypotheses, you might have convinced yourself, are guesses. But really, They're questions. What do you wonder? If you don't wonder anything, what are you hypothesizing about? So let's do some examples. Can we find relevant tables for natural language to SQL? By the row values inside them. Compare that to feature build keyword index of row values. Okay. I could go build a keyword index of row values. But really at the heart of this is like, I want to know the answer to, can I find the relevant tables with this kind of search. Does data extraction work better before OCR transcription or after? You want to get some structured data out of your PDF? Neat. Well, that's simply infra: new transcription node in extraction pipeline. Huh. Okay. Okay. Does import— oops, sorry. For some reason they're bound. Does important judge work better on summaries? That's that EPIC, summarize conversations. And finally, should we re-rank or interleave our RAG results versus 7 meetings with vendors about their re-ranker and how good it is?

So people actually do that. People are having 7 meetings with vendors about RAG technologies. That's, that's the way they're going about it.

I know, I know. Hamel, can you just, out of curiosity, Hamel, would there be any incentivized reason why vendors would potentially want to tell you a particular approach to solving your recommendation problem?

No, no, because they're, uh, you know, they're, they're charities and they don't have any incentives to, to sell you anything. So of course, like, yeah, I can't figure out why.

That's good, that's good. Yeah, I would never, I would never want to sort of rely on the objectivity of the sales. So, all of these are questions. All of these can be foundational pieces in your roadmap.

So, I have a question about this. Is this sort of directives that you have to reframe as questions or does it deeper than that?

I almost think it's the other way.

Okay.

You have to ask questions. You have to have genuine curiosity. You can't be too obsessed with like make AI done, finish AI feature. You have to ask yourself like, what are you actually curious about? Sud in the chat says define better. Sud's on to the next page already. Because the next page is actually we can do this even better. Just providing the questions is not enough, and it will open you up to the most obvious objection. How will we know if it's working? And so next, you also have to add your metrics. What is our table recall? For those of you that haven't spent time with this, recall means the percentage of queries where relevant results are included.

In retrieval.

How do the accuracies change for each extracted attribute? Accuracy usually means the percentage of the time that the correct value or an equivalent value is in, is, yeah, is in that, like, keyword. Is the binary accuracy on this classification task higher? Asking a yes or no question is the gold standard. And finally, does the rank of the necessary resource have a non-trivial effect on the generation? RAG systems return results in order. Does that matter? And so you see, we started out with some questions, but the questions are sort of like an aperitif. This is where you're starting to get to like the main course. You're asking, what is the sort of explanatory power of those questions? How do you turn those questions into something operational? But actually, we're still not done. Because you need to estimate. Those metrics. And this is the thing that we're all very afraid of. Now you start talking to data scientists and every data scientist is like squirming in their chair. No, please don't ask me the effect size. I don't, I don't know. Do we have a baseline? Do we have any other experiments that we ran? I don't know. And that's fine. It's really freaking hard. But you gotta try. Sorry, we correctly retrieve the right table 73% of the time on queries. We think this approach can have a single-digit percent improvement. We currently extract 40% of the attributes correctly. We think that this could add 10% accuracy across our dataset. We currently have a false negative rate at 7% for conversations. We think this could lower it by a whole percent. Where RAG succeeded, Generations fail 15% higher when the relevant doc is central in the retrieval. We think re-ranking could put them first 60% of the time. And so now, you're starting to have things that feel a lot more actionable. If you really—

These are not like just random guesses. These are like, you can get these intuitions by doing some data analysis. Like exactly the 60% of the time is not just, oh, I feel like, no, you can do some small experiments to get that intuition.

Exactly. Exactly. And these are like pre-experiment experiments. These are before you go and really do the experiment in full, you have some small like, kind of like micro experiment to test, okay, here are the cases where it's failing 15% of the time. How many times does it put the right thing at the top? So these are, these are sort of like free work before you commit to this work. You know, keep in mind, one powerful thing is this matrix that people like to draw where they're sort of like ROI and effort. And when you have ROI versus effort, you need to be able to put the points on here. So you do need to be able to say how difficult is to try this new table method, how difficult is it to try this different sort of like structure for the pipeline. And that's going to give you this Y-axis here. But this other axis, this ROI axis, you need that too. And these are giving you that axis. This allows you to put points on this plot and then answer the question, well, if this is really high ROI with super high effort, maybe not right now. But look at this, this thing's huge ROI and not very much effort. Get to work. And so this can really help you with your prioritization. And if there's one thing that PMs do love, it's having really honest conversations about priorities, having sort of the technical team come back and say, this is how we think prioritization should happen and here's why.

Okay, so Rohit's asking a question about— okay, he's saying, does having a pipeline of data or a certain table constructed to have rework populate help, or is it just posturing? I think he was asking, is the way I'm interpreting this is, uh, you know, do you need to have an eval harness in place or eval system in place? Like, when do you, when do you do that? Like how important is that?

Yeah, I'm gonna talk a little bit about like analyzing and setting that up, but, um, I don't think— there's two ways I can interpret the word posturing. One way of saying the word posturing is defensive. I'm in a defensive posture. I want to do things so that when people criticize me, they're the wrong one. Another way is to sort of posture for, I would say, like Agility. And so if things change, you're rapidly able to shift your weight to where it needs to be. Those are both types of posturing. The defensive posturing, unfortunately, we do need to do that sometimes in the workplace. It's a bummer, but I'm not gonna bullshit you and say like, no, you should always be like, everyone's always has best intentions. Unfortunately, people have different prior— like, internal motivations and internal goals. And so I, I can't tell you that you never need a defensive posture, but a lot of this, from my perspective, it enables you to have like an agile posture where you can rapidly pivot and feel confident in those changes. One of the things that's really, really popular in experiment-driven road mapping in things like traditional, I would say like R&D, is that like plans change and you are pretty okay with that. You should be pretty okay with that. It is stressful. I'm not going to shit talk or bullshit you and say, oh, let the plans change every day, no big deal. It's super annoying. But the one thing that I can say that makes it feel less annoying is having stuff like this, having lots of experiments, populating your experiment database with lots and lots of questions and ways that you can measure those questions— sorry, questions and ways that you can measure those questions. And then also how much effect you think they're going to have. So in my opinion, that can help you feel more agile. Okay? Great! I assume you know how to productionize this all, and that's enough, right? We're good? Hamel Husain Hussein— any comments or considerations?

I have a strong feeling this is not the last slide.

Why do you say that?

I don't know. You kind of had this humor about you that was funny.

Hmm, interesting. Is there anything that maybe I didn't talk about that I don't know, it feels absent.

Yeah, like how do you set yourself up for success in doing this extra question, question, uh, asking, answering exercise?

Hmm, interesting.

Yeah.

Okay, okay. All right, so yeah, um, sadly we We've got some more work to do. It's great that we have questions and it's great that we've got those metrics and it's really nice that we would like to say how big an ROI they have. But Hamel Husain kind of already exposed me a little here. How the fuck am I going to estimate the effect size if I don't have any evals? Shit. Alright, let's, let's get into it. We need to be doing evals-driven development. Now, I don't necessarily like love this terminology. I think there's some, some hobgoblins in this, but it's, it's good for now. That's good for now. So before we go any further, I want to talk philosophy. What do great evals do? Trend on Twitter. But besides that, besides that, great evals do not answer the question, does my AI capability work? That's like kid shit. That's kindergarten. Okay? Great evals show you which of your assumptions are false. I want you to think about the capability funnel. When you are trying to build a successful AI capability, something that is at the application layer that is feeding users, you need to overcome a series of steps or assumptions, however you wanna call it. And the final stage, the very last one, is did the AI succeed in the job the user hired it to do? The job to be done. That is the very last stage, though. That is at the bottom. Before that, we have several stages that we have to work through. We have to fight our way through many levels of hell. And that is what I call the capability funnel. Let me give you an explicit example. I would simply build a natural language to SQL bot. Okay, me. I think, I think the first day that LangChain came out, it had a natural language to SQL capability bolted onto it. This is about the most obvious thing that everyone wants to build. So how do you measure it? Well, you could measure, I typed in question, I got back 7, answer 9. Eh, you could do that, but what do you do with that? What do you do with that information? It gives you nowhere to go. It gives you nowhere to look. It doesn't edify you at all other than it failed on this task. But here are some questions. Can it generate syntactically valid queries? Can it generate queries that execute without errors? Can it execute— can it generate queries that return relevant results? What about results that are— match the user's intent? What about ones that solve the problem? So if I ask you, What is our ARR? Well, if it returns SQL that's like syntactically valid, neat. But does it actually error when you run that query? Maybe the join doesn't work. Okay, cool. The join does work. Are the results relevant? Maybe it returns the ARR for all the individual contracts, or maybe they're from last year. So the results may be irrelevant or not my intent. But most importantly, what is the problem that this is solving? Because if I ask you, what is our ARR? And you come back to me and you work at a hot new AI startup, and so clearly the ARR is $100 million. Over the past week. Um, and so you come back and you say it's $100 million. Well, that's great. I'm, I'm really happy that you answered the question right, but does that solve my problem? Why did I ask the question, what is our ARR? It's very unlikely that I just want to know $100 million. I probably want to know it over time. I probably want to know it broken down by segment. I probably wanna know it in a way that I can actually make use of it. Users often fail to provide the problem that they're trying to solve. One of the most amazing things about OpenAI's Deep Research products is it fires back almost immediately, give me more information. Here's some things I think you failed to specify. Now, I'm not gonna lie to you. Sometimes it's like, I think you failed to specify these. And I'm like, I don't want to specify, leave me alone, just answer my damn question. But I'd say 9 out of 10 times, I love those follow-ups. They're super valuable and interesting. And I'm almost immediately like, oh, actually, that's a great point. Shit. Like, I didn't even think about that. I recently asked this question that like was mostly about like industry movement and stuff like that. And it comes back, it's like a particular industry and its growth over time. And it comes back and it's like, do you care about the cultural implications? I'm like, fuck, yeah, I do now. So you can really get a lot of power out of trying to solve the user's problem beyond what they specify, what they communicate. So sometimes, you know, even that's not enough. But, you know, this is mostly pretty. This is mostly pretty. Wade, where do you start the evals? You're building an agent system with all kinds of different pieces for which evals should probably be created, routing, RAG, etc. So you start with evals for the specific bits, or do you start with evals for the final results? Well, where's the value? There's no value in the final result evals. You can't do shit. What am I going to do with it? Like, these ones work and these ones don't. Okay, moving on. Like, you have no, you have no chance. You got to start with the pieces. You should always be assessing every component of your system individually. And then when you ensemble them together, which is a fancy word for putting them in the same bucket, like when you put them in order is where things start to unravel, where you start to see this sort of picture emerge of where things are broken. If you told me, Brian, I only have time to build evals for 2 of my steps or the whole system, I'll take the 2 steps every time. I always, always, always want to know things that I can action on, that I can take some sort of next steps towards.

Yeah, I take an interesting approach to that. I always do an end-to-end to get a sense of like where to zone in, and then I zone in.

Interesting.

So, uh, it's a little bit of a— yeah, it's maybe it's like advanced I don't know.

Sridhar is drawing on my screen. Okay, well, let's see. Lewis asks. Yeah, I think— I mean, Lewis, I won't claim that it's easy to get people on board. But I think there is a— I guess the question is this: what are you trying to sell them? Are you trying to sell them that the product is going to work? Because that's a lot of risk. You're taking on a really intense risk profile there. I'd rather try to sell them we are going to make progress on improving the parts that are most impactful. And you're gonna see how I do that in a second. Ben, skill issue. You gotta try harder. You gotta think about how to break it down into individual pieces. Unless you are literally taking the user's query Putting a system prompt on the top of it and sending it to the model, there is decomposition to be had. Are you routing to different kinds of prompts? Are you using any tools? What kind of context are you providing? Where does the context come from? Is the context dynamic? Are there any other agents that look at the context? Is there a RAG system? Is there a re-ranking step? Is there a, is there an agent that looks at the sort of RAG results and maybe does a reaction? Is there a follow-up? Do you have a reflection after you show the user something? These all sound like components to me.

Yeah, I mean, one way you can, one way you can zone in on components is with error analysis. Like, you know, if you're doing error analysis and when you do data analysis of the errors, you can often see patterns and that will help you identify like, hey, XYZ component is broken, let's write an eval for that. That's what I was talking about before, is like, okay, a lot of times I'll look at the big picture and then I'll zone in, and that's through error analysis.

Yeah. We are definitely going to talk about error analysis here in a minute. It's important to notice that these steps are dependent and that failing one fails all the ones downstream. This is why I care so much about this funnel perspective because, you know, someone says, if you're doing PDF parsing, can you get away with just one eval? Well, I don't know, maybe. What kind of PDF parsing are you doing? I'm doing PDF parsing for one of my side projects. I like take a photo of a piece of paper and I like need to extract certain things and add certain tags and sort of like, you know, I would say like classify it and also create an event. Hmm. I have one PDF. That PDF has text on it. So the first eval is like, does the text that I extract reflect the text in the PDF? Okay. Well, what about like where? On the first page? On the last page? All the pages? What about the tables? What about the images? What does it do in the captions? And that's just on the extraction. What about the events that I want to create automatically? What about the tags? So how do we tie this back to our experiments? Because your funnel of failures are going to point to more questions, and those questions suggest your hypotheses, and those hypotheses are what creates the experiment. And so this failure funnel is actually sort of the breeding pool for your experiments. Looking at your funnel is where you get inspiration on what populates that roadmap. People say things like, You should be measured by the number of experiments that you run. That's cool. Are they good experiments? Are they grounded in anything? Are they things you read on Twitter? I don't know. But this is how you can ground them. This is how you can be confident in those ROI estimates. The granularity of this evaluation means that these are lower variance metrics. And thus you have higher confidence in the value. If you come to me and say, I have 50 experiments, Brian, that's cool, but I want to know, are they the right experiments? Are they relevant experiments? And I don't know how else to gauge that other than trying to estimate their effect size. And I don't know how to estimate their effect size without measurement. I don't know how to measure without actual realistic impact analysis. And that brings us to analyzing, and this is data science. When you're analyzing eval results, you should be hunting down data stories. Just because the number is improving is not enough information. The failure funnel gives you direction. But now, now we get to say the line, so everybody say it with me: what do you do with your failure funnel? What do you do with the results of your evals? Put it in the chat. I know you know the line. Ben knows it. Everybody should know it at this point. Tammy knows it. This is where we have to look at our data. Let's take an example. This is a failure funnel. Now I have 9 steps. This is simple to nuanced, but Everyone can understand every single one of these steps. Did we retrieve the right table? Did we hallucinate any tables? Did we pick the right tables once we got them? Did we make up columns? Did we create the filters correctly? People talk a lot about natural language SQL. But are they looking at where their natural language SQL pipeline fails? All these red bars are value. I get excited when I see these red bars. Now, for those of you that haven't spent a lot of time looking at funnels, I do wanna remind you, numbers here are worth more than numbers over here, because if they fail, early, they have no chance of moving on. So this 41 here is actually quite significant. That's failing pretty early on. There's a lot of steps it still has to get through. There's this concept of kind of uncovering new failures. When you fix failures at the top of your funnel, you may discover more failures at the bottom of your funnel. So when you're thinking about ROI, think about these as having big magnitude effect and these as having smaller magnitude effect. Okay. That's just sort of a little nuance of evaluating funnels.

Yes.

Fixing the leaky bucket early on can have outsized impacts.

What if you have just like, someone's gonna ask this question, so I might as well ask it. What if you have like agentic workflows where all the steps are like so variable and all over the place that it's like doesn't feel linear anymore? It's like just a spaghetti of steps.

Yeah, it's totally okay to ask the question sort of like there are multiple paths, first of all. And second of all, like it's okay to take something that like feels not Let's say, does it look like a DAG? It's okay to say, take something that doesn't look like a DAG and stretch it out, make it linear, take out paths and say, these are the paths that I want to walk down. The point is, if you can't find any paths where you feel like start here and end here and take these steps in between, it's like not clear to me that you actually know what you're trying to achieve. I know what I'm trying to achieve. I'm trying to answer, like, a damn question for the user. And so I need to get there. I need to get there through this series of steps. You can see here a couple of these steps are related to RAG. A couple of these steps here are related to, like, structured output. A couple of these steps here are related to sort of, like, query understanding. There's sort of the standard ingredients. But you just have to ask yourself, what depends on what? There are cases where things aren't completely dependent. It's technically a poset if you want to be very fancy, but you can always get close enough to a linear ordering and move on with your life.

It's really interesting to see this, like, um, in error analysis, I have a heuristic to help simplify things. Like, people get paralyzed sometimes in error analysis. They're like, what error? Where do I look? And I say, look, just take note of the first, the most upstream error. And just write that and just move on.

Exactly.

So this is very cool. It's like, it's like some parallel confirmation. I've never seen that, by the way. I've not seen this presentation before. I'm getting this fresh just like everyone else, so I'm just as entertained.

Yeah. Um, how do you take into account human loop for when we might not have all the data? Well, it depends what you mean. If you mean human loop as in like the thing that you're trying to do for the user, sort of like the job to be done is to provide them with recommendations recommendations, and they're going to take the final answer. There is sort of, did you provide the right information to the user to be able to actually, like, succeed? I'll give you an example from Stitch Fix. Stitch Fix was very human loop. We had humans that were given a bunch of recommendations for clothes, and then they would pick the clothes to send to the user. The human was the person doing that final step. If we didn't give them good enough recommendations to send to the user, then they had no chance of success. And so we needed to evaluate whether the candidates were good enough. Ultimately, if the stylist— is what the human was called in that case— if the stylist failed to give them anything that they wanted to buy, it wasn't on the stylist. It was on the algorithm. The stylist is kind of that last mile But really, you want to be evaluating, is everything I give to the stylist good enough? If you mean instead human loops to sort of do evaluation, that's fine. But you need to start building towards a reproducible process with sort of building up a labeled set of data where you can build your funnel. What I can tell you is this was human labeled. This funnel that you're looking at, required humans to write the answer to the question and then us as engineers to backpropagate all the way through and say, hmm, okay, at this step, this is what must have been there. At this step, this is what was required. Remember, the funnel is assumptions. We assume that the table that is correct is included when we send it to the model. If we don't include the right table from RAG when we send it to the model, Good luck. It's going to pick the right one. If you ask me a question about your PDF and I don't send that PDF to the agent, uh, oops. So you do have to take some human labeling, you know, calories or spend some.

And is there, is there that last slide? Is there a name for that analysis that you like to use? I just like it a lot.

Failure funnel.

Failure funnel. Okay. I'm going to. I'm gonna cite this. I'm gonna—

and I'm gonna talk about this at Data Council even more. So if you're coming to Data Council, which I'll mention later, you're gonna hear an in-depth explanation of the failure funnel. And you might say, is this data science? And the answer is yes, it is. I'm sorry, you might have been like, you know, celebrating the death, as they say, the sort of like the rumors of my death have been grossly exaggerated. The rumors of data science's of death have been grossly exaggerated. What's this on the right, you might say? What, what's going on here? Because this is an example of sort of like trying to understand this very nuanced picture of like why a system at one stage is failing. This is a retrieval system. This is asking the question for a particular retrieval system, how do we kind of find balance of showing the user a whole shitload of stuff versus showing them only a couple of things. Turns out there's a parameter that controls that, and that threshold, or you can call it a threshold, that threshold has different effects. Ooh, look, interesting. There's different metrics plotted here simultaneously. We can trade things off. I don't know what idiot put F1 score on here, but I certainly wouldn't have. The one thing that I will say here is it's important to have these things interpretable. Because this is the kind of thing that you're gonna work with your product manager on to understand the effect of these experiments, to understand the ramifications of what experiments you wanna do next. This is the aperitif. This is the warmup. Later on, we're gonna go deeper. You're gonna go deeper in your experimentation. So you gotta train them. You gotta get them excited. Now that you have a funnel, you want to improve it with your experiments.

It's not— can we linger on that for a second? Like, so do you need data science? Do you need a data scientist?

I don't think you necessarily need a data scientist with capital D and capital S and some master's degree from Georgia Tech. I don't think you need that. What I do think you need is I do think you need people with some data acumen, people who have been analysts, people who have spent time working with data, pulling out narratives. And most importantly, I cannot stress this enough, people who are truth-seeking. People who care about what the data is telling them. Yes, there are many narratives. Yes, there are many ways to look at the same data, come to different conclusions. But remember, you are trying to hunt down the signal in this noise, because as you find that signal, you have something to do. And that's what I'm going to talk about now. You're going to go and improve. You're going to improve it with your experiments. You're going to build experiments that can actually improve that damn funnel. And that looks like this. Here are two experiments. You can use the funnel as states, and you can think of them as transition matrices. On the left is what things looked like before. On the right is what they look like after. And you're asking, I did some treatment. That's a fancy word for I changed shit. I changed things, and now it's different. Is it better or is it worse? A transition matrix is an easy way to visualize that. But actually, you can do two. You can have two experiments and you can compare them. You can say, which of these is better? Is this experiment better or is this experiment better? Hamel Husain says the word rank fusion and I want to kill myself, but he wants to do rank fusion. I want to do interleaving. Okay. Great, we're doing both. Now we know one of them improves the funnel at this stage a little bit, and at this stage a lot of it. The other one improves the funnel at that stage a lot of it. Which do we care about more? I don't know, but you will when you look at your funnel. That's all I got for you today. You can find me—

I was asking a very data science question. How do you think about controlling for multiple comparisons when running these multiple experiments?

So I don't think that you have to worry about it as much because you have established the metrics that you care about. You can do a really fancy thing if we want to be very smart people and pretend like we have PhDs. We can do something where we collapse these into an individual score, and you can say there is one score, which is an aggregate measure of the deltas that can get you a little bit around it. But actually, multiple comparisons is less of a concern here because you are going to make a hypothesis and a statement. You're going to say, I want to improve stage 2. Did experiment 1, what about experiment 2? That's what you're trying to measure. And so you can be a little bit like defensive against multiple comparisons, but I claim that it's like a little bit less of a danger here because you're not, it's not the standard. I've got 80 different metrics for this, like, you know, user engagement and I'm going to pick the one that works. You really should commit ahead of time with your hypotheses. Luckily we did.

So, so one, one question I got for you real quick is most people have a traditional roadmap for their AI. That's why this talk is so exciting. When I walk into most companies, they have like, hey, we have our Kanban and our sprints and our whatever, all the stuff that is of our nightmares. So how do you unwind that to this? Like, what's the right way to get, you know, to the right place? I think you gotta ask yourself the following question.

I've got a new capability that I wanna start developing. Start small, like take, take one bite of the elephant first. And I think the bite of the elephant that you need to take first is, okie doke, we have a new capability that we want to develop. The first thing that you need to do is you need to spell out the assumptions that you need to kind of like clear. What are the hurdles you have to clear to get to something useful? Once you spelled that out, you come to the product manager and sort of say like, Here are experiments that we're going to improve the individual steps in this funnel. And so I think on a new capability that is sort of like greenfield is the easiest way to like start innovating your team on this. But believe it or not, I actually think another way to really effectively do this is take your most important capability. Where you have evals and go and build the eval or the failure funnel for that. Because I'll tell you what opens people's eyes, especially like leaders looking and saying, oh, this thing that I know isn't as good as I want it to be. Now I know where. Now I know why. It's not 60% of our evals pass. Now it's fuck. 12% of our evals fail before the first step. 7% fail the step after. That is something that, like, you— we talked about transacting against early on. Now that is something you could transact against because it's, it's prescriptive. And that's ultimately what we always wished roadmaps would be. It's prescriptive. Roadmaps People want to be dates that prescribe how engineers spend their time. And this is failures that prescribe how engineers should spend their time.

That's homework. That's the homework for everyone. Yes. That has the, these kind of roadmaps is to take one bit of the problem and reframe it you know, and also think about the experiments you can run and do what Brian is saying.

Come to Data Council. Um, I have a discount code here, it's Brian20 for Data Council. You should come. I'm not only giving a talk that'll go even deeper on the funnel, but I'm also hosting a talk track, and Hamel Husain is speaking in the talk track, and I promise you it is going to be lit. That is the only way that I can describe it because one, there's literally going to be a debate at the beginning of the track. And two, like, I am a pain in the ass MC and I'm going to heckle every single one of them. The speakers are from Moody's, Hamel Husain Husain Husain. We've got speakers from OpenAI, from Gemini. We've got a speaker from Meta and from Runway. So you should come. But there's still a secret for everybody here.