In this talk, the speaker critiques the use of rigid, deadline‑driven product roadmaps for AI development, arguing that they are ill‑suited for work that involves discovery and uncertainty. He explains that traditional roadmaps force teams to commit to feature releases before any experiments or customer feedback have been gathered, likening the practice to assigning a syllabus item before learning the material. Instead, he advocates a hypothesis‑driven, experiment‑first approach where each experiment starts with a clear question or hypothesis. He illustrates this with concrete AI‑focused experiments: searching for relevant tables for natural‑language‑to‑SQL using row values, deciding whether data extraction should occur before or after OCR transcription, evaluating summarization quality for conversation summaries, and testing whether to re‑rank or interleave Retrieval‑Augmented Generation (RAG) results. The discussion also touches on vendor dynamics, with Hamel Husain noting that vendors may have incentives to push particular RAG solutions, though he humorously dismisses them as “charities.” Overall, the talk urges product managers and engineers to replace fixed roadmaps with iterative, question‑driven experimentation to better handle the exploratory nature of AI projects.
Chapters
So roadmaps are just very rigid, almost intentionally so. They're rigid to kind of incentivize engineers and sort of benchmark engineers. They're rigid to keep people aligned and in lockstep. If you've ever heard sort of like, I want to make sure this team is delivering, like, what does that mean? Like delivering? It'd be like kind of like delivering on like a cadence that everyone's really excited about. So that they can build other things around it, build structure around it. And so here you go. Here is your roadmap. You're gonna deliver this amazing and novel AI feature. It's due on September 21st. And the reason is because I want us to commit to this launch deadline for these new features that I just came up with. It's been 9 seconds since they came into my brain, and we are ready to commit. Maybe you've had the experience of being told when they're going to launch before you've had a single experiment or a customer conversation. And it reminds me of going to school. It reminds me of seeing that we're doing Lagrange multipliers on test 3. It's in the syllabus. I don't know what Lagrange multipliers are, but like, I guess I'll be learning that this quarter. So you may say right now you're feeling, no, no, no, these are just bad product roadmaps. This is not what great product managers think like. You may say, are the great product managers in the room with you right now? I'm very fortunate. I actually do live with a great product manager. So I do get to say that they do exist with confidence. But despite that, and despite your rejoinder, that I'm fixated on bad product roadmaps, I actually think this is more common, this way of thinking and this approach is more common than you think. There's a popular O'Reilly book on how to product manage. You may know it called Product Management in Practice. What does it have to say? You will likely have more success trying to make those Gantt charts better than you will trying to convince people to abandon them. And this is a really, this is a really like popular and honestly kind of beloved book on product management. And so It's, it's a pretty endemic perspective. I would say that like these product roadmaps that are strict and rigid and hard, they're very, very important. And it is definitely difficult to pry them out of people's hands. No matter how many engineers push back. And again, the problem is you can't roadmap what hasn't yet been discovered. That is the point. And so if you believe that we're doing novel things, if you believe that we are sort of going off into the, like, the proverbial, like, horizon, then you should be anxious. So what do we do? We experiment. And experiments start with a question. Great experiments are driven by hypotheses, and hypotheses are questions. Hypotheses, you might have convinced yourself, are guesses. But really, they're questions. What do you wonder? If you don't wonder anything, what are you hypothesizing about? So let's do some examples. Can we find relevant tables for natural language to SQL by the row values inside them? Compare that to feature build keyword index of row values. Okay, I could go build a keyword index of row values. But really at the heart of this is like, I want to know the answer to, can I find the relevant tables with this kind of search? Does data extraction work better before OCR transcription or after? You want to get some structured data out of your PDF? Neat. Well, that's simply infra: new transcription node in extraction pipeline. Does important judge work better on summaries? That's that EPIC summarize conversations. And finally, should we re-rank or interleave are RAG results versus 7 meetings with vendors about their re-ranker and how good it is. So people actually do that.
People are having 7 meetings with vendors about RAG technologies. That's, that's the way they're going about it. I know, I know.
Hamel Husain, can you Just out of curiosity, Hamel Husain, would there be any incentivized reason why vendors would potentially want to tell you a particular approach to solving your recommendation problem?
They're charities and they don't have any incentives to sell you anything. So of course, like, yeah, I can't figure out why.
That's good. That's good. Yeah, I would never, I would never want to sort of rely on the objectivity of the sales.
