Hamel Husain argues that rigid product roadmaps don't work for AI projects. He pushes for hypothesis-driven development and experimentation instead, and discusses how vendor incentives shape their recommendations in these meetings.
So roadmaps are just very rigid, almost intentionally so. They're rigid to kind of incentivize engineers and sort of benchmark engineers. They're rigid to keep people aligned and in lockstep. If you've ever heard sort of like, I want to make sure this team is delivering. Like, what does that mean? Like, delivering? It means like, kind of like delivering on, like, a cadence that everyone's really excited about so they can build other things around it, build structure around it. And so here you go. Here is your roadmap. You're going to deliver this amazing and novel AI feature. It's due on September 21st. And the reason is because I want us to commit to this launch deadline for these new features that I just came up with. It's been nine seconds since they came into my brain, and we are ready to commit. Maybe you've had the experience of being told when they're going to launch before you've had a single experiment or a customer conversation. And it reminds me of going to school. It reminds me of seeing that we're doing Lagrange multipliers on test three. It's in the syllabus. I don't know what Lagrange multipliers are, but, like, I guess I'll be learning that this quarter. So you may say right now you're feeling, no, no, no. These are just bad product roadmaps. This is not what great product managers think. Like, you may say, are the great product managers in the room with you right now? I'm very fortunate. I actually do live with a great product manager. So I do get to say that they do exist with confidence. But despite that, and despite your rejoinder that I'm fixated on bad product roadmaps, I actually think this is more common, this way of thinking, and this approach is more common than you think. There's a popular O'Reilly book on how to product manage. You may know it, called Product Management and Practice. What does it have to say? You will likely have more success trying to make those Gantt charts better than you will trying to convince people to abandon them. And this is a really. This is a really, like, popular and honestly kind of beloved book on product manager. And so it's. It's a pretty endemic perspective. I would say that, like these product roadmaps that are strict and rigid and hard, they're very, very important. And it is definitely difficult to pry them out of people's hands, no matter how many engineers push back. And again, the problem is you can't roadmap what hasn't yet been discovered. That is the point. And so if you believe that we're doing novel things, if you believe that we are sort of going off into the like, the proverbial like horizon, then you should be anxious. So what do we do? We experiment. And experiments start with a question. Great. Experiments are driven by hypotheses and hypotheses are questions. Hypotheses you might have convinced yourself are guesses, but really they're questions. What do you wonder? If you don't wonder anything, what are you hypothesizing about? So let's do some examples. Can we find relevant tables for natural language to SQL by the row values inside them? Compare that to feature build keyword index of row values. Okay, I could go build a keyword index of row values, but really at the heart of this is like I want to know the answer to can I find the relevant tails with this kind of search? Does data extraction work better before OCR transcription or after you want to get some structured data out of your PDF? Neat. Well, that's simply infra colon new transcription node and extraction pipeline. Does important judge work better on summaries? That's that epic summarize conversations. And finally, should we re rank or interleave? Are rag results versus seven meetings with vendors about their re ranker and how good it is?
So people actually do that? People are having seven meetings with vendors about rag technologies. That's. That's the way they're going about it. I know, I know.
H. Can you just out of curiosity, Hummel, would there be any incentivized reason why vendors would potentially want to tell you a particular approach to solving your recommendation problem there?
They're charities and they don't have any incentives to to sell you anything. So of course, like, yeah, I can't figure out why.
That's good. That's good.
Yeah.
I would never, I would never want to sort of rely on the objectivity of the sales.
