How to Stop Building Products Nobody Wants

Teresa is a legend in the product management space. She has been teaching something called continuous discovery habits for a really long time. I thought I knew what it is, but I can tell you that no, engineers don't really know what that means.

I really got frustrated early in my career with just spending so much time building the wrong stuff. There's nothing more demoralizing than to like put everything you have into an idea and then watch the world not care about it. The biggest mistake founders make is they go, hey, what do you think of my idea? But when you're interviewing customers, don't ask them about your Ask them to tell you about a time when they experienced the problem your idea was designed to solve.

This is an opportunity solution tree. It's like a structured approach that kind of helps you guide your thinking.

Our brain was designed to keep our heart beating and our lungs breathing and our body alive. It's not optimized for good critical thinking. Confirmation bias is real. It's not conscious. This is your unconscious brain filters out the disconfirming evidence before you even notice it. I think we should just build a map where we show people where alumni live.

That is every engineering meeting I've ever had.

Holy crap, Claude is already better than the average product team. Now I have goosebumps because I can give you an AI that raised the floor, and if you work with my AI, that will raise the ceiling.

So I want to introduce you properly because a lot of my audience is engineers, data scientists, machine learning folks, and Teresa, who we have on today, is a legend in the product management space. So a lot of people who are product managers have heard of Teresa. She has been teaching something called continuous discovery habits for a really long time. She started writing about it in 2011. Like the core of what you talk about is how do you talk to customers and how do you synthesize the information from those customers to inform what you're building? Now, the thing that is really interesting is, okay, every engineer and machine learning person, data scientist, we've all heard talk to your customers and everyone thinks they know what that is. They're like, oh, you just pick up the phone and talk to your customer. To be honest, I thought I knew what it is, but I can tell you that no, engineers don't really know what that means. Like most people don't know what that means. Most engineers, they'll like phone up their friend and be like, hey, I'm working on this new idea. Like, what do you think? Do you think it's a good idea? Be like, yeah, I think it's a good idea. And some people, some engineers have Read some books. I think amongst engineering circles, there are some books like The Mom Test people have read. And that only kind of scratches the surface. Like, I think it focuses a lot on interviewing. And then like, you take it to a different level where you put a lot of structure behind it, talk about like how to actually do it, how do I do it on a continuous basis. What I want people to get out of this interview today is, first, I would like to see if you can describe Like what you teach, why it's important. And then also because you are a teacher, you also have been building an AI product that helps you with the teaching. You've been transforming your teaching. And I think that that's super interesting. And then like by the end, I want to also help people understand why talking to customers is really important, like why talking to them in a structured way at a regular cadence is important, and even like how you Might learn about it as an engineer, but also like how it might tie into evals, cuz you like do evals in what you're doing. And that's one of the things I wanna explore with you is, okay, how does this all tie together? Because, you know, you have blended a lot of things together in your practice. So, you know, just to start off with, because a lot of people haven't heard of continuous discovery, and I don't really think they know what it means to talk to your customers. Can you talk a little bit about your body of work and what you're trying to fix with it? Like, what is your mission? Like, what are you trying to do? What problem are you trying to solve?

Yeah, so I think this will resonate a lot with engineers. I really got frustrated early in my career with just spending so much time building the wrong stuff, right? Just so many companies, an executive has an idea, it gets added to the roadmap, product managers, designers, engineers spend a ton of time having to build it, and then it's only after you release it we realize nobody actually cared. And I think I'm a maker at heart. Like, I just really like making things. And there's nothing more demoralizing than to like put everything you have into an idea and then watch the world not care about it. And I think discovery is not a term I came up with. This guy Marty came up with it, and he's really trying to distinguish between the work we do when we're deciding what to build, that's discovery, versus the work that we do when we're actually building it, that's delivery. And I think those terms are really helpful because A lot of business people think all of the work is in delivery and that having the idea is easy, but they forget that like most of their ideas don't work. And it's not that they forget that, it's that they never bother to measure impact. So they actually never learn their ideas didn't work. They just do a bunch of stuff, they fall short of their expectations, they're not sure why it didn't work, they're not sure what worked, didn't work. And really, I think good discovery is just building in fast feedback loops. And this is the thing that I would say applies across all the discovery habits. It's why I call evals a missing discovery habit. Discovery is just how do we build in fast feedback loops so we learn we're on the right track sooner than later? Like, it's that simple. And you know, in your class, Hamel, you and Shreya presented it as like, all you're teaching is the scientific method. That is literally all I teach is, okay, so we have an idea of what we should build. Okay, great. That's a hypothesis. Is it the right thing to build? How do we build in fast feedback loops along the way so that we don't go too far down the wrong road and we learn much sooner whether we're building the right thing or not? There's lots of ways to do this. We have qualitative methods, we have quantitative methods, but if we think about it as like at its heart, it's the scientific method. If you were like, okay, I'm gonna go do this scientific method to get feedback on my startup idea, you're not gonna call your buddy and be like, here's my idea, is it good? That's a terrible experiment to test that hypothesis. And like Most people intuitively know that when we frame it as an experiment, when we frame it as the scientific method, when we frame it as we're trying to get reliable, valid feedback on a hypothesis. And so all of the discovery habits are really designed to teach teams how to build reliable feedback loops.

Can we make it a little bit concrete? Like, how do you know that you may have this problem of like, you're not doing discovery, you're not talking to your customers correctly? A lot of people, again, they think they're talking to customers. They're not. Most people are not. I only realized that after reading your stuff deeply. I'm like, oh, I'm not doing it correctly. It takes a lot of practice, a lot of skill. Yeah. Can you tell me like, what does it look like when it's not going the right way?

Yeah. So one of the things I tell people is if you're not constantly throwing away ideas, if you're not constantly evolving your ideas, you're probably not doing good discovery. And this is really tough because you can do all the right activities, but if you don't come to it with like intellectual curiosity, More importantly, intellectual honesty about your idea may not be very good. This is why I think using the scientific method as an analogy is exactly right. We all have read papers where we look— for those of us that read papers— where we look at the methods and we're like, wow, this is sort of a crummy process. Like, sure, you're a scientist, but you're not really doing the scientific method. You started with a hypothesis, you designed your experiment to confirm that hypothesis, and like, if you have any critical thinking skills, you can see that. Yeah. It turns out this is what founders do, and we do it for a reason. Like, founders are so— already have to be crazy to be like, I'm going to create a new thing in the world that has never existed. That already creates— that already requires like such a leap of faith, of courage, that that strength that gives you that leap of faith, of courage, is actually a blind spot because it's setting you up that when you go to test your ideas, you're testing them from this really confident point of view. Whereas I think to do science well, we have to have a lot of doubt. We have to set up our experiments to be disconfirming, not confirming. And so there's this mindset that I think is really hard for humans. Like, we feel so right, we feel so certain. And I think good discovery is really like, okay, I have to detach myself from my idea and really design my feedback loops to be disconfirming. And this is hard. And I think the closer we get to our work, the harder it is.

And it's really easy to fool yourself because when you say detach yourself from your idea, most people will, again, they will call up their friend and say, what do you think about this idea? They will ask a customer like, hey, they will lead with a demo to say, hey, like look at this idea. And they will get like a false confirmation from those conversations because they're not going about it the right way. You have to actually like dig really hard to get to the truth.

I think about discovery as there's 2 parts to it. There's the generative part. How do I know I'm solving the right problems? And then there's the going after the right problems. And then there's the evaluative part. How do I know my solution actually solves those problems? Let's say you start with an idea. If you wanna get feedback on your idea, the biggest mistake founders make is they go, hey, what do you think of my idea? First of all, not everybody is your target customer. Not everybody experiences the problem your solution was designed to solve. So if you're asking people that are not your target customer that don't experience the problem your solution was designed to solve, your feedback is garbage. Who cares what so-and-so thinks if they don't have the need? And then if you are talking to someone where you know they have the need, or you think that they have the need, that's actually the first thing you have to test. You don't want to say, do you have this need? That's a bad test. You want to ask them to tell you about an experience when they had that need. You're listening for, is this real? I think you have— like, I'll give an example. Let's say I want to start a startup and I want to help podcasters grow their audience, right? I don't know what I'm going to build. I don't have a solution yet, which by the way, this is the right way to be a founder in my view, is Don't just start with a shiny idea because we don't know who needs that idea. If you've heard the phrase, you're a solution looking for a problem, we wanna avoid that. That's like the slowest way to find a successful product. Let's say I just love podcasts. I see a lot of new podcasters flounder. I wanna help them grow their audience. I don't know what I'm gonna build. I don't know why they can't grow their audience. I don't know very much about them yet, but that's my goal. I'm gonna go interview some podcasters. I'm not gonna say, here's my idea, what do you think? I'm just gonna ask them to tell me about their podcast, what they're doing to grow their audience today, what's working, what's not working. I'm also gonna interview really successful podcasters to learn how they grew their audience, because when we find people that are successful at the problem we're trying to solve, that's really good inspiration for how we might solve it for the people who haven't found that solution. And then as I learn about those customers, I'm listening for Where do they have unmet needs? Where are their pain points we can address? Where do they aspire to things? Where do they desire things that just don't exist yet? And that's going to help me understand where can I intervene positively. And it's only once I have a really rich understanding of, like, I call it the opportunity space. It's just unmet needs, like unaddressed pain points, desires. Some people call them latent needs. The jobs to be done framework falls into this category. Right? It's just like, what do people need that isn't being met? And we want to start there and really have a rich understanding of who our humans that we're designing for are. Oh, and then only then do we want to start to look at like, okay, well, I heard this need. It's really hard to find your first 1,000 listeners. Let's focus on that. Well, what are really experienced podcasters doing? Maybe they're coming up with like a content strategy to go viral on social media. And we can start to look at like, I want to figure out a social media strategy to promote my podcast. That's an opportunity. So then we can look at like, okay, what's some potential solutions for that opportunity? And what a lot of founders do is they just build something for themselves. They jump to their first solution and they hope there's other people like them. Now we have lots of success stories where founders built something for themselves and it worked. And that's because there were other people like them. But we have a lot of stories of founders that start with an idea building for themselves And they, it went nowhere. We don't hear those stories because they're not famous.

And every single time anybody has brought up a startup thing to me, in my personal experience, always starts with the idea, hey, Hamel, I have an idea. And I think it's every engineer's experience, almost universally. They're like, you feel like it needs to start with an idea. And I'm glad you called that out. And you mentioned one thing that I think is deceivingly, it sounds simple, but it's very difficult. And you said you want to ask for behaviors and stories. And so can you unpack that a little bit? What is the distinction between behaviors and stories and how people usually ask questions?

Let me first go back to this idea of I'm going to start with an idea. I actually think this is human nature. Our brain thinks in solutions because they're concrete, they're really specific, they're tangible, right? Whereas problems are more abstract and our brains want to get concrete as fast as possible. If you're a founder and you have an idea, that's fine. That's a perfectly good place to start. But when you're interviewing customers, don't ask them about your idea. Ask them to tell you about a time when they experienced the problem your idea was designed to solve. So if my idea is to help you with your social media strategy for your podcast, I wanna ask, tell me about the last episode you promoted on social media. And I want you to walk me through everything that you did. Now, the reason why I'm doing this, there's a very subtle difference here. I could say, Hamel, how do you promote your podcast on social media? That's a general question. When I ask you a general question, I invoke what Kahneman calls System 1 thinking. It's your fast, lazy brain. Now, really smart people hate hearing this, but it turns out our brain was not designed for all the mental activity that we love and enjoy. Our brain was designed to keep our heart beating and our lungs breathing and our body alive, right? And this mental activity is sort of a side effect of that. And so the brain is wired to optimize for life. for keeping us living. It's not optimized for good critical thinking. And so oftentimes the brain will generate— System 1 and System 2 are analogies. I just wanna be clear. Like, it's not like there's a part of your brain that is System 1 and System 2, but they're really helpful analogies. System 1 is your fast-thinking, fast-response brain. I like to think about it as the lazy brain. It's like your brain is just generating an answer so it can go back to keeping your heart beating. And the problem with System 1 answers is that they don't necessarily reflect your actual behavior. So if I say, what do you do to promote your podcast episodes? You're gonna get a fast answer. Your brain is gonna tell you something that you believe is true, but what you tell me won't actually match what you actually do. It might match what you wish you did. It might match what you do some of the time. It might even match what you did like the most recent time, some of it, but you're not taking the time to think through the whole instance. And so there's a lot of context missing. There's a lot of detail missing. And it, when I say reliability, it doesn't always reflect what you actually do.

Do you have a favorite example of this? Yeah.

One of my favorites is I was in a workshop where I asked a woman, I do this to demonstrate it and it works every time. I asked a woman, tell me about what criteria you use to buy a pair of jeans. And she said, fit, price, and she has a favorite brand. And I said, okay, great. Tell me about the last time you bought a pair of jeans. And she said, I bought 'em on Amazon. And I sort of chuckled. I was like, oh, you bought them on Amazon. How did you know if they fit? And she said, well, they're a brand I buy all the time. And I said, okay, have you ever bought the same brand and they fit differently? And she said, yes. And I said, well, why were you willing to buy them on Amazon this time? And she said, because they were on sale. Okay, so what actually came out of her last purchase? She has competing criteria. Fit does matter to her. Her favorite brand does matter to her, but she'll compromise fit for a better deal. And everybody knows like returns on Amazon aren't super— they're easier now that Amazon owns Whole Foods, but at that time they were not easy, right? And so this is very common. Another one I often ask people is if you're super athletic, like let's like runners fall into this category. I'll ask, tell me about how often you run, and they'll say 7 days a week. And then I'll be like, okay, tell me about last week. And inevitably they'll have run— like runners are crazy, so a lot of them do run every day of the week. But if we ask about last week, maybe it was only 5 of the 7 days. And what they forget is like their kid was sick one morning and they got to school late, or they had a meeting early in the morning and it overlapped with their run. Like there's all these exceptions. Our brain thinks exceptions are outliers, but in reality, exceptions happen all the time.

And what I learned from you is there is a large gulf between what people say they want and then what they do, like how they invest their time and where their Actually problems that they're actually trying to solve.

Even more than that, I would say there's a large gulf between what people think they do and what they actually do. So not even what they want, just— we're just talking about behavior, right? If I ask you, what do you like to watch on Netflix, your brain is going to generate a really fast answer. That answer may or may not be different— the same as when I ask you, tell me about the last thing you watched. When we're designing products, we want to design for actual behavior. Now there's some exceptions. If I own a gym, I want to design for your aspirational self because I don't want you to come to the gym. I just want you to pay for a subscription. That's a little different, right? For most of us, we want to design for what our customers actually do. We can't just go call our buddy and say, what do you think of my idea? We have to learn techniques for how do we get reliable feedback to learn what they actually do. The simplest way to do this is to ask for specific stories about past behavior. So tell me about a specific time when— and then even this is hard because if I say, hey Hamel, tell me about the last time you watched TV, your response is gonna be really short. You're gonna be like, I watched a movie the other day. And I have to like do the work of situating you back in that moment and helping you tell me the whole story from beginning to end. So it's actually a skill to collect a good rich story that requires practice. But the benefit of practicing this skill Is we start to get a really reliable feedback loop. When I'm trying to learn, does my customer experience this problem? How do they experience it? What environment do they experience it in? Who are they with when they experience it? I can learn all of that by just collecting a rich story.

That makes a lot of sense. It's almost like if you talk to customers and if you talk to them in the wrong way, it's gonna lead you even more astray. So you think like, oh, I'm doing this thing, I'm talking to customers, I'm doing what people told me to do. But if you do it the wrong way, it's just going to lead you even further potentially away from what you should be doing.

We've all seen everybody like cherry-pick data for their argument, right? We're confirming. We're coming in with a confirming mindset. I'm going to see all the evidence that supports my idea. This is really common. I mean, this is one of the biases. Confirmation bias is real. It's not conscious. That's the thing people misunderstand about confirmation bias. They think that like People blindly ignore disconfirming evidence. That's not what happens. What happens is your unconscious brain filters out the disconfirming evidence before you even notice it. It's not a conscious act. So to overcome confirmation bias, we have to design our feedback loops to be disconfirming from the beginning. And I think this is what people get wrong. They just think like, oh, I'll just go get some feedback. Yep. The problem with that, like, you're doing a good activity of getting feedback, But your brain is literally filtering out all of the evidence that there's something wrong with your idea. And so you're getting no value from that.

The next thing I want to ask you is, okay, so you have this word continuous in front of discovery. So what is the continuous aspect of this? Like, why is the continuous thing important? And what does that mean?

Most digital products are never done, right? So it used to be, we made physical products, we obviously still make physical products. But like, if you work at a consumer packaged goods company, And your product is shampoo. You might come out with a new shampoo every year or 2. This is how software used to work. We used to stamp software onto CDs and put them in a shrink-wrapped box and ship them to Fry's, right? Like, but that's not how software works anymore. Thanks to the internet, software is continuous. We are continuously iterating and improving our products. It's not like Netflix is going to decide one day that, hey, we're done. We're just going to stop building this thing for a while. In fact, we see companies do that. Airtable stopped building their product for a while to build a new AI product and it got real bad. And then they just sold for a fire sale.

So don't do that.

It's a very relevant right now true story, right? Like digital products to stay healthy have to always be evolving. And there's a reason for that. Like the competitive landscape is always changing. Technology is always evolving. There's new stuff just now possible. As we address customer needs, new needs emerge. We often create new needs with the stuff that we're creating. And so our products have to evolve as the market and our customers evolve around it. And so we continuously build, we have to continuously make good decisions about what to build.

I feel like there's a lot That goes into this continuous part. So there's like, how do you interview? Or, and like, who do you interview? How do you schedule those interviews? And your writing discusses a lot of that. Can you talk about it a little bit? What do you have to learn in order to like do this properly at a high level?

The first thing I'll say is there's more than just interviewing. So interviewing is the generative feedback loop. Am I solving the right problems? Assumption testing is our evaluative feedback loop. So am I building the right solution? And a lot of founders actually could just start with assumption testing. They could— the idea behind assumption testing is you take your solution. I like to use story mapping to figure out what my underlying assumptions are. So in a story map, we're mapping out what does a customer have to do to get value from this solution? So literally step by step, what are you expecting your customer to do for them to get value from using your product? And then in each of those steps, we can start to generate the underlying assumptions. Do they want to take that step? So that's a desirability assumption. Are they willing to take that step? Here's a really— people think wanting and willingness are the same, but I'll give you an example where they're clearly different. If I'm in a foreign country and I need cash, I want to put my bank card into the ATM machine and get cash out. But if it looks really sketchy, I am not willing to do that. So your customers have to want to do each step that's required of them. They have to be willing to do each step that's required of them. Willingness is often about trust, but not always. Sometimes it's you're asking them to do a lot and they want to do it, but they're, they're not willing to. It's too much work. Every step we have usability assumptions like, can they find it? Do they understand what's being asked of them? Are they able to do what's being asked of them? That brings into a lot of our accessibility concerns. We can look at, can we do what's required at each step? That gets into some of our feasibility assumptions. Like, is it possible to deliver what we need to do at each of these steps? Sometimes our solutions require Partnering with other people, that can bring up viability solutions like, can we partner with them in a way where the economics work for both parties? That can bring up feasibility assumptions like, can we even integrate? Do we have the APIs needed to integrate well? Viability is also like willingness to pay. Is this creating enough value that the customer is going to pull out their wallet and pay for it? There's also ethical assumptions. So what data are we collecting? Are we being transparent about that? Who are we sharing that data with? There's sustainability assumptions about resources being used and waste. There's even ethical assumptions around who we choose to target and who we might be unintentionally leaving out. So a big one here is most B2B companies focus on large enterprises because that's where the revenue is. But I don't know about you, you live in Portland, I live in Bend. Oregon is very anti-big box everything. We like our mom-and-pop businesses, but who's creating software for them? Not very many people, right? There's some exceptions, but not very many people. It's not where the revenue is. So there's sort of some ethical implications of how we build things. And so one of the things I encourage teams to do is to really start with your idea and break it down into these underlying assumptions, get really specific because these assumptions are a lot easier to test than your whole idea. When we try to test the whole idea, we very quickly fall into the trap of let's build everything and see what happens. Whereas if I get really specific on an assumption, I can collect data on an assumption really quickly, usually like in a day or two.

And I'm glad you brought this up. Like, okay, so someone listening, they might be slightly overwhelmed. Like, okay, Teresa's talking about all these different things. Think about how do I keep track of it? One of the key things that you've created that help people go through this is an opportunity solution tree. It's like a structured approach that kind of helps you go through it without getting overwhelmed and like guides your thinking. Can you talk about what that is?

Yeah, I actually wanna share a story about what inspired it cuz it was really driven by an engineer that I worked with. So I was working at a startup. We built communities for university alumni associations, and at this time we had over 300 universities in the US. We were like the dominant player in this market. And so if you have ever participated in your university alumni community, especially around 2003 to 2012, it was probably our product that drove that. And what's interesting about alumni is they have a lot of expectations from their university about what they want, but they don't really engage. So they want to receive a lot, but they don't give a lot. And we had built this system where alumni could post what they needed, but it was overwhelming people because people asked for a lot of things and nobody wanted to give. And so we were like trying to figure out like, how do we get people to give?

What kinds of things do alumni want, out of curiosity? I'm an alumni of several.

Yeah, help finding jobs is the biggest one. So like, I want to use my university network to like get a job at Google, or even things like, I have a dining room table I want to sell. I'd love to sell it to an alum of my alma mater.

Okay, I wouldn't have guessed that.

This is all the kinds of things we see on Facebook. Like Facebook has Facebook Marketplace. I don't I don't know if they do anything with jobs, but like some people, especially some schools have really strong affinities. And then people really, you've probably experienced this. You meet someone in a bar that went to your school and you instantly have a lot in common. There's a lot of affinity with the people we went to school with, even if it's across decades. So there's, we had this like imbalance in our community. And what it led to was I framed it as like in my head as the product manager, I was framing it as we have to find the right balance. between giving and receiving. So I was like, how do we get more people to give? But our ultimate outcome was, let's increase engagement. So what the business needed was alumni to engage more. I had identified an opportunity of like, let's rebalance this giving and receiving. We're sitting in a room with our engineers brainstorming, and one of our engineers goes, hey, the Google Maps API just came out and I've been playing with it. It's so cool. And I'm like, okay, I'm kind of rolling my eyes already because what does Google Maps have to do with getting more people to give. And he goes, I think we should just build a map where we show people where alumni live. And I was like, how does that solve it?

That's so typical. That is every engineering meeting I've ever had.

Ever, ever, right?

100%.

And it was driving me nuts because I was like, no, we're trying to get alumni to give. How does that get alumni to give? And he was like, no, we're trying to drive engagement. And this would be a great way to engage people. Okay, he wasn't wrong. We just weren't having a conversation at the same scope and we were driving each other nuts to the point that literally this engineer stayed up all night implementing a map of alumni because that's how strongly he felt about it. And I was pissed off because then the next day he couldn't work at all on the sprint he was supposed to be working on. And like, I've seen this problem time and time and time again where like we're all humans in a room trying to collectively solve a problem, but we're not working from the same context and we're not working within the same scope. And so if the opportunity solution tree had existed at that time, that engineer was right. Our outcome was increase alumni engagement. But in my brain, I had identified an opportunity and I thought it was the most important opportunity to solve. There were other opportunities like that guy Seth was trying to solve this opportunity of like, I wanna know who lives near me, which by the way is a perfectly valid opportunity to go after. But we, we skipped the decision of which opportunity is most important. Each made assumptions about how we should solve it. And we argued in the solution space.

No, it's very classic. Like, hey, it's a solutions-first thinking. Like, I want to use this Google Maps API. I think it's cool. Let me find a reason to justify it.

And here's the thing, like, you will never come to consensus when you're arguing in the solution space because you skipped the most important question, which of these problems is most important to our customer? To be honest, to this day, I still don't know. Would alumni have cared more about who lives near them? Or would they have cared more about like getting what they actually wanted when they posted a need? I don't know. I have no idea how to answer that question because we weren't even exploring that opportunity space. We were, all of us were just jumping to solutions. And like I had been talking to customers, so I had my own intuitive sense of what opportunities mattered. And I had the hubris to think that like I was talking to customers, so I knew more than Seth. But the thing is, Seth is an alum and he actually wanted to know which alum lived near him. And he wasn't wrong. Like, I can't tell you to this day I was right and he was wrong. He could have been right. The problem in this story is that we weren't all aligned. We hadn't collected all the opportunities we might solve, put them on a piece of paper in front of us, and had the more important discussion. What do we know about our customers? Which of these is most important to them? Which one should we solve? And agree on that first before you move into generating solutions.

So how did you solve that?

I didn't solve it at that point. I mean, Seth solved it by just staying up all night and building what he wanted to build. I got upset and continued to plan our sprint around what I wanted to build. So we didn't really solve it. We just, you've seen this probably on a million teams, right?

Like, I would say, and engineers, you know, there's another insidious problem, which is there's a currency around like, which technologies do you know? How much complexity you can, you can tame. Sometimes like engineers are incentivized to create new technologies and new Complexity for their own purposes. And so it's a friction. Clear to me, like, okay, from a business perspective, you wanna start with like what outcome is most important, especially now with AI where like you can just build anything you want.

Funny, at the time Seth drove me nuts. I love Seth, but, and actually with time I can look back on it and be like, actually I, like, I was just as wrong as he was. The problem was not who was right or wrong. The problem was the scope of the conversation we were having and where Seth was 100% right. I'll tell you, I've been trying to get engineers to engage in discovery for 20 years. And I actually love that with AI, they are now starting to engage with discovery because they realize their companies are asking them to be more product-minded. And so like, welcome to the club. Thank God we're finally getting there because what engineers bring that most product managers cannot bring is a knowledge of what's just now possible. I have a podcast called Just Now Possible for this reason. Like a lot of innovation is driven by technology evolving and new things becoming just now possible. Look at AI. Like, we're living this every second of every day right now, and our engineers are the best people to understand that. That's exactly what Seth was doing. A new technology came out, and this Google Maps is so like pedestrian now, but at the time it was revolutionary to be able to put pins on a map based on just data. And he looked at that and he got immediately excited and he saw a customer problem he could solve with that technology. I was a product manager that didn't even know that technology existed. So like, I never would've even considered that customer problem because how in the world would we solve it? That technology didn't exist a week before. And so good discovery happens engineers with knowledge of what just now possible and product managers with knowledge of business needs and designers with knowledge of what's usable and desirable and delightful, like all come together and collaborate on what's the best thing to build given all of our expertise. But it so rarely happens in business.

So how was the Opportunity Solution Tree like born out of this?

That story I told you, I think happened in— I can tell you it happened before 2011. I left that company in 2011. So I'm not sure, it could have been like 2007, 2008, 2009, like somewhere before 2011. I thought about it for a long, long time. I was like, what is going wrong in this situation? Like, I framed it as like, I have a hypothesis And I'm working in the scope of this hypothesis, but I haven't done a good job of communicating this hypothesis to the team, and they're working in a completely orthogonal scope. And so we're not finding alignment. But I could not figure out, like, what was in my head that I wasn't communicating. I just thought it was obvious. Like, we have this system where we let people ask for things and other people give. It's unbalanced. Like, clearly we have to help people give. Like, I just thought it was obvious. I couldn't see the context that I had that the rest of the engineers didn't have. And so it took me almost a decade to like go through lots of iterations. And I had started coaching in 2011. So lots of like seeing this on lots of other teams and collecting a lot of data points to realize like, okay, the first thing is I have a lot of customer knowledge engineers don't have. How do we get engineers to have more customer knowledge? Let's invite them to the interviews. Like we can't just do interviews on our own. That was like one thing is like some of the context I have is just learning from customers. Some of the context was business context. How do we get engineers up to speed on business context? And then the thing that really was the catalyst was I read Anders Ericsson's book Peak. So Anders Ericsson is a researcher that studies the difference between experts and novices. And he's— Malcolm Gladwell wrote a book where he coined the term the 10,000-hour rule. The way Malcolm Gladwell wrote about it kind of bastardizes the research, but the 10,000-hour rule is based on Anders Ericsson's research. What he argues, one of the things he argues, is that experts just have many, many, many, many hours of experience. That actually wasn't the most interesting part of his research. The more interesting part of his research was experts have mental representations that novices haven't developed. And the easiest way to understand this is if you look at a chessboard and you're not a chess player, it looks like a random mix of pieces. But if you're a grandmaster chess player and you look at a chessboard, You see all the patterns of how they got to that point, and you can see the moves to like win the game. It's— you have a structured representation of a chessboard that a novice doesn't have. And you probably have experienced this in your own life. Like, think about like evals. You have like a mental representation of types of evals, ways to do error analysis, like ways to set up data that like a novice, it's just foreign to them.

Yeah, I feel like whenever I Look at data, I can smell what's wrong within.

Yeah, exactly. So I read this idea of like, oh, it got me thinking, what's the mental representation I have in my head that a novice doesn't have? And what came out of that was the opportunity solution tree of like, okay, yes, the outcome matters, but there's this whole missing layer we never talk about, which is the opportunity space. And the opportunity space is where we're customer-focused. We're not just using technology to build cool things, we're using technology to solve our customers' problems. And then we need to make sure everything we build is tightly coupled to a need. And then as we do that, we start to impact our outcome. And by the way, we're only looking at needs that we think can drive our outcome. Like the needs, the opportunity space is infinite and a lot of teams get stuck solving customer needs that don't create business value. And then that's also a problem.

And I think, okay, so this opportunity solution tree structure you created, I believe is super popular. Like when I did research on the internet, there's like a cottage industry of even software of like different software that will like help you build these opportunity solution trees that you can like push your data into these like interviews. And there's, there's actually different kinds of software. It seems like it's taken off. It's, it's become a very popular way to think about the problem. So maybe as a little bit of a pivot, it seems like you practice what you preach a lot. And so like, you know, people have been using these tools. I know you've been thinking about, you write a lot about how people are trying to use AI nowadays to help them with this process. And there's a tension between having the AI do everything for you and then having the AI help you. You're trying to teach people. So first of all, like, how did you identify the opportunity, this angle of how you're transforming what you teach? And like using the kind of the methods that you teach to discover that?

Yeah. So I've been teaching the Discovery Habits to teams for a very long time. I think this is my 15th year doing it formally as my own business. And then obviously several years before that coaching and developing my own product teams at startups. I, what's hard, like to do discovery well is genuinely hard in the same way that to do science well is genuinely hard. It's all the mindset stuff. It's intellectual curiosity. It's intellectual honesty. It's continuing to have rigor. What's different between science and discovery is discovery happens in an industry that wants to move at a ridiculously fast pace. Everything in the organization is telling you, don't do that, just build. And so we have this discipline that takes time, that takes rigor, and we're trying to do it in an environment that doesn't support any of that whatsoever. And there's these competing odds. Like, your boss is very certain about what you should build and wants to just tell you to build things. And you're trying to, like, work this process and you really want to do the right thing. You want to learn from your customers. You want to build something that's going to be a good fit, but it's Friday. And on Monday, the engineers need a new sprint full of user stories and you don't have them yet. Right? So, like, it's genuinely hard to do good discovery in industry. Trying to adopt some of the, like, ethos and principles of like good science, but good science moves extraordinarily slow. And like, we don't have that luxury in industry. We have to move fast.

And I wanna be clear, people might hear you say good science and they just go straight to quantitative methods. 'Cause I know the engineers watching this will be like, oh, we need data analysis. Gimme the dashboards, whatever. And like, no, no, there's a lot of qualitative as like dimensions to what you're talking about.

Measuring, instrumenting your product and getting good behavioral analytics is a discovery habit. Right, that is a feedback loop. Running A/B tests, this is like what most teams think. They'll be like, I'll just build both and we'll A/B test it. Okay, you just did twice the work. Like, let's learn before we do all the work whether we're on the right track. Because when we build something and then learn it was wrong, what we lose is all the opportunity cost of what we could have built instead. And that's too expensive. Like, we don't wanna spend that every time. So a lot of the discovery habits are designed to help you learn as early as possible so we don't waste time on the wrong stuff. We have both quantitative habits and qualitative habits, Both are important. Even in science, we have quantitative methods and qualitative methods. We can get into this whole like epistemic philosophy of how do we know things. They both bring something to the table. It's actually really important that we embrace qualitative as well as quantitative, because when we just focus on the quantitative— there's a really good book about this, it's called The Score. It talks about— it's a little bit Gottman's Law. Like once you come up with a measure for something, it stops measuring the thing you're trying to measure. And it's just this whole idea of like metrics can be gamed. and we optimize for the metric instead of the thing we were trying to measure. And so I actually really, I don't fall into one camp or the other. I think we have to embrace both equally.

Is this similar to Goodhart's Law?

Goodhart's Law, that's what I meant. And so like we have to balance both. Obviously quantitative gets us data at scale. Qualitative gets us all the rich context that we're never gonna get from quantitative.

How did you, using your continuous discovery habits, identify like the exact way that you should transform your education To have AI in it? Because there's so many things that you could have just made a chatbot. You could have some kind of gadget thing that does, you know, you could have gone in hundreds, thousands of different directions. So like, don't take me through the process. Like, okay, how did you focus on, and we're going to talk about the solution, but just opportunity that you found.

I've been teaching this for a very long time. And I think it's taught me how hard it is to do discovery in industry. Because there's so many competing factors. I also have learned how hard it is for an individual to learn to work this way. There's a mindset that's required. There's skill that's required. And like, it just takes a lot of effort to develop that mindset, to develop that skill. So just take interviewing as one skill that's required. I have to develop a very curious mindset. I have to detach myself from my own ego, forget what I'm trying to learn, And be totally present with you and hearing your story. That's like a mindset that's hard to develop. Then I have to develop the skill of how do I situate you in the moment, help you recall your memory, help you walk through your story step by step. That takes a lot of practice. So through years of teaching, I see that people get better at it, but they don't get great at it. Like, it's only the exceptional students that put in the time and energy to develop the skill and to really change their mindset. And I'll tell you, I've been doing this for 15 years. I can name like 10 people. They— I know all their names. I've stayed in touch with them because they truly were exceptional students. But it's rare. And I'm not blaming the individual. Like, they're working in an environment where this is genuinely hard.

I feel the same way about evals, by the way.

Yeah, it's just— this is— it's all because the scientific method is hard for humans. It's hard for humans to do the scientific method well. Like, it's so counter to our biology. And so I care a lot about learning outcomes. Another thing that comes from Anders Ericsson's work is this idea of deliberate practice. The way we build a skill is we have to practice the skill in small pieces with expert feedback. And so all of our courses are designed around deliberate practice. So like in our interviewing course, you practice interviewing live in class and the instructor gives you feedback and your peers give you feedback. Well, your peers aren't very good at feedback because they're giving you— they're new. And so the first— when I saw what was possible with AI, the first thing I wanted to do was start to experiment with Can AI give feedback better than peers? And if AI gives feedback better than peers, maybe that even helps your peers give you better feedback and everybody gets better. And so that led to my first AI product, which was my interview coach, where you submit your interview transcript and it gives you very personalized, detailed feedback on how well you conducted your interview. And then what this unlocked was like, okay, well, now that I can give you personalized feedback, I can make this course accessible to way more people because I can offer it on demand. So before we used to have live cohort-based courses. Now it's on demand. You can take it from anywhere in the world. You can find a friend to interview. You record your interview, you send it to the coach. And it turns out some people won't even do that because they're afraid of being a beginner with somebody they know. And so the, another product I'm working on right now is the AI participant so that you can practice with the AI and not have to be embarrassed that you're a total beginner. And then submit that transcript to the coach and get feedback. And then you can, what this unlocks is it unlocks infinite practice with expert feedback.

What's the difference between the AI coach and the AI participant?

So the AI participant will be the person you're interviewing. The AI coach is what grades your interview. Because what would happen— so the dirty secret of on-demand courses is lots of people buy them and almost nobody completes them. This is why people move to cohort-based courses, because you have to come to class. But cohort doesn't really solve it. You probably see this in your course. Some people really engage and do the work, and a lot of people just— they want edutainment. And so I'm trying to get people to build skill through a lot of practice. But there's all these barriers to why they don't wanna practice. They're embarrassed to practice. They don't, whatever. And so I thought like, okay, well you just practice and you get private feedback that will help with the embarrassment. But then there's like this embarrassment of, I don't wanna interview my friend, or I don't wanna interview my colleague cuz I'm a total beginner and I don't wanna be a beginner in front of another person. So now I'm using AI to be your interview participant so that you're in the privacy of your own home. You can practice as much as you want. Nobody sees your interview practice. Nobody sees your grade. Just you. And what this unlocks is it just unlocks really fast feedback cycles, and that's how we build skill. So that was sort of my first foray into like, interesting, this is like an interesting way to use this in class. It has this extra benefit now of I can see learning outcomes. So I see the scores you get from the interview coach, and I can see if your scores go up over your rounds of practice.

There's many different layers to the whole process. There's the interviewing, and then there's the synthesizing of interviews, We'll talk about synthesizing interviews in a little bit, cuz that, that has some very interesting challenges. But like, okay, when it comes to this interviewing that you just talked about, this interviewing practice, the million-dollar question, how do you know it was good? How did you get confidence that you feel like it's good enough?

I mean, this is what led me to your evals class, was trying to figure out, is my interview coach any good? What's nice is that my courses at this time, at the time that I built the interview coach, we were still cohort-based. We limit our cohorts to 50 students, so they're not giant large-scale classes. This was a scale that I literally could read every single coach response and just get a sense for like, is this what I would have said? And in the earliest days, if the coach gave a student feedback I disagreed with, the student got an email from me saying, pay attention to this part, you can disregard this part. In round 1, like in April of last year when I first released this to the first class, I just QA'd everything, every single response. I knew that wasn't sustainable over time, which is why I started to learn about evals, ended up taking your class, and then was able to do error analysis, see where the coach made mistakes, and start to put evals in place. And this, like, resonated with me so strongly because evals are just another feedback loop. So to me, it just felt like, oh, this is right in line with what I teach. I feel like I had the mindset. I could bring the, like, intellectual curiosity and, like, go deep on the analysis because it's very similar to— you guys even talk it about it as axial coding, which is a qualitative analysis tool. So like, I already had those skills, and so it was really easy for me to just dive deep and like be rigorous about where does it make mistakes, how do I measure that, and then how do I systematically reduce those errors.

What are the evals? What do they look like?

There's some specific things that we teach. So a story-based interview has a structure, and the structure is you start with the story-based question. One of the things we teach is at the very beginning of the interview is to situate the person in the moment at the start of the story. So for example, if I ask you, tell me about the last time you watched TV, and you're like, I watched something last night with my family. I'm gonna say, okay, set the scene for me. Where were you? And I'm gonna get you to describe like your living room and who you were with and who was sitting where. I don't actually care about those details, but I'm doing that because it aids your memory recall. It also communicates to you that I want all the detail, which is gonna get you to tell me a better story. And then the next piece is we teach you how to build the timeline. So we're using a lot of temporal prompts. What happened first? What happened next? What happened after that? And then because you're human, your brain wants to— memory recall is System 2, your slow deliberate brain, and your brain is going to want to switch to System 1 because remember our brains are lazy. And so as you're telling me your story, you're going to inevitably start to generalize. You're going to be telling me about a specific time and then you're going to say, we always watch blah blah blah as a family. I'm gonna be like, okay, tell me about last night. Did that happen in this instance? So another skill we teach is to redirect generalizations. Back to the specific instance. These are the 4 elements that the coach grades on. It coaches on— it grades on, did you ask a story-based question? How well did you set the scene? Did you build the timeline? And did you redirect generalizations? And it actually pulls out excerpts from your interview and shows you what you did well, what you did not do well, what you could have done differently next time. And then it gives you a coaching tip. Like some of the errors where we'd get confused, it would think a setting the scene question was a bad building the timeline question. It just kind of miscategorized where they were in the interview. Sometimes the coaching tip would suggest a general question, as you should have asked, tell me about your typical day. No, you should never ask, tell me about your typical day. So it would just like, sometimes it would just get confused between the parts of the interview. Sometimes it would suggest a bad question. And so my evals were just designed to detect these errors, and then that allowed me to reduce them.

That makes sense. It sounds like the interviews have some structure in mind. And that helps you break them down and to evaluate them in, in a certain way. So I think that makes sense. What is the entry point to this kind of coaching? Is it like, uh, you do have to take a course to then avail yourself of this AI? Is it like something you could just like, if I'm one of these people who I'm like, oh, I talk to a customer and I'm probably not doing it correctly, buy the software and like, what is the entry point?

Yeah, so it's part of our Story-Based Customer Interviews course. Which is a totally on-demand course. And as part of that, you get access to the interview coach.

And is that the best entry point? Let's say I want to get started doing this. I'm like, okay, you know what, I'm convinced by this conversation. Like, you know, I don't feel like I'm doing this wrong. I want to do this right. Should you start with that thing?

Yeah, I actually think the Story-Based Interview course is probably the most common entry point for people. I think interviewing is the highest return on your time in discovery, which is a little counterintuitive. You'd think assumption testing would be, but too many teams start with a solution that isn't solving a problem. And Interviewing will help you find that sooner.

Okay. It's like error analysis.

Yeah, exactly.

Same relationship. Okay. Interesting. Okay. So that's interviewing. There's a lot more that, like, kind of in the whole process as a whole. And I don't want to skip any steps. So, but I know one of the important things is synthesizing the interviews and like creating these opportunity solution trees and doing that. And one of the most interesting things in your writing Because you've chronicled like your journey of like introducing AI into the teaching process. So people, they obviously want to take all this information and they want to synthesize it and they want to create this artifact, this opportunity solution tree. And you've written about the tension here of like people have these, you call it magic pills, like they have these magic pills and they're gonna point AI at all these interviews and they're gonna have the magic pill. They're gonna swallow the magic pill and you fought it for a while. You're like, oh, I don't know if you should use the magic pill. You write about how you're gonna give them a better magic pill. So can you talk about that? Like what the trade-offs are, how you let AI help someone without having the right trade-off of like having the human in the loop and so on and so forth. Like, can you talk about all the trade-offs and what happened there?

Yeah, so talking to your customers is just the beginning. Like you're gonna have 3 customer conversations and you're gonna very quickly be overwhelmed with all that you're learning. And so the next step is like, what the hell do I do with all this data? Like, what's my synthesis process? And unfortunately, what a lot of teams are doing today is they're just taking all their transcripts and they're dumping it into a folder for Claude, or they're uploading it to NotebookLM. And I get it. I use AI literally for everything. So like, I get that, like, intuition. The challenge here is there's a lot of ways to synthesize transcripts depending on what we're trying to learn. And unless you're an expert researcher and you've like taken the time to identify your research goals, you designed your research specifically for those goals, you're giving all that context to the LLM, it's not going to do a very good job. It's just going to give you a very shallow— like, it's going to tell you some insights, but I don't— they may or may not be grounded in those transcripts. They may or may not be what you actually needed to learn. They may or may not support the decisions you're trying to make right now. And so like there's a million types of customer interviews and you may not even know the type of interview you're doing. And then you're just throwing, or you might even have like 10 people on your team doing 10 different types of interviews and you're throwing all that to Claude and saying, just figure it out. Like it's just, you're not gonna get good results. And the challenge is I have a lot of empathy for this because remember I said we're being asked to do this really deliberate, rigorous process in an environment that's saying, give me an answer tomorrow. Human synthesis is ridiculously slow and takes a long time.

And I know that people listening to this are gonna be very skeptical. They're gonna say, what do you mean? Like, AI's not good at this? No, they'll be like, no, like my AI is good.

I wrote a great prompt.

I know this. And actually this fascinating, it has a parallel again with evals. Eval has this auto eval thing that's been cropping up lately. And auto eval is where you have an LLM look at all of your logs and try to do error analysis for you. And like tell you what's wrong. But it, like you said, it can't read your mind and it can't infer like what's important to you. So my favorite example is we have this apartment leasing assistant that we like to teach with this dataset. If you throw that into any auto eval system, one kind of behavior that it'll say pass on is objection handling. Because the AI agent, when someone asks to like look at an apartment and it's not available, they'll just say, hey, we don't have anything for you. Have a nice day. And the customer will say, thank you. And LLM thinks that's amazing. LLMs will say, good job. You did what you needed to do. But like, in the product frame, it's like, no, this is a sales assistant. Like we don't, we need to have some level of objection handling. Is there an example that, you know, your side of the world like that you have of, hey, like throwing it, having synthesized interviews and like obviously going to miss something to make it real for people?

I think there's a few layers to this. So one is just how shallow. I teach story-based interviewing, so all of my synthesis effort is on synthesizing a story-based interview. So this is one way, this is one type of synthesis. What I'm listening for in an interview is where do you have unmet needs, pain points, and desires? I'm trying to identify opportunities. So if I give a transcript of a story-based interview to Claude and say, can you identify opportunities? This is actually the very first thing I did, like, A year ago plus as my first AI experiment was like, how good is Claude at this? I was pleasantly surprised. Claude did a reasonable job. It missed things that I found and it found things I didn't find. And so like, that's both good and bad, right? Like if I just relied on it, I would've missed some opportunities that I uncovered, but it actually was a good teammate because it found things I missed. But how an opportunity is framed has a big impact on how we might solve it. So let's say I frame an opportunity as I get too many Slack notifications. This is something probably everybody can relate to, right? And how am I going to solve that? Okay, well, if it's framed as I get too many Slack notifications, I'm immediately going to focus on notifications, the solution. But if I frame that as I don't want to be interrupted when I'm doing focused work, now I have lots of potential solutions, but almost no human says, I don't want to get interrupted when I'm doing focused work. What they say is, I get too many Slack notifications. And so what Claude is gonna capture is, I get too many Slack notifications. And you're missing the real need. Like, you don't complain about too many Slack notifications when you get a notification about an urgent matter that you need to pay attention to right now. You complain about getting too many Slack notifications when you get a notification for something that's not relevant to you right now. So that's really a relevancy need or a timing need. And we're not like, Claude is never gonna pick up on that. And so this is another skill, by the way, that people have to learn to be good at discovery habits is they have to learn how to frame needs in a way that opens up the solution space. So if we frame a need really narrowly, we're now stuck in just trying to fix broke notifications. But if we frame the need to be like not about our product and about what is that, what our customer is actually experiencing, I'm getting interrupted when I don't have time to be interrupted, that opens up way more of the opportunity space. And I don't like to teach an agent to do this well, to like stay out of the solution space, to frame it based on the human need and not based on your product. Like people think like, oh, it does a great job, I told it all about my product. Yeah, but did you tell it all about the humans? That's the important part. And so one thing I'm coming to realize, so I've spent the last, what is it, it's August, I spent the last 6 months working on AI-generated synthesis. And one thing I've come to realize, like so many people have asked, why are you spending so much time on this? I just built my own pipeline in Claude. And here's what I realized. Most teams have never seen a good opportunity solution tree. So you mentioned that this tool is really popular. The problem with a tool becoming popular is other people write about it and they write about it in very shallow ways. And then people read about those shallow ways and they misunderstand the purpose of the tool. And I'll tell you my evidence for this. The purpose of an opportunity solution tree is to help you find the best path to your desired outcome. So the root of the tree is an outcome. Then you interview to learn the opportunity space, and then you design solutions to match that opportunity space. So our purpose is to reach an outcome. We populate it by doing interviews. Do you want to know what the 2 most common questions I get are? I don't have an outcome. How do I use an opportunity solution tree? I haven't done any customer interviews. Can my opportunities come from all these other places? Purpose of the tool is to help you reach your outcome and to make sense of your interviews. People think of it as an artifact that like has value In itself that like having the artifact is valuable. It's the synthesis. It's a synthesis exercise to help you make sense of what you're learning. Synthesis is so cognitively hard and you can scaffold it, but you're not going to remove the cognitive challenge. Like that is the work, but business doesn't give us time to do cognitively challenging work ever. It's a little depressing to realize, but like very few people have time and energy in their workday to do cognitively challenging work. I find myself in the situation where I look at all these teams' trees and I'm like, where do these opportunities come from? They're like, oh, we just made them up based on what we know. It's like saying I made up my errors in the evals.

Yeah, this is like saying you use generic metrics off the shelf.

Yeah, like it's, it's bad. And so I've seen this for years. I've been fighting against it for years. Like it's not changing. And so when I started like collaborating with Claude on my own synthesis, I was like, holy crap, Claude is already better than the average product Claude is better at the cognitively challenging work than the average product person. And when I say product person, I mean product manager, designer, software, anybody who's involved, right? And this was like very disturbing to me in a lot of ways, because like I think humans benefit from doing synthesis. And like I really resisted this, but I have to face reality. I can teach the AI to be better than most teams. And if I can do that, like why wouldn't I make that in the world?

That's really interesting. I mean, that takes a lot of humility, right? Because like at the top level, is Teresa's skill. And then there's the average person's skill down here. And then the middle is like Teresa's AI. You want people to be at Teresa's skill.

And I hate saying this because like it sounds like a lot of hubris, like you suck at this, I'll teach the AI to be better than you. Like, this is just— I don't think it's that humans can't do this or can't do this well. I think our business environment is such that it's really hard for most people to build the skill and find the time to do this really rigorously. And I have seen over and over again that when you do it really rigorously, you learn at such a level that you can build much better products. And like, I want more teams to have that. And then I'll tell you, I read a paper that completely convinced me this was the way to go. A research paper came out about the impact of working with AI and who is getting the best results. And using AI does raise the floor for novices. And this was like across a wide variety of like areas. Like you've never been a marketer, You want to learn how to like market your new blog, AI will raise the floor. Okay, that's what I was already seeing. The part about the paper that really excited me was experts working with AI also raised the ceiling. And I was like, oh, okay, now I have goosebumps because you just talked about like regular people, Teresa's AI, Teresa. Now there's Teresa with AI and that's even higher. It's like, okay, so now I can attack this both directions. I can give you an AI that will help you do the work that's going to raise the floor. I can build teaching tools to help you build your skill. And if you work with my AI, that will raise the ceiling. Once I learned that, I was like, I would be crazy to not invest in these tools.

Very interesting. I mean, as an educator myself, I'm really fascinated, like, how education might transform with AI. And I think it's an important problem to think about. Do you change the format of your courses, like your offering? Did you do anything? Can you talk about that a little bit?

Yeah. So for a long time, I resisted on-demand courses because I really believe in deliberate practice. And our live cohorts was all about practice. We now have, we still have a cohort-based course. Our flagship, like, beginner course is still cohort-based and probably always will be. But we now have moved our advanced curriculum to be on demand. And that's because they're all supported by AI tools. So students submit their homework, they get personalized, detailed feedback. It supports this, like, infinite rounds of practice for people who are motivated to do that. The other thing I'll share is, like, I think I experienced this with engineering and that's what's opening me up to this idea of like, if I build a tree for you, if I do your customer interview synthesis for you with the AI tool, as long as I show you what the tool is doing, like I don't just say here's the answer, but I show you the work in progress, I actually think that is a teaching tool. And the reason why I say this is I have learned a lot about engineering over the last year. Just for perspective for folks, a year ago I was writing some code to like automate my backend business logic, but I literally did all of it in the This is embarrassing. I did all of it in the Amazon Management Console. I wasn't using an IDE, I wasn't using Git, I had no unit testing or integration testing. Like, I just like was piping data from one tool to another tool and just like writing Lambda functions and Step Functions in the console to just make it work. And I was at a small enough scale that like when I had errors, I just went in and fixed things, you know, like it was just duct tape and shoestrings. But as I started building AI products, and especially I have a AI products now in a real production software tool, I had to learn how to be an engineer. And when I first started like working with Claude in like VS Code, I didn't know the AWS command line. I didn't know any of the Git commands. I didn't know, I barely understood like why do I have to add files and then stage them and then commit them? I don't know what feature branches work. Like I didn't know any of this stuff and I didn't need to know it. I would just tell Claude like, can you commit this change? Can you tell me what's in this DynamoDB table? Can you, right? And like Claude did it, but this was at a time when Claude showed you everything it did. It actually drives me nuts that it doesn't do this anymore. And I learned a lot. Like I now know how to use all these tools myself because I watched Claude do it over and over and over again. And so when I'm building AI synthesis, I'm building it in a way where my goal is to show the process so that as you use the tool, you're actually building your own skills at the same time.

And can you talk a little bit about that? So I know that you've partnered with a company called Vistalee that helps with this opportunity solution tree and maybe some other things as well. I think this story of how you built it is one of the best eval stories I've ever heard of. Can you talk about that?

Yeah, so there's this company Vistalee. They've been around for a few years. They're definitely a startup. They basically took the opportunity solution tree idea from my book and built software around it. So they have a canvas tool where you can add tree nodes. Here's the problem they're trying to solve. A lot of people build their trees in like Miro or Mural or Jamboard. I don't think Jamboard exists anymore. FigJam, it's just a digital whiteboard. But the problem is like you need to move things around and then all the arrows get moved with it. So what they did was they built a purpose-built tool just for the opportunity solution tree where there's nodes like you can see here, and there's at the top is your outcome, and there's opportunities, there's solutions, You can add your assumptions that you're testing, and they just built this like really slick interface for opportunity solution trees. And so they joined my community a few years ago. I got to know the founders. The very first partnership we had was we actually integrated my interview coach into their product. And then as I started to experiment more with AI synthesis, we started to talk about what if we brought all that AI synthesis to Vistalee? And so in Vistalee, you can upload your interviews. My AI service will extract key moments and opportunities. So that's 2 of the elements on an interview snapshot. Once you have 3 interviews, we will generate your first draft of your opportunity solution tree. And then every 1 to 3 interviews more, you can choose to update the tree as you do more interviews. And it's all AI driven.

And my understanding is you are the chief AI developer, like you are responsible for all the AI in Vista.

They license my AI services. So when you upload an interview and it generates a snapshot, that content is coming from my AI service. When you generate your opportunity solution tree, the whole tree is coming from my service.

Like a private API somewhere or some endpoint that they're calling and is doing something?

We technically, they actually have my source code. So they, because of data requirements, so they're HIPAA compliant, they're SOC 2 compliant, they follow all the GDPR rules. They're in the process of getting some ISO certification I don't know about. I didn't wanna do any of this stuff. Like, I don't really wanna run a software company. I just wanna be an AI researcher. So the way we set up the deal is it's actually source access partnership. So they deploy my software in their AWS environments. That allows them to keep their European customer data in Europe. I don't have to worry about running multiple AWS accounts or instances or getting all these certifications. And then, so I just, they're part of my GitHub repo and they, when I commit, They deploy in their environment.

Okay. This is pretty sweet. And to come back to your point, okay, these trees. So you wrote about this journey of like, okay, like creating the AI for these trees and back to your point about showing the work. So you don't just create trees, you show people the diffs of the trees as the trees evolve. So you, so you know, from like step 1, step 2, step 3, step 4, how this tree changed over time. And it's like a non-trivial computer science problem. Can you talk a little bit about that? Like what happened, how you Solved it.

So I'll say there's, there's actually 4 AI services that I build. So we'll talk about the different components and then it will help with this. The first is I have to classify your interview transcript. Is this an interview I can even do anything with? And that seemed like a trivial problem because I just want to know, is it a story-based interview? Okay. We launched our beta. We got 500 teams to upload 3 interviews. Of those 500 teams, I think 8 of them had story-based interviews. So very quickly we were like, oh yeah, people don't know how to interview. I forgot this problem. So we had to expand from story-based to just general interviews. And then this is a problem. Can I even identify opportunities from general interviews? So like even on day 1, this turned out to be a way harder problem than I thought it would be.

So why did you relax that requirement of having a story-based interview? You could have gone either way. You could have said like, hey, like, no, you have to have a story-based interview because that's what we teach or whatever.

So my inclination was to say we're just going to support teams that do story-based interviews. We would have had 8 beta customers. Vistalee was like, hey, we need to make money. We're a company and we need to grow. I actually shared this concern. We were concerned the market of teams that do story-based interviewing, and especially story-based interviewing well, would just be too small. So we're trying to meet teams where they are. The way that I reconcile this for myself is I'm coming up with this model of strength of evidence. So if we identify an opportunity from a story-based interview, that's a stronger signal than identifying an opportunity from a general interview. The other thing that I did before I agreed to expanding this was I ran experiments with some of the general transcripts To figure out what can I extract from this and what do I think that quality of evidence is. So we don't extract everything from a general interview. We only extract things that— in a general interview, like if I ask you, what do you like to watch on Netflix? You might answer that question by telling me a little mini story. That could be an opportunity that has just as much strength as a story-based interview, but it came from something that wasn't a story-based interview. So I recognized pretty early on, like even though this isn't classifying as a story-based interview, there are strong enough signals that I can still identify opportunities. Now we actually got 8 different types of interviews uploaded. We got like team meetings, which I think were like— people thought they were stakeholder interviews. We got sales calls, we got usability sessions, whereas we decided we're not supporting any of those out of the gate. Usability sessions we will eventually support, but they will be used to evaluate assumption tests, not to generate opportunities. This has been one of the most interesting things.

Is there a way you could like nudge those people into doing more story-based interviews, or is there a way— how do you do that?

So long term One of the things I'm working on is this other product right now. It's called TheresaBot. We'll see what the name stays in the long term. But the idea is that TheresaBot is a discovery coach that when it can see your work will nudge you towards better stuff. So if I can see your interview transcript, I can tell you like next time try asking this instead. I can just give you like little mini tips to like move you in the right direction. So there's a lot of stuff we plan to do down the road to help people get better quality of evidence. What's been super fun is that like I have a much— like as a coach, I got to work with like dozens of teams at a time and see what their interviews looked like. I only sometimes got to see their transcripts. Like now transcripts are everywhere, everybody records everything, but like historically that wasn't the case. Now I get to see like the actual data, like this is where teams are. So that's super fun and it opens up all sorts of opportunity, like teaching opportunities. Can I nudge you towards better interviews? Can I teach you about evidence quality and like this is a weaker signal than this. There's like so much I want to do, but there's also so much to do. So the first service is I classify your transcript to determine if you're even qualified. The second service is if your transcript is qualified, I extract key moments and opportunities. So key moments are like what happened in the story. Opportunities are unmet needs, pain points, and desires. The third service is we take 3 interviews and we generate a tree from scratch. So the agent gets to decide everything. These are the key moments on the tree, these are the opportunities, and we're just literally building it from scratch. There's a 4th service which is update a tree, and the reason why that's a different service is after we generate a tree, a human could go in and edit things and like rename things, add some things manually. And so when we update a tree, we don't want to lose any of the human edits, but we want to like correct And so there's a little bit of like a more of a provenance problem in update because there's like a base tree that we have to manage what we can edit and not edit, if that makes sense. And so it's only on update tree that we, I build change sets. So my service will take in an existing tree, some new interview snapshots, and it will output a new tree, but it will also output a set of change sets. So we defined a vocabulary of moves that the agent is allowed to take.

Is there a visual of change sets that you think I can find somewhere?

So I output the tree, and then along with the tree, I output this change set where it's literally walking through exactly what the agent did. For— this is a change set for a new tree. We're adding an outcome, we're adding this opportunity, we're adding this opportunity. And so this is a really simple change set. Some change sets are Really complex. Like, what if the agent wants to take this opportunity A and split it into A and B? This opportunity has sources on it. The sources are what interviews did this opportunity appear from. And so if we're going to split it, it also has to split the sources. So when the agent decides it wants to split A into A and B, it also has to decide which sources go where. And the reason why we're doing this— there's other complex moves like a merge, So it can decide to merge A and, or B and C. And in that case, the sources get combined and the children get combined.

This is like a LeetCode hard problem.

Yeah. And I blindly wandered into it.

Yeah.

Yeah. I blindly wandered into it and solved it. So the reason why we're doing this is when you update your tree, I don't want to just say like, click the magic button and then here's a new tree. We want you to see what changes happened on your tree based on the new content. So we want to say like, and we want you to be able to follow the provenance. So we added opportunity B, and here's a link to the interviews that had that. And Vistalee has built a really cool interface for this. Like when you open an opportunity on the tree, it will show you exactly which interviews that opportunity came from, and you can even play video clips of that opportunity so you can hear real customers talking about that need. And this is really important to us. We wanna make sure everything is grounded in real customer data that you can always trace back where everything on your tree came from. And you can see the steps the agent followed to like add things and merge things and split things. And so we, to make this work, we had to define like a whole vocabulary of moves that the agent could make. And then it's not just about producing the tree, but also making sure that you can produce an accurate set of moves that, that the idea is that you can take the input tree, apply the change set, it will generate the output tree. And we talked about doing this because we wanted to like walk the user through like how to update trees. Like we want them to learn how to do this themselves. I didn't realize at the time like how hard of an engineering problem this was. And I had some bugs. The LLM basically would generate change sets that didn't generate the tree. And first, actually, that's not true. My bug was I wasn't having the LLM generate the change set. I was deterministically generating the change set after the fact.

Oh, okay.

So I would look at the input tree and the output tree and try to just create a diff. We were like inspired by Git. Can we just create a diff for these 2 trees? The problem is, is the diff is ambiguous. There's more than one way to generate the output tree, and especially with trees like merges and splits and the order in which they happen is really ambiguous. And so my deterministic change sets weren't always generating the output tree. And so the way I had to fix this is I had to get the LLM to make basically like a scratch pad of all the moves that it was making and then use that to generate the changesets.

And what do you mean by scratchpad?

Yeah, so the LLM, it used to just like, like one step it will do is identify like structure a branch. That's one of the steps it does for branch. It'll add structure to the branch. So originally the LLM would just output the structured branch. I updated it so it outputs the structured branch and the changesets for how it generated that branch. This actually improved the quality of the agent.

Like a text file, JSON file, something like that?

Yeah, it's all the LLM always just outputs JSON. So the tree is represented in JSON, the changeset is represented in JSON. And this actually improved the quality of the agent. Like when I had to show its work with changesets, it reduced a whole bunch of other errors. So that was like a nice surprise.

It was like a chain of thought almost.

It's like a chain of thought. And then it also allowed me to add an agentic loop where it'll generate the, the branch and the changeset. It can call an audit tool and the audit tool will take the input, run the changeset, look at the output and make sure they match. And if there's any mismatches, those errors go back to the agent to fix.

And I don't necessarily recommend this on every kind of problem, but because your trees have this quality, it seems like you're doing some computation, some intermediate computation, and there's like quite a lot of computation potentially to check different things, to do like to audit things. There's this thing called recursive language models. It's the naming of it is very scary, but really what it is, basically it's like giving your LLM a kernel, basically like, like a Jupyter Notebook, like an IPython kernel that can go like fiddle with, come back to you. So it might be interesting. Yeah.

You know, what's been fascinating about this is I've toyed a lot. Like right now, most of my processes are still pipelines. They might, in the pipeline, they might have an agentic loop step, but it's not like fully agentic. And. Part of the reason for this is just cost. Like, I experimented with an agent that had skills for each of the pipeline steps and just let it like try to attack the problem, and it would be like $25 per run. Whereas right now with my pipeline with some agentic steps, I'm at like 50, 60 cents per run.

It works really well. You don't need to fiddle with it.

You know what's fun is that like I set up my partnership for me to just be an AI researcher. Like, I don't have to deal with any of the company stuff, which is really amazing. And so it gives me the flexibility to be like, I'm just gonna try this and see like, what if this was an agent with a bunch of skills? Or what if it was an agent with a bunch of tools? Or what if it was, this was an agentic loop and this was a pipeline? And I just sort of play with different orchestrations. And I obviously, when I release something, I have to care about the impact on cost, but I don't have to do that for my own experiment. I can afford to do a $25 experiment just to see what happens and like to see the difference in quality. And that's, Because there's so much complexity to these layers of synthesis, it's been really fun to like run all those experiments and start to learn like, oh, this brand new thing came out, but it's just not viable. Like it's just too expensive for right now. Or like even just playing with what models are good at what steps and how do you optimize cost on that. I actually just watched the model cascade, the Lightning lesson you did, and I was like, oh, this is perfect timing because I have some evals that I had to use Sonnet for and it's just So expensive. And I was like, oh, but I could do a model cascade. So I'm having a lot of fun just like tinkering with those things. And this is, I think, the benefit of my partnership with Vistalee is like, they're the startup, they're being startup employees, they're running the company, they're doing the sales, they're doing all the data practices. And I'd really just get to focus on the AI research part and like on my domain expertise of the discovery part.

I need to find a situation like that. Maybe like the eval SaaS company.

If you find an opportunity like that, we should talk about contracts because I feel like I ended up in a really good situation and Vistalee feels like they ended up in a really good situation. So I feel like it was a, we found a way to make it a win-win, but it wasn't easy. We spent months talking about the right way to do it. But I think we landed on something that works really well for both parties.

Is there anything you can share anyone that does teaching might benefit from if they're coming to this type of situation? Like anything to think about?

Teaching was never about transmitting knowledge. I think it's really easy to think about it as like you're the teacher and your job is to impart knowledge on your students, but I think that's like a very crude like way to think about teaching. I think about it now as my job is to create spaces for people to learn, and that sounds very hand-wavy, but I'll give a couple of examples of this. Like my story-based interviewing course, which we talked about Yes, there are teaching articles and there's teaching videos, but I know from teaching this course for 9 years now that that helps you learn concepts, but it doesn't help you apply the concepts. The way that you apply the concepts is through structured practice. So my job is to create a way for you to do structured practice. Another way that I do this, like when I started writing about my, all my adventures with AI, people started asking me like, why aren't you creating an AI course? I don't really want to create an AI course because it changes so quickly. Like I don't want to do all the work to like Constantly revise it. And if I think about like, there's so much knowledge about AI, people don't need me to impart more knowledge about AI to them. Like it's already out there.

The revising stuff is so true. I, we had to just completely redo our entire evals course. Yeah.

It's just, it's a lot of work, right? What I realized was like, I can create a space for people to learn without creating a course. So like in my community, I host AI Maker Studio sessions and I host AI show and tell sessions. So the Maker Studios are, we just show up and we like co-work on Zoom. It sounds so stupid, but it reserves space in people's calendars to like work on personal productivity things with AI with a whole group of people that like when they get stuck, they can ask a question. And then a week later, we do AI show and tell where people show up and they show and tell what they built that week. And it's just enough for like a total beginner can show up and get inspired. We have a whole wide range of people with experiences. And so like, I am really changing the way that I think about teaching. I think knowledge is basically free at this point, and I think like the real value is how do you create opportunities for practice? How do you create space for learning? How do you build community around this? I remember I, I went back and I got a master's in learning and organizational change, and one of the things I learned in one of my classes was something like, you don't learn on your own, you only learn in community. And when I first heard that, I thought that was baloney. Like, I'm a reader. I learn a lot alone, but I was thinking about learning as acquiring knowledge. And I think learning is more about acquiring skill and acquiring new habits and adopting new mindsets. And I think all of that requires learning in community. And so I think AI can accelerate skill building because it can help with like infinite practice. But I think there's also this like really human part of it of how do we create community around this and bring people together. And so. I'm not really like extending my curriculum anymore. I'm, because that's like the knowledge piece. I'm extending my curriculum with AI tools to like support infinite practice. And I'm using software to show what good looks like. Like what I love about Vistalee is we can show teams what a good tree looks like, what a good snapshot looks like. Eventually we'll add some coaching nudges into the platform where as you work, you'll get feedback on how to improve things.

And then.

I'm really looking at how do we learn in community.

Tell me more about the community part. What are you experimenting on with community, or thinking about in terms of community, if anything?

I think people learn by seeing and by doing. So, you know, there's this like mantra of like, I do, we do, you do in teaching. So it's like, I'm going to model it for you. We're going to do one together and then you're going to do one on your own. This is like really common and it's grounded in a lot of evidence. So I can describe something to you, but until you see it, you don't really get it. And so some of community is just bringing people together to show. So our show and tell is just come together and show what you're doing. And a lot of people don't know what's possible with AI.

Oh, cool. Okay. So you have this community. It's very reasonable. Yeah. $399 for a whole year. Wow. That's—

I am at the point where I'm very intrinsically motivated by what I do. And for my community stuff, I want to make it as accessible as possible to as many people as possible. It's not my primary revenue stream. It really is just like bring people together and see what emerges. It's like a playground for me.

How many people roughly are in your community at the moment?

A lot of people subscribe for free. There's a lot of things, you know, to be honest, I don't do very much for my free subscribers. They just get emails knowing there's content coming. A supporting membership is where I do a lot of events. A few thousand on the supporting membership level. And then the CDH membership level is a little bit complicated for numbers because we— there's 2 ways to get access to this program. The first way is through this subscription. You can pay $359 a year. I think we have a few hundred people that do that. It's not giant, but this gets you everything in the supporting membership, so all of our events, but it also gets you access to our Slack community. But our Slack community is bigger than a few hundred people because of our courses. If you take any of our courses, you get access to the Slack community for 3 months. And then our core, our companies that work with us get access to the community for a full year. So there's a lot of like overlapping circles for each of these programs. Here's what I'll say. On the community side, I look at it as a 2-pronged approach. The first is, can we create events where you can just come together and learn from each other? So that's our AI Maker Studio, our AI Show and Tell. We do office hours, we do community calls, just opportunities for people to get, come and learn from each other. Because so many people don't know what's possible with AI. And like, for those of us that are like doing this all day, every day, we forget that normal people are still using ChatGPT like Google. And so like, we have people just demonstrate all sorts of stuff. Like someone made a coloring book for their kid, like a physical coloring book, and had no idea how to do that. A lot of just cool projects. And then the other thing I'm starting to experiment with both of these subscriptions is like that CDH membership gets access to my Teresa bot. So this is where I'm running all my experiments of like on-demand coaching nudges.

Okay, you like get grilled by Teresa.

The Teresa Bot started really simple. It was just a vector database with all the content I've ever written. So all my course content, all my book content, all my articles. And it started as like, you could just ask Teresa Bot a question and it will point you to the right content. So it was sort of like a concierge. Because of my work with Vistalee, I was like, well, can Teresa Bot be a coach? And so I started to give it skills. And so I've built Like a number of little AI tools to support our courses. So like I have an outcomes coach, I have an interview coach, I have like a coach that will give you feedback on your research questions and your interview questions. I have a coach, our business fundamentals coach will give you feedback on like how you define your ICP and some other things.

And these are different bots that you tag in like Slack? They're—

All those things that I just rattled off are like AI tools in our courses. And so what I did was I basically gave them to TheresaBot as tools or skills. If you ask TheresaBot like, hey, this is my outcome, can you give me feedback? It's not just finding an article to give you an article to read. It's actually invoking the outcome coach and will give you feedback based on that.

Okay, so there's like some tools in here, like in this thing.

And actually it's not updated on here because like all of this is just like in the last month. I haven't revised these since I launched them. So like the way that I would describe these subscriptions is like both of them give you access to all of my premium content on the blog. The supporting membership level is for people that like want the community events. And then the CDH membership level is for people that want like the 24/7 Slack community, want Teresa bot. And then they're gonna stay in that split. Like supporting will be very driven and CDH membership will be very AI tool driven.

For anyone watching, I would encourage anyone To support Teresa, read her newsletter. I'm gonna be joining this community right after this call. It just came up organically. I'd never talked to her about it before, but one of the increasingly rare creators who like, she actually puts a lot of thought into her writing. There's like no slop that I can see. It's like very nice to read. And that tells you a lot. Thank you, Teresa, for doing this. I'm looking forward to what you create and I'm looking forward to having you in this next cohort. We're going to be talking about a lot of new ways of doing evals that use AI in the loop, kind of very in parallel to what you're saying with discovery.

Yeah, I'm excited. Actually, my next blog post, I'll give you a sneak peek, is I spent 3 weeks about a month ago chasing down an error. It literally started with a customer. I was on a customer call getting feedback. We were watching them. react to their first AI-generated opportunity solution tree. It was awesome because they had hired an agency to conduct their interviews, and the agency gave them a research report, and then they took those interview transcripts and uploaded them to Vistalee.

Wait a second, is it a good idea to have an agency do your interviews?

Most of the time, no. So I'll share, at the very beginning of my consulting career, people hired me to do their discovery for them. This made me really uncomfortable because discovery needs to be continuous. So like an agency will do a project for you, but then what happens, right? And so the reason why I shifted into coaching, I basically, when people said, will you conduct these interviews for me? I would say yes, but you have to send your product team with me so they learn how to do it themselves. So, but what I loved about it for this customer call, because an agency had already like conducted the interviews and done the synthesis and they had a research report, she uploaded those interview transcripts to Vistalead and saw the opportunity solution tree and like her feedback. She was blown away. She's like, wow, you found everything the agency found and more. And then she paused and she goes, and we spent a lot of money on that agency. Already I was like, man, can I just have this as a testimonial? But then she's like looking through her tree and she pauses on one branch and she goes, wow, there's a lot of unsorted opportunities here. I'm gonna have to come back and clean this up. And of course, that's the quote I remember because that is an error. That's a failure mode. And so I immediately, like, after that call, started to look at that failure mode. And I thought it was just gonna be a quick fix, like I just needed to like tweak a prompt or something. And turned out to be one of the hardest evals I've ever written. It's an LLM as judge, and I really struggled to get it to calibrate. And the reason why I actually asked you about— while I was right in the middle of this, I sent you a text about it. I— that was the silly fruit example thing I gave you. The reason why the judge was struggling to calibrate is because there were confounding errors in the samples in the calibration set. The judge would The error was, so on a tree, you have a parent, a node on a tree is like a parent with a set of children. And as the children grow, you should be adding structure. There's probably subgroupings. And so the error is missed groupings. Like there was an opportunity to group these children and the agent missed it. And I thought like, this is gonna be easy. I'll just measure that and off we'll go. Well, it turns out like trying to reduce that error exacerbated another error. And then trying to reduce that error exacerbated the first error. So it's like telling an agent to do one thing created one problem, and then trying to reduce that problem created the original problem. And I spent 3 weeks, like, I think I had to create 4 new evals. I ran 16 experiment variants, like big experiment variants, not like little tiny tweaks, before I found a solution that like was able to reduce both errors. And I learned a lot about, it's so easy to think you can fix everything with a prompt. What was actually required was more of an orchestration. Experiment, not a prompt experiment. And you know, when I first— when I was struggling to get the judge to calibrate, I saw the confounding errors, but I didn't want to fix them. And the reason why I didn't want to fix them is because this was like nefariously tricky. No customer was complaining about the other errors. So I just had a customer complain about this missed groupings. The missed groupings judge was getting confused because there were some poorly framed opportunities, but no customer had ever complained about a poorly framed opportunity. Like they were good enough for the customer. They weren't good enough for the judge to not be confused by them. And so like I really resisted wanting to fix it. It felt like I was wasting my time fixing something that like was good enough. And so like, remember when I reached out to you, you were like, you have to attack the upstream errors. I was like, I don't want to attack the upstream errors. Like a customer doesn't care about those errors. But it turns out I had to fix the upstream errors. And then fixing the upstream error exacerbated the error I was trying to fix. And then fixing that error exacerbated— so there was like this weird back and forth between 2 different evals that like were counterbalancing each other.

I'm gonna create a snippet of this portion of the interview because I can't tell you how many times people, they want to just like create evals for like every single step and they get like all this analysis, everything. I'm like, you need to focus on the upstream error, and everyone resists that. This is a great story.

So my next blog post is the really detailed account of what I tried, what was hard, and then how I resolved it. And the way I eventually resolved it was I have this agentic loop and I just added my evals as guardrails. And so I couldn't get the original agent to be good at both. It would always have one error or the other. And so I took my evals and I run them as guardrails and it gets kicked back to the agent and it has no problem correcting on the second loop. So instead of trying to like make magical prompts that could just do it right from the first time, I just gave it a tool and that pretty much solved it, which is pretty wild.

Really look forward to the blog post. Yeah. Sounds really, really cool. Asked too many questions to ruin it for people. I'm gonna, I'm sure you're gonna share a lot of the like cool details about like what the evals were and like the different iterations of them. When is it gonna coming out?

Either this coming Wednesday or the Wednesday afterwards.

Well, it was really nice to talk to you again, Theresa, and thanks for sharing all this information. I think it's gonna really help people. Especially if they are thinking about like talking to customers. I think you can do anything. Check out Teresa's book, put it on the screen somewhere and check that out. Check out her community. And yeah, look forward to seeing you in class as well.

I'll say, since your audience is a lot of engineers, I know a lot of product teams that like are being taught include your engineers in discovery, but there's like some resistance from engineers because I don't think this is engineers' fault. We've been telling engineers for decades, like, just build what we tell you to build. But I know engineers are makers and they care about what they're building. And I feel like there's a huge opportunity right now. I think most teams are being asked to be product builders, not just engineers anymore across all functions. And I have a ton of free resources at producttalk.org. The very first article at the top of the page called Product Discovery Basics, like, is a great place to start.

Okay. Product Discovery Basics. So it's like right here in the banner, click on it.

And that visual you're looking at is an opportunity solution tree right in the middle. This is just going to cover like the high level. This will give you sort of like, it'll start to build that mental representation of how this stuff works. And then if you're a reader, Continuous Discovery Habits is available in paperback and Kindle. And there's also an audiobook that I narrated, which was a lot of fun. And then if there's one skill to invest in, I would say build your interviewing habit. And so if you want some of that infinite practice, Check out Story-Based Customer Interviews.

Team Discovery Habits. Check it out. I have read this. I also have the audiobook too. It is fun to listen to Teresa. She has a lot of life into the book, so it's great. Well, thanks a lot.

And thanks, Hamel. This was fun.