The UX research reckoning is here | Judd Antin (Airbnb, Meta)

User centered performance refers to customer obsession or user centered practice that is symbolic rather than focused on learning. It's hugely common, I would argue. It's work we do to signal to each other how customer obsessed we are, not because we want to make a different decision. If you're like listeners are like, I don't do that. I'm kind of like, think about it for a second. This is extremely common. Every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study, you know, just to validate our that's user-centered performance. It's too late to matter. We gotta ship it, right? What they want is to check the box. One of my big kind of mantras was, we don't validate, we falsify. We are looking to be wrong. Many PMs, many designers are not in that place. They do not wanna be wrong. They're looking to validate, and that's user-centered performance.

Today, my guest is Jud Antin. Jud helped build the user research practice at Facebook. He was a longtime head of research at Airbnb. and his direct reports have gone on to lead research teams at Figma, Notion, Slack, Robinhood, Duolingo, Faire, and other amazing companies. These days, Judd spends his time consulting, helping companies with organizational challenges, product strategy, design, research, hiring, onboarding, and crisis management. In our conversation, we unpack a conclusion that Judd has come to recently about how the user research field is going through a reckoning. and what needs to change, both within the user research field and how companies leverage user research going forward. Jud shares what the user research field has gotten wrong over the last decade, how PMs and designers rely on user research too often and to answer the wrong questions, where user research will continue to provide significant value and how to best leverage your researchers, why it's important for researchers to think about the business goals more versus just what the users need, What to look for when you're hiring a user researcher, how PMs can be better partners to researchers, and also a phenomenon that I love that Judd describes and often witnesses that he calls user centered performance, where everyone acts like they care about the user, but they're just doing it for show and already know what they want to do. This episode has a lot of spicy takes and will probably upset some people, but Judd is sharing some real talk here that I think we all need to hear. With that, I bring you Judd Antin, after a short word from our sponsors. This time of year is prime for career reflection and setting goals for professional growth. I always like to spend this time reflecting on what I accomplished the previous year, what I hope to accomplish the next year, and whether this is the year I look for a new opportunity. That's where today's sponsor, Teal, comes in. Teal provides you with the tools to run an amazing job search with an AI powered resume builder, job tracker, cover letter generator, And Chrome Extension that integrates with over 40 job boards, Teal is the all-in-one platform you need to run a more streamlined and efficient job search and stand out in this competitive market. There's a reason nearly 1 million people have trusted Teal to run their job search. If you're thinking of making a change in the new year, leverage Teal to grow your career on your own terms. Get started for free at tlhq.com/lenny. That's tlhq.com/lenny. this episode is brought to you by Vanta, helping you streamline your security compliance to accelerate your growth. Thousands of fast-growing companies like Gusto, Calm, Quora, and Modern Treasury trust Vanta to help build, scale, manage, and demonstrate their security and compliance programs and get ready for audits in weeks, not months. By offering the most in-demand security and privacy frameworks such as SOC 2, ISO 27001, GDPR, HIPAA, and many more, Vanta helps companies obtain the reports they need to accelerate growth, build efficient compliance processes, mitigate risks to their businesses, and build trust with external stakeholders. Over 5,000 fast-growing companies use Vanta to automate up to 90% of the work involved with SOC 2 and these other frameworks. For a limited time, Lenny's podcast listeners get $1,000 off Vanta. Go to vanta.com/lenny. That's V-A-N-T-A.com/lenny. to learn more and to claim your discounts. Get started today. Judd, thank you so much for being here. Welcome to the podcast.

Lenny, thanks for having me.

It's my pleasure. So we actually worked together at Airbnb for many years. And as I was preparing for this, I realized how many of the people that you managed went on to do amazing things. So I'm just going to read a list of people that work for you and what they do now. So we had Matt Galavan, who now leads research at Slack. We have Janna Bray, who leads research at Notion. Celeste Ridlin, who leads research at Robin Hood. Rebecca Gray, who leads research at Faire. Hannah Pelagi, who I think was leading research at Duolingo. Luis Berrill, who leads research at Figma. And then Noam, who was leading research at Wealthfront. I think he moved on to something else. What a freaking crazy alumni community and group from this one team that you hired and incubated.

I've never looked at that list, but I'll tell you, I have been so privileged to work with all these amazing humans. I can't take credit for it. They're just outstanding people. And I'm glad the diaspora is out.

There, because these people rock stars. Okay, so the main reason that I wanted to do a podcast episode with you is that you wrote this piece that was titled, the User Research Reckoning is Here. which I understand caused quite a stir in the research community and I think adjacent communities. And let me just read one of your takeaways at the top of your post to give people a sense of what it was about. So you wrote, the user research discipline of the last 15 years is dying. The reckoning is here. The discipline can still survive and thrive, but we'd better adapt and quick. Before we get into the meat of the piece, Could you share a bit about just the reaction to this piece and maybe if it was a surprise and what you expected would happen when you put this out?

Yeah, I mean, I was definitely surprised. I wrote it because I wanted to start a conversation about something I was thinking about. I didn't really know who would read it and in the end, it turned out a lot of people read it. I learned that using the word reckoning may have been a mistake because it inspires a lot of drama. in a conversation that I wanted to be really productive and positive. Overall, I would say though that like the response was very positive. It seemed to resonate with a lot of people who reached out to me. I spent a lot of time talking to teams, to designers, to researchers, but there were also a ton of critiques. I would say some of it was like people thought I was throwing research or researchers under the bus. Like it's researchers' fault. We're doing it wrong, which I don't believe at all. and that I wasn't taking responsibility as a research leader or a design leader myself. And like a real, the most interesting one, I would say, was the anti-capitalist crew. Because one of my points that we'll talk about is that I think researchers need to be more profit focused. And there are a lot of people out there who think that's not, I think they think that's not cool or not research's job. And I'm kind of like, well, what are we doing then if we're not helping businesses succeed? So, but that was the, that was the most surprising critique for sure.

I've, I've worked with some of those people who are just like, why are we growing? Why do we focus so much on growth?

Why do we need to grow this business?

Maybe, maybe it's the wrong industry for them. Yeah.

Yeah.

I'm not a fan of that. Okay. So let's actually dig into the meat of your message and the big takeaway and kind of the conclusion of what you're finding is happening in user research. And I know a lot of this comes from a lot of user researchers have been laid off at a lot of companies. It was one of the hardest hit. teams. And so I think a lot of this comes from that. So, yeah, so let's just start big and then see where it goes.

So, yeah, I mean, everybody who's paying attention has noticed that there have been a bunch of layoffs. And I, I think back in the summer I was thinking, listen, this seems to be hitting UX and UX research particularly hard. Is there something going on? Is there a bigger picture? Is it maybe the, the reason I use the word Reckoning is because to me that's like, hey, a moment to take stock. and triggered by the fact that a lot of wonderful humans may have lost their jobs and many more are afraid of losing their jobs. And so if it's a sign, the fact that research has been hit so hard, it's a sign of what? And so the thesis of my article is really it's a sign that maybe the system is a little more broken than we think. And that research is not driving the value or impact that it should or could. And that's for a bunch of reasons, I think. Some of it is stuff that research can do better. and a lot of it is how research is integrated and positioned in companies. And at the root of all that, I think, is that we're, we're just doing too much of what I would consider the wrong type of research. And what I mean by the wrong type of research is I have this framework that it's in the article, macro middle range and micro research, at least three ways to talk about it. And it's pretty simple, the intuition of what those are. So macro research is like big picture, strategic business focused, forward-looking Innovation, you know, look at the look at competitors, long-term research to understand where the product should go next, stuff like that. And then you have micro research, which a lot of really technical usability falls into this. All the beautiful stuff that researchers do to enable a really high quality, excellent, pixel perfect thing to go out the door. Laser focused research to understand AB test results, stuff like that. And then you have this middle range, which is this blobular place where the research questions are sort of middle altitude. And a lot of the core kind of, let's say, user understanding questions fall here. And a lot of what research is doing is research in that space. It's sort of, let's take a group of people and ask some questions about how they think, feel, behave, how they're using a product or not using a product. And it's just this devastating mix of really interesting To many, including researchers, and not impactful enough for the business. That's kind of the core thesis. Researchers do it because it's interesting, but honestly, and I think we should talk about Lenny, is researchers also do it because it's the kind of work we most often get asked to do.

Yeah, that's exactly what I was thinking. That's what as a PM, that's what I want to get answers to is like, how should we think about this one product? And I totally get this.

Yeah, the questions turn out to be really interesting and there are many cases at many companies where it's super impactful. But the problem with that, those types of questions is they tend to be really, like, they trigger all the worst stuff that researchers experience. Right? So they yield results which are interesting, but sometimes hard to operationalize. They trigger the post hoc bias really, really, really well, where, like, a lot of people can say confidently, like, oh, that was kind of obvious. We knew that already. and they fulfill this kind of need for us to feel and be customer obsessed, user centered without changing anything. And so doing too much of that research to me is a symptom of a broken system, right? And where like companies are really different from each other. I heard from so many after this article and they're like, well, my company in my industry is like this or not like this. But in tech, we spent the last many years hiring, hiring, hiring researchers. And, but maybe I'm sure most of your listeners are familiar with the idea of a Zerp. You know, like maybe it was a zero interest rate phenomenon where it was okay when the money was easy to hire researchers, even though we were not setting them up properly. We were going to set them up to fail. We set them up as a service function. We didn't know what research was for. We didn't know how to, how to really drive impact with it. And, and that's where the reckoning comes from. It's like that era is over. Research, I think is more crucial than ever. Researchers, good, great researchers are more impactful than ever, but it's, it's in a new space.

We're in a new space now. I want to make sure people understand this framework. And specifically, how would you best describe the difference between this middle range research and macro research?

So middle range research is usually focused on a more specific set of research questions or a constituency. So if macro is like, let's understand the overall competitive landscape, let's do a concept car type project where we really look ahead, Let's get involved with strategic planning, which is a wonderful thing for researchers to do, you know, do TAM studies, other things like that. That stuff lives in the macro space. The middle range space is like, what's a good example? We want to know how Airbnb hosts feel about their payment options. That's like a really interesting, reasonable question, right? And we can go out and do research on that, but it's not that specific. It's not really targeted at a business problem yet. It could be, right? Maybe that's a result of the research, but it yields these kind of like middle range insights in which we've learned things like, well, hosts want flexibility about their, you know, payment options. I'm making this up, but, you know, and that's a good example where it's like, it's not that that's not an interesting set of questions. It's just not quite pointed enough in order to like, and it's not framed in the language of, the funnel or the business strategy or the OKRs. It's not quite enough aligned enough to that. It's too blobular in that middle level and it ends up not driving impact.

I think it also leads to a lot of the things as you describe, people don't like about research. It delays everything. You have to wait for the research to be done to have an answer to make a clear decision. It also creates this issue that people complain about that PMs And product teams don't want to just make a decision on their own. They're like, I will get this additional data point and make sure research tells us this is the right answer instead of just trusting their gut. I guess maybe along those lines, this may be going off a little track, but what's your advice there for say product managers or PMs or product teams to not necessarily rely on research for that middle research?

I think the reason why so many PMs ask for those middle range questions is because they haven't really gotten deep with their researcher in a way which can leverage it for maximum impact. if the question is like, hey, Judd, you just pointed out like a bunch of problems. Like, can you be more Solutions oriented? Well, the solution is simple, but not easy to me. It's that we need to restructure the way we make products in a way which integrates research much more fully. Like, it looks like consistent relationships in which researchers and the work and the insights they provide are a part of the process from beginning to end. and like, I think, Lenny, you as a PM, that's how you worked. You know, I remember you, like, I know who you worked with, you worked with great researchers, but honestly, most product processes are not that way. And so that's when you like, research is a service function. It gets called in right at the end. It's reactive in the sense that a researcher in the room listening and participating in the conversation could have a ton of impact on framing exactly the right question that will drive maximum business impact, maximum product improvement at that moment, and then go do to it quick and get back and we're on to the next. But they weren't there. The relationship wasn't there. They're not engaged in the, in the project from the beginning. And, and, and that's the number one root of the problem. As long as research is a service discipline, I think we're going to be.

Stuck in this spot. When people might be hearing this, on the one hand, it's research has been not as helpful to teams as they thought. And researchers have been spending time on the wrong thing. On the other hand, your advice is integrate research from the beginning, make them more involved. Throughout, and I think that might confuse people. What's your, how should people think about, like research is actually more important, you should integrate them more deeply?

There's a vicious cycle that's been happening, right? It is from, from where I sit, and this is what I hear from, you know, many, many researchers and research leaders, which is a lot of companies hired a lot of researchers with great intentions, didn't quite know how to integrate them, you know, and research is a, UX research is a kind of a newer discipline. So maybe that's not surprising. We're still learning how to use it. Cool, let's evolve. But a lot of companies hired these people, but they hired them into kind of like a service discipline. Very reactive, not in the room, not integrated in the way I said. And so they had less input on the questions to ask, or they're included, but only at the end, right? And then they're unable to build those direct relationships to be there in the room to actually drive the questions and insert insights, because a good researcher is like the repository of insights you need. for growth, but they're not there. They don't participate in the decision. So they end up redoing research. They have jobs to do. So they do research that is too reactive. It doesn't matter. And then it doesn't, you know, it's less impactful. Researcher or executives kind of conclude that therefore researchers are not as impactful, and then they get sidelined or laid off and the cycle continues. So I think the short circuit is the constant engagement. If you take a great researcher and you insert them consistently in a product process, I feel confident that researcher will drive product improvement, metrics, impact, growth, all the things that you want to see as a PM and a product leader. It's just, that's the exception, not the norm these days.

This may be a hard question to answer, but when people hear, if you have a great researcher, here's how you approach it. What are signals that your researcher is great versus not great? What are some things people could look for to tell them, like, oh, maybe I have the wrong researcher in my.

So the best researchers, I think, are first of all, multi method. The first iteration of user research was primarily a qualitative discipline, but a strong opinion that I have is that is largely one of those models that needs to evolve. It's not that qualitative user research is no longer important. It's that the best researchers have five tools. I think they have five tools. And those five tools are Number one, what we would call formative or generative user experience research. So looking ahead, innovation focus, really open ended, maybe more ethnographic. Let's go out into the field and talk to hosts and guests on Airbnb. Let's see people using our product in the field, stuff like that. So that's formative. The second type is evaluative, right? So more like usability testing. The third tool is basic rigorous survey design. Right. It's the best scaled way to get responses from communities small and large. You can get a lot out of really well crafted surveys, but to do that you have to have the fourth tool, which is applied statistics. The best research know a little bit of stats. Like you can't interact in a world of A/B testing without knowing basic statistics. And then in the old version of this, the fifth tool was SQL because I think good researchers need to be able to run their own queries. these days, so much of that is dashboarded that the fifth tool may now be prompt engineering, you know, which is a thing we could talk about. But I think it's some, maybe that's the fifth tool is somewhere, is it technical skills that fall in between querying your own data, understanding it very well in companies that are awash with data.

And then interacting with generative AI. Amazing. That's such a cool list. Okay, so just to play back formative, generative, innovative skills to think bigger and come up with new ideas. Usability.

Yep.

Yeah, usability. How did you describe it? I have a different word here. Evaluate.

Evaluative.

Right.

So like we're evaluating products and doing more really, that's kind of the micro level.

Yeah. Survey design being really rigorous about it, applied statistics, and then SQL/dashboard/prompT engineering. Right. Maybe just one last question along this thread. Also big question, but any advice for how to evaluate these skills slash Interview for them. I know this is its own deep topic, but any advice for someone trying to find this person?

I've interviewed hundreds or thousands of researchers, and the way I usually approach that is you want a researcher who's got a Swiss Army knife, because if all you have is a hammer, then everything looks like a nail. And so if you give in the context of an interview, let's say a researcher a pretty juicy, open-ended research question, and you want to see how they handle it. And a good answer is usually multi method, right? We're not going to handle it in any one way. We're going to say, well, here's a couple of ways we could deal with this. Here's how we could do this in a day or a week or a month. I mean, we usually don't have a month, but sometimes big research projects go on for that long. And here are the different sets of methods that we can use. So see where they go. It's actually pretty simple. Most researchers are deeper in one than the other. And sometimes you can make up for those five tools with the team, right? So you have experts who who are kind of T-shaped, but maybe deeper in one or several of those ways. But when I built a team at Meta and at Airbnb, that was my goal is individually as researchers build up those tools and then as a team build deep expertise that would fill all the gaps.

Coming back to the main premise of your post, one of your big takeaways is researchers need to be much more business oriented thinking about what helps the business versus the user, which I think a lot of researchers will feel really weird. Can you just talk about your, your kind of takeaways there?

So much of user experience practice, not just research, but design too, is focused on empathy, right? And very user centered. This is beautiful. I'm not saying that we should abandon that. I think what I'm saying is there's an overlapping then where you have the user and profit or the business and what researchers need to do is be way more explicit about finding that overlap. So one thing I often, when researchers ask for advice, they're like, well, what should I do to be more business or profit focused? I say something like, did you read the last quarterly report? If it's a public company, did you listen to the shareholder call? And they're probably like, no, it's full of a bunch of language I didn't quite get. And I'm like, mm-.

Yeah.

So there you go. That's the language you need to learn. Scour your Google Drive folder, your internal folder, and look for all of the documents that are about this quarter or this halves or next half strategy. What are the OKRs? Understand the metrics and the funnel, the conversion funnel, know it back and forward, because then what you're doing is you're proposing, if you're in the active conversation, you're saying, cool, I hear you asking that research question. I've identified this is exactly the spot in the funnel. where I think we need to do work, right? There's an opportunity here or that competitor is eating our lunch with this group of users. I know that because I read the competitive report and I understand it deeply. So those are skills that some researchers have and a lot are building these days, but historically, last 15 years, it hasn't been a thing we've been as focused on. And I think that's an evolution that needs to happen.

I think a lot of PMs listening to this are gonna be like, hallelujah, this is exactly what I've been trying to convince people of. It's what I've been trying to convince my researchers of. Design often falls into this.

But Lenny, the opposite is true too, because you gotta take the average PM who lives in that land all day, every day. And what they do is not in the Venn. I think those are people who are also performing customer centricity and performing user centeredness a lot. when they're really not interested. And so this is like not about researcher, this takes two sides, fixing this broken system takes everyone, researchers, PMs, designers, everyone at the company, but also the way that organization is structured and integrating itself in a different way. Everybody's got to come to the table.

Such a good point. And you have this actual term that you call user centered performance, where it's the performance of being user centered. Can you talk about that? And then just what advice you'd give to PMs that hearing this or like, yes, I love everything you're saying and then not realizing maybe they're too far in that extreme.

So user-centered performance is a term I made up because it's fun to make up terms. And it refers to customer obsession or user-centered practice that is symbolic rather than focused on learning, right? So it's hugely common, I would argue. It's work we do to signal to each other how customer obsessed we are, not because we want to make a different decision. And if your listeners are like, I don't do that, I'm kind of like, think about it for a second, right? Because this is extremely common. It shows up in explicit ways and implicit ways. So explicitly, I would say every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study just to validate our assumptions, That's user centered performance. It's too late to matter. That PM is not interested in being wrong at all. It's too late in the game for that. We got to ship it, right? What they want is to check the box. So any check the box style research is a wild example of user centered performance. I would argue every researcher has probably had to do executive listening sessions. You know, cause a lot of PMs, founders, product people, but designers too, they want to get close to the customer, right? And so like, can I do some focus groups? I want to be there. I want to ask them questions. This is 97% performance. It's well intentioned, but it isn't focused on learning. It isn't going to drive better outcomes or more impact. And then there's all these implicit ways that people engage in that kind of user performance too. A lot of it comes down to cognitive biases, confirmation bias, ego. Like one of my big kind of mantras was we don't validate, we falsify, right? We are looking to be wrong. That is the mindset you should use when you're approaching insights and research. I want to be wrong. I want you to do research that shows we were off base in the following ways. Tell me exactly how and why in a way that allows me to fix it quickly. But many PMs, many designers are not in that place. They do not want to be wrong. They're looking to validate, and that's user-centered performance.

Oh, man. I think a lot of people are hearing this and feeling exposed.

Exposed.

I feel like you're like this deep-throated person coming from sharing these things people don't want to talk about.

I know.

There's this quote in your post I'm going to read. Product managers love to ask for middle range research that they can use to justify decisions they're reluctant to make on their own. User designers love to ask for middle range research because it fits their model of what proper design process should look like. Executives love to ask for middle range because they don't really understand what research is for. It helps them do performative user centeredness. In the end, they will decide based on their own opinions.

There is an important place for intuition in product development. Of course, like the best designers, researchers, product people, develop strong intuition for the product. But like you got to understand intuition is where all of those biases lie. It's where all your blind spots are. And what great insights people do, what great researchers do when you're next to them all the time is they'll expose you. I don't have to be the deep throat, right? Because you have somebody whose professional job is keeping you honest is probably the wrong way to put it. But, you know, as somebody who's, who, whose capabilities are about expanding your horizons, making it so that your intuition is constantly improving. You don't have to rely on it when your intuition and the evidence sort of collide in a way that either affirms or falsifies the product decision you made. Now something really good is happening. So, you know, and the other thing that is inherent in that quote is, you know, I, at Airbnb, wore many hats over the years. I was head of research two different times. I was head of design for guest products, and my last job was I was head of the design studio, so UX research, UX design, writing, localization, they all reported to me. So I've seen this from many disciplinary angles in the UX field. And researchers aren't the only ones who are guilty of this. Like I would say design has a ton of performance and it comes from the fact that we have like figured out user centered design, this process or design thinking, which, which IDO popularized. that's what we're supposed to do, right? Bezos told us that we, as PMs, had to be customer obsessed, right? So that's what we're supposed to do. It's a really common and damaging thing when we are not genuinely, we don't genuinely have that growth learning mindset. And it's easy to sideline researchers. We don't need them in that situation. We've got our guts. Isn't the gut where a great PM, a great founder, needs to have that gut? And they do. But they need to be open to the fact that your gut is limited and biased and narrow and wrong sometimes.

The two sides of this is trust your gut opinion. I don't need research. I don't need data. I have opinions and my own experience and I'm going to use the product. And let's just go with what feels right to me versus pure data driven, research driven. For designers that are maybe listening for product managers, do you have any advice for just like where to fall on that spectrum and just how to best leverage research to inform that opinion.

Yeah, I taught a class at UC Berkeley this semester on leadership, and we talked about that a lot because, you know, great leaders develop intuition, right? That's part, it's the, it's the, it's the pattern matching part of experience, right? Where you develop heuristics which allow you to make good judgments, even if you can't quite explain where that judgment came from, right? That's what the gut is. But it's also, like I said, where bias comes from, where we're cognitive, like all the cognitive biases, there are a hundred, there's a list of 151 of them on Wikipedia. I won't name them, but like all those thorny things that lead us astray, you know, the behavioral economists and social psychologists study, those live in the gut. And so the advice is, you know, when you are looking to check your gut, you have to do that thing. A lot of, a lot of your listeners have probably read, thinking fast and slow. System one, system two, right?

I have it here right under my laptop actually holding out my laptop screen.

That's so appropriate, Lenny. Um, so the secret is not that sexy, it's system two, right? So you engage that slow methodical process in which you do analytic thinking as a means of checking your gut. Slow in the grand scheme of things, right? Slow meaning not a split second decision, not like months of analysis. That's not what I mean. the other thing you can do, and there's really great research on this, is you bring in the wisdom of the crowd.

Right?

So the wisdom of the crowd is a phrase a lot of people are familiar with, and it works in a specific situation. The wisdom of the crowd works when the people involved with the decision are bringing diverse sources of information and judgment to the table. Obviously, if everybody has the same sources of information, then it doesn't matter how many people are out there. So if you want to check your gut, get a bunch of different guts together, get a bunch of different people in the room who can bring evidence and intuition to bear and have an open, like direct and kind conversation in which we might disagree. You know who's great at that? Researchers.

Leading those discussions essentially and getting a bunch of people's opinions.

Yeah, this is the structural solution I'm talking about, Lenny, is like, I never asked for research teams to have their own separate OKRs. I said two things. Number one, what's the teams? Okay, like shouldn't the PMs, the engineers, the designers and the research, everybody should have the same set of metrics for success, right? Because either we're doing it together or we're not. And then I said my metric for success is when they won't have that meeting without you. That's my metric for success. If they cannot have that decision making meeting without the researcher there, That means you've developed influence, strong trusting relationships, you're an active participant in the process, not just sort of somebody who provides input into someone else's process. And that is when researchers can have huge impact. I think of the PM role in.

A similar way, even though people won't have these meetings without PMs, because they're often at the center of a lot of the stuff, but you want to be a PM that people want on their team. There's a lot of teams that are like, we don't want any PMs, we don't need project managers. They just get in the way. And I find that that's only the case when the product manager is not great and not really good at their job, because most great PMs just make everyone's life easier.

They do. The grease, I love it.

You mentioned also that before we start recording that the biggest challenge for user researchers is their relationship with their product manager. Can you speak to that and what you've seen there?

I'm wary of overgeneralizing, but I can tell you that from my experience and from what I hear, the product research or product insights relationship is one of the most challenged. And I think it comes from the fact that fundamentally many researchers are just not included in the process that, that PMs are running. And then there, actually, I kind of thought I, I did some asking around before, before this podcast, and so I, I thought, There are some tropes that researchers have about PMs that are worth PMs knowing, right? Just like four or five of them, the things that researchers know PMs say, which drive us nuts because they're not true. So the first one is that research just slows us down. Research is too slow. This is bullshit. A great research team can do research in a day, a week, or a month. you know, it just depends on what you want to get out of it. You know, like if you, you know, how much, how, how much detail do you need? How many people do we need to talk to? What is the depth or breadth? Do we need to go to seven different countries to talk about our constituencies in Latin America? Well, that's not going to happen overnight, but we don't often need that. The other way to look at that is that, is it slower to get it wrong and fix it than to take a hot second to do the work to get it right the first time? you know, so that's BS like research, good research doesn't slow us down. It speeds us up.

And also just along those lines, like a big part of your premise is you don't need to do as much research as people are doing, like this middle research that a lot of the time is put into.

Yeah, research can go super fast. I think especially so the macro level research, I hope what it is is tied to things like annual planning processes. We did a thing at Airbnb several years that we called the It was like insights 2019, insights 2020. They were concept car projects. And we spent quite a long time synthesizing the entire year's worth of insights from every place we could get them and then developing with designers and engineers, like a concept car for five years in the future. So that's a long process. But the micro level, there's so much business value to be derived there, so much business value. And it can go so fast, Lenny. Right? It can go so fast. You can have, you know, you can have results in 48 hours on these things. Like we did a thing at Airbnb. There's a famous story, which I'll only tell in the abstract because I don't want to out anything, but it's like the, we call it the multimillion dollar button. And basically we did research which revealed that people weren't going down the purchase funnel because they were afraid The calls to actions on the button was making them afraid that it would initiate a purchase when really it was just taking the next step. Right? We changed the text on the button with help from our amazing content design or UX writing team. We basically changed seven characters and made Airbnb millions of dollars.

Right?

Because what we found out was really simple. It was just like, Hey, this button feels scary. the CTA on the button feels scary. So that's a great example of how micro, and that happened in like 48 hours. We would discover that inside or 20 overnight, basically. And we were like, Hmm, maybe we should test some other CTAs. We did the conversion, like we added like 1%, which is really, really hard to do, you know? So that's a quick example of how that type of quick research can drive a huge amount of business value.

This episode is brought to you by Ahrefs. Many of you already know Ahrefs as one of the top tools for search engine optimization. It's used by thousands of SEOs and companies like IBM, Adidas, and eBay. What you may not know is that there's a free version that was made with small website owners in mind. It's called Ahrefs Webmaster Tools. It's free and it can help you bring more traffic to your website. Ahrefs Webmaster Tools will show you keywords that you rank for and backlinks that you can get. It also automated site audits to find what issues prevent your website from ranking higher on Google. Every detected issue comes with a detailed explanation and advice on how to fix it. Visit ahrefs.com/awt, set up a free account, connect your website, and start improving it. That's ahrefs.com/awt. So just to make this even clearer, I think this middle research zone is the stuff that does slow people down, I imagine. It's like, what are the challenges hosts have with payments on Airbnb? What you're basically saying is spend your time doing the micro stuff, like usability research, and then the bigger stuff that's part of overall planning. That's part of the planning cycle. It's not like every project you're working on, you need to have a whole research project on.

Exactly. The micro research should be much more common. A lot of researchers think that that's sort of scut work. you know, that like usability is something junior researchers do, I completely disagree. Like, I think we need to get back there as an industry and be like, when you make a product easier to use, when you like discover problems with functionality, business metrics we care about will go up. I've seen it happen, right? But it's, that's not just work for interns and new grads, that's for sure. And then the planning process, like absolutely, if we're integrated from beginning to end, We can help. And the thing about that middle range, I think you're right. That's the stuff that makes the sort of stereotype that researchers is slow. And a lot of times it's also because it's just not pointed enough. The researcher can also say in that moment, I have studied the business plan. I know exactly where, like I've seen the metrics trend. I have an idea about exactly where that's going to go. We still need to do that middle range research. The question is valuable, but it's now very pointed. and that the time is worth it.

Amazing. Okay, I want to hear the rest of these tropes.

Okay, so research is too slow is the first one. The second one, I can do my own research. Why do I need researchers? And like, that's true as product people. I hope, you know, you are engaging with customers and listening well, but no offense, garbage in, garbage out. You know, like the thing is anyone can talk to a user. that does not constitute research or insights work, you know, because one user can be powerful, but one user can be idiosyncratic. And a researcher knows how to get to the heart of that really quick. They know how to take that conversation and understand and situate it in a way which means like, sure, like democratize research. That's happening. There are tools out there that will let anybody get customer feedback, voice of customer type stuff. But a researcher is there to help you turn garbage into, you know, into something that's not garbage. So, and avoid the bias that can come from like you just reaching out to your cousin's family and then doing whatever they thought you should do to the product. So that's the second trope. The third one is AB test everything, right? And like, like AB tests are great, but one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that like flipped a stat sig. And then they're like, cool, I was significantly down over this course of time for these users. And then they just start speculating about why that is because the A/B test rarely tells you why it changed in the way it did. And then this endless flywheel of A/B testing goes. And I'm like, Hey, you don't have to guess. I know somebody who can get you an answer to the, or at least evidence that addresses the question of why did we see the test result we did in a very short amount of time? Or you could use your customers as guinea pigs and throw more experiments at them over and over and spend a long time on it and come to the same place in the end.

I think a similar critique that PMs often have is A/B testing is conclusive. Scientifically, statistically, user research is just talking to a bunch of people. Why would I trust that? What is your best way to help PMs realize that this is actually very valuable data and you should listen to it? It's not just this, you know, a story here and there.

Yeah, no, I think they're both right. A/B testing is as close as we can get to causal, making causal claims about products, right? Research is usually not oriented towards making causal claims or it should not be. But those causal claims rarely tell you how and why things happen. And if you want to develop, if you want to not make that mistake again in the future, you need to know how and why. If you want to build a better product in a way that doesn't just answer this narrow question that an A/B test answered, you need to know how and why. And so you kind of need both, like beautiful partnerships between data scientists and research and insights people, like are, are, I think what we're going to see in that next evolution. And if you set that up, that, that sort of virtuous cycle up, If you set up the engagement where those people are involved from the beginning, you don't make those mistakes. You get the causal relationship, which is valuable for one reason, and the hows.

And whys, which are valuable for other reasons. Awesome. Okay.

I think there's two more tropes you had. So one of them is a simple one, which is everyone loves to quote that it turns out a totally apocryphal Henry Ford quote about if I'd asked my users, it turns out the best of our knowledge, he did not say that.

And really? Yeah, I know.

Isn't that sad?

You know that?

I know. Sorry to burst your bubble, Lenny.

Oh, wow. Does anyone say anything?

I feel like every quote is- Is it boggable now?

I know.

What is reality? Jeez, can we- well, let's just- okay, maybe he said that. He certainly believed that. That's what the historians say. But the reason that makes researchers so angry is because that's not research. That's not what researchers do. A researcher who's going to ask customers what they want is a bad researcher. You need a different researcher. Like that's not, I've never done that in my career. No one on my team has ever run a study that's like that. You know, so, so that, that just makes researchers mad. And then the last one is about post hoc bias. It's, we knew this already. That was obvious. And, I think a lot about this book, which I would recommend. to your listeners. So the author is a sociologist at UPenn named Duncan Watts, and the title is Everything is Obvious if youf Already Know the Answer. And it's about hindsight bias. He kind of makes the argument that we rely too much on intuition, heuristics, and pattern matching in a way that sort of is inappropriate to our experience. And it leads us astray. It's kind of a form of self gaslighting. and it happens because we end up sort of selectively remembering things and then constructing narratives around them in a way which makes us feel like we already knew that when we infected not. And he talks about this other, one of those cognitive biases called the narrative fallacy, which is the idea that people love to make convenient, simple stories about the past. If I asked you about your career, Lenny, and how you got to be this amazing podcast host, you'd be like, well, Let me tell you about this series of events. And we do that. It's part of how we make sense of our lives and the information around us, but it would probably be a lie in the sense that we all twist the evidence we have to fit the narrative we want to be true because it's simple and lovely and makes us happy. This is going to sound self-serving, but.

I find I'm the opposite. I'm like, I have no idea how this all came about. Here's some things that kind of happened and somehow I ended up here. But maybe I'm being very modest and try to not Give myself any credit.

That's beautiful.

Thank you for these tropes, by the way. This was fun. I didn't know you were going to do that. So that's a fun little collection we've got here. I wanted to ask about, there's this tweet by Patrick Collison that I've brought up a couple times on this podcast that I think is really interesting. And his tweet is this. In my opinion, the best product will stem from a very strong mental model of the domain and user. User research can help you get such a model and validate it along the way. But it's important to view the syllogism of UXRs model as of user research to improving your mental model of the user to what product you should build versus user research tells you what product to build. Does that resonate in any way? Thoughts on that way of thinking about user research?

Yeah, I mean, there's a double-edged sword we talk about a lot in in the research community, which is about making recommendations, recommendations for design, right? So the best research doesn't like leave it at that, right? It tells you, and it's like the what, the so what, and then the then what. But the problem with that is some researchers go too far in the other direction where they're like, we ran the study, it yielded these insights, and therefore this is what we should build. and everyone else on the team is like, Whoa, whoa, whoa. Right. Glad to hear your thoughts on the matter, but like, there's a lot going on here. Maybe we should talk about it. Like, so, and that makes perfect sense. That's like a failure of communication. And I think that it like speaks to the thing that Patrick is saying is like, good research can sometimes tell us exactly what the problem is and exactly how to fix it. An example of that is the multimillion dollar button I told you about. But in a lot of the bigger picture questions, especially the macro ones and maybe also the really pointed middle range ones, the point isn't really, this is exactly what we should do and this is exactly what we should build. It is let us develop a framework which is based on actual evidence, and then together as a team figure out how we want to experiment our way to a successful product.

To close the loop on this specific thread, what is your advice to teams, researchers to help move out of this kind of reckoning and to move forward and help the field both from a user researcher perspective and also from just like a company that maybe laid off a bunch of user researchers or is trying to decide what to do with the researchers.

Thank you for asking. I think I said to you earlier, and I feel some pressure as maybe the first conversation that you've had specifically about research on this podcast.

Yeah, I think so.

I want to help. I believe so much in this discipline of research and insights. And I think there's that, you know, when I said the user, the UX research discipline of the last 15 years is dying, I didn't mean that I think research is dying. Far from it. I think that there's a version of it which we're now moving past and into a new version. We're going through an evolution, as many do. And so the question for me is like, how can researchers and the companies and the other people with with whom they work, create a new version, a different version and evolution, which is hugely impactful for the business. And so the advice I give to researchers about that is develop diverse research skills, remembering like the five or five and a half tool list that I mentioned earlier, really go deep on that business knowledge. So speaking the language of product and business and metrics and understanding exactly how to use your insights like a scalpel, Building those strong relationships, which is not a thing that researchers can do by themselves. It requires two-way engagements and also in a way which allows researchers to do fewer things better. So most researchers that I know are working on teams where they're like, I'm the only researcher and there's like, I have seven PMs and 20 designers and, and like, I'm trying to do 10 projects and no one's going to do a good job that way. So researchers have to learn with their partners about how to say no and focus on the most important But that's like only half of it, right? That's the research side. I have like two thoughts about, about what companies should be doing. The first one is, it's a little bit of an aside, but not really. Like one thing I learned by, through the responses to, to the article was everybody came out of the woodworks from the variety of insights disciplines that are out there. Cause I come from a tradition of user experience research or user research. But there are many insights disciplines in many industries. And they all wanted to claim one type of research or another and say like, oh, well, we over here in consumer insights or market research have been doing that well for years. And, you know, there are many insights disciplines and generally I think creating silos is stupid. Actually, I'm curious what you think, because here's the number one thing I heard when I joined Airbnb and you were there. is I did a quick listening tour where I talked to a bunch of product people and they all kind of said the same thing. They were like, listen, we have all these different people throwing insights over the transom. And it's great. We want to hear from the data scientists, from the product specialists, from the customer service people, and the voice of the customer, and reset, whatever, all that stuff. But they're all coming over the side and we don't know what to make of it. It's like too much. And that as much as anything is an argument For companies to stop siloing research disciplines. So when I joined Airbnb, I like set out to create an integrated insights function where it's like, let's do UX research. Let's talk about the market and competitors when we have to. Let's integrate smartly with data science functions. Let's integrate all the stuff we're getting from customer service feedback. You know, we, we brought over what was then the NPS program and, and sort of said like, Hey, like if we're getting customer feedback there, let's all just like, use it all to fuel this one insights machine. So that's the first piece of advice I give companies. And the second one, without being a broken record, is to think differently about the broken cycle. So integrate researchers into a unified lean process. So if the researcher is not there from beginning to end, if there are not strong relationships between product people and design people at every level, engineering people at every level, and somebody who's there partner, their insights partner. We're going to fall back into this problem where we're just a service discipline. We're not extracting the maximum value. It comes too late. We don't know what questions to ask. We're ignorant about what research can do. And so creating that integrated lean process where a researcher is arm in arm from the beginning is the most important advice I'd give.

That last piece may be the answer to this next question, but the question is how can product managers be better partners to user researchers slash get more leverage out of user researchers?

I think that is in many ways the answer, making sure that they are creating a process for the product, for their products that integrates user researchers and insights from beginning to end. Also, being willing to partner with the research on the ruthless prioritization. I used to say that a full plate for a researcher was probably three things. to big projects and a small project, right? Like a side project. More than that, your researcher is probably not doing a very good job. And a project may take 48 hours. That's okay. But so they need your help to prioritize. They need you to participate. Great PMs will take the time to be with researchers, to go into the field, even to travel. Did you ever do that, Lenny?

I did. I went with Luis, who Introduced to who like came up with this basically told me to chat with you about this topic.

Thanks Louise.

Thanks Louise. We did a whole tour to Paris, our whole team, the leads of our team, went to Paris to do a bunch of focus groups and a bunch of research behind actual mirrors. I've never done that before that trip and it was amazing. We learned a ton.

Can I tell you a quick story about behind the mirror? So this is back from when I was at Facebook and it was the high times there was like 12, 2012-13, and News Feed is really taking off, ads are going into News Feed. And I was working, I was the leader of a team that was working among other things on how to address post quality. How do we think about what's a good post and how do we get feedback about it? And so there was kind of a team of engineers that thought one thing that you can do on Facebook is hide a post, right? So they were like, this is easy. Let's look at the posts that are hidden the most and use that as the signal of what's a good post on Facebook. Seems kind of reasonable, right? And like something tripped me on this one. And so I was like, so I did two things. So the first thing I did is I looked at the distribution of hiding by user and found out that it's power law distributed, like everything on the internet. There are A few people on Facebook who hide a ton, and then most people don't hide at all. And so then what we did was we said, let's find, we call these super hiders. We call them super hiders. And so we said, let's find super hiders around the office and we'll get a super hider in, and we'll do like a really traditional user interview. I just want, we just want to see. So literally the first person who walked in, I remember because this is a person who had those fingernails that are so long, you don't know how they can do cup touch screens. But they did, they were amazing at it. And it was one of those rooms with the glass. And I insisted that the ENG directors, the product people, and they were willing, whatever. So everybody's behind the glass and I'm there with them. And the excellent researcher is in the room and they come in and we're just doing a traditional think aloud study. And so they, hey, can you open up your Facebook app? We would just love to see you, you know, what your experience is like. And so they open up Facebook and we're looking and they look at the first story. and they hide it and they go to the second story and they hide it. And this went on for a while. And like this, she's definitely using Facebook, but every time she'd read, finish with a story, she'd hide it. And the people in the back room were starting to chatter and they're like, wait, what? What is happening right now? And like good research that the good researcher that this person was, like they let it continue and they're like, oh, can you tell me what you're thinking right now? Come to find out, She was like, well, I hid that story because I'd seen it already. The model she was going for was inbox zero, which was like sad because it was infinitely scrolling. She would never get there. And the reason I like that story is because the people in the back room had their minds blown. It was not that we assumed that was common behavior. This person could have been unique, but it was enough because those people were there experiencing the research that N of one allowed them to burst their own bubble and realize, okay, we can't think so naively about hides as a signal anymore. And we came up with a better solution.

That is an awesome story and such a good example of you don't need statistical significance to get massive insights. One, example just gives you a, wow, this might be exactly what's happening. Let's go validate that versus like, we are confident 100% this is what happened. I love that. It reminds me actually in the mirror, in the mirror study that I was talking about in Paris, there's a Facebook element to it too. We were trying to convince hosts how to feel more comfortable accepting guests who are booking instantly. And one of our theories was if they were connected on Facebook, they would be more comfortable letting someone book instantly.

Yeah.

And we're just like, Hey, why don't, what if you were to connect Facebook and see if they're friends? And everybody in Paris was very afraid of connecting and giving Facebook any data way ahead of like what the US hosts were feeling. Yeah. So it just made us, made it very clear nobody wants to actually give Facebook any data.

So it was very anti Facebook at that point. Yeah, that's so interesting. Germany, Germany and, and, and France were always our like bellwethers for, for what the rest of the world would be.

Thinking with private data privacy concerns. Oh man, okay. A couple more things. So a lot of this started with a lot of layoffs within user research. And I think between the lines, there's a sense of teams don't need as many researchers as they hired during the Zerp era. I think a question in everyone's mind is just like, how many researchers do we need? What is a good ratio? I imagine there's not a simple answer here, but just what's your general advice to companies of how many researchers is right?

So this is the thing I, I've thought a lot about, especially in my role as the head of the design studio. That was like my fundamental question. It's like, you have all these writers, designers, researchers, like, how do you structure them? How many and where and who works on what? And the organizing principle for me was always relationships. You know, you have enough when the people who need to have a constant research partner have them. And I would much rather create pain in that situation than spread someone too thinly. So my advice was always like, don't try to create a researcher to cover this entire product space. pair a researcher up with, you know, somebody who's gonna involve them in a consistent engaged process and let them go to work and see the impact they're gonna have, but protect their time. And then other people are like, wait a second, that person's doing great work. I want some of that. And creating that pain for them, like, because it's, it's a pain of loss, like is the number one way to grow headcount. That's how I always approached getting more headcount was not arguing abstractly for why research is important. but by asking partners who wish they had it to do the arguing for me. And so there isn't a, you're right, there isn't a clean answer for like, hey, this is how, this is the right ratio because it really depends on the nature of the product. Like, is it a, is it an early stage product? Is it a late stage project? Are we talking about a startup or a late stage company? But, you know, I would argue there's always room for a researcher. Lenny, I'll tell you, and I use this in a keynote talk I gave lately. You published recently a list of, I think it was about 20 B2B companies and their first 10 employees. Do you remember doing that?

Absolutely.

Do you remember how many researchers are in anywhere on that list? I'll give you a hint. It's between zero and two. It's one. There's one researcher on that list anywhere. Anywhere. And that's messed up to me. Now look, It's just these 20 companies and like they're, you know, each is in their own space. So I'm not going to overgeneralize, but there is an in a researcher can drive incredible value no matter what stage a company is at because a good researcher makes you go faster, not slower, and they drive impact because they answer questions which are impossible to answer in any other way. That's true if you're a startup. It's true if you're a late stage company. Now, if it's your first 10 employees, you know, like one, one researcher is going to go a long way. as you grow, making sure that you're matching up researchers so that they have strong partners in the key parts of the business is the best way to figure out if you have enough.

Interesting. So your advice is as you're starting a company, your pitch is that you will have a lot more leverage and move faster hiring a researcher versus generally an engineer is what you'd be trading off. Essentially, that's what most of the hires end up being.

I am reluctant to overgeneralize, but I would say and I would say I know in many, many founders who are in startup mode are like, I know what I need to build. The problem is that I need people who can help me execute. And I think that's right. And so everything's a trade off, but remember, imagine that you could have that Swiss Army knife at your disposal. As you, maybe you've got like an MVP out the door and you're looking to make your first major iteration, or like many startups you need to pivot. this is where it's like, hey, you don't have to do that alone. We deify startup founders who pivot appropriately, but I think that is what we might call moral luck, where we deify the ones who got it right, and even though they made exactly the same decisions as the one who got them wrong. And the fact of the matter is, if you have an insights person with you who has that Swiss Army knife of tools, you're not in it alone. You don't have to guess. Ultimately, it will still come down to a tough decision that you and founders have to make. But you can have evidence that bears on that decision, which you wouldn't be able to get any other way.

To close out on this, and I have just a couple more questions on this thread, I think one of your big messages to researchers is you can be empowered. It's up to you to do the right sort of research and to move your career in the right direction, not become a researcher people need. And there's this quote that you have, at the top of your post where a lot of the reaction, or I guess the way you put it is, I know what you're thinking. They just don't get it. We're so misunderstood. Our plight is to deliver insights that users used to drive business value while we're forgotten. Never driving the roadmap, no seat at the table, consistently miscast, only to be laid off in the end. And what I'm hearing from you is you can change that. You can push back on doing research that isn't actually contributing. But let me ask you, what's your kind of leave lasting, I don't know, advice you would leave researchers with to be successful.

Yeah, it's tough to be operating in a broken system. And so I feel that response where you feel kind of powerless. But I think that's not likely to lead us past this moment to the next evolution of research. So that's kind of where it's like, I don't blame any researcher at all for being in the spot they're in. It's been a tough go, right? However, crying about our lot, you know, is like not going to get us anywhere. So like, I think the point of the, the, the article for me, and, and this is advice I give companies all the time when I do consulting with them, it's like, Hey, we can set this up in a different way, which responds to the current environment in a way which will drive a huge amount of impact. Now that takes companies making the right choices. It also takes researchers owning up and developing skills, right? pushing back, understanding what research can have the most value, developing the skills and the knowledge and language around the business, like becoming more influential, being excellent communicators. Like it's one of the things I would evaluate the most in hiring, especially research leaders, because I needed them to show and teach by example is like, can you, if, like, it isn't just rigorous research. It's like a tree fell in the forest and no one was there to hear it. you know, you need to communicate it effectively and you need to do it in a way that's appropriate to the audience. Cause if I'm, if I'm talking to you, Lenny, it's different than I'm talking to Brian Chesky, you know, at C at Airbnb. And so I gotta be able to give that presentation effectively and get right to the heart of it and speak the right language. And so if you're a researcher, it's not hopeless. Actually, the, the discipline, the future is so bright and we can help it along by continuing to develop these different.

Skills as companies build a, a model that's more inclusive. Awesome. Okay, I have one just random tangential question about NPS. You have strong opinions about NPS, and I just wanted to hear your perspective on the value of NPS, your experience with NPS.

Yeah, I do have a strong opinion about NPS. I like to say NPS is the best example of the marketing industry marketing itself. And I think I'm going to, the problem is this threatens many people's livelihoods because there's an entire industry of consultants and software providers that want you to believe NPS is a useful and accurate metric. The problem is, so the consensus in the survey science community is that NPS makes all the mistakes. So it's a garbage in, garbage out problem. So the likelihood to recommend question is bad for a whole variety of reasons. So it's bad because it's a 0 to 11 scale. It's bad because it's usually unlabeled. So we label the polls, but that's not the gold standard for research. It's bad because it's 11 items, right? And there's a couple of problems with that. Number one, we find that precision goes down after five items on average, maybe seven. Number two, especially on mobile, if you're taking this survey, what percentage of those options are below the fold? Right? We are not going to get accurate survey data, right? And so from a survey perspective, it's really bad. There's also this intuition, which is like, are you a person who will let record, you know, like, Are you, are you, how likely are you to recommend Windows 11 to your friends and family? I am not a person who goes around recommending operating systems. The question is fundamentally flawed. The argument is that's the, that that question is a good indicator of loyalty, but there's a really simple solution. Lenny, customer satisfaction, a simple CSAT metric is better. It has better data properties. It is more precise. It is more correlated to business outcomes. I wanted to prove this. This is something that survey scientists know and marketers don't want you to know. And so I did, we did the work with Mike Murakami, who led survey science at Airbnb and he's still there. Great researcher. And we basically redid all that work to find out if all that stuff was true just for Airbnb. And it is. It's simple.

Don't ask NPS, ask customer satisfaction. And the customer satisfaction question, what's the actual question for people?

Overall, how satisfied are you with your experience with Airbnb? Or it could be some version of that, which is like, Overall, how satisfied are you with your experience with customer service when you had a problem? You know, so there could be a more specific version of that question, but those questions have better properties. And, you know, a lot of people say, well, hey, everybody's using NPS, right? So at least it gives me a benchmark because I can compare my NPS to industry NPS. The problem with that is the research shows that NPS is idiosyncratic. So it goes up and downs in ways that we don't understand. And there's a lot of inconsistency in how it's asked that creates variations in the data, which means it's not apples to apples. So you can't even compare your NPS meaningfully to somebody else's.

I love these hot takes. I'm curious to see who comes out.

Of the woodwork too.

People are gonna be so mad, Lenny. I love that. I think, yeah, people, like, I've heard this many times and people don't talk about it. Okay. Is there anything else you wanna share or leave people with before we get.

To our very exciting lightning round? Can I, yeah, I wanna add one thing, if I could, because this has come up on your podcast. a few times recently, which is about the idea of people doing their own product walkthroughs. Like, so should a PM just rely on their own dogfooding of the product and their own walkthrough to figure out how to fix it? And a couple of times recently this has come up like this, and I think the consensus seems to be sort of yes, this is a good thing. And I have a contrarian opinion there too. which is that I think it is really important for everyone to dog food their own products. The problem is related to relying on your intuition about those products, which is the thing most PMs have trouble with is realizing you are nothing like the user. You are nothing like them in ways that will bias the way you think about what's good and bad in your product in ways that you can't necessarily recognize. Like some things with a product, some problems with a product, you need a pulse. to recognize. And like most good PMs that I know have a pulse and so cool, but a lot of them require like context of use, priorities, constraints that you just don't have and you can't imagine purely on the basis of your own usage. So what I think that means is that you should definitely dog food your own product. Doing product walkthroughs to identify lists of potential issues is a great thing to do. Prioritizing that list, figuring out which ones are more or less a problem and for whom is an area where you should be extremely wary of relying on your own sort of opinion, expertise, or intuition when you are dog fooding your own product.

Thank you for sharing that. It's definitely come up a bunch on this podcast, so I think that's an important lesson for people to take away. Anything else before we get to our very exciting lightning round?

I appreciate you, Lenny. Thanks for having me on.

I appreciate you, Jed. Well, with that, we've reached our very exciting lightning round. Are you ready?

I am ready.

What are two or three books that you've recommended most to other people?

I recently read a book by a business book by Barbara Kellerman called Bad Leadership. And what I love about it is that we spend a lot of time talking about good leaders, and she really dives into like the worst leaders and what makes them bad leaders in a way that I think is really valuable for everybody. I'd also recommend, I read a lot of fiction, so two recommendations there. One, recent Pulitzer Prize winner, Demon Copperhead by a Barbara Kingsolver. It's like an outstanding read that also is like really sad and moving and illustrative, especially when you're, if you want to understand rural poverty. And then completely other side of the fiction spectrum, if you, if you're interested in science fiction, which I am, read the Murderbot Diaries. It's like about a sarcastic robot, a sarcastic killer robot in who doesn't love them.

I love these fiction recommendations. I feel like we need more of.

These on the podcast.

Yeah, everybody goes to business books. Yeah, absolutely. What is a favorite recent movie or.

TV show that you really enjoyed? We recently watched the Last of Us and it blew our mind. I watched it after I played the video game after long last. If you are a person who plays video games and you haven't played the Last of Us, play it. If you don't know, the show is based on the video game, not the other way around.

Do you have a favorite interview question you'd like to ask candidates that you're interviewing?

Think of a topic that you had to explain lately that was the most complex, and then explain it to me like I'm five. And there are a lot of ways to vary that question, but the reason I like it is because I think, and I've asked this question to, you know, VP and C-suite candidates in like multiple disciplines, and sometimes it's related to a conversation. Like I might ask them to explain something complicated about Quantum Computing or music theory, or it could be a complex business decision. But I want to see if somebody can break a complex problem down in a really simple way and give me an intuition for it in a short amount of time. I think that is a differentiator between good and great for many people.

Do you have a favorite product you recently discovered that you really like?

Yeah, this is a really weird one, but my whole family started indoor rock climbing recently. And there's a challenge you have when you top rope, which is that you're looking up all the time. So they make these glasses, which are called belay glasses, and they have an angled mirror embedded in the lens so that you can look straight ahead and the view you see is up towards the person who you're belaying. And I just thought that product is so perfect for that. That's a niche problem and there isn't a better way to solve it.

Do you have a favorite motto that you often come back to that you share with friends, either in work or in life, that you find useful? Yeah.

So this is gonna seem like pandering, Lenny, but I don't know if you remember a conversation that you and I had. It must have been eight years ago. I remember where we were sitting and it was about stoicism. Do you remember this? Anyway, we had this conversation.

I don't, but I was into stoicism for a while.

I know you were because we talked about it. And so the motto comes from stoicism. which is basically focus on the things you can control and ignore the rest. And, you know, a lot of people think of this as, you know, a lot of people think of this as the. The serenity prayer or the serenity saying. That is. That was a 20th century invention. But Epictetus was writing about this, you know, BC. And I think about it all the time. So much of the stress and pain and worry that we have in life comes from things we can't control. So I try to let those things go.

Amazing. I learned that lesson from seven habits of highly effective people and just the importance of thinking about these circles of you can control, you can influence, and you have no control over. There's no reason to think about those other things. Absolutely.

Judd, this is everything I hoped it would be.

We got into some really good stuff. I'm excited to hear how people react. Two final questions. Where can folks find you if they want to learn about what you're up to? Actually share what you're up to these days and how people can find you. And then also, how can listeners be useful to you?

Yeah, thanks for asking those questions. So people can find me at judanton.com that's the best way to find out what I'm up to. These days I'm a consultant. I help people with UX strategy, org design, and crisis management. Somehow I love dealing with other people's dumpster fires and I've found that I'm constitutionally good at it somehow. So judanton.com is the place to find that. I also write, I write a Medium post that you can find at onebigthought.com and you'll find a lot of the topics we talked about today, including the original post that started this at onebigthought.com. if there's one thing I could ask your listeners to do is to get next to your researcher. You know, like, I just, I just think if you build those relationships and involve a researcher and insights person early and often, beautiful things will happen for you and for the business. So that's my, that's the thing everyone can do for me.

I love that. I've always done that. I loved my researchers that I've worked with, many of them reporting to you. And so beautiful takeaway. Judd, thank you so much for being here.

Lenny, thank you. It's been a pleasure.

Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lenniespodcast.com See you in the next episode.