
A short YouTube talk by Mike Lyons (uploaded May 29, 2025) using Netflix's 2011 attempted split (into streaming and a DVD service called Qwikster) as a case study to explain three kinds of product feedback: opinion-based, experience-based, and market evidence. Lyons contrasts the strengths and weaknesses of each type, cites concrete harms when market and experience signals were ignored (Netflix lost 800,000 subscribers and its stock plunged 77% within 23 days), and advises combining all three to make better product decisions. He also references a supplemental tool called 'Master the art of the critique'.
Questions this video answers
May 29, 2025
2011
800,000 subscribers
77%
23 days
Chapters
You might remember in the early days of Netflix streaming that you could actually still get DVDs by mail as well as stream movies online, all under your one account. Then in 2011, their CEO Reed Hastings made a bizarre decision. He split the company in two. Netflix, well, they would only do streaming, and a new company called Qwikster, which Netflix created, would handle all the DVD rentals. Suddenly, customers who used both services needed two different accounts. They had two different passwords. They even had to pay two separate bills. But even worse, they couldn't see their viewing history or ratings across the two services. That makes zero sense. Now, there were warning signs kind of everywhere. Customers began to post all of her social media saying things like, this feels like a terrible idea or why fix something that's not broken? Their customers were sharing opinion based feedback and Netflix largely ignored it. Then their longtime DVD customers spoke up. They were saying things like, I've spent years rating movies and getting recommendations, now I lose all that. And they were sharing real experience with the product. This sort of experience based feedback, well, Netflix largely ignored that as well. But the most sort of obvious evidence was in their own data. Their own numbers showed that most customers used both services. And their viewing history proved that people regularly switched between DVDs and streaming to watch what they wanted. Well, that was market evidence. And yes, they ignored that too. Within just 23 days. So less than a month, Netflix lost 800,000 subscribers and their stock price crashed 77%. They had to publicly apologize and kill off Quickster before it even launched. Getting real feedback from customers isn't just important. It sometimes can be the difference between success and total failure. But you know, not all feedback is created equal. I want to talk to you about three different types. And knowing the difference between these three is crucial. First, there's opinion based feedback. You hear it all the time. These are opinions like, users will love this new design or another opinion. Nobody would ever use that feature. Or my editor saying, my shirt is too pink. It's a little pink. It's too pink. Opinions are everywhere. And opinion based feedback, well, it's easy to get, but it can be dangerous to solely rely on it. And you know what, it's especially tricky when those opinions when they come from powerful people in your organization. Think about like a CEO that says, I know users will want this, and then suddenly everybody's scrambling to build it as a po. When you hear something that sounds like opinion based feedback, you want to be polite but but firm. You might say something like, hey, thanks for that input. Let's do this. Let's test this out with some of our customers and see what they think. That sort of sets the stage for what's important to us and what we're going to measure. Okay. Then we have experience based feedback. Experience based feedback. It helps you avoid repeating old mistakes. This type of feedback is tricky though, because timing matters. Something that was true two years ago might not be true today, heck, even six months ago. With how fast things are changing and context, well, that matters as well, because what works for enterprise customers, it might fail completely. With your B2C market, I'm sure you've heard someone say, we tried that before and it failed. But as you go deeper, you might realize before was five years ago with different technology and different customers and even different needs. But today, the idea could potentially work well. So that's a bit of a tricky one. Okay. And finally, there's market evidence. This is the good stuff. It's like cold, hard facts. Not just someone saying, I think users hate this feature, but actual data showing 60% of users drop off at step three, for example. It's not someone saying people are complaining more. No, it's clear numbers showing that support tickets shot up 40% last month. And it moves us away from language like users love this page to data proving that they spend three times longer there than anywhere else. Market evidence is powerful because it shows us what people actually do, not what they say they'll do. I was in a meeting once where everyone was convinced users wanted this new feature. But when we looked at the numbers, looked at the data, nobody was even using the old version of that feature. Now, I know that's not enough to make a decision on its own, but we also considered that nobody was asking for it in our support tickets, and we couldn't find that anyone was working around it. Basically, we were solving a problem that didn't exist. The best decisions, well, they use all three types of feedback. Opinion based feedback, it helps you explore new ideas. Experience based feedback, well, that helps you avoid past mistakes. And market evidence, well, that helps you verify what's really happening. I've got another tool for you called Master the art of the critique that breaks this down perfectly and I've attached it below. It shows you how to spot these different types of feedback and use them to build better products. Just remember Netflix, they had all three types of feedback telling them to stop, but they didn't listen. So don't make the same mistake. Pay attention to what your customers are saying. That's opinion. What they've encountered, their experience and of course, what they actually do. The market evidence Understanding these three types of feedback is important to the evolved step in our value proposition design. You can't adapt and improve your product unless you know what's working and what isn't. And you can't know that unless you understand what feedback to trust and how to use it.










