Mike Lyons describes a medical-records company that spent 4,000 hours building features nobody used. He outlines five ways to catch this: watch users work, track support calls, measure task completion, compare before-and-after speed, and see if people abandon your solution for workarounds. The company's nurses ditched digital forms and kept paper allergy lists instead.
Questions this video answers
4,000 development hours
1) Watch users in real time; 2) Track support tickets/help-desk calls; 3) Measure task-completion rates; 4) Compare before-and-after metrics; 5) Check if users stick with the new solution or find workarounds.
40% of users abandoned the forms about halfway through
Before: 30 seconds; After: 45 seconds
500 degrees for 12 minutes
Chapters
A friend of mine's a product owner at a medical records company, and she once told me that they were sure that their team had delivered exactly what was asked for, but customers weren't using it. The code was perfect and the features all worked, all the tests passed, and they even delivered on time and throughout the project. Their metrics looked great, and so by every sort of traditional measure, they were killing it. But something was missing. After about six months, the users actually went back to their old manual processes while the new system just sat untouched. She told me the company wasted 4,000 development hours building features that solve the wrong problems. Now they face an expensive rebuild with angry users and a shrinking budget. When we tend to focus only on delivery and not value, we were focusing on the wrong area. Yes, checking acceptance criteria, it's important, but it's not enough. Perfect delivery of the wrong solution, well, it costs you money. Think about ordering a pizza. The acceptance criteria might say it needs cheese and sauce and whatever specific toppings you want, as long as it's not pineapple. And that it should be cooked at 500 degrees for 12 minutes if the pizza arrives exactly to spec but tastes terrible. Did we really deliver value? As product owners, we often focus too much on checking boxes and not enough on checking results. Every story marked done needs an immediate value check, not just a criteria check. So let me give you five ways to assess value delivered. First, we want to watch users in real time. Don't wait for reports or surveys. No, sit with them. Observe how they work and notice where they get stuck. Second, track support tickets and help desk calls. If users need help doing basic tasks, you've got a gap between what you delivered and what they need. And third, measure task completion rates. How many users start a process but don't finish? Where do they drop off? That tells you where the value begins to break down. Fourth, compare before and after metrics. If it used to take a user 10 steps to complete a task, and then you deploy a new feature, and it still takes them 10 steps, did you really deliver value? No, no, no, no, you didn't. Finally, check if users stick with the new solution or find workarounds. This is a big one for me. When people go back to their old ways of working, it's a clear sign you've missed something important. These things give us a clear, actionable way to assess value while staying focused on our outcomes rather than our outputs. Each method helps spot gaps between delivery and actual value. My product owner friend at that medical records company, she's the one who told me about these steps. And here's what she did. First, she sat with the nurses during a shift change. This was like a prime time for record updates. And they spotted nurses opening multiple screens just to enter basic patient data, even though the new system was supposed to simplify this. Second, they noticed that help desk tickets were clustered around medication reconciliation. And the nurses. They kept asking how to view historical medication changes, which is a basic need the team thought they'd solved. Third, they tracked completion rates of admission forms. They found 40% of users abandoned the new digital forms about halfway through falling back to paper, they found out that the autosave feature wasn't working as expected. And fourth, they timed common tasks before and after the new system launched. Blood pressure entries now took 45 seconds versus 30 seconds in the old system. Faster technology, but worse workflow. Finally, they found nurses keeping paper backup lists of patient allergies. Not trusting the new system's display, the team had built a technically perfect allergy tracker that the nurses found hard to scan through quickly. Each of these checks revealed problems that normal acceptance testing missed. The features all worked perfectly. They just didn't work for the actual users. And finding these issues six months late, that's six months too late. You need to spot them immediately. Don't wait for formal releases or even the end of a sprint to assess value. The best product owners I know, they keep a simple daily routine. They watch users and ask, is this making things better or is it just different? They make assessment a habit. When a developer says this story is ready for review, they don't just check the acceptance criteria. They check if it actually helps users do their work better. And they share what they learn with a team. Teams need to understand not just what they're building, but how it's impacting users. This does more than create better solutions. It actually creates more motivated teams. Now, assessment, it's not about finding fault. It's about finding truth. The truth about what users need, what works and what doesn't, and most importantly, what to do next.
