In this talk Andy Campbell critiques the common practice of judging training programs solely by completion rates and satisfaction scores. He argues that these metrics answer the wrong question when stakeholders ask whether the training actually solves the underlying work problem. Campbell proposes a proactive approach: before building a course, define concrete outcome measures that reflect the real issue (e.g., fewer requests with missing information, reduced turnaround time, fewer repeat questions). He suggests gathering baseline data by reviewing existing reports or sampling a representative set of work artifacts, then comparing the same metrics after the training while being transparent about any concurrent changes (form redesign, instruction updates, process tweaks). He also emphasizes talking directly to the workers and managers to capture qualitative insights that numbers miss. If the training fails to move the needle, Campbell advises honest analysis of why – perhaps the problem was misunderstood or the training didn’t address the real pain point – and using those findings to guide the next iteration. Completion rates remain important for compliance, but they should be a secondary metric, not the sole indicator of success.
Chapters
The training's out, people are finishing it, the scores look good, and everybody's happy with the launch. Then somebody asks, so is it working? And you look at the report again because that's the information you have. I think that's an awkward spot we can get ourselves into. We know who finished, we know who passed, we might know whether people liked it, But somebody's asking whether it helped with the problem, and we haven't really worked out how we'd know. Let's say people keep submitting requests with information missing. Managers are sending them back, everything takes longer, and somebody asks us for training. We build it, people take it, and the completion numbers look great. Okay, are the requests any better? That's what I want to check. Are fewer coming back with that information missing? Are people getting stuck in the same place? Are they still asking the same questions? And I'd much rather have that conversation before we build. Just ask the manager, how are you noticing this problem now? Maybe they're reviewing the requests themselves. Maybe there's a report Maybe they can show us a few examples. We need some idea of what was happening before we try to say it's improved. It doesn't have to start with an enormous dashboard. If there's an existing report, great. If there isn't, maybe we agree on a way to review a reasonable sample of the work before and after. The important part is that we're looking at the same kind of thing. Instead of comparing whatever happens to make the launch look good. Of course, then we have to be honest about what else changed. If we deliver the training, rewrote the instructions, and fixed the form, we can't announce that the course did all the work. Maybe the form was making the mistake ridiculously easy in the first place. I'm happy if the whole thing helped, I just want us to describe what happened accurately. If it didn't help, that doesn't mean we hide the report and hope nobody asks again. I'd want to go back and look. Did we misunderstand the problem? Could people find the reference? Did the practice prepare them for what actually happens? Did we explain something beautifully that wasn't causing the issue? That would be frustrating, obviously. Nobody enjoys finding out the thing they worked hard on didn't do what they hoped. But at least now we've got somewhere useful to go with the next conversation. I talk to the people doing the work too. What's easier now? What's still a pain? Their answers won't tell us everything, but they might explain something the numbers can't. Completion still matters when people are required to finish the training. Keep track of it. I just don't want us to stop there when the original request was about something going wrong at work. Before we start building, I'd like us to be able to answer, if this helps, what should we see happening differently? Because I'd rather work that out at the beginning Then stare at a completion report after and try to make it answer a question we never designed it to answer.
