In this video Michael Kaminsky explains why statistically significant results can still be completely misleading, especially when they arise from exploratory data slicing. He illustrates the problem with a marketing campaign example where a specific demographic (women 35‑44 in the Midwest) appears to click ads at twice the rate, leading leadership to reallocate budget based on that insight. Kaminsky points out that a p‑value threshold of 0.05 means a 5% false‑positive rate, so examining many slices (e.g., 40) will naturally produce about two “significant” results purely by chance. He defines HARKing (Hypothesizing After Results are Known) and p‑hacking, describing how they turn random noise into seemingly causal stories that can drive costly business decisions. To avoid these pitfalls, he recommends three safeguards: (1) label unexpected findings as “interesting” rather than definitive insights, (2) keep track of how many data cuts were made and interpret significance in that context, and (3) test the finding with a pre‑specified, targeted experiment before acting on it. The overall message is that a hard line separates genuine insight from a narrative built on convenient data.
Chapters
Researchers estimate that 24 to 40% of findings related to business strategy wouldn't hold up if you actually analyze them properly. How does that even happen? Picture a marketing team that runs a nationwide paid social campaign with new creative centered around their rebrand. After the campaign, someone digs into the results and comes back with an insight. Women aged 35 to 44 in the Midwest click the ad's call to action at twice the rate of other cohorts. That insight then goes into a presentation deck and the leadership team that gave the rebranded creatives a thumbs up, loves it. So they decide to shift more paid social budget to that segment next quarter. Digging into and slicing up campaign results like this is fine, but the problem starts when exploratory findings get put into a deck as confirmed insights about our key audiences. Then budget moves before anyone checks whether that insight is real or just the result of statistical noise. And here's why. The p-value threshold for statistically significant, it gets typically set to 0.05, and that implies that we accept a 5% probability that a winning result result is actually a false positive—one that happened by chance alone. So if you run analyses over 40 slices of your data, you expect about 2 to look like winners, even though there's nothing real going on there. Those are false positives, and they're often used to inform next quarter strategy. There's a name for this in the academic community. It's called HARKing, an acronym that stands for Hypothesizing After the Results Are Known. You saw the answer first, and then you constructed the question you sought to answer around it. P-hacking, another well-known statistical workaround, and HARKing are related. P-hacking when you keep tweaking an analysis until you get a good result. Harking is when you take a result and then you build a story around it so it looks like that's what you were testing all along. Harking sounds benign, but it can be expensive. A brand might shift budget toward a segment that was never actually driving incremental outcomes for their KPIs, but just happened to look that way in one slice of the dataset. Then next quarter, they underperform their projections and nobody can figure out why. But the reason is that the original insights are coming from harking. They're not real. The final lesson here is that when you If you find something unexpected in your data, you should do 3 things. First, call it what it is. Say, "We found something interesting," not, "We discovered that X drives Y." Those aren't the same, and the latter implies that you are looking for it and that you have causal signals. Second, count your cuts. If you slice the data 40 ways and for 2 segments you got statistically significant results, it's not really a surprising finding. It's a 2 out of 40 finding, which is exactly what you'd expect to get by random chance with a p-value of 0.05. These winners might be the result of chance alone. Third, put your findings on trial. Run a targeted experiment to test the finding of your analysis and pre-specify the hypothesis before you see any new data. Until you do that, it's more like a lead, not a conclusion. There's a hard line between a genuine insight and a story backed by convenient data.
