Is Your Company Reading Data the Wrong Way?

AI is transforming the world. And it starts with the right compute. ARM is the AI compute platform trusted by global leaders, proudly NASDAQ listed, built for the future. Visit arm.com discover. Welcome to the hbr ideacast from harvard business review. I'm kurt nickish. You're a business owner and you're interested in reaching out to new customers. You know that data is important. I mean, that's clear, right? So you put out a survey into the field asking what kinds of products your ideal customers are looking for. You get that data back and you have a clear decision made for you as to which direction to go. You develop and sell that new product with a big marketing push behind it, and it flops. But how can the data be wrong? Today's guests believe in data, of course. But they see major ways in which over reliance or under reliance on studies and statistics steer organizations wrong. They found that leaders often go to one of two extremes, believing that the data at hand is infallible or dismissing it outright. They've developed a framework for a better way to discuss and process data, to interrogate the data at hand. Michael Luca is a professor at Johns Hopkins Carey Business School, and Amy Edmondson is a professor at Harvard Business School. They wrote the HBR article Where Data Driven Decision Making Can Go Wrong. Welcome. Thanks so much to both of you.

Thanks for having us.

Thank you.

Are business leaders relying too heavily on data to make decisions?

I don't think that's quite the problem. One of the things that really motivated Michael and me to get together is that I study leadership and leadership conversations, particularly around really difficult, important decisions. And Michael is a data science expert. And our mutual observation is that when leadership teams and leaders are using data, or teams at any level are using data, they're often not using it well. And so we've identified predictable or frequent errors, and our idea was to help people sort of anticipate those and thereby do better.

Is it more of a data science understanding problem here or more of a having the right culture to discuss the data correctly?

Well, that's just it. We think it's both. But I'll just say, you know, in a way, my side of the problem is we need to open up the conversation so that it's more honest, more transparent. We are in fact, better able to use the data we have, but that's not enough. And that's a lot. But just getting that done will not ensure high quality data driven decision making.

Mike, data has kind of been all the rage, right? For at least the last decade. I feel like it was 10 years ago or so that Harvard Business Review published this article saying that data scientist was the sexy new job of the 21st century. A lot of places make a priority of data to have something concrete and scientific. If they're getting better at collecting and analyzing data, where's the decision making problem here?

We're certainly surrounded by data. There's growing data collection at a wide variety of companies. There's also growing research that people are able to tap into to try to get a better sense of what the broader literature says about questions that managers are grappling with. But at the same time, it's not really about just having data. It's about understanding both the strengths of the data that you have and the limitations and being able to effectively translate that into managerial decisions. There are a couple of challenges that we discussed in the article, but they all kind of come down to this idea of once you see an analysis, and the analysis could be coming from within your company or from something that you've read in the news or from a research paper, how do you take that and understand how that maps to the problem that you have at hand? And that's the decision challenge. And this is where effective conversations around data and having a framework for what questions to be asking yourself and what questions to be discussing with your team come into play.

In your interviews with practitioners, you identified that there's kind of two big reactions to, you know, this data that's been collected, internal or external. As you just said, where do those reactions come from? Like, why are we seeing that?

As you said, Kurt, data is the rage.

And we have more than ever.

And we have more than ever. Right? So it's sort of, you know, you can really understand the why. Okay, great. You're telling me there's the answer. You know, everybody should get a pay raise and that'll make us more profitable. Okay, I'm just gonna do it. Or, yeah, that's nice literature out there, but really we're different. I think we see both modes and they're easy to understand. Both are wrong. Both need to be more thoughtful and probing in what applies, what doesn't apply. What does this really mean for us? And we believe there are good answers to those questions, but they won't pop out without some thoughtful conversations.

Analytics or any empirical analysis is rarely going to be definitive. I think the conversations need to come around. What are the outcomes that we're tracking? How does it map to the things that we care about? What is the strategy they're using to know if an Effect that they're saying is causal actually is. I think those conversations often don't happen, and there's a number of reasons that they don't happen in organizations.

So you're advocating for kind of this middle path here where you really interrogate the data, understand it, understand its limitations and how much it does apply to you, how much it can be generalized, which sounds like work, but you've laid out a framework to do that. Let's start with that. Where the data comes from, internal or external? Why is that a key thing to understand?

When we think about external data, there's exciting opportunities to take a look at what the literature is saying on a topic. So, for example, suppose that you are managing a warehouse and trying to understand the likely effect of increasing pay for warehouse employees. You don't have to just guess what the effect is going to be. You could take a look and see other experiments or other causal analyses to try to get a sense of what people have learned in other contexts. And then you as a decision maker, could think about, how does that port over to your setting? Now, in thinking about how to port over to your setting, there are a couple of big buckets of challenges that you'll want to think about. You want to think about the internal validity of the analysis that you're looking at, meaning, was the analysis correct in the context in which it's being studied? So is the causal claim of wages on, say, productivity, is that well identified? Are they outcomes that are relevant there? And then you want to think about the external validity or the generalizability from that setting to the setting that you're interested in, and think about how closely those map together. So I think it's both an opportunity to look more broadly and what the literature is saying elsewhere, and to bring it over to your setting, but also a challenge in thinking about what's being measured and how to port it over. Now, for larger companies especially, there's been a growth of internal data. So you could think about Google or Amazon or other large tech companies that are tracking exorbitant amounts of data and often running experiments and causal analyses. Those come with their own challenges, thinking about what is the metric we care about. So it's slightly different challenges, but related, but then sort of zooming out. What you want to think about is combining what internal and external data do we have and how do we put it all together to come to the best decision that we can, to get.

A fuller picture, really, in a way, what we're saying, which is pretty simple, but I think really Profound is that you can't just assume, you know, if someone tells you, here's a result, you can't just take it at face value. You have to interrogate it. You have to ask questions about causality, was it an experiment or not? You have to ask questions about, you know, what was actually measured and what's the context like and how is it different from my context and all the rest. And these are things that scientists would naturally do and managers also can do and get better decisions as a result.

It's a lot of basic statistics skills, right, that not everybody has. It sounds like you kind of want that capability across the team or across the decision makers here, and not only housed in a data analytics team in your group, for instance.

Yes. And it's not that everybody needs to be a data scientist. It's that data scientists and operating managers need to talk to each other in an informed and thoughtful way. So the managers need to be able to learn and benefit from what the data scientists understand how to do. And the data scientists need to think in a way that is really about supporting the company's operations and the company's managers.

So maybe just one quick example. There's this famous ebay experiment that looks at the impact of advertising on Google. And what they found is largely the ads that they had been running were not effective at generating new business coming into ebay.

And just to spell out this ebay experiment, they had been advertising in markets and seeing more sales there, and they thought the advertising was working, but they were basically just advertising to people who were going to be buying more from them anyway. So the effect of all that advertising spend was pretty muted.

That's exactly right. So they had been running billions of dollars of ads per year on search engine ads. So they had actually brought in consultants to look at this and try to analyze what the impact was. And initially they thought that there was a positive effect because of the correlation. But then by thinking more carefully about the fact that ads are highly targeted, that led them to run an experiment to get at the causal effect of ads. And that's when they realized that many of the ads they were running were largely ineffective.

And so was this a correlation causation problem? Essentially, at its core.

So for ebay, there was a correlation versus causation problem. Then you could think about generalizing that to other settings, other types of ads on ebay, other companies that want to use this result. In fact, even within that one experiment, when you dive a little bit deeper, they found certain types of ads were slightly more effective. Than others. So you could find corners of the world where you think there's more likely to be an effective advertising and change your advertising. So it's correlation, causation, and then trying to learn more about mechanisms or where ads might work so that you could update your strategy. Then as external companies saying, here's this new evidence that's out there. How do I take this and adjust either my advertising strategy or my approach to measuring the impact of advertising?

Tell me more about the disconnect between what is measured and, and what matters. We all know that you get what you measure. We've all heard that. Where do managers often go wrong here?

Such a challenging problem. And actually earlier we were discussing the fact that many things are measured now, but many more things are not measured. So it's actually really hard to think about the relationship between one empirical result and the actual outcomes that a company might care about at the tail end. So, for example, so imagine you wanted to run an experiment on a platform and change the design. You change the design and you see more people come. That's one piece of the puzzle. But you really want to see what's the long run effect of that. How many of the customers are going to stick with you over time? How happy are they with the products or the engagement on the platform? Are there going to be other unintended consequences? And those are all really hard things to measure. We're left in a world where often analyses are focused on a combination of important things, but also things that are relatively easy to measure, which could create pretty important disconnects between the things that are measured in an experiment or an analysis and the outcome of interest to a manager or an executive.

Amy, when you hear these problems like disconnects, you could also call that miscommunication.

Absolutely.

From an organizational culture perspective, how are you hearing this?

So I hear it as a. I think there's a general need to go slow, to go fast, and there's a strong desire to go fast just in everything, you know, data. It's a modern world, things are moving fast. We want to get the data and then make the decision. And we write about the fact that it's this issue we're talking about right now that making sure that the outcome we're studying, the outcome we're getting data on is, is in fact a good proxy for the goal that we have. And just that getting that right, then you can go fast or go faster. But really pausing to unpack assumptions that we might be making, what else might this design change encourage or discourage what might we be missing? Asking those kinds of good questions in a room full of thoughtful people will more often than not allow you to surface underlying assumptions or things that were missing. And you know, when a culture allows, when an organization's culture or climate allows that kind of thoughtful wrestling with very ambiguous, challenging, uncertain content, you'll be better off. You'll design better experiments. You'll draw better inferences from the data or studies that you do have access to.

We've talked about the disconnect between what's measured and what matters, conflating correlation and causation. Let's talk about some of the other common pitfalls that you came across in your research. One is just misjudging the potential magnitude of effects. What does that mean? What did you see?

Well, we talk about our general lack of appreciation of the importance of sample size. Certainly any statistician knows this well. But intuitively, we make these errors where we might overweight an effect that happens in a very small sample and realize that that might not be representative to a much larger sample. So how precise can we be about. The effect that we're seeing is very much dependent on the size of the sample.

You suggest a question to ask there. What's the average effect of the change? To get a better sense of what the real effect is.

I think for managers it's thinking about both what the average effect that was estimated and also what the confidence interval is to get a sense of where the true effect may lie. Thinking about confidence intervals is important both before and after you conduct an analysis. Before you conduct an analysis, anticipating the uncertainty in effects is going to tell you how large of a sample you might need. If you're going to, say, run an experiment after an analysis, it could tell you a little bit about what the range of true effects may be. So a recent paper looked at advertising experiments for a variety of companies and found that many of the experiments that were being run didn't have the statistical power to determine whether it had positive or negative ROI.

So they'll hear, okay, it was up 5, you know, sales were up 5%. Oh, great, you know, let's do it. Let's roll it out. But in fact, that up 5% was well within what's called the margin of error and may in fact even be negative. It's possible that that advertising campaign reduced interest in buying.

Right.

We just really don't know.

Based on the sample size, overweighting a specific result is also a common trap. Can you explain that?

Yeah, it's a confirmation bias or a desirability effect. Or sometimes if a result is just very salient or it kind of makes sense, it's easy to just say, okay, this is true, without pressure testing it, asking what other analyses are there? What other data might. We need to have more confidence in this result. So it's kind of a variation on the theme of the magnitude of the effect.

One common pitfall is also misjudging generalizability. Why is this problematic?

So we talk about that example in the article where there's SVP of engineering that was talking about why he doesn't use grades in hiring and says, well, Google proved that grades don't matter. Now let's put aside the fact that we don't know how Google exactly did this analysis and whether they actually proved that. It doesn't matter in the Google context, but it's a pretty big leap to then say, because they've shown this in one context, that that's going to be poured over exactly to the context that the SVP was thinking about in his company. So I think what we have in mind here is just thinking a little bit more about the relevance of findings from one setting to the other, rather than just kind of porting it over. Exactly. Or dismissing it altogether.

What's a good strategy to break out of that when you're in that situation or when you see it happening?

Yeah, so you can't see me smiling, but I'm smiling ear to ear because this really falls squarely in my mind territory because it's so related to, you know, if you want something to be true, it can then be even harder to, you know, tell the boss. Well, hold on here. We don't really have enough confidence or. So this is really about opening the door to having high quality conversations about, you know, what do we know, really? Curiosity led conversations, what do we know? What does that tell us? What are we missing? What other tests might we run? And if X or if Y, how might that change our interpretation of what's going on? So this is where we want to help people be thoughtful and analytical. But as a team sport, we want managers to think analytically, but we don't want them to become data scientists. We want them to have better conversations with each other and with their data.

Scientists in teams, as data's being discussed, how, as a leader, can you communicate the importance of that culture that you're striving for here? And also how, as a manager or as a team member, how can you participate in this and what do you need to be thinking about as you talk through this stuff, because it's definitely a process. Right?

Right. I mean, in a way, it starts with framing the situation or the conversation as a learning problem solving opportunity. And I know that's obvious, but I have found if that's not made explicit, especially if there's a hierarchical relationship in the room, people just tend to code the situation as one where they're supposed to have answers or they're supposed to be. Right. And so just really taking the time, which can be 10 seconds to specify that, wow, this is a really uncertain and fairly high stakes issue for our company and it's going to be important for us to have the best possible bet we can. So what do we know and what are the data telling us and what do we need to learn? And really probing the various people in the room for their perspectives and their interpretation. So I think starting with that stage setting and then like we write about leaning into questions, you know, we provide a set of sample questions. And they aren't the only questions or even a cookbook of questions, but they illustrate the kinds of questions that need to be asked. Tone matters. Tone needs to have a feeling of genuine curiosity. Like, ooh, what outcomes were measured? Not well, what outcomes were measured? You know, were they broad enough? No, it's, you know, how broad were they? Did they capture, you know, any chance that there were some unintended consequences and so forth. So it's gotta be approached in a spirit of genuine learning and problem solving and viewing that as a team sport.

When can you lean into the answers?

Right. Cause it's, you know, there's never gonna be the sort of perfect answer. The crystal ball. There are no crystal balls. So it's a very good question.

It seems like to be really good at data driven decision making, you have to be analytical and you have to have those hard skills. You also have to have the soft skills to be able to lead these discussions among your team and do it in a psychologically safe space. I mean, it definitely sounds hard.

Yes.

And you can see why a lot of people go the easy route and say, oh, that doesn't apply to us, or yes, that's the gospel truth. What's your hope out of all of this?

Well, I think my hope is that we all get more comfortable with uncertainty, start to develop the emotional and cognitive muscles of learning over knowing, embracing learning over knowing and then using the team. This is a team sport, so that's kind of. Those are mindset things. And then so that we get more comfortable with a mode of operating that is really just test and iterate. What do we try? What did the data tell us? What should we try next? Life and work in kind of smaller batches rather than these giant decisions and giant rollouts. There's going to be more navigating the uncertainty, I think, going forward. And we need people who are, as you said, analytical, but also curious, also good at listening, also good at leading a team conversation so that you actually can get somewhere and it doesn't have to take forever. We can have a conversation that's quite efficient and quite thoughtful and we get to a sufficient level of confidence that we feel now we're able to act on something.

People talk a lot about things like quote, unquote big data or large scale analytics. And I think there are a lot of interesting innovations happening there. But I also think there are lots of contexts where a little bit of careful data could go a long way. So I think when it comes to many managerial questions, thinking about is this a causal inference question and if so, what is the question we're trying to answer from a team perspective, my hope is that people will be focused on trying to answer a question that could then inform a decision. And by thinking about the analytics underlying it and being comfortable with uncertainty, you get to a more effective use of data. And that's both the internal data that's sitting within your organization, but also the growing amount of external data that's coming from academic research or news articles and thinking about how to synthesize information from these different sources and then have good group discussions about how to effectively use it.

Mike and Amy, this has been great. Thanks so much for coming on the show to talk about your research.

Thank you.

Thanks.

That's Cary School Professor Michael Lucca and Harvard Business School Professor Amy Edmondson, co authors of the HBR article Where Data Driven Decision Making Can Go Wrong. Want some more data points? We have nearly 1000 episodes plus more podcasts to help you manage your team, your organization and your career. Find them at hbr.org podcasts or search HBR and Apple podcasts, Spotify or wherever you listen. Thanks to our team, Senior Producer Mary Dew, Associate Producer Hannah Bates, Audio Product Manager Ian Fox, and Senior Production Specialist Rob Eckhart. Thank you for listening to the HBR IdeaCast. We'll be back on Tuesday with our next episode. I'm Kurt Nickish.