Being data-informed vs. data-driven w/ Milene Darnis, Product Manager at Heap

Hey everybody, this is Mike DelCito, co-founder of Product Collective, and we're back for another industry interview. Welcome to May. We're finally in May. I'm very glad to be in May and not March or April, so I'll leave those months behind. And, but no, I appreciate y'all joining us. For those of you that are joining for the first time, I'll give a little bit of information about what these industry interviews are all about. So we at Product Collective organize these industry interviews. We do these twice every month on the first and third Wednesday of every month. And really these are just conversations with product leaders from all over the world, and we cover a different topic every single time. And they're an opportunity for me to ask questions, and I have a lot of questions of my own, but they're also an opportunity for you to ask questions. So you'll notice at the bottom there are two kind of features there. There's a Q&A and a chat. The chat's really just for you introducing yourself. Maybe you can share where you're from, what product you're working on, what part of the world you're in. In fact, you can give me some assurance that you can see and hear me by maybe in the chat just letting everybody know where you're from, just what part of the world you're from. It's fun seeing people from all over the world in these industry interviews, so you can always do that in the chat. The Q&A section, that's for you to bring up questions that, you know, you want to ask our guests, and I'll be looking at that. I'll start with a few questions of my own, but again, I would say this— your questions are usually better than mine. So, you know, fill that Q&A section up at any time. You could even do it now if there was a question that was on your mind. And I'm seeing a lot of people in the chat mentioning where they're from— Northern Virginia, Berlin, Germany, Denver, Colorado, Ottawa, Canada. Yeah, it's just for me, it's always really cool to see people coming from really all over. London, Pittsburgh, that's really close to where I'm from. So I'm from Cleveland, Ohio, and we had a beautiful weekend this past weekend and sunshine today, so I will take that for what it is. But, you know, at Product Collective, you know, we do a lot of other things aside from these industry interviews. We look at ourselves as a community for product people, so there's a lot of different ways for people to get engaged. We have a Slack channel where there's 10,000+ people that were invited to be a part of the Slack channel. So every day trading ideas, best practices, that sort of thing. We have a weekly newsletter that goes out to about 30,000 people every Friday. And again, different focus every week. And then we organize conferences. So yeah, these times are a little crazy for us because our conferences are usually in-person conferences. But everything's going virtual these days. So we actually just announced a couple months back the first slate of industry virtual workshops. and those have been going really, really well. We've run now almost, I think, probably 8 or 9 industry virtual workshops with everybody from, um, Gib Biddle, who's the former VP of product at Netflix. He actually has one coming up again next week, to another one that's actually coming up, April Dunford, the author of Obviously Awesome, and she's, you know, the person to go to when it comes to product positioning. So on our website, industryconference.com, it's really easy to just find virtual and you'll find the industry virtual workshops. And then we have more things planned coming up. So some things that I can't announce quite yet, but by next week I might be able to. So I'll keep in touch with y'all on that. But I want to get back to industry interviews. I'm really excited for today's session. We're gonna be talking on things related to data and Milan— ah, I knew I was going to mess this up. Mylène? Mylène, thank you so much for joining us. This is a topic that I was kind of excited to cover because you hear about people talking about, like, we have to be data-informed. Or I hear some people talking about, well, no, our decisions are made— are data-driven decisions. And sometimes, you know, I always wonder, like, is that the same thing? Is it different? Do people that say those things understand the differences between them? So I figured This is a perfect topic to cover, and I'm glad that you're joining us. So thank you very much for joining us.

Thanks for having me.

Yeah, well, I guess maybe we should start off there. So, you know, I guess how would you define when people do say data-driven or people do say data-informed? You know, how would you define those and how are they different or are they different?

Yeah, sure. So I think to begin with, like the period we're living in, I think it's awesome that we all have access to data. So now, no matter whether you're a product manager or you work as like a data scientist, of course, and the marketing and operations, like you're very likely to have access to data. And I think this is where like the term data-driven came from. But then people started almost like going too far when it comes to data and waiting on the data to tell them what to do. So they would start to blindly follow the data. And I think this is where the term data-driven started having kind of a negative connotation where you actually don't challenge the data. You just like blindly follow it. And this is when we started introducing a term like a bit more subtle, like the difference is very subtle, but saying, oh, actually, instead of just following the data, I'm going to challenge it and also use it as a tool in my toolkit to answer questions. And this is what I think the definition of data informed becomes, where you're not just blindly following it, you're actually using the data to better inform your decision-making process.

I love that. I love it. Well, and it's like if we are to blindly follow the data, how do we exist as product people? Right. What's our job now? Right. People could just run a little analysis and just from the data alone, the decisions are already made. So I would then imagine that we as product people, we should be striving to be data-informed, but not necessarily data-driven. I mean, I guess, would you agree?

Yeah, I do agree. I think also it depends maybe on your size. So if you're a very large company, you're going to have millions of users and it's going to be much it's much easier for you to be data-driven and mostly rely on the data to make your decisions because you can afford to. You can run A/B tests very quickly, you have tons of data at your disposal so that you can make better decisions. I think when you're a smaller company, this is when it's a bit more crucial to be data-informed and basically say, okay, I'm going to use data. I have some data about my users, about how they're using my product, for instance, but I also will rely on other inputs to make my decision-making process. I do also think that this is sometimes easier said than done because today we live in a world of an abundance of data. So there's so much data that it's sometimes easy, like as product managers, we have so many things to do. So once you have the data, you might think, oh, okay, this is good. Like I have data, I can go forward. Like, you know, I checked that box. And I think this is where it starts, like the subtlety comes into play and it starts being sometimes a bit dangerous to just blindly follow the data because you want to be able to challenge challenge it and you want to make sure that first of all, you're using a trustworthy dataset, that you have confidence in that data, and also that you're actually asking the right questions and that this data is informing your decision in a way that's not too biased to begin with.

Yeah, yeah. Well, and one of the things I've always wondered about it too, because you mentioned how we have this abundance of data now at our disposal, and that's so true. And there's a lot of different platforms now that we can use to get this data to us, how much data is enough data? Like when should we, even though we can get so much data, we often have to make decisions quickly as product people. Like sometimes those decisions made without necessarily having, quote, all of the answers. So how do we sort of manage that? Like what's your take on that?

Yeah, I think this is a difficult question. And again, I think it really depends on your size and what kind of decision you're trying to make. It's not going to be like a one-size-fits-all, like, oh, you need 1,000 data points and that's probably enough. That's never going to happen. So again, if you're a larger company, enough data might actually be millions of data points because you have hundreds of millions of users. And if you're a smaller company, enough data might just be like 3 or 4 data points, but that actually really tells a very important story. Story. And I think this is something, for instance, that we do at Heap and that I really enjoy doing where you say you have a hypothesis. So you say, for instance, that I believe that this certain segment should be using this specific feature. And then you observe the data and you see that, oh, actually there are people in the segment that I was exactly targeting that are not using the feature at all. So now you have these data points and From there, you probably don't have enough data to conclude and just say, oh, this feature is not working, but you have enough data to dig deeper. And now you can talk to these customers. You can ask them like why they're not like using this thing that I built specifically for people in their like segment. So the one size fits all, yeah, there's no one size fits all. So it will really depend on like, yeah, your company size and what decision, kind of decision you're trying to make.

And I'm glad you brought that last part up too about, you know, hey, now you have at least some data, then go talk to people because qualitative data is data too, right? It's not when we're saying data informed, like we're not just saying it's the numbers. It sounds like, you know, you're a proponent of utilizing qualitative data too, right?

Totally. I think it's definitely a mix, like quantitative data, qualitative data. We also run a lot of surveys and I think, yeah, it's like all about like making sure you're not over-indexing on one part. And making sure that when you're using the purely quantitative data, you're using it with rigor. And for instance, if you run an A/B test, you have confidence that you'll be able to reach the necessary sample size to conclude that A/B test. Otherwise, you probably shouldn't be running it in the first place, but using other methods.

Yeah, got it. No, that's great. By the way, I'm seeing a lot of good conversation in the chat. Keep that conversation flowing, everybody. It's good to— You know, when the topic comes up, feel free to go back and forth like y'all have been doing. I think that's great. And by the way, just another reminder, if there's questions that you have for Milan that you can ask, like, just go to the Q&A section. I will do my best to look at the chat and see if questions are coming up there, but if you can, just put it right in the Q&A and I will get to it. I will, I will even prioritize it above my questions too. So fill that Q&A section up so we can take advantage of this time. Time with our guests. One thing I did want to ask you is for people that are here and they're, you know, maybe nodding their heads, you know, in front of their computers, but they're like, okay, I want to do a better job of getting in the mindset of being data-informed. Like, what recommendations would you give to them? Like, how do you kind of properly get in the mindset of actually putting that into practice?

Yeah, I definitely think that there are small tips that you can use to ensure that you're being data-informed and that you're using using your data the right way. And so the first one you might want to do is make sure that you define correctly your metrics. So to begin with, you need to use proper metrics so that you are telling a hypothesis and they're driven by your metrics. And so this way, you actually will continue to— sorry. So this way, you will— have defined that in advance, and you will use the data to validate or invalidate this hypothesis. So a typical example might be making sure that you look at retention instead of acquisition, if that's what you're trying to achieve. I think this is a common pitfall that now a lot of people are trying to, like, are starting to really understand, where sometimes one metric can come at the expense of another one. And so if you just look at acquisition, for instance, you might forget that too much acquisition will negatively impact retention in the long run. So setting up upfront the metrics properly I think is super important. Another tip that I like to use is making sure you challenge the data because you sometimes don't challenge it enough. Like you see data trends that seem surprising and you think, "Oh, okay, I guess that's how it is." And then you go following this data and this is where you start to maybe blindly follow the data instead of challenging it enough. So if your data is telling you a story that is super surprising, I always encourage people to challenge this data. Oh, actually, is it really the right dataset? Did I segment things in a way that tells a wrong story? I think it's super important to be critical of the data that you're using to make your decisions. And lastly, I do also believe that sometimes you will hit limitations, things that are not easy to measure. And I think this is where it's important important to have a good conversation with your team to basically decide whether you are still comfortable moving forward without having a full picture using data, because there's just no data. And a good example of that, for instance, might be if you are prioritizing between making improvements to an existing product or shipping a brand new product. If you were purely data-driven, you're always going to prioritize the improvements to the existing feature because you have data for that. But if you're actually data-informed, you're going to understand that in one situation you have a lot of data, in the other one you— in the other one you won't. And you'll have this discussion with your team whether it's worth doing it.

Yeah, thank you for that. I, by the way, see some good questions coming in. I'm going to get to them right after this. The only question I wanted to ask you before I do is just, you know, we're Even if we do sort of accept this mindset, it would probably be even better if we can help others that we're working with take this mindset on as well and sort of empower the rest of the organization. And I'm curious if you have sort of advice on how we can better, you know, just sort of empower the rest of our organization to take on this data-informed mindset.

Yeah, definitely. And I really love this question because I strongly believe that data and the product data shouldn't be something that only product managers own and have access to. Like, yes, maybe as a product person, I spend a lot of time understanding how are people using my products and looking at adoption of new features, for instance. But I think like the entire company should be striving to having hypotheses and validating whether they're true or not using data. And one thing I've seen people do is say, oh, we're going to buy all the best tools and then my entire company will become data-driven or data-informed. And saying I'm going to buy the best business intelligence tool, the best user behavioral tool, the best warehouse, like all these things that are super useful and say, oh, and then that's it. And now my company is data-driven. I actually think it's not that easy. Yes, these tools are great and they're super important, but I think the most important thing is to teach people how to ask good questions using the data. Now the tools are becoming super intuitive and easy to use, but if people in your company don't know how to ask the right questions that can be validated with data or not, it's going to be very challenging to actually make data-informed decisions in your company. And to take an example, for instance, let's say you want the marketing team to be more data-informed. And so you might want to say that if they just have the tools, they might be tempted to look at something that some people also call vanity metrics, things such as number of visits to the blog. And this is great, like I mean it's probably a good indicator, a good thing to look at, but it's likely that it's not going to be very actionable. But then if you tell your marketing team, oh actually, like the goal of the blog is to drive conversions and to drive people to actually buy our product, then now your metric instead of being just the number of visits becomes the conversion rate. How many people go from the marketing site to actually purchasing my product? And then you like— it's by like showing how you can ask questions this way and using more accurate and precise KPIs that you're going to kind of instill this data-informed culture in your company.

Thank you. I'm getting some questions, and so I do want to go to some of these. So let's see, one of the questions was, how do you balance challenging the data versus falling into the trap of ignoring data or massaging it until it meets your preexisting expectation. Do you have thoughts on that?

Yeah, yeah, I do. I like this idea of massaging the data. I think there's a quote I really like that says, if you torture the data long enough, it will confess. And I think this is very true. Like, if you torture the data long enough, if you have a preexisting bias in your head of like the data should tell a certain story, you will find data that tells your story. And this is definitely like the bad thing that you don't want to be doing. And so I think challenging the data will also depend on how much data you have. If you have a large dataset and things are very surprising, I will really spend time challenging it to understand why I'm seeing something that I was not expecting. Because with so many data points, it's likely that there's going to be an error in logging and how things are instrumented, for instance. And it's important to get to the bottom of that because you want to continue to use these indicators and these metrics going forward. I think when it's a small dataset, on the other hand, like, like an advice has been given by a mentor in the past is that the best solution is probably to pick up the phone, right? Like ask your customers what is happening, try to understand why, like you're seeing like this surprising data. That's probably the easiest way to challenge the data rather than like spending too much time fixing something for a very small dataset.

Thank you. I'm going to go to another question here. Let's see here. So, this might be— all right, so another question is just with everything that's going on right now, you know, potential exists for data trends that have, you know, that's never been seen before. So, do you have suggestions on effective ways to separate new trends from incremental changes that you might have made? So, maybe the way that we've been looking at data in the past, maybe it's changing on the fly. Do you have any sort of good advice on how to maybe deal with some of that?

Yeah, I think it's definitely difficult and challenging. I wonder if we will go back a few months or years from now and look back at this period and say like, this is just a big outlier and we should completely ignore it from our analysis. Like, it's really hard to say whether some of these trends are here to stay. Or if they're just short-lived and they will disappear once this crisis is over. I do think something that we've been doing, for instance, and it's super important, and we also advise our customers to do the same thing, is to make sure once again that the right metrics are being tracked. And I think in days like today, for instance, retention is more important than ever because it's going to be very hard to get new customers because people are not willing to spend as much as they used to, and people are being much more cautious. But if you can retain your existing customer base is probably going to be critical for the continuation of your business. And so making sure that you look at the right metric and that you're informed by these and not maybe looking too much at the noise created by other metrics is probably the best way you have to survive.

Yeah, that's great. Thank you. Oftentimes, I know as a product person, I make a lot of mistakes. I try to learn from mistakes. When it comes to data and everything we're talking about here, what mistakes do you see made maybe most often and how can people avoid some of those mistakes?

Yeah. So one thing we touched on was having a preexisting bias of like what you think the data will say. So if at the end of the day, as a product person, you're like completely convinced that you're going to do something and you're using data just to validate that you're data-driven and that you've like checked that box. Don't do that. Like basically you're not gonna— like it's not bringing anything extra and you're just going to be finding like this data point or this trend that justifies you doing a project. So, and this is hard to do because everyone wants to use data and everyone wants to like show that, yeah, I not only have a strong product intuition, but I also validate it using data. But I think this is where you need to maybe step back a bit and say, Okay, like maybe I actually want another opinion, someone who disagrees with me so that I can come into the problem a bit more unbiased and formulate a real hypothesis that they will try to validate with data. Or like be completely honest and say like, it's going to be hard to find data around this specific thing that I want to build, but I believe it's the right thing to do for the company. And then once we ship it, once we release that new feature, for instance, We're going to look back and look at the data that then gets created to, like, close the loop and say whether, yes, it was a good decision or not. But sometimes you have to kind of take those bets.

Thank you. I'm going to go back to audience questions here. And there's a lot of them. So, you know, keep those coming. I'm going to try to get as many as I can in the next few minutes here. Let's see here. OK, so this is one. So somebody was curious about your perspective on somebody that is, for instance, you are a product person at a data analytics company. What's your view on it on somebody that is a data scientist and a product manager? Like how much data should a product manager be able to be comfortable with managing versus somebody that's a data scientist? That is their job. So I guess like where, like how much data should we be great at analyzing as product people?

I think in a perfect world, every product manager is paired with a data scientist and is able to go really deep into the understanding of the underlying data. I think the reality today is that we just don't have enough resources to do that. And it's very often that you have more product managers and sometimes not even a single data scientist at your company. And so I think this is where it's important to also make sure you have the right tooling so that as a data— as a product manager, you can use data in a way that's not too time-consuming. I've seen, I've worked at companies where everyone was expected to write SQL. And I think on the one hand, okay, this is great because that means that people are learning new skills and are also very self-sufficient to answer their data questions. But at the end of the day, I don't think that scales because it's too much to ask for everyone in the company, including marketing people, product people, to just be completely fluent in SQL and be able to answer all their questions this way. So I think it's finding the right balance between having all the right tools. So as a product manager, you can use data, but you don't spend like half of your week like pulling this data and messaging it to understand it. And at some point, like typically what we see is that at some point companies become large enough that it's time to bring on a full-time data scientist to take care of most of that work.

Yeah, thank you. I am going to go to a couple more audience questions if we can. But the one thing I did want to ask you before we, you know, before we kind of close out is just with all of this that we're going through, like if there is one big takeaway from all this that you want people— because I don't know exactly how many more questions I can ask, we're coming to the end of the time soon— but what is that one big takeaway that you do want to make sure people can get out of this in particular?

Yeah, I think I would like people to take away that data is very powerful, data is very important and critical But with great power also comes great responsibility. And so as product managers, as product people, it's our responsibility for using data the right way and not just blindly following it and not challenging it, but instead using the power of data in a way that's scientific and rigorous so that you use it as a tool in your toolkit of decision-making process.

Yeah, no, I thank you. I appreciate that because I think that for a lot of us that are watching this that are again kind of accepting of this information, sometimes you take in so much, so much information could be information overload too. So I wanted to make sure we kind of capture that at the end too. And then, you know, so we have time for a couple more questions. This one I think is a good one. It's just, you know, where you talk about experimentation a little bit and the person was wondering if you could share an example This could be just an example everybody might be familiar with, or in your own experiences, that maybe an example for a hypothesis that you were able to experiment with, and you know, kind of using data from that experimenting, like how were you able to best sort of leverage that?

Yeah, sure. Let me think. I think in general, like, yeah, if so, I'll start with one thing. As a B2B company, it's harder for us to run experiments because we have smaller customer size. I used to work at Uber, so of course, like thousands of experiments were run all the time. And I think in general, like the keys of a successful experiment was to say that you have like a right hypothesis upfront. So for instance, like we had the experiments around making sure that for drivers, like that they would, like around number of trips that would be completed by a driver on a given day. And then, so you have a strong hypothesis. And another thing that I think I've seen that is super important is have guardrail metrics for that experiment. Because the problem of, for instance, having a driver complete too many trips on a day is that that person might be tired, might start having negative reviews, all these things that were not in scope of what you were initially intending to do, but might be negative consequences of your experiment. So I think a successful experiment has that upfront hypothesis and also has a set of guardrail metrics that are being monitored as the experiment runs so that you ensure that you're not negatively impacting other things.

Thank you. No, I appreciate that. And then let's see, I'm going to do one more question, I think, before we kind of close things off. And then let's— so this kind of goes back to making hypotheses. It says, would you say it's best practice to define your assumptions and hypotheses first? And how might you go about doing that? And maybe that will be— we've gotten a lot of questions from the audience, so maybe that might be the last one.

So the question was how important it is to define your hypothesis or how to do it.

Yeah, exactly. Is it important to define those first before trying to take in data, I guess, if you're starting that process?

Yeah, I definitely think so. And I think it goes back to what we were talking about that if you're not doing that, you will have some kind of bias whether you realize it or not. And you will start looking at the data with this bias and basically you might be— you risk misinterpreting it. And I think actually not interpreting the data correctly or looking at wrong data because of that can be sometimes more dangerous than not using data at all. So again, it comes from making sure that you're cautious when you use the data and you do it in a way that's safe, so driven by an initial hypothesis.

Okay, well, thank you. I appreciate that. Well, Milan, this is— this has been awesome. This is a great conversation. I appreciate— I appreciate having you with us, so thank you very much. And yeah, I— we have plenty of industry interviews coming up soon, so for everybody else, I'll be letting you know about those. Thank you for being here. Also, and Milan, you might probably don't know about any of this happening. Everybody, I'm sorry. I know in the chat there's some weird things going on before, and I was trying to do my best to— I booted that person and tried to do my best to take care of it. And I, we, I think I had the settings set up so that you people had to register in advance and we had their information, but I'll be doing a much better job of trying to make sure we can avoid that in the future. So I'm sorry that that happened, but But I thought this was a great conversation and from the chat, everybody seems to really appreciate the convo too. So what I'm gonna do is I'm gonna email everybody once we have a video of this conversation up, which will just be in a little bit. And yeah, Nilan, I just really appreciate your time. Thank you so much for being here.

Thanks a lot, Mike.

All right, everybody have a great rest of the afternoon and I'll see you next time for another industry interview. Thank you.