EP 517 - Amplitude Fmr. VP of Product on Leveraging Data Literacy and Storytelling in Product Management

This podcast is sponsored by Mighty Capital, an early stage venture capital firm that capitalizes on the growing influence of product management in corporate success. Notable investments include Amplitude, Grok and Canela. Hi, I'm S.C. moadi, founder and Board Chair of Products that Count, the producer of this show since 2017 and a 501c3 nonprofit that helps everyone build great products and engages over 500,000 product managers worldwide. Products Account celebrates product excellence through coveted awards that honor great products and the professionals responsible for their success. It accelerates the career and rise to the C suite of over 30% of all product managers globally by providing exceptional programming including this award winning podcast for free. Learn more at Products that Count.org in every episode you'll learn the secrets of a great product leader. Let's meet today's host.

Welcome to Product Talk, the podcast from Products that Count. I'm your host Mark Bails, the Director of Technical Product Management at VNS Health and today I'm joined by Ibrahim Bashir. Ibrahim has been building and shipping software for over 20 years. He was most recently an executive at Amplitude where he served as the VP of Product. Before that he scaled new products at Box Service infrastructure at Twitter and the Kindle business at Amazon. Ibrahim also writes about the discipline of product management and craft of cross functional leadership. You can consume his latest musings via his substack newsletter. Run the Business Ibrahim welcome to Product Talk.

Thank you for having me. Mark. Excited to be here.

It's great to have you with me. Thanks for joining me. Today we're going to talk about data literacy in product and the kind of context that I want to set for our audience here is around an evolution that I think I've been able to observe and I think you have seen as well in your experience with Amplitude in particular. Most recently, originally there was a problem around getting access to data to inform some of the decision making that product managers need to make on a daily basis. More recently I hear many product leaders saying that actually there is an abundance of data and access is very easy in many cases. And actually the challenge is now making sense of the data in order to take effective actions. So we want to talk about whether it's possible to have too much of a good thing in this context. Ibrahim, if you wouldn't mind, can you just start by telling us a little bit around that evolution in terms of data availability and access?

Yeah, I would say, you know, the actual challenge has always been what you just articulated, which is making effective decisions and taking meaningful action around your product. I would say when people first approach that problem, the messiness of the data, the accessibility of the data, the availability of the data was the low hanging fruit was like the obvious non starter. So there was a lot of energy expended to solve that problem. And I think that problem has been solved to a significant degree. Right. Most companies at this point understand they need to track all the exhausts from their customer interactions and their user behavior and collect it in some sort of centralized warehouse or like. They understand they need to pipe that into other purpose built systems that can do deeper analysis. And they understand every function at the company, whether it's product design, research, marketing, sales, wants slightly different slices of that data because they care about slightly different versions of the customer journey. And so I think there's almost a proliferation of tools and data management applications to make sense out of the data. And so now that a lot of companies are past that, they're now struggling with how do I make sense of it? And I think the tricky part is the focus should always have been on how do you create a culture of database decision making. And I think now that the access question is out of the way, now people are struggling with that and now it's become, well, I have too much data.

Yeah, I think that's really interesting insight. And you mentioned this is almost a saturated market in terms of the access to data and the tools for some of this as well. And I think that's a great point. Would you say that data literacy has become an increasingly valuable skill for product managers as a result of this evolution?

It has and it hasn't. I would say data literacy is always going to be in vogue because if the job is essentially making judgment calls, right? And if you can get better at making judgment calls, whether that is through quantitative data or qualitative data or some combination, then you're going to be better at the job than not. It's better than just guessing. So I think data literacy matters. I think what's happened is a lot of the infrastructure and tool management around the decision making has become a substitute for data literacy. So when people say they're data literal, what they're saying is, oh, I know how to organize a taxonomy or I know how to do a query. And I'm like, those are elements of data literacy. But the height of data literacy to me is if I gave you some metrics around retention and if I gave you some pipeline analysis, if I gave you some user behavior cohorts, could you make reasonable guesses around what is worth doing next or how to slice the data next. Right. I'll give you a simple example. One of the most valuable skills that I think a really good product analyst or data scientist brings to a table is knowing how to slice the data to make sense out of it. And that's something the tool is not going to do. Or maybe with AI the tools are heading in that direction, but they just have a knack for knowing, well, you know, this metric might be up because there is a holiday right now, or this metric might be up because of promotion. And so they have an intuition about how to rotate the Rubik's Cube and look at it from a different angle to try to figure out where to go next. And I think that's what data literacy means to me is do you have that intuition? And sometimes that just comes from reps of looking at the same data set for a while. Right.

That makes a lot of sense. And I hear you in terms of the data literacy is consistent from qualitative analysis previously to more of the quantitative analysis now. Surely there's a slight difference though in terms of how those skills are applied. Right? Like if I think of qualitative as just doing observations or interviews and consuming that data and taking judgment on it, that is very different to some of the querying and taxonomy that you were referring to as well.

Yeah, I didn't mean to imply their equivalent skills. There is definitely different practice around each. For example, the qualitative, you'll see that, that being the purview much more of research teams and design teams and even PMs who tend to rely on that more. And quantitative comes more into play when you have hard data, when you are looking at things that are more data oriented, like raw metrics, like KPIs, like SLAs, or even things like retention and whatnot. But to me, blending those two things is the most powerful and you can use them in different ways. For example, one of the concepts I like to use is macro to micro. So you might from a high level customer conversation, get a hypothesis or an intuition at a macro level and then you go into the data to validate and explore that at a micro level. Right. Or on the flip side, you may be looking at metrics on a quarterly view or an annualized view and see a trend and then you go, may go talk to certain customers specifically about how that trend is manifesting in those users at a micro level. And so I think qual macro plus micro is a really, really powerful way to look at it. Because one of two things is going to happen. Either they're going to tell the same story, in which case you found something to build around and capitalize on or they're going to tell you different stories. And reconciling why you're hearing different things verbally from customers versus in the data is always an interesting thing for product teams to go look at.

I love that. And you started off just by saying qualitative is still huge in research, quantitative in other areas of product management. I was wondering if you can maybe go a little bit deeper if you can share kind of where you see qualitative research and data being used.

I have a framework I like to apply which is, I think there are, you know, let's just call data is more the raw fuel. What you're trying to get out of the data is what I call insights. Right. Insights are what ground your decision making and your product direction. And so I think there are really two types of insights. There are directional insights, like we should go after this segment versus this other segment. And then there are precision insights versus this has hit a certain threshold and now matters. And I think depending on the type of decision you're making. Right. Let's say the decision is we have a very successful first product and we have a couple of different options of what our next product could be. Directional insight is very valuable there. Right. And you can get directional insight from talking to your sales team, marketing analyst, customers, et cetera. Let's say you're running an experiment and you want to know whether this experiment should be rolled out to your entire user base. Statistical significance, Tension metrics. Precision matters a lot there. And one of the anti patterns I see in a lot of organizations is their blanket operating in directional metrics or their blanket operating in precision metrics. And you don't always need to be in one mode versus the other. It depends on the context and it also depends on the tool. You might have a tool that's a directionally useful tool and when you try to get precision out of it, you get frustrated. You might have an ultra precise tool that is really hard to wrap your head around. And when you give it to somebody who's just looking for directional context, it doesn't help. Right. And so I think you have to find the right tool for the job and the right context you're operating in. And that's one of the questions I always like to ask is when people start arguing, arguing about, well, is this data precise? I'm like, do we need precision?

Right. That's really interesting. Thanks for sharing that. I've not really Heard insights described in that way before, but that definitely thought provoking.

Yeah. And I would even say the market breaks out that way to a certain degree. I think if you think about very digital, native tech forward experimentation driven, everything is a structured release, constant metrics, measurement, scorecards, et cetera. Those types of companies like to operate around precision.

Yeah.

Which can be very, very good at operationally improving a flywheel and getting maximum value out of it. Right. If you think about more traditional companies where digital is a channel, they tend to think more directionally. And I'm painting with a broad brush here. But what can happen is because of the lack of precision based analytics that sometimes traditional companies use, they fail to capitalize on emerging trends in their user base. And because of the obsession with precision that a lot of digital first companies have, they failed to see obvious things that their customers customers are telling them, but it's not in the data per se.

Great insights. No bad use of the word insights in this context. But yeah, thanks for sharing. I think that's really interesting. Precision versus directional insights. So kind of switching ever so slightly into where product analytics should fall within a product organization. I do often hear a lot of focus on the product operations team primarily being responsible for product analytics, meaning the actual implementation of the technology and the development of the insights and then actually interpreting the insights and acting on them. So I wanted to ask, where does product analytics live in your opinion?

Yeah, before I answer that question, I just want to connect back to something I said earlier, which is, you know, everybody's procured some of these tools and they're hoping for change. Whenever you think about introducing a new way of operating in a company, there's usually a tooling component, whether you built it or bought it. Right. And then an operating or ritual component, which is, are we going to change our practices around this? And so I think a lot of the frustration with lack of ROI on some of these tools is people buy the tool and then they don't really go through the change management exercise and they're like, well, nothing happened. My life didn't change. So to bring it back to your product operations question, it's really interesting. I have a lot of empathy for my product operations friends because a lot of times they're caught in this tug of war between how the PM's on the ground want to operate and how product leadership and executive leadership perceives best practices to be implemented and ProductOps gets put into we bought a tool, we're not operating the way the tool requires us to operate. Change the culture of the org. And I think that's a complete abdication of responsibility from the product leadership team to put that on Product Ops. In an ideal world, Product Ops should say this is how we operate. Can we introduce tooling or process to enhance that? Or I see as deficient in this way. Let me raise a conversation bidirectionally. But what's starting to happen more and more, let's just to paint an example is there's an org that is completely shoot from the hip, no data in their decision making. New chief Product officer comes in, says we're going to be data driven. They buy a tool off the shelf. They tell the Product Ops team, you have six months to implement this. And then the Product Ops team now has to be the bad guy trying to convince people the way you've been building software is incorrect. I need you to do 50 things that you're not even understood, why you should do them or how to do them well. And if you don't do that, then you're just going to constantly get reamed and called out in every meeting. Right. And so I don't know if I'm actually answering your question of where should it live. I think there is a version that can live under productOps. But I think the more important question to me is how do you create a culture where using data to make decisions is the norm and celebrated and ProductOps is a partner in doing that? But I don't think Product Ops should be fully on the hook. The simplest example I can give you is the number of times I've seen people say, well, Product Ops is going to make sure we have a Monday morning meeting with a dashboard and the dashboard should be filled out and there's a spreadsheet. And then the executives themselves have not really internalized what it means to use data to make decisions. That's just a silly situation, right? It's like you're preaching something but you're not doing it yourself. And so I think you have to lead from the top. And I think people have to see that culture. And some of the ways you can actually do that, speaking as a former product leader, is to actually hold the line is to say, hey, when somebody comes to you with a product idea instead of just green lighting and you say, hey, I want to understand the user research and what the metrics are saying. When somebody says, let's end of life this feature, it's too costly. It's like, well, let's look at the ROI of this feature before we do that. Right? Yeah, before Somebody says, hey, let's do a marketing campaign around this thing we launched. It's like, let's make sure it's actually adding customer value. And so when you start to introduce that into the vocabulary as a leader, then people go to product ops and say, hey, I'm not going to be able to get this through without having data and a story around it, so can you help me?

And that then presents the case for the procurement of a tool of this type.

Right, exactly. And so I think people start with procurement and mail in the change management and I think you should start with the change management and the tool is secondary.

It's almost like when we take market products to market, sorry, in B2B world, and we speak with the customers to help understand their operational readiness for a software tool within their organization, we make sure that they have processed built in around the procurement of the tool that we are selling to them. It's almost like we need to look internally at our product or to determine our own operational readiness for a product analytics tool.

Yeah, I mean, just speaking as somebody who wants their software not only to be bought, but successfully deployed and renewed and then referenced, you want to make sure the change management is part of it, whether it comes through documentation, training, professional services, a partner. The other interesting thing I was going to mention was there are a lot of ideas in analytics and data science thinking that can be applied to qualitative insights. What I mean by that is you gave the example of B2B products. You know, a lot of times companies will say, I have 10 customers who want this new product, we should go build it. But we all know the reality is if you are a successful B2B product company, your customer base is not homogeneous. There are different cohorts and segments. There's early adopters, there's people who've barely deployed, there are enterprises, there's mid market. And so anybody can find any list of customers to make a case to do something. But the right question to ask is, are we building something for a legacy cohort that's fading out of our segment? We're building something for an emerging cohort. Are we building something squarely in the middle that's going to be heavily utilized? Right. And so that's a way you, without getting very data precise about it, is you can sort of paint a picture qualitatively and say, look, the feedback I'm getting is from the customer base we want to bet on. Right. And so I think applying those kinds of ideas can be very, very useful in any kind of Research quality as well.

Thanks for sharing. It's actually a very nice segue to kind of the next point I want to discuss with you, which is around, like, the interpretation of data and statistics. Because, I mean, particularly in sports, if we just look at data and we see the data and it says, somebody had a great game, right? Or the fan that's watching it in the stand says, no, they had a nightmare on that specific day. The data can be interpreted in many different ways, essentially. And two that I am very familiar with. The recency bias. Confirmation bias. Right. Just two examples of things that I am always very, very aware of when I'm trying to interpret the data. What are some of the other challenges that product managers could expect to face, and how do they mitigate some of those potential biases?

Yeah, I would say, you know, one of the first things that comes to mind is anytime you're trying to interpret data or a metric or a spike or a dip, that's usually what happens there. There's a baseline and then there's a shift up or down, and people want an explanation of what's going on. In my experience, all the hoopla is for nothing because usually there's two reasons why it's dipped or spiked data error. Right? Like something's gone wrong in your pipeline or seasonality. Right. There's a shift in holiday behavior or you did some sort of promotion. And so those are two of the things I try to do before I get excited in any direction is just eliminate the obvious stuff. Right? And that's why it's actually really, really important to have good monitoring around your data pipeline to get ahead of those things. Right. Like, if you found out you stopped recording an event in the metrics review, that's the wrong place to capture that. Right. So that's the first thing I would say, is just do the basics. The second thing I would say is it actually takes a lot of reps to understand the life cycle of a metric and get familiar with it. Right. It's kind of like, I'll just use the example of, like, you get a new car until you've been driving that car for a few weeks, you don't really have a sense of, like, how does the acceleration feel? How is the turning radius? You kind of need to, like, get comfortable in the seat before you can actually drive it fully. Right. And so I think a lot of times people are trying to interpret metrics, and I'm like, how long have you been tracking this metric? And they're like, oh, a week. And I'M like, that's. You're making big conclusions from small sample size. Right. You almost need to go through a full whatever your life cycle is of that metric to get a feel for it. And then you start to understand how it tends to spike and dip. Right? So, for example, one of my former bosses who owned a lot of the metrics before I came in, when I came in, I would react to every blip on the dashboard and he would always tell me things like, oh, you know, Q2 is always hot because of this. Or our customers customer base is heavily media centric. So when the Olympics happen every four years, this tends to spine. And I'm like, how could. You can't know these things until you've lived it for some amount of time. Right, right. And that's why I'm actually pretty excited about this kind of interpretation as the place for AI to help. A lot of people are excited about AI to do basic things like, oh, can you translate words into queries for me? And I'm like, those are efficiency shortcuts. But the really interesting thing is for it to be like, look, I've observed all the analytical expertise in this organization. I understand the entire corpus and knowledge base of how you guys look at metrics, and I can apply that in an instant to anything you're looking at is really powerful. So one example I'd give you is, you know, with my former product team, this would happen at least once a week. They would email me or slack me a chart and say, oh my God, look at this. And I would stare at the chart and I'm like, what is it that I'm looking at? I don't see it. And what was happening there was not that I'm dense, but they're so deep in the metric, right? The consumption metric, the activity metric, they look at it literally five times a day that any shift they can perceive. And I'm just look. I look at it once a month and I'm like, I'm not seeing it. But then there are metrics that I look at like pipeline or renewals or whatnot, where I'm like, I can tell anything before it's happened. Right. And so it's a little bit like animals can feel earthquakes. And it's like they're attuned on a different level. And so I think that's the thing I think about interpretation is how do you get people attuned to the vicissitudes of the metric as quickly as possible?

I think that's a great use of AI I've never kind of thought about it that way. And interesting, because when you started off here, I was thinking to myself, that's you kind of gain that in any way other than through experience. And yet you followed that on with saying, this is a potential use case for AI. Right. So it is.

Because I'll tell you, I mean, like I said, I think a lot of the AI energy right now is around automating workflow, which is, which has value. It's a productivity use case. But you know, one of the smartest things I did was when we had a data scientist looking at our metrics, I would just shadow the guy. I'd be like, hey look, Monday morning as you're preparing for the meeting, can I just sit with you and understand? And he would just be doing stuff. And I'm like, hold on, hold on. Why did you segment that way? And he's like, oh, because this is always. And I'm like, oh. And then he's like, you know, double clicking. I'm like, why do you know to double click there? And as he would sort of do his thing, I would take notes.

Yeah.

And then by the end of it, I had like a 10 page doc of like, this is how this guy's brain works. If I could feed that into an AI, I have data scientists in a box, Right. And so that I'm very excited about that possibility at some point.

That's incredible. And then good idea, good advice for others who do have the luxury of data scientists there to shadow them as well, I think. Very practical tip as well.

One more thing on interpretation, which is at some point not all interpretations carry the same weight, right? So for example, hey, look, we're trying to guess whether this trend will continue and whether we should roll this out. It could be a simple two way door. It's like, let's say you move forward and it turns out to not be the case. You can roll it back easily versus should this be our next product or should that be our next product? That's a very hard decision to unwind. And so the thing I would say is one is interpretation is fundamentally not a precise science. But not all interpretation decisions are heavyweight decisions. And so this is purely my approach, but it served me well is I tend not to overthink simple decisions even when the data is not super obvious. Because I'm like, we can backtrack easily here, but when it's a major decision, I want to look at qualitative data, I want to look at quantitative data, I want to talk to all the people who help pull the data together. So as an example, I don't want to be on the receiving end of a report that says, hey, we're going to build this thing sales set. Right. I go talk to the top salespeople, I go talk to the customer sales. Because I'm like something gets lost in translation. And so the biggest point I'm trying to make is based on the heft of the decision. Make sure you even check how the interpretation work was done.

Great advice. I think that makes a lot of sense. Very practical as well. You mentioned around kind of using data for non hefty decisions essentially. Right. And there's an element of intuition there as well. Have you got examples where data supports or maybe conflicts with some of your intuition that you could share?

Yeah, absolutely. I would say the example I was giving before of you hear something from a customer but the data doesn't show it, or you see something in the data but you've never heard it from your customers. There's a lot of reasons for that. So I'll speak in broad strokes.

Sure.

On one hand, if you're getting access to customers who want to talk to you, more than likely they are fans and advocates and power users. And so one of the things I've seen happen again and again is the product team is talking to customers and users. They're doing the basics, but they're talking to one cohort over time that is just very familiar with their product. And the data is showing you the entire universe. Right. And so the data is showing that people are struggling to activate, struggling to onboard. But the power, like I love this product, they're raving about it. That's a very common pattern. And so one of the things I always challenge the team on is can we mix it up? Can we make sure we are bringing in new users and new customers and not talking to the same captive audience again and again and again. Right. That's number one.

Yeah.

Number two is, you know, the life cycle from idea to released can be quite long. Especially if you're working on B2B products like I have for the last seven, eight years. One thing I'd say is sometimes the market moves fast. Right. And so if you, if you saw a trend and you started building around it and now it's six months later, you might be releasing at a time when that trend is on the downward side versus on the upward swing. And that can be painful for your go to market team to then try to capitalize because they're steering into a skid versus going up the hill. So that's another one. And then the last thing I would say in terms of interesting situations where the data and the feedback don't qualify is my view is people in product leadership are there because they have good pattern matching ability. And so that's one of the things I look for, is like, what do I just know to be true about user behavior and buyer behavior over time? That doesn't feel right. So a simple example I can give you is I fundamentally believe that a fast workflow in your product matters. It matters. I can't tell you how much it matters, but I can tell you it matters because I have been on the receiving end of enough customer fees feedback, where eventually they say, I love this roadmap. I love that you guys are innovating, I love that you ship fast, but the product is so slow, none of it matters.

Yeah.

And inevitably there's a reset on the product side where they're like, oh my God, how can we let this happen? Quality has fallen by the wayside. We need to have monitoring and instrumentation and blah, blah, blah. How does that happen? It happens because you look at those metrics at a blended user level. So you're like, the average user is having a good experience. But the tricky part, especially in SaaS, is the heaviest consumers, the most powerful users tend to do the most customization, tend to have the most data, tend to have the most users, and so their experience actually drags the most. The data is not going to tell you that or unless you're looking at the data in a very sophisticated way. And so that's one example I've had happen to me multiple times is I'm like, all the metrics look healthy, all the thresholds look healthy. We're within sla, but our most valuable customers are extremely upset with us around quality, around performance along reliability. And so those are some examples of where there's a dissonance. But now I've even started keeping a list of those common dissonance scenarios to try to get ahead of them.

Thanks for sharing. I think they're great examples. Many people can probably relate to listening to this as well. You've mentioned in a couple of your talking points here. Firstly, you mentioned data errors, which very common, very, very common. Huge challenge for our data teams, even when putting the data together for us as well. You also mentioned about macro and micro. I want to kind of switch this and maybe use those two examples as an example of data quality and completeness as well. And you started speaking about looking at Blended rather, or segments specifically. I think it's one of the pragmatic product management foundations is that you hear from your noisy 20% but not from your quiet 80%. Right. And the fact that you actually get a response from 20% of your users, but you don't get a response and therefore don't have Any data from 80% of your users potentially puts you at risk for building for that 20%. So are there actually insights that you can take from missing data is an indicator of poor quality data?

Absolutely. I think you know, one thing about missing data and one anti pattern that I see people falling into is this desire for comprehensiveness and completeness. They're like, I have to track everything, I have to measure everything. Until I do, the system is not ready. And my question is, do 100% of your users use 100% of your features? Absolutely not. Right. So I actually did a North Star workshop with a product leadership one time that was. They were completely wrapped around the axle and like, we don't know what the metrics should be and we don't know what the key workflows are and the critical user journeys. And they were essentially a food delivery business. I did a simple exercise with them. I said, imagine your company is having the worst outage in the history of the company. Everything is offline. They're like, oh, my goodness. And I was like, you go to your engineering and your reliability team and your, you know, support team and you say, we got to bring things up. And they say, well, you can't turn it all back on. We got to start with the P0 stuff stuff. And they were like, yeah, that makes sense. I was like, what is the P0 stuff? And they said, well, we ingest the catalog of what you can order from our partners. You gotta be able to browse and add to your basket checkout and then some sort of delivery estimate and confirmation. That's P0, right? It's like, great. That's it. Nothing else matters. Nothing else matters. I didn't hear gifting, I didn't hear installment payment. I didn't hear any of that. I just heard five concrete steps that are your core workflow that generates 80% of your revenue. They were like, yes. I was like, that is what you should instrument. That is what you should monitor. That is the customer base you should talk to regularly is the people who flow through that pipeline. Because if you just really, really make that simple and beautiful and fast and repeatable, you have a solid business. Right? And so I think all this energy that's spent on, oh, we're introducing this new product and do we have instrumentation around it? And I'm like, like zero people use this thing. Like, why are we obsessing about it? So that's one. The second is, and I'll acknowledge there's no silver bullet here is every metric, every North Star, every data driven approach you're going to take has this tension between it's really simple to understand, but it might be meaningless and it's really meaningful, but it's like too hard to wrap your head around.

Yeah, right, Yep.

And so you can have like, hey, we have a metric on a dashboard that everybody at the company can look at. But it just kind of like we don't really understand why it moves up. Right. It's so simple. It's like activity, it's a good thing. And I'm like, is it? Is it? Do you know that it really matters? On the other hand is, you know, we have users who are doing X features four days a week within a blah, blah, blah. And I'm like, okay, nobody can like query this metric other than five people at your company.

Yeah.

And so there is always a tension. And this goes back to what kind of mode is your company in? Are you in the mode of we're trying to build a muscle of getting data literate and making decisions and checking in on that metric. Then go simpler. If you're in the mode of we have a handle on it and we need to like, we're making 50, 50 decisions all the time and they need to be right more than not on balance, then go towards precision. Right. And so that's something I wanted to acknowledge is, is that spectrum of simple to complex metrics and why some companies land one way versus the other. The last thing I'll say on that is to me, product management at the heart of it is about choice. It is about choosing where to focus and spend energy and R and D capital versus not. Right. Like you said, where are you not focusing? And so it's very, very important to say, like, if we're spending something, if we're spending time building something for users that don't exist for a market that isn't there, for a problem that is invalidated, for a solution that has no budget, those are important things to validate as well. On the negative side, Right?

Yeah.

That is the most wasteful thing a product person can do is to build something for which there is no utility or value or revenue for the company.

Completely agree. And it sounds so simple when you put it as Black and white as that, actually. But many, many of the time I think we do over complicate things and lose sight of that North Star metric.

Yeah. And you know, people don't do it maliciously. They do it in a very well intentioned way. And I think this, you know, topic for a future podcast, I think that happens because the PM's on the ground don't really understand and haven't really digested what the strategy of the company is. Right. They're off saying, customers are telling me to build feature X, I'm building feature X. And you said you wanted it grounded in research and data. I haven't, I have research and data. But at the top, the leadership has realized customer base X is fading. They don't have buying power, we need to go to Y. And so then there's this tension of people are like, I did all the things I did, all the steps. I wrote the prd, it has data. Why are you not green lighting my thing? And leadership is like, well, I just don't feel like that's the right place to do because they're getting different data and they have different pattern matching from people in the buying position. Right. And so I think it is on PMs to ask leadership to write down and disseminate strategy in a way that they can interpret. And it's on leadership to actually do that regularly because the market changes very rapidly. Right. So you don't want to be optimizing your metrics and your decisions around a segment that just doesn't matter anymore.

Aligning your product strategy to the business strategy. Right. If they are not even the same, some small companies, the product strategy is the business strategy. Right. And some of the larger companies, you might have multiple products and then there's.

Even more pressure to make sure it's aligned. I mean, like, so we're not talking abstractly. One really concrete example I can give you. Every company with any kind of digital presence has some sort of change in behavior and boost during COVID And then they all started building like that was the new normal. And then now I have seen for dozens of accounts that I've talked to that, oh, we had this Covid spike. And then that behavior did not continue. But can you imagine, you know, for 18 months people are like, well, this is the new normal I need to build around. And then now they've built a bunch of stuff where those users don't exist, that behavior doesn't exist. That's not the workflow anymore.

Great example. I think that summarizes that Section really nicely. Thanks again. We've spoken about how data literacy is not new. It is still valuable, always has been valuable, always will be valuable in terms of the PM and the choices that they have to make. To use your words earlier, if I was to call data literacy a superpower, I think within some of the marvel situations, you can see that superpowers can be used for good or evil. For example, in the business context, do you think that data and data literacy can be weaponized?

Well, absolutely. Let me answer that part of the question and then I want to go back to the superpowers. It can absolutely be weaponized. I mean, we were talking about it earlier. Once everybody figures out the game, which is, you know, I have to show X amount of change in metric by building this feature and Z number of customers who have said they want it, that almost becomes a rubber stamp process, right? And, you know, leadership once again sort of takes its eye off the road by saying like, yeah, the, you know, all the boxes are being checked. But it's like, but if everybody knows that, then how are you actually choosing what to do and not to do? Because every single feature will have some data around it and some qualitative feedback around it. So data can absolutely be weaponized. Two is, you know, I forget who said the quote, but it's something along the lines of if you torture the data enough, it will confess to anything, right? And so you can look at a large enough data set or slice it in a particular way to tell any kind of story, right? And so I absolutely think it can be weaponized. And then three, and this is, is, I think, very insidious. One of the ways it can be weaponized is by not having it. So somebody stumbles onto something that is a genuine real thing that the company should pursue. But it's very early, there's no data around it, and somebody says, hey, I have a sense that this is where we should go, right? And the leadership team, either because they don't want to bet their careers on it, or they're just playing out of fear, right? Say, well, you don't have data, so we're not going to do this until data which creates this vicious cycle of people are like, let me go get some data. And then it's like, well, it's not enough data, right? I don't really trust this data. And I think, you know, that's a disservice to people doing genuine work. And I think if as a leader you're not bought in on something, you should just say, I Just I'm not bought in. And at some point I get to make this decision, even if it's purely emotional or a judgment call. So I think it can be weaponized even when it doesn't exist. Now I want to go back to the superpower. Is data literacy a superpower? Yes. But I think there's a larger superpower, which is storytelling. That is the ultimate superpower. If you want to be an excellent PM in any organization, storytelling matters. I think the nuance comes in. In certain organization, storytelling is about presentation skills. In certain organizations, storytelling is about utilizing data. And in certain organizations, storytelling is about using customer anecdotes, or storytelling is about using internal influencers. You know, I coach and mentor a lot of heads of product, and one of the most common things people come to me frustrated by is they're like, I really feel like we should do X. And the CEO won't listen to me. And they're just like, you know, I'm just very Fred up and frustrated.

Yep.

And I always ask them, who does he or she listen to? Who do they listen to? And inevitably there's a person, there's a whisperer.

Right.

They're like, well, you know, there's a co founder or there's this chief of staff. And I'm like, well, have you tried telling the story to them? And how did they absorb information and stories? And usually what ends up happening is you're not using the right channel and you're not using the right medium. So it's like, hey, I made a presentation and I emailed it to my CEO's chief of staff. And I'm like, okay, okay. Like, is she going to understand an async presentation? Right. And so that, that's. I think a very, very important thing is storytelling is the superpower. But you have to tell the story in the format that the audience can receive it and in the channel where they congregate.

I love that. I think that's very true. I think storytelling has more recently come into one of those top soft skills for executive leaders. Actually, it's a huge, huge superpower. I would agree with.

Yeah. And, you know, just as somebody who grew up on the PM ladder from senior PM to, you know, VP, I will say sometimes there is frustration between PMs as peers because one person is like, I did all the deep dive. I did the data, I did the research, and my idea is not getting greenlit. And this other person barely checked the box. Their thing is like, like, full steam ahead. I don't get it. It's unfair. It doesn't make sense. We don't actually follow the playbook. But really what's happening is one person is excellent at storytelling and there's less pressure on them to prove it, and then the other person is deficient at storytelling and they're trying to overcompensate with data.

Yeah, I like to say that in a slightly different way of actually sometimes the presentation is as if not more important than the content, actually in a lot of cases.

And that goes back to, I think so much of the focus around insights is around the data and the hygiene and the pipeline and accessibility. But I think the real focus should be on what does this mean, how should we interpret it and what action are we taking. Right. And so I think there's a higher order level of operation as an org that we want to operate at around data.

I really like kind of how you tied all of that back to the data piece at the end. Because data alone doesn't tell the story. The interpretation of the insights enables a story to be told. I think so. Really nice way to kind of tie that out.

Absolutely.

So, Ibrahim, we've spoken a lot around how data literacy is not necessarily a new skill or a new need within PM world. We have spoken about how data availability, accessibility has grown and evolved over time within the PM discipline. We've spoken through many great examples. Again, I'll thank you for sharing some concrete examples in many different scenarios here of how data can be used to tell a powerful story and make good decisions. I want to kind of give you the final word here. What would you like to leave our listeners with with regards to data literacy and product?

Yeah, I just want to offer some empathy for all your listeners. I'm sure there are PMs, there's product leaders, there's non product people. I would say the empathy I have is, you're right. Data literacy has always been a skill and a discipline that's been around for a while. And tracking data around your customers and their usage is not new. What is new and what keeps jumping by leaps and bounds is the volume of data, the dimensionality of the data, the ability to connect disparate data sets, and the almost real time speed at which the data is coming, coming in. So you know, if you rewind the clock, it's like people are like, yeah, I was dealt with data, but I would get it once a week in batch, right around one dimension and I would just have that data set. Now people are like, I'm getting data every second right. In enormous volumes. Such that the tools can't keep up. Right. I'm able to then cross reference that data around user behavior with my CRM, with my data warehouse, with some third party attribution data that I bought with ads, with anonymized, de. Anonymized, authenticated. And there's just so many breadcrumbs to make sense out of. And so it starts to quickly become like a insurmountable problem because you're like, there's so many ways I could try to reason about this. Right. And so I really empathize because the data's exploded, the tools have gotten better, but then the expectations are out of control. Right. And so I go back to if you can get really good at knowing what is worth exploring and slicing and interpreting and what to ignore, it'll be really well served for your product career.

I appreciate your time and your insights. Thanks for joining me today.

Yeah, absolutely. Thanks for having me.

Mark this is SC Moadi again, founder and chair of Products that Count. Thank you for listening to this episode. If you like what you heard, be sure to leave us a review on your favorite platform. If you including Spotify and Apple. And please check out our other resources at productsthatcount. Org. They're designed to accelerate your product career and make product the most important function in business. Until soon.