Product Management Metrics w/ Brian Crofts, Chief Product Officer of Pendo

Hey everybody, this is Mike Belsito, co-founder of Product Collective, and we're back for another industry interview session. Thank you very much for being here with us. I'm excited about this because I think this is a topic that is just always on the mind of product people. You know, we're always wanting to be data-driven, but are we doing the right things? I was just talking with our guest about that, and this is one of our most RSVP'd sessions. So I think all of you seem to be excited about the content as well and the topic at hand. So I want to get into it. But first, just a couple of housekeeping things. If this is your first Industry Interview, welcome. We're glad to have you with us. I want to just kind of point out a couple of things first. We at Product Collective, we do these Industry Interviews. We also have our newsletter. We have an active Slack channel with a lot of product people. And we also put on a couple conferences every year called Industry. So that's what we do at Product Collective. These are particularly fun because we get to sit down with product leaders from really all over the world. In fact, our guest today is, is coming in from Israel. So that's a lot of fun. It's also a chance for you to get to interact with them. So you probably can see a couple of things. You could see chat and you can see a section called Ask a Question. I would love it if you'd introduce yourself. Actually, now, if you could see and hear us okay, maybe you could say hi in the chat and just let us know where you're tuning in from and, and maybe what kind of product you're working on. That's always, always fun to see. But if you have questions for our guest today, I really want to direct you to the Ask a Question section. So that Ask a Question section, that's where you could submit a question, and I'll be checking up on that really throughout the entire interview. This will go until 1:30 Eastern, so it's about half an hour. And I love— I have questions teed up, but I love getting to bring in your questions. In fact, I'd rather bring in most of your questions. So be sure to list your questions there. Some folks ask if maybe they have to jump off, you know, hey, is this going to be recorded? It will be. In fact, you could come right back to this URL right after the session and you should be able to watch the recording. But we'll also follow up with an email and I'll give you a link to that afterwards. So that's just a couple things to keep in mind about these sessions. And if you're new, again, if this is the first time, I definitely encourage you to sign up at productcollective.com so you can take advantage of all the resources. We can invite you to upcoming sessions like this. We've got some awesome ones coming up with Bob Mesta. Architect of the Jobs to Be Done framework, with Scott Belsky, who actually just wrote a book a few months back called The Messy Middle, Dear Jack Bauer of The Skimm. So there's, there's a lot of these sessions that you could sign up for. We do these usually the first and third Wednesday of every month. So that is fun. And then again, you could check out industryconference.com and you could, you could keep tabs on what we're doing with industry. We have our European conference in April, and then we have our global edition where we'll probably have about 1,200 people with us in Cleveland in the fall. And that Jason Fried of Basecamp will be there. Grammy and Academy Award-winning artist Common will be there. And our guest is from Pendo. Pendo is our presenting sponsor for both of those conferences. So we're really grateful for that. And actually speaking of Pendo, Product Collective, we collaborated with Pendo on something. Actually, we did it last year and we doing it again in 2019. We did a study with product leaders and we put out this report, or about to put out this report next week, called the State of Product Leadership. And I'm going to be co-hosting a webinar with Jake from the product team, or from the Pendo team, on February 21st. And we're going to be going over our findings. There should be a little call to action button that'll lead you right to the webinar page. So if you, um, want to check that out and listen in on the webinar that we have planned for that, that'd be great. Or if you sign up at Product Collective, I'll be emailing you next week when the report's out. So we'll definitely email you the report we put together, nice ebook with Pendo on that. There's a lot of pretty insightful learnings on that. So definitely check that out too. But with all of that, I, you know, now that we have sort of housekeeping out of the way, I want to get into the topic at hand, and that is metrics for product people. And To discuss that, we have Brian Cross, who's the Chief Product Officer at Pendo. And thanks a lot for joining us, Brian.

Yeah, it's great to be here.

Well, I know that this is a topic, first of all, that you've written about in the past. I know this is something that you, you know, sort of near and dear to your heart. And it is something that definitely is of interest to me because I find my— I found myself in the past as a product person where, again, people are leaning on me to be tracking things and we should be data-driven, but Again, it's just like, how do you know what to be tracking or the things that we are tracking? Are these the right things? And I guess just to start off our conversation, maybe I could just ask you, as product people, how can we figure out what the right metrics actually are? And I, you know, maybe it depends on the types of businesses we have or the products we're managing, but maybe we could just start there.

Yeah, it's a great, great place to start. And again, thanks for having me on. I think this is a great topic of conversation. It's fun to see people from all over the world joining in— California, India, Seattle. So it's great, great to have all of you guys on board as well. And definitely I welcome questions. It looks like there's already a handful. So, you know, I— there was actually already a question. I was looking while you were talking and there was a question about, you know, what kind of framework do you use to be able to know what you should measure and how you should measure it? There's just a couple of really simple ones that I like to follow. And really it's two, which is first and foremost, you know, are you a B2B app or are you a B2C app? And there is some major distinctions between the two. If you're B2C, you're talking much more about a linear experience, you know, so everything is about conversion funnels. It's about how do you optimize the perfect path, the path of least resistance, the how do you reduce friction. And typically when you're B2C, the volume of data, the volume of users is much more significant. So you can get to statistical significance really quick and will change the way that you operate almost on a daily or weekly basis. The idea of experiments comes into play a lot. And so B2C is just fundamentally different than B2B. And so if you're B2B, you're thinking a lot more about retention and engagement and active use and sentiment. And because, you know, for B2B, these are apps that they're spending, they maybe rely on to do their job. And they, you know, so how active they are, how engaged they are is really, really important. So I first, you know, just really calling out there's a major difference between the two and it's important to understand. And then second, the kind of the other framework, and you mentioned jobs to be done, you know, this is another lens that we have as well, which is, you know, first try to understand that the job that you're trying to solve. So for example, if we're talking B2B and I'm trying to improve engagement, I need to understand certain metrics that are in engagement. And if I'm focused on sentiment, then I need a little bit more qual and quant. If I'm focusing on learning, you know, how to improve onboarding, there's a certain set of metrics that I want to look and measure there. So that's the two frameworks I, you know, number one, B2B versus B2C, and two, you know, what questions do you have? What job are you trying to solve? And there's a series of metrics for each one of those.

That's awesome. And yeah, for those of you that aren't as familiar with Jobs to Be Done. It's become so popular with product people, so probably everybody here is, but we do have that upcoming session with Bob Mesta, so that might be helpful to check out. Somebody did ask, you know, and this, this would be one of them, like going through the whole Jobs to Be Done process, but some of them— some— somebody had asked, what are— are there exercises that you go through to try to figure out metrics? So that might be one of them. Are there, are there other sort of exercises or sort of things that you're doing to try to, you know, get to a point where you're like, okay, this is an important metric for us to be tracking?

Yeah, absolutely. So I think, you know, one of the most important sort of tactics that I have with my team or with any team that I'm working with is just, you know, so number one, just understanding that sometimes when you say metrics and you need actionable metrics and you need to measure this, measure that, you need to be data-driven. It can sometimes be overwhelming. And so the number one thing that I try to start with is just asking questions like, what are we curious about? For example, like, do we know the adoption of our product or our feature when we launch it? So for example, you know, we spend, if you think about it, we spend tons of time and money and our most precious resource, which is engineering, all to build maybe a new feature or a new set of functionality, and then we don't know how to measure if it's successful or not, or we don't know, we wonder what the adoption would be or what the adoption should be. Or, and so I think just to keep it really simple, it's what questions do you have, what do you wonder about? And then I think the, this kind of maybe a higher level activity is, is really going through and, and being clear about What is the true benefit of your product? What's the value of your product? And then how do you measure things like speed to value or time to benefit? And then, you know, after that, it's like, well, how do you actually measure you're achieving that benefit? And so I think, you know, like there's a lot of good ways to get started, but that's the one that comes to mind, you know, first and foremost is just to like sit down with your team and brainstorm a bunch of questions. And then you can say, well, how do we, now how do we go get these? Versus I think, look, I'm in the world of building tools for product people and around analytics. And so obviously we want people to use these tools 'cause they're super important. But like I always ask people to step back and just ask questions. And you know, you don't need to think right now, is it a leading indicator? Is it lagging? Is it actionable? Is it not actionable? Just what questions do you have and start there. I find that to be the most helpful. And what's surprising is people have a lot of questions and they wonder about a lot of things. And the good thing is in today's world versus when I started product management, it's really easy to answer these questions.

That's great. Yeah. And that, that does sound like that'd be certainly useful as a first step. I'm curious, once you at least have a hypothesis of, okay, these are, these are the right metrics, like let's start tracking these. How widespread, you know, within the organization should they be shared? I mean, I've been, I've been some companies where I walk in the door and, you know, there's a big screen and it's got all sorts of dashboard-type things, which some might be company things, but some are actually product. Then I've been to some companies where, you know, I've asked them about metrics and who else in the organization has them. Oh, we can't share them with the marketing team. Like, we don't want the marketing team to know everything. Why don't you want the marketing team to know? So I'm just curious, like, as product people, how, you know, should we be sharing these with the rest of the product org, broader than that? Like, where does it sort of, where does it go?

Yeah, no, it's a great question. So I think just a couple of statements or thoughts. Number one is, you know, as organizations, as departments, we need to be more transparent. Like, you know, first and foremost, we need to be able to share what's working, what's not working well. And so if you're, you know, if you're like worried about showing what's happening in your department, I think that's a bigger issue. It might be a cultural issue, it might be a personal issue, but number one, we've gotta be transparent, we've gotta share this stuff. And then I think, you know, second is, you know, I think one of the things that as product management is evolving and is becoming, you know, as like we like to say, product is the new marketing. Like we kind of owe and have a big responsibility, I think, within our organization to help educate the rest of the organization about what to measure and how important product measures are and how important it is to analyze all these different things that we'll talk about. And, and again, like one of the things that I always struggle with is, you know, sales, marketing, that is, that's that's something that's been going on for a long time. And so we know how to measure those, we know the questions to ask, but we don't on product. And sometimes us as product people, we don't know what to ask. And so if you are measuring and you, whether it's good or bad, if you are measuring it, it's really important to share. And, you know, as you said in the beginning, like it's hard to improve something you don't measure. And so it's just the best practice that that, you know, to share and to be open and to get it on your Looker dashboards or whatever your company uses to go over metrics. Like, have it on there. Take a seat at the table in terms of metrics and start to continue to show the value that product management is to the organization. And we can't just be this organization that says, "You don't get it. It's, you know, it's— there's a lot of things that happen. You know, we're working with engineering. We're doing all— it's hard to measure." And we just need to move, like, if we really want to establish ourselves as a legitimate function, which again, we have been over the years, like, we need to bring metrics and we need to run our department like the rest of the business does. And so I'm super passionate about it, of course. But yeah, I think there's an education aspect and then there's a doing aspect and we need to see more of it.

That's great. And by the way, just as a reminder, everybody, I have several more questions here, but keep loading up that Ask a Question section because I definitely want to get to more of your questions. For one of mine though, right now, you know, you mentioned before, you know, the comment you can't improve what you aren't measuring, right? So, you know, part of us tracking these metrics is to figure out a gauge of whether things are working. And if things aren't working, You want to improve at it. Is there— what's your take on, you know, improving at things that we see that are starting to not work versus over— you know, I also don't want to overreact. I don't want to see like, oh man, an indicator's showing that this is going down. How do you know when to act on something versus not overreact?

Yeah, well, I think, you know, kind of going back to some of the earlier threads, like, number one, like, I don't think we're I don't think we're yet in jeopardy of overreacting. Like, I think it's first, like, let's act, you know. And if we start to get to that place where we're overreacting, we can correct ourselves. Now, what I think you might be referring to is if, you know, if there's some kind of, you know, false positive or some indic— you know, indicator that it's successful or something's working, but it's really not. Like, anything like that, that obviously can lend itself to overreaction. But, but if we can just start acting more and, and even start to cross that line of overreacting, I think I'm totally cool with that. But I think, you know, it goes back to some of the fundamentals that we were talking about, which is, you know, what are the questions that you have? As you start to answer them and start to pull together data, that then leads to more questions, and you start to go on this path, and you start to get more curious and ask more questions, and you start to find yourself in a place where, you know, maybe, maybe you don't know what to do at that point. But I think, you know, one of the tactics that we've used is, you know, whatever metric we might be looking into or following, you know, one test that we always do, or quick brainstorm, is let's come up with 5 or 6 things that we can do right now today to try to improve this. And so just like anything, you know, we're as product people, we're super comfortable brainstorming and coming up with those next steps. And for me, like the best thing is as like a product leader working with product managers, the best thing for that I see, like when I kind of dub you as like a, you know, an amazing product manager is when you come to me and say, all right, we had this goal. For this feature launch. We launched the product, and what we've noticed is this segment adopted it more than this segment, but overall we're lower than what we anticipated. And so what I did is I reached out and engaged with the customers, whether it's in-app or outside of app, and I asked a precise question based on some unusual behavior that I saw from certain individuals or certain users. And then once I got back that feedback, I was able to come up with a few hypotheses and now I'm ready to run an experiment, or I just ran an experiment and it, it, you know, it turns out that the reason why they're not doing XYZ is because they don't understand this. And so all of a sudden you have a product manager that really understands that the goal is not just to launch new features. The goal is to add value. And so when we see that adoption isn't what we hoped it was, or maybe what it historically has been, we then go on this path of figuring out the why. And then once we understand the why, it's now we can do something about it, something that's actionable. And for me, that's like music to my ears, where people understand their core job, which is to build stuff that creates more value to the user for whatever industry or whatever product you're in. And so, you know, it's not just— you're not at that place where you don't know what to measure. You're not at that place of you've measured it, but now what? You're in that really great place where you have insights, you have a bunch of ideas of how to act on it based on hypotheses, and then you're really managing a product. That's what it's all about. And I think, you know, too often we do PM but we don't know what's happening after that, or we launch features and we get stuff shipped and then we move on to the next thing. And that's obviously what we— we're missing a big part if we're not measuring.

Yeah, I know. I appreciate that. I have a couple awesome audience questions that are coming in, so I want to— I want to go to a couple of those. One of them has to do with metrics, not so much on our product. We talked about kind of some product health metrics, how we get there. But product team metrics, um, you know, the question was, you know, if you are, or, you know, what kind of metrics should you use when you're actually developing the product? You know, should we be tracking things like, you know, how, how often we're taking things off of our backlog? Or, you know, I've in the past, you know, worked with a CTO where, you know, he's really concentrating on like the efficiency of the team. Like, what's your take on those kind of metrics?

Yeah, so, so a couple of things. Like, um, I actually haven't done it for a couple of quarters, so it reminds me I need to do it. But there's a couple of things that come top of mind. One, just the product team in itself. And so I have a handful of questions that I survey my team, and it's really about, do they understand the job, you know, or the— what they're trying to solve? Do they understand that? You know, number two is mostly about, do they feel like they're able to leverage their skills to be able to accomplish that? Do they feel like what they're doing ties to the bigger goal? So that allows the bigger company goal or whatever, vision or mission. And so that allows me to assess like, are we aligned? Like, are we headed in the right direction? That's just a pure like engagement. So that's always great to check in on. We also like, in my team right now, every week at the end of the week, you kind of do a recap of how the week went. What you accomplished, what went well, and just an overall happiness score, like how happy are you, how, you know, how good do you feel. And so that's just data that I get on a weekly basis. And so, you know, if I know somebody's consistently, you know, 4 out of 5, 5 out of 5, and all of a sudden they're at 2, I know how I— that's a, that's a great data point for that. I can go react on that. Now it's a bit of a lagging thing, but because we do it every week, I have a really good sense of kind of the happiness of the team and do they feel engaged? And so I think about like quarterly, you know, quarterly surveys, weekly survey, whatever you can do to get the pulse. But of course, that's just qualitative data. And we know data itself isn't as good as when you combine it with just talking to your people, making sure that everything is going okay and having those face-to-face one-on-ones. Now that's kind of just the pure, like, this is the product team. I'm getting data. I'm trying to learn how we, get better at XYZ, how they're engaged, but how engaged are they? But when you think about the overall product team, the product and engineering team, you know, in terms of metrics, like if you don't know, like each of your, say, call it feature teams, if you don't know their, what, you know, their velocity or their, what they can achieve, you know, sprint to sprint, and then you're not measuring, okay, if this is our velocity, this is our velocity, this is what we can hit in terms of a sprint. And then you're not measuring whether or not they hit that velocity or hit that commitment, then that's bad. Like you need to absolutely understand the throughput and the output and the velocity of your product and engineering organization. So, you know, what we do is for every feature team or every team or every Scrum team, we, every 2 weeks, you know, we look at those metrics and see if teams have, accomplish their commitments, if they've hit, if hit their velocity or not. And so we measure that religiously. In fact, we, it's part of our health dashboard for the company. So as a company, they see if these Scrum teams are actually hitting their commitments, if they're hitting their velocity. And, and that's super important because then that, of course, is a trigger, a leading indicator of if we're going to hit our commitments for the quarter or for some big feature release. And that then enables the non-product, the non-engineering people to say, hey, wait a minute, you've kind of been showing you're off track for a few sprints now. Are we worried about this launch that's tying with this marketing campaign? So absolutely we need metrics to be able, just like any team, it can't be some black box. We need to be able to measure the efficiency, effectiveness of our product and engineering team. So yeah, fundamental. If you're not doing it, start doing it today. Start understanding your team's velocity. Start understanding their output. Start putting goals. Start measuring it and make it visible. Make it— make people accountable. There is, you know, somebody said the other thing, something to me the other day, which is like, well, how do I hold my engineers accountable? Like, they're so protected, right? They're the most precious resource and there's almost a negative unemployment rate with with engineers and I'm like, well, apply social pressure. Like, let everybody see, you know, how they're doing against their commitments and it'll change. And whether you're in demand or not, you wanna hit your commitments. You know, that social pressure is super important.

Yeah, that makes a lot of sense for sure. This is another audience question and it's definitely the most upvoted question so far. Um, and, and this is one that's always on my mind. How, how do you distinguish the important actionable metrics that, again, might vary from business to business, but versus the, the vanity metrics? There's probably metrics that we could easily track and show people, but really they're, they're not as meaningful. How do you sort of distinguish those?

Yeah, I, you know, I, I totally get the question, and I've always struggled with kind of what's the confusion on this one? Not to alienate any of the people that asked the question, but to me it's a very simple test, right? Like, is it actionable or is it not, right? Like, can you and your team— so for example, if you are looking at, let's say, call a vanity metric a number of visitors, and you're looking at that number of visitors and you see that it's growing, it's going up, well, Okay, what do you do about that? What is— now hopefully it's going up and not going down or whatever, but like, is that— if you sat in a room with other product managers, other business leaders, and you said, what can we do to get this number up from a product perspective? Like, what are you going to do? Like, I, you know, now, um, something like that's similar to that, which is monthly active users, so kind of that DAO-MAU ratio of daily active users over monthly active users, that starts to kind of move towards that realm of actionable because I could say, well, look, you know, our goal is to take DAO-MAU from 25% up to 50%. And so what you're talking about is how do I make my product more engaging on a daily basis? How do I create, let's say, more transactional value in my product versus I need to just come in once a month to do this job and leave. Again, it depends on what you're trying to solve, but all of a sudden even something like Dow Mao becomes super, it becomes super actionable, right? And you can brainstorm a bunch of ideas of how do you increase daily active use for a B2B product, for example.

Yeah, it almost sounds like, you know, It's like first, when you have a metric, sort of challenge yourself. When I look at this metric, do I automatically know, is this positive or negative? And if you're looking at something and it's hard to answer that question, that might not be that useful of a metric, maybe. At least maybe that's one thing to do.

Yeah, and somebody here on the thread, looks like Barry said, you know, banning the metrics, does it solve a customer need? Like, you know, I think it's a great question and it kind of goes back to, as you pointed out, like the jobs to be done. For all of the products that you guys are managing, what is the core value that you're bringing to your customer? Why are they hiring your product? Again, job-to-be-done language. But at the end of the day, if you can't figure out how to tie it back to that, then you just kind of got to go back to the whiteboard. So, you know, I think, look, it's, you know, vanity, like, look, in our product, in our dashboard, we have vanity metrics, right? You know, I don't think vanity metrics are bad. Let's just call them what they are. And, and again, the litmus test is get a bunch of smart people in the room and brainstorm what you can do about how to improve this metric. Yeah.

Well, we have pretty much time for one last question. Time's kind of flown by, but I want to kind of combine a couple of things here because this, this question sort of came up in the, in the ask a question section, too. But, you know, there are some companies— I would say there are rarely companies, right? It doesn't happen as often, like Pendo, where Your CEO, Todd's a product person. Like, he came from the product world. Like, he gets product. I have definitely come into contact with some companies where the executive is not a product person. So their, their idea of tracking metrics might be, to you, might be a vanity metric or might be thing, you know, they're tracking things that are important for the business, revenue, profitability. How can we help for the, for the companies where the executive the executives might not be tracking the product metrics and that might traditionally have not been important to them. How can we help our executives understand the importance of real true product metrics?

Yeah, no, it's a really great question. And, you know, I think to me it's kind of a two-part answer. Number one is we have one really good thing going for us, which is this idea, which is a broader topic in and of itself, that product is becoming the new marketing, meaning At the end of the day, we live in a don't-tell-me industry or world. We live in a show-me. And so we have, whether it's B2C apps, B2B apps, we show people, we tell people, they see these apps that change our lives and we talk about them. And that's the number one way to drive business. That is the number one metric, you know, is word of mouth. And so there's really no number two after that. Like, it's all about word of mouth. And so, you know, what, what, you know, CEOs or business leaders of, of, you know, uh, these products, these digital products, if they don't understand that, then at some point something's going to happen, whether they're either disrupted or they're outperformed by somebody else. I mean, that's fundamental, and I think that's becoming more and more understood. So that we have all of that going for us, which is more CEOs, I think, are going to be asking, gosh, we need to measure product. Product is way more important than I thought it used to be, or whatever it is. So we have that kind of macro thing happening. And then I think in terms of what can individuals do, I think it kind of goes back to the first part of that conversation, which is part of it is we need to educate, you know. You know, one of the, one of the things that I did at my last startup where we were dealing with the CEO that was, we'll call it less product savvy in terms of metrics, in terms of technology. And I just walked into an executive meeting. I had 15 minutes to do an update. I said, We're gonna do, you know, we're gonna go through and I'm gonna kind of talk about some mythbusters in the business and the product. And I'm gonna kind of put a myth out there like, you know, this type of customer doesn't use our product or only this type of person use the product or nobody uses this part of the product. And then you either prove that to be right or you mythbust it. And all of a sudden you kind of start going down this path of like, Wow, these metrics or these insights are like, I want to go do something about that. And all of a sudden you're starting to change the culture of, you know, going from only sales numbers or marketing numbers actionable to, wow, this product metric data, whatever, is super, it's helpful and it's actually driving action and driving powerful conversation. Like, why is our engagement, why is DAO now so low? Or why is engagement low? Or why is our NPS so low? And then you can take some, you can take initiative. So I think about something like that. Take about, take, take all the myths or the things that are being talked about your product and put it to the test and do some myth-busting. And I see it as, you know, it's fun exercise, but it's also that education that we talked about. And then it shows that you're really trying to evolve and take your company to the next level by being data-driven, just like sales, marketing, and all the other departments are.

That's great. I love that. I think that's an awesome sort of exercise. We could all, we could all run our own Mythbusters games with our executives. I think that's great.

Absolutely.

Yeah. Well, we're past the time that we had with you. So I just, first of all, I just want to say thank you for being here and thanks to everybody for being here. But Brian, I know that other people have questions that maybe I didn't get to. There might be more that pop up. How can people keep tabs on you? I know you do some writing. You're on Twitter and other forms of social media. Can you maybe just let people know how they could keep tabs on you? Yeah, absolutely.

I mean, if, you know, you can follow me on Twitter @briancrofts. Of course, I'm on LinkedIn, like any of the mediums these days. And we were talking about before, like, if you— I would also love if you have questions, hit me up. But if you also disagree, if you disagree with some of that, I would love that debate, that conversation, because I think that's how we look. We're all in this together. We're all trying to rise the tide of product management and how we run our departments and how we influence the world and the business. And so I love the debate, the discussion. So hit me up on LinkedIn, Twitter, whatever. I'd love, love to engage.

So awesome, awesome. Well, thanks again, Brian. Thanks everybody for being here, and we'll see everybody in 2 weeks where we have a session coming up with Scott Belsky. So I'll see y'all then.