A Product Manager's Toolkit w/ Varun Nagaraj, Researcher at Case Western Reserve University

Hey everybody, it's Mike Belcedo, co-founder of Product Collective, and we are back for another industry interview. I'm excited about the interview that we have today because we're going to be talking about a topic that I think everybody would be interested in, and that's the toolkit of a product person. This is, you know, we're all reliant on all sorts of tools and platforms to help us do a better job as product people. And our guest Varun is actually done quite a bit of research on this. Varun is an academic, but not really. He's actually a product person himself and has been the CEO of product-driven companies in the past. And we'll get to talk to him in just a second. You could already see him, so you could do a virtual wave to Varun. Before we get to the conversation, though, just a couple housekeeping things. First of all, my connectivity might seem a little choppy today for whatever reason. So if I kind of fade away. Forgive me for that. We'll try to figure that one out in the future. But if you have— when we do start the session, if you have questions that you want to ask Varun, go ahead and do that in the Ask a Question section. I'll actually be monitoring that. And so I'll be pulling questions in throughout. But yeah, there's a specific section for that that we like to use. The chat, however, that's a perfect place to introduce yourself, you know, share what product you're working on, where you're from. Actually, Maybe let me know if you can see and hear me okay in the chat right now. That'll— if I see some people coming through in the chat, that might give me some reassurance that at least some people can hear us. But yeah, definitely do that. The other thing is we just announced some future industry interviews that are coming up. So you can actually go to the main industry interview page, which my partner Paul could probably link to that in the chat. There's all sorts of sessions coming up with Hunter Walk of Homebrew. You know, a top seed stage VC and Hunter's a product person. We also have one coming up with John Zeratsky, who is one of the co-authors of the book Sprint with Jake Knapp. Sprint, of course, coming from the whole GV team. So there's all sorts of sessions that are coming up. I think we have some that are going all the way out through September. We're almost booked through October. So I'm excited for that. So definitely register for those maybe after this session if you haven't already. And then, yeah, we have Industry, you know, working on that, and that's coming up, and all sorts of exciting things going on there. So I'm not going to talk too much about that, but you can all check it out at industryconference.com. I think the last price sort of increase of the season is at the end of this month, so if you want to save a little bit on tickets, you could do that there. You can also email me, and I might be giving with a discount code if you use it before the end of the month too. So you can hit me up for that. So with all of that out of the way, I want to officially say hello to Varun. How are you?

Good, Mike. How are you doing?

I'm doing really well. I'm doing really well. And Varun is actually a return guest, actually maybe the first return guest for Industry Interviews. But there's a good reason for that. You know, Varun, I mentioned you're, you know, you're a product person throughout your whole career, yet you have focused a lot of your time now through, you know, being a PhD candidate at Case Western Reserve University. You also do teaching, I know, at Boston University, but you're actually researching product management, which there's not really a whole lot of people out there that are doing that. So anytime that you talk to me about the research that you're doing, I want to make sure we can expose that to the folks at Product Collective. So anyway, welcome back, Varun.

Thanks, Mike. Really nice to be able to share the research with a community like Product Collective. Thanks for associating with us.

Well, you're welcome. And, um, and I do want to point out to everybody, you might already see there's a survey link, and Varun will talk about this, but there's a survey link. If you click on that, we definitely encourage you to do that. It'll help, um, give even more data to Varun for the research that he's doing. Um, but you'll actually get something out of that. And again, that'll come up later in this discussion as well. But I did want to point that URL out to to everybody. So anyway, Varun, I wanted to— I know that this particular chat, you know, we're going to talk about the product person's toolkit. And maybe we could just start by, you know, maybe you could just mention how did you start to dig into this in the first place and kind of what led you to starting to research this?

Okay. There seems to be some feedback. And I've got that input coming in. I was wondering if all could mute themselves.

I just muted myself. Participants shouldn't— they shouldn't actually be— their audio won't come through, but I'm going to mute myself when I'm not talking.

Okay. Okay, Mike. You know, when you and I tested it, we didn't have feedback, but I'm feeling— I'm hearing feedback right now.

So yeah, I am hearing a little bit of echo now from on your end, Varun, but I don't— maybe on your screen you might be able to turn volume down or something like that. Maybe that will help. I don't know.

Okay, try that. So it's a couple of things. As you mentioned, I've spent the last 3, 3 and a half years researching different aspects of product people and product teams. And we had a chance to share my findings and research on design thinking as used by product teams last night. And you know, there were some surprising findings about design thinking not always delivering what its proponents claim it's going to deliver. And so I continued down the path of looking at things to see if in fact they do deliver the benefits they're supposed to deliver. And tools that people use became a logical, uh, next choice here. Um, so if you look at tools, marketing and sales people have their set of tools. Uh, you know, there's a marketing stack, there's a sales stack, there's a CRM stack, and so forth. But there is no generally accepted definition of what a product stack is. That people can use. Okay, I'll put the mic—

Varun, if you wouldn't— let's see, if you wouldn't mind, maybe is there a possibility to use a different mic or use the computer mic or something? Maybe that might help.

I am using the computer mic here. Let me see if I can turn off the camera. I don't know.

Um, I don't know, you could— and maybe you could start— I know at some point you're going to start to talk about also some, um, you know, with the presentation, so maybe that's a good time to— you could pull that up at least.

Okay. So looking at tools, I tried to put together the tools that product people are using and really picked up the data for it in 2 or 3 ways. One was—

if you wouldn't mind hitting the full screen on that so we could get a better view of that, that would be great.

I have my— So I have the full screen. Okay, does that work?

Uh, for some reason it is, at least for me, just coming through as, um, just, you know, like we're seeing your PowerPoint screen but we're not seeing the whole, um, kind of the whole screen there. I suppose people can try to zoom in if their computer allows them, but I know we were doing the dry run, you were able to click something and it kind of went full and I'm not seeing—

Yeah, that's what I'm doing now. Okay. Okay, cool. Okay, I apologize, guys, for these snafus. We did do a dry run 10 minutes ago.

Well, we can— we'll send— we can send a portion of this presentation to everybody too. So if it's hard to see, if it's hard to zoom in and hard to see a portion of this, know that we could send that everybody's way.

Okay, and, and this content is also available inside of the questionnaires now. So from a tool category point of view, working at a workshop that Pendo, which is a tool company, had several months ago, and then working off of the internet and with tens of other product practitioners, uh, here was a starting set of tools and tool categories that I came up with. So these are the tools that product, these product people are using to help with different aspects of their job. So let's just walk down sort of the big broad categories, and on the right-hand side I have examples of companies that are offering products that fit into that particular category. So let's begin with sort of the one that's probably most popular, which is the notion of Understanding how a product is being used, right? So these are product analytics tools or session tracking tools, and I combine them into one category here because the real purpose is for the product people to understand how their products are actually being used by customers on a real-time basis and potentially to improve adoption because low usage might indicate lack of awareness not necessarily that it's a bad feature. And that's the reason adoption tools, I believe, belong in the same bucket as the analytics. And in here are companies that you probably are using, like a Pendo or a Mixpanel. There are session tracking companies like FullStory and Hotjar, and then there are folks that focus on adoption, like WalkMe and so forth. So that's one category of products. And I've also seen here people using Segment in order to embed that code into their products and then using Tableau or Looker or other kinds of analytics front ends to analyze that data. Right. So there's a couple of ways one could go about truly understanding how the product is actually being used, and that provides insight to the product. Person about their product and its efficacy. There's a new category of products that are emerging. Some of them come from the world of testing, so like Optimizely and so forth. And here the idea is you're doing feature experimentation, right? You're looking to see if customers prefer A, and that was the beginning of the A/B testing. But this category evolved sufficiently to also include sort of full-stack testing, right? You actually can construct a full product suite with potentially different backend flags and see what the scalability implications and the performance implications of different, uh, aspects are. So that also provides— the third category that I think folks are using increasingly is around incident management. A rich source of information for product people, what customers are saying when they're using it and not quite doing it right. And so the service desks and Zendesk become valuable sources of information as well for product.

Hey, Varun, real quick, sorry to interrupt, but I'm wondering if— are you— is there a presenter mode that you might be able to put it in, maybe in the view or trying to see the up top, probably slideshow.

Yeah, I have, Mike. I really apologize. I mean, I'm doing what I normally do.

Okay.

And I don't know why you went into slideshow and hit the—

okay.

Yes. Okay. Yeah.

All right. We'll have to— we will, we will follow up on this one afterwards. Sorry, everybody.

Okay.

Sorry, everyone.

Thanks for my apologies.

No, no worries.

Um, So some of the other categories are pretty cool as well, and they go from more the quantitative to the qualitative side. So prototyping has advanced well beyond just vision and sketch and balsamic for internal conversation, to also being able to implement it so that prototypes can be used really to gather more quantitative feedback. And I think that is an increasing use of prototyping to actually see how people react. In a more precise fashion. Idea and feedback capture ranges from user voice types of capture of information to things that are more crowdsourced. So IdeaScale is a nice way of going out and actually getting the voice of the customer or non-customer, if you will. And then there are obviously other things like Aha Ideas and Usabilla and so forth. Few companies are using UX research and testing services. So here you have a camera that's trained on a panel of people, and you as a product person can actually see them interacting with some sort of a demo or a prototype that you put in front of them. Perfect way of getting people who are not your— but it does cost some money in order to get that kind of panel set up for you. The part that I think is missing, honestly, a lot is the net crawling integration part. You know, one of the concerns I have as I watch product people is almost too much of an obsession with the current customers and the current usage. And while that is great, I do think it can make you somewhat narrow-sighted. And so really having a set of tools that is more broad-based and bringing in relevant information information outside of Google Alerts would be a good contribution to the industry, but I just don't see enough tools in that space just yet. So maybe for one of you, that's a great new company idea or a great new product opportunity that you guys might want to follow up on. Um, project management and workflow management are starting to blend together. As you look at Jira, they're adding discussion threads to them, threads for discussions or channels. And obviously Slack is, you know, immensely popular. They're kind of complementary products, probably focus more on collaboration, one's focused more on timeline task management, but they probably go hand in hand. And forever there's a set of roadmap tools that are out there that you can potentially use to just communicate pretty roadmaps, but ideally they evolve as well to help you make decisions and prioritize and rank and figure out what to work on, not just how to present what it is that you're working on. Um, so the question is, why do we need so many tools, right? Why are people actually using as many tools as they are? Uh, and what tools in fact are best for the job that you have at hand? So people are looking really balanced with things, you know, and our job, product people, has always been about and in this case it's balancing familiarity with unfamiliarity, right? Because you don't want to be blindsided, biased by what you know, but you then don't want to give up what you know in order to just pursue the total unknown. There's a balance there, the familiar and the unknown. There's a balance between insight that helps you refine your thinking versus that insight that totally disrupts your thinking, and both are necessary. There's a balance between qualitative interviews and discussions and data, and quantitative data that is actually very objective, not subjective at all. And then there's a balance between having people participate in the roadmap creation and having their opinions heard and having dialogue in order to really create, versus things that are more direct, actionable. So the right software tools really need to provide this kind of a balance, and no one tool is going to do that. And that's the reason why you need a tool set or a toolkit that makes sure you can cover these. And you know you have it right when you can create a roadmap that balances organizational alignment on one hand along with insight. Now this is really important because on the insight side, the more unfamiliar the situation, the greater the opportunity to come up with something really cool, really different. But that also makes people nervous. That makes your organization not entirely believe that you're on the right path because it is just different from what they've done. So finding that balance between something creative, something new, but yet something comforting that people support is a very, very tough balance to achieve. And that's one of the things that distinguishes a great product person from an average product person, in my opinion, the ability to strike that balance. The second balance that I think a product manager is looking to strike is the balance between its usefulness to a customer, which we all can immediately get feedback from, along with its novelty or newness. Now, in the perfect world, you come up with inventions or innovations that are both novel and useful, but in reality, it's hard to achieve that balance. So in my research, I've really found that these are the two balances on the roadmap that people are looking to find, and tools help you get there by increasing your breadth, your reach, by promoting dialogue and execution. And so forth. So that's sort of the sort of pieces around product tools and what we're seeing out in the marketplace there.

I, well, I appreciate you walking us through that, and that's, that's helpful. And I know, again, we'll share this deck, a version of it, with everybody afterwards. So there are parts that you couldn't see or it's hard to see, know that that'll be coming your way. If you have questions for Varun, again, you can ask them in the Ask a Question section, and I'll keep a tab, or I'll keep an eye on that. To start, though, I do have a couple of questions for Varun. So I know, you know, I've worked for a couple of different companies as a product person, and they were very different from each other, right? And so my role was similar for both, basically leading our product efforts. And these were early-stage technology companies, relatively small teams, you know, range from 25 people total in the whole company at one company to just under 100 for the other. And the product teams were both, you know, just a few people. I'm curious, you know, for the product person, whether you're in my— whether you are in the situation I just described, maybe you're working for a large organization and there are lots and lots of product people. But when you're evaluating different tools and you're trying to build out your product tool where do you start? You know, knowing there's all these different categories of tools, there's all different types of, you know, different companies within each of those categories, it seems pretty daunting. And I'm just curious if you have feedback for a product person based on some of the, you know, things that you've uncovered in your research and in talking to folks. What advice would you give?

Okay. I think there are some basic tools that any product company is going to have. So let's just walk through kind of the sequence in which, um, product or product would put tools together. So from a project management point of view, whether it's Jira, it's Asana, one of those kinds of tools that includes in there, you know, Kanban boards or the ability to manage workloads I think those are tools that people begin to use quite early. So I'm consulting with two companies, both of which are less than 100 people at this point. One of them uses Jira, the other one uses Asana for their task management and product and feature. So all the features they're working on are listed on that particular project management tool, as are the assignments. Now following that up, people typically add Slack or something equivalent to that, so forth, or they use the conversational capabilities inside of Jira and Tana, right, so that there is some kind of conversation happening about a feature, its requirement, its relative priority, and so forth. Is it perfect? Does it result always in amazing insights? Maybe not, but but it results in clarification at least as to what something might mean. It also allows salespeople, for example, in there and add their input, or business people to basically participate in that as well. So I think those are the two starting points. Now, the moment you have a product in the field, no matter how small, you're going to have a trouble ticketing system of some sort, right? So you're going to be using a Zendesk, It might be Jira, so on and so forth, because you do want some means of communicating with your customers, even if there are only 5 or 10 of them. So I think that becomes possibly the third category. So incident management, um, the third category that people— and then after that, I think it really depends on whether you're a B2B company. Uh, it might depend on whether you're really interested in how people are clicking and interacting with your product. If it is a UI-heavy product, maybe products like Kendo and Material become more important. If it is not that UI-heavy a product, they're less important. Depending on the number of salespeople you have or businesspeople that you have that are talking to customers, you may want to have products like UserVoice capture information in real time. If you have a lot of B2C customers, you know, if you're Xbox, I mean, or a game player, maybe something that's making games, you may have passionate people that want to give you customer input. So there might be a small company where it's still worthwhile to invest in the UserVoice capability because you actually have hundreds of people that want to give you input. So I think it's really So there's a core that I think makes sense for everybody. And then what you add on, I think, really depends on the business. Yeah.

Well, let me— I know we're actually almost towards the end of the session as it is. So, I mean, if there's another question anybody has, you could ask in the Ask a Question section. But the one thing I did want to ask you before we wrap was in the research that you've done so far, and I know you're really in the middle of the research now, right? It's not as if you know, you're sort of, you've gotten started, but there's still more work that you're doing. Has anything come up though that's particularly surprised you or was something that you weren't necessarily expecting?

Yeah, I have a sinking feeling about tools in one way, right? Even though this question is about tools, that there might be a danger again, depending too much on tools, just as there was a danger depending too much on design thinking, which was my biggest study. It is possible that tools lead us in a particular way that we don't intend them to, right? So tools may lead us in the direction of incrementalism. Tools may lead us perhaps in the direction of really focusing more on coordinating rather than something that's truly surprising and different. But I could And all the research that we're doing that I'm hoping that the audience members are kind enough to participate in is the following. I'm trying to understand the impact of different types of tools on different types of outcomes. And so there's a short 5-minute survey, my URL is out there. And I really would appreciate it if you guys could take the 5 minutes in order to do that. The definitions of all the product categories are in the survey, so I hope, you know, you guys find it useful, not just in terms of contributing to research, but there's also content in there that I hope can help you. And my point, you and I will do with Product Collective, at least I know who we're working research with, is we'll take the aggregate input, we'll provide it back to the community, so we're able to say within the Product Collective community, here is kind of the usage of tools today, so people can benchmark themselves against today's usage. And we'll also try and identify what tools are delivering what return, and I think we're going to find one or two surprising, and the tool vendors may not like what we find, but that's the advantage of us being users rather than vendors, right?

That's right. Now, and you know, somebody had asked where the survey is. I just, the there should be a green button showing. We try to set it up that way. If it's not showing, I just put the link in the chat. So definitely check that out, definitely take 5 minutes. And, and again, what we thought would be interesting is, you know, I'll follow up right away with the deck so that everybody could have that. But later on, after we get some feedback through this survey, Varun had offered to write up a post that we could feature at Product Collective, and we'll everybody that was in this session. I'm also shared out with the rest of our community too. But if this is something that's of interest to you, if this is something that you're working on building out your own toolkit, definitely take that survey. It would be helpful. And, um, and then yes, we'll, we'll get you some back. Um, so again, everybody, sorry that we had some issues, um, today. We, you know, sometimes technology happens. Kind of run with it. So I appreciate you bearing with us. Um, the upcoming industry interview sessions that we have, those should be exciting, so I hope that you'll check those out. Um, Varun, thank you so much for joining us, and yeah, I appreciate you bearing through all of this as well.

And thank you, and guys, I apologize for whatever stuff would have happened to me on my computer side. My bad. No, thank you.

And like I said, it's— I— it probably on my side, who knows, doesn't really matter. I just, this is a lot of, this is good information. I want to make sure we could get some of it to the attendees. And I appreciate you starting to uncover it in this session. And we'll, we'll do some, like I said, some follow-up posts on this as well. So thank you, Varun. I really appreciate it.

Thank you so much, guys. All right.

See you, everybody.