Implementing Design Thinking w/ Varun Nagaraj, Researcher at Case Western Reserve University

Hey everybody, this is Mike Belcedo, co-founder of Product Collective, and welcome for another edition of Product Lunch. Hopefully some of you have been here in the past and have enjoyed some of our Product Lunch sessions, but if this is your first time, welcome, and I will actually share with you just a couple details about what these are all about, and then we'll get right into getting to know our guests a little bit as well. But Product Lunch, these are live video Q&A chats that we have with product leaders. We do these twice per month. Usually it's the first and third Wednesdays of the month at 1:00 PM Eastern or 10:00 AM Pacific. So, you know, sort of around lunchtime. The idea is you could bring a brown bag lunch and sort of enjoy your lunch while we talk product. So these are a lot of fun. We have actually a couple of others coming up. In fact, we have the next two that are already planned and advertised. We have one with Claire Vo of Optimizely in two weeks where she'll be talking about experimentation, but experimentation at scale. So for those of you that are at larger companies and you love the idea of experimenting, but it's like, hey, we deal with thousands of customers, you know, sometimes it might be hard to actually experiment with them. Claire will talk a little bit about that. And then we also have Ash Maurya, the creator of the Lean Canvas, author of Running Lean, awesome product person. He's actually gonna be speaking and doing a workshop at Industry this fall. The conference that we have here at Product Collective, but he'll be focusing on a topic that he loves talking about, which is loving your customer. That takes place in 4 weeks. So we hope that you could join us for those and you just go to productlaunch.com. You can even sign up right after this session and we hope that you do. But for now, we're going to dig into this week's Product Launch session and just a couple of sort of housekeeping type things. If you have a question for our guest, whether it, you know, just anytime you think of a question that might be relevant, there's a section, it says, ask a question. You'll notice there's the chat, but there's a section that says, ask a question. Use the ask a question section. If you put it in there, I'll be monitoring that. And from time to time, maybe it'll be throughout, in this case, actually might be towards the end of the session. I'll look for questions there as well. And then in the chat, many of you are doing it right now, feel free to introduce yourself. You know, let folks know where you're from, what kind of product you're working on. That chat's a way to, you know, connect everybody. But again, if you have questions, use that ask a question section. So the only other thing I'll mention is we're coming to the close of a season for Rocketship.fm, which is the official podcast of Product Collective. We're actually opening up the veil right now, and Bob Mesta, who's one of the co-architects of Jobs to Be Done framework, He's helping us do Jobs to Be Done interviews for industry. So you could actually listen in on the pre-interview meeting that we had with Bob, the actual interviews, some of the post-call sessions that my partner Paul and I had, sort of opening up the veil and letting everybody listen in on those. And you'll get a sample of that through episodes at Rocketship. And later on, we'll release the full interviews. We definitely want to do that. So with all of that, I want to introduce you to our guest today, Varun. Varun, how are you?

I'm doing well. Thanks, Mike.

Awesome. Well, thanks a lot for joining us. For those of you that are tuning in, I've known Varun for the past couple of years because Varun has been studying. He's a PhD candidate and an Innovation and Design Fellow at Case Western Reserve University in Cleveland, Ohio. And that's actually where I'm an adjunct professor. So I've gotten to know Varun a little bit. But Varun's a product guy. In fact, he's been a product person most of his career. He was CEO of Sierra Monitor, which was an Internet of Things company, CEO of Net Continuum, which ultimately got acquired by Barracuda Networks. And he was a product strategy consultant at PwC and was a partner in the practice there as well. But more recently, he's focused his product life in an academic sense. And so at Case Western Reserve, which I had mentioned, but he's also a digital innovation fellow at Boston University. He's written a book very recently, helped with a MOOC course. So Varun, you've spent most of your last couple of years really on the academic side of product management. Would you say that's fair?

Yeah, I think that's fair. It's a change of pace, but a good change of pace.

Yeah. Well, I think it's, I mean, for me, that was one of the cool things when I met you. There isn't a whole lot of academic research that's been happening within product management. So when you had shared with me that this is something that you're passionate about and this is what you are doing right at the moment. I know for us at Product Collective, we wanted to get involved because we think that kind of research is really important, especially since it really hasn't been done much before. So I'm excited because really now this is sort of the unveiling of a lot of research that you've done over the past basically year or so. Is that right?

That's right. And a lot of it with the help of Product Collective members, and that's why I'm particularly excited to actually share the results back with your community. They were very instrumental in the data that went into this research.

Yeah, well, that's awesome. I'm glad that that was the case. Actually, for any of you that are tuning in, if you received an email from us at Product Collective asking you to take a survey, that's what this was all about. Or maybe you were at Industry and you got to meet Varun in person there and take the survey in person. That, that was this research. And so I'm excited to dig in. I'm just curious, maybe we could start with what's your take on— because I know that a lot of this research is sort of around design thinking in a way. And I was just curious, Why do you feel that academic research around design thinking is an important thing?

It's a fair point. There's a long body of work academically on design thinking, you know, that really harks back to how architects and real industrial designers used to approach problems. Now, the world of practice adopted some of those concepts and really turned it into its own thing, if you will, you know, with the Stanford D School and IDEO and so on. And the concern is that sometimes these things become fads and trends. And so right now you could argue that design thinking is an overused word, that it's viewed as a trend that, you know, you just throw design thinking at everything and probably everybody means something a little different. So what I wanted to do was to sort of unpack design thinking and look at it from the point of view of a product team to figure out what exactly does it mean And does it really help or not? And if it does help, how does it help? And so the survey was really about trying to collect data to come up with a quantitative analysis of these various relationships between design thinking and results.

And for those that are interested in sort of the process of the data collection, like how did you go about collecting that data? I know we were involved a little bit, but there are certainly other ways too. Maybe you could talk a little bit about that.

Sure thing. You know, the data collection was through a survey instrument that took roughly about 10 or 15 minutes to complete. You know, there's a science to designing a survey so you're not playing your hand. You know, you don't want the person responding to know what hypothesis you're testing or not. And so you kind of mix it up a little bit and you're trying to construct, you know, what are called constructs with related answers and so on. Such. So I administered it in a couple of ways. One was online to the Product Collective community. One was at the industry conference, and I got about 110 responses really from Product Collective members that were usable, meaning they were complete. Unfortunately, 200+ people took the response, but if they drop out halfway through, you know, it's not something I can use. And then I picked up another 130 or 140 responses through my own LinkedIn network. By going onto the LinkedIn forums on product management and design thinking and soliciting input from those folks over there. So of the 247 responses that made up the study, I would say about 170 or 180 of them were filled out by product managers, and the rest were filled out by, you know, product leaders and others that would have similar positions in order to be able to answer the types of questions I asked. So we collected the data over a period of a month or so, and then after scrubbing it, after scrubbing 300, 400 to get down to 247, that's when I then used structural equation modeling. For those of you, the product managers who are more quantitatively oriented, it's a form of doing multiple regressions at the same time with looking at model fit and so forth. So that was the mathematical approach I used in order to drive, the numbers that I'm looking forward to sharing with you.

Awesome. And well, and I want to get into those numbers, but maybe before we do, could you take a step back? Because you said something earlier, which is that design thinking as a, you know, as a thing, as a term, you know, it kind of can be considered to be overused. I'm just curious, do you have like a definition of, you know, just how you define design thinking for the purpose of this study before we sort of dig into the research?

Yeah, that's the right first question. If you don't mind, I'll actually put up a slide that hopefully helps, right, with the quick definition as I used it. Hold on, give me a second. I'm not sure why. Yeah, no worries.

And I love seeing in the chat people from so many different places. Looks like I saw Rhode Island, New York City, California, Poland, Dallas. So thanks everybody for joining us.

All right, so can you see the screen that says Defining Design Thinking?

I can, yes.

Yes, okay. So, you know, people look at design thinking generally in one of two ways, and that's on the left-hand side. You know, people think about design in terms of what's called the design attitude. You know, so these are beliefs that you have. It's a way in which you approach a problem, your level of comfort in various situations and so on, right? It's a very psychological way of looking at design. And this is actually the most academically correct way of looking at design. It's as an attitude that a person carries. There is also a whole school of thought coming more from the practice world about design thinking as a set of tools. So here you're thinking of mind mapping, customer journey mapping, you know, so these are tools that are considered design thinking tools, right? Even brainstorming would be an example of a design thinking tool. So the question was to sort of put these together, and the way I define design thinking for this study was to say that these design attitudes and these tools are manifested or used in day-to-day practice by the product team, right? So these are essentially, these become principles that get embedded into what a product team does on a daily basis, these attitudes and these tools. And the way you can tell if in fact these have been successfully embedded into a product team's day-to-day routines are by looking for sort of 4 signs, if you will. And those are the 4 bullets on the right-hand side around empathy, meaning does the product team's day-to-day routines involve a deep sense of empathy with stakeholders of all kinds, right? And empathy is not just asking what do you want, but what is it you're trying to achieve? Why do you feel the way you do? And so forth. Generativity is about being really open-minded and very comfortable with lack of answers. Representations is about using tangible physical things in your conversation to clarify matters, you know, a prototype, wireframe, these would be examples of representations. But again, you're using them to clarify the problem, not necessarily to sell the solution, right? So it's not a demo as in, a demo for sale, it's a demo for conversation enablement. And iteration is obviously the ability to take feedback and do something with it on an ongoing basis. So the idea here really was that a team that is practicing design thinking is going to be high in terms of empathy, generativity, iteration, and representations in terms of what their day-to-day activities are. So that was basically how, Mike, I defined design thinking.

That's great. That's helpful. And I think that sets the stage well. One question that came in, and I don't know what you are allowed to share and what you're not, but it would be awesome if— might there be some version of this deck we might be able to share with people afterwards? Maybe you could think about that.

Sure thing.

That would be great. And if we can get something like that, just for everybody's sake, we will go ahead and email everybody afterwards with a version of that deck.

Okay, that sounds, that sounds good. Great.

So yeah, I mean, Varun, I guess I'd love to dig into kind of what your findings were and, you know, just because I know you put a lot of work and it took a lot of time to go through this study and to build up the questions and actually find the people to respond to, but what, yeah, like what did, what did we learn from this?

Sure thing. So let me, let me switch to some slides again, simply because they have some numbers and it's easier to probably look at them. As opposed to just talk to them. And once again, hold on. Okay. So this was basically the model that I looked to test, and I'll just take a minute to describe it because then the numbers a line around this. And this is typically any kind of a causal model would look something like this. So the hypothesis here really was, if you can follow my mouse, that the application of these design thinking principles, you know, the 4 things I talked about of empathy and iteration and generativity and representation, would have a couple of effects. One, it would basically improve the utility or the usefulness of the product, hence you see that positive sign. And greater amount of product usefulness, meaning product fit in the customer environment, would lead to greater market success. And I define market success as a combination of short-term business success and long-term capability building. And, and surveys had questions that basically factored into these things here. I also hypothesized that design thinking principles being used would result in greater product novelty, novelty being something new and sort of out of the box, if you will, that the team comes up with, and that greater novelty would result in greater success, right? Seems intuitive. I also then said, in addition, there are probably other effects between design thinking and market success, meaning not all the effect can be explained through these two pathways of utility and novelty. Then there are other effects that we're not testing, which will be significant as well. And then what this product unfamiliarity angle does here is referred to as a moderation, meaning that in the presence of unfamiliarity, meaning unfamiliarity being 1, if you will, that these relationships would actually be stronger, meaning the design thinking principles would be even more useful when you're dealing in an unfamiliar environment in order to create useful products. That's what that plus says there. And that this unfamiliarity would actually get in the way of creating novelty meaning design thinking would not be useful when you're trying to create novel products in an unfamiliar situation, right? So these were sort of the hypotheses that I tested. And then based on the multiple regressions and the structural equation modeling, here's essentially what I found. So the first finding was that design thinking does in fact influence the product's market success. So as a blanket statement, is design thinking a good thing to do? The answer is yes. And what these numbers are saying, and again, I apologize for geeking out too much on you guys, but here's essentially what it's saying. The design thinking principles, as you can see, what that number is, is a regression coefficient. And that's a pretty strong regression coefficient.

Oh, and Varun, real quick, on my screen, I still see the hypothesized model. So you might need to click over to the next.

Okay, will do. Let me just go to the next slide. Maybe just tell me if— does this work? Yes. Excellent. So here's basically what you're looking at. You're looking at design thinking principles strongly driving the usefulness of the product. That's what that regression coefficient is, meaning a unit of 1 increase in principles will increase utility by a unit of 0.45, which is a pretty strong number in these kinds of regressions, and these stars simply mean that these are significant values at certain values of probability. So design thinking principles, you know, strongly drives utility, and utility strongly drives market success, right? And that shouldn't be too surprising. You look at that and say, well, it kind of makes sense, you know, if I'm really engaging with the customer and modifying my product on a regular basis as I get feedback from the customer, the usefulness and fit is going to be greater, and I've really built something customers care about, and therefore my success should be higher, right? So that's kinda what that's indicating. This is a more interesting relationship. So let's look at principles to novelty. So it's slightly weaker. So using design thinking does help create, you know, new kinds of products, but look at this here. Product novelty is actually not significantly related at all with market success. In fact, the sign is negative, though it's not significant, meaning something weird is going on here, right? Design thinking is creating novelty, or what you would think of typically as innovation, but that is not translating into market success in some fashion, right? So that's sort of strange. Then you sort of look at this number here of 0.505, you know, where my mouse is, and what you're saying is, well, there's a strong correlation also between design thinking and market success that is not explained by these things. And that may be because of other reasons, as in maybe the team bonding gets better, maybe motivation increases, You know, there are other reasons why products are successful besides whether they're useful and whether they're, you know, cool and novel or not, right? So a blanket takeaway would be design thinking is, final assessment, useful in driving market success, right? But these pathways merit some more discussion, and that's what I'll show you a little bit here. If you see this next screen, Mike, can you see this product utility as mediator? Okay. So what this is saying is, as you would expect, design thinking drives utility, which drives market success, right? As you would think about it. Then what I did was I modeled unfamiliarity on top of this. And I said, okay, what if the situation, the problem facing the team is really unusual or unfamiliar? You know, is this relationship still valid? And the beautiful thing here is that this relationship is not only valid, it's actually stronger. And, you know, I apologize for not being able to see these numbers below here, but those would be essentially the regression coefficients under conditions of familiarity and unfamiliarity. And under conditions of unfamiliarity, this relationship is actually a stronger relationship. So this is again, a good thing for design thinking, saying design thinking is going to produce successful outcomes, through customer fit, and the more you engage in design thinking in an unfamiliar situation, the greater your customer fit will be and the greater your success will be, right? So it's a, as you would expect, it's a good thing to do when you're confronted with unfamiliarity on this pathway. But here's the one that is I think of most interest, which is the pathway between design thinking and success through novelty is unexpected. So more design thinking is not resulting in success through novelty. You know, something weird is going on. And when you increase unfamiliarity, this relationship falls apart even more so, right? So what it's saying is that design thinking through fit and therefore success makes sense. Design thinking creating novelty and having market success, something odd is going on. That is not a straightforward relationship. And so this is a relationship I dug into just a little bit more. And this is pretty darn fascinating. And I apologize again for geeking out, but let me take you through what this is saying. So this is basically looking at the relationship between novelty, which is the x-axis, and success, which is the y-axis, right? And what you're seeing are what are called U-shaped or inverted U-shaped curves, meaning, So if you look at the green, as novelty increases to a point, success increases. When novelty gets too high, you know, one can imagine something being totally bizarre, right, meaning it's brand new but it's just weird and bizarre, success actually drops. Now the difference between the green line and the blue line is the following. The blue line is what you see for those projects that had a high degree of design thinking. The green line is what you see for those projects that had a low degree of design thinking. So what is this saying? It's saying that, hey, listen, you know, novelty behaves in a weird way, meaning too much novelty can possibly be bad. But that novelty that was created with design thinking, meaning the blue line, is actually a better novelty than the novelty that was created without design thinking, right? You know, you can see that it's higher. You can also see that the curve flattens out, meaning you can actually get more and more out of the box and weird and strange and still not get penalized because the act of design thinking made sure that your innovation was relevant, right? So it brought a practical aspect into innovation or novelty that was not available to those that were not using design thinking. So that's sort of what the angle there, There is, and I'll just go to one more mic and, you know—

And hey, Varun, if you don't mind, would you mind, see if you are able to minimize the notes underneath? That might make the screen even a little bigger.

Okay, okay.

If that's possible.

Did that help?

Maybe a little bit. So that's, I think we can see these words at least, so that should be okay. Oh, thank you.

So, so, so, you know, I certainly threw some numbers at you, but at the end of the day, we are you and I in my past life, we are practitioners, right? So what does this all mean, right? So I tried to boil it down into a set of dos and don'ts. So my first do, and I think this is a reasonably broad statement that applies to all of you on the webinar, is that design thinking has absolutely value, right? So don't just dismiss it as a fad. So do incorporate these principles into your routines. And again, what were the principles? They were the principles of empathy, generativity, and iteration and representation, right? So having those be part of your routines is a good thing. It does result in market success. Design thinking is absolutely going to help you develop more useful products. They're going to be better fits with the environment. Design thinking is also going to help you deal with unfamiliarity. Right? So those are all good things. However, if you have been tasked with coming up with something out there, something that is so creative, so different, you've been tasked with coming up with the new Walkman before the Walkman existed, or you're coming up with a new MP3 player before the MP3 players ever existed, and so on. If that's your goal, to create something that is novel, not just something that is a better fit, with customers, do not just depend on design thinking. You need to be looking elsewhere. And maybe it's the elsewhere as in an amazing visionary leader like a Steve Jobs. It may be elsewhere in terms of looking deep into R&D and research labs as a way of pulling out that kind of novelty. But do not count on design thinking to produce that level of novelty that you're looking for. However, design thinking is still going to help you even if you're pursuing novelty, because this novelty that you're coming out with is going to be actually more effective in the presence of design thinking. It's going to be more useful and have better commercial prospects if you're using design thinking. So all in all, I would say design thinking, you know, is a good thing to do. Think of it not as a set of tools, but think of it as a set of principles that you actually embed into your day-to-day practice. And I think good things will happen in terms of fit and in terms of realizing novelty that actually can succeed. But do count on other ways of creating novelty if that's what your mission is. So roughly speaking, Mike, that's about the, you know, that was kind of the genesis of the gist of the research.

That's awesome. Well, and I know we only really have a few minutes left, so we have a couple audience questions. I want to make sure that we can ask some questions to you as well. Somebody had asked the question, can you talk a little bit about how do you measure something like product novelty? Like how is something like that even measured?

Yeah, so the novelty scale was basically the way I defined it was in terms of level of newness. So for example, if you think about a 1 to 5 scale, and I'm just talking about one question here, right? But if you think of a 1 to 5 scale, if you say we basically fixed bugs that we had in the product, let's say that's what the person had responded to. You know, every respondent basically described one single product or project that they had worked on, right? The attributes of the product, what they did, what the results were, and so forth. So if they picked a project which was essentially a bug fix type of a project, or a very, very minor update, that was over there, and a product project like that would be considered a less novel project, right? There's very little novelty or newness about a bug fix. Something that basically is a first ever in an industry, right? The first kind of feature or the first type of product delivered to the industry, you know, and perhaps borrowing from techniques that are not even used in the industry. Would be an example of high novelty, right? And you can imagine things in the middle, you know, catching up with competition. Yeah, that's probably somewhere in the middle. You know, it's novel from the company's point of view, right? They're doing it for the first time, right? So there is— so it's more novel than a bug fix, but from a customer's point of view, you know, all you're doing is catching up with stuff that people already, you know, your competitors have out there. So the questions around those kinds of things to being able to create a novelty score for a given project.

Gotcha, gotcha. And somebody else had asked sort of along the similar lines in terms of like, hey, how do you sort of define that? But, you know, how do you, how do you define market success? You know, when you're in that area, is it strictly quantitative? Is it qualitative as well? Kind of what goes into that?

So market success in this study was actually a, what's referred to as a second-order construct, meaning it built on two separate ideas. One idea was short-term commercial success, which was measured with a combination of, you know, 3 or 4 questions. And short-term commercial success asked, the questions were, did you achieve or exceed the financial expectations for this product? Did management consider this project a success after 1 year, right? So those would be the kinds of questions that would measure short-term commercial success. And then I had a set of questions around long-term success, and long-term success would be doing this project allowed us to develop capabilities which were subsequently useful in our business, right? So something like that would be a measure of long-term success. So I created essentially, you know, two first-order constructs, short-term and long-term, and they hung together well enough with the model fit that I combined them into this second-order construct called market success, that had both a short-term and a long-term element to it.

Got it, got it. Well, thank you for that. Well, we're— I'd love to keep talking, right, but we, we're kind of already up on our time. I do want to make note, um, Chad McAllister in the chat, and I'll share this with you later, Varun, but he had a, an episode of Everyday Innovator, which he's the host of, that might be of interest to you. So I'll send that your way. Um, but But for other people that have questions or they think of something later, do you mind even if I share contact info so they might be able to reach out to you and dig in a little bit more?

Absolutely. Being a researcher is a lonely life and any kind of interest that anybody shows is good interest as far as I'm concerned. The other thing is I really do, Mike, A, wanna thank you for your support throughout, but your audience is, is really excellent. And there is yet another research project on the use of software tools by product managers that I'm kicking off. And I think this could be of great interest to your audience as well. And I might very well reach out to you and your Product Collective membership around this study, as in, you know, does using all of these fancy tools, whether it's Optimizely or Pendo or UserVoice and so on, I mean, does it help? And to what degree and when does it actually help? So that might be one that I might come back to you and the community on, and I'll be happy to share the results back either in a webinar or an industry or whatever is convenient for you guys.

Well, you know how I feel about that. At Product Collective, we, we're just big supporters of the research that you're doing, and yeah, any way for us to get involved, or when after it's all sort of said and done, to unveil the research, you know, we'd love to be a part of that. So that That would be great. And I appreciate you taking the time for us, Varun. And yeah, just thanks so much for walking us through this today.

Thanks a lot, Michael. It's totally my pleasure. Thank you.

Awesome. Well, thanks everybody for tuning in too, especially coming from all over. This video will be immediately available for stream as soon as I hit the end broadcast button. So you're welcome to revert back to it. And I will share some version of the deck that Varun's able to share. I'll send that via email pretty soon. But so with that, I hope to see everybody in a future product launch and Have a great rest of the day.

Okay, thank you.