Strategyzer Webinar with David Bland: Testing Business Ideas

Hi, everyone. Welcome to today's Strategizer Strat Chat webinar. We'll be starting in just a minute. We just want to take a few moments to think about what you might want to get out of today's session. And if at any time you have questions during the webinar, please feel free to use the Zoom Chat box. And we'll try to get to your questions throughout the webinar.

So I think.

We can get started. Welcome again, everyone to today's Strat Chat webinar. Today's session is about testing business ideas. I'm Lauren Kantor. I'm your moderator. I run the content studio at Strategizer. Our host today is Alex Osterweilder. He's the co founder of Strategizer. And our guest is David Bland. He's a founder, author and innovation coach. And Alex and David are working on next book in the Strategizer series called Testing Business Ideas. And throughout the session, if you want to, if you have any questions, please feel free to use the Zoom Chat. And no need to take any notes, record this webinar. We'll be sending out the session later this week. So with that, I'll pass it over to Alex and David.

Awesome. Thanks, Lauren. Thanks for having me. Thanks for joining, Alex. So, yeah, we're really excited to talk about our upcoming book, Testing Business Ideas that's going to be out this fall. And I just want to share some content with you all and start a conversation. And we're really excited about it. So we wanted to get everybody involved. So the first thing is how do I reduce risk in my business idea? We keep hearing that time and time again. I know Alex hears that all the time. People want to create something new, whether it's a startup or a corporation. And we're trying to go into kind of an unknown, right? So it's not a known market. And we're trying to figure out, you know, what is this market? Can we deliver this, you know, is there a business model there? Can we generate more revenue than cost? And are there infrastructure, you know, concerns as far as we can support this? And I hear it a lot at the team level. And Alex, I'm curious, do you hear this at the exec level or what do you hear out in the world when you're, when you're consulting on this stuff? Absolutely.

And I think the good thing is the way you framed it. How do I reduce risk in my business idea? Because often people think about building stuff, right? I have this idea. I'm Just going to build a smaller version of it. And if you frame it this way, well, let's see what the risk is, how risk is this idea, what needs to be true for this idea to work. And then you break it down into smaller chunks into these hypothesis, you. You can manage in a very different way. So I really think this idea of risk, framing it in terms of risk and smaller chunks that you can test resonates a lot. And I'm super curious to see what you're going to present and have a conversation about this, because working with you on this next book has been pretty fun.

Yeah. So I do think there is an element of risk in the sense of it's just too big to think about. Will we succeed or will people buy it? And Grant, granted, those are risks, but you need to be able to break it down. And so what we've been working on together has been this, essentially a library and a field guide for experimentation. And it's went through several iterations, and it's based on almost like 10 years of experience helping companies do this and over time trying to figure out how to make this actionable for people when they say, okay, I want to reduce risk, but I'm not really sure what's available. And, you know, if you only have one tool in your toolbox, let's say, you know, or three tools, you know, usually it's landing page and surveys and interviews that's going to help you reduce some risk, you know, but it's usually mostly desirability risk. And then what else is even available? And some people don't know. And so we kind of do a landing page and then it's, well, now what? And there's a bunch of things out there that you can do. And I think just giving people something actionable is. It's almost like giving you superpowers, right. So you can choose what. Okay, what's available to me when I have this type of risk, it's like, oh, there's all this stuff available to me. How can I. How can I run those and help feel better about my path versus only going back to the same ones over and over again? Because, you know, you might be completing experiments and that might feel like progress, but it's very dangerous because it's not necessarily progress tied to your riskiest assumption.

So this is a question, David, can I ask you a quick question? Because this library is really complete, right? There's a whole series of things that you can do beyond surveys, beyond landing pages, beyond interviews, which are kind of the go to things or then building something. Were you surprised when we formalized this into all different types of experiments, how many we ended up with?

I am. And I'm also surprised at how the strength of evidence kind of played out. So for example, I made some assumptions even going in and I've been doing this a while, which was I had, I kind of had a feeling that when we were doing discovery work and open ended, the evidence is usually a little, a little weaker. And it really played out when we started categorizing things. When we're doing something we call them validation experiments in the, in the book. But when we're doing things, we're putting something out in the world and people are responding to it and there's a real value exchange. You could just see there on the right, the evidence strength is kind of off the charts, you know, because if you are delivering value and even if it's manually right, you're kind of testing it end to end and people are paying for it. But when you're doing, you know, things like surveys and interviews, you still need to do those, but the evidence strength is weak. So I think I kind of felt that with the teams I was advising over the years, but I always tried to prevent them from let's say, let's do some interviews and then let's build a whole app or a whole product. Because you're going from this weak evidence strength to something that's really high fidelity, that costs a lot of money and if you get it wrong, it's really hard to recover from. And you know, I always try to get, to incrementally go through it and I think it was really telling for me. The other point about evidence strength, that was a big learning aha. For me and Alex, you can chime in here too, which was, you know, you can't. It's not an overall evidence strength. Sometimes even if you did something like a landing page, if you're asking for email, that's relatively weak because anybody gives out their email now. And so it's not, it's some currency that's really weak, you know, and. But if you're asking for payment information that's really strong because they're giving up payment information for something. So even in the gradation of like variants of an experiment itself, there are different nuances to strength. So that was probably one of the biggest learnings for me was when we went through categorizing these.

One of the things that actually helped me do when we were working on the book and this idea of strength of evidence, you know, was also answering some questions and realizing, well, you can test exactly the same hypothesis.

Right.

Willingness to pay with different levels of strength of evidence. You use an experiment like an interview, people say, yeah, of course I'm gonna pay that. I have a budget. That's relatively weak evidence, or it's a quote. But then the exact same hypothesis that they're willing to pay for something, maybe even a specific price, you can get stronger and stronger evidence. So you're basically working on the same hypothesis, willingness to pay. But your experiment gets more and more complex. If you want, with your certainty and the strength, the evidence gets stronger and stronger. First you had a quote from an interview, then you might collect some emails and then you go to simulated purchases.

Right.

Which seems trivial, but actually when I look at teams, they're not as sophisticated as that, Seeing that they can test the exact same hypothesis with several things. They might just do a landing page and that's it. So that was a big eye opener for me as well. And then connecting that with the library has been brilliant.

Yeah. I think one of the things that I hear time and time again when I'm working with teams all around the world is, oh, is this a one to many thing, like one hypothesis to many experiments? I think they feel like it's a one to one. I have a hypothesis, I run experiment and I give thumbs up and I move on. And it takes a little more rigor than that. Right. You have to try different things. And are you really proving or refuting this thing, basing off one experiment? Typically not. I've never really worked with a team that just ran one experiment and then uncovered millions of dollars and was super successful. It's always this winding path of, you know, what are the different things I could do?

So, David, before we move on, before we move on to the first experiment, there's somebody asking, and this is a good question, could you explain how do you measure strategic risk? So just breaking it down maybe in the three risk categories that we have in the book might make it a bit clearer for people before we get started.

Sure, yeah. So there are three major themes that we use. Alex, you have more too. But for the book, we're focusing on desirability, which is, if you look at the canvas, it's really about the value proposition. Your customer relationships, your channel and your customer segment. So think of that kind of top right corner of your canvas. For viability, we're looking at revenue and cost. So think of the bottom of your canvas, those two boxes and. And Then for feasibility, we're going beyond just technical feasibility and we're talking about, do you have the right infrastructure? So that's key activities, key resources and key partners. So are you able to kind of deliver on that strategic risk? And then also the value prop canvas, which is pretty much all desirability. Right. So your customer profile and your value prop seamed in. So when we're framing strategic risk for this book, we're very much talking about, do people want this? Is there a market need? Which is all desirability. We're talking about, is this a viable business in viability? And then we're talking about, can we support the infrastructure for this to create this business, which is very much feasibility, can we deliver?

I think it's important to point this out because sometimes people, they just build a smaller version of their idea of the product, actually. And this is not about product alone. It's really what we want people to think about is we're testing a business. We're testing a new growth engine. We're not just an R and D lab testing if we can make a product. That's the big thing. We're not just testing if the market wants it. We want to know, can we create real business growth financially around this? And that includes an entire business model from the start. So there's this belief sometimes people say, oh, startups, they just need to test the product. That's BS Because I can build a successful product for you tomorrow if I can spend all the money of the world. It's really about creating something that creates value for customers, but is also embedded in a scalable business model. And this is, I'm curious before we go into this first experiment, I think it's a good one to show this. David, I'm sure you've seen teams that forgot that it's about building a growth engine, a business model, and not just a product. And then it failed once they validated the product, but they couldn't turn it into a business.

Yeah, I think most of our experimentation knowledge out there has been purely focused on user research and product experimentation. And so I'm always taken back when I'm working with teams and the light bulbs go off and they say, oh, wow, I can experiment on the back end of this business. I can actually experiment my way through, you know, can we do this and how we make money on this? And I think that's why we kind of felt the need for something like this book, which was there's a gap in knowledge out there in the market of holistic. Experimentation around your business and you can successfully kind of do user research and desirability experimentation, but then you still might fail because you can't support the infrastructure or you get the price points wrong. And I think, I don't know, I think some people have anxiety around testing with price, but you can really do it in an ethical way and do it to maximize your price point. But I just. Yeah, it's really interesting to me how I feel like all that, all the knowledge out there is around kind of user research and desirability testing.

Okay, let's get started with this wizard of Oz, right? Some, some of the experts know this, but probably not an experiment that is used a lot. Probably the most underused experiment.

Yeah, you know, one of the things. So wizard of Oz has been out there for a while and it's basically you're kind of creating and delivering the value manually, kind of like in the movie. Right? But basically. And unlike concierge, where concierge is similar, but it's obvious someone's involved, so there's a. There's more like a concierge service where the people are visible. And when we start breaking this down, I realized one, not a lot of people do this. I think it's one of the most underutilized experiments out there and it generates almost the most learning. And I think it's just because people have a hard time like, how do I break this down and what does it look like? And so what we started doing is kind of breaking it down for you and saying, okay, so what does this test? Well, it tests like all three themes. And that's something that we really understood pretty early on in the book writing process was it's not that these three themes are in isolation and it's not that there's one experiment per theme. A lot of these kind of span all three. So if you're receiving orders and manually creating value and delivering it and they're paying for it, that's all three themes, right? You're really doing all three. And it doesn't scale, so it's not ideal for scaling, but it's great early on learning for figuring out, you know, is this really this value exchange real? And so when you start breaking that down, you look at costs. And for the most part, you know, because you're doing this manually, cost is not, not really high with regards to something like this, you can keep costs low because you're doing it manually. Again, you're not doing it at scale, so you don't have to worry about scaling you know, people and all that. So it's basically pretty cheap to do this and it generates a lot of evidence strength. So out of all the experiments like this is one of the stronger ones because there's a real value exchan there and you're testing out the back end, you know, can we, can we deliver this? And then when you start breaking down the setup time and the runtime, yeah, it takes a little longer to set up and maybe you run it a little longer, but it's not something that should take you months. Right. It's something that you can kind of spin up. If you're at this point in your process, you kind of, you've ran some lower fidelity experiments and cheaper experiment to inform this kind of bigger one where you design the end to end process, process. And then for the capabilities you're going to need, you know, almost if it's not, it doesn't have to be all separate people, but you're going to need, you know, design, product technology, legal marketing, you're going to need to be able to deliver the value, right. And if you skimp on one of these too much, you know, let's say you don't have any design. Well, if the design isn't at the right believability, right. You're not delivering value that's, that's usable and viable, then, then it's going to fail. So we give some advice and deeper explanation in the book. But basically you need to be able to kind of pull together the right kind of capabilities if it's just you or if it's a team to kind of run through a Wizard of Oz in a way where you can deliver value and receive value and generate the kind of evidence you need.

So I like this a lot. Quick question, David. So we did this right for all of the different experiments. What was one of the biggest surprises for you when we broke it down into runtime, evidence strength, setup time, cost and the capabilities. Did anything kind of surprise you when, when you systematically went through these experiments?

I think for capability specifically, you know, I always recommend cross functional teams. So I say product design, engineering, that's what you need, that's what you start with. And I think as I really dug in and that does cover most of your bases. Right. But it doesn't always cover things like legal marketing. Some of the things that might not feel like they're core to you, but they need to be in a good shape for you to be successful when you run your experiments. So I think that was one of the learnings for me, going through and breaking these down was, you know, product design. Engineering is a great starting point, but I think it's almost like we need to evolve our definition of cross functional team to be everything that we need to ship and learn in the market and de risk our business. And for that for you, if it's hardware, you know, you're going to need hardware design, you're going to need like industrial design, you're going to need hardware engineers, you're going to need to be able to, you know, for your context, pull that together. If you're doing software, it's a little different, right? You have software design and engineering. But for the legality part, if you're in a heavily regulated environment, there's some things you can and can't do. So you need to consult legal for marketing. I mean, it's a great thing to say, I'm going to have a Wizard of Oz, I'm going to run this. But if you don't have any people, they don't know about it, it's not going to succeed just because you didn't have attention. So you can't neglect something like marketing. So I think for me the capability section kind of jumped out in addition to not one experiment per theme, which I think naively maybe walked into this field like that. That's like a nice clean break and that's how this would all shake out. But in reality they overlap themes. Cool, cool.

We have an example for this one, right?

We do. So I want to kind of break down a little bit more of what some of the insights you're going to get from the book are on doing something like wizard of Odds. So again, behind the curtain the stuff's going on. It's not always visible to the customer. Right. So it's kind of, it could be a physical curtain or a digital curtain, whether it's software or hardware. And the idea of like, okay, we need to prepare for this experiment. I think so many experiments are ad hoc and we just kind of do them, right? There's some initiative, we get excited about it and we just run off and do them. But if you're doing something like a Wizard of Oz, plan it out. What are all the steps I need to do to kind of receive an order and process that and give it back to a customer and receive payment for it? Have a board, have some way to track it. If it's offline, literally just have a board on the wall where you're tracking the steps. If it's software, have a board up where you can track the steps and just basically make sure you have, you know, it's not completely ad hoc. It's something that you've planned out and you can track and measure. And then when you execute it, you know, update, update your board as you go along. Document how long it takes to complete things, do customer satisfaction interviews, survey, csat, you know, try to. Try to get qualitative feedback as well from the customer to see is this the right, you know, viable kind of level of fidelity for what your wizard of Oz is trying to learn, and then take the time to analyze all the results. You know, did people actually find the value? How long did it take us? How costly was it? Where were there delays? Right? Are there blockers in your process where you have to figure out, hey, before we operationalize this and we want to scale it, this could be a killer for our entire business if we don't kind of solve this and kind of use all that learning manually, delivering value to help inform your automation if you plan to go forward and scale it. And, you know, the biggest pushback I get on this when I talk to people is like, why don't have the time? You know, I don't have the time to do something like this manually. And it's kind of crazy when you think about it. You know, it's like, oh, I don't have time to go through this, but I have the time to scale it to an infinite amount of customers and not know where the risks are. You know, so you have to make time. Yeah, yeah. So for the example, I'll get to that. But basically for the pairings, I also we give some example, for some examples of like how you would pair customer or how you pair experiments together. So for example, on this one we could say, well, you're running a Wizard of Oz. What goes well with that? It's almost like a menu. And it's like, well, you can have a feature stub. A feature stub is something on your website or digitally where they click and it's not ready yet, but if they sign up, you can deliver the value manually to them. Not at scale, but if they're that interested without building out the whole product, you can use something like a feature stubborn. Another thing you could do is a brochure. This is really easy, especially for service or other products where you can literally have a brochure and there's a call to action on it. And if they call you manually behind the scenes, deliver the value to the customer. And then obviously one of the ones that are More popular is something like a landing page. And I think people may be, I know it's really popular kind of experiment, but the idea of if they submit an order digitally, you can behind this digital curtain, create value and send it back to them. And they really don't know it wasn't software, right? So you can really use landing pages to your advantage that way too if you're not further along. For existing product I would do something like a feature stub, but for a landing page it's something kind of new. And then afterwards there are other things you could do, right? You could do some kind of mashup MVP where you take parts of what you learned and automate it with existing tech so it starts to scale, but you don't over invest in it. You could do things like referral programs where you have, hey, do you really love this experience? Can you refer a friend? There's something you could test around, kind of virality and kind of engine of growth there. And then you could do something like crowdfunding, right? So if, you know, people really want this, can you crowdfund it to, you know, to drive. To drive to more demand? And basically, you know, it's not an experiment in isolation, you know. And then we've laid this out into sequences too, which are much longer in the book. But the idea of it's not in isolation. There's stuff feeding into this experiment that we've done before that we've learned and there's stuff coming after this that we can do. And I think that's another one of the bigger learnings for me working with teams is that it's really hard to make this repeatable and they're not sure what comes next or what should I be doing before this. So for the example is a fun one that we have in the book from Topology. And so basically they have augmented reality for glasses and they customize and measure your face and basically they'll customize glasses and ship them to you. So it's really, really cool tech and also kind of a blend of software and, and hardware. But they started off as a pop up store, which I think is fascinating story, right? So a pop up store in San Francisco, they said, look, so pop up store is basically, you know, a temporary store that you use to kind of sell, but you can also use it to validate and to learn and discover. And you know, they just wanted to find out, hey, before we expend all this capital in developing a sole solution, like are people aware of this and are they even willing to have some kind of high tech approach to the solution. And so the experiment is they created a pop up store and they intercepted people on the streets. They would talk about their, you know, the fit of their glasses, you know, the people that were really interested, they kind of guided them into the store and then, you know, they really had kind of low expectations. But you know, they were trying to not get too excited about it. But after two hours they sold four pairs, you know, average of about $400. And while that was interesting evidence for them, I think the big learning for them, the insights based on speaking to them was, you know, the qualitative insights were even more valuable than the sales. So, you know, the idea of what exactly was, you know, they were kind of symptom, aware of what, but they weren't really sure what problem, you know, what was causing the problems on their face. And while it wasn't fitting and the idea of like, oh, this is how we need to dial into the specific problem because they kind of understand it a little bit, the symptoms, but they don't really understand the overall overarching problem to why the glasses, why the glasses don't fit. And then they kind of use this pop up store, they ran other pop up stores, they kind of use this to inform the, you know, their landing page. You know, the text on their landing page come from a lot of those great customer quotes and the interviews they conducted in their store. And so they're able to kind of generate all this learning from a relatively short time box experiment where, you know, there was a value exchanger in the sense of, you know, granted they didn't have all the product available yet, it was kind of behind glass, so they weren't necessarily handing pairs over. But the idea of while people are going to give us their email to sign up, they're really interested, people are willing to pay, they're willing to give us feedback into what the actual problem is. Things like this, I feel like you could be really creative and tap into your team's creativity and do something fun that also will help you learn and de risk your business idea.

I like what you just said about creativity because we put all the creativity on ideation and whatnot. But the point is that we should probably be a bit more creative in how we test our ideas in the whole series of experiments that we do. And I think here we have some good examples of creative experiments. I think the creativity for testing is underrated. I'd say that's actually the hardest part. Right. So go beyond interviews and surveys and landing pages to really get some very strong evidence for creative one experiments. And in particular in highly regulated industries. Right. You can't just do landing pages. Doesn't make a lot of sense. So creativity is underrated when it comes to testing.

Right. And we have other examples in the book too that are not, you know, they're more kind of tailored to B2B into hardware and regulated. And we have some sequences that we've laid out for those industries because we do work with those industries. And I think overall it's just. I think you're exactly right. The ideation seems like, well, that's where the greater creativity happens. No, I mean it's. Yes, it's one part of where the creativity one place. But there's so many things you could do when it's like, we need to learn this. How are we going to learn this? And if you can just pair that up with a series of things that are available to you, it's pretty amazing what teams can come up with. And they come up with like much more creative things than even I could come up with. And I've seen a lot of different opportunities out there. So it's really, really cool. And see them get excited about it and feel like, all right, I'm going to learn about this before I just go off and build it. Because I'm really worried because we've all built something that nobody cared about probably in our career and it's just a terrible experience. And so the idea of I can use my creativity to help de risk something I think is really powerful.

I love the way you framed it. Right. Creativity, we believe it happens in one place, but actually happens throughout. It's awesome. Lauren, you've been following the questions. Any questions you would want to highlight for. For David and myself.

David, I think people are curious as to what you think the difference between startups and corporations and even B2B or services industries. Are there different considerations you need to take when picking an experiment and conducting the experiments?

Yeah, I see there's probably more overlap than people realize because I do work with both or many. So, for example, I see a lot of the practices startups use are also the same practices corporations use. It's just a little different context. So we do give some advice in the book on things that you're probably worried about if you're a corporation. Number one thing is brand. That seems to be, you know, I work with companies where their brand is over 100 years old and so everything's about the brand. And I think in a way that almost that culture creates this behavior where we can't release anything unless it's fully polished. It's better than perfect because we can't damage the brand. And if you notice there are a lot of kite. There are a lot of companies are addressing that by kind of sidestepping the process a little bit by saying we're going to create a sub brand or an off brand or we're going to create a new company to just test this idea to see if it works and then we can bring it on brand or we can kill it if it doesn't work. And so you know, when you search online and you see all these different. There's a lot of Indiegogo campaigns that are branded this way, a lot of Kickstarter campaigns, a lot of landing pages, a lot of things are digitally out there where people, you know, they're not necessarily leading with the brand. If you trace it back, you can see, okay, this is powered by such and such, but the brand isn't something that should stop you. Right. It shouldn't stop this behavior just because you're afraid of not putting something out there that isn't perfect. So that's that kind of context we're putting in the book. As far as, look, if you're at a corporation, your rules are a little different, but they may be more similar than you realize as far as what's available to you. It's just you have to do it within the constraints of the system. Whereas a startup doesn't have those constraints because their constraints more around resources and money, but not so much about their brand because they really don't have one yet. So yeah, in the book we try to lay that out for teams to give them some advice.

Yeah. I would add one thing that as a difference. Steve Blank often mentions this one, that startups can break the law. They shouldn't. They might go to jail for that, but. And corporations can't. Right. So you want to get your legal department involved as early as possible, not afterwards. You want them as allies. Right. So I think the best companies we see, they actually have lawyers who help the teams with designing experiments. And not just in pharma, where it's just mandatory.

Right.

But also in other areas. So if you don't have legal on your side, they're just going to come knock on your door and shut you down. Down. So you want to make sure as an individual team that you get their buy in up front as much as possible. And if you're a company that's already a Little bit advanced. You actually want a legal team that is here just to do that. In particular in Europe now. Right. With the whole GDPR where you need to make sure you treat the customer data right. So otherwise you have a problem. You really want to get the legal team involved as an ally. I think that's a big one. A big difference that we should point out. Lauren, any other questions? I have one that came up, but I want to see if there's anything you want to highlight first.

Sure.

David, I think people are curious. You hadn't. I know the book covers a lot of different metrics for measuring experiments, but I don't know if you want to go through the pop up store or any of the experiments you've already shown about what actual metrics you would use to measure the risk.

Oh yeah, I don't have them in the stack but basically in the book we have, we give recommendations on specific measurements per experiment. So things like basically like cycle time. So I talked about a little bit wizard of Oz. I dived in that. I dove in a little bit where we talked about, you know, how long did it take you to complete the tasks, where were you blocked, you know, what was customer satisfaction. So for each experiment type we kind of break it down into specific things and serve that up. So that, that is in the book. We do give guidance and then we have a strength indicator on each type. So like I said, you know, if people are saying things to you like at the pop up store, if they're giving you good feedback verbally, that's relatively weak. I mean even it's great qualitatively and you might use it later, it's relatively weak evidence because they're just saying right with their purchasing it's really strong. So that stronger sense of evidence. So yeah, in each experiment type we have that broken out separate spreads.

I'd like to make a quick comment about one of the questions. If we can go back quickly to the wizard of Oz. Somebody was mentioning. Oh, wizard of Oz. Maybe I just want to put up the slide very quickly of the, you know, with the curtain. Somebody was saying, yeah but this looks like it's more for low, you know, investment products. I think it's actually quite the contrary where it costs a lot to build a system or to scale a system. That's exactly where you want to use the wizard of Oz.

Right.

Because you're basically not going to build the entire system. But the customer still feels like he's dealing with the real thing. But behind the scenes you're doing things manually. So I think actually wizard of Oz is a very, very good one, very, very good experiment for platforms and products where the investment cost is very high. So after interviews, you might go to something like wizard of Oz to get stronger evidence than just the interviews. So a lot of companies that we work with, they say, yeah, but it's going to cost me a million dollars just to make a first prototype. That's exactly the type of place where you want to be creative and use wizard of Oz. And generally I'm curious about what you think, David. I see a lack of creativity and still you get them started. Once they've done it once and came up with the creative experiments, the floodgates kind of are open. But before that they always feel like they need to build this expensive thing. So what's your experience with this? Also in relationship to industries where the investments are pretty big upfront, so you want to look at different kinds of experiments?

Yeah, I mean, it depends on your context. So one of the companies I worked with out here in Silicon Valley, we were building mobile apps and they're really successful mobile apps. But before we got to the app, we did a Wizard of Oz where we had a landing page and people submitted what they wanted. We manually designed them with a designer on the back and sent it back and gauged viability in regards to was this the right quality is what they asked for? Did they use it the way we thought? Because it's for social media and it's for social media managers and to learn. And so we used all that learning pretty early on in doing it manually without them knowing a person was designing it and use that to design the actual application. So, you know, when you think of how apps are designed today, you know, the UI is really, the UX is really important. So if you have too many things in there, it could, it could be just too confusing for people and it'll fail that way really expensively. And so what we learned was, you know, people only care about a couple things and if we actually added all these other features we wanted, they don't even use them, they wouldn't want them. So we used what we did manually to help inform the app. And I won't name the app, but it's one of the five star rated apps in the App Store and it's used by hundreds of thousands of people. But it'd been really expensive to build that in a way where we added all the bells and whistles that people didn't need. So we use wizard of Oz in that case. To kind of learn what do people really care about. And then we tailored our design to that. So we already had some evidence it wasn't just sitting in a room going wouldn't it be cool if. And that's a really dangerous phrase, as my friend Jeff Golfdoff says in software development. So you don't want to stack it up with a bunch of wouldn't be cool ifs. You know, you want to have some evidence to say is there really a demand here? And so in that way it can help you really from making a million dollar mistake.

Lauren, I think you've seen a couple of questions that, you know, going in one direction or another. So we have, we have people with completely diverging opinions, right? 180 degrees differences. What are some of those things that came up?

Sure.

David, people are asking about the, a lot of the topics we discussed are about business model innovation or focus on business model model innovation rather than product innovation. And then some are asking how you would use the testing for services. So focus on products versus services.

Yeah, so a couple answer a couple of this. So basically product is part of this, but it's not all of it. Right. So there is some product experimentation in the book and stuff that you can use for product experimentation, but keep it in context of the overall business model. So over, like the book is focused on testing business ideas and, and your product or service is going to be part of that business idea. So it is a core part, but it's not the only thing. So we do address that in the book. And then services, in some ways they might be easier to test. Right. So I don't have all the experiment types here laid out for you today obviously, but concierge things where you just do things really hands on early to get the learning you need before you scale services because sometimes services are going to be more people and money. Right. And hours. So there are techniques that you can use for services in the book as well.

And David, what about for projects that might be in existing companies that tend to have long time frames that maybe there's, you know, it takes a lot of infrastructure to build. Is there a better test for that?

Well, I mentioned it slightly on this one with the feature stub. So companies like Google do this all the time, right. Where it's like this is a feature, it's not available yet and it's quick test. It doesn't mean you base your entire decision off the test, but it is a point of evidence on whether or not we should build this thing or not. And so you can do that with pretty standard technology today, fast. You don't really need a dev team. You can just literally paste something in and have a dashboard, right. With some of the product offerings today. And so there are ways in an existing company where the learning loops are really long to get learning faster. And I think that was one of the big ahas for me working with teams was, you know, I hear back from product managers, well, we can't wait this long to learn. And one, I love hearing that because, wow, they have this, like, hunger for learning and we need to learn faster. And it's not coupled to our release cycles. But two, like, what are. They got kind of curious, like, what are the options out there available that would help us shorten that learning loop? And the good news for you, and we'll do this digitally after the book as well and have a page for you, but, you know, what are the resources available to me if I'm tied to these long release cycles? How can I get the learning faster? What are some products and services that'll let me do that? So things like feature subs are more for, you know, I want to test a feature or an option that's something existing and do it quickly and get the learning I need.

Yeah, I'd like to add to that one because there's this common myth again, and I'm repeating myself, that we need to build something, literally a smaller product of the thing to actually find evidence and learn, right. So if you are in an industry with very long product development cycles and heavy investments, that's a reason more to be insanely obsessive with testing before you put anything into feasibility, Right? So when the cycles are long, you need to be really, really sure about jobs, pains and gains of your customers. You need to be, you know, really have very strong data in terms of what is the most important job pain or gain. Because often, you know, when I hear people saying, oh, we know our customer really well, you start asking questions, you ask for evidence. Why do they know that this job is more important than that job to be done? Why do they know that this pain is bigger than that pain? And they usually fall apart. They don't have the evidence to say that. Right. So I think in particular in industries with long cycles and very heavy investments, that's exactly the place where you want to be obsessed with, with getting the jobs, pains and gains, right. Before you put a single dollar into feasibility, before you put a single dollar into building. And I think one of the attractive things that I found really amazing working with David, on this book is that there's a whole series of experiments that you can do beyond the interview to start to understand customers well before you go into building stuff. Right. So this one, I get a bit passionate about it because I see way too many mistakes of people starting to build much too fast.

Yeah, I agree. I work a lot with the automotive industry and have in the past. And depending on who you ask, the cycle time on a car from idea to production is about six to seven years. And think of the investment. Just hold up your phone and imagine what that looked like six or seven years ago. Your phone looked very different. Right. I think, I don't know what I was using. Maybe a Motorola Razor, maybe, I'm not sure. But the idea of, you know, if that's the pace of technology change and you're trying to validate something like a vehicle in automotive, you know, some of these techniques can help. And that's what I do. I help automotive companies figure out, you know, what's the need for this thing before we actually invest in it. Because, you know, integrated into car, with all the industry infrastructure and then all your point of sale contacts, all this other stuff that could be really, especially around anything like connected car, autonomous.

A good example of that, just while you're talking about it. The Porsche Cayenne was conceived with that in mind. They actually had willingness to pay conversations before they built anything. So they started to understand the different pieces in the cars that customers would be willing to pay for. So not just do you want this? Which is already a bad question in itself, but they were trying to figure out how much people would pay for the different elements in the car before they built any kind of physical prototype. I found that very impressive. And that's public information. Right? So willingness to pay conversations upfront, that's. That's pretty exciting to see.

Yeah. So a couple more things before we get into more Q and A. So one people keep asking me, well, is this just a list of experiments? And it's not. So I'm sharing some of the experiments with you today. But in addition to the field guide and the library aspect of the book, there's a lot around the book as well to help you make it kind of keep your momentum and choose and theme and look at your organization. So this is just a taste, right? So over the years, you know, we've kind of codified what are the kind of ceremonies, what are the meetings we need to have to make this, keep this momentum and make it repeatable. And so one thing is like planning it's planning. Just like everything else, you know, you're going to have to plan, you're going to have to plan before you run off into your experiments. They're not just ad hoc experiments. So as part of you keeping momentum, schedule your planning sessions for your experimentation just as you would your delivery. And then stand ups, I mean, standups are pretty core to most businesses now. And even if you're not doing them every day, we recommend every day. They're also important for experiments. Each experiment, when you start breaking them down, even something like wizard of Oz, the taste I gave you today, you have to assemble a team, you have to lay out all the steps, you have to get the marketing legal involved, you have to plan it all out. All these tasks kind of roll up and it's work. So as you design these experiments and pull them together and run them, you should be tracking your work and helping each other as you do all the tasks for each experiment. So you need some kind of ceremony to help keep that on track. Otherwise it's four or five weeks in and you're still at the beginning of your experiment design. Also learning, and that's a huge part of experimentation is not just doing them, but also what did you learn? So giving the space for the team to kind of come back and synthesize your evidence and get insights and help. Drive to action. You know, I think I keep hearing this in the market today of learning. We have to learn faster than everyone else. And that is true, but you also have to turn that learning into action faster than everyone else. So, you know, the market isn't going to reward you for just learning, right? It's going to reward you for building something that's, you know, high value, that provides, you know, solves for a need or a job for the customer. And so, you know, learning is great, but you also have to make space to kind of really learn and really drive that to action.

Let me just share a small anecdote there because let's have a good laugh about strategizing. Well, so I remember a couple years back, my co author, Yves Binyar, who's not involved in the business at Strategizer, said, you know, you guys are great, but don't forget that, you know, it's not about learning, it's about action, it's about building business. So we were doing a lot of learning, a lot of great experiments and it's so easy to get caught up in experimentation and learning. At the end of the day, all, you know, you really need to do is make Progress from idea to business, which sounds like a trivial thing, but often we kind of get caught up in learning. So I want to share this silly kind of little story of failure at Strategizer of us learning too much and not acting enough.

Yeah, it's interesting. I feel like over the last decade we've kind of moved from fail fast to what you need to learn fast to what you need to learn to put that into action fast. Right. So it's not about failing, right? I mean, yeah, you're going to fail in experiments. That is the nature of the experiment. Some of them will fail actually some should fail and need to fail, but it's not. You don't focus on the failure and failure feels. I mean we have this saying in the Valley which is we need to embrace failure. And I'm like, I don't know, I don't know if I need to embrace. Like I'll give it a fist bump, you know, maybe you know, acknowledge like a nod to failure, but I'm not embracing it, you know. So I need, so we need to move from that to a learning mindset and then we need to take that learning mindset and drive it into action. And that's where you're really going to see this payoff. And then so also part of that is retrospectives and again this is pretty common and even Kaizen events for more kind of lean manufacturing companies. But basically take the time to figure out how we're going to get better at this. And so you know, as part of this is kind of training like Eric Reese said is training your intuition. And so for example, you need to get better experimentation and if you don't give yourself space to talk about what you need to do to get better, then you're not going to get better. And so the idea of oh well, we forgot the instrument analytics, well that's a big deal because you can't get the learning you need. What are some steps we could take next time to help prevent that from happening? And then finally you need some kind of deciding decision maker session also with your stakeholders. They feel very uncomfortable when team just goes away and starts running all these experiments. You need to be able to kind of package up your learning and, and present it back. And you know, if you're a startup that's pretty much like, you know, when your VCs pretty should leave you alone, right? And then you come back and say this is what we've learned, you know, with our funding etc. But if you're a corporation, you know, this is coming. You know, you're not pitching a vc, you're pitching a VP maybe. Right. So the idea of, you know, how do we package our learning, communicate upstream? Because there's this balance. You have to act, you have to walk here with. We need to de risk the business idea we're working on, but we also need to kind of teach people around us this new way of working and communicate how we're making progress. And I think the mistake I see with some teams is they get really excited about the process. Right. And the process is interesting. I mean, obviously I'm passionate about it, but, you know, your execs aren't going to care about the process as much. They're going to care about the results. And so you need to be able to kind of communicate, hey, these are the hypothesis we had about our business idea. Here are the experiments we ran. Here are the evidence we generated. This is what we learned from that evidence, and this is what we want to try next. And if you can frame it, you're helping them understand. Wow. Okay, Help me understand why you chose that, why you chose that recommendation. Is there any other part of the business that can help you out on stuff that you're blocked on? Right, so there's some really powerful stuff here with regards to not just doing the cycle once. That's kind of like level one. We want to get you to level two or three where you're doing this repeatedly. Maybe you have a portfolio of teams testing out business ideas and you're not just doing experiment once, you're doing this repeatedly over time and systematically de risking your business ideas. So we're giving advice like this in the book too, where we're trying to lay out ceremonies for you and say, this is what we've seen over hundreds of teams across our experience and this is what we think will work for you. As far as starting point, I want.

To emphasize two things here because I think this is an extremely important kind of visual that you put up and I see kind of mistakes in two of them. One is in the learning that teams fail to try to learn every week. Look back at the experiment they do. And I like the way you often frame it with teams. One experiment a week.

Right.

But that they try to learn at the end of the week. What are the insights we gained and how do we turn that into action? Meaning what do we need to change in our value proposition or business model? Right. So that weekly learning in action is something that in particular in large corporations, less in startups. I don't See that as regularly, they would go along four weeks and do some testing and then after four weeks, sometimes even worse, eight weeks, kind of start digesting and making decisions about their value propositions and business models. I want to emphasize that weekly cadence is extremely important in terms of learning and action. It's the first one I want to highlight. The second one is around deciding. I see many, many teams when they have these meetings with key stakeholders, the decision makers, at the beginning, when they're inexperienced, they fear presenting the evidence. They want to make the idea look as good as possible. But it's not about the idea, it's about the evidence.

Right.

To say, here's what we learned and here's why we're confident we should be doing this. Here's what we don't know yet, rather than saying, you know, the traditional way, ah, beautiful idea, beautiful spreadsheet and all that kind of beautiful. It does require, of course, that we help the leaders, that we often need to train the leaders to look at the right thing, not that they're looking at the shiny object. Oh, this is a great business model, you know, great business idea. It's very shiny, has a beautiful spreadsheet. No, they need to look at the evidence and help the teams evolve. So those two things, I think are where we can make most progress still in corporations.

Oh, I agree. And it happens even on small things like. Like interviews. Right? We'll do interviews, but then all the notes are strung out in Google Docs or pieces of paper or word docs, and then we don't actually synthesize it. And if you don't do that, it's kind of dangerous because then you just pick out the things you like to hear to validate your own ideas and then you run with them. So even something like as simple as interviews, taking the time. And we do this in the book. We teach you how to synthesize the results. I think that it's just a practice we need to get involved in. There's a discipline to testing that maybe isn't really applied everywhere. So not only are we focusing on the library, we're also focusing around the library of. Here are the ceremonies you need. We go into leadership. We go into org design a little bit too. You can't solve everything in one book. But I at least want to share what we've seen as patterns that can help you. And then, you know, if you're reading this book and you have to convince your boss, right. You need something as far as making the case of, okay, well, we think there is a way we could do this and present progress and share what we've learned. So I'll pause on this for a second because, you know, this is kind of the core of the book. But what other questions do we have? I guess Lauren from the group or if Alex, if you have any other questions or comments.

I think Lauren has a couple lined up.

Right. We don't.

The time flies. We're 50 minutes in already. Right. Really, really fast. Lauren, what are one or two questions you'd like to highlight?

David, a bunch of people are asking. You know, there's a whole industry around market research and a lot of large companies spend a lot of money on that market research. How would you weight the value of market research versus the evidence you'd get from testing?

Yeah, I think there's still value in market research. What I'm seeing is taking that expertise and trying to customize it and shape it in a way where, look, we need to do market research, we need to do it quickly. So I see much more of an emphasis on speed nowadays. And so I have some corporations that are still partnering with their market research firms, but they're just the nature of the partnerships a little different. They're saying, look, we need to learn this by next week. We can't wait three months to kind of create a focus group and recruit people and perform it and record it and create these high fidelity kind of learnings. We want to learn fast. And there's a trade off there. I think there's also somewhat of a danger of outsourcing all your research in the sense of one, it has an expiration date. So if you're building all your research off something you paid for a couple of years ago, things have changed probably since then, depending on your industry. Even insurance. Right. I was just meeting with insurance companies this week and their world has changed even the last few years. And so research reports that they have. Yeah, that was interesting learning at the time, but it's not really actionable today. So I'm not discounting or discrediting traditional market research. I do think there's a need for that. But I do think teams are prioritizing speed over everything. And so if we can just generate stronger strength of evidence over time through research and generate our own evidence, we're just going to feel more confident about that if we have some, at least guidelines on how to do it. So even in your own organization, you may have researchers that just tap into them. You know, they're the experts. Right. And you can help them understand. Well, here's what we're trying to do at the pace we're trying to do and what can you provide? So you do have to balance kind of traditional methods that might be a little slower to, we have to go faster. Because if we take a really slow approach to this, you know, by the time it's even generated, it might be outdated in some, in some of these.

Industries I would, I would build right, exactly right on top of that. And say in particular in the early stages where you need to make fast decisions makes absolutely no sense to outsource this, nor internally, nor externally because you're the one who makes the very rapid decisions. You don't want to have some kind of four week market study that's going to cost you in terms of time internally or a lot of bucks, a lot of money externally. In the early stages you need to be in the driving seat. Right. Steve Lank even likes to say you need to see the pupils dilate of the people you're talking to. I wouldn't go maybe as far. But then in the later stages I think it does make sense. When you already reduced uncertainty a lot and you're pretty confident that you're going into the, the right direction. You could hire an agency or maybe the internal team to do a bit of a larger study. That could take a bit more time, you know, two, three weeks. But you definitely don't want to do that in your first sprints.

Right.

The first eight weeks, probably I'd say to three months. You should really be doing this yourself and rolling up the sleeves because you're the one who are making decisions. You need to see that learning and sense, you know, in which direction you should do this. Imagine a startup outsourcing their customer research. No VC would ever fund them. Right. So this shouldn't be the case internally either. Any other questions, Lauren, that came up that you want to highlight?

David, I think you've been talking about the strength of evidence and here on this list of experiments you can show the relative ranking of evidence strength. But I was wondering if there's an overarching way to kind of define what is strong evidence and what is not.

Right? Yeah. So here maybe getting a little more detailed, we have to be mindful over time. But basically there are variants of strength. Right. So weak is again, it's kind of closing the say do gap. So if there's, if it's verbal, qualitative, Right. It's generally weak sense of evidence. And if there's a transaction or something that's more quantitative, it's usually stronger. So qual and quant is one way to look at it. Another thing that we've done, we understand that it's not just a one shot thing. So if you look at the some of the circles here, you'll notice they're outlined. We even learned as we go through, you know, experiments that some aspects of the way you run it can generate weak or generate strong. So there's kind of a range and variation thereof. So we address that in the spreads in each experiment. Alex, you wanted to expand on that at all?

No, I think you did an excellent job. What I'm amazed is how fast these 15 minutes went through. We were hardly able to, you know, go through all the content of the book. I think this, this has been brilliant. And I'll hand it over to Lauren to wrap it up. I already want to say thank you, David, for this webinar and thank you for actually, you know, joining us for writing this book. It's been an absolute pleasure. I want to be very clear. David has done the heavy lifting on this. I tried to infuse some of the knowledge that we have, but this has been a brilliant, brilliant collaboration together.

Yeah, it's a lot of fun.

Thank you, David. Thank you, Alex. This is a great webinar and thank you all the participants who had some really great questions and interactivity. As you can see on the screen, we're coming out with another book. But first off, David's book, Testing Business Ideas will be coming out in late fall, around November. If you want more information on that, I posted a link in the chat and also on the Strategizer webpage, there's information. And then Alex and team are writing another book called the Invincible Company, which is basically about tools about invincible companies that have reinvented themselves and how they've reinvented themselves over time and again. Strategizer will be posting this recording of this webinar. We'll be answering more of your questions on the blog. If you have any more questions about Strategizer itself, please have a look at our website. And thank you so much again.