Discover & Validate Breakthrough Innovations w/ Steve Cohn, CEO of Validately

Hey, this is Mike Belcedo here, co-founder of Product Collective, and back for another industry interview. Thank you everybody for joining us. I just wanted to give a little what these industry interviews are in case there are some folks that are joining for the very first time. So these are live video Q&A chats that we have with product leaders. We do these twice a month, usually next month might be a little different because we'll be in Dublin, Ireland for our European edition of our conference, INDUSTRY. But I'm really excited to have Steve Cohn of Validate.ly with us today. And we'll get into his segment in a little bit, but first, hey Steve, how are you?

I'm great, thank you. Thanks for your time.

Thanks for joining us. I just, again, for those that haven't been with us before, a couple of things I wanna point out. You see there is a chat section and definitely encourage you to use that chat section, introduce yourself, share what product you're working on, where you're from, that sort of thing. It's always fun to see people from so many different places join us. Also, you'll notice that there's a section called Ask a Question. This Q&A chat's going to be a little bit different in that I've sat through a presentation that Steve gave in the past, and I thought it was really cool. And I asked Steve to join us for this, and Steve's actually going to go through that presentation. So it'll just be the first part of our industry interview, but then we'll get into the Q&A afterwards. So if you have a question teed up, just list it in that Ask a Question section and I'll look out for it there. Also, for those of you that are, you know, joining, maybe you saw this on Twitter or LinkedIn or something like that, we'd love to have you join Product Collective. This is our community for product people. We have all sorts of stuff that help product people all the time. So you can go to productcollective.com for that. Also, hopefully we can meet you in person sometime at Industry. Industry, the product conference, is something that we've run the last few years. We're expecting about 1,000 people at our global conference in Cleveland, Ohio this fall. But also, like I mentioned, we're off to Dublin, Ireland next month, April 23rd and 24th for the first European edition of Industry. So really excited for that. We're getting down to crunch time. But I would get all that out of the way because really, y'all came here to listen to Steve and to really interact with him and learn from him. So I'd love to get right into it. Steve, thanks so much again for joining us.

Great, my pleasure.

Now, you and I connected for the first time. I, we were both at the Product Leader Summit in New York, really just a couple of months back, and I enjoyed chatting with you. But then the day after we initially met, I got to watch you give this presentation. And, you know, when you and I were talking just beforehand, thought, hey, I know this isn't how we normally have these industry interviews, but if I enjoyed that so much, why not let everybody else get to see that? So Are you still cool with getting a walkthrough of that presentation? I thought that'd be great for everybody.

Absolutely. It's about a 15-minute presentation, maybe 20 minutes, and then that should leave plenty of time for Q&A and follow-up discussion.

Awesome. Well, I enjoyed it. So, and I assume that you'll give a little bit of an intro of what you do and what you're up to validately in the presentation too.

Yeah, absolutely. So, you know, thank you very much. Did you want me to start going now, or—

Yeah, you could go right into it.

That's cool. So, uh, my background, just so you know, is I'm actually not a product manager by, by, uh, job title. I'm an entrepreneur. Um, so I've started 3 companies. I sold, uh, my first 2. Um, and I've learned a lot along the way, a lot of scar tissue. Um, a lot of, uh, uh successes and failures. I'm a very reflective person, so I, I take a lot of time to try to understand why things fail, why things succeeded. And I collected some thoughts into a presentation around discovery as well as validation. And that's a subject that I'm very, very excited about and I'm passionate about. I love— I really spent a lot of time thinking about it. And so our mutual connection is a guy named Dan Olson who put on the conference. And Dan's great and wrote Lean Product Playbook, and he's great about connecting people of like minds about how to, this exact topic area, how to build products the customers love, how to do it as efficiently and as lean as possible. And I've known Dan for a long time, and so we got connected, and so I hope you enjoy. So let me share my screen. I'll walk through this real quick, and then we'll do some Q&A. All right, so I just want to make sure everyone can, everyone can see my screen now.

Yes, I can see it.

Great.

Okay, y'all can see it in the chat, feel free to mention it too.

Great. All right, so 3 things I'm going to talk about today. So the first one is going to be, um, you know, just, um, about me. I'm going to give you 30 seconds, a little bit more detail about me, um, how to discover breakthrough innovation ideas. So I'm going to give you some ideas. You guys might already have some, but hopefully something that I give you might, might jog some ideas. I try to make it as actionable as possible, so as tactical. So this is not theoretical. These are things that I'm going to talk about today that you can do today and tomorrow without major, huge cultural changes. These are going to be tactical, actionable things. And I'm going to talk about validation, which is something that I'm very excited about. And we— I'm going to share some practices that we do. I'm going to give you a case study as well with my own company, which is called Validate.ly right now. And Validate.ly is a user research and testing platform. All right, so real quick, I talked to you about my past. So I always start with this slide because it's my family. My family is my core. As Type A people, we can get so wrapped up in, in, in success and drive and our drive. But it's really important to understand the people that kind of help you be happy and live your life. So, this is my core, but my passion is starting companies. And, you know, the first company I started, I ended up having a success. It was a messy process, but we went from a $9 million valuation when we kind of— we merged with the company early in our lifetime, and we went from a $9 million valuation to a $6 billion valuation. And so for people who haven't been on a ride like that, that's quite a ride. You know, that's a huge spike. We took some money off the table, but ended up, interestingly enough, we couldn't find a— or after I left, the team couldn't find a business model that worked, and they ended up going to zero. So it went from $9 million to $6 billion to zero, which is a heck of a ride. And you learn a lot both going up and coming down. My second startup, but in the middle, after I left that startup and things looked great, like around the point we were getting a $6 billion valuation, people were throwing money at me saying, you know, leave that company and we'll fund you to start your next company. And I didn't even have an idea. I had an idea, but I didn't have like, I didn't have a team, I didn't have a product, never anything. And I built it all the wrong way. I just started building, okay? And, um, uh, you know, I learned, uh, a tremendous— it was a failure. I always sold it, but for pennies on the dollar, and it was a failure. And, um, I probably learned more from, from the failure than the success. Um, and so what I did afterwards is I spent about a year interviewing product and design teams at some of the best companies in the world and saying How do you do this? How are you? How do you build products that customers love? What do you do? And if I had to summarize kind of what I heard, it was this great quote from Albert Einstein that says, if I had an hour to solve a problem and my life depended on the solution, I'd spend the first 55 minutes determining the proper questions to ask. But once I know the proper questions, I can solve the problem. And basically the point, you know, my takeaway from the point is is that once, you know, it's about, you know, measure twice, cut once, right? So do your research up front. Don't jump to your solution space, what Dan Olson says, you know, the solution space and the problem space. Spend a lot of time in your problem space, really understanding the pain that you're trying to solve. And that's what led me to validate Lean. I think that was a mistake that I made with my second company is I just jumped to the solution space. I wanted to build something, I had an idea, and I figured if I just do it, it would be great. But I didn't spend enough time with customers upfront. So when I say, you know, when I say I saw this tweet on Hacker News, so we have a great team and we have capital, but we can't find a good idea. And I responded, you're at, you know, you're asking the wrong question, right? You don't start with, you don't start a company with an idea. You start with a problem to solve, right? And I think that's, you know, important to understand for, you know, for product people too, whether it's a new product that you're starting a new company or a new product within an existing company. So a new big feature. At Validately, we have— we refer to features in our roadmap as, as boulders or pebbles. So pebbles are little small things that, you know, you know, are nice to have or make the use case a little bit easier and all that type of stuff. Those are great. And then boulders are big things, big initiatives. And so where do you find these boulder ideas? My argument here today, or my ideas I'm going to share with you, is that you don't actually find it by looking at your product. You find these bolder ideas, these big breakthrough ideas, by talking to customers and asking them what they do before they log into your product or after they log out of their product, right? So the idea here on the right is collaboration, and then before on the left is, I don't know, file input or upload or whatever it is. So you can take any use case, and if you actually sit down with some of your existing customers and say, just, you know, Tech, for a second, I just want to ask you a little bit about, you know, how you, how you, you know, what you do before you log into a product and then tell me about what you do after that, you know, the product. You might actually be amazed what kind of breakthrough ideas. I'm going to give you a case study with my current company. I'm going to share with you not only what we did, what we learned, but also the results. Okay, because case studies are important. So when we first started, we started with this idea of how to do faster and better research, right? So there's source testers and there's run tests, and there's moderated and unmoderated and mobile and desktop, and then there's, you know, all kinds of different things. And so we built a product to do that, and that was where we started. And we were bumping along, going nicely, growing nicely, but, you know, we plateaued a little bit. Uh, we had some churn issues at one point we were dealing with, um, but, you know, we kept building the product and we were bumping along nicely, but I didn't feel like we had something really revolutionary, really innovative. Okay, and so, um, what, what we've spent doing— and actually on the call listening in is Juan, our VP of Product and UX— and we spent a lot of time doing, uh, what's called ethnographic research. So ethnographic is, you know, means in the wild, right? So I, I like this picture, uh, not only because she's, you know, in the jungle, in the wild, uh, you know, a little play on words, but also because she actually climbed up into the tree to get an even closer look of the animal that she was observing. And I think that's really symbolic of what you need to do with ethnographic, right? So it's not just simply about going on site with the customer, but really, really getting into, at a high level, real observation at a detailed level with what those users do. And there's a great, there's a great story, maybe you've heard it with TurboTax. So, you know, when Intuit launched TurboTax, they actually had their product managers go to the big box retailers, wait on the aisle for someone who actually, you know, for someone who was about to pick up the box for TurboTax, follow that person to the line, to the checkout line after they purchased it, walk up to them and said, hey, I have a kind of a weird question to ask you. 'Can I actually go home with you and watch you install this software?' And so they did it, and they actually watched them install the software on their, on their own home laptops— actually, probably that back then it was desktops. And instead of bringing them into their lab or, you know, something like that where there's the perfect setup and perfect environment— so that's ethnographic. We did a bunch of ethnographic, and Here are some good questions to ask. And, you know, so what do you hate about how you— if it's a B2B, kind of how you use this product right now? What do you hate about your job specific to this product, using this product, right? What do you hate? What would you know? What do you like about using this product? What's missing from using this product? Do you need anything to do anything before you log into our product? Whether it's consumer or B2B, describe it, what takes time, what's frustrating, etc. What do you do after you log out, right? Describe it, what takes time, what's frustrating, okay? And when we did this, what we found is that people who used our product often were UX designers. Some product managers started using it, but they started using it after we've evolved the product. I'll share that in a second, but initially our use case was UX designers and and UX researchers. What we found is they were spending about 60% of their time not doing UX. They were actually spending a tremendous amount of their time, the bulk— I don't know if you guys can see my pointer— but on this collaboration piece. So there was definitely some setup time that we could take some time off, but really it was after they'd run a test, this collaboration piece was taking up a massive massive amounts of time. And so, so we, we studied that for about 9 months. Juan spent about 9 months interviewing customers. I, I listened in, so he had executive buy-in, um, and, uh, we, we basically got a deep, deep, deep understanding of the problem space and then started prototyping and putting prototypes in front of customers. And we evolved the value prop to not just being source testers and research, but then collaboration and reporting. So we increased our collaboration and reporting, and that became a huge part of our value prop. And we, we took over what are called insight reports and, you know, really evolved. So did that work? Did this work? Well, this is, this is our kind of revenue chart. Our monthly revenue chart. Um, and it's not on scale, but the scale is pretty decent. Um, and so when we introduced the, um, that big red mark is kind of where we introduced the collaboration stuff, and, uh, that's where we started taking off. So we really, we grew extremely fast. Um, we more than tripled revenue last year, and we actually did it while being cash flow positive. So we didn't just grow by, uh, by burning a ton of cash. We actually we're able to grow and do it smartly. And not only did we, by doing that and understanding the use case about collaboration, change our trajectory, change our value prop and how we differentiate from customers, but we also found new customer verticals. So what we found is that product teams have these challenges, marketing teams, and even interestingly enough, randomly, HR recruiting teams reached out to us and said, hey, you you know, we have, you know, we actually do a lot of interviews and we take notes and, you know, so we have this same challenge of interviewing people and then sharing what we learned and collaborating with different teams about what we learned. And so we opened up new customer verticals as well as changed the value prop simply by starting with, tell me about what you do before you log in or after you log out, and what are the pains around that, and not being locked into the artificial boundaries of the existing product at the time. Okay, so then how do you— all right, so let's say you do that. How do you validate that your solution space? Okay, and you know, and this is where I spend a tremendous amount of time, and I've learned a lot really around this, and we actually— I'm going to share with you an idea that actually we do on a, we do on a daily basis with my company. So, and we sell to large enterprises. So if we're a startup and we can sell to large enterprises with this technique, you can do it as well. And actually there's a lot of case studies. So a big focus is validating pre-build. Okay, so the biggest mistake that I think I see product people doing when they're trying to validate something is they simply ask, you know, would you use this, right? Do you want this? Yes or no? And people were like, oh yeah, it looks great. Okay. And whenever I'm in large groups, I would say, okay, what's the problem with this, with this, with this question? And we go back and forth and usually maybe after about 5 or 6, someone will get it, but sometimes no one gets it. And the answer is the problem with this question, although really this is what you want to know. The problem with this question is there's no teeth to it. So there's no cost to say yes. And in the case— and actually, there you can go even further. There's some value in saying yes just to get you off my back, right? Maybe I'm not interested in talking about it anymore. I want to end this research session. If you ask me, do you want to use it? I'll say yes, right? Because it didn't cost me anything. And so what we found is that we introduced this concept called a cost question. So it must cost the user something to say yes. So when we do our interviews, we found that there's 3 things that really can, can cost, can be a cost. One is your time. So when we do our interviews, we don't actually pay people for their research time. So people, customers will do research with us and we don't give them incentive because what we find is that if there's a real problem, if there's a real pain that we're solving, they'll give us their time for free. And if it's not a real pain, and they're not interested in this problem space, then they won't give us. And that's really what we want to know more than anything. The second is their reputation. I use Twitter as a symbol, but really it's about social sharing. So reputation can come in the form of inviting teammates. It can come in the form of, you know, social sharing for consumer products, etc. And the last is the form of budget. So one of the things that we love to do and we do, and we think is important, is we try to sell, uh, when it's still at a prototype. So we literally will work with a client, a customer, set of customers to understand the problem. We'll then go back to them and say, hey, here's a prototype of, of how we're going to solve this problem. And they say, and we say, you know, is this good? Does this match what you need? And they say yes. And then we say, okay, we are, you know, here's the price tier that it's gonna It's gonna be a new price tier for you to do this. And again, we only do this for boulders, not for pebbles. Are you willing to pre-purchase? And the answer is we get customers to pre-purchase. And so there's always skepticism in the crowd whenever I present this, they say, oh, customers would never, you know, buy this. And of course I always point to something like Kickstarter where even on consumer side, people pre-purchase on just a prototype or a video or something like that. But there's also tons of B2B examples. I know we're a B2B business. We don't have— we're not this massive company. We sell to large enterprises. Fortune 50 companies are some of our biggest customers, and they pre-purchase products that don't even exist yet either, because there are prototypes that we give a defined timeline of when they're going to be delivered, but they'll pre-purchase it based on demand. And so that type of putting that type of roadblock, that validation roadblock in place is important because it's— you want to hear no. You want to hear no in your early validation efforts because then the next question is, why is it no? So hearing that is really— no, hearing no is really important because you want to get that before you start building. Because what you might find is that's when the good stuff comes. When you say, OK, that's great. Tell me why you don't want to purchase this product, even though you said it was really great and going to value. So much value and really help you. Why are you not ready to buy it? Well, because, you know, I would need this thing and I need to get this other person to buy off on it. Oh, that's great to know. Can I, can I talk to that person right now? We're asking for reputation. Can you, can you introduce me to that person who also needs to sign off and make sure I capture their needs and use case? Right. But if you, if you, if you don't make it cost something, right, if you don't make them sign a form that says when you release this, I will purchase it. You might not get that piece, right? You might not get the real objections. So the cost question is important. And I leave you with this, you know, this validation, you know, demand rules, the golden rule, which is, you know, you are not the user, you are biased. I learned that with my second startup. And then in the golden rule, I say is, you know, what a customer says they will do, what a customer says they actually did, and what customers actually— what a customer actually does are three different things. So make it cost something to say yes. So let me repeat that. What a customer says they will do, what a customer says they actually did in the past, and what customer— but what a customer actually does are three different things. And it's not that they're intentionally trying to be misleading, But, you know, certain circumstances of a user interview can lend itself to those things being different. And so I say make it cost something to say yes. And I talk about time, your reputation, and your money. And these are all practical things that we do at Validately, and it has really changed our growth trajectory. So that is the presentation.

Awesome. That was great. I appreciate that. And actually, I have questions myself kind of teed up, if that's all right.

Fire away.

So I am super curious about, and I love the fact that you at Validate.ly, you actually sell products before they're ready, right? Like you're doing that pre-selling as a part of validating whether that product should exist. How much is enough? You know, because I could— I'm sure that there are some folks that are watching where it's like, well, maybe we can do something like that. But, you know, it's different to sell it to one person versus 10. Like, at what point do you feel like, all right, you know what, we should be rolling with this?

So I'm a big, big, big— we are— our company is a big, big, big believer in persona segments. So the answer, my belief is if you can get 10 to 12 customers within a persona to pre-purchase, then you've got something great. You've got something great. And if you really nail it, if you really nail the need, the solutions, the problem space, and you put a prototype in front of people, you can absolutely get them to pay upon delivery, to pre-purchase and say, as soon as you deliver this, I will upgrade. And buy this, right? And with some real teeth to it. And so about 10 to 12 within a specific persona group, I think lets you know that you've got something. Awesome.

You know, I'm curious about, you know, there's some companies and I've been with some of them that sort of fall in this camp where they say, oh, okay, listen to our users. All right, yeah, we'll put together a customer group, you know, we'll put together a customer advisory board and sort of that's their way of getting feedback from customers. But the reality, there are all sorts of ways to get feedback from customers. Knowing that there's all these ways that exist, what's a good mix? Like, I'm trying to think of for people that are just getting started, right? Like, they want to go back to their company and say, we should be doing more of this. What would you suggest they start? What kind of mix might be right for somebody that's just getting going?

Yeah, so I, So first of all, my belief is customers are exceptional at describing their problems and they're really poor at describing solutions. And the reason why— and they're not supposed to be great at describing solutions because they haven't thought about your product, right? They haven't, you know, they're just jumping off the top and they don't know what's technically capable. It's not their job to figure out solutions. So sometimes you're in these user interviews and they'll say, hey, these are Here's an idea. You should do this. All right. And then it's like, well, thank you. Thank you. But help me understand what problem you're trying to solve, like what problem you have that has led you to this feature idea. And so I think the first thing is making sure you understand. I'm not a fan of focus groups personally. So I think that was part of your question around, you know, customer advisory boards are typically, you know, we have like 5 people come in or whatever. I'm not a fan of focus groups. I think I remember getting a quote one time I thought was funny and accurate. You know, focus groups are great, a great way to ask 12 people a question and get one answer. I think there's a lot of groupthink in a focus group. There's a lot of, yeah, what he said, which is not really very helpful. I want to hear. And so, so one-on-one interviews are great. I think also problem with focus groups is they're so painful to— and so much effort to organize 10 people, 5 people at once. What we found when we go into the best companies, like some of the best companies in the world use our products. So we talk to, you know, Google and Amazon and Salesforce and like the— I would describe them as, you know, research, user research is in their DNA. It is not an activity that happens occasionally. It is a part of the process. It is a part of how they build everything. They, you know, it's not, it's not like a once a month, once a quarter, we need to get a customer advisory board. They have built into their organization, into their culture, a constant approach to talking to customers, putting something in front of them, testing, iterating, testing, iterating, building, testing, iterating, building, prototyping, testing, iterating. It's just, it's constant. It's not— and so, um, you know, we try to, you know, emulate those exact— that exact approach as well, because I just think that, again, it comes back to my, my views on this was formed when I when I, when I interviewed a lot of some of these great companies and how you do this, how, you know, what is it that you do. And, um, uh, you know, it just seems like it's a part of their DNA. Um, and you know, it's funny when I talk to customers, um, who don't do testing, a lot of testing and research, and they always say, well, we don't have time to test. But they always say, and I always say, well, the reason you don't have time to test is you don't test. And you're building a whole lot of stuff that no one needs and wants. And, you know, you're not a feature factory, right? Technology is not a race to see who can crank out more features, more story points like that. That's not— if story points was the goal to like figuring out how to build a great product, then it would be really easy to optimize. Everyone could do it, just crank out story points, you know, and, you know, It's not about that. That's not how you build great products that customers love. And, you know, every product person loves to say, you know, less is more. But then it's like, okay, well, that was fun to say that, but now let's just figure out how many points we can jam into this sprint, you know? And the reality is, is that when you, when you study these great product companies and you understand it's really— that's not what it's about. You're not a feature factory. And it's about really deeply understanding the problem space and building that culture of, of rapid, of constant communication, testing and iteration before you get into the engineers' hands what to build. Yeah, yeah.

Well, that's great. Well, we're at the end of time, which goes by pretty quick, but I appreciate you sharing all of this with us. Again, I enjoyed the presentation, so I was Happy to share it with everybody here. I think you shared it on your slide, but what's the best way for folks to— obviously you're on Twitter, saw that in the slide. What's the best way for folks to interact with you and get back in touch with you?

Yeah, so you can always— I can post my email address here, stephen@validate.com. Sorry, Stephen. Not. So you can always email me, stephen@validatedly.com. You can hit me up on Twitter @SPCone, and happy to discuss.

Awesome, awesome. Well, thanks again, and thanks to all of you for being here.

I appreciate it.

Hopefully we'll see you in a future industry interview. I'm actually— the next one coming up was with Ash Morya, and which we're excited. He's a speaker coming up at Industry in October, so we're pumped to have him. Hopefully you could listen. We're actually coming towards the end of our season on our podcast, Rocketship.fm, so hope you'll check that out as well. And yeah, hopefully we'll see some of you in Slack in the Product Collective community. So with that, thanks again, and we'll see you all soon.

Great, thank you.