Testing w/ Users to Achieve Product-Market Fit w/ Mike Mace, VP of Product Marketing at UserTesting

Hey everybody, it's Mike Belcedo here, co-founder of Product Collective. Thank you for bearing with us. We still might be having issues, so, so sorry for that if that's the case. I haven't been able to hear our guest Mike, and Mike has awesome info to share with us today, so it will be frustrating if you all can't hear him too. But here's what we're going to do. We're going to give this a go. I'm going to— it could be that maybe I just can't hear him on my end. I actually have Mike pulled up on my phone here. Mike, if you don't mind, just say hello to everybody. We're going to see if anybody else can hear you.

Hello, can you hear me? Hopefully you can. This is so frustrating. Oh, you can hear me.

Everybody can hear. All right. So you want to know what? This is my fault because for some reason on my computer, it's not allowing me to hear Mike. But if you can all hear Mike, that's totally all right. I did. I unplugged and plugged back in. But thank you for that suggestion. It's okay though, we're gonna, we're gonna give this a go. Thank you for bearing with us here. I'm excited for this. I'm gonna keep Mike though, if that's okay, on my phone, so you might hear a little bit in the background. Maybe I'll mute my mic when he's talking, but hopefully that can work out really well. So thanks again. I'm gonna keep all of the introductions really short, but the only thing I'm gonna say is we have— we do these product launch sessions twice a month. We have some already planned. So we have Ash Maurya, who he's one of the speakers at Industry. So that's coming up. That's going to be great. Varun, who is a PhD candidate, is one of the only PhD candidates in the country focusing on product management. So he actually did some academic research that he's going to share with us. So that'll be a little bit of a different product launch session. But I'm really excited for that. And, and for those of you that haven't been a part of the Product Collective community, definitely encourage you to join. We have a Slack channel of 4,000 product people. We'll let you know about, you know, we have a newsletter goes out every week. We'll let you know about upcoming events. We have a few of those coming up with our industry conferences. So anyway, with all that out of the way, I do want to welcome Mike. Mike, thank you so much for being here.

Oh, my pleasure. And thank you everybody for sticking around through these technical difficulties. It's moments like these when I'm embarrassed to work in the tech industry.

Well, you wanna know what? Again, I believe it's not your fault, Mike. So, it's okay.

It's all of us. It's okay.

I'll check that out on my end. But let's get right into it. You're with a company called UserTesting, but obviously, you've become a bit of an expert about user testing, the act of user testing. And I'm just curious personally, I've been involved with early-stage startups. I've been involved with companies that are a little bit larger, sort of at the scale-up stage. What stage should user testing, you know, be an activity for us as product people? Maybe you could just talk a little bit about that.

Yeah, glad to. And it's a great question to start off with because a lot of people get confused. You know, they'll hear the word testing in our name, and what you usually tend to assume when you hear testing is, oh, okay, I should use this for beta testing, QA testing at the end of the development process. You wait, you do all the design on your app and everything, you get it built, and then you put it into user tests. That's actually— a couple of people are saying, okay, some people can hear me and some people can't. Sorry about that, guys, for those who can't, I'll keep talking for those who can. Actually, doing user tests at the end of development is absolutely the wrong thing to do. It's a terrible mistake because this is not like QA testing. You're really using user tests as a quick and convenient way to get inside the heads of your customers. And so what you want to do is you want to do testing as early as possible in the prototype phase. You can even use user tests when you're doing agile discovery. So before you've even figured out what the product is, You just want to interview customers about their problems and their lifestyles and what sort of things are they trying to deal with. And the user tests, um, don't even think of them as tests. Think of them as interviews. They're a really convenient way to get feedback from your target customers, um, very, very early. So you use it for discovery right at the start to get feedback from customers. And I can go into detail on exactly how to use it. You use it in the prototype stage to test, uh, mockups. You can even do sketches and stuff like that to get reactions to it. If you do that right, you can perfect the user experience before you even start to code, which then saves you time on development so you're not going back and changing things. If you do find for other reasons that you need to change part of the user experience during development, you can do some quick user tests to validate that you've made the right changes, that you haven't screwed something up. If you do it right, you do zero of these user tests after development. You can just focus on finding bugs. You don't have to sweat whether you got the user experience right because you know that you did. The other time you use it is after release when you start getting analytics back and you'll see something that you can't understand what's happening, like, oh, they got to the third screen and then they bailed. Why did they do that? They should have kept going. You could either guess and do a bunch of A/B tests and spend 6 months, or you can do some, you can do some quick user tests, and those will enable you to then very rapidly find out what's going on, figure out what you need to fix, and then get back to business. So rather than using this after development, it's best to use this before and during and then after release.

Thank you. That's awesome. And let me ask you this. Is there a time, like, as somebody that is doing this, like before the release and after the release, just like you had mentioned, How do we know what's enough? Like, is it possible to do too much? Maybe you could talk a little bit about that.

I think it's, it's possible to get diminishing returns. So what'll happen is in— you will, you will eventually get to the point where you start getting the same feedback. And so what I recommend, and this is one of the differences between this and traditional research, is Ooh, it says something disconnected, but I'm gonna just keep going. So sometimes people think this is like traditional market research, and so what I'm gonna do is I'm gonna do 50 tests because we need something that's gonna be statistically significant. And what you'll find when you go to watch all of those videos is that in fact, you, you start seeing the same things after you watch 5 of them because people are running into the same problems. And so the best thing to do is to do this in really little batches, you know, maybe 4 or 5 people at a time. Watch those, and as soon as you start to get pretty much the same feedback, um, stop. You found it. You found what you need to work on. Go fix it and then do some more tests. So it's really tiny iterations. It works well with agile. Because you can set these things up and usually get videos back within a couple hours. So you can literally do something in the morning, test it out over lunch, and have some videos to watch when you get back. And so you can do really rapid iterations. And so tiny little samples to verify that you're doing the right thing and then move forward to the, to the next thing. That's the, that's the right sort of pattern to hit.

That's great.

And by the way, I want to know— I can't hear you.

Just meant—

oh dang it, so our phone call dropped. We were using the phone call to try to do this. So what I need you to do— speaking the host now, everybody else stay on— I need you to call me back on your phone. Okay, okay, so he's gonna do that. This is, this is fun, it's an adventure. So anyway guys, what you want to do is— there we go, all right. So try it again.

Okay, are you— can, can you hear me, Mike? Great. Yeah, awesome. Thank you. Um, and everybody else, I, I think, I think everybody can hear me. So, um, again, sorry for the issues there. What I was going to also recommend, everybody, you see an ask a question section. Um, if you have questions, and some of you are doing this, which is great, I might go to some of this right now. If you have questions for Mike, I have loads, but I'd rather get to your questions. So you can ask them in the Ask a Question section. And I'd love to go to one right now. Somebody had asked the question, do you have any tips for user testing with users that they're not very enthusiastic about giving feedback? Maybe they're not so tech savvy. Like, how can we help pull out those awesome tidbits that we need from folks like that?

Yeah, it's, you know, a lot of times what people do is, even if they're, even if the people who are testing for you are not, are enthusiastic, they don't know how to speak aloud or to think aloud. They don't know how to speak aloud. But what they'll do is they'll stare at something and kind of think, and then they'll do something. And what you really want to do is coach them to just just talk to me as you're going along. And that's whether— the nice thing is, you know, if you're doing it through user testing itself, we kind of coach them up ahead of time and they know that they need to behave this way. And so hopefully they're going to do the right thing. And if not, you can complain to us and we'll find you another tester. But if you're doing it yourself, what you want to do is just brief them ahead of time to just give me a stream of consciousness, talk about what you're thinking, talk about what you're seeing. If you have any questions, get them to do that. And if it's a live session, you know, you're just like sitting across the table from somebody or you're on video conference, you can And you can just keep prompting them like that. Hey, tell me what you're thinking. You know, give me what's going through your head. And usually, they'll start to open up a bit. Because most people, they like to, they like to share their ideas. You know, it feels flattering to them. And as long as you make them understand that their thinking and their thought process is important to you, then they will usually want to open up for it. Frankly, the other thing is, It also helps if you're compensating them a tiny bit, even if it's just like a $5 Starbucks card or, you know, $10 gift certificate on Amazon or something like that. People will feel like because you're doing something for them, they will feel obligated to kind of go out of their way to try to help you, even if it's not a lot of money.

That's great. You know, I saw a question come in and this is a I can empathize with this one. I can relate to it a lot. It was, hey, look, I'm a product person. I understand the value of user testing. But our CEO, you know, they, they, you know, maybe they're in their mind, it's like, well, hey, if we ask people this stuff, they might think we don't know what we're doing. Like, we should know this stuff. So let's not risk it. Let's not ask them. Is there a way to help sort of manage up within the organization so that we could get the value out of user testing, but also, you know, get some champions internally for that?

Yep, yep. So a couple of thoughts. First of all, in a lot of cases, the resistance is that you'll have somebody who's in a senior position who thinks they know everything. You know, like, I've been working here for years, and I understand the customers better than anybody else. Let's just, you know, save some time. Let's just build what I say we should build. And The best way to deal with that is find some way to run a couple of the tests, and inevitably they will, they'll come back with something that you didn't know because you were making an assumption about the customers or something that was obvious to you is not obvious to them. And that happens a lot where you'll have some assumptions that you didn't realize you were making and then you discover these things. So the first one is, you know, and I've had this conversation with some product managers who said, you know, I really thought I knew our customers. And then I did the test and oh my God, I found out it was XYZ and we had no idea and we were so embarrassed. It's like, ah, I can't believe it. So that one is just find a way to get a few of the tests. And usually it's the video of the people themselves talking. That's the key thing. When they say, oh, hey, I don't, you know, I don't understand this, or I thought this was so-and-so, or I thought that icon meant save when you thought it was intuitive that it meant delete, you know, and look what happened in the app. So just one of those experiences of, oh, I've been humbled by the actual behavior and thinking of real customers, usually is enough to convert people. Now, the second one is worrying that you're going to look bad because you're asking people to give you feedback when you're supposed to know it. And what I recommend is you don't ask them. The best way to do user tests is you just show and you say, tell me what you think. What would you do when you see this? You know, I just want to see how you react to this. You know, just, just talk me through it so that you're not losing any face. You know, you're not, because you haven't said that you're, that you're not knowledgeable. What you're saying is, oh, hey, I'm interested in you. I want to hear from you. I care about you. And usually people will view that as an indication that your organization is a caring organization, that you're open to your customers. And so you actually gain credibility by doing this rather than, than having it look like you don't know what you're doing or something like that.

That's great. That's great. I saw another question come through and then in the chat some folks are helping answer that question, which is just, hey, if our users are busy, our customers are busy, how do you, how do you get them to find the time, you know, to actually do a user test? So just ideas for getting folks. And I know your company can help with that. If people are doing these on their own though, do you have other ideas? And I want to call out in the chat, somebody had mentioned that they knew a UX person that will go to Starbucks and put a sign on the table that says, I'll buy your drink for 5 minutes of discussion. I thought that was pretty— that's one interesting way to get folks. Mike, I'm just curious, do you have other ideas as well?

Sure. Yeah, well, the simple self-serving one is like you said, you know, come to us and we do all that for you. That's why we're in business. But Yeah, the Starbucks test is kind of a standard thing to do. And what you can do actually is get a little pile of Starbucks gift cards so that they're sitting right next to you so that people see that you're going to hand this, you know, hey, let's talk for a minute. I want you to, you know, I want you to look at this. Let me hand you my smartphone. I want you to look at this app, or I want to ask you some questions. Just sit down and talk with me. So the plus of that is it's, it's very convenient to set up, you know. Second, it's a proven methodology. Downside, you can't control the demographics. If your customers are not an average Starbucks person, you're not necessarily getting the people that you want to get. Second thing is it does consume a fair amount of your time because you're sitting there working with each person that you're getting to and interacting with them. You know, one of the nice things about the user tests that we do is that they're asynchronous. So you come to us, you define the, the tasks you want performed, the feedback that you want to get from people, and then that's sent off. You know, our system takes care of it. You just get the videos back and you can watch the videos later at your convenience. So it's a balance, but definitely the Starbucks thing, you know, I've talked with companies who've done it other places. So in San Francisco, for instance, they would do it at the Ferry Building because they know there's a lot of traffic coming through there. And apparently the Ferry Building people won't throw you out when you're doing it. You know, you got to be sure you're not going to get kicked out. But, you know, Starbucks, the nice thing is if you bought a bunch of their their gift cards and you're giving them out to people, they don't care. You're giving them money, so that's fine. You basically rented the table.

I, I remember one time for me, we, we had a product that I was managing that was a fitness tracker, but it was, it was for the corporate market. And, um, we did user testing at a mall food court that it's just right in the area of Cleveland where the only people that are really, uh, you know, business people that are out at lunch. So it's like keeping your target market in mind certainly helps too. I imagine.

Yep.

I had a couple of my own questions, but I wanna make sure I got to one that got some votes here as well, which is just, you had mentioned before, hey, it's great to do user tests before you're releasing and then after releasing. Is it better to do it with the same people or do you try to get different people at different stages? What are your thoughts on that?

I like to mix it up as much as I can. And we call that fresh eyes. What'll happen is if you test on the same people over and over, eventually they turn into insiders. You know, so they've learned from previous versions of your app, your website, how it works, and they will become more and more like your own employees. I mean, if you want somebody who already knows your product to give you feedback on the product, you can just talk to the person who works next to you, you know, or talk to the sales reps or something like that. It's usually best to try to get fresh feedback and from people who've never seen it before. And because presumably you fixed some problem that was aimed at people who are new to it, haven't seen it before. Oh, they can't figure out section 3 or something like that. And then you want other fresh people to come in and see how it works. So it's okay if it's something where you had to do a lot of recruitment or they need the context of previously using it, or if you're after your current customers. You know, a lot of companies that I talk to say, no, no, no, our, you know, we've got a big learning curve. I need to go back to people who've already gone through the learning curve. That's fine, do what you want. But if all other things are equal, it's better to get completely fresh people.

That's great. And I know we're— even though we got a late start, we're kind of coming up towards the end of their time. So I would— I was curious, Mike, if you could share common mistakes that you see, because we're all learning at this. So if there are things that you've seen happen over and over again, you're like, oh man, I wish wish people would know, like, this isn't the way to go. Maybe, maybe that would help us. And, um, and, and also there's sort of like, if you want people to walk away with one thing from this product launch, you know, what, what would you want that to be?

Yep, good. So number one, um, you don't have to test on that many people. You know, I can't tell you we have to have— well, at least it has to be at least 40 or it's not going to be statistically significant. And then you've got 40 15-minute videos that you need to watch, and you're going to find that they're all repeating each other anyway, and it's just, it's a really bad waste of time. The second thing is testing too late, you know, and again, you know, I had a heartbreaking thing when we were talking with one of our customers who had shipped a mobile app, and, and they'd done some tests that satisfying to customers. And I said to them, you know, hey, you know, you had this research that showed that it didn't, it didn't work. Why did you guys end up shipping the app? And they said, well, because we did the tests at the end and we didn't have time to fix the problem because we had a schedule. We had to ship the app. It was almost done. And so we just shipped it. And it's like, gosh, you could have, you could have saved all that effort if you'd just done those tests a little bit earlier on a prototype. So, so test early, test in tiny increments, and test often. That's the most important stuff. And really don't think of it as testing. Think of it as getting feedback. That's really the key thing. Now there was a second part to your question and I forgot what it was. I apologize.

No, just like as we're wrapping up here, it's just what's the one thing you want to make sure that everybody remembers when they walk away? Sure.

So I think the other thing is this is a really, really flexible mechanism. So You can use what we call user tests to do anything from interviews to even ethnographics. So for instance, we had a company that, um, they were one of the clothing companies aimed at, uh, teenage women. And they said, we, we'd really like somebody to just take their smartphone and go into their closet and talk to us about what's your favorite clothing and why, where did you buy it? Why did you buy it? What makes you feel good about it? Cause we just want to know how they think. That is super easy to do with our systems. You don't think of it cause you think, oh, I'm going to use this to test a user interface, but it's really easy to set up a test where you just say, Talk to me about this subject. You know, I'm not looking for feedback on a website. I just want you to tell me what you think about this, or take your smartphone and record, show me how you do X, or show me what location is X. And so a lot of that stuff that you would think have to be really complex ethnographic studies with a professional researcher who goes out with a camera and lives with somebody for 3 weeks and stuff like that, you can actually get that feedback super fast and super easy. And just be moving on with your insights. And so I encourage you to really take advantage of the flexibility of these platforms, whether it's through us or some other company. You can get so much insight so quickly now that there's really, there's really no excuse for not getting the insight.

Well, Mike, I really appreciate you spending the time and sharing with all of us. Thank you so much for being here. And thank all of you for being with us as well. If I would love for you to check out other upcoming product launch sessions, Also, if you're a podcast listener, the official Product Collective podcast is Rocketship.fm. We're in the middle of an awesome season right now. I also saw the host of the Everyday Innovator podcast, Chad McAllister, was on as well. So check out that podcast. That's a good one as well. And, and yeah, thank you again, Mike. And hopefully we'll get to chat soon and I'll see everybody else at the next product launch session. Thanks, everybody.

Thanks, everybody.