Validating your Alpha and Beta Products w/ Emily Hosselman, Marketing Director at Centercode

Hey everybody, it's Mike Bellocito from Product Collective, and we're here for another product launch session. So I'm super psyched that you're all with me today. This is a topic that I've been looking forward to because it's something that I'm no expert on myself, so I definitely want to learn a lot today, and hopefully everybody that's here, we all can learn as well. And I have a special guest, you could already see her, Emily Hasselman of Centercode, but Before we get into meeting Emily and getting to learn a little bit more about what she's up to and diving right into the topic, just a few sort of updates, or at least sort of announcements, I guess you could say, particularly for those that are joining us for the very first time. These product launch sessions, they're meant to be interactive in some way. Like, we— I have questions for Emily, but I'd love to take questions from the audience too. So there's a special questions and answers section right in the platform there. If you have questions for Emily, feel free to enter them there at any time. If you see questions that other people are entering that you like, feel free to upvote them. I'll be monitoring that from time to time, and so I'll pull questions in sometimes when they're relevant, sometimes I'll wait till the end, but definitely do that because it's a perfect opportunity to ask anything that's on your mind there. Also, just a couple of sort of upcoming things that I want to let everybody know about. We have another product launch session that's coming up August 2nd, right at 1 PM Eastern, and I'm excited for that one too. We'll have Gene Ngo from Pentaho, and the whole topic there is when customers need analytics. So I know being product people, all of us, you know, oftentimes we're asked to look towards the data and analytics is a part of that. So that should be a great session. Also, and you can see already a link, there's that green button with the link. But we launched our podcast a few weeks back. It's called Build Launch Scale. I'm really excited for it. I think we have 6 episodes that are already live. We're releasing them once every 2 weeks. So check that out. I'd love it if you took a listen. If you happen to like it, you know, tell your friends about it, maybe give it an awesome review. That would be really helpful. But with all of that, well, and of course I can't forget, you know, if any of you want to meet other awesome product people in person, Of course, we have our conference coming up in now less than 60 days, IND-SUM. So, you could go to indsum.com, I-N-D-S-U-M dot com. I actually put it in the chat there if you want to learn more about that, but it's been— the momentum's been great. I think we have about 600 people that are slated to attend. We might even get higher than that. So, that should be great. It's mid-September, 14th and 15th. Actually, the working sessions, which are the pre-conference workshops, those get started on the 13th. So if any of you have questions about industry, you could always email me at mike@productcollective.com. Also, one other thing, if there's any— just so you know, all of these sessions are recorded. So these sessions, just immediately afterwards, you could use the same URL and you can actually pull this up. It'll just be there waiting for you to download. So if you ever have any issues or anything like that, feel free to access the recording afterwards. By the way, I noticed, yeah, it— so also another, if you do have any issues, usually a simple reset works. Hopefully you all can see me and hear me though, because we are going to get started. And I want to welcome our guest Emily, and Emily is from Cleveland. By the way, I'm wearing my Cleveland shirt with industry coming up so soon. It says, I liked Cleveland before it was cool. I'm coming to you from Cleveland, Ohio, but Emily, welcome. Thanks a lot for joining us.

Yeah, I'm happy to be here.

And yeah, we were just talking about it. You were from, born and raised here in the Cleveland area, right?

I am, yeah. I love Cleveland and my family's still there, but now I live in sunny Southern California.

Well, Southern California is not a bad place to be, that's for sure. So I wanted to get right into the topic, but first, maybe you could just give everybody a little bit of a snapshot of, you know, the work that you do at Centercode and especially, you know, how you, how you work with folks on helping out with alpha and beta products too.

Yeah, absolutely. So I've been with Centercode for 5 years. We are a customer validation company. So we help tech companies get customer validation of their products through alpha, beta, field testing. That kind of in-the-wild testing. So we have a software platform that companies use to run their programs themselves. So companies like GoPro and Fitbit have massive beta testing programs built on our platform. And then we have a services group that does it all for them. So we have learned a ton of great lessons over the years on how to run these kinds of tests properly. And, and I'm going to be talking about a lot of those tactics today. Today, but I've been with the company for 5 years, like I said, and I've written on this topic, I've spoken on this topic. I am very passionate about testing your products with customers and making them better and making them amazing for launch. So I am super excited to talk with you today.

Awesome. Well, yeah, I want to, you know, one thing that I was curious about, because I love testing with customers too, getting that customer input, we all know it's so important. In the whole product process. But, you know, sometimes when we talk about our alpha or our beta, like those words get tossed around sometimes, maybe even interchangeably, probably when they shouldn't be. What— maybe you could just start off, what is the difference between the alpha and the beta? Why is it important to make that distinction?

Yeah, absolutely. We see a lot of companies that kind of just paint the whole process with a single brush. So they call it alpha testing, they call it beta testing, and, and then they just go. So alpha and beta testing are two very distinct methodologies. Alpha, they, they have some similarities. So they're all about getting real products into the hands of real people in the real world and then using it over time and giving feedback on that experience, but they achieve very different things. So alpha testing happens a bit earlier. It, the product might not even be feature complete and you're really focused on quality and evaluating the stability of your product. So you're looking for major bugs, massive usability issues, interoperability challenges, but you, you have more technical users testing it. So maybe super users or even internal employees. Some companies use the alpha phase for dogfooding with their own employees. The beta phase, on the other hand, you really need to get that into the hands of your target market because you want to evaluate customer acceptance and customer satisfaction with the entire product. So you need to be feature complete. You want to walk testers through the product and, and they get the full experience so they can give feedback on not only quality and interoperability issues, but, you know, do they like the feature set that is there? Do they, do they use it properly? You know, are there parts of the onboarding experience that are confusing? And there are other methodologies too. So there's field testing, which is more natural use over time and feature adoption. And that gives you backend data for machine learning and that kind of thing. And then there's market testing as well, which is comparing different products in the field side by side. So if you want to see how you stack up to the competition, or you're trying to delve into a market, you want to see what's already there, you do a side-by-side market test, but they all involve out in the wild, real people using it over time and, and giving feedback. So they're, they're really powerful stages if you do them properly, but a lot of companies, like you said, either don't have processes for it or kind of jam it all into one and end up getting kind of frustrated with the results.

Yeah, for sure. Well, and you started to get into kind of what makes for a good alpha user, beta user, at least for some companies, you know, how they use those stages. Maybe can you talk more about that specifically? I'm interested in like, what are those characteristics that make up the perfect alpha customer, the perfect beta customer?

Yeah, and that, that actually really depends on the type of product that we're talking about. And we have a poll down at the bottom if you guys have a chance to respond about the types of products that you guys manage, and that'll give me a good idea of kind of how to angle the responses as I'm answering some of these questions. So right now it looks like we're heavily on the B2B software side, which is an incredibly challenging type of product to recruit testers for. So I'm gonna angle some of my advice kind of that, that direction, but I'll check back in on the poll as more people answer. But really what you're looking for is enthusiasm, attention to detail, and then good communication skills. So if, if you have a tester giving feedback and they're like, all right, the, the product didn't install properly, and that's the extent of their bug report, that's not helpful, right? You need context, you need detail. So you're looking for testers who are willing to kind of put in that time, give you a complete picture, give you detailed feedback. And when I say enthusiasm, don't mistake that for positivity. You aren't trying to pull in all your super fans into your beta or alpha program, because all they're going to do is say, I'm so excited, I can't wait to use these features, your product is amazing. But you really want to get feedback from people who maybe aren't major users or, or aren't maybe the most positive about your product, but really passionate about your product type or passionate about how your product fits into their life and are going to have really strong and detailed opinions about how they use it on a day-to-day basis. But in terms of kind of the differences for alpha tests, you're looking for technical people. You want people who are going to be okay using a feature-incomplete product. They're not going to get frustrated the moment that they run into a bug and just throw up their hands and say, well, too bad, can't use it. They really need to, to be willing to put it through the wringer. Then on the beta side, you're looking for those passionate people that also really fit the, the different demographic and technical requirements of your target market. So if that's not super technical users, or maybe that's only 20% of your target market, then you need to find some non-technical users too, who are really going to be able to say, okay, this, this got really confusing, or, you know, I don't understand how this feature is supposed to work. I don't understand installation, and get feedback about that experience that you wouldn't get from the technical users. That, that answers the question.

That's super helpful. I mean, one of the things popping into my mind is, where do you then find these alpha or beta users? I mean, I know if they're already using our product today, you know, maybe that's a little bit easier. We could kind of go to our customers, but particularly if maybe they're not, like we need to start to get them, you know, on board.

How does that work? Yeah. So the, the first piece there I would say is you don't want to start from scratch every time. You want a standing community of people you can pull from. Otherwise you're reinventing the wheel every time you're trying to get that message out there fresh every time. And it can take you more time to find the people than it does to actually run the test, which makes it start to feel really frustrating. So we recommend for all our customers to have a standing community that they're always pulling people into, always profiling people so that they can run to those technical users, you know, the moment they have an alpha build and pull them in and start going in, you know, a day or two instead of it taking a month to find them. In terms of how to find them, like you said, if you, you have a customer base, that's a great place to start. I'd work with your support team and your sales team to work it into your process. So if you have an onboarding process, work beta into that, a beta signup opportunity. If you have a support process, work that in. And once they support, uh, finish up a ticket, say, hey, would you like to join our beta program and give feedback on future versions? You know, we'd love your input. Or, hey, you're a new customer. We, we'd like to make our product better. Would you like to opt into our community? And building that into kind of some of your other customer experience moments can help you build that community. Now that said, you don't just want existing customers in your beta program because then you're missing out on the fresh faces, the people who can give you feedback on first impressions, who can talk to you about the onboarding experience, and all of your customers, that's, that's already passed. So you need to be out there. As well, putting it up on your social media, putting it on your website, going through local trade organizations, you know, and, and then there are also organizations and communities that exist. So for example, we have a community of 200,000 pre-profiled people, and on our website BetaBound, which is our homepage for our community, we will announce any company's beta test for free. as a way for them to get great profiled testers that we know are enthusiastic and in the right mindset. And that's just kind of a give back to, to the overall community and strengthening of the community thing that we do. So there are other organizations out there like that that can help you find, you know, connect you with their communities. So it needs to be an ongoing effort. You need to kind of think through the other touch points that you have, but over time, building that community will make it so much easier so, so much easier to find the right people quickly when you need them. Yeah.

And thanks for sharing that resource. I mean, that, yeah, that's a good place for a lot of people to start too, it sounds like.

Yeah, it's, it's really helpful and our beta testers are amazing. And so I highly recommend that people take advantage of that.

That's awesome. Now let's say we found the people, we have them, we're about to start our testing and whether it's at the alpha or beta stage, What, what kind of feedback should we be collecting and how do we differentiate the, the stuff that we absolutely need to get? Like, this is the feedback we have to have versus the nice-to-haves really, right?

Yeah. And this is another big challenge that companies face because they try and shove it all in at once. Right. And they go, all right, well, we'll put it out there and we'll start getting feedback and then, and then we'll, we'll kind of figure it out later. And that can be, A, overwhelming, because you start to get all this feedback and you don't really have a plan for it. And B, really frustrating because now you're saying, well, I kind of got some feedback on this objective, but not really. And maybe it's not enough to actually make a call or a decision. And so it's a, it's a big challenge. And I would say first, it depends on the objectives of the test. You want to stay focused. So if you're running an alpha test, you're focused on quality and you really need to focus your, the feedback you're collecting around evaluating the quality, stability, and operability of your product. If you're running a beta test, that's where you're focused on, um, customer satisfaction, customer acceptance, and you're collecting feedback around that. Now, that's not to say you can't collect a wide variety of types of feedback. So, and that's, that's kind of what makes customer validation different than say, you know, a usability test where you are watching a screen capture of how they navigate through the product. And that's the single way that you're getting feedback. In a customer validation test, you're getting bug reports, feature requests, journals, surveys, tasks, online discussions. So you can attack an objective from multiple different angles, but you're focusing the feedback that you get to achieve those specific objectives. So if you're writing a survey, you want to write it around quality, if, if you're running an alpha test and you want to write it around customer acceptance, if you're running a beta test. If you start to try and cross those objectives, that's where you start to really muddy the message and your ability to actually make good recommendations afterwards.

Got it. Got it. And I was going to ask a question actually before I do that. Just another reminder, if you all have questions for Emily, feel free to write in the question and answer section. I'll keep checking back there. So keep that on your mind. But before that, you know, so let's say we're asking the questions, we know, you know, we're getting the feedback that we think we need, we're specifically now, you know, focused on the right kind of feedback, but when is enough enough? I mean, I know for some people that's a big challenge to say, okay, we, we can now move past this stage because we've collected enough feedback. How do you— yeah, like what advice would you give to somebody on that?

Yeah, there's definitely a tapering off point, and it's different depending on how you're your participation rates are and what exactly you're trying to achieve. So it also depends on if you're doing multiple releases during your beta phase or your alpha phase. So this is actually most often with alpha, especially if you're running an agile alpha, alpha program. So you might be doing releases every 2 weeks, in which case you can keep going with that cycle as long as you're doing new releases. But after you have a build in the field for about 2 weeks or so, then you're going to start to see kind of a tapering off of feedback, and, and that's when it starts to become less valuable. With the beta phase, it's, it's probably going to take a couple weeks for you to walk your customers through the entire product. We recommend kind of having a different focus for each week, and then you really can make sure that your testers are giving that feature, um, the, their focus and attention, um, for that entire period. So that could, if you have a complex enough product, that could take 2 months. Um, if you have a really simple product or maybe you're just putting out a couple of new features, that could be a week or 2. Then if you're talking about field tests, we have customers running field tests that can last up to a year because they're trying to get really, um, a lot of data backend telematics to feed their machine learning algorithms to see how people use the product over time. So really what it comes down to is you want to be able to make strong quantifiable product recommendations at the end of your test. And that's true for alpha, beta, field, any customer validation test you're running. So you need to get enough feedback that you can make, confidently make those recommendations and have the data to back it up. And if you're not at the point yet where you feel strongly that you can make those recommendations, then, then more feedback still needs to come in.

Yeah, no, I appreciate that. And thanks for bearing with me when I, when I went dark here.

No worries.

By the way, there are a couple questions coming in, so I have a couple more, but might as well take a couple from the audience right here.

Yeah.

For— there's one question about specifically engineers on the team having a tough time understanding the differences between alpha and beta. And so the question was just, how can we be more— how can we effectively communicate better to that team, to the team of engineers, the good validation strategies so that those technical folks, they can understand the differences? Do you have any advice for somebody like that?

Yeah, and, and I think part of it is getting them involved early and really understanding what they need to to be able to do their job well. So, um, our bug reports, for example, are incredibly detailed, and the way we do that is by collecting a bunch of information about their technical environments— users' technical environments— first at the beginning of the test, and then attaching all that information to bug reports during the test, and then sending it directly to Jira, um, so that the technical people don't even necessarily have to touch the product, all the information is being sent to them in kind of a carefully curated and complete way. But having kind of the context and having, having it tied to kind of their objectives and, and their language in their world is really helpful. So that's why we like to kind of bring them in early, say, okay, how do you guys look at bugs, for example? How do you define a blocking issue versus a stopping issue. So we have blocking issues, which is a customer can't use the product at all. We have stopping issues, which is they can't use a feature, and then we have lower priority issues from there. So having them help you kind of craft some of those definitions can make it more meaningful to them when the feedback starts coming in. And then also, I, I think it's, it's valuable to show them kind of some of, of the data that backs it up. So we do impact scoring on every piece of feedback that comes in for us. So we look at the popularity of the feedback and then some innate things like severity or, or the feature that it's touching. And we combine that together to provide an impact score. And then they're not just looking at a list of bugs, but they're able to look at, okay, these are the top 10 bugs that we feel are going to have the most impact on, on the usability of the product. So this is where we feel we need to focus our time. And if you can start kind of providing some quantifiable, um, heft behind that, instead of just saying, all right, we got these bugs, these are the ones we say we have to fix, that can provide something that's a little easier for them to grasp when, when they're given this feedback from kind of this mass of people that, that aren't professionals in their eyes.

Yeah, got it. I appreciate that. Well, here's another question that came in and maybe it's along similar lines, but it was more about, Hey, sometimes you will go through the process, we'll give our testers an end date, but then the software group, they don't want to necessarily cut it off at that end date. So it just keeps getting extended and extended. What's the, what's the best advice in that for that situation?

Yeah. So there, there are a couple of different ways to handle that because we, we obviously experiencing that as well with our customers. You know, we were about to come to the end of the test, we've set up all the expectations and And they say, oh, wait, wait, wait, we need this other thing, or, you know, we have a new build or something like that. So there are a couple of different ways that you can kind of handle it. A, planning from the beginning for that can be really helpful and having a very clear, okay, this is what we're doing each week. This is how we're making sure we're getting complete coverage of the product and keeping them involved in that throughout to say, hey, is this still the right schedule? Is this still the right track so that that doesn't slip? Sneak up on you can be really helpful. But phases are kind of a fact of life with this sort of thing. So if you've been running a test for 8 weeks and then all of a sudden your team turns around and they're like, all right, we want to run for another 8 weeks, that you risk burning out your testers if your testers thought that they were done. So either what you're going to have to do is increase the incentive. We give thank you gifts to all of our testers that provide feedback. So that they feel incentivized to do, do another round. And then you need to build up some good messaging to that tester pool and say, all right, the feedback you guys have given us has been amazing. We've made some really important changes to the product. And now we want that feedback to make sure that we've, we've really heard you guys. And so we'd like to do another round, have them opt back into that. The other option is to end that test and do another phase with completely fresh faces. Say, all right, we're going to bring in some new blood. We're going to, you know, have them regress those fixes from the last group, but bring in some new people that are excited so that you can have that fresh enthusiasm from the, from the start. So it is kind of a fact of life. It's something you're probably going to have to deal with. But on the plus side, that means that your software group is getting value out of this feedback. So you want to make sure that you're, you're leaving yourself open to, okay, that was valuable, that was great, let's get more value out of it so that, that way you can justify a longer test next time and kind of start it from the beginning so that it doesn't sneak up on you.

Awesome, thank you. I'm gonna do one more audience question, then I'll have another sort of the final question that I did wanna ask you before we wrap up. But this question is about, the customer's likelihood to purchase and whether that plays a role in this. So the question is, what specific questions or scales might you recommend when you're doing product testing with customers to quantify whether they have a likelihood to buy? Does that have a place in the testing phase? Maybe you could talk a little bit about that.

Yeah, so that's, that's a question we get a lot from our customers as well. And that is a really tough thing to quantify in that environment for a couple of reasons. One, they've already experienced the product and they've experienced it in a curated way. So on the one hand, they're experiencing a product that isn't perfect. They— it might not be feature complete. It might not, you know, look the way your product is going to finally. So, and it might have a frustrating experience because that's what you're trying to polish out. So we generally don't recommend having sales leads be a core of your beta testing group, because that could really, hurt the sales process. If, if the experience isn't quite where you want it to be and sales is trying to close the deal, it could end up hurting you more than helping you. And if you're talking about trying to find price points and that kind of thing, you've probably walked them through a curated experience with the product. So they have, they don't have that natural experience with your marketing messaging and to give you an unbiased idea of whether they would buy it and for how much. So we generally don't recommend asking for, at the end of a test, to say, how much would you pay for this product? How likely would you be to buy this product? Because they've already experienced the product, which customers buying your product won't have that benefit. So you can't really trust that feedback to, to be accurate to what the experience would be. So unless you're testing out, say, a trial version or what your, your free trial would be, something like that, then you have more of a leg to stand on to say, okay, now you've, you've had this experience like a free trial customer would be. How, how likely would you be afterwards to purchase this, recommend it, using an NPS score or something like that to gauge satisfaction? That can be a lot more reliable than, than straight up asking, would you buy this product? Because it's just not accurate.

Yeah, that's helpful. I appreciate that. Well, really, I mean, we're almost at the end of our time, which of course it always goes way quicker than I'm always planning for it to go, but I guess that's a good thing. But the final question I do want to run by you is a lot of times we can all learn from mistakes of others, and I imagine in helping so many folks specifically on alpha and beta testing, you've probably seen your share of mistakes too, even from experienced people. Could you talk about what some of those common mistakes are so that we can all not do those things?

Yeah, of course. Yeah, we've seen it all. I mean, over the years, just, just products that haven't worked, products that didn't end up getting released, products that— yeah, we've seen everything. So I'd say a couple of my biggest tips are poor planning. Maybe they go into it thinking that they can do everything at once, or they don't know how they're going to use the feedback, or they don't know, they don't have a schedule, they don't have objectives, they they don't have a plan or workflows for how they're going to use their feedback. Just think it through beforehand and really sit down and say, what are we trying to achieve with this test? What do we want to walk away with? Who needs what feedback in what way to actually be able to do something with it? And if you're just kind of going into it saying, well, we're just going to put it out there and see what we get back and kind of go from there, you're setting yourself up to have a whole bunch of data that you don't know how you're going to use. And it's really hard to prove the value. And do anything with the feedback if you don't have that plan beforehand. Um, the second piece is getting feedback from the wrong people. So I know it's really tempting to get feedback from friends and family, to get feedback just from your employees. It's scary to put your product out there before release, but if you're acting on the feedback of the wrong people, you could send your product in the wrong direction. And that's true with customers too. If you're only listening to your super users, you're only listening to your really vocal customers, they could be sending you in the wrong direction too. So you need to, to really make sure you're getting feedback from the full breadth of your target market. Um, and then the last piece, and this is kind of obvious, but not doing anything with the feedback. We've seen lots of companies, they spend all this time collecting feedback so that they can kind of check the beta box or check the customer validation box, um, and then go ahead and release without doing anything with it. And then they get all this feedback once it's out in the field and they're like, oh yeah, we kind of knew that was, that was coming. But they didn't do anything to prevent it. And even if, if your, your response is improving your documentation or prepping your support team for the issues that you know are coming, that's going to make a big difference in the customer experience. So even if you can't change the product, you can still do a lot to prepare your team for what you know is coming. And that's still really valuable.

Yeah. Well, thanks for sharing those. I think it's great for everybody to at least be aware of as we start going down the path of alpha and beta testing. And I know that there's questions that I didn't get to. So, but I know you and Centercode, you're active on Twitter. Maybe you could share how folks can get a hold of you there and maybe keep the conversation going.

Yeah. And people are welcome to email me. I'm emily@centercode.com. I'm happy to give advice. And we have a ton of free resources and blog posts and all that on our website that can be incredibly helpful too. So, but don't, don't hesitate to reach out to me on Twitter, over email. I'm happy to help.

Awesome. Well, thanks so much for being here and thanks everybody for being here for today. This was a great product launch session. Hopefully I'll see everybody at the next one again, August 2nd at 1:00 PM with Jean Ngo from Pentaho. So until then, we'll see everybody later and thanks again.

Thank you.