How to use Customer Feedback for Continuous Product Improvement with Centercode and Peloton

So just to get started here, I'd like to start with a quick introduction of who I am and why I hopefully should be talking to you about this. My name is Luke Freiler. I'm the CEO and co-founder of a company called Centercode. Prior to founding this company, my background was on the engineering side of business and specifically focused on UX and software UX eventually. I started my career at Samsung, moved over to Ericsson, two very large enterprises, and at both of them I saw I saw a need for what I'll ultimately be talking about today. In that journey, I experienced both the hardware and software sides of the business, working on both consumer and business products. So I had a nice, well-rounded experience leading into the foundation of this company. And I would definitely encourage you to reach out to me either direct through email or just connect with me on LinkedIn, and I'd be happy to keep up there. Now, in terms of Centercode and who we are, we have a mission to effectively drive product quality as well as innovation. Through very high-engagement customer tests in a methodology called delta testing. And that's what I'm going to be hopefully introducing you to today and taking whatever questions you might have. I'm also going to be talking to Yandara from Peloton about their experiences in this space as well. So very much looking forward to that. Now, in terms of what Centercode actually does, there's really 3 components to our business. The foundation of the company is a platform that allows companies to host these types of programs, to recruit targeted audiences, to collect feedback from them, very high engagement tests, and then ultimately derive insight to drive again the quality and the innovation in their product lines. With that, we also support it with a large tester network. So we have individuals around the world who have basically come forward to say they would like to help shape the technology that they use, the technology that solves the problems that they have, And we help connect those people to these various companies, typically either high-growth technology companies or modern enterprises that have sort of progressed into their digital transformation. And we work— we provide those testers to them as part of the service to help improve their products. Finally, we offer managed programs where we actually run these types of programs on behalf of other companies. So many large tech companies and non-tech companies that are entering the tech market that are bringing these new products to market will actually work with us to manage these entire engagements for them. Now, to just sort of cut to the chase, what is delta testing? And it's very simple. Delta is the next generation of beta. It's aligned with agile development of products, and it really leverages technology to drive the quality and the innovation directly from those users, but throughout an entire product lifecycle. In short, beta testing is a very old idea that is very much aligned with how products used to be developed, waterfall development, whereas Delta is a modern solution aimed at leveraging technology to work with the way products are developed today. So in terms of why this is necessary, first of all, products are becoming far more sophisticated effectively across the board. Software, hardware, doesn't matter. It's becoming more and more complex as it solves— or serves to solve more problems. At the same time, the audiences for these products are typically becoming more mainstream. They're becoming less patient. Now, on top of that, products, especially software, are becoming even more connected. It's virtually impossible to have B2B software that doesn't rely on and interact with other software. So products becoming more connected is a challenge, but more importantly, you typically have no control over all the products they're connecting to. It's just a slew of moving targets that are growing by the day. Finally, what I consider most important is that products are now evolving and growing. It's not sort of a fire-and-forget strategy to get a product out the door. Audiences expect products to be progressive. And again, both on the software and hardware worlds, where we do work with and see both, um, it— products just don't stand still anymore. You don't release something and then start working on the next thing and release it another 18 to 24 months later. Rather, when you introduce something to the world, it almost certainly becomes a platform that is built on over the course of months and years. Now, the use cases for this concept and what this methodology applies to is reasonably diverse. Anything that effectively leverages an audience for a result, starting with product launch. This is the goal of driving a successful launch through target market feedback in, again, a very well-formed methodology. This is the equivalent to a beta test, what you might call a field test, user acceptance, friendly user trials. Companies use many, many names to describe this, but Delta provides a reusable, scalable method to achieve a successful product launch with customer feedback. Now, once we've gone through that launch, it then moves to an iterative model, and delta helps with product growth. So this is leveraging existing customers to mature a product. So this would be more of a continuous beta aligned with agile development, or you could even think of it as an ongoing customer advisory board that's deep into the product and helping continually improve it. Now, something else that we see commonly with this methodology is what's being dubbed a Customer Zero program, or what you would have probably previously called a dogfooding program, or in many cases, alpha testing. This is really driving, or driving to create a product culture within a company through engaged employee testing. So this is going beyond just introducing it to a market, but rather earlier introducing it to your employee base to get them closer to the product and also help shape that product where it makes sense. Finally, the last methodology here, or the last use case here that this methodology is used for, is competitive analysis. It's very typical to take this method and gather intelligence on a competitor's product through their customers. So this is effectively a market test where if you're looking to enter a new market or further understand a competitor's product to help shape your own and better solve the needs of those customers, those shared customers, then the same methodology applies to do that. Now, in terms of the results, one of the goals here is to have very explicit actionable results. And what that turns into is 3 channels of feedback. The first is what needs to be fixed in the product through issues. This is useful to engineering, quality, and support. What needs to be improved in the product through idea generation? This is for product management, for UX, for CX. And then finally, what are the heroes in this product? What needs to be promoted through praise? Most interesting to marketing, sales, and executives. So these three channels of feedback are explicitly what comes from delta testing. Each with their own audiences and their own value. But at the end of the day, one of the core tenets within this space is ensuring that everything is prioritized. You know, there's a couple major truths in our space. One is that you cannot address everything. And number two is that not every issue is created equal, not every idea is created equal. So it's incredibly important to be able to prioritize these. And this system uses a concept called impact scoring to effectively look at the frequency and the severity and various other signals to prioritize all this feedback against each other so that you can optimize your efforts. Now, in terms of how it works, this could be an hour on its own, so I'm going to do a very, very quick version. Each delta phase starts with a simple phase scoping. And what this basically is, is breaking down your product into a series of features and experiences that you want your audience to go through. And with that, each is going to have a very simple basic activity that's going to inform how they test that thing. This can be very exploratory. I just need you to use this feature by name, or it can be very explicit. We need to make sure this thing connects to this other thing and that successfully works. Now, beyond that scoping, basically mapping the product for whatever happens to be tested in that phase, um, we're looking at ongoing tests typically. So it's more tester replenishment and having an on-demand group of testers available that want to be testing that you can bring in as testers may fall off over time. So we scope our test, then we replenish. From there, it's about tester engagement management and typically using technology to do this for you. So effectively ensuring that testers know what you're currently focused on, what's important to you, ensuring that they're testing that, and then they're producing feedback in the form of issues, ideas, and praise. So once that feedback comes in, it's about presenting and implementing those results into the team. And most importantly, this happens at a cadence that is typically aligned with whatever the sprints are within that company. So in many cases, this is weekly. Sometimes it's biweekly, whatever happens to make sense for those companies. Now, the impact of this is broad. Typically, what we've seen in moving from a traditional beta program that's very ambiguous, using a variety of sort of single-purpose tools, point solutions, we're typically looking at about 3 times the feedback. With this methodology, that feedback is prioritized and actionable, and it's gathered in far less time. So as a result, this costs far less. Now, on the increase side here, what we see in a business is typically a higher quality product, user experience or customer experience being improved. Collectively, a friendlier, higher quality product results in higher reviews, better sales, customer sat, and earlier product planning capabilities. On the other side, we're looking to reduce quality cost, we're looking to reduce development time, the internal bias that typically surrounds all of these decisions, the time to market being reduced, the support costs being reduced, and of course, at the end of the day, lowering churn. Now, that was intended to be a very, very quick intro, but the real goal today was to actually talk to someone in the business, in the space, that does this professionally for their career. And I'm going to be speaking with Yandara León today about this. She's actually a senior program manager of testing at Peloton. And you can connect with her on LinkedIn here as well. But with that in mind, I hope she's here and available and unmuted. But Yandara, welcome.

Hey, thank you.

So before I get to the questions, let me just throw a little bit of a level set out there. For those that don't know, which I assume is probably a dwindling group, Peloton is an incredibly successful sort of leading-edge fitness brand. Making a growing variety of fitness-related products, both hardware and software. They do have a very successful SaaS offering. And very similar to companies like Zoom, they were on the right side of COVID right? As everyone was sort of home and looking for more things to do with their time and trying to stay in shape, Peloton became one of the de facto awesome solutions to help do that. So I imagine it's been a pretty awesome year at Peloton in most cases. Yandara, on the other hand, is someone that I've actually had an opportunity to work with numerous times over the years. And she was someone that when we had the opportunity to speak to you immediately came to mind as someone that's been doing this for some time, seen a lot of success and grown her career as a result. So I really wanted to give her the opportunity to speak. So that said, Yandara, welcome. I'd love to just start with, you know, how long you've been at Peloton and what your sort of role and responsibilities looks like there.

Yeah, so I joined Peloton in July of last year, and my title— I'm a senior program manager for the field testing team. That's what we've named our team at Peloton. Yeah. And did you want to go back into my career?

Yeah. So I'd love to know kind of how and where you started in customer testing and where it went from there.

Yeah. So Um, kind of stumbled upon it. So I started actually as an associate for a newly formed team. Um, it was a newly formed beta testing team at GoPro. Um, so I just came in and they had, uh, adopted a tool called Centercode. And, um, I was curious about the tool. I was just helping them out. Um, and so I started to learn the tool and was able to turn around and help train the other two associates that were on the beta testing team. With Centercode. And then eventually I started to manage the beta test, and I was there for 5 years. And then from there, I went to Pax Labs and helped them in a similar fashion with beta testing. And then most recently, I found myself at Peloton.

Awesome. And we were excited when we heard that, uh, Yandara had been picked up at Peloton. I believe, and you can correct me if I'm wrong, but your introduction there was actually relative to kind of a user group we held at one point, right? Is that where you, you had met Richard originally?

Yes. So Richard is my boss and we met at a conference that you guys had held. You used to hold like a yearly conference and pre-COVID. Exactly. Made acquaintances, stayed in touch on LinkedIn, and it definitely has worked out really well. It's fun to have that story. I've been working with Centercode for 7 years and it just kind of like keeps— it's a theme in my life for sure.

Awesome. So how has this testing kind of helped with, with product quality and innovation in Peloton and both the time you've been there, but also obviously what you know of before, you know, its brief history before you?

Yes, our team is fairly new, um, and we've just had really good support from the company and especially from our org, and it's quickly just become a part of the product development life cycle and the delta testing plans. We've been able to establish processes that have helped embed the customer experience and their voice, if you will, into the product. And it has also helped conversations in defining future products as well.

So do you currently have any sort of employee testing program in place?

We do. So I think like most companies, lots of people have like an unofficial dogfooding. And so by forming our team, we've been able to formalize that process. And this has allowed us to extend testing to teams that are outside of the normal product development. So we engage a lot with the marketing team, member support, and legal. And that's been able to help us bring in a little bit less bias when we're at the early stages of product development. But we still want it in people's hands and in testing. It's been great to be able to utilize all the teams throughout the company.

Awesome. So the big obvious question, and obviously relevant to your space, but what is the story with the COVID at-home testing, the remote testing? How has that impacted you, especially as you've joined Peloton obviously in the middle of it?

Yeah, it's definitely different to start at a company fully remote. I've never met anyone besides Richard in person on my team, but I'm pretty used to working remotely. So like my day-to-day personally hasn't changed, but from our team's perspective, we have had a few logistical challenges. I think that's pretty common with a newly formed team as well as like a new pillar in testing that we've established. Um, but luckily, like I've said before, the company and just teams are who are working with us are really enthusiastic. And we've been kind of been able to embed those logistical challenges into like our project schedules and just by adding like a day or two into the test plan.

But from the tester perspective, is COVID increasing your feedback? I assume just people have so much time.

Yes, we've seen some really great feedback. Um, just our tester participation has been very high and people definitely like to work out at home right now, so that's been a great advantage for our team for sure.

And how hard is it for you guys to find testers?

Um, I— not that hard right now. Yeah, I haven't seen numbers like this before. It's great. It makes my job a lot easier for sure.

That's awesome. So have you guys talked about what happens when COVID ends and things might get a little more challenging from that front when people return to their normal lives?

Uh, yeah, I think we'll just have to roll with the punches, see how it goes.

Yeah, no doubt. Yeah. So are there any sort of core challenges that you face today in collecting feedback, or at least actionable feedback? What is it you're getting out of that stuff right now?

So just in my experience, like, our team definitely sees and collects a lot of data and a lot of feedback. So one of the biggest challenges is figuring out what to do with all of it and how to present all of it. Lots of times we're very much honed in on what's the product that we're developing at the moment, and we want to bubble up, of course, all that feedback to the, to the teams. And so we navigate through that and really like present that. But then we also have to make sure that anything else that we've gathered isn't lost in the mix. And so we have, we have like a repository of feedback, if you will, and we kind of, we definitely try to make that available to all the teams. But it's a good, it's a good challenge to have. Just having too much data, or it's not too much data, it's just a lot of data. It's great to have. We just hear a lot of things and people who are testing for us tend to be very enthusiastic about what they're testing and they want to tell us all their ideas.

I'm curious, which teams do you interact with the most regularly? Like, who is sort of your hit list for implementing what you've learned?

So product, product owners, we try to work very closely with them as well as we are within the quality org actually. So other quality pillars, we work with them very closely. And we also have seen just a lot of curiosity from upper management. They want to know what the product is, how it's being received, and that's just been great to have that support as well.

Awesome. Okay, um, at this point I'm trying to split this session in thirds, so I'd love to move on to some questions from, from the field here. Um, the first one I'm seeing on here, and it's a little vague, but I'll let you, uh, field it. It says, can you start with the basics and explain what exactly the field testing team at Peloton does, maybe with a real-world example?

Yeah, so we like to get the products into customers' homes prior to launch, and the idea is that we're testing a few things. In our case in particular at Peloton, we can be testing the delivery process, the setup process. We can also be then testing the product, and the idea is that we give enough time so the tester is experiencing it as if they purchase it themselves. So we're not going to ask them to complete their testing in one week. You know, it's not realistic. Everyone maybe works out at a different cadence. So we're going to give them weeks of time, and then we check in with them every week. We may give them a little bit of guidance like, hey, make sure to navigate within this menu, or make sure to take at least 3 30-minute classes. 'How many classes a week?' or something like that, that gives a little bit of guidance without giving them a script, if you will. And then we, we ask at the end of the week, like, 'How, how was your experience?' You know, 'Did you, did you feel like there were any issues?' And then throughout that, they're also able to submit any issues in the CenterCode platform. They're able to submit praise, and then we just get that plethora of feedback there. So the true gist of field testing is making sure that products are in home and with real people that are outside of the company and getting that feedback so that we're not missing anything when it comes to launch.

Got it. Let me move on to another question here. This one says, by inviting members to test who aren't involved, so like legal or marketing people more directly, You said the goal is to sort of remove bias. Do you have any other tips on how to sort of remove bias from a test?

Yeah, it kind of goes back to what I was saying about like that scripting and not scripting. Like there's definitely a balance there. You want to give them direction and you want to make sure certain things are tested, but you can easily find yourself just becoming like RHEL testing, if you will. So you have to kind of pull back a little bit and see what's important, see what's really relevant to the tester and keeping the tester top of mind because truly, like, they're the customer and kind of like giving them their space to actually use the product and then making sure that they know how to use the platform and how to give that feedback just so that we can also hear what they have to say.

Another one on here is, is we don't have a dedicated person running our betas or tests. Who's a good fit if we're looking at existing employees? I can field this one. This is an interesting question we see a lot. The most common person in a company that actually runs these types of tests today is actually product managers. More than half of product managers actually run their own tests. And most tests, you know, this is sort of a new field in general. It's always been something that's kind of ambiguous and been done by different people within the org. It really is the sort of the high-growth technology companies that have teams for this. Nora listed some great examples that she's worked at. Other companies that do this kind of thing, the Fitbits and Rokus of the world and people at Google and whatnot, they have teams for this. But most companies that aren't of that sort of high-growth mindset that kind of grew up in Agile, it's typically product managers that are doing this on their own. It's not necessarily ideal because they have a lot to do, but often they really are supposed to be the smartest person in the room relative to the product. And one of the neat aspects of these tests is that they are really a direct reflection of a product manager's responsibility. They touch the technology, they touch the customer, and they touch the business aspects. So those three things are very, very important to product managers. This being a reflection of that gives them an enormous amount of intelligence as well as a direct touch to their customer base. So we see a lot of product managers who are directly involved. Beyond that, we typically see this living in one of two orgs. It's most often in a, in a quality-related org, just as Yandar has seen here. The other one that we've seen some interesting success in, and I've personally found pretty fascinating, is when it grows out of support organizations. And the reason for that is, is if you look at sort of what support staff does, It's very similar to the skill set you would need to do something like Yandara's position. You know, you have to be customer-facing and, you know, willing to speak with and engage with customers, but you also have to have some level of technical acumen to be able to work with their feedback and whatnot. So we've seen some really great examples of this being born in a support organization who also recognizes the direct benefit to them, right? If they help get fewer issues out faster, or they help understand the issues so they can get ahead of them when the support calls or emails come in, then it's immensely valuable to them. So they're often the ones that have made the connection between, hey, this product has gone continuous. We can't just run a beta test once a year and kind of fire and forget. So that's kind of how they're seeing it. Another question on here. Was it hard to get support? I guess this one is more for you, Yandara. Was it hard to get support from the company when the field testing team had just started and been formalized? And obviously you said you started— joined shortly after, but I feel like you were probably still part of this. And how sort of flexible was the company in kind of adopting what you do?

My experience at Peloton has actually been pretty unique in that I feel like the ask to create our team was from the top. And so the support has been there from day one. I have seen where it's not necessarily that case in every company. But with that, here at Peloton, we've just been able to really kind of grow and define what it means to field test and beta test and delta test, whatever you want to call it, at Peloton. It's been great.

So the next question would actually be interesting for both of us to answer, because I would love your experience both at Peloton and maybe even before. But I'll throw in my own at the end here. The question is, how are test teams measured? What are some of the KPIs?

Do you want to go first?

Um, I would prefer to go second because mine is a little future-facing, but I am curious how your teams have been measured at, you know, sort of the last 3 companies you've been with doing this.

I mean, launch is obviously like a huge factor, what it is after the fact and see how smoothly that launch goes, how those reviews, if you will, that end up online. But I think within the company, or like in the product development cycle, we definitely agree upon those KPIs within the team and, you know, give a standard as to what we think is okay from the tester if we're giving them a 5 5-star rating, maybe we're looking at 4 stars and above for that, for maybe like a particular feature or just like their overall sentiment in the product. So we've been able to gauge that and be able to use that as a standard right now. I think that's what's worked for us. And, you know, if we need to take action because something hasn't met that mark, we definitely do.

Awesome. Yeah, sort of measuring star ratings after the fact is something, and sort of movement, uh, evolution of those star ratings is something we see a lot. Um, the reason I was interested in answering is actually this has been a pretty big mindshare for me recently. Um, we have a big release of our platform coming out where one of the big takeaways is actually trying to solve this problem, because the reality is being sort of a nascent industry many companies don't really know exactly what to measure and what impact the team should have. So what we're doing is actually boiling it into 3 metrics that we believe are going to be very valuable to anyone who's sort of existing in this space. Um, the first one is basically a grade for the project, and that grade is based on engagement in 2 forms. One, are testers completing the things that you ask them to complete? And two, are they giving you rich actionable feedback as a result? And we have sort of a scoring paradigm built behind that that literally produces a grade, a letter grade that tells you how well your engagement is, is in that test or how high your engagement is. The second metric is then basically predicting the success of the product, and this is a score that is then looking at the relationship between the feedback that you've gotten. So if you think of it as issues, ideas, and praise, the 3 things I described earlier, um, each having sort of a different stake in that— praise being good, issues being bad, and so on— we can actually use a pretty simple algorithm to get a sense of how that product benchmarks against your other products. Then the third one— and I'm running out of time, so I'm going to finish with this— um, is effectively a test impact score. This is looking at those first 2 scores and saying, okay, what did the product look like at the beginning, and then now what does the product look like at the end, understanding that Some ideas might have been implemented or experiences changed, some issues fixed, and so on. And the delta between those two— not to be clever with it— but the delta between those two is the impact of that test. Where was the product and where is it now? And by doing this, we can give you a benchmark to measure all your products against each other. So with that said, I am at my time, um, and I could talk about that for another hour and have a great time with it, but I know that's not on the cards today. So I definitely want to thank Saskia and Caitlin for this opportunity, and again, encourage anyone here to connect with me on LinkedIn or just reach out to me, Luke@centercode.com.

Thank you, Luke. Thank you, Yandara. This is great. Like Luke said, if you guys have more questions or want to reach out to him directly, please feel free to do so. The presentation will be available and so will the session, so the key takeaways can be in there as well. And we appreciate you guys coming. Thank you again, Luke and Yandara.

Thank you. Thank you, Yandara, very much.

Thanks. Have a good one, guys.