Scaling Experimentation for Real Revenue | Muqtadaa Miandara

Hi folks, how are we doing? My name is Mukta Damodara. I'm the digital growth manager for The Upbound Group. We're based out of Dallas, Texas. You'll have to forgive me, my head is stuffy and I feel like I'm hearing everything from a million miles away, so if I sound insane, that's why. But yes, so my name is Mukta. I am the digital growth manager at The Upbound Group, and what that means is that I'm responsible for everything related to growth and optimization and personalization within our organization. So how do you test? What do you test? Why do you test? And then how do you make those into longer-term, more scaled experiences? So for just a quick pulse, who in this crowd is already testing or trying to build a program that has A/B testing involved with it? Show of hands. Okay, only a few. You'll see there's kind of a few key things you hear a lot of when people are trying to build or scale a testing program from scratch, right? Because growth and optimization teams, they're not born with unlimited budget or headcount. They're very often what I would call a sideshow that gains traction. So some of the common issues that you see is that they're underfunded, siloed and dismissed as just testing. There's a laundry list of illegal statements that I have put up here on the left. These are things that I hear that I hate, things that are like, well, we know we should do this. It's best practice. Some things are just a priority.

Cool.

And so you're God and you know everything. Amazing. What are the winning numbers of the billion-dollar lotto while you're at it? Like, no, you don't know anything until you've proven it. And so that's where testing comes in, because it's not just innovation and exploration, it's de-risking and it's validating, right? But who am I, and why am I talking about all of this? I never meant to be in this field. I studied software engineering, failed out of that, studied UX design and research, did a lot better with that, really thought I was going to go into, like, quantitative research. And then right after graduating, I really needed a job, and I applied to be the designer on the team that I'm on now. Did not get it. Somebody else got it. They're way more qualified than I am, and that makes a lot of sense. But they called me back and said, hey, do you know what Google Tag Manager and Optimizely are? And I was like, I will on Monday. And here we are 5 years later somehow. So that's how I got into the space. Not totally certain what it was when I started, had to learn a lot on the fly. But at the end of the day, it was the same skills around creativity, rapid iteration, and critical thinking that sort of guided my growth in this space. And so the proof is kind of in the pudding. These are the things that, that I have accomplished in the 5 years that I have been working on the growth team at Upbound Group. We were incredibly siloed to begin with, and you can see that over time we have grown from a sort of marginal impact within the org to being a team that delivered in 2024 $10.4 million in incremental revenue impact from just one line of business for the web. We have three lines of businesses. If I get to work on all of them at some point, I don't know what I'll accomplish, but it'll be cool. I just need more resources to do that, and this is helping me build the business case for that. So, uh, the question then becomes, where do people fail to deliver the promise of testing? What are the common problems facing teams that are trying to scale growth programs, optimization, personalization. So picture this: you have a small team that's testing whatever they can see, leadership that doesn't see the impact or potential, a design team that struggles with data-driven feedback, analytics stakeholders that question every single number you present to death, product owners and developers who guard their spaces with jealousy and proprietariness, marketing partners who feel like you're stepping on their toes and messing with their attributed outcomes. If any of that is even marginally familiar to you, then you're not lacking ingenuity equity, passion, or creativity, you lack integration. So ask the following questions: Why are you testing things that the business doesn't care about? Are there OKRs or other strategic business priorities that you could be optimizing towards? Can you identify the key stakeholders and the metrics that matter most to them? How regularly are you sharing results with leaders? If you don't shout out your success, no one else will do it for you. And if you don't do it in terms that matter to those stakeholders, they won't care what you're shouting about. How well are you collaborating with other stakeholders, your design teams? Are you breaking their design systems, ignoring their research? Have your analytics and BI teams signed off on your attribution? Are they the ones that helped you develop it in the first place? Is your tagging sound, your data layer well validated, your event capture proportional, trending in the right, right direction? Does it all dump to the data warehouse? These are questions that help you identify whether or not you're integrated in a way that gets people on your side, right? Because otherwise, you're talking to a wall, and talking to a wall means you're not going to get anything done. And finally, and this is a big one, product teams and developers can be very proprietary. We know this. It's their space. It's their technology. They've built it. They've championed it. So, make them your champions. Get ideas from them. Have them drive your product backlog. If you're not already integrated with a product org, become their collaborator, become their best friend, become their business partner internally. I feel like I'm going fast, but you know what? That's fine. There's plenty of time to talk afterwards. So there's a little bit of a how-to guide on what it takes to embed yourself in an organization, right? At Upbound, we had the luxury of executive sponsorship kind of from the beginning. Our CMO was— interested in the idea of test and learn on the web, though cautious, marginally skeptical. So to reassure her, to get others on board, we had this weekly audience with leaders from dev and analytics and product and marketing and merchandising. And in it, we got to talk through what it was we wanted to test, what we had tested, and we helped set the roadmap on enhancements and feature releases. Now, we weren't invited to this meeting to do that. Initially, we were invited to this meeting to take input and give— Zero output. But we were like, well, we have input to give back. We have results that tell you you should or shouldn't do this thing you're thinking about doing. So if we're good stewards of the business, should provide that feedback. In time, we realized we needed to reach a larger audience to build leadership buy-in. So other parts of the org who were hesitant to collaborate or who wanted to guard their space and process, from us weird upstarts on the third floor, we ended up barreling our way into the monthly web update for the executive leadership team to get, you know, a direct injection of executive leadership input. Right. We were able to tell them this is what we worked on and this is what we delivered. We tested the following items on the website and it delivered the following tangible results that you see downstream in our analytics. But we're thinking of testing the following. As executive leaders, does this matter to the business? Does this matter to you, to the industry? So that elevated our strategy. It elevated our standard of reporting, and it forced us to zoom out from the little bubble we worked in of web funnel to business funnel, right? Industry funnel. The challenge that we face then is you're now presenting weekly to a mixed group of stakeholders. Monthly, you have office hours with specific groups of stakeholders. You have an executive leadership update, you have a quarterly business report. All of a sudden you need to start customizing your outputs, right? You can't just copy and paste the same slides across all these decks. There is content that is relevant to some stakeholders and not to others. So I don't remember what it says on there, but more specifically, like, you're not going to start talking about the product team's budget allocation to the marketing team. To the marketing team, you're going to talk about the attribution of their return on advertising spend and how you helped boost that in a given variation, right? And so on and so forth. So customizing those metrics is going to make a huge difference in specifically pushing the right messaging to the right stakeholders to make them your collaborators, and your champions, right? And finally, one of the things of note is that very often testing teams become get-shit-done teams, pardon my French. They're, you know, problem solvers, innovators, and you'll find that you're driving initiatives that by all rights are not yours to drive, but you're in a well-suited place of relative flexibility to where you're not hamstrung by traditional process or the quagmire of the daily rut, And so you can help break patterns of inefficiency and energize others to be excited about growth through your work and through improving workflows in general. Because when you start optimizing one thing, you can't help but look at everything through an optimization lens, right? Just because your own— your, your real impact is I optimize a website or an application. You also inherently start to look at the world around you and say, This workflow sucks, we can improve it, right? This communication is really tedious, we can improve it. You start optimizing everything. So, part of how you get people on your side is you want to make sure that not only are you taking input from people, but the people you take input from, you're giving credit to, right? If the marketing team suggests a personalized experience based on some marketing spend that's going, really cool, you should absolutely celebrate that they helped generate uplift for your program. We, you know, saw a, call it, $12 average revenue per user uplift in this personalized experience. That's a crazy number. $12 is huge. Sorry, pulled that out of my ass. So sharing credit is what's going to make people feel like, hey, there was a tangible return on my investment. In this team. I worked with them, I got something out of it, and it helped me deliver a result that my leaders will care about. Right. And for you within an organization driving growth, that means they're all the more likely to work with you again in the future to support additional investments in tools and headcount when you need it, because they know that you can deliver on results. And back to what I was saying earlier, tying your wins to their KPIs. Again, you don't want to speak about like return on advertising spend to the operations team. They don't care about marketing. You don't want to speak about product allocation of budget to marketing. They don't care about that, et cetera, et cetera. And finally, a rising tide raises all ships. So just if you do this right, you can position your growth-oriented team as the engine behind other people's success. Granted, there are models within orgs that are very centralized, which is kind of more of what I'm talking about. But there are also growth models where there are experimentation specialists distributed across the org. It's very decentralized. In that case, it probably looks a little different from what I'm describing here, but the larger overarching theme is still the same. If your team or if your process is the place where people go to get a win and drive learnings and growth, you will never want for opportunity. Opportunity. So regardless of if it's centralized or decentralized, if what you do and how you do it is what's driving that growth, you'll have the outcome you're hoping for. Um, hmm, hmm, hmm. There we go. So one of the things that comes with delivering on growth is making sure that what you're saying is validated, that people trust you, that there is credibility, that what you are delivering on is agreed upon by other stakeholders. One of the challenges our growth team faced initially was our results dashboard in our experimentation tool would say one thing, and then people would go and start poking around in Google Analytics and then in the data warehouse, and they would say, none of this matches up. What's happening? So we had to undergo an exhaustive effort to make the data line up. And it's become easier with newer tools around data and analytics that integrate and make things smoother. But ultimately, growth is more than rapport and relationships or embeddedness, though all of that is incredibly important. The credibility where people trust your outcomes is what will drive growth and autonomy. So data hygiene and QA is going to be probably top of mind, where if your data is not structured in a way that matches with what the business needs, if people don't trust that it is well validated, if the architecture doesn't make sense or isn't delivering consistent outcomes, you're going to run into problems. Replicable ROI. So revenue is is probably the most useful metric in experimentation. And I say that because it's not a great metric for deciding if individual tests win, but it's a great metric for deciding if a program is doing well. Does it deliver to the bottom line? You could test all you want and increase conversion on funnels all you want, but if all you're doing is generating empty leads or less qualified leads, then who cares, right? But actual money made is what matters. So actually, my team had to build a model in partnership with our digital analytics team because we're, we're, we're a payment-based model, like similar to subscription, or we're in the lease-to-own, rent-to-own space. So like people pay over time. So that initial payment that we capture is not useful in terms of being able to track the trend in value of a customer. So we had to build models based on when a user converted, where they converted, with what type of product they converted, and what type of conversion it was. And then from there, attribute some estimated lifetime value to it and build a revenue model for them. And then using that average revenue and the number of users in an experiment or in a variation, we could build revenue per user and then measure the incremental difference between variations. What was the uplift or the degradation. And this helped us to better attribute outcomes when, like, test results were confusing, right? Where we'd see uplift on certain metrics and a sharp downswing on others. Well, look at the bottom line. Did it move the needle or not? So that was very helpful. Finally, just something to acknowledge is the reality that leaders will often only listen and make changes when they have to. So be consistent, be really hard to ignore. Even with my team and my program's successes, we still have to fight for resources. We still have to justify our wins. And that's just kind of the game, right? So either you stew in it and you get kind of bitter about it, or you play the game and you play it better than others and come out ahead. These are— this is a case study. We Pivoting a little bit in the direction of what experimentation offers and then what personalization offers on top of that. We have been working on like building better customer experiences for a very long time. Last year, this is by the way, a 50-year-old brick-and-mortar business that only added checkout functionality in 2021. So we're catching up, I swear. But adding a recently viewed items component on our homepage for returning customers. Technically a marginally personalized experience, but we called it a test because it was a very broad segment of customers, right? It was like one-to-many instead of one-to-everyone, right? So we tested it. It did great. It generated, I think, $129K in incremental revenue uplift over Q4 of 2024. But we had at the same time a thought of What if we presented this in similar sort of recommended or preferred components to more qualified customers? And so we did the same thing a little bit more expansive, but we did it to a select group of users that we thought were more qualified that had already gone through and gotten approved or were signed in and had an approval because you have to be approved to shop with us. And so despite technically being a smaller customer group, or visitor group, I should say, it ended up generating nearly triple in terms of revenue and had a tangible uplift on transactions because, of course, we all know delivering the right content at the right time is the most important aspect of all of this. So, and one of the things that we observed in this, because last year was the year that we really spun up a personalization program because Everybody heard it in conferences in Q4 '23. So all of 2024 was spent on personalization because of course it was. One of the things that we observed in conducting this effort to stand up a personalization program was we only really got going with it in end of Q2, beginning of Q3 of '24. And when we broke it out in our revenue reporting, one of the things that we discovered was it was $10.4 million was the incremental revenue we attributed through the year. And of that, $5.1 or $5.2 million was personalization, which was only half the year. We didn't do it through the whole year. So knowing that I'm futzing the numbers a little bit there, that implies that personalization is roughly twice as valuable or personalized experiences are twice as valuable as one-off individual tests. That, of course, is also a little bit spurious of a statement. I'll edit myself as I say that because ultimately you can't achieve qualified personalizations without testing your way into them, I don't think. So I've talked myself in a circle here. But my point here being that if you're not testing to personalize to release, like well-built experiences, you have the wrong workflow, right? That's kind of what I'm trying to get at there, where yes, you can continue testing till the crows come home to roost. Is that a thing people say? You can test until the end of time, but inevitably you'll run out of things or low-hanging fruit or even like bigger swings that you can take and you'll need to start customizing experiences. And there's a lot of value in doing so. So in order to grow, scalable growth comes down to people, process, tools, and data. My team is currently experimenting with improved customer attribution so that we can drive our contextual multi-armed bandits through our testing tool Optimizely, which is a self-healing personalized experience that optimizes which users see which experience based on correlating outcomes to those customer attributes, which is a really cool technology, right? Our data team is working on machine learning models for predictive scoring. What events correlate to success on your funnel? On our website, users who visit 2 or more product pages are twice as likely to convert to a transaction. Okay, cool. So now we know to start optimizing towards users hitting more product pages at minimum. That one seems— that's a really obvious one. But there's other deeper insights where we're like, oh, if a user does the following chain of events, they're most likely to do the next step. So how do we get them to follow and fall into that funnel? Also, we're looking at targeted promos based on where users are in their journey. Hesitant return buyer— hesitant return visits where people keep looking at the same products or keep looking at specific categories, trying to populate customized, personalized promotions for them. Ultimately, just trying to get ahead of where the customer is and giving them what they need to get them across the finish line. And finally, of course, in case anyone hasn't heard the term enough today, AI. I am a firm believer that the AI hype bubble will burst. I don't want to deflate anybody's bubble here or make anybody feel bad, but like Not everything needs to be AI, but what should be AI is largely not customer-facing. What should be AI is what makes your creative team move faster and generate assets better and more flexibly, makes your marketing team stop living out of Wrike or Jira and move incredibly quickly to get social posts out the door without having to question the how or the why or where the verbiage is or whatever. We're doing that right now. We're embedding AI to start speeding up generating creative assets for our homepage panels. We're using customized prompting to generate experiment documentation about results and briefs and hypotheses, et cetera, et cetera, et cetera, trying to offload as much of the scut work as possible to our AI tools. The hope is to get to the point where our agentic AI tools can really handle a lot of workflow. And rely on those automated agents to act as a pseudo team member of sorts. So once you have done the work, if your team is successful at getting embedded in the organization and you are actively given the opportunity to give input on the direction of the organization, what comes next, what needs testing, what feature releases need to go out. Take the opportunity, right? Like grab the bull by the horns and take the biggest swing you can to mix as many metaphors as possible. So, so what does that look like? That, that means essentially you absolutely should be advocating to test and learn for everything. People will say again, well, some things you just do, and the answer is not when there's hundreds of millions, if not billions of dollars on the line. You don't just do things when people are dropping a quarter of a million on something, a million, 2 million, a billion. Those are numbers that are stupid, and you can't be stupid with numbers that are that stupid big. Validate, de-risk, advocate to feature flag as many things as you can, advocate to test dummy versions, door-knocker tests, tests wherever possible. Advocate for optimization of product releases. Some things have to happen because of legal requirements. You have to release this new bank verification feature because you're in a regulated space that says you have to have this in your checkout funnel. Okay, then let's also immediately post-release start testing to optimize that adoption rate or that product utilization rate. There's a million and one ways. To use testing to grow in any capacity, in any workflow. You can even turn it internal. You have internal systems, internal tools. We have our merchant portal tools. We have our, our custom point of sale tools. These are things that we assume work for us because we defined their needs and built them for ourselves. They absolutely could use improvement. Our internal point of sale system is very custom because we're in, again, a very legally regulated space and we have to do very specific customer record management and all this stuff. It is not the best solution. We built it. It's the best solution that we've had so far. That does not mean it is the best solution. So we are actively working on what are the tests and then what are the outcomes that we attribute to measure the impact of that test, because it's not going to be a customer review, it's going to be a store associate that says, well, this was not better, so I don't know why you released that. Right? Like that kind of feedback is harder to collect and to attribute to a revenue outcome. So it's harder to justify sometimes to leadership, but it can be done. It just, it becomes a question of who do you have to collaborate with to make that happen? For us internally, it's our operations teams. So we have to work very closely with them to figure out How do we attribute that? Ultimately, testing is as much a go-to-market strategy and a growth strategy as it is anything else. So to sum it up, because I think I went quicker than I intended to here, but it's fine. Make your work impossible to ignore. You want to integrate, you want to prove, and you want to scale. And I could read what's on the little parts, but I can't because it's too small on the screen for me. Me, so we're going to pretend that I did. All of what I've been trying to say here basically is that there is no reason that a testing team shouldn't grow in an organization as soon as you line up just a couple of key things to start delivering validated, verifiable results. There is no leader in any business outside of ego that will say, I don't want to make more money. Everyone who is a good steward of any business will say, I do want to make more money for this business because that reflects well on me. You can be the path to doing that if you are directly chairing or guiding a testing organization to grow. So thank you. I believe That should be it. I don't know if we're doing questions.

We are going to do Q&A.

Wonderful.

I'll join Preston in these rooms.

Oh, phenomenal.

But also, if they aren't being added, we can simply go around the room and see who may have any questions about this wonderful presentation. But first, let's give you a hand.

Oh, thank you. Appreciate it.

Are there any questions? Yes, coming.

I think your kind of guidelines for becoming embedded in an organization are important for any function and any individual.

Sure, absolutely.

So thank you very much for that reminder. I wondered if you could comment a little bit on what your team's connection is with creative and creative A/B testing. You talked about getting creative teams out of Wrike, but oftentimes this A/B testing out in, let's just say, a paid media world digital world does put a lot of burden on creative and how you partner with them to make that more efficient.

Yeah, that's, that's one of the places where we're taking a 50-year-old advertising engine and saying, please use AI, just shaking them by the shoulders and saying, for the love of God, open up Gemini. Just do it. Especially like with Nanobanana and like these other models that are getting really, really good at asset generation. Like our main focus was we started by challenging our creative team on the assets for the website. We didn't— we're not looking at anything paid or social yet because we know that's a whole different beast. But just for the website, our creative team was generating some of the most staid stuff I've ever seen where it's like, okay, I get it. You have a lot of in-store and pop and everything else you have to generate. The web deserves good assets, so how do we get them? So we did a bunch of user research first. Usability tests on home screens that had different types of imagery, product knockouts and text and promos. And we started identifying that there's promos, deals, products, categories, like different types of creative imagery that you can promote in a retail space. Right. And then from that, which of those A should be the number one and B, what style should they be in? So we settled on for our customer, lifestyle imagery is king. Which has kind of been a trend for a while. So then how do we get that? Because our merchants don't supply us those images. So then we start taking their product knockouts, dropping them in Gemini and saying, please do something with this. Give us a fall-themed living room with this couch and boom. And we just have to tweak a few things, make sure it works for our merchant partners, and then we're able to get those live on the site. And our creative team was— their flabbers were gasted, to say the least. They were absolutely surprised that it could go so quickly. And it's a simple matter of taking those assets, upscaling them, making sure they're web ready in a good, like, you know, like a WebP or AVIF format and then going from there. So I have been basically content managing our homepage for the last 6 weeks since we started successfully testing this, which is not a great use of my time, but it has proven to our creative team that they need to learn to move faster because if I can do it without living in their systems, in 2 tests, they have to be able to figure it out. So there's a little bit of a challenge there. Like I'm throwing down the glove a little bit to be like, figure this out. If I can, you can. And it's not— I'm not saying be impolite about it, but also, you know, push. Other questions?

Hi, I'm curious what tools you suggest emerging brands use to figure out their customer segmentation for A/B testing. As you mentioned earlier, when you're really targeting, you can have more efficient results. So how can smaller brands do that?

So there is a misconception that you need to have all of the customer data in the world to start personalization. And to be fair, a mature personalization program absolutely requires more customer data. Crawl, walk, run, right? You got to start where you got to start. If you don't have the data schema to understand your customer, that's fine. You do know a few things off the bat. First visit, returning visit, existing customer, new customer. right? Like has viewed product, has not viewed product, these behavioral markers that aren't really demo-related. They have nothing to do with their demographic information or their interests, but they do tell you a lot about where in the customer lifecycle they are. Has a customer just landed on your site for the first time? From what channel? What does that tell you, right? Are they social? Okay, social customers very often prospect. What is— do you need to give more education on your value prop then, right? We also look very often at like handoffs. So personalizing content on our homepage based on if a user has reached certain points in our funnel. So a user has gotten through approval but then never converted and they come back. Okay, we got a signpost to them that, hey, you're, you're a qualified customer. Finish, finish the journey. What are you doing? So that's where I would start. And then I would say in terms of tools for better attribution over time, I mean, there's a lot of contenders in the CDP space, customer data platform space. I would say a lot of those are CRMs that are claiming to be CDPs, so I would be cautious. I would look for a tool that is specifically oriented on customer data, not on like their, their core value prop is customer data, not just like an email tool that has evolved into a customer data tool because yes, they have a lot of what you need, but do they connect to the web? Is it easy to activate on that data? Is it easy to surface that data or pipe it other places? So for example, because Optimizely offers their data platform as well, ODP, We've demoed it before, haven't been able to convince my leaders to sign off on it, but it does a lot of real-time segmentation and suggest audiences and things like that. I think there's a lot of a case to be made for any customer data tool that is modern and using machine learning to sift through some of that data for you as well. Of course.

Fantastic. I'm not sure if we have time for another question, but if you want to grab him after this, he'll be available hopefully. But thank you so much for that presentation. It was very robust, and it was awesome. We'll give you another hand.