#49 – How To Design Winning Holiday Experiments

How do you determine which elements you need to prioritize during the holiday season? I think we cannot answer that question without talking about the strategy, the experimentation, and a B testing strategy, or the CRO strategy during the holidays. And the strategy that I'm going to talk about, we've learned the hard way probably in the last 10 or 11 years. That's when we start focusing on the holidays and what kind of testing we can do during the holidays. And I have to tell you, our first experiments that we've launched during the holidays, as I reflect back at them, we knew that we have to have offers, and we were testing different offers, but at the same time, we said, since people are highly motivated, they're coming, they're looking for a deal. We wanted to test simple site functionality. My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal.

Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with the growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts. Okay. Another episode of CRO Live Hour. We are back.

It's been about three weeks without the CRO Live. Oh, wow.

I didn't know that.

Not like publishing any episode. Yeah, it's been three weeks. I think it's about time we do, like, a recording.

It's funny, I remember why three weeks ago. That's when envesp and thinkpie were super busy. I think it was one of my most stressful weeks at Invesp in 10 years or something like that at Figpie. So I'm like, simba, I can't do it. And then you are out. Do you want to share any news with our audience?

Yeah, of course. I took a vacation for two weeks. I came Back a different person. I became like a dad for the first time. I now have a son. I'm now a dad. So it's been interesting being like a first time father. There's no like template how to be a father. You just have to try and take advice from other people who are already dads. They've also like been giving me tips on how to become a father or how to do certain things which they've been helpful.

Yeah, it's like everybody has to figure it out on their own. The most important thing that you will do for your kid, but you're just trying to figure it out on your own. But by the way, the first kid is when you're just confused and you're like asking for advice and what do I hear? Second kid is, oh yeah, I figured it out by the third kid is like, hey, first kid. Can you take care of the third kid? Leave me alone.

Yeah, it's been interesting so far. I think he's three weeks old today. So we've been having like less and less sleep every day. Waking up after, after about every 12 hours. But I'm getting used to it bit by bit. So it's been cool so far.

Congratulations. And thus now we are back at the CRO live hour. You took vacation for two weeks, but I don't know if it counts as vacation because I think it's just exhausting when you have tubeborn.

But such as mentally I rested, but physically I didn't really rest.

Can you actually mentally rest when you're physically tired? I don't know. But anyway, we are back. I'll see what are we talking today?

Today we are talking about winning holiday experiments. Like how do you come up with different winning holiday experiments? We also had a webinar sometime this week. Was it like two days before? And we're going to send out like the recorded replay. I also had you talk about some of the experiments that we launch here at Invest for companies that we work with during the holidays. We're going to be talking about that today. I have about three questions that I'll ask you. Just let me know when you're ready, then I'll just.

I'm ready, man. Just fire away.

Okay, cool. So the first question is how do you determine which elements of an E commerce website to prioritize for experimentation during the holiday season?

How do you determine which elements you need to prioritize during holiday season? I think we cannot answer that question without talking about the strategy, the experimentation and a B testing strategy or the CRO Strategy during the holidays and the strategy that I'm going to talk about, we've learned the hard way probably in the last 10 or 11 years. That's when we start focusing on the holidays and what kind of testing we can do during the holidays. And I have to tell you, our first experiments that we've launched during the holidays, as I reflect back at them, we knew that we have to have offers and we were testing different offers. But at the same time, we said, since people are highly motivated, they're coming, they're looking for a deal. We wanted to test simple site functionality just to see which one worked better. And some of the stuff was okay, hey, dropdowns versus radio buttons. Which one, with somebody who's highly motivated, would that have an impact testing that functionality? At some point we tried to test branding plus discounts and offers versus just offers and discounts versus just branding itself because we had one client who refused to do any discounts during the holidays. And they're like, hey, I don't care. Black Friday Cyber Monday, we don't give discounts. And over the years, I think what we've learned is that the strategy needs to be simple. If I may say, don't overcomplicate your holiday testing and experimentation strategy. Remember the goal during the holidays, during Black Friday Cyber Monday, probably until the end of the year, our goal is to maximize our money coming in, the ROI that we're getting from all the sales that we are generating. I'm not necessarily trying to learn a lot about my customers. Maybe if I'm learning, I'm just trying to learn which offers work best for them. Now, having said that, that really dictates what kind of experimentation we're going to do during the holidays. So usually whenever we're sitting with a company that we work with, we say, okay, so what are the offers that we're going to be focused on this year? And some companies go overboard with the amount of offers. I'm like, okay, don't complicate it. Let's just stick with two to three different offers. We'll test them out, see which one works best. And for each one of those offers, we will create different creatives. Now have the offer different offers from different angles. Three different creatives in terms of testing, we always have a banner at the top of the site that just works very well all the time, especially during the holidays. You combine that with making sure also on the collection pages or the product pages for E commerce that you mentioned the offer. We like to use scarcity and urgency as well, but they need to be authentic. So you're just not inventing, trying to trick people. And in the cart we just repeat the offer again, we apply any discounts automatically and we run those experiments. Usually our experiments launch midnight, 12am Eastern, that Thursday night. That's basically the start of Black Friday. And they will continue. We're constantly monitoring. And I must add that if the site has enough traffic, which most of the companies we work with have the traffic and the conversions, we are using multi arm bandit algorithm. For those who don't know what Multi arm bandit basically is an algorithm that reallocates traffic between the different variations you have in a B test based on which variation is generating more conversions. So the goal from an or multi arm bandit is to maximize your revenue during the testing period. We run all of that testing usually until Monday morning. We look at our winners and we use those. Now we started with three campaigns. We usually take the winning campaign and that's going to be our Cyber Monday that we're going to launch with. Now we have different creatives that are already ready based on all of this, prepped two months before the holidays. So Cyber Mondays, okay, here's our winning campaign. And then we continue running that probably Monday, Tuesday, sometimes until Wednesday. So what am I testing? I'm not complicated. The testing. I am going to be testing my offers. I'm going to be testing something globally across the website. I am going to be testing few elements on the placements of my offers on the collection pages, the placement and the copy of my offers on the PDPs and on the cart page. Outside of that, don't test, don't get really too fancy. People are highly motivated. Give them what they want. Just tell them here's what I have and give them what they want. In the webinar that we've had with Adam. And then it was just really fascinating because Adam focuses on email marketing and the holidays. He's like the first portion correct. Like he drives that traffic to the website and we try and convert that traffic. There are some strategies that he mentioned that I found fascinating. And then of course I talked to Ayat and she's like, oh, doing this for a while. I'm like, okay, great. So as I'm disconnected I guess because by the way, usually I have conversations with our team members regularly starting November. No one talks to me. Everybody just heads down working with their clients. They're just too busy to actually have a conversation. But one of the strategies is, let's say you're working with a brand. So an email will come out from the CEO of that brand telling people that, hey, holidays are around the corner. We are going to be running offers and then they give a calendar. So, hey, Friday we're going to run this promotion on this category and Saturday this promotion on this category and Sunday this promotion on this category. So they're testing different promotions on different categories and maybe in the last three to four hours they might even increase the discounts that they're giving. I thought that was fascinating. That was interesting. And of course it goes back to also the other piece that he said that I thought I'm like, oh, this is really cool. Segmenting the email lists and knowing what offers you make at what time to which segment.

That's a good point. I just want us to dig a little deeper into like your explanation. How many tests do you usually run within that period? Let's say maybe from Thursday night or from Thursday morning going into Black Friday. How many tests do you usually like, launch. I'm just asking this because maybe there might be someone who's listening to us who also have an E commerce site and they don't really know what exactly do they test and how many tests to launch.

So let's start with the assumption to answer that question, that you're going to launch a test on both mobile and desktop. So like one test that's going to address traffic and then you can. Your A B testing software will allow you to segment the results between mobile and desktop. So we're going to launch what we call a responsive test just to simplify life. If you have traffic, you might actually run desktop separate than mobile. Let's say it's one. So we will launch a test site wide. That's in the header site wide. I'll launch the test probably on my top landing pages, slash or. And the homepage that becomes important. If the homepage doesn't get enough traffic, lots of traffic, then I'll focus on the landing pages. If no, the homepage gets ton of traffic. Landing pages are different. So those might be one or two tests. So that's up to three tests until now. We will launch a test on the category pages, a test on the product pages, and a test in the cart and checkout. And usually the Kotlin checkout is one test that we run. So up to six experiments if you're running what we call a responsive test. So it's one test that's gonna run for both mobile and desktop. If you say, you know what? No, my mobile is gonna be different than My desktop. So I'm launching six experiments for mobile, six experiments for desktop. But of course that starts with the assumption that you have enough traffic, enough conversions. Sometimes I like just to combine them just to make life easier. One responsive test for both devices. You just need to make sure that your AB testing software allows you to segment your results based on the device. Correct. Hey, we've run test, but I want to see what is the conversion rate for mobile and what is the conversion rate for desktop. And I want to slice and dice the data and I have to do like a shameless plug over here for FigFi because we provide that out of the box. You run a responsive test, we allow you to segment not only by device, but by the type of traffic that people are coming, specific campaign and all that.

You also answered it like very well. And I'm just thinking like when you run like a B test on desktop and mobile devices, like maybe at the same time, doesn't that mean more development in QA work for your team, given that you have to be moving fast during that Black Friday period?

Yeah. So this is interesting because this is always a discussion with our dev team. Ultimately, I think it's about the same amount of time if you're going to do separate mobile desktop. Unless you're going for a completely different experience, usually the first platform that you're going to program the A B test for will take you the longest. So that platform, let's say you have an experiment and it's going to take eight hours. Let's say you're developing the test first on mobile. When you go to all the tests for desktop, you already have most of the code. You need to make a few changes. So it might take a couple of hours. So in total you've done 10 hours. Now, if you're going to do responsive, it is probably going to be close to 10 hours. So it's about the same, by the way, whether you're doing responsive or you're doing combinations. Sometimes actually doing it as responsive cuts down on the time by 10 to 20%, which is fascinating. It's not exactly equal. You're not launching two tests, you're launching one test. And the time is not the combination of both. It's about 20% less. But remember, we're talking about not too complicated experiments. We're talking about banners, we're talking about special offers. I don't anticipate that development time for any of those experiments will be long. Most of these experiments are what we call low effort experiments. Low effort experiments are experiments that take less than four hours to develop. That's what I want. If you have an experiment, this is something that we do. Whenever we have an experiment, we classify it as low, medium, high effort. It's important for us because that really helps us understand how long it's going to take to implement this experiment. Low effort experiment is less than four hours. Medium effort experiments is anywhere between four to eight hours. And high effort experiments is more than probably eight hours. Sometimes it's usually around 16 to 24 hours. If I see that there is medium or high level of effort experiments in my Black Friday, I'm stopping. I'm like, hey, what's going on here? Why do we have high effort experiments or medium even? Because our experiments should be fairly straightforward. We're not trying to learn a whole lot about our customers. We're trying to maximize our revenue during the holiday season.

That's a good point. Just one more time, let me just try not really pushing back, but I just want to hear what you will say.

Go for it. Push back, my friend.

Some of the web elements that help increase conversions on desktop may actually affect conversions on mobile devices. What I'm trying to say is that what works on desktop might not necessarily work on mobile. So if you launch sendcat on both devices at the same time, did you think that might end up affecting one device than the other?

So I agree with the fundamental premise that the visitor who comes to your website on mobile is in a different mindset than visitors who come to your site on desktop. I completely agree with that. I think that's a correct assumption. I think we see that all the time in our testing. And that's the reason whenever even we launch a single test, responsive test for mobile and desktop, we have to segment just to really understand how mobile visitors are behaving compared to desktop visitors. But there is always a but correct. Here's the thing. Holidays are a little bit different because we're focused on offers and offers and the visitors are also coming and they're highly motivated and they're searching for discounts. Let's be very frank and honest about it. So it's all about how do I show them the best offer in the best way on mobile and desktop. So while people are a little bit different throughout the year, I think the holidays is a little bit less, especially if we're just focused on offer testing. So that is something that we want to keep in mind. Now there's a whole other complexity. Let's say you launch mobile versus desktop or you launch a responsive test you launch mobile versus desktop and offer one number, one wins on mobile and offer two wins on desktop. What do we do in that case? That's the question. Or even in responsive, you launch a responsive, but then you're segmenting the data and you see the same thing. Offer one wins on mobile and offer two wins on desktop. What do we do? Holidays are very special because we are trying to maximize our revenue. So I would actually say, you know what, for mobile, this is the offer that we'll have for desktop, this is the offer that we have. This is because we have very limited time. So we create this what I call a split in the path for mobile, for desktop. Because really the data is showing us that they are different outside of the holidays. It's a completely different discussion. Because outside of the holidays you're not testing offers. Correct, you're testing, you might be testing functionality, messaging. And the minute you see that mobile versus desktop are a little bit different, you actually have to stop and say, okay, how many visitors are getting to us on mobile? How many visitors are getting to us to our site on desktop? We usually like to go with one message, even if we end up saying, you know what, this version is really winning on desktop. But really desktop doesn't matter to me. Most of my visitors are coming on mobile. I am going to go with the mobile version and that's going to be even the default on desktop. Why do we do that? Because splitting the code base into here's mobile, here's desktop sounds great for marketers. Hey, we want different experience, but maintaining that is not so easy for developers. And remember we talked about one test, correct? Oh, here's the experience for mobile. Here's the experience for desktop. 300 experiments later, maintaining the website could be horrendous. It's just asking for trouble. Unless fundamentally the website is architected to handle those two different experiences. Again, as you notice. But unless. Caveat. So there's a lot of thinking that needs to go into that.

I agree with your explanation and it also makes a lot of sense. It makes a lot of sense because I'm also thinking that like some of the people, they might visit the website using their mobile phones and then maybe they purchase on their desktop. But what if like they did that, like they took that same path and they see like the offers are different from what they saw on the mobile device and what they saw like on the desktop, you guys, I'm just thinking might be far fetched.

Happens all the time. Happens all the time. By the way, typically we say it Is what it is. Offer here, offer there. It is what it is. Just the amount of effort that takes because it's a completely different visitor. It is the same visitor, but for the AB testing software, it's a completely different visitor. Unless you have very intelligent AB testing software that says, oh, you came from here. I can look at your signature, I can compare it, I can, like, match it. And this is a lot easier if you have what we call logged in user. So I'm like, oh, it's the same user id. Okay, now I showed you this offer on mobile. Now you're logging in, coming back to the website, I'm going to show you the same offer. Not everybody is logged in on devices. That's number one. This becomes a much bigger problem. By the way, the mobile versus desktop. Different offers or different experiments, variations when you're testing pricing. Ooh, this item on mobile is for $80. On desktop, it's for a hundred bucks. By the way, this happened to me just a couple of days ago. So I somehow joined. Not somehow. I actually looking forward to it. A book club. So a whole bunch of friends reading a book together. And I'm just busy all the time. And also I visited my doctor and he's not moving enough. And I'm like, really? I need to move. I guess I am getting older. Yes, you're not in a good. You just need to start moving. So I'm like, okay, let me get an Audible subscription. It's like $12, $14. And I'm like, okay, I can do it. And I get one book basically per month. I go, and now I click on their ad from Google Ads and I land and I see a flicker where initially it starts by saying, you start your subscription with $4 a month, but it's an A B test. And I see the Flickr and then the version that I was getting served showed me that, oh, start your subscription at $14. And I'm like, no, I won the $4 offer. So I'm like, okay, let's go back. I knew exactly what's happening. They're running an A B test. And I got stuck with the I'm not getting the offer. I'm paying $14. I'm not getting the $4. And I didn't know any other details. So I'm like, okay, let me go back to, like, Surge Chrome. And this was on my mobile device, incognito mode. I come, and again it flashes, and I'm stuck with the 14 offer. I'm like, really, guys? I want to See it. So now I want to see. So I go, I open mobile device, I go to Safari Search. So by the way, because they're running an A B test, I clicked on their ad three times, I made them pay multiple times and then I got the offer and it's $4 or 4.99 for four months. So I'm like, oh, I'd take that. So I subscribed. But again, yeah, I am aware of what's happening. I wonder another visitor who's not aware, who will see that the price flashes in front of them. That's horrible. Usually I don't care much for Flickr effect. I just say it's a fact of life. But in this case, the 499 versus 1499 was so clear that I'm like, oh no, that's the difference. Correct. Of $40.

So I also use audible, but I don't pay top baths. I think like I got a deal where I pay dollar for the first three months. So that's like dollar to credit. Okay, so I think like it's also based like on the location.

Hold on, hold on. Now I feel like I need to. I'll do the $4, then I'll figure out the location. It's funny because Spotify in Turkey, when we were living in Turkey was like so cheap. I think it was $3 a month or $4 a month. And then when we came to the US it was like $15. And initially I'm like, oh, let's continue using the Turkish plan. But then they looked at our IP and they're like, oh, you are in the U.S. we have to switch you. And I'm like, okay. So then by the way, next time, what I've learned is if you're in Turkey, you get the Spotify subscription. You pay for a full year. So even when you come to the US it's still Turkish money that you paid anyways.

Okay, moving on to the next question. Are there any key metrics that E commerce businesses should track specifically during holiday experiments? And how do they tie into the long term success?

I read multiple times where people argued that you want to look at conversion rates or you want to look at revenue per visits. Revenue per visitor. I think by the way, in GA4 now they have a different fancy name. I don't even remember. It's for revenue per visitor. Initially when I saw it, I'm like, what is that? And then I look at the formula, I'm like, dude, why would you change the name of the metric that we've used for the last 15 years. It makes no sense to me. The same thing, by the way, when they, like now they remove bounce rates and they replace it with engagement rate. I'm like, really? Why? Why? But again, I guess that's their product, so they get to do stuff like that. But I digress. Some people like to look at, you know, rpv, some people like to look at conversion rate. I actually look at something else. Profitability. And that's a open conversation with every client, every company that we work with. You have to have a full strategy. When it comes to the holidays, you might be losing a little bit because you have lost leaders. So you want to bring people in knowing that I know that if somebody places an order with me, they will place another order within 30 days or 25 days. So you need to understand how frequently your people are ordering from you. So, yeah, I might be losing money on the first order, but I know that's on average, somebody will place an order with me within 45 days. And within a year, maybe they'll place six orders. You have to have enough data set to do that analysis before you run a specific offer or discount for me and for my team as a CRO, like, hey, the higher the discount, the easier it is for us to increase conversion rates. But that is a stupid business strategy. I always sell every company we work with. I can increase your conversion rates to 100%, offer your products for free. Everybody's going to buy, we're done. Stupid. We don't want to do that. We want to think about profitability, lifetime value of a customer, the frequency and the recency of our customers and how often they place an order. That is the framework that we need to look at. Yeah, I am tracking all the other metrics, but I think the work of a conversion rate optimization specialist is a bit more complicated. To pull the data from analytics to look at the recency, frequency, lifetime value of a customer, to say, okay, here's the data that we have, here's the offers that make sense, even in instances where we talk about, okay, we're going to offer free shipping. This is an experiment that we launched a while back. I don't know if I would test it again, but what is the threshold that would offer the free shipping? Correct. I want to think about that. But that requires you to dig through the data. You want to look at your abandonment rate cart and check out abandoned rate, because if somebody had clicked and added an item to the cart, they're highly motivated. Okay, but then they abandon what Is that abandonment rate based on average order value? My abandonment rate is 50%. Okay. No, you think it's 50%. Let's look at the cart value and let's look at the abandonment rate. What is the abandonment rate? When the cart value is around $40, $45, $50, 55, 60, you might find out that the abandonment rate at $40 is only 10% and then the abandonment rate at $90, it's actually maybe 80%. So thinking about that, then saying, you know what, listen, maybe if we offer free shipping at higher level orders, then I'm going to reduce that abandonment rate from 80% maybe to 60%. So again, there's a lot of data and lots of insights that you can pull from just saying, oh, a generic number to digging through it in that analysis and then basing your business decision using those metrics.

That makes sense. Could you discuss the role of user feedback and customer behavior analysis in shaping the design of holiday experiment?

I don't know. It's going to sound a little horrible. I don't know if user feedback plays a huge role in the holidays in the big scheme of things. Correct. Because again, users are highly motivated during the holidays. I start with that premise. They're coming because they are looking for an offer. Now, user feedback matters to me a lot and figuring out which offers work best and this works more for established brands. Look at the messaging, look at the offers that worked really well last year and the year before. Sometimes most of the time you don't have to reinvent the wheel. Let's look at the creatives that worked really well. Let's look at the offers that worked really well. Let's look at the offers that lost as well. Maybe we want to test with one of them and we want to avoid the rest. That's feedback that you're getting through the data is what mattered to me during the holidays. Actual customer feedback. I don't know how much you will do of that now. I might stand corrected. By the way, somebody might say, actually we do quite a bit of user feedback during the holidays. I would be interested if one of our listeners says, actually you need to consider this angle. I'm like, okay, that's something that I'll be interested in learning about. What do you think, Simba? Do you think user behavior and user feedback matters during the holidays?

I think like when it comes to direct feedback from customers, I think they can highlight maybe their preferences, their pain points and expectations during the holiday season. Like you can reach out to them. Maybe when you do those Kind of customer interviews, jobs to be done, customer interviews. And when you hear, like, much more about their preferences, pain points and expectations, then that's how you can take that feedback and also try to inject it in your quality season offers. I guess, like, maybe that's also something that you can do even when it's not like a holiday.

Yeah, that's the challenge that I see with this, because I should be doing that all the time. I don't know if there's anything special for the holidays that I'm like, okay, yeah, this is particular to the holidays. I think you'll do that throughout. And I think, by the way, the other thing, it depends, I guess, on how large the organization is. The holiday is always a mad rush. I don't think many people use their feedback, but it will be interesting if we learn something different.

Those are the questions that I had for you. Unless maybe you want to add something.

No, you always ask me that question and I'm like, usually by the end of the episode, I'm like, I'm tired. But that was an excellent discussion. I want to ask our listeners, wherever you listen to this podcast, leave us a review. By the way, we should put some effort to getting more reviews for the podcast, and maybe that should be one.

Of our goals, considering our listenership. Like, we have a lot of people who listen to us. And I also had some people reaching out to me on Valentine saying that, oh, they enjoyed their listening to the CRO layout. But then it is at that moment, then I should say, oh, okay, why don't you just drop a review with JPEGs?

Yeah, yeah, no, I agree. Are agree. Thank you, everyone. And until next time, hot testing.