#47 – Black Friday and Cyber Monday CRO Strategies For E-Commerce Websites

There are a few lessons that you learn when you run experimentation programs during Black Friday Cyber Monday. Let's start with the first one. That if you are not running experimentation during Black Friday Cyber Monday, you are leaving money on the table. It's as simple as that. My name is Carlos Slade, and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we'll talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high, testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts. And another week, another CRO Live hour. Can't believe it's almost the end of the year. I don't know how I feel about this. I have a weekly ritual where I water plants every Monday. Today is Thursday and I forgot to water the plants. I can't believe it's Thursday already. Crazy, crazy week, crazy time. And I can't believe we're starting to talk about Black Friday Cyber Monday. Simba, how are you doing? New episode of Sierra Live Hour.

I'm great. I think nectarine definitely moves fast when you're busy. It's crazy that like I've been at invest for four years and I have seen four Black Fridays and I know, like how team gets so busy was like, I remember at some point there were days some of the team members, they were working like throughout the night.

Yeah, yeah. They don't sleep. Basically. There are some team members who actually don't sleep. And we are on shifts, actually during Black Friday Cyber Monday. 12 hour shifts where we switch back and forth. The first time we focused tremendously on Black Friday a few years ago, we didn't really think about making shifts, so everybody stayed up. Not a very good idea because I think we made a couple mistakes. When somebody's working Continuously for like, you know, 72 hours. You have the right. Make mistakes at that point in time. But it's funny, by the way, when somebody joins invest, one of the things I do as we make them the offer, you know, to join us, I run them through our PTO policy. Here's how much time you take off. Here's the official holidays at invesp. And I pull them up and I read them, and I think most people don't notice, so I tell them, please notice that Thanksgiving, it's not a day off for us. And everybody's shocked. They're like, whoa, what do you mean it's not off? I'm like, I'm sorry. This is our busiest day of the year, as a matter of fact, from 12 o' clock Eastern to that Thursday night or Friday morning, I guess, and until Tuesday or Wednesday night, it depends on what campaigns we're running. That's almost like for most of the companies we work with, that is 40 to 60% of their business during that period, which is crazy, you know? And I'm like, yeah, we're working with companies and they're up and running and running campaigns and monitoring campaigns. We're doing the same thing. So everybody understands it. Although they're like, really? I'm gonna have to pull this off. I'm like, yeah, I'm sorry. There's not a whole lot I can do about that. So I was going to ask you.

Like, about the process and everything, but it's good that today we're talking about Black Friday, so I'm sure we might not maybe finish everything in one episode. We might need maybe another episode. We might also need. Maybe there. Maybe if you can also like, invite Ayat to join us because she does a lot of Black Friday optimization, so that might be helpful.

Yeah, it will be good to have her kind of join us. We just need to figure out the time because she is busy with client meetings at owing to the point that I'm like, yeah, I dare not ask her. Hey, can you join us? But we'll, we'll, we'll manage to squeeze something out of her.

Okay, cool. So today we're talking about Black Friday. You've been doing like, Black Friday conversion optimization strategies for years now. Last year we did above 37 million for clients. I'm not sure about the previous year.

Oh, I think the 37 million of additional revenue during Black Friday was actually the previous year. Not 2022. 2021. 2022 was above $70 million of additional revenue added on top. Kind of regular conversion. Like, you know that anyways during Black Friday. So we added about $17 million more dollars during that period.

Yeah. So that's interesting. So it should be great, like maybe if you kind of like walk us through the strategy. So the first question that I had for you is, what are some of the most significant lessons you have learned from your past CRO strategies during Black Friday and Cyber Monday campaigns?

There are a few lessons that you learn when you run experimentation programs during Black Friday Cyber. Let's start with the first one, that if you are not running experimentation during Black Friday Cyber Monday, you are leaving money on the table. Simple as that. Which is, by the way, counterintuitive to what many companies typically do during Black Friday Cyber Monday, where they say, hey, we have a code freeze. We're not going to be changing any code on our website. Most companies, especially if they are larger, by the end of October, they say code freeze. No changes counter to what you should be doing during Black Friday Cyber Monday during that season, you have a highly captive audience that is ready to convert. And your job as a marketer, your job as somebody who's running E commerce, your job as somebody who's running SaaS, because even SaaS now joins in Black Friday Cyber Monday is to try to get the most out of those visitors who are highly motivated, who are looking for a deal to say, oh, we're going to code freeze. I'm sorry, that was fine 20, 25 years ago. And the reason technical teams said code freeze, and I understand why they tried to say, you know what, we don't want to introduce bugs. And instead of doing well and increasing our sales, we introduce a bug and then that causes a drop in conversion rates, that causes conversion issues on the website. So let's not touch the website. The fact that we might run into an issue during the holiday season does not justify that we are going to sit around and do nothing. And mind you, I understand that development teams are not stopping their work. Right? They work on features, but they don't release them until post holidays. But what I always tell every company is have a good quality assurance program in place, have your technical teams ready to respond to any issues that might come up as a result of a bug, because you are bound to run into an issue there. And instead of saying, oh, well, we're not going to risk anything, say, no, guess what? And people are going to hate me for this. No, we're going to push really hard and maybe, you know, just like how we have it at invesp, you know, we have shifts and people working around the clock because it is a make or break for most retailers. For most SaaS companies, I would say most retailers, perhaps even more important than SaaS companies, you want to push and you want to capture. And we also have to kind of associate what happens during the holidays and during the season with the fact that the economy everywhere is not doing really well. Last Black Friday Cyber Monday were not really as good. The previous years, the growth there was not as good as people had hoped. So we don't have the luxury of sitting around and saying, well, yeah, we'll just wait and we're going to do code freeze. No, we're not going to do that. Now. No one has ever been fired for hiring an IBM. Correct. It's like you do the same thing and you're fine. And I guess in normal times, I understand that, but we are not in normal times. Let's talk about the US Economy. Growth for the last quarter was not as people had expected. You know, I think it's about set of 2, 5. It came out to 2.1%. We're looking at net overall growth. So we got to maximize the results that we get from the upcoming holiday season. That's number one. Number two, kind of piggy backing off of the first point. Be ready and be prepared. If Black Friday Cyber Monday are very important and essential, critical seasons for your business, don't leave the prep work for the campaigns that you're going to be running. The offers that are going to be running, don't leave them until beginning of November. You're opening yourself up for disasters that could happen. For most companies that we are working with, we started the conversations talking about Black Friday Cyber Monday from kind of, we're starting them actually next week saying, hey, what are the creatives that we're gonna be running? What are the offers that we're gonna be running? Having that discussion starting. We're not gonna finalize everything from the first week. It's a discussion. It might take anywhere between two to four weeks, but we like to have most of our strategy, at least 85% of it, done by the end of September. Part of that, by the way, as well, people don't pay attention to that is making sure that whichever company we're talking to, whichever offers we're thinking about, whichever creatives that we're going to have and we're going to run, we're also talking about logistics, delivery. Do we have enough items in stock? Nothing worse. We worked with this one retailer, absolutely amazing. Black Friday Cyber Monday campaign, they missed out on a small detail. The item went out of stock, like, you know, two, three hours into Thursday night. And I was like, really? So we had to wait list the item, and people were joining, and I'm like, oh. And then they couldn't get the item. It took them about three more months to get the item, and it was just a mess. So Black Friday Cyber Monday need a holistic approach. But that approach and that strategy should be done as early as possible. Now, that does not mean that, hey, let's say it's mid October. Let's say it's end of October. I'm like, it's too late. No, it's never too late. As a matter of fact, one of the very first, very strong campaigns that we've done was with Z Gallery, the large furniture retailer. And we started working with them on Black Friday Cyber Monday about a week. Only a week before Black Friday Cyber Monday was not a good idea. But the lifts that they saw from the experimentation that we did during that those three, four days period translated into millions of dollars. And actually, Karim Oskay, who was the VP of marketing at Z Gallery back then, did an amazing case study with us talking about just kind of the results that they get from Black Friday Cyber Monday. The third lesson and kind of links into kind of thinking about Black Friday. One of the things that we tried early on is to say, hey, do we focus on the brand now? As I say it, I'm like, okay, that's a little silly. Black Friday, people expect discounts. That's really what works. It's people are coming shopping for deals, trying to highlight the brand and whatnot does not necessarily make sense. You might experiment with that, but in our experience, it's not very helpful. So that's the third point, the fourth point. Remember that it's not only about the sale. Correct. You need to have a full strategy, because you might capture new demand that you don't typically have. And the question is, it's not enough to convert those people because we might be giving them very heavy discounts, but what is the full strategy to get them to order again, back again for us afterwards? Your sales are not limited to that Black Friday Cyber Monday, really. Your sales start at Black Friday and continue to post Christmas with the New Year's offers that you might have. So you need to have a full strategy. Okay, I've converted them a Black Friday. I have another offer. How can I maximize the revenue that I'm getting from that visitor? That means sitting back, strategizing. Coming up with full cycle, full length marketing campaigns on how we convert those visitors and bring them back to our shop.

I think those were really good lessons and I hope like people are also like listening to this news there. I read it somewhere that you can't really use like the Black Friday and Cyber Monday insights outside of that period because like the kind of visitors that you get during Black Fridays and the typical visitors that you get like any time of the year. What's your take on that?

I'd love to say no, that is not true. But it is somewhat true. Most of the experimentation and conversion optimization and AB testing that you do during Black Friday is not necessarily focused on generating insights. Most of it is focused on maximizing profits and maximizing revenue. Really, that's what it is. Again, you have a highly motivated visitor. That visitor is motivated by discounts. What kind of discounts are you offering me now? You can use that. You know, let's say you have a large quantity of an item and you want to move it out, you might be able to do that. Now the question is, so those visitors are motivated, your job is to trigger them to act. Correct. At that point in time that they came. People are shopping for deals, they're walking around looking for deals. So how do we actually trigger them to act on our website and to act right away? The lessons there of, oh, our visitors respond really well for buy one, get one free. No, our visitors respond for scarcity and we're talking about true scarcity. You know, our visitors respond well to time limited campaigns. It's amazing because sometimes you think that these concepts work really well across domains, but that's not always the case. Correct. It depends on the type of visitor that you have. We work with and we've been working with them for a long time. An apparel retailer targets ladies who are age 55 and older. So an upper middle class average order value is about close to $400. The visitor is motivated during Black Friday Cyber Monday. Correct. But what works with that particular brand doesn't work with another brand. Again, apparel brand targets ladies who are age 25 to I would say about 40. Average order values about $125. Two completely different and they sell different items. But the strategies and tactics that work on brand number one, the luxury apparel brand don't work really well on brand number two. More on the lower investment, lower price tag per item. So yeah, are there deep insights? I don't know if there are deep insights that you can drill not to like, you know, something similar to when you do experimentation and a B testing. But at the same time, you can understand the triggers that work really well for your visitors. I think that's what Black Friday Cyber Monday can do. But your approach to Black Friday Cyber Monday, even holiday season, and I think that's where fundamentally experimentation is different between, you know, very high season, high demand to traditional season, normal, average demand. Your approach is, let's maximize the revenue during that period and we compliment that with a full strategy that really helps us now. Okay, so the visitors are coming right now. They're highly motivated. Okay, we've converted them. We might not be making as much money off of them because we're offering them discount. How do we actually bring them back so we can make more money off of them?

Okay, that's a good one. So, but how can you use, let's say, maybe the insights from the previous Black Friday campaign on the next one? Is there like a way that you can do that?

Yes, there's a lot to be learned from previous campaigns. Don't throw those ones away. You're constantly trying to optimize. So in 2021, we've tested different concepts. Take one brand, like an already tested brand versus specific offer, limited time offers. So there's different concepts that we've tested and we saw that, okay, well, you know, there are winners over here. We saw consistently that the winners are limited time offers. We've noticed that offering items with like, you know, a 10, 15% discount and really encourage people. We've noticed that offering discounts above 50% converts at the same level at the 70%. Which is fascinating. Correct? Because, okay, we like the 50% discount. And anyways, we're moving items that we have stockpiles of. So kind of taking some of those lessons and saying, okay, so we know this is the kind of offer that worked really well for our customers, you know, so that is going to be the main control that we're going to run against. Now. How do we beat it? What are different creatives that we are going to be focused on? What is the messaging around those creatives? So Your baseline for 2023, your baseline for this year is going to be what won last year. And now since I have a baseline, I know it did well for me last year, I need to beat it. So I'm going to have, anyways, a lot more conversions during Black Friday Cyber Monday during the holiday season, so I can afford to introduce four or five challengers to my control. What's fascinating is, is typically whenever running an A B experiment, even for high traffic, high conversion sites, typically we don't like to introduce more than two, three variations, challenging, maybe four if we're feeling like, okay, listen, this site, I was looking at one of the sites and you can introduce 10 variations and you can conclude within a couple of hours they just have the conversions correct. It's funny because somebody was telling me is like, not only can I introduce 10 variations, 10 challenges to the control, I can go for what we call an mde. So usually, what do you expect the experiment, what kind of result do you expect to generate a 10%, a 15%, 20%? The higher your expected lift, what we call the MDE minimum desirable effects, the faster the experiment concludes. So an experiment that you say, oh, I expect this experiment to generate 10% lift will run maybe as an example for 10 days. If you say, well, no, same experiment, but I expect it to generate a 20% lift, it will conclude in five days or even less. So usually with most web experimentation, we go for an MDE anywhere between 10 to 20%. It depends on the side, depends on the conversions during Black Friday, and it depends again on the side. You might have 10 variations, you might go for an MDE of 1%, and you still generate results within three, four hours. Not everybody's blessed with that level of traffic and that level of conversions. But guess what? When you're at that level level, a 1% even lift or 2% lift is millions of dollars. But for the average Joes, the rest of us, we're going to look at what won last year. We're probably going to introduce four or five variations, try and beat it. And the idea there, by the way, and I think this is fundamental to keep in mind, it's not just a matter of, oh, well, here's one variation. Oh, I just changed the color on the button. Oh, I have a banner on the website. And instead of black, I'm going to go with red. Okay. I mean, you get to do that, but don't expect massive results as that. Each variation you introduce is focused on triggering people. And there's a fundamental psychology of conversion optimization principle behind it. Silly example, but just kind of to complete the answer, maybe I'll test a site wide banner. And one banner is going to talk about 30%, you know, limited time offer whatnot. Maybe the second, the second variation and the A B test could be, oh, instead of the black banner, we're going to go with red. I'm like, yeah, but really, is that the variation? Like, you know, okay, I guess you can test that, but maybe something more meaningful. The banner will Have a countdown timer. Correct. Now I'm going through something a little different. So you need to think about really substantial changes in the variations that you are testing against the controls, which is what won last year.

That's a good one. Maybe one other question that I have is we are talking about AB testing and all these things. What do you usually like to test during next Friday? Because there are certain pages that you probably don't test and there are certain pages. Do you only just focus on manual pages or you also get into some pages?

We test all over the place. On E commerce website we are testing everything and mainly what you look at is like we're looking at our top landing pages. People are landing on those pages. If we're running any specials, if we have email landing pages, those are prime Candace for testing. But we run also our testing on product pages on the cart, on the checkout ask area. That is we test on the homepage and the collection page. So we're testing all over the place. Now the trick there because you're running lots of experiments, correct? Sometimes if we're testing the same hypothesis. So that's a multi page test. So I'm testing let's say a site wide banner. Correct. It needs to be consistent and that's a site wide test. But let's say I'm running a special offer where I'm going to test it, then I'm going to show it on the collection pages, I'm going to show it on the product pages, I'm going to show it during the checkout. So you have to be careful with the design of your experience. Because I've also seen teams where they run one test on the product page, a completely different test on the category page and different tests on the landing pages. And I'm like, you know, there's a problem because it's one visitor who's going through this journey. But you're testing different things and your data may or may not be accurate. So you want to be fully aware. Test everything but fully aware of the possible interaction between your different experiments.

That's also like a good answer. I'm also like thinking as I'm asking you these questions, I'll be catching some of the tips from the answers that you're giving me because they are really good answers and we have to share them all over social media. One last question that I have for you which is what aspects of CRO are particularly unique to Black Friday and Cyber Monday compared to regular E commerce operations?

I do think the amount of prep work that needs to go into the Black Friday Cyber Monday holiday campaigns. Definitely not usual, like, you know, to start prepping campaigns with. The level of intensity that we go through is very unique to the holidays. That's number one. Number two, whenever we talk about the holidays or high season, by the way, lots of times, especially the holidays, Black Friday Cyber on Monday in particular, we go against many of the CIARA principles that we have and we go against fundamental principle of conversion rate optimization. Typically, whenever we're creating an experiment, we are always asking, hey, what are the deep insights that we are aiming to learn? What's going on here? What are the deep insights, I dare say during Black Friday Cyber Monday? That's not a question we ask, it's a completely different question. That question is, how do I maximize my revenue? Yes, there are insights, but really they're not driving the experimentation program. In this case, the driver is a lot different. This is not to say that's when you're doing your typical experimentation, a B testing, you're not looking at maximizing revenue, you always are. But usually the insights first kind of in the driver's seat and the revenue is the person sitting next to the driver. It's flipped during Black Friday Cyber Monday where the revenue is in the driver's seat. And on that note, also not very typical. Most of the experimentation that we do during Black Friday Cyber Monday uses multi arm bandit algorithm. Most of the experimentation that we do outside of Black Friday Cyber Monday are using Bayesian statistics, two completely different methods. Bayesian statistics says, you know, here's the control, here's the conversion, right? Here's how many variations I have. Here is the confidence interval that I'm aiming for. Here is the what I expect the test to generate during using a multi arm bandit algorithm. I'm saying, hey, software, please maximize my revenue. I have four variations. And what the software does, if your software supports that, which I think most a B testing software supports these different algorithms. Some of them you have to pay additional money for to get that feature. But if you are serious about Black Friday Cyber Monday, I highly recommend using that. What the software does, it's constantly monitoring the results and redistributing the traffic. Let's take a simple example. Two variations, one control, I guess, one variation you can extrapolate from there typically is 50, 50 correct, 50% to the control, 50% to the variation. That's how you start. But because you are running a multi arm bandit, what that does says, oh, you know, the variation is doing really well. Let's give it 2% more traffic. So now it's 52 to 48% this traffic distribution. Oh no, the control's doing better. So let's change this. And I'm oversimplifying this but I think you get the point. Oh, let's go to like you know, maybe 49, 51%. Oh no, no, no. The variation now is really doing really well. So it constantly evaluating the traffic which is by the way was something that Google Ads does. I think Facebook algorithm also kind of maximize the traffic, maximize the conversions that you are getting as a result of running the experiment. Those are kind of fundamental changes. Correct. We don't do multi arm bandit during the year unless again by the way somebody has a specific, let's say campaign for Mother's Day or Valentine's Day. Multi arm bandit works really well when you have a campaign for a specific period and you're trying to maximize your conversions. That's really what we're aiming for. So that's when we deploy that software and I have to plug in figpy over here because that is a standard feature that we have in figpy. If you have FIGPI now you will get the multi arm bandits but you will only benefit from multi arm bandit if you have the conversions. Correct. If you're generating, I don't know, a thousand conversions, you might not see benefit from multi armed bandits because those are kind of really tough statistical decisions. And if you don't have enough conversions, the algorithm is not going to work for you. So that's the reason I say like if you have the conversions and you need your variations to be. Let's go for something strong. There's a difference between the different variations, between the control and the variations that are challenging which we, we just talked about. That's cool.

So for the testing tools that you usually use during Black Friday maybe just for some people who might want to get into this, I heard that you mentioned figpi and also like earlier on today when I was listening to your other podcast you were also talking about gives you the details of what products that people bought or something like that. Can you kind of like maybe just touch on it a bit and then.

It turns One of the things that we've noticed that about at this point about 70 to 80% of the customers who are using FigPi, there's about 3,000 different companies using FigPi, maybe actually even bits more in the last few months because we had so many people sign up. Most of them are E commerce now. Typical a B testing tool. Here's the control here's the variation. We're done. Goodbye and good luck. Well, if I'm an E commerce operator, I'd like to know more data about the experimentation for this variation. Which products are selling better? Correct. That's deeper analysis. This variation does really well for products 1, 2, 3, 4 does actually not so great for products 6, 7, 8. So that the amount of data that Figpy collects because it's a full analytics engine that is really monitoring your e commerce website. It just provides you with the rich amount of data. And then this has been released actually quietly to a few of the e commerce companies. Actually it's released to everybody, but it's hidden. Not many people know how to access it because we're kind of monitoring the results, looking at the reporting because typical reporting and AB testing tool is number of visitors, number of conversions done. Goodbye and good luck. Now imagine for a variation now you need to show the different categories and how they're performing, the different products and how they're performing. There's a lot more data. So how do you present that in a meaningful way where people are actually able to make actionable decisions based on the results that we're collecting? So that's something that is just fascinating. I think the couple of customers who we've talked to and showed them the data is like, hey, it's already running on your website. See, they're like, oh my God, this is a just a trove of data that you're providing us. And you know, there's always this fine line of providing too much data where somebody says I don't know what to do with this versus oh, this is actually actionable. I can actually benefit from this data. So that's one thing that we're focused on in figpi. I'll mention a few other tools I know, I mean I run fake PI, but I have to mention a few other tools that we use during Black Friday Monday. Specifically vwo. We love vwo. Fitpy is not right for everybody. I mean, VWO is also another solid tool. It's sort of strange, but not many of the companies I work with use optimizely. So that's kind of fascinating because at some point we had a lot more companies using Optimizely and optimizely is kind of leader in the space. But I answered somebody who sent me a message earlier and they said, hey, what's. What do most of your clients use that tell them it's either FIGPI or vwo. Those are kind of the two tools that people use.

Okay, sure. That's great. Those are the questions that we had for today. Unless you'd like something to add.

No, I think I'm excited about this upcoming holiday season. I am watching carefully kind of the US Markets. I mean, that's what I watch because I think where the US Goes, the rest of the world follows economically. So I'm watching carefully just to see how things will play out in the next three to four months. I think the early indicators that we're seeing are promising now. We have lots of political instability all around the world, even in the US you know where next year is going to be? It's going to be interesting because we'll have an election year. Tempers usually flare and there's lots of polarization. What's the impact of that on the economy? It'll be interesting to see how that plays out. But let's see how things pan out in the next three to four months. Well, Simba, thank you, sir. Until next time, happy testing.

Ciao. Bye.