#54 – Post-BFCM Experimentation Strategies

What is the difference between the holiday CRO program? I don't want to say program, but the experimentation that you do during the holidays and the one that you do after the holidays. Because that, in my mind I'm trying to think, like, right now we are focusing, like, on the holiday experiments. Then at what point does that switch enough to say, okay, cool, now we're going back to, like, no matter what changes. Exactly. Is it like the data that you look at or something different?

I think your full analysis of what happens holiday versus non holiday. Let's start by visitors. The mindset of visitors. Completely different. Correct. My name is Carlos Slade, and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing experimentation and conversion rate optimization.

Each episode Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do AB testing results fade over time? How do you go from low to high, testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do and certainly like Simba does, subscribe to the CRO Live Hour podcast today. Wherever you listen to your podcasts. Another week, another episode. CRO Live. Simba, how are you doing?

I'm good. Did you see the CRO live hour numbers?

I saw the CRO live, our numbers, but I'm a CRO, so there is no baseline, like, oh, 100% increase. I'm like, from 1 to 4 or to 5. That's 400% increase. Did the numbers look good, though?

I think that shows also, like, an improvement because one of the things that I noticed is 90% of people listen to CRO. They discovered the podcast this year. So it was like, oh, cool. That's great.

That is awesome. That's awesome. We've been doing this now for two years probably, by the way, two years into the CRO Live Hour. And I wonder sometimes I'm like, okay, the format, is this the best format? I enjoy talking to you, but I wonder, hey, is there better formats? Okay, so I think that we're putting good content out there. But you know what I always say, you might put out the most amazing content if no one knows about it. You are putting the most amazing content that no one knows about. So no one cares. So I wonder how can we promote the podcast even more? Even CRO Live hour even more. So those are some of the questions I've always asked myself.

Yeah, those are good questions to ask. But one of the things that I also, I admit that we really haven't promoted the podcast like the way like it is growing. It's just like happening organically and of which it's good. And in the past few weeks I began to see the CRO Live hour being mentioned amongst like CRO Podcast, of which that's a good thing. Like it's appearing like on different lists. Yeah. So which means like people are seeing it.

Yeah, it's funny because. Okay, so you can always just do something. This is horrible. I'm going to contradict myself and just keep on doing. It's almost like pushing like another marketing flywheel. It's very painful and very slow and it's going to grow organically. Sometimes you can short circuit some of that growth, can make it a little bit faster by adding some more promotions, some more marketing. It helps you get to the other side maybe a little bit faster. So that's the question that I ask myself.

Yeah, that's true.

That's true.

I see like where you're coming from, the Black Friday sub Monday, all the bars, all the business. I wanted to say that it's behind us, but I don't think like it's really behind us.

Yeah, I don't know if it's behind us. So here's how I judge. By the way, I just grew fake pie. The traffic that figpai gets, of course that traffic just shoots up. So if we are processing, I know what do we do? Anywhere between 3 to 5 million visitors per day across all the different domains on FIGPI. That jumps up to 25 to 30 million per day. Black Friday, Cyber Monday. So it just increases tremendously. Now our busiest time, by the way, is Black Friday. That's extremely, extremely busy. Slows down over the weekend. Sunday slows down. Monday shoots up back again. Similar levels for most websites, similar levels to Black Friday and then slows down. But what I've noticed this year, and you probably have seen this, many companies have started the Black Friday three, four, five days before Black Friday, which had, by the way, conversion optimization implications. When you start something so early on and then they don't finish usually like no. Cyber Monday finishes Tuesday nights weeks and until Wednesday. And now I'm looking at some of the customers on FIGPI and they've extended until Friday. They are trying to get as many sales as they can compared to last year. So it's just an interesting trend. There's some sites that are up there, sometimes sites that are down, and sometimes some sites within the typical range.

You have been doing this for quite a long time. Are there any changes that you saw during this Black Friday compared to the previous one?

No. I think in all honesty the biggest change that I see for this particular Black Friday is that the number of retailers that are jumping on it. And people might say, huh, this is a little bit strange. There was a segment of retailers that for the most part just refused to join. By the way, Black Friday, Cyber Monday. They're like, we don't do this, we don't need to do it, we're fine. We don't want to dilute our brand. This year they jumped on the bandwagon and I think reflecting it was interesting because I think that's also an indicator of how the economy is going because you probably are getting squeezed, your sales are not as high as you would like them to be. So I think many companies are jumping on the Black Friday bandwagon in otherwise for the tactics and whatnot. Very similar. I think at this point companies have gotten used to how to run Black Friday after doing it for so many years. You focus on the basics, you have really good offers. But there are realities. I was talking to a CEO of a company and his Black Friday, his November sales 2023 compared to November sales of 2022. So we're comparing kind of month, year over year, which is by the way a horrible comparison from a conversion optimization perspective. But that's what every CEO does. That's what every VP of E Commerce or VP of Marketing or CRO, a chief revenue officer will do. The other CROs and his sales were down by about 15%. So he's really stressed out about the fact that the sales are down. But the reality of it, by the way, sales this year for that particular brand for what they sell are down about 30% month over month, year over year, month to month. So October to October, 22, 23. So yeah, November increased, but it did not hit the numbers that they were hoping to see. And some of it's by the way, there's a whole other forms that will just open up. Some of it is the fact that last year they offered 50% off discount. Those were most of their campaigns this year. They offered 40% off and 30% off. So there were less discounts now, typically, by the way, Interesting. So if you're offering less discounts, that means, okay, yeah, maybe you've sold less, but also you're making more money. Correct. So it balances itself out. The only problem is that the fundamental assumption that you will make more money by offering less discounts if your cost of goods sold remained the same, you're buying something for $10 in 2022, you're buying it for $10 in 2023. Last year you sold it for $40. This year you sold it for $45. Okay, so you sold lusiness, but you made more money. The problem is, no, I bought it for $10 in 2022. I'm buying it for 15 and $17 in 2023. That's the reason in the first place, I can only give a 40% discount or a 30% discount. So some retailers are getting squeezed from both sides. Correct. Consumers are not buying and then their manufacturers and suppliers are increasing the cost. So it's a rough spot for some of the retailers. Some of the retailers today.

It's quite like an interesting topic to talk about post Black Friday experimentation analysis. Because like, that's what you are doing right now. Like you're analyzing what didn't go. So just like maybe to kick it off. What are some of the lessons that you learned on this Black Friday? If there are any new lessons that you think, oh, this didn't really happen last year.

I think the biggest lesson, it's going to sound very silly when I mention it to you. What's the biggest thing that people do retailers use during Black Friday? Is those urgency experiments. Correct? Oh, the countdowns. How many days remain until the offer runs out? And it's funny because some companies started their sales 10 days before Black Friday. Guess what? If you have an urgency indicator, this actually is counter. I come into your site on November 15th and you're telling me, oh, this offer that I started right now is going to continue for the next 11 days. Guess what? I am not buying from you. No, I am in no rush to buy. And it's funny because we've had that discussion with some of the companies that we work with and we told them. Really? You're telling people that they still have 10 days to place an order? Isn't that counterintuitive? And they're like, no, we need to push people to do that. And I don't know how Many companies that are E commerce sites that try to do that, that experiment, trying to use a countdown like an encounter, those lost significantly. So I think almost every retailer that I saw that tried that had to stop those experiments. And then they said, okay, we're done. I'm like, no, no, no, we need to rerun those experiments if you want to experiment with them. But guess what? Run them a day or two at the most before your deadline, because then you're really adding urgency. But if you're telling people you sell days now, people are going to walk away. So that was a big one. I think few retailers learned that hard lesson, silly lesson, because the problem with this type of experiment, when you mess it up. Introducing this experiment with the goal of increasing your conversion rates, increasing your sales, and by telling people you still have 10 more days to place an order, guess what happened, what you did to your sales, you slowed them down. So it's not like, oh, we just ran an experiment and we learned something from it. This could hurt you. That's just, you need to think about experimentation. The other big thing that we're doing now, and I spent actually the morning working with some of our team members, is looking at, at the revenue impact of the different experiments. And the beauty of this is, and this is very specific to Black Friday, is we have experiments that ran for two to three days. Correct. Very specific experiments. I started on November 21st. I concluded November 25th or whatever time period that we ran the experiments. I can look at how the control performed. I can look at variation one and how it performed. Variation two. I can. When I say how it performed, I'm looking at the conversion rate and I'm looking at the average order value. So I know, for example, the particular company that was looking at their data just prior to jumping on C arro live hour. The control generated $220,000 of sales for them. That was in one day. The variation did about $320,000, so about 100k more. The winning variation, because they had multiple variations. So you can right away, if you're working, if you're a CRO agency or you're going back to your client, or if you are a CRO specialist working within a company or CRO strategist, you can go back to your boss or to your stakeholders and tell them, guess what, my variation just did a hundred thousand dollars more in sales for you. Which is beautiful. Correct. You don't extrapolate. You're just looking at exact data. Here's how people went to the control. Here's how many people went to the variation. So you can come back if you're running multiple experiments and say, you know what? I have helped you generate X number of dollars all across. So that particular company that we're looking at just in the span of two days did almost an additional half a million dollars in sales. Five experiments and about half a million $550,000 in sales, which is absolutely amazing. Correct. In two days, this is additional on top of their sales. So that's a portion that we do. Now here's the thing. For every winning experiments, we can calculate the impact as like, oh, here's what this did. We can extrapolate from that to say, you know what? Okay, so let's take a very simple case. I had a control and I had variation one, variation one. And let's take an experiment where you had 10000 visitors. So 50000 visitors wanted to control 50, 000 visitors went to the variation. The variation increased conversion rates and it increased it only for those 50, 000. Correct. I saw the variation. The 50, 000 visitors that went to the controls. You didn't see the benefit there. That's how you did your measurements. So what we do is we say, hey, if, okay, so we've learned a lesson over here. If we had run all hundred thousand visitors through the variation, what was the revenue impact? It's even better. Correct. Because it's actually higher. So if this variation had made us $100,000, if we had ran everybody through that variation, we probably would have made $250,000 of additional sales. Okay, so yeah, would have, could have, should have, but didn't because we still R50% through the control. Yeah, there's a lesson here. Guess what? What's the difference between this variation and the control? What did we learn there? Next time, this variation is going to become our baseline for next year. The thought behind this variation is going to be our control. We're going to try and beat it. So because the potential is really large. Yes, we made a hundred thousand. We had the potential to make $250,000 of additional sales. And what do we learn from that and how do we deploy that? So those are the lessons and the discussions that we're having right now.

Yeah, those are good ones. I noticed. Are there like any other metrics or any other data points that you pick out that you should prioritize collecting after Black Friday besides maybe the revenue ones? Because maybe I know Black Friday is all about revenue, but are they like.

Any other Simba, let's move to this way. I was talking about our CRO team because people on CR team from different backgrounds and they have different points. I appreciate that tremendously. There are people who love the analysis. There are people who love the analytics. There are people just focused on the numbers and they were trying to look for deep insights. And I told them, all power to you. I always love the fact that you're trying to draw some deep insights. But in our case, we're working with clients on invesp. I tell them the CEO, the VP of marketing, VP of E Commerce, or the owner that we're dealing with does not care at this point in time about all of your insights. The only thing that matters to them are the revenue number. What matters is how much revenue. It doesn't, by the way. Usually we say, we come and we say, hey, we increase your conversion rates by 10%. We don't do that. During Black Friday Cyber Monday, our conversation is different. Here's how much more money we made for you during Black Friday Cyber Monday. Some of our purest CROs are so uncomfortable with that. No, that's how we pay for your salary. Keep the clients happy. You've made them a ton of money, and guess what? Everybody is happy. So that's the reason you want to look at those numbers. This is particular for us, by the way, for the holiday season. It starts from Black Friday, and I would say about the 20th, 21st of December, when the cutoff for shipping dates, and then we go back to learning. So right now we're just very focused on increasing sales. We're asking and we're evaluating. What are some of the campaigns we've ran last year? How can we beat those campaigns? What are some of the campaigns and the creatives that we are doing this year? How can we do those better? That's the focus right now. And it's pure and simple. More money so the CRO program can pay for itself and learn the rest of the year.

That's good.

That's good.

What is the difference between the holiday CRO program? I don't want to say program, but the experimentation that you do during the holidays and the one that you do after the holidays? Because, like, in my mind, I'm trying to think, like, right now we are focusing, like, on the holiday experiments. Then at what point does that switch enough to say, okay, cool, now we're going back to like, normal. What changes exactly? Is it like the data that you look at or something different?

I think your full analysis of what happens holiday versus non holiday. Let's start by visitors. The mindset of visitors. Completely different. Correct. I'm coming to your shop to buy during the holiday. I'm expecting to buy things at the discount rates. I've been planning to buy a frying pan. You're going to laugh about this. Just good old pan to fry some eggs forever. And sometimes I'm like such an impulsive buyer that I'll buy really quickly. But I've been telling myself, you know what? I probably should spend more than my typical 20 on buying like a pan. I do a whole bunch of research, and the one that I like eventually is $130. I'm like, no way I'm gonna spend $130. I'm sorry. Not to offend anybody, but my whole probably cookware set. But now I'm trying to upgrade. I saw the, like another pen that I. And I waited. I actually waited. Didn't buy it on Black Friday, waited until Cyber Monday to buy it. And Instead of spending $120, I probably spent 70 bucks for me. So my mindset was very focused on as a visitor. I want to get a deal. I want to move on now. It's funny because for that side that I bought from, if they're smart, guess what? I bought one from you. I should probably be buying more. So what's my lifetime value for that store? That's really where the analysis happens. Great. We sold it something for 70 bucks. We probably barely made any money because it was free shipping, too. I don't know how much. Most places are making 40% off any item that they sell. So it's 120. They barely made a couple bucks. And add some shipping on top of that, they might have made money or loss. But guess what? If I'm the marketing person in that company, if I'm the head of marketing, if I'm director of marketing, I am doing full analysis and asking for those new people who just bought from us, how can we get 30 to 50 to 60% of them to place an order with us at regular price in the next 30 to 60 days, what's going to be there? So again, going back visitor is going to be different for us. I've mentioned this before. Whenever we're running experiments outside of the holidays, we're trying to learn. And with every experiment, there is some data that supports it, There's a hypothesis, and then there's a whole bunch of questions that we're trying to answer as a result of the experiments. Maybe five or six different questions. Great. We Finish the experiment. We say, okay, what happened here? What did we learn? And we add more questions during the holiday. If we're being very frank and honest, first, we're not trying to learn. Second, our experiments are very simple, straightforward. You know, we're not doing something really complex. They're just very focused on the offers and they're focused on incentivizing those visitors who are already highly motivated. We're trying to incentivize them to actually act and act right away. The minute we're done with the holidays and we've made the money that we want to make, then we make the switch. We go back, okay, let's look at the website where our visitors leaking. Let's prioritize and let's figure out how we can learn about our visitors, giving them better experience, making more money as a result of that, and improving conversion rates. So it's a shift in the mindset for us and it's a shift in the implementation of a mix between the two.

How many experiments do you usually run? Maybe for one E commerce company, let's say maybe from Black Friday up to the 28th of December.

Oh, man. At a minimum, we're probably doing 10 experiments. At a minimum. I was talking with one of our developers and for one website, since the last seven days, he had launched eight experiments in basically less than a week. By the way, granted, that site gets a ton of traffic, gets a ton of conversions, so their data supports us running that many experiments and segmenting the data and not having any issues around data pollution between different experiments, but eight experiments in these five days? Actually, again, the site traffic supports that and we know that we're going to continue running. So if you take that, okay, we're doing eight experiments in five days. You're talking about the next 30 days. We're probably going to be launching anywhere between, I would say, 25 to 35 experiments for them. Smaller sites we probably are doing, I would say since Black Friday, until we close out the holiday season, we're probably doing anywhere between eight to 12 experiments that's going to average that we're going to be running. Now I remember something. Those are all unique experiments. I might run an experiment trying, let's say, a new design for the homepage, trying to motivate visitors to place an order. The control has V1, V2. Okay. The V1 does really well. I might do an iteration on that experiment. Correct. And try and just tweak it a little bit more. That doesn't count within the eight to 12 experiments that we're aiming for. That's a small tweak. We already know that V1 is doing really well, but we think we can beat it a little bit more. So those are really small iterations where the conversion optimization specialist is saying, okay, I've already got a 6 or 8% lift for my winning variation. Can I up it by another 2 or 3%? That's. That's what we're trying to do. Original experiments. Yeah, 8 to 12. If you think about kind of iterations on those experiments, you might double the number easily.

Yeah, yeah, yeah, I can see that. How many people are usually, like, involved in the analysis of those experiments? Because, like, right now, like, I'm thinking eight experiments in the past five days. That's for one client. Maybe that CRO specialist is about four, five clients. So usually how. How many people, like, are involved in the analysis of that and if you can mention their roles so that, like.

Listeners can understand each one of our CRO pods is managing anywhere between, I would say four clients to five clients. We don't even. Four clients. Probably. We max out there. We have a couple of pods that are managing more. And during the holiday, it's a team exercise. It's not that the CRO strategist is the one who is doing. Or she's the one who's doing the work by themselves. No, they're working very closely with their UX designer. So it's a pod for us is the CRO strategist, the UX designer, the developer, the QA specialist. There is a project manager that's really. Project management is extremely important. And I don't like to use the word extremely. I'm like, oh, what's the difference between important and extremely important? Yeah, it is extremely important, this one. Because when you run that many experiments, there's no space, no room for making mistakes. So project management becomes really important and everybody's chipping in, everybody's discussing, everybody's looking at the results. This. You've been with us for a long time, Simba. We don't take that weekend off, like Friday. We're working because we have to look at results. We have to make adjustments. That Saturday, Sunday is going to continue. This weekend I think the team is going to be off. Correct. Because last weekend they were working. Things are going to slow down Friday, so they'll catch their breath. I would say this weekend, next week. But then we're back at it again a week after that.

That's so intense. But in a good way. Yeah, but in a good way. In a good way.

It's highly rewarding by the way, when you do your analysis. And oh, I just made one pot for one client, did about $550,000 of additional revenue. And that's their smallest clients. They've done a lot more. So I know that's one pot. Probably did close to $6 million of additional sales in the last week or so. They're very happy and their clients are really happy. It's funny because we had one company, I'm actually, as we speak, I just have to send you the feedback that they sent us. Let's see if I can pull it up. So yeah, let's pull it up. So basically we're trying to do check ins with clients just to see, hey, how did it go? So here's the response that I got. We can skip the meeting, but I will just email you my feedback. Look, okay, first off, the CRO, the particular CRO is absolutely amazing. I sure love having him as the main point of point of contact and having him on weekly calls. He is a superstar. He is really great. I feel really blessed to be able to talk to him each week. And of course, Ayat is super smart and a guru of convergence rate optimization. As much as I like the guru. So an A plus 10 out of 10. I'm really impressed with Invasp. They are the best. Listen, Simba, if you make somebody two, $3 million in a week, you all get like that. But also, by the way, the reality, we've had couple clients, their sales are not that great. It's a mixed bag. I would say we had 25% of our clients that did absolutely amazing. 50% of the clients, 60% of the clients that did just normal what we expect. And then there's about 10, 15% that didn't do well. It's a mixed bag. When you do really well, you just feel so good. Oh, we've done this. This is absolutely amazing.

Yep. I just want to put you like on the spot because I want to come back to this question again, like maybe in the next few weeks, in 2021, during Black Friday for combined clients, it was close to about 37 million or above last year. I know it was above. How much do you think for the holiday seasons investors made for its client.

Last year was absolutely incredible. Close to $65 million. Correct. So 37, 65. I'll be very happy if we hit the same numbers as last year. The reality of it, so much more tough markets. I think it's more realistic. And then by the way, it's the same number of clients, which is funny. We keep the same number of clients. We might have one client additional, one client less. But I think I'm a happy man. If we do $55 million of additional revenue this year for clients, if we do what we did last year, it's just difficult. The market is a lot tougher. People are not spending money and they're really careful and when they're searching for really good deals. So we'll see what the data takes us to and shows us.

Yeah, that would be interesting. I'll ask again, like this question maybe sometime next year. So I'm sure by then you've done. They've done like the full analysis for all the clients. So I think those. Yeah, you're saying something.

It's going to be interesting because we'll see how the data, like, pans out about mid January. That's when we finish all the analysis, everything, looking at everything and see how things played out. So it should be an interesting year.

Those are the questions that I had for you. Unless you have some things to share. I always have time for those.

Yeah, I know. You always ask me the question. No, I'm actually super excited about just the holiday season. A little bullish, really. On 2024. I know I just mentioned this is a rough economy out there, but I think the trends are all in the right direction. Yeah, there are some companies that are struggling. That's just the reality. It's not easy. But at the same time, I think there's enough indicators that are telling us, okay, things are heading in the right direction. We'll see. It will be an interesting conversation by June of next year on seeing how the economy goes. So until then.

Yeah. Happy testing.

Hey, happy testing. Thank you, Simba.

Thank you. Bye.