#14 – Website A/B Testing Process

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation, and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts. Gosh, do we have two hours? No. Let's simplify this. Assuming that we've already done the research, we've identified a usability issue, a conversion problem, or revenue opportunity on the website, we have a solid hypothesis. We've created designs for that hypothesis. First question, how many challengers do I need to include in this AB experiment? Do I create 10 designs? Do I create two designs? Do I create 100 different designs? Wouldn't recommend 100 different designs because your designer will hate you. Usually, the number of designs that we include will depend on the volume of traffic, the number of visitors coming to the website. But as a good rule of thumb, we don't exceed four different challenges to control. Even if I'm working with an ebay or a target, even if I'm working with a site that gets millions of visitors every day, still, I want to be very focused. Each one of those designs, of course, is an implementation of the hypothesis that I have. So we're very strict that the designer is taking the hypothesis he understand or she understands understands the problem that we're trying to address. They understand what we're trying to do over here, and their designs are true to the spirit of the hypothesis. Then you need to do your calculations. Okay, so I've created three different challenges to a control. I need to do my calculation, how long I'm going to be running the test for. There's calculators out there that will help you determine the sample size based on the type of statistical engine that you're going to be using, that calculator is going to tell you that you're going to run the test for a week, for two weeks, for five weeks, for six weeks. Now, it's very critical that you don't run a test for less than a week and you don't run test for more than four weeks. So you might have to tweak your parameters there. But do that before you launch the AB experiment, because one of the worst things that could happen is you launch the experiment three days into it. Oh, we've reached confidence. Let's stop the experiment. No, you don't do that. You actually determine the sample size and you're going to run the experiment for that period. Now, let's say your calculator tells you that, well, you're going to run the experiment for three days or for a week. You think just based on the type of orders that you have, that you actually be running the experiment for longer. So sometimes I say, yes, we'll run it for longer. The calculator says a week or three days or five days. We run experiments for 10 days. But we have that as a set rule. We have also what we call early stop rules. We are launching three Challengers. What if one of them is absolutely doing horrible, is actually reducing our conversion rate by 30, 40%? What do we do in that case? So we have what we call early stop rules, because there is no need to like, oh, no, I'm going to run the experiment for the next 10 days because that's what I decided. And this variation that's getting 25% of our traffic and reducing our conversion rates by 50% is going to continue running. That's being stupid. You then launch the experiments. We have some rules about launching the experiment. We never launch experiments on Fridays. Horrible idea to launch things on Fridays because they tend to break over the weekend. And then your team is going to spend the weekend validating that. Oh, what happened over here? We launch experiments usually during the early hours of the morning, and then we monitor the experiment very closely. Go back to my variation that's doing 50% worse than the control. The first question that we have is, do we have a bug in this implementation? We've qaed our experiment prior to launch. We do a ton of quality control. But if we see a variation that is consistently doing really absolutely bad, we want to ask, hey, what's going on here? So that's when our quality control engineer jumps in, does additional qa. We let the experiments run until we hit the timeline and at that point in time, when we reach the period that we've decided, that's when the team has to make a call. Did we achieve confidence? How did the different variations perform? We have an experiment that has four challengers or three challenges to control. One of them is a clear winner. Maybe the other two are not doing so well. We actually take that winner and we run it against the control in a head to head match. Maybe not for full two weeks, because now it's just kind of validating those results. One of our goals is to make sure that whenever we have a winner, we actually have a winner. We don't have a false positive. Sometimes we have a clear winner and we have a variation that lost. But we think that it should have performed well, so we might include it in the next iteration of the experiment. At some point, the team will have to make a decision based on the results. Do we launch a second iteration of this experiment with fewer variations? Do we iterate on the results? Do we pivot away from the results? That post test analysis is very critical in your CRO program.