Bonus Episode 2 – What Causes A/B Tests To Fail?

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live, our podcast today, wherever you listen to your podcasts.

Hello. Hello, everyone. Welcome to yet another welcome to a short episode of the CRO Live Hour podcast. I'm your host, Simba Dube. In today's episode, we'll be focusing on a B testing. What really causes a B test to fail. We know that the success rate of a B test, especially in the E commerce space, is around 12 to 15%, which means the majority of a B tests that are launched by E commerce companies, they fail. What causes them to fail? So I had a chat with Khalid Saleh about this, and then he mentioned a couple of interesting things. This is what he had to say.

Fundamentally, your goal is not to run AB testing for the sake of AB testing. If you're just, hey, let's test this, let's test that. Without actually doing the proper research of why you are testing something. We have enough data, we have enough insights, so we are testing this particular element. Many of your experiments are not successful because you're just randomly testing things like, oh, I've heard this, let's test it. Oh, let's test that. Oh, we should change this, we should change that. Not a good way to actually spend your time and resources. Number two, you need to understand that a B testing is not for every company. First, you need to have the right numbers to be able to a B test. So there's a certain number of conversions that you need to be hitting every month in order for you to do any type of a B testing. And perhaps more challenging is you need to have the right mindset to be able to run an AB testing or an experimentation program. The goal whenever you are doing AB testing is to get an indication from visitors from people who are coming to your website that, oh, we like this design better than the control, than the original design that you have. The minute you get that inkling, that indication, then you can go back and improve it more and more. If with every a B test, you're going to spend hours upon hours perfecting every variation, thinking that, oh, this is going to be the most perfect design, and I have five of them that are competing with each other, well, guess what? There is a really good chance that they all end up losing. So all this time that you've spent perfecting the designs and make sure that they look good was a waste of time and people told you we actually don't like it. So that's another reason. A third reason that I see where people struggle and why most AB testing fails is that many people don't understand statistics enough to be able to understand AB testing results. People either don't want to put the effort because statistics scare them, or on the other extreme, they make the AB testing statistics so complicated that you think that you need a PhD in statistics. I would say I can probably explain the statistics that you need for AB testing, and I would say three to four hours, that's the most that you need as a marketer. But you have to understand those core concepts. So when you are reading the data, the results of your A B test, you're actually doing sound analysis. You don't need to learn how to program in R to understand how to analyze AB testing. I need to be able to understand enough about statistics that I'm like, okay, I can have an intelligent conversation. Sometimes the algorithm and the tool that you select to run the A B test can be a cause for why your A B test fails. Without getting too technical, there's what we call frequentialist versus Bayesian statistics. Correct. Two different methods of statistics. Neither of them is good or bad. They're just different ways of looking at data. Bayesian statistics, for example, doesn't have false positives. This is where you run an experiment. You think, oh, we have a winner. It's absolutely amazing. It's going to increase our conversion rates. But it was a false positive. It was not actually a winner. That's something that you need to be aware of. There is something within frequent that we say, one tail, two tail. And again, different testing software uses different methods. I have a design for a particular page. I come up with a new design for. For another page on my website. And when you run an a B test, your goal usually is to say, hey, did this new design that I've created help increase my conversion rates? So that's what we call one tail is it's one direction. A two tail thinks, did it help increase my conversion rate or did it actually reduce my conversion rate? So people say, why does it matter? Well, two tail is a lot more strict. So if you're using an AB testing software that uses two tail statistics, there's. It's a lot more restrictive. So you might have AB tests that look like they've lost and they might not have lost. One tail versus two tail matters a lot, by the way. For example, in medicine, I have a new medicine I'm going to test, and I want to see whether this new medicine helped actually cure a disease. So that's one tail. But I am also highly interested to see if this new medicine actually caused negative effects. Correct. So I definitely want, for any type of medicine, I want to do two tail analysis. We've talked about using visual editors and the flicker effects. People get frustrated with AB testing. Original page loads, flickers. The new variation loads again. Visual address on a B. Testing platforms are good in theory, bad in practice, and then finally, I think at a very higher level. If you think about experimentation, AB testing and CRO programs, they have two goals. One goal as a result of running this program, to understand my customers better. Another reason I want to generate more revenue, more roi. If you're not achieving both goals, you have a problem. So I always say generate results. Make sure that people who brought you on board, whether if you're an agency or if you're an internal resource, that they see the value of this program. It's not enough for you to say, oh, this program was really valuable. If your stakeholders don't believe it was valuable, they don't see the value of it. And again, the value could be measured in different ways, but if they don't see that value in a clear, measurable way, then your program might fail.