#21 – Failed A/B Tests: What Went Wrong and How to Fix It

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today. Wherever you listen to your podcasts.

Sometimes a variation loses a challenge or loses against a control. And it surprises us. It surprises us because the change that we made in that variation was so small and minute. They were like, we didn't really expect such a loss, especially if the loss shows. Well, it lost by 15%. It's like, really? That doesn't make sense. The question is, what's happening here? Because that's the first assessment that I want to do. And it all depends on the hypothesis that we had and the implementation of that hypothesis in the test. If we think that the implementation, if we are watching session recordings and we see positive signals, if we're looking at not the primary conversion goal, but we're looking at secondary conversion goals, and we're saying, hey, this variation, although overall it lost, there's other indicators that still makes us not so confident in the data. And we want to bring it back again. Now, when you bring it back again, do you bring it as is, or do you do a couple of tweaks to it? That's a decision that the conversion optimization team has to decide. It's like, maybe we need to tweak it a little bit based on some of the session recording that we saw that, oh, you know, we've had a small, minor change or error in the way we implemented this. I always say good experiments with interesting question end up with even more interesting and smarter questions. Correct. Because that analysis is just very powerful.

And it's very lively. And every once in a while, by.

The way, you run an experiment and you sit back and it's like, okay, I don't know what's happening here.

I know my hypothesis.

But visitors surprise us in certain ways, so that process analysis is a critical component of any experimentation program.