#32 – How To Analyze A/B Test Results

There are days when you run an AB experiment and the results are crystal clear. You know exactly what to do. What we are going to be talking about today is not one of those days. We are talking about days where things are confusing, things are not clear. And if you do CRO and if you do AB testing, you run into that. So we'll be talking about that. My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of your conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today wherever you listen to your podcasts. Simba, how was your week?

It's been a good one. Also a busy week because creating different landing pages for figpi and at the same time working on the marketing side for invesp. So actually like a good thing for me because I get to learn about a whole bunch of things, you know, like I learned about all sorts of marketing things from both sides, from the agency side and also from the SaaS side of things. So it's a good learning curve.

I love that. Yeah, it's fascinating what's happening with investment. I think we can share no big deal. But with the Google updates. Correct. That happened recently. We saw a drop in some of our SEO ranking. So the team has been working day in, day out trying to kind of remedy that. You've been kind of in the weeds trying to figure out what's going on there. Thoughts, ideas, anything you can share.

We have some of our best performing articles in terms of traffic. They dropped, they lost like a lot of traffic soon after the update. That happened in March and we've been trying to update some of the content even though, like some of the content had been updated recently. So we had to try to re update it and just add some new content just to make it look so interesting and to refresh it, to give it like maybe kind of like a new list of life or something like that. But it seems like we still have to do more than that, more than just updating the articles because there's also like the technical side of things where you have to kind of like investigate and see how good or how healthy is our SEO on the technical side of things, rather than just the updating of content. So we've been working on that with the team updating a bunch of articles and I've been also following some of the top SEO expects about what they think about this update. And it seems like most of the people, they really haven't like really, really figured out and a lot of websites, they also were hit by the update. So it's an interesting development. But hopefully in the next few days, not weeks, we might recover the traffic that we lost.

I've seen Invesp hit 200,000 visitors a month and I've seen Invesp.com hit 800 visitors a month. You know, so I have seen the wild ride of SEO. But that's the reason, I think the investments that we've been making lately in our newsletter and kind of driving more and more subscribers. We're getting almost 200 subscribers, 200, 300 subscribers per week to the newsletter, which is fascinating to me. It's always good to see that because our subscribers are our subscribers wherever we go. Like now they're. Yeah, we'll use different email platforms, but really they are our subscribers. So that's what I love to see. But. Okay, Simba, what are we talking about today?

Today we're talking about analyzing a B test results.

One of my favorite topics.

Yeah, yeah. We have about four questions to answer. I'm sure. I know that this is going to be a good session.

Well, let me tell you, I was looking at the questions earlier. The best intro that I'm like, there are days when you run an A B experiment and the results are crystal clear. You know exactly what to do. What we are going to be talking about today is not one of those days. We are talking about days where things are confusing, things are not clear, and if you do CRO and you, if you do a B testing, you run into that. So we'll be talking about that. So shoot away, my friend. Let's see.

So when you run like an a B test, right. The way I think about it, there are three outcomes. The Test will either win, lose, or maybe the results will be inconclusive. Like those three outcomes. I just had to use the word win or lose. But I've seen like a lot of CROs going against those words. So the test will either win, lose, or the results will be inconclusive. So let's just start with the inconclusive result first. Can you explain to us what it means if people say, like, the results were inconclusive and what's the way forward after that?

When we start with an A B test? Let's take the very first. When you have an A B experiment, you've done some analysis on a landing page, you've done some analysis on your website, and you say, I have a problem here. The current design, the current status quo has a problem in it. I think I can come up with a better design, provide better usability, better user experience, and hopefully better conversions. You run the AB experiment, you're splitting the traffic. Let's say you have 10,000 visitors, 5,000 see the original design of the particular page, 5,000 see the new design, and then we're comparing the results. It's a pure statistical comparison. Those 5000 people that came in placed 50 orders. Those 5000 people that came in and saw the new design, they saw 75 orders. Are we seeing more orders in the original or are we seeing more orders with the new design? Now, the fact that the original Design gave me 50 orders, the new design gave me 55 orders, it doesn't mean that I necessarily have a winner. Correct. Because maybe if I run the test longer, those results will shift. It depends, really. It's like. And it's a very random process. And you want to make sure at the end of the experiments, when you look at the results, you want to say, well, for sure I have a winner here. Whether it's the original or the new design or no, the results that I got are just based on chance. If we run the test again, I might get different results. So we measure that using different methods. But confidence is a good way to measure it, to say, you know, how confident are you in the results that you saw as part of your AB experiment? Now, typically we run an experiment, and this has been an industry standard, is until you hit a 95% confidence. That means that I'm not going to get into explaining what that means because people will get really upset with me every time I try and explain it and I try and oversimplify it. They say it's statistically is not correct what you're saying, but 95% is kind of the standard. It's either chance to win or a confidence level or 5% significance. What does it mean when you have non conclusive results? You ran the test, you split the traffic. Either the original won or the new design that you've deployed have won. However, when you look at the probability to win, the confidence level, the significance, you might see something around 55%, 60%. So you say, you know what, yeah, there is a lift over here. My new design is actually showing me an increase in conversions, but really it's most likely that this is due to random chance. If I run the test again, the original is performing at the same level as the new design that I've done. Now remember, I've gotten to this point after I've done initial analysis, I've done some coding, I'm running the experiment, I'm doing the analysis and at the end it is non inclusive. You cannot say for sure that you have a winner in this case. And that's when people get frustrated because they're like, oh, why did this happen? Now let's break this down a little bit further. First off, one of the very first things that they ask people like, you know, what's probability level up to what confidence level are you running your experiment? Are you running it up to 95%? If you run a test and you see that a variation has a 85% chance to win, is it as good as a 95%? No, it's not as good. However, it's also not as bad as a 50% chance to win. Remember, the 95 threshold is a very artificial threshold that we set altogether. You know, CROs, actually not even CROs have set this threshold. It's been set by statisticians for a long time, but it's a very artificial threshold. The best example that I always tell people about where statistics really matter and they are a matter of life or death, is if you look at the Food and Drug Administration because you don't get the drug in the market put out unless there's an AB experiment that runs correct. There's the population. We are going to run this and then we're going to see how people react to this medicine. And is it really, are we seeing positive results as a result of this drug? The confidence level that the FDA requires for most drugs is at 95%. There are some drugs that require a 97.5 confidence level. Harvard had a very interesting paper where they said, you know what, this random threshold that we've set really should be re examined Based on the type of medicine that we are looking at, what do they mean by that? If you have a medicine for, let's say cancer, somebody who's terminal, somebody who's going to basically die in the next six months, they said, really aiming for 95 or 97% doesn't make sense because the other alternative to the person without the drug is that they'll die anyway. So we might as well lower that threshold to an 80% level. However, let's say for an over the counter medicine, you know, cold, common cold, headache, they said, you know what, maybe we need to be a lot more strict than 95%. Maybe we should be aiming for 97% or 99% as well. I go back to typical A B testing. So I say, okay, what is the threshold that you've set? I have seen some companies where they've run an experiment, they hit the sample size and they say, you know, what variation B or the challenger to the original did not win. Yeah, it had like, you know, a 10% lift in conversions. But when we look at the chance to win, it's at like, you know, 90% and that means we have inconclusive results. And I'm like, hold on, what do you mean you have inclusive results? Yeah, it's not as good as 95% or 99%, but. But I will take a 90% chance to win. And if you think that your 95% is the cutoff, well, I say like, you know, maybe I will be even more strict and I'll put the 99% chance. So I'll reject all 95%. So it becomes really, really silly. So that's the reason I want to look very carefully on what it means that I have inconclusive results. Number two, I want to make sure, of course, that my test have hits, you know, the sample size that is required. Correct. That we've ran it long enough. Because sometimes people run the test for way too long or way too little. And what I worry about them for the most part is people running the test for very little time. We work with companies where you take the variations, look at the traffic, you look at the conversions and you calculate the required sample size, how many people really should see the experiments. And literally the sample size calculator will tell you, run this experiment for two hours. I don't want to run an experiment for two hours. I'm never going to run an experiment for two hours. Yeah, according to, purely based on statistics, based on numbers, I can run the experiment for two hours. But that doesn't make sense. So usually we want to run it for at least a week to two weeks. Now, let's say you've reached a test and it's non conclusive. What do you do? To me, I go back and I evaluate. Was it the hypothesis that was incorrect or was it my implementation of that hypothesis? 9 out of 10 times the hypothesis is solid and sound. It's just the way you've implemented it. Correct. The way you've expressed that hypothesis on the page, that is what was inconclusive. That means you go back to the drawing board and you say, okay, well, what can we do to improve our implementation of that hypothesis? I remember one time we worked with an insurance company, a large insurance company, and we're looking at their quotation funnel, where like, you know, you put in your car information. You put in your information, you get the pricing. The pricing was complicated. How you reviewed the pricing and the final price was just really complicated. We ran an experiment and the team saw that. We did usability testing, we looked at session recording, we look at funnel analysis. There is a problem. So the team decided, let's oversimplify this. And they oversimplified. Like, no, they simplified it. They ran the test, it lost. I was like, okay, well yeah, no big deal. We went second time at it and again it lost. And I'm like, guys, come on. Okay, maybe people like the old complicated. And by the way, you can always rationalize. Because then I went to this rationalization mode where like, yeah, maybe because it's an insurance product and people expect it to be complicated. And maybe you oversimplified it and maybe there were things that were hidden. Now that we're making it very clear to people, maybe that's what caused a drop in conversions. I rationalized why we lost. The team just completely ignored me and they said, you know what it's called? Leave us alone. They went a third round and again they lost. And at this point I'm annoyed, the client is annoyed, everybody's annoyed. And now I'm bugging the team. I'm like, can we just move on from this side? Obviously we could not figure out and your hypothesis is incorrect. The fourth iteration again had a 32% lift in conversions. So the team absolutely believed in the hypothesis and the research. And they said, you know what, it's just our implementation of this research. We've missed it the first three times. We're going to go back at it now. You need to have a team that is very confident. Correct. To be able to stand to the Client, but even to be able to stand to me and to say, hey, leave us alone, we know the work that we do. And I love that. I'm like, oh, wow, this is right. They came back for the fourth time, correct. So they had inclusive results, inconclusive results, lost even to the original. But they came back because at the end of every experiment you need to decide, do you iterate, do you pivot? Where you say, you know what? Okay, yeah, I was wrong. And this I think depends on the quality of the research that had gone and how certain you are about the hypothesis that really drove this experiment altogether.

So you said that if the results are inconclusive, then you have to revisit the hypothesis and check it was correct. Or maybe you implemented it wrong.

How many times do you have to.

Reimplement the hypothesis to kind of like think, okay, it seems like this hypothesis wasn't working?

That's a really good question. I wish there is an easy way to say, you know, we've implemented this hypothesis enough times to say like, let's pivot, let's iterate, let's do this, let's do that. I think it depends really on, based on experience. What I've learned over the years is no matter how skilled a CRO is, a conversion rate optimization specialist is you bring somebody else with a pair of fresh eyes and they look at it and they say, how do you do this? How do you do that? So that's the reason sometimes when we get stuck, lots of times, by the way, the first intuition that we have when an experiment either fails or it has non conclusive results is to say, you know, it is the implementation of the hypothesis. So we'll go back and say, okay, here's how we draw. So usually the CRO is working with the UX designer to come up with different ways to implement. Now, on the third or fourth try, that's when you usually say, you know what, I need somebody else to take a look at this. Now I also want you to remember something. Most of the AB experiments that we start will have multiple variations. Each one of those variations is a different way of implementing the hypothesis. So I can pull up any a B test that we're running right now. There's probably three or four challengers to an original. Each of them. They're all within the sphere of that hypothesis. They are all an implementation of that hypothesis, but in different ways. So by the third experiment that you've run, you probably have hit that hypothesis in nine different ways. That's when usually we say, you know what, let me find another CRO to take a look and show them the results and see pick their brains. And I'm also willing to say, you know what, let's investigate further because this whole thing, our implementation of the hypothesis, was correct. So there's something in the hypothesis that's missing. Let's dig a little bit deeper. Let's ask more questions. Armed with the fact that we've ran multiple ways, each of them is coming at the hypothesis from different angles, let's go back to the drawing board and now think about the hypothesis even further.

Yeah, that makes a lot of sense. Let's look at a different scenario. Right, You've launched a test. It hit like the right sample size. It ran long enough, maybe for more than a week or maybe for more than two weeks. And then there was one winning variation. What do you do? Do you just go ahead and implement it or. There are certain things that you have to do.

So you've ran an AB experiment. You've made sure that the experiment ran for the sample size that is required. You've ran it maybe for a week, two weeks. So everything looks good. You have a winner. The chance to win is at the right level. Do you implement the AB experiment or not? Quick answer. We don't. And people always find it very strange. So ran an experiment on the product page, have a winner. I am super excited. Most of the times people are saying, hey, let's move on, let's implement this. Well, this is a single AB experiment. Guess what? I firmly believe that with every winner, it's going to change and it's going to be challenged again. The minute we have a winner, we know that we are going to be running a second and a third and a fourth AB experiment against that winner. This winner is going to continue running and we're going to direct the traffic to it through the A B testing software that we are using. So we say, you know what, 100% of the visitors are going to see this winner. And we're going to monitor the metrics. That's number one. Number two, I have a second iteration. Other challengers that are going to run against this winner. So usually within a week, sometimes two weeks, sometimes right away, we're running a second A B experiment where that winner now is the control, is the new control. And now there are other challenges that are hitting that winner that are trying to beat it. Now. People find it very strange and they say, why don't you implement the winner? I tell them, because the page and the winner will continue to change. I don't need to implement something. I am going through a cycle of iterations. So I start with the Design in State 0. I have a winner. Guess what? I run another experiment. It's my to win again. But by the third experiment, we'll probably have another winner. By the third or fourth experiment, I will have another winner. Usually we ask for a winner, the final winner of all these different battles to be implemented after two to three months. That's when we come back to the company that we're working with and we say, you know what? We've hit this testing cycle. The winner has changed three or four times. Now it is time to implement the new winner. That's going to be the new baseline. That's the way we like to do it. There are some companies who say, you know what? Now we'll implement with every iteration. But I think that slows down the program because the IT and dev resources from the company that you're working with might be busy. And by the time they get to it, I'm like, no, I'm not gonna wait. I need to be moving, and I need to be moving really fast. So that's the reason we hold off. We test again and again and again until we reach a point where we say, okay, now we've exhausted this round of iteration, this round of testing. Let's go ahead and implement now. The only caveat to all of this is that sometimes people take this to a whole other level of waiting. We've worked with a company, we're working with them right now, where they have not implemented winners since 2017, 2018. Horrible idea. Almost like, you know, every year we have that discussion with them where we say, guys, we're dying over here. There's 33. I actually counted it about a week ago. 33 different AP experiments running on your product pages. That's just not a good practice. Now, judging from the outside, everybody will tell me, oh, yeah, that is silly. Why are you doing that? So I'm like, there's always kind of correct specifics of why this happened and how we landed here. Is this a good, healthy situation? No, it's not. Are we doing something to fix it? Yes, we are, but avoid that situation. So usually whenever we work with a company, we tell them, if we give you a final winner. So after all of our rounds, we ask you to implement it within a couple of months. We want you to commit that. I've tested multiple iterations. I have multiple winners. I have the final winner for your product pages or your homepage or landing page, I want you to commit to implementing it within two months. And usually the companies I work with are open to that.

That makes a lot of sense. So what if there are about two or three winning variations? Is it like still the same way that you did with the first with the scenario you just explained?

If I have multiple variations, multiple winners, there is a good chance then I'm looking back at those variations and I need to understand the logic between them. And then I need to understand where is the business going, does one of those variations that have won. So if I have multiple winners and I have one of the variations that actually matches the direction that we want to go with and it's one of the winners, I'm like, okay, well this variation has a preference, but still it's not something that I'm going to implement. Correct. If I have two to three variations, then most likely I'm going to be running the test again. I will pick maybe the control again. Although it lost three variations, I'm going to run it again because I want to have that confidence and then I'm going to pick one or two of those variations to run again. The idea there is to have confidence and to make sure that the results actually make sense. But having two or three winners is a very good situation to be in where you say, ah, okay, multiple variations have beaten the control. Where's the business going? What is even sometimes personal preference? Correct. Because if I have multiple winners, we've had that instances where we had multiple preferences and we say, okay, so they're all performing well. We've ran the test even a second time and they continue to be winners. That's when we say, okay, which one do we prefer? Where are we heading? Are we doing something in the next six to 12 months that we want to kind of think about? And does one of those variations take us closer to that direction? That's what you want to consider at that point in time. But it's an ideal scenario. I love to have scenarios like this where it's a lot better than a test with non conclusive results. I call them winners and losers, man. I call them winners and losers. People are like, you know, oh, it's like, oh, it's not. I don't care what you say. As a CRO, that's what businesses are thinking about. Correct. I run a marketing campaign. It was either successful or it was a failure. And by the way, a win or loss doesn't mean that I'm going to go in a certain direction or in a Different direction. I might sometimes go with a loss or sometimes actually, you're running Facebook Ask and whether for fake buy or invest. And what do you tell me? It's like, yeah, the cost over here is too high, is going to go down. I'm like, yeah, it's fine. There's a lot more to running an experiment and to running a marketing campaign than the pure results. The pure results are kind of like, you know, guiding metrics. But there's more. I'm like, well, I want Simba to learn this, I want Simba to learn that. I also want our team to experiment. I want to. There's many, many things. But at the same time, imagine like, oh, we're running a whole bunch of Facebook campaigns. But, yeah, don't share with me the cost per lead and don't share with me like, you know, the cost per subscriber we don't need. That is cold, stupid business. Or a business with amazing budgets. None of the clients that we work with have unlimited budget, unlimited resources. But, you know, some CROs operate in their own universe. They create their own universe because experimentation is hard. It really is. There's harsh realities and. And if you're running experiments and it's constantly losing, they're like, oh, yeah, I want to learn, but I want to win as well. I love to win. So anyways, let's go back.

Okay, before I jump into the next question, I just want you to ask something which will lead us to the next question. I see, like, a lot of CROs, or a lot of people, generally, they don't really celebrate. In fact, there's nothing to celebrate when a test doesn't win or underperforms. Let's talk about the lessons that you can learn. Can you learn something from a test that didn't do well? Because most of the people, they don't really like, celebrate those.

Yeah, I mean, I don't know if I would celebrate it. Every test for us, 99% of the test for us will have things that did not win because every experiment has multiple variations. And those multiple variations are an implementation of the hypothesis. Yeah, you might have one winner, but you have two, three other variations that lost. They are an implementation of the hypothesis, but they did not generate a win. And yes, we're happy when we have a winning variation. But I think real CRO, the process of conversion rate optimization, starts when you ask yourself the tough question. Okay, great, I have a winner over here. But why did V1 and V3 lose while V2 won? What happened there in V1? And V3, now, some people would like to say, oh yeah, we've learned from this. You don't learn from this, from the fact that V1 and V3 have lost. All that you do is you start creating a hypothesis about your visitors of why V1 and V3 lost. Hypothesis is a lot different than learning. Still, it's an assertion. We think that way and the hypothesis will only be validated through additional AB testing. So you might say, well, V1, we think V1 lost because people coming to our website don't like scientific mentions. And we think V3 loss because of this particular design pattern doesn't work for our visitors. Again, that's a single experiment, two variations in it, both of them have lost. You're making hypothesis now. Walking away from the experiment without thinking about those variations that have lost is a complete waste of time and money. You need to think about those variations that have lost. But at the same time, to think that you actually have concrete, solid learnings based on a losing experiment or losing variation is also a complete waste of time and money. What you do is you have those hypotheses now you will validate those hypotheses through additional AB testing. Just like, hey, maybe we'll test this again and maybe we'll hit on this type of variation in a different experiment. Now, if this hypothesis has been validated through multiple A B experiments, every time we have introduced this design pattern to our visitors, it lost. And I've ran it like, you know, 10, 12 different pages, 1012 different locations. Every time it loses. I think you might have a good, like, you know, good solid insight over here. I go back to the scientific mention because we've ran that with one of the customers, one of the companies that we're working with. Every time we mention scientific research in a headline in benefits, it just lost. We're like, do people actually hate scientific research? We've tested it again and again and again over a year. We just are trying to validate, just see. And for that particular website, for those particular visitors, yes, the mention of scientific research would actually cause a drop in convergence. But that happened after multiple experimentations. I couldn't make that statement after the first experiment. No. I had to validate it multiple times. And I think it's fascinating. By the way, sometimes people want to come to really quick questions and they say, oh, there's an experiment, I've learned from it. I'm like, no, it really can raise interesting questions for you to ask. And it really also helps you kind of create this full path of experimentation. Correct. Because to me, it's a tree that kind of goes in different areas and you're trying to validate different things about your customers. It really helps kind of understand our customers better.

I had to ask that because there's this common phrase in CRO. People often say test to learn and not test to win.

So it's kind of like, yeah, test to wanna learn. You're always learning. You learn when you win, you learn when you lose.

Okay, so the last question is, what should you do when the test loses? Do you discard the variation without doing any further investigation or no, I think.

We'Ve answered this correct. If a test loses, go back to the drawing board. Ask the questions, what happened here? You're always asking the questions, what happened here? Do I create the hypothesis? Maybe even before it's like, hey, yeah, the implementation was not very good. I want to add something that most people don't pay close attention to. You've run an experiment, you start with the hypothesis. There's three different variations. Correct. They're all an implementation of the hypothesis. And you have to have a clear understanding of your hypothesis and what you are trying to do. And you want to look at the results with kind of careful eye. To say, really, this variation caused a 40% drop in conversions? Or this variation caused a 40% lift. By the way, this happens for both a lift or a drop. Sometimes I see people celebrating something like that. I'm like, what did you do in the test? Because that's very unusual when you see such a huge increase or decrease. I want to see what they've done. Now, sometimes the change is substantial and justifies in my mind that, okay, yeah, you've changed things drastically. Sometimes the change is so small, and I'm like, really? So you think by changing this one headline or by changing this small thing that I couldn't even notice on the screen, we have a 40% drop in conversion. And they say, yes, the data is very clear. My intuition at that point is to say, look at session recordings and do some more quality control of your test. Because I think there's a problem with the data. If the change is so little and you're seeing the results are telling you that, oh, there was something magnificent. Now, this is, by the way, contrary to one of the people who talk about conversion rate optimization. He had always said, it's not the size of the change, but it's the size of change in the minds of the visitors. I always thought that was. It sounds really good, but it really means nothing. No, the change needs to be obvious. People need to notice it. If you made a change and no one noticed it and you think that's going to help you improve conversions by 15, 20%, okay, you haven't done enough experimentation. So it's one of those problems where people say things that sound really cool, but the reality, they break on the rock of reality whenever you test them. So look at your results, make sure that the results actually correspond. And if you're seeing that the results, whether it's a positive lift or a drop in convergence, is really large and the change that you made is really small, most likely there's a problem in the quality of the testing that you've done outside of this. Ask questions and create hypotheses about the losing variations. They'll say, yeah, that is correct, but now we need to examine our results and examine and create new hypotheses for those losing variations. And again, more hypothesis, which means more testing and more testing. And then some of that testing is going to validate why or hypothesis of why something lost or it's going to give us more questions to ask. Here's something cool that I always say. Good CRO starts with good questions, ends up with more interesting questions.

That's a good one. I think those are the questions that we have for this week.

We did talk about a B test results. So myself, Simba, are always living on the land of LinkedIn. We are going to have LinkedIn Live next week. So drop us a message if you have a question, if you'd like us to answer it. Until next time. Thank you, Simba.

Sure. Bye.

D.