#45 – Debunking A/B Testing Myths

You should have seen the description that I put in in LinkedIn. I'm like, myth, reality, or it depends. Really my least favorite word, by the way, when it comes to marketing is it depends. Because I'm like, ah, man, gotta take a stab. Some of them are myths, some of them are it depends. Some of them are reality. So, you know, let's go through that so I can tell you whenever we hire somebody, and I've got through this process, like I mentioned in the last couple of weeks, trying to bring on a senior CRO. Statistics do matter. I don't expect you to have masters in statistics, but I usually ask very specific statistical questions that I think are really easy. And if you spend anywhere between two to four hours, you should be able to answer those questions. My name is Khalees Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult out CRO questions, dropping insights like it's no big deal.

Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we'll talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answ answer them here. And if you love conversion optimization like we do and certainly like Simba does, subscribe to the CRO Live hour podcast today. Wherever you listen to your podcasts, Simba. Another week, another Seattle live hour August 17th. Has it been.

It's been really good week in terms of work. There's been like, a lot of work to do. I used to think that, like, I was doing, like, a lot of work, but by now I feel like, yeah, there's a lot to do and then a lot of pieces to make sure that everything, like, is going well. But I'm enjoying it. I'm learning every time, so that's good.

Yeah, it's just fascinating to me how investment, how fake pie have grown over the years because you joined us when we had decided we were going to grow the team, but we're not going to grow it to the point that's where it's at right now. I think we crossed the 30 team members.

I think we might have crossed even, like, 40, including the freelancers that we have.

Oh, definitely. Yes. Yes. If you look at that, we're definitely over 40. But, you know, one of the questions that I've been thinking a lot about is the culture of the company. I wish I have a very elegant answer to talk about the culture of invesp. And this is horrible to say maybe I know only about 25% of the culture of the company because the team is spread all across the globe. So, yeah, okay, so we sit in meetings every day, but at any point in time in a month, I'm probably meeting and talking to 25% of the team members. So I haven't talked to some of our marketing team members now in, like, three, four months, which is absolutely horrible. So just kind of trying to keep track of everything and thinking about the company and the culture and the growth. I don't want to wait until we're 100 team members. I don't know if we ever want to grow to that point, but I'm just kind of thinking about that. It's like, you know, who do we hire? We're about to add, I think, three new team members. So I get stressed out, by the way, whenever we add new team members, because of all the different things that we do at invesp, hiring is the hardest thing. The easiest thing we do is CRO, by the way. It's like, oh, we figured this one out.

Hiring.

And we've just modified the hiring process. So I'm interested to see the results of the hiring process. But basically, that means. Also I've spent the last two weeks interviewing a ton of people. Little overwhelming, but fascinating nonetheless.

Yeah, it's interesting. Like, especially also for me, I remember time when I joined, we were just like, about 15 people or so in the office, if I'm not mistaken. Or there were other people, like, who were working remotely. But in the office, we were just, like, maybe just about 15 or less. But right now it has grown, which is a good thing. And also with the marketing team, we're able now to do, like, a lot of things that we didn't do during the first year that I joined. So that's cool.

And you know what's fascinating, because I remember the time, like, everybody was in the office. I mean, reflect back at that time because you've joined around time or were in an influx, trying to figure out how to hire the right people and whatnot. And unfortunately, I think the hiring. Many of the hiring decisions we'd made back then were not correct. So lots of people left us. But the people who stuck around, they've been around since then. We have team members down with us for many years. I think the longest somebody stayed with us was Gagan, who was with us from 2009 until 2021. And that's when he decided he's gonna completely quit marketing. He's gonna focus on his family business. But he was with us for a very long time, almost 12 years. So fascinating. But I basically spent the last two weeks interviewing a ton of people and just that interview process and then knowing who to look for. Preparing to bring on a new CRO for invest is fun. It's stressful.

How's it going, like on the side of FigPi? I just can't imagine like you're trying to juggle two companies at the same time.

Think by has been very interesting. So we used to do a single monthly release. It's funny. What do I have behind you? Experiment, fail, learn, repeat. But I have assigned somewhere else. Move fast and break things. That's a horrible, horrible idea. Especially when you're playing with data. I mean, we have companies with millions of visitors coming to their sites and we're looking on. They're using figpy to run experiments on their site. Can move really fast. I need to make sure that that integrity is there. So we used to have a single monthly release of all the new features. We've moved from a monthly release to a weekly release. And oh my God. First it's fascinating because now we're able to push the team to move a little bit faster. But at the same time, the growth there has been really incredible. More and more companies signing up for figpi and some are really large companies that I'm like, oh, wow. And they're comparing us to some of the big players, which I've never expected to be compared to some of the top players. So I'm like, okay, this is really cool. Because we've carved a niche for ourselves. Like, okay, here's what will work really well. And lots of times people ask us like, hey, what do you recommend? Sometimes we recommend optimizing, sometimes we recommend vwo because those are really good tools and I love those tools. So especially vwo. It's a tool I've known from the very first, probably few months when the tool was released. So we've recommended it quite a bit. But now if VW is not the right fit, then in that case, companies look at figpies they might look at some of the other competitors, but those are kind of the companies that we're looking at. So the growth at figpi has been absolutely incredible.

Nice, Nice. That's cool. So today we have an interesting topic. They're not like really questions, but they are myth that people have around a B testing. What's your thought when you just did those?

You should have seen the description that I put in on LinkedIn. I'm like, myth, reality, or it depends. Really my least favorite word, by the way, when it comes to marketing is it depends. Because I'm like, ah, man, gotta take a stab. But that's the answer to everything.

That's the answer to almost like 85% of marketing questions.

That's true. That is true. But people absolutely hate it. And I can understand why they hate it. Some of them are myths, some of them are independent, some of them are reality. So let's go through them.

Cool. So we have about six of them that we'll try to talk about here. So the first one is, is a B testing complicated and does it require a lot of statistical experience?

So I can tell you whenever we hire somebody, and I've gone through this process, like I mentioned, in the last couple of weeks, trying to bring on a senior CRO. Statistics do matter. I don't expect you to have masters in statistics, but I usually ask very specific statistical questions that I think are really easy. And if you spend anywhere between two to four hours, you should be able to answer those questions. I like to ask, hey, you've run an experiment and you have a control and you have a challenger to it or a variation and you have a winner. It's a 12% lift with a 95% either confidence or chance to win. It depends on the statistics statistical model that you use. And you're now sitting across the table from a CEO or a VP of marketing and they look at you and they ask you, what does 95% mean? I want to hear how people explain it. Now, granted, I do believe that most people don't know how to answer that question, even people who work in the field. And by the way, that included me. At some point I actually had to bunker down and just do some research to really fully understand it. And that I think happened back in 2013, 2014, when I'm like, you know what, I'm computer science, so I've taken some statistics class, but I'm like, I need to delve a little bit deeper into that. And back then we had an absolutely amazing lady who was our head of statistics. She actually teaches statistics now at a university, but she helped me really through that. So I want to listen to how people explain that. Where I see is like, you know, some people really don't have a clue on how to explain it. Some people try and they explain it in what I would say, acceptable way, not exactly the most correct way. And some people, like, don't have a clue at all where they. I'm like, they just avoid the question. I'm like, okay, yeah, you can say that. You don't know it. Sometimes people say things and then they're so certain and they are so incorrect that I find it entertaining. Yesterday I was interviewing somebody, mind you, he only has two years of experience in CRO, and he tried to explain it, and he's like, well, it depends on the statistical method. I'm like, oh, good. I like where he's going with that. Oh, it's like, you know, if you like frequentist or Bayesian. And I like frequentist. I'm like, okay, that's fine. I'm like, why? So you threw a couple of big terms, correct? And you should be able. If you're going to knock the door, you should be prepared to hear the answer. So I'm like, oh, why? You just sort of opened yourself up and he just stopped fumbling the question left or right and didn't know. And then the question that I follow up with is like, well, 95%. Why not? 99%? Why not 90%? So going back to the myth, you don't have to have a master's in statistics to do conversion rate optimization experimentation. But I do believe that you need to understand basic statistics so you can speak intelligently and you can understand the data that is in front of you. And you should be able to explain it to the stakeholders, Whether somebody internal or somebody external. You should be able to explain that. Okay, here's what this means. There are some amazing, amazing CROs who do not understand statistics, by the way. I have to say that. And I know that because sometimes they talk about the statistics, and I'm like, okay, you haven't done enough research. That doesn't take away from how great of CROs they are and how great they can, like, how great the testing ideas that they come up with. I think they will be so much better if they invest a couple of days understanding statistics because it's going to take them to that next level of experimentation and conversion rate optimization. One of the things that we do now, whenever we hire a CROs, we have A test project. And part of that is around, I don't expect anybody to memorize statistical formulas. I'm like, I'm sorry, no one memorizes statistical formulas. If you ask me, hey, how do you calculate standard deviation for something? I'm like, really? I don't know. I can look it up really quickly and tell you. But you should be able to explain what standard deviation is and you should be able to explain what significance is. Lots of times people even confuse significance with confidence and use them interchangeably. No, they're not the same thing. So it's fascinating because I'm almost sure somebody could be listening to us and saying, khaled just made a mistake in statistics. Hey, I know enough to be dangerous. I know enough to explain. I'm probably more than enough after the intensive year that I did in statistics. But do yourself a favor. If you're going to do experimentation, you don't have to get really advanced, but at the same time understand the basics. I think that will make you so much better.

CRO so asking, maybe for someone who is listening, what do you think, like, is the difference between significance and confidence? Just like in a few words.

So the formulas, by the way, that you get for all statistical formulas, whenever you run an a B test are around calculating the significance. How significance? This difference that you have, you have a control and you have a variation and then there is a lift. How significant is that result? Another way to think about it, kind of like if we're dumbing it down, how random with the results where the results that we got, did they happen just because, hey, we got lucky in this experiment and that percentage of randomness. So usually you're aiming for a significance level of 5%, 0.05, or if you're going for like, you know, 0.01, these results being random is about 5%. The chance of these results being random that, hey, we just got lucky. Yeah, we made changes, but we got lucky. I don't know how it's only 1%. Confidence is 1 minus significance is 1 minus P. If the significance is 5%, then the confidence is 95% and vice versa. Correct. So that's how you calculate both. Lots of times people confuse them. Significance is smaller, confidence is high, significance is the significance of the possibility of randomness and the confidence, the less random the results are, the more confident you are in the results that you are getting.

Moving on to number two, the higher the testing velocity, the better the experimentation program. What do you think about that? True or false? It depends.

It depends how about if I say this, a good testing program will have high velocity. Correct. High velocity does not mean that you have a good testing program. It's not the other way around. I always said you need to learn how to crawl before you walk. You need to walk before you run. And to me, testing velocity is about running. And experimentation is a marathon. And if anybody who tells you, well, no, it doesn't really matter, we can just do one experiment per month. One experiment a quarter. Yeah, I guess you can run a marathon, but I feel bad for you if you're just walking 100 meters a day. It's a painful marathon. Correct. And you're going to get really frustrated and tired. Now, you need to understand that the quality of the program starts first. And I see a lot of people focusing on the fact that, oh, testing velocity is not a goal. Oh, because then you're just testing things for the sake of testing them. No, that's a myth. We're not testing for the sake of testing. We're not testing for the sake of velocity. And if you're doing that, then you're being stupid. Then there are no pills for stupid. Start with good experimentation, start with good ideas, start with good research, come up with the problem statements, come up with the hypothesis, but then move fast. Because things change as well. If you're doing one experiment a quarter. And even if that experiment is really insightful, guess what? The market is constantly changing. People are constantly changing. Can't wait. There are experiments literally that we run them, they win, we rerun them, we always have this where we take the winner and we run it against the control. In a head to head match, say we have a winner, let's just make sure we run it again. We're aiming for at least a 95% confidence, sometimes 99% confidence. It depends on the type of page. We have a winner, we deploy it, it's doing really well, and we come back maybe seven, eight months, maybe a year later and we test it again and would have lost because things are constantly changing. And if you don't acknowledge that fact, then we probably don't live in the same universe. You want to move fast, making sure that you have the right methodology, the right process in place that allow you to move fast. Velocity is not a goal in itself, but velocity is a component of successful testing program. And what's funny, by the way, is how do you define high velocity testing program? At some point, and this is back probably 15, 16 years ago, high testing velocity program was, hey, we're doing one to two Experiments per month. If you say that to somebody nowadays, they'll be like, really? I mean, that's sort of very slow testing for us. Even I wouldn't consider five experiments a month. The high velocity testing program. The minute we start talking about six to eight. Now we're going really fast. I was looking at the data from one of the companies that's been with EnvespNOW for about two and a half years. And we've done in two and a half years, 372 experiments. That is high speed testing velocity. And guess what? The challenge there is not coming up with ideas. Oh, my God. There's a ton of ideas. And every time you run an experiment, you look at the analytics, you look at heat maps, you can come up with more really solid ideas. But when you run that many experiments, there's a completely different challenge, which is making sure that you're finding the right insights, that you're keeping track of all the data. Because it's a lot easier conversation when you only have 15, 20, 30 experiments. You crossed 370 experiments. Oh, my God. And we have systems. But in all honesty, I'm finding out that our systems at 370 experiments are not operating as well as they used to. We're talking about doing 50 experiments in a year because now we're talking about 180 experiments in a year.

Yeah, that's quite like a huge number. Moving on to the third myth. You have to run a test until it reaches statistical significance.

You don't run the test until it reaches statistical significance. That's the wrong way of doing a B testing. Before you start, before you launch that button, you plug in all your numbers into your a B testing duration calculator. I'm running a test on the homepage. I'm running a test on the cart page. I'm running a test on subscription page. Here's how many visitors are coming to it. Here's the conversion rates for this test. Here is the expected lift, when we call the mde, kind of the fancy statistical term that I expect as a result of running the test. And here's how many variations. Tell me. Oh, great. Test duration calculator. How long I'm going to be running the test for? Oh, you're going to be running the test until you collect 300,000 visitors or 60,000 visitors. Really depends on those parameters. And you can play some of them. You control how many variations I have in an experiment or what kind of lifts. Some of them may be a little harder. How much traffic is the Page getting. Unless you're driving traffic through paid and say, oh, we're going to drive more traffic based on that, it will tell you how long you run the test. You run the test for that duration, and then you evaluate, did we achieve statistical significance or statistical confidence or not? Do we think that we have something there or not? And you plug in your numbers. The duration calculator tells you that, okay, you need to run this experiment. It's based on the number of visitors, which means number of days as well. We've ran it for that long. You look at the data. Did we reach significance or not? Did we achieve significance or not? And then you have to evaluate. Okay, well, yeah, we were aiming for 95% and significance. We're at 90%. You have to analyze. Do we launch? Do we change something in the experiment? Do we say, you know what, 90% is good? Sometimes people freak out when I say that. No, you can evaluate whether 90% is good or not because again, significance measures the randomness. Correct. Of this result, there's 10% that we just got lucky. You know, is that acceptable? Because there is risk. There is risk by deploying this, that, oh, it was actually random, and we didn't really have any uplifts. How much are you willing to take that? Randomness in business, Forget about the agency side, but being a business owner running two companies, I take more than 10% chances of randomness all the time. I'm willing to take probably even 30%, 35% chance of randomness just because to me, the name of the game is moving fast. But again, that's about the culture. That's about even bigger kind of the analysis that we need to do and the insights we got from an experiment.

The next myth is you should a B test everything before making any changes on the site.

I absolutely hate it when somebody constantly like, oh, you should test this, you should test that. You should test. Yes, test. But you don't test everything. There is a reason for that. Sometimes when there is a bug, I don't need to test the fix. I'm like, dude, come on, you know, for God's sake, password field is broke. I don't need to be testing it. Or there's a JavaScript error. Oh, well, you know, we have a fix for it. Let's ab test it. I'm like, really? I mean, yeah. Okay, great. Because of JavaScript error, people cannot submit the order confirmation. And I'm going to run an a B test where I'm splitting the traffic. And then you think that I'm joking, but I've seen that where it's like, oh, this is an absolute slam dunk of an AB test. This should not be an AB test altogether. It's a bug fix. You don't need to be fixing bugs now. We need to distinguish correct between bug fixes and UX improvements. They're completely different. A bug is a bug. Bugs should not be AB tested. Now, UX solutions, UX enhancements, conversion enhancements, revenue opportunities, I test all of those. And since we're looking at taking a detour, why, I don't like when people say, oh, you should test this, you should test that. Yes, I am all for testing good ideas, but to me, an a B experiment starts with a smart question and ends up with smarter questions. Right? What is the research? Why should I test this? Just because you think it is. I guess that's good enough for some people. But guess what? I have limited time, I have limited resources. I need to be very conscious of the experiments that I run. Again, it goes back to the same velocity. We just talked about that. Think before you test. So make sure that you have good testing ideas. Where. Okay, so what's the question that we're trying to answer? Okay, so did we answer this question? And what's the next question that we should be asking? That's my answer too. You should test everything.

The next one is, is a B testing only for websites with a ton of traffic.

It does not have to be a B testing. There are websites where it doesn't make sense to do any type of a B testing. We're talking to a company and they are a client of invesp. They get seven conversions a month. Seven. Fairly large client, by the way of invesp. Now, it doesn't make sense to do any type of AB testing because in that case, oh my God, I told him, I'm like, let's just do one experiment with the 30% expected lift and one variation. And this experiment is going to take about 200 days to conclude and then we'll see you next year to run another experiment. It does not make sense to run experimentation. Now, in this particular case, it was a landing page and that was the conversion rate for it, and that's it. But think about this. I was talking to somebody. They run the experimentation program at Microsoft specifically for Xbox. And if you're on a landing page that sells the games for Xbox, your typical investment approach is like, oh, I need to test different versions of the landing page and I want to see which of these designs had helped increase the conversion rate order purchases correct. Because Microsoft is just Microsoft and they're large, they don't have the ability to do that. So they're like, oh. When we're running tests on the homepage, our goal is to get the visitors just deeper into the site. If they're on the pdps, we need to get them to click on the add to cart. So they're doing micro level testing. Now the nice thing about micro level testing, it makes life tremendously easier because it doesn't matter how many conversions, you might be getting 200 conversions. But if we're looking at a micro step from the homepage to another page on the website, guess what, you'll have probably a ton more conversions. Now in theory, you know, it's almost like you have a piping system, plumbing system around the house and something is blocked. So I'm going to open that pipe, more visitors are going to flow. It doesn't always, by the way necessarily mean that you're going to get a lot more orders. Sometimes you don't. And then like you say, okay, well we got a lot more people to come from our landing pages to the product pages, but not enough people clicking on the add to cart. What do we do there? I'm like, that's an interesting question. What promises do we make them? Are we fulfilling the promises? What's the breakdown of the traffic? There's lots of questions that you do so you don't have to have the traffic. You can get really creative. But sometimes if you're dealing with a website with 5, 7 conversions, maybe a B testing, not maybe AB testing is not the right solution for you. That doesn't mean that you don't do conversion optimization. That doesn't mean you don't do experimentation. It's just that last piece of conversion optimization is not going to be part of the program.

I think like for some people, the confusion when you say that you can't do AB testing, they kind of like think that you can do conversion optimization of them, of which they are kind of like two different things. So the next one is a B test is only as good as its effect on conversion rates. How far trip that?

Oh gosh, that's like a kind of foreign. No, an A B test can teach you so much about visitors. As a matter of fact, we love winning a B test. Anybody who tells you, oh like it doesn't matter to me whether a test won or lost, they probably don't have enough winning tests, you know, just to start with. I love winning experiments. But again, as important to me as, hey, what did we learn from the test? Because testing philosophy behind a B testing is I am going to make the user journey on my website a lot better and at the same time. So that's user experience. I work for business, correct. It's not charity. So I want to make them more money. I was having this discussion with a senior UX designer. He's like, oh look, it doesn't matter. This tension always between the user experience and the business and user experience really is what should win. I'm like, really? He works for an E commerce website. I'm like, I will give you the best user experience on the face of the earth. Your visitors will love you. He's like, really? I'm like, oh, I can do that really easily. He's like, how? I'm like, just offer everything for free. Best user experience. As a matter of fact, this is so good that people are even willing to ignore your whole job. Your business is going to go bankrupt. So it's not tension between user experience and conversion rates and conversion optimization. They're both working together. Better user experience, keeping an eye on improving conversion rates, improving revenue. Now, we've mentioned this before. Really. For me, it's the growth that matters. Conversion rate is a very tricky metric. Like to say, oh, my conversion rate is 1%. No, it's not 1%. Your conversion rate may be over the last month, might be 1%, but conversion rate yesterday is different than your conversion rate today. It's a bell curve. There's a distribution for that conversion rate. It is anywhere probably between 0.9 to 1.105 and it depends from one website to the next. The beauty of conversion optimization and experimentation is like, oh, here's an experience as the website as it stands right now, I'm going to present random segments, equal segments of visitors with a different experience and I'm going to compare their behavior and based on that I'm going to judge, are we seeing metrics, better metrics? That could be many, many metrics. Conversion rate is one of them. But you might be looking at rpv, you might be looking at aov, you might be looking at other metrics, user flow through the website, all of these.

It's not.

Conversion optimization should not be summarized in one metric. It's a lot more sophisticated. I was doing a demo for figpi and somebody asked me, are you guys using AI and how does that play with figpy? And I told him, well, really, we're spending quite a bit of time on using AI for ab Testing. And he's like, really? He's like, that seems to be the least important part. And I'm like, oh, interesting. Why do you think that? He said, because it's very clear. Control variation. You ran a test. I'm like, really? No, there is so much of like control variation, but there is like, you know, probably 10 different metrics that we're tracking. And oh yeah, the control did better in this one metric, you know, did worsened this other metric and what's the. The impact and its conversion rate is one metric. We're tracking a ton of other metrics. I told him it's a huge matrices that we're looking at to evaluate the performance of an a B test. And that's really where AI matters when it comes to a B testing. He's like, oh, I didn't know. I didn't think of it this way. I usually just run with one goal. Control versus variation. I'm like, well, okay, the more sophisticated your experimentation is, the more you probably need. You can do it humanly, but AI will be very helpful.

That makes a lot of sense. I think, like, those are the myths that we had today. If you have heard of any crazier ones than the ones that we had.

On the list, those are kind of the typical myths that we hear. I think experimentation, kind of just to finish with that is a way of doing business. I've mentioned this before. It's a way of thinking, it's a way of doing business. And companies who embrace it. It's beyond just conversion rate. It's the way to do business. The way you like, you release features, the way you evaluate features. And it doesn't necessarily mean that it's going to take you down a certain path, but it definitely is almost. Think of it as the lantern or the light that you walk with so you know exactly where you're heading as opposed to. It's like, how did we get here? You start with good experiments, you know where you're heading. You're able to look around, you're able to make data informed decisions.

So those are the myths that we had today.

Hey, Simba, until next time, happy testing.

Ciao. Enjoy your day.