#51 – A/B Testing and Decision-Making: Lessons from the CRO Trenches

I would say those are the toughest type of companies to conduct CRO and AB testing for. Otherwise, if you're willing to listen to people, if you're willing to let your visitors guide your decisions, I think CRO experimentation is just easy to do.

Yeah, I do agree. You touched on the next question when you were explaining, like, the situation you had, like this previous client in the board. So my next question was, maybe you're going to answer it differently or you're going to clarify more. How do you handle situations where a B testing results conflict with stakeholders intuition?

My name is Khalil Saldeh and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing experimentation and conversion rate optimization.

Each episode, Khalid amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today wherever you listen to your podcasts. Simba. Another week, another CRO Live hour. It's been two weeks since we recorded. It's been.

Is it two or three?

I don't know. It's horrible. One day you had a problem in the building where you lived. One day I was traveling and last week I forgot also something major happened where I'm like, okay, we're not going to be able to do this, but we are back. Excited to be back. How's life, Simba?

My life is good. Don't ask me about Manchester United, but my life has been good.

I don't know what's going on. I don't think there's a quick solution, but sometimes you get stuck in these slumps where just everything goes wrong. And by the way, whether it's a B testing or soccer or football, sometimes things just don't go the right way and it's just, okay, you need to pause, you need to step back and look at what's going on. And you need to make sure that you have the right ingredients in place. If you have the right ingredients, it's fine. Everybody goes through ups and the downs. But if you don't have the right ingredients, that's what you need to ask. Are we talking AB testing? Are we talking soccer? What's happening with Manchester United? Right ingredients? Just bad time or. No, there's a whole bunch of problems.

You know what the funny part is? I now have a son and I was actually thinking of initiating him to Manchester United age so that whenever he grows up, he would know he grew up supporting United. But right now, yeah, it's not looking good for us. There's no point in doing that. So I'm not sure. Maybe I'll come up with something different.

Yes, we went through a really bad slump last year. Correct. I just could not wait for the season to finish. It's like everything that could go wrong just went wrong. It was just like the last three months, like, oh, yeah, here we wake up. So I was like, really, man. So anyways, let's talk. Let's talk CRO. Let's talk. Well, how is the marketing team at Invesp and ThinkFi? How are you guys doing?

I think we are doing good. We've been just like, trying to put out some SEO files. Like, Google keeps on like, rolling updates and then I think I can say that we were hit hard by the update. So we've been trying to recover from that. And then once we started like that, we are recovering bit by bit and then they roll out another update. So we're trying to just going back and sorting out like our content on our blog. And it's a lot of work. But I think like, by the end of in the next few weeks, we'll see like, how our effort, like, pan out like in the next few weeks. So it's interesting. So that's usually what's happening with the marketing team at Invest. It's been good, though.

Excellent, excellent. I'm glad to hear that. It's just interesting time. Fake Pie has been, of course, on fire since Google Optimize has gone. It's been just exciting.

You know, I was looking at the FIGPI numbers, like getting over 40 new signups every week. That's a good thing.

Yeah, definitely, Definitely. And I'm like, okay, it's the right signups. And then what's interesting is there's lots of enterprise accounts that are signing up. Those are huge wins. Correct. An enterprise account is almost equivalent sometimes to 200, 300 lower tier accounts in terms of the payments, in terms of subscription, and of course in terms of the work that we have to do as well. But it's exciting to be getting all these accounts and expanding the team as well. But let's talk AB testing. Let's see what questions do we have today?

Okay, cool. So today we have a about four questions about AB testing. You've been doing a B testing since 2006, or maybe before that, before you even started invest. So if I recall very well, we have investors had like more than 30,000 A B tests so far. So I'm sure, like, you won't have any trouble answering any of these questions.

At least that's the theory. No, it's funny. Can I just tell you something?

Yeah.

You would think after doing experimentation for 17 years that you'll have all the right answers to the different questions. No, no. We're constantly changing and modifying and adding new things. And I spent some time in the last three weeks, four weeks, working with our team. So we have now we've created a research team that looks at the different projects that we're doing, some of the methodologies and we're like, oh yeah, we can modify this. This is not good. And I was just fascinated by some of this stuff. I'm like, oh, this is really cool. Like a new analysis, new ways of looking at data, new ways to prioritize items. So yeah, you never say I have the right answer. Maybe you're just constantly in this research phase.

Okay, that's good. The first question is, how do you balance the need for quick decision making with the requirement of conducting data and reliable AB tests?

How do I balance the need for quick decision making and the thorough, reliable A B tests? Fundamentally, there's something wrong with that question. And the way the person who's asking us not to offend anybody the way they are thinking about AB testing, because ultimately this question has an assumption behind it that if I'm a business, I just need to make decisions and move on to the next stage. And if I do AB testing, I'm going to be slow. Especially if I want to do reliable AB testing. Thorough AB testing is going to be a slow process. This goes back to the way you are making decisions. The decision process is not about, oh, we're going to go to this path and then move on and then we're going to go make another decision. When you are conducting experimentation a B testing, the decision is we are going to test this path. Correct. So the decision process is a little Bit different. You're never deciding on a path. You're deciding on testing two different alternatives that are available and then seeing which one people, visitors, customers like more. And then based on that, we say, oh, we're going to go down that path. Am I just kind of splitting words and the difference is really very minute? No, not really. I think think it goes back to the way you think about doing things. The best example that I can give is from our work with ebay. The first experiment that we did with ebay, and I mentioned this so many times, took about six months to implement. It was just extremely slow. Everybody had 23, 27 rounds of feedback, and everybody on the face of the earth, okay, not everybody. Lots of people had given us feedback. We want to modify this, we want to modify that. And then the experiment lost at the end. That's because they were thinking that, oh, we need to run this test and it needs to be thorough, it needs to be reliable. Fast forward, two years later, the way ebay did business is a lot different. Now we have pods, and every pod releases a feature. 90% of the time. They release it as an a B experiment. We're going to test, we're going to see if this is what people like, and then based on that, we're going to change. That required a really huge mental shift because, oh, now we're testing. Everything's gonna be tested. You don't come back and you don't come and you say, this is the way we're gonna go. No, no, no, we're gonna test this versus that. We're gonna a B test. And then based on that, we are gonna decide. And really, the minute you do that, you are making quick decisions, but you are also making quick, informed decisions, which I think is really important. Now we had a new CRO, she just joined us this week, and I think we need to get her. By the way, on the show, we have a whole bunch of new CROs who just joined us. Absolutely. She's been doing CRO now for about four years. And one of the things, she looked at some of the testing practices that we do and she came to me. I like to check in on our CROs, especially that first week when they are with us. And she said, khaled, I have a concern. I'm like, oh, sure. She said, I was with the team and I was looking at some of the experiments and they were mentioning that, oh, here's the time. I'm going to run this experiment for a week. I'm going to run and some Other people were saying, oh, I'm going to run this experiment for three weeks. And she said, for me, that is bias in the testing process if you're changing it. She comes from the school that I always need to launch the test to run for two weeks. And it starts on the same day, finishes on the same day it started on. So that was her philosophy. I told her, that is fine in principle. However, let's change the way we think about experimentation. There are companies we work with nowadays and their whole philosophy about experimentation is not to find necessarily a winner and an increase in profit or anything like that. There are companies that work with that. They run the A B experimentation to validate an idea. Is this idea worth actually our dev team spending time and our design team spending time hashing out, fleshing out, adding more details to it, or it's just really not a great idea. And what's interesting about those, there's two companies that do that. What's interesting about their approach is they're constantly thinking about the time it takes their dev team to develop a particular feature. So we might run an experiment and they get a ton of traffic. We might run an experiment after three days. They'll say, you know what? Based on our calculation, we can run the experiment for three days. We always tell them, no, three days is not enough, really, at least a minimum a week. But their philosophy is like, no, it doesn't really matter for us because all we want to see is if there is a general direction, a positive impact of developing this feature. They're like, oh, we're actually going to develop it right now because all we've done through our AB testing is an MVP approach. Now they're saying, okay, we have enough signals from people that are interested in this. Let's develop it some more. And after we develop it some more, we're going to test it again. So the philosophy is a lot different. That's one set of companies that we work, another set of companies where we're very strict about the numbers and the analysis and we run every everything for, I would say, two to three weeks. That's the philosophy that we have. Third set of companies where we say, oh, no, experiments are going to run for four weeks because maybe they do not have enough conversions to conclude in a less period of time. So really, yes, it's important to get to reliable answers, but it is within the context, the business context, correct, to make sure that ultimately AB testing is serving a purpose. And that purpose is helping you figure out the right direction, the right decisions to make based on your context.

Yeah, that's good. That's a good explanation. You mentioned like a lot of interesting things there. One of the things that she said is like at the end you were talking about this client that and then you were advising them to at least run a test for at least a week. And sometimes when I think some of the CRO or AB testing best practices, I know that you hate that term and most CRO people hate best practices, but they say you should run a test for at least a week, for at least two weeks in order to know. My question then is, is it important to run it for two weeks? Or what matters is when you hit the right sample size or you get like the required significance.

So remember, you calculate the sample size prior to starting the experiment. So that's number one. The challenge happens in two cases. The one case that's like, okay, I don't know what to do over here. When you calculate the sample size and experiment needs to run for six months and you're just sitting there scratching your head because you don't want to run an experiment for six months, there are so many factors that can pollute the data that you just need to rethink about the experiment, the factors that go into creating that experiment. So that's one other extreme end is when you have enough data and you have enough conversions and you can run the experiment. We've worked with companies where I can run the experiment really for an hour and I will have statistical significance. Two hours and I have statistical significance. Would I do that? No, that would be stupid. Because again, here statistics purely based on numbers are telling you are giving you one answer. But you need to step back and think, well, if I run this experiment and let's say your duration calculator tells you that you need to run the experiment for a day. Okay, I know that people who are shopping on my website, coming to my website, maybe to subscribe, maybe to shop, maybe to fill out a lead gen on maybe Saturday, behave completely differently than people who come my site on a Tuesday or a Wednesday. And I can figure this out by the way, by looking at my analytics consumer behavior on my website. There's many ways that we can figure this out. So what I'm saying is if I run this experiment on a Saturday, I'm going to get one result and if I run this experiment on a Wednesday, I'm going to get a different result. Although in both cases the sample size is accurate, correct? I mean, I've already hit my sample size. You're telling Me, I need to run this experiment for a day. Trying to, for a day over here. So it's important that we run the experiment. Now, here's just to complicate things. For the length of time we assume a buying decision takes a buying cycle. This buying cycle might be two weeks, might be three weeks, might be four weeks, might be sometimes longer. Correct. So that's where the challenge is. But what we like to do, we like to run experiments two to three weeks. And okay, so maybe the buying cycle is a little bit smaller. Maybe the buying cycle is six weeks. So are we getting really accurate data? That's a challenging question. I'm going to put it to the side because really it's not only philosophical question, but there is an impact on how you answer and how you think about that question and your answer to it. What I think is within those three weeks that I run, I'm going to have enough sample of people who, this is their first time coming to my website and looking at my products and evaluating. I'm going to have enough people who have already come to my website three weeks before and looked at it, and now they're in a different stage in the funnel. So to me, two to three weeks is really what we aim for. Ayat, whenever she runs experimentation, that's the role that she has for the team. But again, sometimes we do run experiments for a week and we think a week is enough. We have what we call early stop rules. Okay, when can you stop the experiments? Just to validate some of the data that we are. Some of the data that we are getting. One of the things that we like to do quite a bit, there's multiple rounds that an experiment goes through. The first round, you might have three or four challengers to the control. So I have a winner and I might have like first winner, second winner. I take those and I run them again in a second round. So, yeah, I ran for a week. Now I'm going to stop, I'm going to take the two winners, I'm going to run it for a second round for another week. So we're always validating work in a validation process, but we're also moving fast enough.

So the reason may be just for someone who doesn't really understand or doesn't really know what's the point of doing second, third round after experiment, Is it purely validation or there might be some other reasons to do that.

I would say it's a mix of two things. Validation is a big one. We want to make sure the results are consistent. I Don't want to just run an experiment. Oh, I have a winner. Oh, it's significance. It's the sample size. Let's implement. We like to validate again. Now we always take the winner from the first experiment and include it. And sometimes we say, you know what? Okay, so we have three versions. V1, V2, V3. They're all challenging the control. V1 is the winner. Okay, we're going to include it automatically. We might say V2 is a small variation of V1 and we're not going to include. But V3 was a completely. A moonshot, something completely different. Let's bring it and give it another chance to see if we can beat the original or are we seeing consistent results. So it's analyzing the trends, the data trends that we're getting from the first experiment and the second experiment.

I know for invest, the majority of the clients are E commerce, but we do have sometimes clients from different industries. So I have this question for you, which is, are there specific industries or types of products or services where a B testing tends to be more challenging?

Yeah. I'll just mention something which is fascinating to me because we've always focused on retail. E commerce, that's our bread and butter. You'll know us by that. What's fascinating, I was doing a survey of the companies that we work with. I would say 5 to 60% of them at this point are E commerce and the others are lead gen and SaaS. That's a complete shift in our client makeup. Is it wise? It's not wise. That's a business discussion. But we're seeing a lot of companies who are reaching out to us who are not necessarily E commerce. And with that there are challenges that come. Yes, there are some industries where it's a little bit challenging. I think there's two sets of industries where it's a little bit more complicated to do conversion rates optimization. One, the industries where the conversion happens offline and it takes a while for the conversion to happen. Those are a little bit challenging because. Okay, you're running an experiment. Okay. And this is usually for a lead gen. Correct. So I'm just generating a lead for you and I'm showing them. It's like, okay, here's a landing page. Here's version one, here's version two. Okay, you've converted, great. But I merely generated a marketing qualified lead. Sales still needs to qualify them and then sales need to close them. How can I attribute that back into the testing that we've done? And some of that I can tell you it gets really complicated on how you tag somebody based on the fact that they've seen a particular version, push that data from your A B testing software to the CRM solution that you have, and then based on that, eventually you look, we're working with one company where the sales cycle is about six months. Imagine this, I'm running an experiment in January. I'm not going to actually be able to close on those people who filled out the form until June anywhere. That's challenging. And it also impacts the way you do experimentations because usually, okay, we've run experiment in general, we want to run a second one in February. So how do we keep track of that? So that's challenging and you want to keep that in mind. The second type is companies where copy is very important for them. E commerce copy is really not very important. I've always believed that messaging is extremely important for E commerce. But if you are a lead gen type website, if you are a SaaS website, copy is critical because people use the words to say, oh yeah, this is a product or this is a company that I want to use. What's the problem with copy? People are very particular about copy. I can show you a sentence or I can show you some copy. And then you think to yourself, oh, this is some really good copy. And somebody else sitting there, they say, oh, this is the absolute worst copy that I have seen. I remember we worked with a SaaS company, Enterprise SaaS, and we were working with the VP of marketing. She brought us on and we knew that we had to change the copy on the homepage. So we did a whole bunch of jobs to be done interviews and we usually like to do the jobs to be done interviews because that's a gold mine of copy that could be used on the website. And we used those to create the copy, fine tuned it, sent it to her, she absolutely loved it. And then she said something that threw me off. She's like, oh, I have to run it by the board. Run it by the board? What do you mean? Oh, we have a board of directors and they need to read the copy. I'm like, okay, she sends them. And it was one of the funniest experiences. I mean, funny in a bad way, funny in a good way. Because I would get an email from somebody, oh, that is such an absolute amazing copy. It truly expresses what we stand for in a clear way. And I get an email from somebody else. He's like, I don't know how to say this, but this is some of the shittiest copy I've ever seen. I'm like, wow. So I was talking to her and I told her perhaps we should have had a conversation with the board about our approach to copy and how we do it. And unfortunately, I don't think she was strong enough to be able to stand. She had loved the cop, but she was not strong enough. And most people were not strong enough because the person on the board was the largest investor in the company, did not like the copy. He's like, I would never use that. And I think he's the one who influenced the decision. And we felt that we were going through an uphill battle. So we said, you know what, it's not worth it. We just literally walked away. The project was a lot lengthier, large project, it took us forever to close it. But then, and I've decided at that point in time, no, I'm not going to put the team through that. And I knew it was going to be a battle and I think they were so annoyed with us, they're like, oh, you can't walk away. I'm like, listen, I know exactly where this is going. CRO is difficult, naturally. And then you add on top of it that your management is getting involved in every little detail. You're not going to see the benefits. You're going to hate CRO, you're going to hate us, and we're going to hate working with you. I don't want that. I'm like, you can hire probably another company that would listen to you and do whatever they want to do. I would say those are the toughest type of companies to conduct CRO and a B testing for. Otherwise, if you're willing to listen to people, if you're willing to let your visitors guide your decisions, I think CRO experimentation is just easy to do.

Yeah, I do agree. You touched on the next question when you were explaining like the situation you had like with this previous client and the board. So my next question was maybe you're going to answer it differently or you're going to clarify more. How do you handle situations where a B testing results conflict with stakeholders intuition?

It is a little bit different because the assumption over here is we've actually managed to run the AB test. The results are completely different than the intuition of the stakeholder. Correct. That's a lot different. In the first case that I just mentioned, we've worked with companies where the stakeholder refuses all together to run an A B test. You tell them, hey, this is a version, you know, we're trying to test it. It's a short test. We're just going to get some feedback. Nope, I'm not willing to test it. That's very hard to deal with because in that case you really then if that's your attitude and that's your mindset, you should not be doing conversion optimization. You should not be doing experimentation. You need to think if you know it's all and you know what your customers want, then don't hire a CRO company, don't have a CRO team and don't complain about a low conversion rate on your website. That's just kind of the way it works. Now I like actually the case where an AB test result is different or contradicts the intuition of the stakeholder because those are teaching moments. You come back to the stakeholder, you show them the results and many times lots of the stakeholders I would work with, they would tell you that I'm surprised by this result. Some that have dealt with this and the team dealt with this will tell you, hey, can we run this experiment again? I'm just not convinced. And we would do that, we would run it again to validate the data. Remember, our approach is to validate. Constantly letting validate. Now let's say we validate and again, the results of the A B test contradict the intuition and the direction that the stakeholder needs to go in. In that case, you're done with the experimentation question. You are faced with a business decision question. All we can do as experimentation teams, as conversion rate optimization team is give you the results and help you make a data informed decision. You can choose to go against the AB experiment, you can choose to say, you know what, yes, V1 actually increase our conversion rates by 5%, by 10%. However, I'm going to go with version B and I'm making a business decision knowing that my conversion rate is going to drop by 3%, by 5%. It's fine by me. You're acknowledging that and you're saying, I'm going to go with it. We've dealt with situations where we run an experiment, you have a winner and the stakeholder looks at the winner and they're like, oh, I absolutely hate it. We're like, okay. So we show them the data, we retest and it's the same winner and we go back at it again. Now we're trying to iterate and we run through three, four experiments and we're just constantly doing really better. We've dealt with situations where the stakeholder comes in and says, you know what, I know you've helped me. And I know this increases conversion rates, but I don't like it. I want to change it completely. We've dealt with situations like that. I tell them, hey, it's a business decision. At that point in time, I gave you the data, I showed you what you're going to be losing. Beyond that, it is completely your decision to make.

Yeah, that's true. When you were answering that, I was reminded of this other article that I once read. I think it was from Microsoft. Yeah, it was from Microsoft. They just opened it up about the A B testing flyover. So I just want to read like just a small paragraph. It kind of like ties to what you are saying. So it says if stakeholders disagree about something, it's probably something worth testing. By choosing something over which there is a disagreement, you are likely going to choose something where insight has value.

I love that. Yeah, yeah. When there is disagreements, it's probably worth testing and seeing.

Okay, cool. So the last question that we have is what is your advice for organizations looking to establish a culture of experimentation? How can they encourage teams to embrace testing and data driven decision making?

This is a tough question and I know there are many companies out there, many CRO agencies who say, oh, the culture of experimentation. Getting an organization to adopt and really commit to that culture of experimentation is not very easy. It's not, by the way, a decision that is made at the top and it's not a decision that's made at the bottom. Correct. I worked with companies where the CEO says we're going to be experimenting and the team says, oh yeah, great. So they resist every experiment and they just make life hell for the CRO team, whether, by the way, internally or externally. And I've also worked with companies where like lower level people say we're going to start experimenting, but they never sold it to their top management. And in both cases I think there's failure. Experimentation is not only a decision that you make, but you have to prepare your organization for it. Why are we doing it? What do we expect as a benefit? What are the implications of doing it correctly? Correct. I mentioned ebay, where we've changed even the team structure and now everything that you do, no dev team, you're not going to just be releasing features, you're going to be creating experiments to test those features and you're going to do MVP approach. That's a different way of running business. That's not your typical development cycle that you go through. I do think, and I've learned over the years, yes, I will always have conversations with the stakeholders. But if a stakeholder tells me, okay, yeah, I'm ready to sign and let's go ahead and we want to get you guys on board, I would tell them, no, I need to have a conversation with your team. One of our most recent clients, somebody. Absolutely. I absolutely enjoy working with them. And they've been trying to be a client of us since August. So now November, three months. This is very unusual for us, by the way. Usually somebody's interested if they match the criteria. We have a couple of conversations with them and we move them on. For us to actually delay signing up a company for three months is very unusual and I did that intentionally. So they are a VC backed company and we did an analysis of their website and we saw that there's lots of opportunity. But here's the thing. I'm having a conversation with their CEO and he says, you know what? We are doing $20 million and we want to grow in the next two years to $100 million. To me, that was a red flag. I'm like, okay. And then I was walking him through the process and I felt that, oh, yeah, yeah, okay, that's fine. Did not. He did not pay close attention to the process. When they said, hey, we're ready to sign. Why are you delaying things? I told him, I'm like, I want to just have a very clear conversation with you. You want to go from 20 to 100 million. If you're expecting us to do anything there, that's not going to happen. Yeah, every once in a while you get lucky. But I'm like, $20 million company is a lot different than $100 million company. CRO, if it's done correctly, will help you. But we're not going to quadruple your business in two years. We're not going to quadruple your business in five years by ourselves. This is a full marketing effort and I don't want you to have the wrong expectations. Here's somebody who's very committed to say, no, no, no, I'll do it. I'm like, you need to also have the right expectations. That's number one. Number two, I'm like, let me walk you through the process. And I have to give him credit because I walk him through the process. He's like, I know it's. He just literally says, I know this is. I'm like, okay. He's like, why are you showing me this again? I'm like, I just want you to understand the process. He gives a thumbs up. We still had conversations with the VP of marketing and the VP of products. And they were part of that initial conversations. And I told them the same concerns that I've had. I want to make sure that everybody understands, had conversations with their marketing team just to understand. I'm like, okay, here is where how we're going to handle this project. So we are on the same page and we deliver what you are expecting. That's how you create a culture of experimentation. Now, one way I measure a culture of experimentation. Let's say we worked with a company for a while and then they decided to not work with ENVESP anymore. It happens. I want to see do they continue running experiments or not? That's one question. Because if they say no, we're not going to continue running cells, we fail. We as an INVESP team failed. If we not convince them, if they decide to continue running experiments, I'm like, okay, at least we've got them over that hurdle. What's their approach to experimentation? I remember seven, eight years ago, we worked with one company. They said, oh, after a few months they all we want to do CRO ourselves. So they did it for a while. And then their CEO called me, he's like, hey, could you jump on a call? I just want to let them show you just some concepts. I jump on a call with him, he's showing me some experiment concepts and I'm like, what's the hypothesis? What are you trying to test? He's like, oh, I'm trying to figure out if this is better or that better. I'm like, why are you testing it all together? He's like, oh yeah, I don't know, I just think. And I'm like, okay. So I walked him through the theory behind everything and why he was really testing something. But it was just very silly. But to me, I had a conversation with the team where I told him we failed because all he got with from our experimentation, all the work that we've done with them is oh, is it A or B? And without all the research that goes behind it, that's a failure on our part. Correct. We did not help him see a true value in the culture of experimentation and the research that needs to be done. All we sold him on is let's a B test. Correct. It's settling bets between two people sitting at a bar. No, that's not conversion optimization. I think we do a much better job now in building the case for every experiments and the post test analysis that excites people. Every company that we work with, we'd like to Include everybody in the like in a. Hey, here's the theory behind it. Here's what we've learned. Here's what we've learned about the customers. Some appreciate it and embrace it, some don't. But those are steps toward experimentation, towards a culture of experiment.

Okay, that's good. So as you were also like explaining, I was also thinking of the title of this podcast episode. So I'm thinking to use as a title lessons Learned from running 30,000 plus a B test. I just want us to end this, like, in a different way. I just want to put you like in a spotlight just for a minute.

Yeah, no, I want that. Yeah, go for it.

So I just want you like to give me like five do's when it comes to a B testing. Just say like, do this and then five don'ts. Maybe it can be, don't a B test, like button color.

Oh, my God. This is not easy. Just to put me on the spot like that. Do this, do that. Okay, so let me think about the dos. If somebody wants to run an experiment, let's start with some fundamental things. Number one, make sure that you understand why you are AB testing. So do that, spend some time on that. That's number one. Number two, make sure that you have the numbers to run an experimentation CRO program. This was going to be different from one company to the next. That's number two. Number three, make sure that you do the research that is required to come up with the a B test experiments. Because just merely randomly picking things because you just woke up, had a dream about something, probably not a good idea to run experimentation program. Number four, understand that if you embrace experimentation, there's going to be many ideas that are going to come. You're going to be flooded with ideas. So how you prioritize those ideas is critical initially, probably not very important. Eventually it's going to be very critical to have a good framework. And number five, what did you learn from running an experiment? Always ask that question, what did I learn from running an experiment? Because if you're not learning, yes, it's great that we've increased conversion rates, but what did that experiment teach us about our customers? How does that experiment impact how we do marketing, how we approach our markets? Last five, Can I add even a sixth one? That just came to my mind. Understand that and then do this. You must understand that your experimentation program is going to go through phase, the first phase, probably the first three months. Small experiments. Correct. Crawl before you walk. So we're going to be focused on usability. Issues. Fixing some issues around the website, testing them out. The program is going to mature now. This is going to mature between 4 months to 12 months is one type of program. After 12 months it's going to be different. You want to have the right mix of experiments between experiments that are UX focused, small experiments, iterative experiments, experiments that are maybe substantial. Oh, we're testing like new concepts around the websites. I always give the example of social proof and different ways to display social proof and I just did a case study on that recently. That's more substantial versus experiments that are completely disruptive. Changes the way that you do business. A good example of that is like, hey, let's test different pricing for our products. Those are rough experiments and you need to have the right mix between those experiments. Not all of your experiments need to be UX or simple experiments. Not all your experiments should be disruptive. It's the right mix. Now with the right mix, that's going to be based on the program and how mature it is and based on the company and how mature it is. So that's six different points in terms of do's for the A B testing program. What did you want?

Don't don'ts. Five don'ts.

Like I just reversed do's. I just reversed the do's. I was like, don't do that. No, let's do five don'ts. Don't expect conversion optimization program would deliver the earth and the moon. So have the right expectations whenever you are starting a conversion rate optimization experimentation program. Number two, don't run experiments if you don't have the numbers. Don't run experiments for six months, for 12 months. Don't run multiple experiments if your site traffic does not support those experiments. You can do conversion optimization without having AB testing. A B testing is a portion of conversion rate optimization. It's not conversion rate optimization. So don't confuse the two. Number three, don't spend a lot of time designing an experiment. You need to spend a lot of time researching an experiment. But you don't have to get an experiment to the perfect pixel perfect state before you launch it. The goal from an experiment is just to validate the concept. Are my visitors, are they really even interested in this or not? If they're not, then we're gonna move on. Number four, and people fall into this a lot. Don't just run an experiment and tell, oh, we've hit significance. Oh, let's stop the experiment. This is not the way you do statistical analysis when it comes to A B testing. Now you don't have to have a master's degree in statistics, but at the same time you need to understand the basics. So we don't run experiments until we hit significance? No, we start by calculating the sample size and then the sample size. Then we'll see. Hey, did we hit the MDE that we expect? Number five. Don't expect that your typical developer who had done development for you for many years is going to be necessarily a good a B test developer. They are two different creatures. Your typical developer versus the a B test developer. Two different creatures. They think differently, they handle things differently. There are few people, few and select who are good at both regular development and ab test development. Lots of time. They are completely different. It's almost like hiring a designer who did catalogs on paper, mail catalogs, and expecting them to do absolutely amazing designs for the web. Two different things. Or somebody who'd done graphics and that's really. They're absolutely amazing graphics. And you tell them, oh, come here, build me this website. That's not their forte. Same thing when it comes to a B test development versus regular development. That's your five don'ts. How about that?

They're good. I've got length, something to use for those YouTube shots and TikTok. So that's good.

Oh, okay. You got me there.

Yeah.

Awesome. Thank you, Simba. And we shall continue next week. Thank you everyone. And until next time, happy testing.

Sure. Bye.