#27 – A/B Testing Goals & Questions To Ask Before Choosing An A/B Testing

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today wherever you listen to your podcasts. Simba, how are you doing, man?

I'm good.

This is CRO Live hour, so if you are watching us, this is your first time or if you've watched us before, make sure that you comment, tell us where you are listening us and watching us from. And let's talk CRO experimentation. Starting next week we will then broadcast on YouTube as well. So we'll do both. And I was telling Simba, sort of funny. So I usually wear glasses for those of you guys have seen my pictures or listened to the LinkedIn Live before and I've decided to wear contact lenses. So my age, I have one eye is for reading, one eye is for seeing far away. That's very confusing to my brain because my brain is trying to sort out which eye to use at which distance. So I'm struggling to read. Sometimes I'm like, well, this is very confusing. Just to add a little bit more excitement. So away from Man United and Liverpool, how was your week? What's the latest and greatest in the land of investment, the land of fake buy? How's marketing going?

I think marketing has been going very well because both for investment, for figpi. Because just before this recording I was in a like meeting with you and other team members and we're looking at some of the core metrics for both INVESP and figpi. I think maybe it's just as we're seeing like a drop in the past few weeks, we're seeing like a drop in traffic. Our blog, it gets Like, a ton of visitors for the website, but we're seeing, like, kind of like a drop. But we are beginning to pick up slowly. And hopefully, like in the next few weeks or days, we think we'll start, like, going up in terms of marketing for both, I think we're doing well. We're trying some channels that we have never tried before. Some are working, some are not working. We're going to all keep on experimenting and seeing what works.

It's always fascinating to me, kind of which channels, because I've had some channels where they were absolutely horrendous for investment. Now they're steam. We've had some channels that worked. And I'm like, okay, not doing so well. And it's funny because some companies think of marketing as set it and forget it, you're done. And it's never like you set it, you optimize, you evaluate, set, optimize, evaluate, and so on and so forth. And I'm actually excited about a couple of things. We've hired an agency to actually help us. We are B2B and we are B2B for fairly larger companies. I mean, the smallest company we work with is probably doing, I would say, 5 to $10 million in annual revenue and grows up from there. So is Facebook a good channel for us? Don't know. Here's a tidbit that I'm going to tell you. Simba, one of our competitors did run Facebook ads previously and it was not successful for them. And the way I judge it, because they've killed the ad after they ran them for six, maybe 12 months. So is that telling about the industry? I don't know. We will see how it works for us, but we probably also have other goals that we want to. By running the Facebook ads.

Yeah, I think some of the channels that used to work maybe before COVID they're kind of like different from the channels that do work now because I know, like some of the companies, they used to run a lot of Fogo ads campaign, but right now, some of them, they're like, kind of like focusing on video specifically, maybe TikTok. I think it also maybe depends with the objectives of the company or. Or maybe with the type of the company, but I see, like a lot of people they're now turning into video.

Interesting. Yeah. It'll be interesting to see how this plays out. So I'll keep asking you every week, but guess what? Okay, so next week we're not going to have Ciro Live hour because I'm actually going to take a vacation for the first time, very long time. So I'm going to be off for a week. So it's horrible to say, but I'm very anxious. I know the team is capable, but you know, to be away for almost a week, like, oh man, that just feels strange. It feels good. It feels strange. So we'll see how I feel.

Will you be off, off, off, or maybe you'll be just like creeping in. I can't like really take off, off, off. I try sometimes to stay away from my laptop but then I just like, oh, how's that campaign doing? How's that website doing? How's, you know, like, I just like to check and see stuff.

I mean, the theory is I'm going to be completely off. But you know, when you launch an A B test, they always tell you don't look at the data, don't look at the results. And I tell them I'm a peaker man, I look at the results. So I think I'm still going to be on but hopefully, unless I see a crisis, I think things can wait for a week. So we shall see.

Yeah, sure. So let's get to this week's questions. So, yeah, this week we have two questions about A B testing. Okay, so the first question is how do you choose goals that matter to your A B testing?

How do you choose calls that matter to your A B testing? So that's a good question. Typically there are three things that you want to keep in mind whenever you are setting up an A B test. In terms of goals, those are what we call macro conversion goals, micro conversion goals, and then revenue goals. So you want to keep those three things in mind. But before you start choosing the goals for, for an A B experiment, you want to take a look at your website and understand how many conversions the site is generating. Let's give a scenario. Let's say that Your site generates 400 conversions a month. Well, in that case, whenever you are running an A B test, you understand that when you are setting up a macro conversion goal which is an order confirmation if you are an E commerce or subscription, if you are a SaaS, things are gonna be slow. It might take you anywhere between three to four to six weeks to run experiment based on the page that you are testing, that's a lot different. If your site is generating on the one extreme, let's say 3,4000 goal conversions per month or even more because then you can feel free and you know that you can run experiments and you can probably reach significance in a very short time period. Anywhere from two weeks to three weeks. A lot different if your site is only generating let's say 50 conversions a month or a hundred conversions a month. In that case, you know that if your primary goal for an experiment is a macro conversion, somebody actually converting on the website by placing an order or subscribing or filling out a lead generation form, it's a very slow process. And in those cases, which is a small segment when you should run AB testing when you don't have enough conversions, micro conversions become really important. What's the micro conversion? It's a step in the process. So let's give an example. Let's say we are running an experiment on a product page for an E commerce website. A macro conversion is somebody actually placing the order with the website on the add to cart, going to the cart page, clicking on the start the checkout, going through the checkout and placing an order that is micro conversion. A micro conversion for somebody who is on the product page is merely clicking on the add to cart. Correct some micro step. It's a small step because a person, in order for them to submit that final big yes, they need to say small yeses, small yeses as they are going through the process. Yes, this is the right product for me. I'm going to click on the add to cart. Oh yes, the price in the cart looks fine, the shipping hand looks fine. I'm going to click on the start the checkout. All of these are micro conversion points. Typically when we set up an A B experiment, we will have a mix between macro conversions and micro conversions. The companies that we work with for the most part have enough conversions. So we always look at order confirmation. The place where somebody submits the final order or if we're working with a SaaS company, they submit the subscribe. Now for us that is the primary goal that we evaluate the experiment. But in addition to the macro conversion, we will ton of micro conversions because we want to see for the different variations that we have that are running against the control. We want to see, hey, did any of them actually increase or perform better? When it comes to micro conversions, it's important to remember that particular variation. Performing well on a micro conversion doesn't mean necessarily that it's going to perform better on a macro conversion. Let me give you an example. Maybe I am a site where shipping charges are. I was talking to a company little earlier. They ship lockers and then they sell large lockers where the shipping cost is very high. And if you don't show the cost of shipping on the product pages. Guess what? You're testing different designs. One design shows the actual cost, the other design doesn't show it. Well, the design that doesn't show it is going to do better because you're not shocking people with this, oh, it's going to cost $500 to ship this item. So naturally people are clicking a lot more on the add to cart. However, when people get to the cart page and now we're showing them the actual cost, guess what happens to the variation that didn't show the price people end up abandoning. So the overall conversion rate for the variations is different than what a micro conversion is. So primary evaluation usually is based on upon the macro conversion, the primary conversion goal for the sides. But we do sub evaluation and it's important that you try and understand the story and the rationale of why particular variation performs better when it comes to micro conversions. But it doesn't do all too well when it comes to macro conversions. Makes you think about really interesting questions. Doing the analysis there and whenever you're doing the analysis you're trying to decipher in mind is like, hey, so why is it that visitors are performing this way? Sometimes it's very obvious, the example of the locker that I gave and sometimes it's not so obvious and it's like, hmm, I am actually not sure. So by the way, question time. So if you are setting up your AB experiments, are you setting up multiple goals? How many goals are you set up? Would love to see that in the comments. One important thing to keep in mind is try and limit the number of goals that you set. We used to have a conversion rate optimization specialist who worked with us who loved setting up goals. So for every experiment she might set up 20 different goals. In addition to the order confirmation she will go, and it's like, oh, I want to look at the clicks on the add to cart and I want to look at the interaction over here and I want to look at like, you know, do they start the checkout and I want to see the flow for the different steps of the checkout. The problem is when you set up that many points almost as if you did not set up anything because it becomes overwhelming to evaluate all these different micro conversion points. So I've always told her, yes, it's good to have multiple goals, but don't go overboard with it because at some point you'll get stuck in analysis paralysis and you wouldn't know what to do. So I like to limit the number of goals to anywhere between four to five goals, including the primary conversion goal.

For the AB experiments, that's a good detailed answer. So you like to limit to about 45 goals maybe for someone who's listening or who's trying to learn more about a B testing for the first time. Right. Which are some of the goals which people should track when they are running an a B testing? You mentioned like maybe a primary goal for E commerce is order confirmation. Right. Like that's the primary, that's the micro. What are some other engagement metrics that you think they should maybe try to track?

So the metrics that you track will depend on the page. Look at the page and say we understand the primary conversion goal. It's an order confirmation, it's a subscription, it's filling out a lead generation form. But what are the different steps for the visitor that the visitor must go through in order for me to get to that big yes on a homepage? The primary conversion maybe is to go to a particular to the collections page and then from the collections page to go to the product page on from product pages that clicks on the cart. Now let's say we're running an experiment on the home. Would I set up clicks on that cart on the product pages as a micro conversion? Probably not, because that is way down the funnel. But I might set up a collection page view as a goal, a product page view as a goal and a CART page view as a goal and an order confirmation as a goal. Let's take another page. Let's say we are running an experiment on a collection page. So again my macro conversion is the order confirmation. But I have other micro conversion goals that might be clicking and viewing the product. Correct. That's important to me. Maybe that's a question mark. I will also count the clicks on the add to cart as another micro conversion goal. But maybe even and it depends on the type of experiments that you are running on a collection page, you might even choose to look at the interaction that people have with the different elements on the page. Maybe I'm introducing an experiment where I highlight the filtration, highlight the sorting on the collections page. That's my goal. So I'll definitely be measuring those. Am I getting a lot more interaction or am I seeing less interaction? Let's take again, we'll go back to the collection. Maybe you're adding a quick add functionality. Correct. Quick shop functionality where instead of somebody navigating all the way to the product pages, I wanna see by adding this quick shop functionality am I getting a lot more clicks on cart. So I will actually be Looking at that as a micro conversion goal and seeing the final impact in terms of.

Micro convergence as well as I was listening to what she was saying, I think I picked out like two mistakes that people make when it comes to AB testing goals. The first one is maybe they add in a lot of goals and then they end up being in a analysis paralysis as you have mentioned. And the other mistake that I picked is they might maybe set up goals which are not relevant to the page that they are maybe launching a test on. Right. Are there any other mistakes that you think people should look out for maybe in terms of when they are setting up their A B testing goals? Maybe something to do with goals also on Google Analytics or something like that.

It's sort of interesting because every time we set up an experiment, we actually sent the data of that experiment to whatever analytics platform that the site that we're working on is using. So if it's Google Analytics, we are sending the experiment data to Google Analytics and then we're looking at a ton more goals. But you just have to be careful and understand that whatever analytics platform that you have is probably have a different way of measuring these goals to your A B testing platform. So just, you know. So for example, if it's Google Analytics, if it's amplitude, if it's Heap, if it's Omniture, for example, the data can look different. So just kind of be fully aware of that's number one. Number two, when it comes to goals, you need to make sure that sometimes whenever you're setting up goals, one of the things that we do during the first, I would say four to eight hours, making sure that those goals and their setup is getting so you're actually recording conversions for those goals. Now in theory, the A B testing platform that you have has a way for you to your goals so you don't have to do it with every experiment. So I'll just mention, for example, with figpi, we have a way so you can define the clicks on the add to carts, the clicks from the cart that clicks on the collection page to go to the product page. You set that up once so you're not setting it up with every experiment. But if your A B testing platform does not support that every time you are setting up those goals, you might run into issues. So make sure that you are collecting the data. And then finally, sometimes your original the control doesn't have some of those goals in it. So for example, let's say we're adding a quick shop functionality altogether. The collection page doesn't have the Quick Shop functionality. So naturally, because it doesn't have it, it's going to report zero for that goal. Your variations naturally will win. Correct. You just added a new design to the variations that are challenging the original. So you're not comparing apples to oranges. You say, oh well, the control has zero clicks. Of course it has zero clicks because that functionality does not exist altogether. And that's the reason I'm evaluating maybe on product page views. I'm evaluating on the order confirmation as well. You can't just set it and forget it. When it comes to goals, you have to be very thoughtful in what goals to select, how to evaluate them, how to track them, and then how to make the final decision whether you have a winner or not.

Yeah, I know that some of the A B testing tools out there, they kind of limit the number of goals that you can track. Pay experiment, Pay a B test. Do you think it's actually like a good thing or it's not? I don't know. What do you think about that?

An A B testing tool, limiting the number of goals. The only reason that they're doing it, not because it's the best practice, but it's additional data points that they track. It's a way for them to have an upsell.

Correct.

I'm going to limit the number of goals that you have to five. Well, yeah, I understand you're limiting it up to five because if I pay more, I'm going to be able to track unlimited number of goals. That's number one. Which means you're not doing it because of statistics or you're trying to make my life easier. You're trying to get more money. I understand it's a way to do an upsell. I know a couple of platforms, for example, that introduce free a B testing plan, but they only allow you three or four goals. Well, most AB experiments might have probably more goals to track than that. So I don't like, I don't like the limits. It's a sales approach more it's a sales driven approach as opposed to, oh well, here's it's a best practice and that's the reason we are trying to avoid that. So I understand it. I don't like it. I like to set up as many goals as I can and I like to make the conscious decision, oh yeah, in this experiment I'm going to have. In this experiment I might have one or two. I probably will never have one. Always have multiple unless it's a final step in convergence. I like to make that decision myself.

Yeah, yeah, yeah. It kind of like makes sense that way. So this question I think will take us maybe to the next question. Okay, so the next question is, what are the questions you must ask before choosing an A B testing tool?

What are the questions that you must ask before choosing an A B testing tool? So let me see, I have 1, 2, 3, 4, 5, 6. Here are six questions that you must ask before choosing an AB testing tool. Number one is how does the tool match the requirements that you in terms of business? Number two, how does the AB testing tool impact your website performance over time? Number three, what statistical model does the AB testing tool and use to determine a winner in an experiment? Number four, what are the core features that a B testing platform or tool offer and do they fulfill my requirements? Number five, what are the additional features that I get as a result of using AB testing tool? And number six, what is the price? Those are the six questions I always think about when it comes to choosing an A B testing tool. So let's jump right in. Number one is how does the A B testing tool match my specific requirements? You must start with understanding your requirements. And when it comes to understanding requirements, you need to think about, well, what type of AB testing am I going to be doing? Am I going to be one of those companies that are going to run multivariate testing or not? Am I one of those companies that are going to be running multivariate testing? Does the A B testing platform support or not tell you? With the 25,000 A B experiments that we've ran and we probably even exceeded that number at this point, I just need to update it. We've stopped running multivariate testing since 2011. So it's been almost 11 years of doing experimentation, A B testing for companies as large as ebay and Target, Special Olympics or Reilly to smaller companies. We don't do multivariate testing anymore. And if you are interested to learn why we don't do multivariate testing, leave a comment below and I will answer that. Another question that you want to ask. Another requirement that you want to think about is whether you have a web app and whether you are going to be running a B testing on the web app. If you have a web app and you want to run AB testing on it, that limits your options from the get go. Not every AB testing platform supports AB testing on web apps. So that's something that you want to think about because that really is going to limit the market and it's going to really drive you to three or four different options. Now you might choose to use an A B testing platform for your site and a different A B testing platform for your web app. That creates some problems. What I've seen in my experience, most companies, I dare say maybe 99% of the companies, even those who have a web app, when they start A B testing, at least for the first year, sometimes for the first two, three years, they are very focused on optimizing their site. They don't focus on doing a testing for their web app. There are exceptions to that. The ebays of the world are focused both on optimizing their site as well as doing AB testing on their apps. So start by your requirements and understand if the features that the A B testing platform supports all your requirements or not. And if it doesn't, then you need to make the decision how important those requirements are for you to selecting that A B testing platform or not. Number two, you want to think about the impact of the A B testing platform on your site performance and page load time. Now, when companies are first starting to do AB testing, they don't think a lot about the impact that an A B testing platform will have on their site. Assure you that within the first 30 days to 60 days, they start thinking about a lot of time and it becomes a huge issue for them. Why does that happen? Remember, whenever you are doing an A B test, your site must load first. Then the A B testing software script must load and then that A B testing script must decide do we keep the original page that the visitor had just viewed or do we serve one of the other variations? That time lag between the page loading the A B testing platform, script loading and then the decision to serve a particular variation to the visitor causes what's known in the industry as the flicker effect. Flicker effect is when you see the original page appearing first and then it flickers and a variation appears instead. The time that it takes for the A B testing platform to load and script to load will make the flicker effect look much worse in some instances, or not noticeable in other instances. I have seen some of actually I'll say with figpi, since we work with figpi, that is a huge concern for us. FIGPI takes less than 180 milliseconds to load and for returning visitors, visitors who viewed the test and then they came back, FakePy takes less than 40 milliseconds to load. Some of the other A B testing platforms out there, some of the enterprise A B testing platforms out there take anywhere between 700 milliseconds up to 1.2 seconds. That is 1200 milliseconds to load. There is a major delay in how long these platforms take to load a particular variation. So you have a very heavy flicker. What is the impact of that flicker on your results? It's up for discussion. There will always be a flicker, but I want to minimize it. It's just life whenever you are doing A B testing. But I also don't want heavy flicker. One last thing that you want to remember. The heavier your page is, the larger what we call the dom, the heavier the page is, the longer it's going to take for that page to load. And then the A B testing platform will have to load. We've worked with companies where their servers are really slow. The page is extremely slow. They are using images that are not optimized. So naturally the original page has to load first and it might take a while. And then right away there's a quick flicker. But that is not the problem of the A B testing platform. That is more of a problem of the actual side that you are working on. Number three, investigate the statistical model that the A B testing form uses to determine a winner. There are two different schools when it comes to statistics. Frequentious statistics or Bayesian or Bayesian statistics. We're not going to get into them. There are pluses and minuses to each model. The model that most modern AB testing platform use is some variation of the Bayesian model. The benefit of using Bayesian statistics is it limits to a good extent the chance of having a false positive. What is a false positive? It is when an A B testing platform tells you that you have a winner. You think you have a winner, but in reality you. You do not have a winner. So you think, oh, variation one increased our conversion rate, while in reality variation one did not have a meaningful impact on your conversion rate. Even worse, variation one actually reduced your conversion rate. So we don't like false positives when it comes to experimentation. There is a reason why most A B testing platforms are moving to Bayesian statistics. The challenge is some of those points are really not very clear on the model that they use. So they use some variation of it and then they use vague terminology about the model that they use. I don't like that. So if the statistical model matters to you, then make sure that you ask questions around that. Number four, evaluate and ask about the core features that the A B testing platform has and make sure that it matches your expectations. Some of the core features that I think about is the ability to target multiple pages on the website. So I want to run a test on all of my collection pages. I want to run a test on my product pages. I want to have concurrent tests run with what we call exclusion groups. So if somebody. If I'm running a test on the collection page, I'm running a test on the product page. If somebody is in one swim lane, if somebody saw the test on the collection page, I want to automatically exclude them from being included in the test that is running on the PDP or the homepage or the cart page. So having the ability to run multiple experiments at the same time and being able to exclude visitors from one experiment and include them in another experiment is something very powerful and something if you are serious about conversion optimization you will run into and you will need think about scheduling. I want to have the ability to schedule my AB experiments. I've already had to run in a specific time to stop at a specific time. This is really important. Let's say it's Thursday afternoon. What time is it now? Thursday, 12:30. We've QA'd an experiment. It's all said and done. We have a rule. We don't launch experiments on Fridays. We don't launch experiments on weekends. So guess what? I am gonna schedule this experiment to kick off 8am or 7am Monday morning. So I wanna be able to schedule that as opposed to having to wake up at 7am and launch the experiment Then being able to set up multiple goals, I expect that. I wanna understand if there's any limitations We've talked about some of the A B testing platforms limit the number of goals that you set up. Am I going to be able to integrate with whatever analytics form that I'm using? Am I going to be able to integrate with Google Analytics Amplitude heap? You mentioned that integration. Am I going to be able to segment the data? Great. You're showing me that. I've ran an experiment and here's how the control is performing, here's how my challengers or variations are performing. But I want to segment that. I want to see how those variations are performing when it comes to organic traffic, paid traffic, social media, because there might be some insights there. Another core feature that is important with the A B testing platform. Show me how my different variations are performing. When it comes to bounce rates, revenue is very critical to evaluating. I'm not making a decision based on the revenue a particular variation is creating, but I'd like to see it's a second guiding metric. I'd like to see how my variation impacted the size revenue per visitor in terms of the average order number five is what are the other features that the A B testing platform is offering me that I didn't even think about? And when it comes to that, I am evaluating features such as heat maps, session recording on site polling if that is something I'm interested in. I'm a big fan of using one platform where I have my consistent data no impact on my site performance in a single place. I don't like running multiple oh, I want for session recording I'm going to use this platform and for heatmaps I'm going to use different platform and for personalization I'm going to use a third platform and for AB testing I'm going to use a fourth platform. That is a recipe for disaster and it's a headache for any marketing team that is handling all of that. Again, this will depend on your own requirements because there are companies who will choose different platforms and there is philosophy behind that. There are some companies who say I'd like to get single JavaScript, drop it on the website and done. The sixth question I want to understand pricing and I want to understand the content length that the A B2 Sim platform requires me to when it comes to pricing, you want to take into account the visitor growth and the seasonality that you might have because your visitors might you might be on an upward trend in terms of increasing visitors. How much does it cost me to run the platform? 100,000 visitors? Usually I like to look at a million visitors. Is it going to cost me $1000 a month? Is it going to be $4000 a month? And what do I get for the 1000 or the 4000 that I'm spending? And what is the commitment? Do I have to commit to an annual contract? 12 months contract? So I'm stuck actually. And I'm committing to paying close. If you're paying $4,000 over a 12 month period, that's a $50,000 subscription or am I paying $1,000 on monthly basis? Both models are out there. There's different platforms that require an annual commitment and they are pricey. They are right for some sites. But those are the six different questions that I want to think about before I select an A B testing platform.

One of the points that you mentioned, I think it was the first point where you said like if people want to learn more, they should maybe comment. But for our podcast listeners, in case they want to know, why did you stop multivariate testing 11 years ago?

That's a good question. Why did we stop multivariate testing. 11 years ago. We're one of the first companies to jump on multivariate testing. We absolutely loved it. You set up, you have four or five different elements on the page. You set up four different variations for each one of those elements, and you end up with 100 variations without really thinking about it. And if the site has the traffic, it's great. Now, 11 years ago, the only statistical model that everybody uses, what we call frequentist model, which meant that we're running the test, we think that we have a winner. Sometimes we thought 20 different winners, but those 20 winners were all false positives. So that was one challenge. The second challenge, which is interesting, more important than running the actual experiment, is the hypothesis behind the experiment. I am testing this because I have a hypothesis on how visitors will interact with the page that is great. Have a winner one out of 100. And now you need to ask yourself a question. Why did this particular variation, with all these possible changes actually 1. Why did it be the other 99 variations? And what you find with multivariate testing, you're really, you don't know how to build on experiments. So we found ourselves very excited about the wins. But we also found that it really killed our effort to think about the next experiments. We could not do deep analysis and understanding of, okay, why is this a winner? Even more important, why did the other 99 variations lose? What is it that we did wrong? I mean, there's 99 of them, sometimes there is 200 of them. So that's the reason we start avoiding multivariate testing. Instead, we choose to do AB testing, very focused. We might have anywhere between two to four challengers to the control where we're able to say, here is the theory behind each one of those challengers. All of those challengers are inspired by one hypothesis, but different ways to implement this hypothesis. And now I can do meaningful analysis and I can figure out my next steps after this one experiment, because this experiment is in a chain of other experiments. So doing the analysis is a lot easier when you're doing straight up AB testing as opposed to multivariate testing.

That makes a lot of sense. That makes a lot of sense. It kind of like reminds me of one of the stories that you usually tell about maybe launching a test which had, like, more than close to 100 variations on them. Was it VWO, where you had to call, like, the support team to help VWO?

Man, it was O'Reilly OReilly, our friends at O'Reilly OReilly, the book publisher, and this is early on. I mean, we're talking about 12, 13 years ago. So the tools were still. I mean, VWO was really just taking off and we're excited. Okay, so we're going to test and we'll do multivariate testing, multiple elements, multiple changes. VWO was not designed or something like that. And we had a very smart development team. But the VWO was breaking constantly. It just could not handle that many variations. You can see preview and at the end they were just starting out. So they really loved us because we're bringing them this large accounts and we're bringing them all these large accounts. But the tool was breaking. So I'm like, guys, we're struggling using VWO on these large accounts. So they went in and set up the experiment with us. And I still remember an email that I got from them. It's like we set it up, don't touch it, don't do anything. If you want to make any changes, we will go in and make those changes for you. And that is the experiment. I think that experiment had about 26 different winners nowadays. I dare say probably they were all false positives. We thought we had 26 winners. We were super excited, but we really didn't understand statistics back then. So nowadays I look back at some of the stuff that we've done 15, 16 years ago and I'm embarrassed by it. But I guess that also shows that we've made a lot of progress in the last 16 years of her musp's life.

So does it make sense to say that the more the number of variations, the higher the chance of false positives? Right?

Yes. So it depends on the statistical model that you are using. But if you are using frequentist statistics and there is an actual formula that takes into account the number of variations that you have, and there is an aim for 10 statistics that escapes me. But at five variations that you're introducing and if you're using frequentious statistics, your chance of having a false positive is near 50%. That's scary. That's before even doing any type of testing. Just by introducing more variations, there's a chance of a false positive. So when you use bias on statistics, you avoid that altogether because bias on uses completely different statistical model. But in addition to the false positive. Correct. You want to think about the post this analysis that you are going to do. I mean, I know I've written an article about the multiple variation and the impact on false positive. I don't know if I've ever published it or not because it had more statistical formulas in it. It's probably worth researching our site to see if we published it. And if we didn't publish it, the article is ready. Because if I remember it was like, you know, 10, 15 pages by the way. Just kind of explaining and showing lots of examples of this. Might as well check it and see because we might have another article to publish.

Yeah, yeah, yeah, yeah, yeah, I'll check it out. I think those are the questions that I had for you for today. Unless maybe you had any other information to just.

I think that's it. I'm excited. By the way, I just came back from. I'd been speaking at Popcon maybe for near 15 years and I've spoken at Popcon where me and two other speakers and there's two attendees. So three speakers. Want two attendees. I've done with conversion optimization was not. My session at pubcon was not only jam, there were people standing to that point. The interest in conversion rate optimization, the problem that I see, I love the interest that people are like saying, oh CRO, how do you do that? The problem is very similar to SEO. There's a lot of not very correct information out there. People have incorrect notions about how to improve a website conversion rate. I still get questions that are too basic. Oh, how about the color of the button? And I'm like, okay, well, I mean, I'm not saying the color of the button is not gonna impact you, but you probably wanna think deeper on how to actually improve a site conversion rate. That's one experiment. I wanna run 100 experiments per year, if not even 200 experiments per year. Okay, I'll give you the color of the button as one of them. As silly as that is. How about your other 199 experiments? Why don't we think about that? Of the people who attended my session, 50% are running a B experiments, which is fascinating. On average, I think they were doing two experiments per month. I actually polled the audience just to see there is one gentleman who was running close to 20 experiments per month now. 20 experiments per month. So great effort, but not the right sight. If you're going to run that many experiments, you need to have the conversions because you need to be able to segment your data and it will make sense to run that many experiments.

That also makes a lot of sense.

Awesome. Simba. So next week I'm off no CRO live hour next week. So we'll come back the week after that. Thank you everyone for watching and we shall come back to you. Until next time, be experimenting.

Sure. Bye. Sa.