#44 – A/B Testing Velocity – Is It That Important?

False positives happen when the a B testing stopper tells you that you have a winner, but in reality you do not have a winner. The question is how do you balance testing velocity with the risk of false positives? I think they are two completely separate domains. My name is Karl Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, this is a very, very toug questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts. So we are live again another week, CRO Live Hour. By the way, Simba, since you never told me how many episodes of CRO Live Hour we've done, I went back to Riverside. That's where we record our podcast slash, the weekly live LinkedIn Ciara live hour. And I counted and we have 44 episodes there. We've recorded probably at least twice as much because we only halfway through, we switched to Riverside August 3, 2023. Simba, how's your week?

It's been a good week. Going strong. I never thought that it will grind for this long, to be honest. I'm sure, like, you also didn't imagine my kids getting this far, right?

Yeah, I mean, oh gosh, like, you know, we're going almost a year and nine months. It's exciting. I enjoyed the conversation. So it's a way for us to talk C R O and at the same time catch up away from our weekly marketing catch up. So before we jump in, you just attended the session with Patty Galloway. Yeah, the brains behind lots of the successful YouTube channels. Give us some insights, man. Because I could not attend the meeting. It's funny, I registered for it and then I'm like, can somebody attend this meeting? Because I'm too Busy. I have other meetings scheduled.

It was kind of like a full house of 130 people attending. Some of the questions that were asked, they kind of like relate to the questions that I had going in. So that was like a good thing. I was taking notes. It was like the some of the insights that were being said during the meeting. And one of the things that he spoke about was having multiple niches. And we said, like, it's kind of like a problem. You should just have like one niche because you can't share different audiences. For us, like, if we're focusing on CRO, we should just next stick to CRO because we have to be known for one thing. And the other thing, what he also spoke about that we also don't usually do is scripting. Maybe it's also because of the mutual follower videos. You were saying there's no editing that can save a bad script.

I do scripting. I didn't. Oh, my God. Like, I can pull up a whole bunch of scripts over here. Okay. CRO live hour maybe is not very scripted, but all the other videos I record, oh man, without the script, I'm lost.

Yeah.

So he was saying that there is.

No good editing that can save a bad script. You should script like all of your videos have a plan. One of the things that you also mentioned is like each and every video, especially like in the introduction, you should have like a book. So one of the things that I also remember from taking Dave Gerd's course, the laws of copywriting and also like emphasized on hooks when it comes to copywriting. Well, it seems like to be also the same issue when it comes to YouTube videos. You also have to have like all kinds of hook. So it's one of those things that I also have to take time to think about. He also mentioned that one of the important things is storytelling. People are drawn to stories. Like, that's also like one of them laws of copywriting. Like, if you can tell like a good story, people are drawn to that in one way or the other. I think we also have to rethink the way that we do like our videos. We have like, videos which are insightful when it comes to conversion optimization and other things to do with digital marketing. But I think we have to work on our storytelling a bit. It's something that we can improve on. Not to say that it's bad, but I think, like, there's also room for improvement. One of the things that he also mentioned was that someone asked him about what is the most important thing is it the editing? Is it the scripting? Is it the thumbnail? Is it the title? And then you say like the most important thing when it comes to YouTube is the topic idea that you have. It's never the editing. They gave like an example of someone going to mourn and then like here's the camera and the camera falls off. People are going to watch that video where they stop edited very well or why? Because like they will be interested in the idea of someone being unknown. I think those are some of the things that were interesting that he spoke about.

It is fascinating. And as I think about that and I think about our YouTube channel, which has been growing and we're only at 1500 subscribers. But it's good because it was stuck at 500 subscribers forever and now we focused on it and we managed to grow it by a thousand subscribers. And every month there's more and more people. One of the metrics that we track is kind of the subscribers. We're constantly learning what applies in one channel, sometimes applies another channel. So in general, copywriting. Yeah, the same rules apply. Correct. But a little different. Because think about email marketing. Many guys do quite a bit of that. The biggest thing, your subject line. Because if somebody doesn't open the email, doesn't matter what offer you have inside. So same thing with YouTube thumbnail and the title also kind of the contents of the email or the content of the video and how you deliver that. It's the same fundamental problem, but kind of in a different medium. So fascinating.

One of the questions that also was interesting was how many times should you upload on YouTube? Some people were saying once a month, some people were saying once a week. All that he had to say was that the YouTube algorithm doesn't credit the number of times up not. But it looks at like the idea of the content. Do people really like the videos that you put out?

Yeah.

So basically that's what you said.

Yeah, he knows YouTube a lot more than I do. But I'm going to come at it from a different perspective, especially kind of at an early stage. I think sometimes you have to put a lot. If you're coming up with like one bad idea a month and then another and whatnot, it's going to take you forever to learn that versus, hey, produce a lot more. Sometimes volume, especially in the initial stages, kind of helps you figure out here's what makes user click, here's what works. And then based on that, you can start going more specific. Especially when you're trying to figure out a platform just before we Jump in. And I'm sure people are not interested in this, but, you know, we also enjoy this conversation. Are you guys ready for the Premier League to come back in what, 10 days? Are you ready?

I'm super ready. Like, been waiting for you. We've brought in a lot of good players, so hopefully we'll perform better than we did last time.

Sounds good. I'm not going to comment. We shall see. In like 11 days, either Simba is going to be a man or I'm going to be a happy man. One of us has to be happy, you know, both of us happy. It's a pun because we are like, no support competing teams. So what do you have for us for CRO Live hour today?

So today we'll be discussing about a B testing velocity. So we have about five questions. So I'll just make sure the first question if you're ready. Sure.

How about it? Let's see.

The first question we have is, how do you balance a B test B with the risk of false positives or false negatives in results? But before you answer that, can you also explain to those who don't really know what a B testing velocity is?

Yeah. So AB testing velocity is a measure of how many experiments you are launching in a certain period. Usually we'll look at it in a month time, or sometimes you can even go down to how many A B experiments you are launching in a week. I was talking to somebody and they're responsible for experimentation at Sephora, and they launch about 80 experiments per month. That's some heavy experimentation that's going on there. And they attribute anywhere between 15 to 20% of the company revenue back to the experimentation program. And they have some very sophisticated measures to do that. And in addition, kind of the software also helps them do that. So velocity measures, okay, how many experiments am I launching? So that's one way. And since we're explaining, we're also. Let's talk about false positives. That's when you run an experiment. And let's take a simple example. I'm running an experiment on a homepage. I have a new design for the homepage that I think is going to just break the bank and make us a ton of money. You run the experiments, you're splitting the traffic between the original design, the current design that you have, and the new design, the A B testing software, after you run the experiment for certain period of time, comes back to you and tells you, hey, this new design is actually a design that helped you increase conversions. So you're happy you're excited. False positives happen when the A B testing software tells you that you have a winner, but in reality you do not have a winner. Even worse, false positives happen when you think you have a winner, but really you have a design that actually causes a drop in conversions. That's absolutely horrible because the whole idea of AB testing is to find winners, to learn what you want to find winners. Now the question is, how do you balance testing velocity with the risk of false positives? I think they are two completely separate domains. You could be running a single experiment per month, you could be running a single experiment per quarter or even a single experiment per year, and you might end up with false positives. That has nothing to do. So the testing speed or testing velocity does not relate to having false positives. The issue of false positives, though, is an important issue in a B experimentation. A B testing, you have to pay close attention too, because false positives threaten the whole program. Especially if a CEO or a CMO or VP of marketing brings you on. They have hired you or they've hired a CRO agency. They're looking for results. And if you end up with false positives, it's a lot worse, by the way, than running a program. And you end up with losers like, okay, well, yeah, this idea was not good, but we've done the research. False positive a little bit scarier because there's this high hope and high expectation that the test showed you. And then you take that winner, you deploy it on the website, and it's like, hey, our conversion rate dropped. Actually, what happened over here, it does a lot more to discrediting the program and the A B testing experimentation and the A B test that you are running now. How do you avoid false positives? Because again, A B testing in the web is nothing new. We've been doing as humans A B testing now for a long time. And I always mention every drug that you take goes through a process of AB testing where we have a control sample, correct. Okay, here are people, we're going to give them a placebo, and here's other people that we're going to give them the drug and that we compare the data and whatnot. And usually in the US the FDA go through a very rigorous process, requires every pharmaceutical company that's about to bring a new drug to the market, it asks them to go through this rigorous process to make sure that we don't have false positives. Because if we're putting a drug out, we want to make sure that's actually going to help people. So how do you ensure that there are certain things in statistics that help you really design your experiment and your a B test that you don't end up with false positives? Number one is you want to calculate even. Even before you start the experiment, you want to calculate the sample size that is required to run the experiment. Based on the different variations that I have, based on the data that I know about the conversion rate for the original state of my website. Here's how many visitors I'm getting. Here is what I expect this experiment to generate in terms of lift or improvements in conversions. It will tell you that you need to run this experiment until you hit 30,000 visitors. 500,000 visitors, 100,000 visitors. Now, one of the biggest mistakes that people do is they run the experiment, they did not calculate the sample size, and they keep on running it until the A B testing software tells them, hey, you've achieved significance. Oh, you have a winning variation. No, no, no, you don't do that. What you do is you say, great, thank you. A B testing software that you're telling me that my variation is actually helping conversions. But because I've calculated the sample size, I'm going to continue running the A B test until I hit the sample size. One of the biggest complaints that PhD professors complain about their PhD students is the PhD students are going to come with thesis. They say, hey, we believe X, Y, Z is going to achieve something. And then the professor is going to tell the student, they're like, well, go ahead and do some research. So what the PhD student does is they will go out and they'll start collecting data from people because they did not calculate the sample size. I'm going to first ask this question, maybe to a hundred people, and see, you ask it to a hundred people and the data does not match your thesis. So you say, you know what? I'm going to ask 20 more people, oh, guess what? Now the data is changing. Oh, I'm gonna ask maybe 50 more people. Now the data actually matches my thesis. So you're not starting with a clean slate. You're just manipulating the data to get to a point of view. Same thing could happen with AB testing. That's the reason we calculate the sample size before we run an experiment that will help you avoid false positives to a great degree. Number two, you need to look at the statistical method and any type of a B testing software that you are using. There's two different schools when it comes to calculating statistical confidence. There's what we'll call frequentist statistics and Bayesian statistics debate around them has been around now for over a hundred years. Let's not dwell really deep. Bayesian statistics ensures to a higher level that you do not have false positives. But remember, even with Bayesian statistics, if you're not calculating this apple positive, going to end up with that. So it's kind of multiple things that you have to do. Start with calculating the sample size, choose the right A B testing software, and remember that at the end you ran the experiment, you have a 95% chance that the winner is an actual winner. There is still 5% chance that the data that you've got was due to randomness. We ran the test V1 variation one had helped us increase conversion rate. With 95% confidence or with 95% chance to beat the original. We're saying 95, not 100% correct. That means that there is a 5% chance that this win was due just to some random elements that had nothing to do with your experiment. So there's always a slight chance of the experiment generating a false positive. When you have that framework, it doesn't matter whether you're launching a single experiment or 80 experiments, just like the case of Sephora in a month, you have to go through that framework that's going to help you reduce false positives. But I will add one other thing that we do. We launch an experiment, we do the sample size calculation, we run it. You know, we're using Bayesian statistics, we have a winner many times. What we do is we come back and we say, oh, this experiment had three challenges, three variations to the control and v1 is the winner, v2, v3 lost. Then we take the winner and we take the control and we run them against each other in what we call a head to head match. Let's revalidate the results, let's make sure that we actually have a winner because there was this 5% chance of having some random results. Because we're aiming for 95% confidence. Let's run the experiment again. So multiple layers or safeguards to ensure that we avoid false positives.

That's a good one. I've been thinking, like as you are explaining it, do you think a high testing velocity can be used as a metric or maybe like a yardstick to measure the maturity of an experimentation program?

Again, two separate creatures. In all honesty, I remember talking to one startup and they're launching about 30 experiments per month. So I was impressed. I'm like, wow, 30 experiments per month, that's impressive. And it was their CMO and she's like, hey, can you talk to the CEO? Because we want to validate the testing program. Now, when you're doing 30 experiments per month, I'm like, oh, wow, you're watching an experiment per day. But I know there's a lot more thoughts that goes into creating experiments. So I'm thinking that they're a large team, because to launch an experiment per day takes quite a bit of thinking. And then I'm talking to them. I'm like, well, how big is the team? And they're like, oh, it's only five, six of us. And I'm like, hold up. How are you coming up with the same ideas? How are you implementing them? How long does it take to do all that? They're like, oh, no, we just decide every week that we're going to launch a whole bunch of other experiments. And I'm like, okay, so you have a small team. I have a red flag there by the fact that you're deciding every week. Oh, next week we're going to test 1, 2, 3, 4, 5. That shows me that's a program that is more spaghetti on the wall than a mature program. And then I ask them like, hey, so what kind of side traffic you have? They're like, oh, we have about, like, you know, 3,000 visitors and maybe 10 conversions a month. And I'm like, hold up, hold up, Paul. Let's really come to a full stop and a halt over here. I'm like, why are you even running experimentation A B testing? They're like, what do you mean? It's a way of doing business. I'm like, yeah, for the right business, you don't even have enough convergence. I'm like, how long does it take you to conclude experiments? They're like, well, no, they never conclude. I'm like, hold on. So you're launching 30 experiments per week, Spend very little time thinking through the experimentation and how you come up with the ideas. You never concluded them. But I'm like, so how do you decide? They're like, oh, we just build more experiments. And I'm like, okay, what you have over here is not the mature a B testing program. When you have what you have over here is a hot mess. Two different creatures. I would say the signs of having a mature program is that you need to crawl before you walk, before you run. I don't start with a company with really high velocity, but my aim within two to six months is to get to a higher velocity. If a company is doing one experiment, you know, per quarter, my Goal for the first month is to do maybe two experiments per month and then go to four experiments per month. I might just stay at four experiments per month. Because really the process around pre test analysis, how do you come up with ideas and post this analysis, how do you analyze the results is taxing. It takes a lot of effort and you will not have a mature program without spending most of your time at those two stages. The actual AB test implementation is not that complicated, but it's the thinking that needs to happen before and after that takes a while. And that's how I judge a certain level of maturity. Correct. Of the program. There are other levels that really are factors or indicators that help you judge the majority of a program. How much buy in we have from other people besides the ones who are running the program. How does the AB testing program help us define the different features that we're going to release on our website? Because lots of times a B testing happens in a corner. Oh, we have a win. But the development team is building features, releasing them, kind of two different universes. It's like, okay, so how helpful is that? I was talking to a VP of marketing. It's one of the companies that's been with us now for a couple of years. Then she was talking to another sister company of theirs and she was telling him, you know, one of the biggest benefits that we've had with our AB testing program with invesp is the fact that we use them to decide on which features we're going to invest development time in. Now we don't ask more developers to work on anything until we run an experiment, and we run experiments as an mvp and then we say, you know what, this is interesting. Okay, we're getting enough signals that we should really invest more time in developing this feature. Sometimes we might run an experiment and they get a ton of traffic. You're talking about tens of thousands of conversions every month. So we reach statistical significance really quickly. But for them, sometimes the win is like, okay, well it's like one and a half percent. And they'll say, you know what, for the amount of effort and time and deployment, because they have a very complex side, it's not worth it for them. Now you talk to them about, hey, this experiment generated a 12% lift. Guess what? They're talking to their VP of Development right away to say, we need to work on this. This is going to impact our bottom line directly.

I need to do like to ask you something and I need you to either say yes or no. Sure, it depends. Yeah, because Like, I have a follow up to this. So do you think a B testing velocity is important? Yes or no?

Yes.

I had to ask you this because, like, I remember one of the articles that I read and they were going against the idea of having like a high testing velocity. So I just want to read like just two sentences of what they said and then you let me know what you think about it. They are talking about the problem of having a high testing velocity. So they say if you decide to test many ideas quickly, you are sacrificing your ability to really validate and leverage an idea. 1. Winning a B test may mean quick conversion, rate, lift, but it doesn't mean you have explored the full potential of that idea.

No, that's about poor implementation of high velocity. What did I tell you? Thank God I said that. It is all about the pre test and the post test. You have to do that. Velocity in itself is not a goal. But velocity is a sign of a mature program. It's not the other way around. A mature program will have high velocity, but high velocity does not mean that you have a mature program. It's all about the thinking portion of the AB test. Before I launch the experiment, after I launched the experiment, there's a lot of work that needs to happen there. To think that, well, I'm just going to do AB testing just for the sake of AB testing really, really fast. No, don't do that. That's just, oh, we're just going to do random things and you might as well just sit still, do the thinking. And you have to do the thinking. And that's the hard part of experimentation. The assumption that because I'm going fast, I'm not doing enough analysis. No, that's a fallacy. I guess I was gonna say that's a false positive, you know. No, you have to do the thinking. And the beauty of this approach is when you have to start with a roadmap to know where you're heading with your testing program. And we'll talk about this maybe in a little bit later in this episode, but you need to know the general direction. However, a mature testing program will also always look at the insights that you've gathered from the experiments. And then you'll say, hey, we found this experiment. Do we pivot? Do we iterate which direction we need to go? That is a mature testing program. You add that on top of testing velocity, you have a winner.

Yeah, that's like, that's. Some people argue that when they be testing lightly to test fatigue among users affecting their overall Experience on the website. How do you strike a balance between experimentation and user satisfaction?

Listen, user satisfaction ultimately is the goal of the whole experimentation. More satisfied users, higher conversion rates, higher customer retention. It's as simple as that. You can trick few users for a little bit of the time. You're not going to trick everybody all the time. And you cannot sacrifice user satisfaction just because of AB testing. One of the fundamental premises of a B testing, I am trying to find a better user experience. I'm trying to measure user satisfaction through AB testing. Yes, conversion rate is one indicator now, a lot of tests and user fatigue. No, I'll put that aside. I mean, you know, everybody's on Facebook, everybody's on Amazon, everybody's on Google, everybody's. The issue is not the number of tests that you're running. I mean, Sephora is a perfect example. 80 experiments per month. You know, some of those other companies are launching thousands of experiments a year. It's not user fatigue. Again, it's confusing. Two different things. You know, the issue there is when you're trying to trick people to improve conversions. You're using AB testing to trick people and you say, hey, this improved our conversions, but it killed user satisfaction. You lose credibility. You see it, by the way, on lots of the travel sites. Correct. I'm not gonna mention anybody, but you're tricking people. Three hotels left or three rooms left, people eventually caught on and you have a major lawsuit in the EU against what? Mentioned against, I think it was booking.com because of all these trickery that they were doing. But that's not an issue with AB testing. That's the tool. Correct. A B testing is a tool. There is a fundamental wrong business practice that's happening. It's getting implemented through AB testing. Yes. AB testing tells you that, hey, by tricking people and showing them that there is scarcity getting more people to convert. But after a while, people learn and people come back to your website using incognito, the famous, oh, you're getting special offer for 10 minutes and there's a countdown of that 10 minutes and you forget about that website, you come back to it an hour later, you refresh and then countdown restarts again and it's like, okay, so you're just really tricking me to convert. That's the problem. It's not the AB testing portion of the program.

Okay, this one is an interesting one. How do you ensure a B testing aligns with the long term vision of the business?

Good question for us. Every quarter we're having a conversation with the Companies that we work with and we say, hey, what are your KPIs that you are responsible for this quarter? What are you trying to achieve? Now some companies are very well structured and they have very clear KPIs, OKRs, whatever you might want to call it. Some companies are like, I want to grow. So let's talk about the structured companies where we're like, okay, what are you trying to achieve and how does my testing program align with those KPIs? Because some of the experiments that I do need to clearly link up with the KPIs and business growth goals or business goals that you are trying to achieve. Otherwise again, the AB testing is done in one corner and the business heading in a different direction. So that's number one. And I think this has to happen. This is one of the ways that you get buy in from everybody. Hey, one of our goals was to increase average order value. Okay? So we've tested all the, we've run all these experiments on our product pages, on our collection pages, on the cart and the checkout. And we're able to increase average order value by X percent. So I've helped you if that's is one of your KPIs. Correct. Bigger things. Sometimes we think that here is the direction that the business is heading towards. Here is the vision of what we want to achieve. Great, let me use AB testing to validate that with my user base. This is a more interesting approach. Sometimes we know that, okay, here's the direction that the product management team, product managers want to take the product into. So now we're developing experiments and A B tests around this vision and we're validating and taking the insights from those AB experiments to say, okay, well maybe like, you know, this feature is really important to people. This feature is not very important. We're using the data to inform the direction that the business is heading towards. And sometimes in those cases you run an experiment. Okay, so this is the direction that you want to head in. Let's start with an MVP approach. We're going to run an experiment and we come back and we say, hey, by the way, people are absolutely hating this. It's fine. Sometimes you say, you know what, yeah, I know people absolutely hate this, but this is the direction that we want to head towards. But we are making that decision based on data, knowing that this is going to cause a drop in conversions because there is a bigger goal strategy that the business is trying to achieve. And all that experimentation is going to do is a tool for you to make data Informed decisions. That's, I think, how you align business objectives with a B testing.

What if maybe the business objectives are kind of not maybe online? I have a business and I want to improve food traffic to one of my location stores. Can I also use, like, maybe experimentation or CRO to do that?

You can. It requires a lot more thinking. One of the interesting stories that we've had, we signed up this client. They've been around since the 1920s. They have stores all across the Northeast, apparel stores. They had just started the E Commerce division. They're doing about close, I think, to a billion dollars in sales. Store sales, retail store sales. And E Commerce was about $20 million. So this is like 2%. Correct. Of what the brick and mortar is doing. So we come in and we're pushing and we're doing a lot with our own customer retention. And we worked together for the first year and we saw E Commerce grow from about 20 million to close 75 million. That's almost 300% growth. Million problems that came with that. Correct. Count logistics and the support because tripling the business. Lots of times people say, oh, this is great. I'm like, yeah, yeah. What you need to think about all the logistics that have to go with that. And it wasn't only the work that we did, but experimentation was fundamental to that. We ran into a completely different problem that we did not think about. Where the store managers came and complained. He said, they're like, hey, your online stores, you're the online E commerce division. It's taken away business from us offline. I just sat there and I'm like, hold on. So I did my job really well, and now you're complaining about that. That was an interesting discussion, but that discussion happened because we were not thinking. We were just focused on the KPIs for the E Commerce division. We're not thinking about the brick and mortar. So we said, we don't want to get anybody upset. Let's sit together and let's figure out how the campaigns that we're launching and the experimentation that we're doing online can help you offline now it gets a little bit challenging because you're using coupons, you're trying to figure out omnichannel setup. Correct. For somebody who visited the store, looked at certain items, how do we actually bring that experience offline? Some of it worked really well. Some of it was horrendous because people felt. And this happened many years ago, some people felt, how on earth do you know that? I was looking at this particular skirt and now when I'm walking to the store, you're identifying which items I want. That did not work all too well. I do think with hindsight, yeah, there are some privacy issues over there that we're not paying close attention to, but coupons, you know, and like, you know, store locators and how do you identify, you know, those visitors and how do you link the data of people buying offline to online? All of that can really work together and must work together for the success of the business.

For most companies, A B testing typically focuses on optimizing existing elements on a website. So how do you approach the exploration of entirely new ideas and innovative concepts through Tesla?

I love that question. Maybe we'll finish with this one because it's a really good question. First starting out and this might be the first couple months, it might be the first six months, it varies from one company to the next. I think your goal is to fix the funnel leaks that you have in a website, the usability issues that you have on a website. And usually it is amazing to me because you would think that even with larger companies that have dedicated marketing teams and merchandising teams and E commerce teams, there are still issues where people struggle on websites. So your first phase is just to get as many of those low hanging fruits, quick wins out of the way. Let's just get those out of the way. That doesn't mean that you finished all the UX issues, but no, you finished the easy fixes. And then you want to look at your testing program and you want to break it into percentages. To me, 50% of my experimentation, 50% of my a B tests are still focused on usability and user experience. On my website, 25% is based on the experiments that I've ran. The tests that I ran and they're informing, it's like, hey, this was interesting. Maybe we need to iterate. Oh, we need to pivot, we need to change. Oh, we've learned something from here. We need to change something else. So that's another 25%. The final 25% is what I call disruptive ideas. Something really new that we are testing now. Disruptive ideas, let's say you're doing four experiments a month. So one of those experiments is going to be disruptive. What's fascinating is that your 50% of the program that's focused on user experience and usability and UX and whatnot will have more wins, by the way. If you know what you're doing and you have enough experience, you'll have wins. But the impact on conversion is not necessarily high. You might have a win 4%, 5%, you know, 8%. The disruptive ideas, which are that 25% of your testing program will have lower win rates. One out of those four or five experiments actually won. But when they win, the increase in conversions can be really significant. Now, I was just looking at an experiment and it took us a while to confirm that with a large company. And it's fascinating because we still need to collect data. So we're about halfway through the sample. We're running the tests globally across the website. Changing really the way users interact with the sites. Major change. And there was lots of resistance to it halfway through the test. At this point, we are a 70% lift in conversions. I have never seen 70%. I haven't seen 70% in the last 15 years. In all honesty, I'm usually celebrating, you know, 10%, 15%. I'm like, oh, we've done well. I was looking at the data and to the point that I've asked the team like go back and check and validate. That's where actually capturing right data. That's correct. And they're like, yeah, you know, the data in FIGPI is correct. The data is getting sent to GA4. We validate, we place orders, we check the cookies. Before we celebrate, we need to make sure that we have the right data. And I assure you this is a test that's going to run a second time. Even after we finish and we have a winner, we're like, hey, let's run it again, head to head match just to make sure that we are seeing an actual lift. And what's fascinating about this test, because it was a disruptive idea, it's running across the website. Every visitor who comes to the website is in that a B test. It's a lot different than an experiment that's running on product pages or checkout because that experiment runs only for people who get to the product page. In this case, no, we're talking about every single visitor that comes to the site. And that's, I guess about 1.5, 1.6 million visitors a month. So I'm excited about the disruptive idea, but I'm like, okay, yes. And by the way, this company's been with us for eight months. This is probably the sixth disruptive idea. The previous fight.

Oh well.

And this one was kind of the biggest one. Lots of resistance and it's paying off. So we're on a call and we'll just finish with this. We're on a call with their VP of marketing and she was super excited and she was the one who was pushing back the most on the idea. So we shall see. You should ask me about this in a couple of weeks to see what are the final results for it.

Yeah, sure, I'll certainly do that. I'm not sure like if you still have 10.

No, I think we have more meetings, man. Tuesday is my meeting free day. Thursday is my everybody will meet with holiday. So I have so many meetings. I'm like, no, I get stressed out thinking about all the meetings but maybe we'll continue with the questions next week as well.

Sure.

Awesome. Well thank you everybody who's listening. The one thing that we ask you is to share this episode with somebody who you think will be interested in conversion optimization, interested in E commerce and until next time, happy testing. Thank you Simba.

Ciao.