#22 – CRO Challenges Faced By Most E-commerce Brands

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts. Happy New Year, Simba. How was your new year?

I'm kind of being, like, optimistic that this is going to be a good year for invesp. Looking at some of the changes that we have made in the past few weeks, trying to focus on some of the things that we're not focusing on. For this podcast, I'm not sure, should I call it like a podcast?

Yeah, it is a podcast. It is CRO Live Hour. It is a TV YouTube show.

How was your holiday?

I got sick. Oh, man. It's funny because I took the COVID vaccine booster. I don't remember what. And I took the flu shot. I felt good after them, and I'm like, you know what? I'm good. I'm inoculated. It's all good. And of course, two weeks later, New Year's Eve, I was sick like a dog. Oh, my God, it was horrible. So I had like three, four days where temperature and stomach and it was just an unpleasant scene. But here I am kicking and screaming. My kids had time off from the 20th of December. Today they are fully back at school. So I'm a happy man, you know. Yet another reason to be happy. I'll be in the middle of a call with the company that we work with, and they're playing a soccer match next to me. I'm like, guys, really in the middle of my call. They're like, yeah, where do we play? I'm like, I don't know. Not my problem. But definitely not near my office just go away.

I've never experienced that yet. I'm just like guessing maybe that's a good problem to have or something like that.

It's a blessing. It's a blessing.

We have like a bunch of e commerce questions around CRO.

We can take one at a time, our typical style.

What are the most common challenges that e commerce businesses face when it comes to CRO?

Before I jump right in, I just want to mention this quickly. This is Khalid and Simba. We are with invesp. This podcast is about helping e commerce companies improve their site conversion rates. Now we do so by sharing the best tips, strategies, techniques on improving conversion rates. On making money online by recording these weekly videos and sharing them. So if somebody is new here, make sure that you subscribe, you follow, you like There are five main challenges that an e commerce company usually faces whenever they start thinking about CS CRO. Whenever they start implementing CRO, their numbers. Whether they have the right numbers or not to do CRO in the first place. A shift and a change in the mindset. Because CRO does require a change in the mindset. And we'll talk about that. Identifying what to test because there is this thought that yeah, you should test everything, which is a recipe for disaster. The speed of testing. How fast are you actually testing things? And number five is the analysis. Okay, so we've done some CRO, we've done some A B testing. What next? Those are the five things and we'll drill through them. So the very first one is the numbers. Not every e commerce website can do CRO. We would love for every e commerce website to do CRO. But meaningful CRO, and by meaningful CRO, I mean CRO that is rewarding. CRO that actually generates results for the e commerce company requires you to have the right numbers. Lots of times people talk about, well, how many visitors are coming to your website. That does not really matter. What matters is the number of conversions that your website is generating in a month. What I typically tell companies is that if you have less than 500 conversions per month, you can do CRO. But it's going to be slow and it's going to be a little painful. So you have to really understand that it's going to take you a while to get results and analyze results from any type of CRO work that you are doing. People sometimes find that strange. Well, ultimately the goal from doing any type of CRO work is analyze the website, figure out problem areas on the website, create designs to fix those problems, and then you don't just deploy those designs on the website, you A B test them, you have 10,000 visitors coming to your website. You're going to split them 5000 view the old design, 5000 view the new design, and then we compare the numbers. Why does the number of conversions matter? Because the number of conversions will tell you how fast those A B tests come to conclusion after you launch them. At a minimum, you need 500 conversions per month. Less than that, it is going to be slow for you to see those A B tests conclude. Now, between 500 to 1,000, you can definitely dip your toes into the sea of CRO. More than 1000, you are starting to see some really good testing above 2000 conversions per month. That's where magic happens. If you have more than 2,000 conversions per month, if you have 3,000 conversions per month, you should be able to consistently deploy a B tests, get results really quickly, and improve your site conversion rate really quickly. So that is number one.

Before you jump to the second one, I just want you to clarify something. When you talk about the numbers, most of the time when I hear people talking about the numbers that are needed to launch a CRO program, they are talking like in terms of traffic, they don't usually like, refer to conversions. What if my website is getting like, let's just say 25,000 visits per month, but my conversions are way less than 500, so does it mean that I can't do conversion rate optimization again?

Whenever you're launching an A B test, you are launching, in scientific terms, what we call a controlled experiment. You have the original site, the original design, and the new design that are challenging each other. It's almost like, you know, I have the original sites and now I came up with almost like a medicine to fix it. Well, in order for me to judge the quality of the medicine, guess what? I have to see how many people, if it's in real life, how many people are actually getting better by using this medicine. It doesn't matter how many people. I'm just offering the medicine to. What matters is like, you know, how many people are actually impacted by that. That's the reason the number of conversions is really what matters. If somebody talks about the number of visitors, then they probably have not done meaningful conversion optimization. They probably don't even understand statistics. One of the things that I do whenever we're interviewing somebody for a CRO position, one of the very first questions that I like to ask them is to say, so I have a website and I want to decide if I'm going to Run AB testing on it or not. I want to do conversion optimization on it. What do you look for? And one of the very first things I expect them to talk to me about is the number of conversions. If they don't mention that, then I know that I'm interviewing somebody junior or somebody who doesn't have experience working with conversion optimization.

That makes a lot of sense. You can move on to number two.

Number two, the mindset. Typical e commerce website and how e commerce platforms are built. It's really designed with the mindset of, okay, let's take a page, let's create the perfect design that we can find and then code that design and then put it on our website, then move on to the next page. Let's figure out this new feature that we want to release, let's come up with the best design for that new feature, put it on our website site, and then move on to the next feature. It's a mindset that is in constant hunt of perfection. It is a completely different mindset when it comes to conversion optimization. Because conversion optimization, you come to the website and you say, you know what? I need to identify problem areas on the website. I've identified the problem areas on the website. Next thing, I will take the first problem area. I will take a scientific guess, I don't know if you can put those two words together or not on the best way to fix that problem. And after I come up with that best way, I'm going to ask my users to judge the quality of this new design. I don't have to get this new design that I came up with to perfection. There is a 90% chance that people will reject it. They will absolutely hate it. They will tell me, although this new medicine that you just provided to us, this new design was supposed to solve our problem, is really not good for us. That's the reason we ab test. If I know that 90% of the designs that I'm creating are designs which people are going to hate, guess what? I don't have to get those designs to a perfect state. I'm just testing them out. And people, my visitors, those who are coming to my website, are going to give me enough signals so I would know whether this design is good or not. So now we shift from let's get this design perfect into what is the theory behind this design. Let's get it out there as fast as we can and let's test it out. Many teams, design development management owners, struggle with that concept because they want to get everything perfect. Well, guess what? The Time that you spend in creating a design has no relation to the fact whether that design is going to help you increase conversions or not. You can spend literally four or five months perfecting a design, making sure that it looks really good. And then when you test it out, you find out that people actually absolutely hate it. So instead of spending four months, why don't we spend maybe a week, maybe we should spend four days instead of spending a lot of time. And if people like it, guess what? We go back and we, okay, now that we know people like this design that we put out, we're not 100% satisfied with it. But guess what? People like it. Now I'm going to improve it. So it's highly iterative process and that is number two.

But I also have like something that I would also want you to clarify here. When you say the design doesn't have to be 100% perfect, of course, like there's no design that's 100% perfect. Don't you think that maybe some people might push back and say if it's not like 100% perfect, that means that it's going to kind of like affect the results because you are pushing something that you know that is not perfect.

I love that question. So I'm going to answer that question by an actual case study from Netflix. The first time Netflix launched an A B test, it took them two and a half months to create that first a B test. They spent weeks analyzing the data, spent weeks working with their designers to come up with the best design. And the co founder of Netflix was excited about this. They launch it and he's like, really? We spent two and a half months and people absolutely hate it. Do this again, they go a second time. This time they spend two months, again, it loses. So now he's a very scientific minded guy and he's like, you know what? Okay, so four months or four and a half months, two designs and people absolutely hate them. How can we actually speed up this process? Because it seems like the quality of our designs have nothing to do whether people like them or not. So they cut down the time to one month. They're like, okay, we're going to spend a month now we're not going to spend two months perfecting the design. They launch it, it loses. They go at it a fourth time, again, the design loses. So now he's like, okay, this does not make sense. Let's cut down the time it takes us to create a design into a week. And then they start saying, you know what? Yeah, we're still losing, so why don't we cut it down? They get to the point where they are creating a design in one day. And sometimes he said, you know, they were doing silly things because they said, you know what? How perfect a design is has nothing to do with whether it's going to win or lose. Now they started doing silly things. So sometimes they're testing things like, you know, images with copyright statements on them. They put out banners with spelling mistakes on them. And you can argue that, hey, you're doing something that is actually going to cause your test to lose. But they've tested both cases and they saw that there is based on data, there was no significant results. What I find very insightful is if you have a good idea, if you have a good concept, if you have a concept that is going to make the user experience on your website better, people will send you enough signals to tell you you are on the right path. And that's when they say, you know what? Okay, now we have enough signals from people and we need to go back and iterate on this design. Remember that with every experiment, you're not tracking one metric. You're tracking many, many metrics. You will track the conversions. Okay, so design A versus design B, how many conversions are we getting? But you're tracking a ton of other metrics. How people are clicking, where are they clicking, how far are they scrolling, where are they navigating to? So there's lots of metrics, lots of data points that will help guide you to make that final decision.

That's a good explanation. It also created a lot of questions in my, in my head. Yeah, I'm just going to ask like this last one and then you move on to the next point. If you say like, the quality of the design doesn't kind of like determine the result that will come out. What matters the most is the user experience. But in a way, doesn't like, the quality of the design also influence the user experience? Because of course, like, if the design is bad, that means the user experience is also going to be. Not going to.

Is it?

I know what you're going to say because you're probably going to use maybe websites like Craig List, websites like Wikipedia that do have like bad designs but are doing well.

Yeah, we have to separate user experience from design. They are two completely different things. People lots of times think about user experience as the beautiful design. No, user experience is about making the life of the user easy. It's allowing them to perform the task they came to perform on your website in an easy manner with very little cognitive load. That's what's a good user experience. You might have the most beautiful design, but then I try and perform a task because that's what I'm trying to do on your website. I'm trying to perform something that is so complicated, I can't see the elements. I don't know where to click. It's a beautifully designed website, but the user experience absolutely stinks. Separate the two. What matters when it comes to conversion more than design is messaging and copy. We're visual creatures. That's what humans are. So, you know, we can judge designs. I always tell any designer, I feel bad for you because everybody thinks that they can judge design. Just like clothes. Correct. You look at somebody and everybody has an opinion because we all wear clothes. So everybody has an opinion about clothes. Actual usable website allows users to perform tasks that they want to perform really easily. And then it's beautifully designed.

I think now I'm satisfied with the answer. You can move on to the next point.

Okay, so we talked about the numbers. We talked about the mindset. Number three is knowing what to test. Okay? So I am convinced. I am going to go ahead and test. I believe in a B testing. I want to improve my site conversion rate. What should I test? And you have this mindset that people go through from, okay, we're not doing any testing into this complete shift in the mindset. Let's test everything. Okay, well, yeah, if you have unlimited time, unlimited budget, test everything. But the reality of it is most of us operate in a world where we have limited time, limited budget. So identifying what to test is very critical. There are different methods that every marketer, every company should use to identify what to test on a website. Those methods include looking at data. So doing some data research, looking at analytics, looking at heat maps, looking at session recordings, doing user research, whether it's one on one customer interviews or doing on site surveys or on site polling, competitive analysis. Don't copy their ideas, but get inspired, maybe steal some of their ideas. Say, you know what, this is a really interesting concept. I wonder how my visitors will like it. But I'm gonna maybe modify a couple things in it. We go back to the point of saying the mindset. I don't have to copy it 100%. I'm testing a smaller concept and people will send me enough signals, whether they like it or not. Coming up with those ideas is very critical. And I think that's one of the areas E commerce companies struggle with. So we wanna make sure that we Identify those areas, Use the right methods to identify all possible areas that we want to test. And then we need to use some sort of a framework to really score these ideas. To say, this is number one, this is number two, this actually goes to the end of the line. So there should be some sort of a method. And there's plenty of frameworks in the industry that people should use. But you should not just randomly wake up one day and say, well, I'm going to test this. And then the next day I'll say, I'm going to test that. I was talking to a small startup out of Spain and they're like, we've been doing a B testing and it's great. And we're launching 30 A B experiences experiments per month. And I'm like, wow, I mean, that's impressive. 30 AB experiments per month. Well, how are you guys coming up with the ideas? They're like, oh, anybody can test anything. The CTO believes in testing. And that's kind of the mindset. We've been a B testing longer than that company has existed. Most of the guys who are working at where still in high school when we first started, even, maybe even not even high school when we started AB testing. You want to know what's even worse than that? Is when I look at this traffic going back to number one, I'm like, well, how many monthly visitors are you guys getting for like, you know, 30 AB experiments per month. That's impressive. How many? They're getting about 10,000 visitors with about 50 conversions. I'm like, hold up, 50 conversions per month and you're doing 30 A B experiments? I'm like, how long are these experiments taking? They're like, well, yeah, some of those experiments have been running for like, you know, like eight and nine months. And we're just waiting for results. And I'm like, and what do you do? They're like, oh, we just pause them and we start other experiments because is we want to experiment. And I'm like, okay, now you're testing just for the sake of testing. And really you're wasting your time. Number three is knowing what to test, coming up with a solid process to identifying areas that you want to test.

So you say like, you shouldn't test everything, but if you have like the resources, okay, maybe you can, maybe one of our listeners or one of the people who listen to the podcast, they would want to know exactly. Like on an E commerce site, what are the five things that you recommend people not to test out? Just like, without mentioning the colors of the button because, like, everyone talks about it. But what are the five things that you think that people might be wasting their time when they're testing those things?

I don't know if I can think of five things, but let's give it a try. Testing pages with low traffic. Basically, a page that has 100 visitors per month and has no impact on your conversion rate. You are probably wasting your time testing things that people don't notice that you are testing in the first place. So running all tests, I remember that one company said, hey, we're struggling with this one a B experiment. Can you come and take a look at it? And I'm like, yeah, just send me the two URL's to the original and the variation. Let me take a look at them. They send me the URLs, I click on them and I'm like, I couldn't really tell the difference. So I emailed the guy back and I said, hey, I can't really see the test. Like, no, no. He's like, maybe. Like, the test is really implemented really well. So it's coming really fast. It's there. I go back, I'm looking at the two pages and I have two monitors. So I'm looking at this monitor, looking at this monitor and comparing the two pages couldn't really the difference. And I'm like, hey, can we just get on a zoom call? So we get on a zoom call. And I'm like, dude. I'm like, looks the same. He's like, no. Notice this small thing. I'm like, are you serious? This is what you're testing? It was so small. People wouldn't. I was looking for the test and I couldn't notice it. I'm like, do you really expect your visitors will notice this test? And he's like, yeah. And I'm like, well, that's the problem. Your tests need to be very obvious. So you're really testing things to see. Okay, well, yeah, we're testing this change, and this change needs to be obvious to judge whether this change is actually going to have an impact on your bottom line or not. Number three that I see lots of people do is instead of testing top of the page areas, they go to things that are bottom of the page. Why do they do that? Not look at their heat map? They did not look at their scroll maps to see, yes, this page is getting 100,000 visitors, but only 5% of those are actually scrolling all the way to the area that you are testing. So you're not testing 100,000 visitors. I'm sorry, you're actually running the test for 5,000 visitors. You are wasting your time. Your most critical areas are that we have on the top. One of the pages that I'm weary of testing on e commerce website is the homepage. People always find it very strange when I say that if you have the right SEO and if you have the right paid campaigns, whether it's PPC or paid social, the homepage is not a very critical page on your website. Your traffic should be coming to many pages across the website and you discover when you look at your analytics that your homepage might be getting less than 1% of the traffic. That's what the numbers are telling you. But politically, which page does everybody look at? Everybody looks at the homepage. The CEO looks at the homepage, the VP of marketing looks at the homepage, everybody in the company. And every time you make a change to the homepage, battles happen and like, you know, people get enraged and I tell people, I'm like, I don't want to go through that battle. I'll leave the homepage for the political battles and I'm going to move on to more bottom of the funnel pages. Those are the ones that matter to me. Because if I can improve the conversion rate for your product pages, for your cart page, for your checkout, I can right away improve your revenue. If I improve the conversion rates for the homepage, I'm only improving the conversion conversion rate for 1% of the traffic. It does not really matter, especially if you're a large e commerce website. I'm going to mention number five, which might be a little bit controversial. The order confirmation. Oh man, people love to mention oh people is like the gold mine. Like, you know, the order confirmation. Now if you are an E commerce company that is built on upsells, Correct. It's like, okay, we're going to upsell them after they bought. So we have an initial offer that is small and then I'm going to try and convert you. I understand that you're testing the order company confirmation. There is a design to the order confirmation page and there are certain things that you want to do on the order confirmation. But people who converted, they already converted. You already got them to convert. And by the way, testing the order confirmation has its own set of complexities. Don't focus on that. If I'm running a testing program for a company, I dare say probably in the first 12 months I might not run a single test on the order confirmation. I'm not saying it's not important. But I have bigger fish to fry.

Yeah, that was a good list. I literally have no questions for this one.

You did put me on the spot. I'm like, oh, man. Can I think of like.

Yeah, that also also goes on to show that you've been doing this for a long time.

So how many did we mention until now? We mentioned the numbers. We mentioned the mindset. We mentioned what to test. Let's jump into number four, which is the speed of testing. So now you have the numbers, you have the mindset, you figure out what to test, and then you start testing. And I talk to companies and they're like, yes, we are trying to improve our site conversion rates. We are running a B test. Well, how many a B tests are you guys running? Well, one or two a B experiments per quarter. Quarter. And I just pause. I'm like, one or two per quarter. I'm like, why is it so slow? And you get a plethora of answers to why and it doesn't matter. Correct. Because from one company to the next, there are excuses. But the reality of it, you are not serious about improving your site conversion rate. If you are only running one or two experiments per quarter, that is too low in order for you to see meaningful results. Let's run the numbers. Let's say you are running two experiments per quarter. So in a year, you're doing eight experiments. Let's say that you're absolutely amazing and you're coming up with ideas that actually have an impact on your bottom line. So let's say that 50% of your ideas, 50% are actually successful. Now, this is actually a crazy success metric. Most companies don't even have 10, 12%. It's just easier math. Four of your testing ideas improve your conversion rates. Okay, this is great. Now compare that to a company that is, let's say, simple company, doing four experiments per month. Month. So in a quarter, four multiplied by three, that is 12 or four. 12. That's 48 experiments. Correct. Per year. We'll apply the same 50% success rate. You have 24 successful ideas versus four. Which company is serious? Testing, again, is about a commitment to say, you know what, conversion rates really matter. If your site conversion rate is less than 4 or 5%, you really have to do something about it in 2023. It's just not acceptable. I'm not saying that's going to be an easy journey. No, it's not. Otherwise, everybody's conversion rate will be 4 or 5 or 8%. If you identify a problem, you have to do something about it and doing like one or two experiments is just not being serious. It's almost like taking an antibiotic for 10 days. You have to take two pills every day. Okay, so that is 20 pills versus well, I'm gonna take one pill here and one pill there. And it's like, why am I still sick? Well, duh. Come on.

That's a good one. But I suppose maybe there's also like someone listening, also trying to, to improve or to increase the number of tests that they are running, maybe going from 6 to 10. What are the things that they should focus on in order to increase the number of tests that they launch? Because maybe they might need to add some resources. They might need to look at infrastructure of the tools or something like that.

There's a lot that needs to happen to go from, hey, I'm doing two experiments per quarter. And so I'm going to be doing 12 experiments per quarter. It's a complete shift in resources, tools, mindset, schedules and all that. So I always tell people, learn to crawl before you walk. It's just the reality. Don't expect to go from like, oh, we're doing one or two experiments, you know, per quarter into 12, just right away. That's just a recipe for disaster. No, we're going to take it slow. So we have a plan. We know that our goal is to get to 12 experiments per quarter. So we are going to get there. But we need to understand that there are certain things that need to be in place in order for us to succeed. We need to understand our current team and their structure and what is the skill set that is. Is missing from the current team. We need to understand that the tool that we have and do we have actually a tool that will support us to go to that level of experimentation. We need to have a very clear plan. This quarter we are doing two. Next quarter we might be doing six experiments. The quarter after that we might be doing eight experiments. And you really have to have the full structure around these experiments. Now, the last thing that I want to add is you want to think about, and you are just starting out, quality versus quantity. Lots of times we argue that quality really matters. I completely agree. Quality really matters. But if you are just starting out and CRO is brand new to you and the company and everybody's committed to experimentation as CRO, I would dare say maybe quantity matters in the first six to 12 months. Let's just train the muscle. Let's train the muscle to get it out there. I want to mention a study and I know I'm going to butcher the study, by the way. But they took a whole bunch of guys who are basically cooking, making recipes. They would ask them, they split them into groups. In one group, like in basically over 30 days, they just kind of taught them how to make the perfect recipe. And they said, like, you know, at the end of the 30 days, here's the one time, go ahead and cook that perfect recipe. You've studied, you've done everything, you've learned the techniques. The other group, they didn't spend a whole lot of time teaching them. Them, you know, every day they had to cook something. Every day they had to cook something. And then at the end of 30 days, they gave them the final recipes, like, okay, let's go ahead and cook it. What they saw is that the second group that was testing smaller things every day and making silly mistakes did a lot better than the original group that had one try correct. They were. They spent the whole month studying and analyzing. Most of us learn by doing, at least initially, get that muscle working correct. We will worry about the form and the quality will matter. You don't want to continue three years down the line. We're just throwing things at the wall, hoping something would stick. That's idiotic. But initially, at least, let's just learn how to walk correct. Can I think about the baby initially? He'll take some steps, you will fall. The baby will take some steps and fall. At five or six years old, you don't expect them to continue falling. You expect them to be able to run and improve like year after year.

You mentioned something to do about the team structure. I just want you to talk about maybe the makeup of the team. What do you think is an ideal CRO team?

I know if there's an ideal, ideal team makeup as much as ideal functions. There are different people who handle different functions when it comes to experimentation and CRO. So you must have somebody who's going to do the analysis. In our case, we have a CRO specialist. He or she will do the analysis to say, you know what, here is a problem. Here's another problem. Here's how I'm going to prioritize these problems. I'm going to conduct the data research and I'm going to conduct the user research. I'm going to look at analytics and heat maps, and I'm going to talk to users. I'm going to do some of the competitive analysis. That is what our CROs are tasked to do. You might call them CROs, you might call them business analysts, but their Job is to identify conversion opportunities, conversion leaks on a website. That's number one. You have to have a designer who's like, oh, we've identified a problem. Well, how do we fix it? You know, the CRO or the specialist or the analyst will work with the designer to come up with a design that is absolutely created to fix that particular problem. I was interviewing somebody in the last couple of days who was an actual UX designer, designer in a different life. And now he's a CRO. Absolutely amazing. He's like, I don't work with designers. I'm an amazing UX designer myself. If you have somebody like that, you're lucky. Most CROs that I know, including myself, are absolutely horrible. I keep on reminding my kids every day their drawings are much better than mine. So I can judge a design based on UX principles, conversion principles, but don't ask me to design it because my designs would suck. Then number three, you have to have the developer, somebody who's going to take that design and actually implement the ability test. Not every developer is an AB test developer. Most people think, well, I have an amazing developer in my company, an amazing E commerce team. An amazing developer is not the same as an amazing AB test developer. It's a completely different mindset. That's a whole other video that we probably should talk about the difference between an A B test developer and a regular developer. Lots of times whenever I mention this, I get lots of slack from the developer community and I'm like, hey, I used to be a developer. I used to be a software architect. I know what I'm talking about. But I know and I've hired both and I continue to hire both. So you should have somebody who is responsible for the quality control. We have the A B test. It's ready. We need to check, you know, are there any bugs in the A B test? The goal of an a B test is to increase the site conversion rate. Okay, this makes lots of sense. But if the A B test has a bug, it sort of defeats the whole purpose. Correct. And it's like now, instead of helping improve conversions, now we dropped conversions. For us, we always have a project manager. Every project should have a project manager. Somebody who's responsible for the deadlines now. Okay, Khadid, you're mentioning like a million roles. No, because your CRO could be your CRO and your designer could be responsible for the quality control and could be responsible for running the project. Correct. You probably, at a minimum, you need that marketing person, CRO person and a developer next to them, those two. Absolutely. Minimum. If you have the right people can run a program now, is it going to be as effective as having five people? No. There's just the reality five people means things are moving a lot faster. We're able to test things a lot faster, and we're able to execute and then see results faster.

You had mentioned the numbers, the mindset. You should know what to test, the speed. We're now going to number five.

Number five is the post test analysis. We're not running experimentation, AB testing and conversion optimization just for the sake of running it. Every experiment has two goals. One, the declared obvious goal is to improve conversions. We want to have higher conversions. Great. And every experiment should get you closer to that goal. It also has a really second important goal, which is helping you understand your customers better. And that's the reason, I think knowing what to test and designing experiments in a mindful way. Being mindful whenever you're designing experiments is critical to the success of the program. After every experiment, we ask a question. So what did we learn about our customers based on this testing data that we see right now? And what actions can can we take? I'll give you two examples. We were testing different messaging. We worked with Dish Network, which is a large satellite provider in the US we're testing different messaging on their landing pages to see which messaging works better for visitors. We have winners. Okay, so what's the next actionable step? Well, you guys are sending all these flyers in the mail, and you're doing these newspaper inserts also for your service all across the U.S. yes. Well, why don't we take that winner design that we have and let's go ahead and actually use it in those flyers and in those advertisements that we are doing on paper. Is it going to work as well as, you know, it worked on landing pages? So we actually even ran an A B test for these flyers that we're sending out in the mail. And for the newspaper inserts, the statistical analysis was a lot harder because we have to pull lots of data, lots of Excel files. But in 17 out of 18 campaigns, the winners online were winners offline, which is absolutely powerful. So, yes, you take actionable insights from your testing and you deploy it. Another example, we've tested different headlines on a landing page. We have a winning headline. Well, the question is, what does that headline tell us about our customers? Maybe our customers care a lot more about influencer endorsements more than scientific research. That actually was one of the results for one of the experiments that we ran. Okay, so this is really interesting. So we need to change that across our website and not only across our website, we need to change that in our ads as well. Your test results should inform actions that you take on the website. Post tests. Another example we ran recently for a company, they sell beautiful, beautiful pajamas. We did a test for bundles and where we bundled multiple items, not only did it increase average order value, but conversion rates increased by almost like 30, 40%, which is incredible for an A B test. And we're like, oh, oh man, people love bundles. We didn't think about that. We were trying to sell one off and trying to figure out a way to increase average order value. And we roll out the bundles across the website. So that works really well. Every experiment should inform you of what to do next. Lots of times people test and then they look at the test results and like, okay, nice, we have a win here. But they don't know what to do with that win afterwards. And that's a huge mistake.

Since we are also talking about the post test analysis is how difficult is it to learn from an experiment that lost? Because like people always say, you always like learn something from each and every test, whether it wins or loses. But is it that easy?

It's not easy. I always love it when people say stuff and you really tell them, show me an experiment that loss and show me what you learn from it. And what you see is they make lots of assumptions. So here's the thing. You have an experiment, you start with a hypothesis and then the experiment fails. Let's simplify it. Let's say we have an maybe experiments, we have an original, we have a new design that we've created to fix a problem that we've identified. That was our hypothesis and APPLAUSE does that mean that, well, the problem that we've identified was not a real problem? No, There's a good chance it is a real problem. It's just the way you've implemented the experiments did not work for people. Let's make it even a bit more complicated, a little bit more interesting. We have a page, we've identified a problem in it and we said, you know what, here are three different ways to fix fix that problem. Approach one, approach two, approach three, they are all different ways to fix the problem. Approach A wins, great. So that means the hypothesis was correct. But then approach two and three, they both lose. Does that mean that people actually hate those approaches? No. We can start making hypothesis assumptions about our visitors. We'll say, maybe people don't like approach two. Maybe people don't like Approach three, we need to test this out. We need to validate that in other experiments. Simple example, website doesn't have trust elements. You see that there's a high bounce rate on the website. That means people are coming, leaving the site right away. There are trust issues. Okay. One of the best ways to do that is through trust signals. Well, what does that mean? Well, you know, we have endorsements that we can mention. We've been mentioned in magazines. We have some influencers who mention us. We can mention the number of people who bought from us. Us. We can show the number of reviews that we've got. Five different ways to send a trust signal to visitors coming to our website. We test them out and we find that whenever we use influencers, people actually like, you know, that that particular design that focuses on influencers and the endorsements that we got from celebrities loses. Let's say our target market hates celebrities or they hate influencers. No, but we'll put a question mark around it because we will test this idea in other places. Places. Maybe we find out that people, you know, the design where we showed that we've been mentioned in. I know the Times in USA Today wins really well. Does that mean that works really well? Is that the hypothesis about our visitors? Maybe, maybe not. Although it's a winning design. But we'll put the question mark because we'll validate that. We'll use those trust signals and other pages on the website and other experiments to validate. If I constantly, you know, run tests where I'm using New York Times and then I'm using USA Today and I see that whenever I use those icons and I use the mentions that these newspapers have of my brand, I see that my conversion rate goes up. I'm like, you know what? My visitors absolutely love those news outlets and they love the fact that we're mentioned there. Now I can put that as a fact about my customers. Again, you're constantly building a customer profile, making assumptions about customers, adjusting them, testing them out.

There's also a good explanation. I'm still digesting it, but like Simba.

We actually had like, what, six, seven questions to start with.

Yeah.

And we managed to do only one question, but I think we've delved really deep.

Yeah, that was a good conversation. I also enjoyed the chat.

Well, thank you everybody for listening. We shall come back next week. Quick reminder. So the channel and the podcast is about helping you improve your site conversion rate. And we do that by sharing the best tips, techniques, strategies on improving the site conversion rates, about making more money online. So if you are new here, please make sure that you subscribe and follow and like and comment. And until next week.