#30 – Should You Be Launching A/A Tests?

This gentleman was launching the AA test for a whole month. I'm like, oh, you don't have any other experiments waiting to run on this page? He's like, yeah, I do, but we're just waiting for this AAA test to finish. And I'm like, an AAA test will never finish. Really? You know, in theory, you should not achieve statistical confidence for the test or significance in this test. My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each EPIS episode, Khaled, amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal.

Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very toug questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today, wherever you listen to your podcasts. Well, here we are. CRO Live Hour, March 2023. Should you Shopify or shouldn't you? AA tests and a lot more. Simba, how are you doing?

I'm good. I'm actually thinking of, is it episode number 80 something or 90 something? I'm not really sure about that.

I mean, Simba, I'm not gonna say anything. I keep on telling you we need to count the number of episodes. Let's just count the number of episodes. I think it will be really cool to know how many episodes we've done until now. So we've done a year, a year and a half now.

Close to a year and a half.

Yeah, we definitely pass 50, but yeah, I wonder actually where we're at. So, anyways, how was your week, Simba?

Still the same as it was last week. Been tied up past few months. I've been like, working on both sides on the invesp side and also on the fake PI side. And it's a good thing because I think this past two months have been, like, very good for figpi, compared, like, to the previous Time, because a lot of people are showing interest in figpai. They're signing up for figpi. They're loving figpai. So it's good.

It's funny because I just did a demo for fakebi just prior to this call, and it's a large company. And then he was looking at it, and then he's like. He looked at the pricing. He's like, don't really recognize that you guys have all these features. Absolutely love them. And then he was asking about the support and he's like, yeah. He's like, this all makes perfect sense to me. So how do we sign up? And I'm like, oh, it's a nice way to start the CRO Live hour with, like, you know, kind of closing somebody signing up on figpi. So it's just interesting. It's just been interesting. And I think you're seeing the signups for FICbuy are up 100% month to month. So that's absolutely amazing.

Yeah, that's a good thing. So how about you? How was your week?

My week was a lot busier than I anticipated and VESP was very, very busy. It's almost like every company just decided to say, hey, we're going to start conversion rate optimization. So invesp was super busy. So we've onboarded, I think, four or five clients in one month. Usually we like to keep them to one or two clients. We're still a small agency, so kind of like, you know, keeping that in mind. And of course, figpi, man, the amount of demos, I mean, at some point there was one day where I had 12 demos booked and I almost died. I love it, but I was just like, ugh, you know, I'm tired.

The good thing, that's a good kind of abuse, right?

Oh, I will take that.

Yes, that's good. So today we're talking Shopify. We're talking AA testing. I see, like, a lot of people are complaining about Shopify, that their products are being removed on Shopify. Is it Shopify Plus? Yeah.

You would think that you have more rights by being on Shopify plus, but really you don't. You're just paying more for more features, but you don't have more rights on Shopify Plus. You might have a little bit better supports. And by the way, it's fascinating to me, in 2009, 2010, if you had asked me about the INVESP clients, I would have told you 90% of them were either on Magento or moving to Magento. That was kind of the name of the game that was the new kid on the block. And at that point in time, I think it was OS Commerce that was kind of the other big platform that people were using. There's a couple of enterprise level platforms. That's where I came from. Kind of the enterprise side of E Commerce. But Magento is the new cool technology. Fast forward 2023, everybody's on Shopify or moving to Shopify, which is fascinating. And I think it's almost like a natural progression of technology, by the way, where a tool becomes really cool, there's the early adopters, then everybody jumps on it. And I think Shopify is going to go through this very important point of figuring out how do they actually meet the demands of their customer base and add new features without slowing down the platform, making it easy to use. You're not going to please everyone. That's just the reality of building a platform. But that's fine. As long as you know your core customer and you go after them, you're all good. Correct. I think Shopify is going to go through that and it will be fascinating to me in 2030 to see who we're talking about. Are we continuing to talk about Shopify or is there going to be a new platform that says, oh well, Shopify had fallen short in some of those areas? That's me and the history of Shopify. But yeah, there has been lots of complaints. By the way, majority of invesp clients, majority of Figma customers are on Shopify. It's fascinating. Shopify, BigCommerce, Magento. I think on the e commerce side, Legion, WordPress, I think Drupal, I don't see many Joomlas anymore. Maybe just the way people are reaching out to us. But one of the fascinating things is the complaints that you mentioned and we just saw it earlier today. By the way, the complaint that people have is, hey, we woke up today and Shopify had removed 20, 30, 40 of our products. And they warned us that if you add those products you will be removed from the platform completely. That's a scary thought. As an E commerce store operator, manager, owner, VP of marketing, and I don't know if there is a solution around that they own the platform and they can choose to have you on the platform or not.

That's kind of like interesting. But for platforms as big as Shopify or for brands as big as Shopify, I feel like at some point they can afford to make some certain mistakes because they are big and they can also easily get back on their feet. That's just what I think, yeah, I.

Mean it's fascinating to me. So I'll mention another big brand that many E commerce stores rely on is Amazon. They will have like, you know, they have their Amazon as one of their channels and Amazon could remove some of your products. There's some issues also around Amazon and if you are a one like you know, small, tiny e commerce store operator, you're under the mercy of Amazon. Correct. They removed some products. They had some customers complaint about the shipping from your side. Sometimes things get removed just because Amazon decided to remove them because of a policy or something like that. The one time that I saw Amazon respond really, really quickly, like in an auction as opposed to like in a story told, is when one of the P and G, Procter and Gamble products, some of their products were removed and it's P and G. Correct. I mean they're big, they can throw their weight around and oh my God, yeah, the Amazon reps are back and they're like, oh yeah, we're bringing it back really quickly. We know how much money you're spending with us. Same thing with Google Ads. Correct. I mean, although there are some companies that try to butt heads with Google, but Google controls also the platform, they run the ads and at some point they can just tell you goodbye and good luck and you don't have much of an option, which is scary thoughts by the way. All of these are rented properties. Correct. You're renting the access by your own subscription. Correct. So they own it. There's risk there. Now I know that if you have a chief strategy officer, if you're a big company or if you're a business owner and you're thinking about strategy, there's always the risks that you want and usually hosted platform, maybe it's on your risk list, but it's really not on top of that risk list. So something to think about now. I think the one thing that we have to add is that these companies that had their products removed, they were in a specific industry, so they were in like no supplements. And I think the issue, not necessarily due to Shopify, by the way, it came from the fact that Shopify relies on Stripe for their order processing and whatnot. And I think Stripe has some really strict rules around processing orders and whatnot. But again, Stripe, another platform that is just humongous and monstrous and they can do whatever they want. Another platform, PayPal, people complain where PayPal had taken money out of their accounts or puts a hold on their accounts, 150, 200, $300,000 for a small E commerce company that's a lot of money and can put some people out of business. Yeah. So diversifying maybe is the name of the game. I don't know how you would when it comes to E commerce platform.

I definitely wonder how many small companies are reacting to this. Like just a week ago or two weeks ago there was this. In fact there's always like this talk of TikTok getting banned in the US a lot of small companies would get like affected and also now shopify acting up. So that's kind of like interesting to see how plays out.

Yeah, I mean it's fascinating to me by the way, because this is one of the things I've done. Again, second attempt remote Facebook from my phone, which I think is great. So like I logged into Facebook yesterday after like not being on Facebook for like four or five days and like all these messages are there and I'm like, oh, so comforting. But I've removed TikTok. I mean, I tried TikTok for a while and I got addicted to it. It's very addictive. It's built to be addictive. So I'm like, I don't need to be addictive. I have a personality that can easily get addicted to anything to be very careful. But I understand the concerns around TikTok and they're at the same time kind of TikTok is trying to say, oh, look at all these small businesses that are relying on us for ads and whatnot. And then I think the founder CEO of TikTok was in Congress testifying. And of course, I mean, you know, those guys don't have a clue. I mean I've worked a lot like in a previous life with Congress and the Senate and like, you know, the White House. So I know the caliber of the people they are the front facing. They don't do lots of thinking. I mean, it depends on the person that you're talking to, like the senator or the congressman. But really lots of thinking is done by the staffer. So the staffer writes a question, the congressman is writing it. And sometimes I'm like, dude, that was a stupid question and you're going to go viral on TikTok for asking a stupid question about TikTok.

So before we get into the questions of the day, you also mentioned that you tried to delete Facebook. Right. And maybe LinkedIn, but you're also part of some other interesting groups on Facebook. So I'm not sure. I wouldn't delete Facebook just because of the groups that I'm in. That's the only reason why I'm on Facebook, I don't post a lot. I just go on those groups and just read a lot of stuff, get to learn a lot of things. For instance, the Exit 5 group. Actually good group for marketers.

I agree. I mean, I do miss some of those groups. Exit 5 is a great group to me. I actually even like, you know, CXL is a competitor of Invest, but I love their group, although I haven't logged in there in ages. Who else? I think those are the two, three groups. I like the Shopify plus group just to see what people are talking about when it comes to Shopify. So, yeah, I miss out on learning from those groups. But here's how I remedy that. First, I log in Facebook on my desktop. So I can control that. So it's like during specific hours, as opposed to having Facebook in my face pun unintended when it's on my phone. Correct. So at least I control that. So I can still access, unlook, and just scroll really quickly when it's on my laptop. In all honesty, maybe like, you know, I'm missing out on information, but it's a tax that I pay. I must have gotten two to three hours back every day. Imagine how much time you gain. And I'm like, oh, wow, so much free time. I can do this, I can do that. Like, there are certain things imagine that I can do before. So I've seen my productivity improve. I am a little bit more on Twitter, but I'm also selective of like, you know, it's like, okay, I just read maybe a comment here and there. Are you, like, using Twitter regularly?

I just go there and I have certain people that I follow and I don't tweet a lot, but I just go and view other people's tweets about marketing and stuff. But I think Twitter and LinkedIn, they are far much like my favorite social platforms right now.

I mean, it's interesting. I keep on saying, I mean, LinkedIn is great. Again, each platform comes with its pluses and minuses. And I'm like, I just wonder if this platform is going to survive for two, three more years. I guess it's an experiment. It's a big AB test that Elon Musk is running. So we shall see. We shall see.

Okay, so let's get into today's questions. So today we're talking about AA tests. And the first question that I have for you is, should you be running AA test?

One of the very first things that we do whenever company starts doing conversion Optimization using envespr Conversion Rate Optimization is launching AA tests all across the website. So it's a very first step that every CRO should do. Now why do you do that? There is a good way to do it and there's a bad way to do it. What are the pluses and the minuses? Those are the things that we can discuss. What is an AA test? It's a simple a B test where you test the page against itself. What does that mean? Whenever you're setting up an A B test, you say, okay, well I want to run a test on the homepage. And you go to your AB testing software and you say, I'm running a test on the homepage, I'm running a test on my product pages, I'm running a test on the CART page. And then the A B testing software is going to ask you, okay, so what is the new design, the variation, the challenger people give it different names, that is going to be running against this original design that we have currently on the website. So you tell it, actually there's no new design. I'm going to test the page against itself. This new design has no changes whatsoever. In theory, you launch the test. So let's say you have 100,000 visitors coming to the page. 50,000 will view the original page. Correct. And then the other 50,000, as they're coming, the AB testing software is going to say, oh well, yeah, let me show them the new design. But the new design is really a copy of the original. So that's the reason we call it an AA test. We like to launch AA tests at the very beginning of every project because we like to take measurements across the website to understand how the site is performing. Now think about this. Typically, most sites use analytics. Most, if not all websites use analytics to track the site performance. They track the number of visitors coming to different pages, the number of conversions. They track the bounce rate and the exit rate. An analytics package, whether it's Google Analytics, amplitude heap, you mention it has a way to define what a visitor is, what a bounce rate is, what an exit rate is, what a conversion is. And an A B testing platform may or may not match that definition when it comes to visitors sessions conversions that an analytics platform has. So one of the reasons that we launch AA tests when we start a project is to get measurements. Let's launch an AA test on all the product pages, on all the collection pages, on the homepage, on the CART page, on the checkouts. I'm trying to get an idea about two metrics that are important for me whenever I do AB testing, One metric is the number of visitors that are coming to that page in a day or in a week or in a month. The number of conversions that that particular page or class of pages is generating for the site. Now everybody talks about the site conversion rates. You know, oh, my site conversion rate is 1%, 2%. I've mentioned previously, forget about your site conversion rate and let's break it down. Let's take a look at mobile and desktop. It's a very high level. But let's break it down even further. If I'm launching an experiment on my product pages, then I want to know prior to launching that experiment how many visitors are coming to the product pages and what is the conversion rate for the product pages. And by saying the conversion rate for the product pages is the number of visitors that come to the product pages and actually convert. Place an order so it's from the product page all the way to the order confirmation. You may or may not know that number using your analytics, but analytics in this case doesn't matter a lot. When I launch an AA test, guess what? The A B testing software is going to help me set up and say, okay, here's how many visitors are actually coming to this page and here is the page conversion rate. So it's really a nice way of getting measurements for different pages on the website. In theory, the conversion rate for the original and the conversion rate for the challenger, which is a copy of the original, should be the same. Correct reality. They're not always the same. Which by the way highlights a whole other problem with AA tests. That's why we launch a test from the get go.

I understand. Maybe just to recap what you're saying, it's also like a way of trying to configure and see if everything is set up very well. Right. Like the metrics are matching with like your analytics tool or maybe your Google Analytics or your amplitude tool. So my question now is how long do you run those AA tests?

Let me actually also mention another benefit of why you would launch a side benefit of launching an AA test. For every experiment that you launch, you have to set up different goals. Correct? For the experiment I'm running, let's say a traditional AB test. Oh, here's a new design for the product page, a new design for the cart page. And part of the setup is to say, well, what is the goal of this test? Oh, I want to increase conversions. So you need to have a definition for increasing conversions. Now in every experiment we have what we call macro conversions. Primary conversion goal, I wanna get more orders. That's the macro. Correct. But in order for me to get to that macro conversion, I have to go through several steps. Clicking on the add to cart button, starting the checkout, going through the checkout. Those could be all micro conversions. The beauty of starting a project and setting up a 8s is you go ahead, you are being forced to set up all these goals. We like to do that, to set up all these different goals so we have their definitions. So now as we launch more actual A B experiments, not AA experiments, we already have those definitions so we don't have to figure them out a month into the project. Now, one thing that you want to pay attention to whenever you're launching AA tests is don't try and match closely the number of visitors or the metrics that you have in your A B testing software to your analytics. They are not, they are two different creatures. They are cousins, maybe far not first cousins because you will waste so much time trying to match the data and it doesn't match and you're going to get lots of headache and heartburn. So just move on. It's not going to match. There is going to be differences anywhere between 5% up to 15, sometimes 20% between the two. The one thing that they should match, by the way, between your analytics and your A B testing software, the one metric that should match closely, or I would say 4 or 5% difference is acceptable, is the number of conversions. If you launch an AA test or if you launch an A B test and your A B testing platform is telling you that you have 1000 conversions and your analytics is telling you you have 500 conversions, you have a problem. Correct. Because a conversion is a conversion. Somebody placed an order, somebody subscribed, somebody filled out a form. Number of visitors might vary, but the conversion is a unique and discrete event. So we want to match it. Now, is it going to be 100%? No, that's probably, I would say within 95%, give or take. So there's almost like a 5% up or down difference between the two. And there's many reasons. It's like a discussion for another time duration of an AA test. Some people and I saw this recently, somebody had asked me to review something and they were running an AA test for like a month. And I'm like, dude, why? Why for a month? He's like, well, I'm just taking measurements. Yeah, I appreciate that you're taking measurements, but remember, the goal of an AA test does really drives how long I'm going to be running it. I just need to get some metrics around. How many visitors, how many conversions is our page is generating? Usually I can get those metrics within two to three days. If I'm being really generous and I have the time, then I might run the AA test for a week. Now, when you launch an AA test and this gentleman was launching the AA test for a whole month, I'm like, oh, you don't have any other experiments waiting to run on this page? He's like, yeah, I do, but we're just waiting for this AA test to finish. And I'm like, an AA test will never finish. Really. You know, in theory you should not achieve statistical confidence for the test or significance in this test. And he was just running it and I'm like, no, no, no, don't do that. A week maximum. That's all. You get the data that you want. So that's what I recommend.

What happens if there's like a huge difference between the results? Let's say the other one gets about 1,000 conventions, the other one gets about 200.

Okay, 1,200. I'll be very concerned. Correct. There will be instances where the AA test results are different. I'll take that for granted. And that difference is anywhere between 3% up to 10, 15%. The higher the difference, the more concerned I am. So I always tell people the reason the gentleman had asked me the one month AA test because the difference was 12%. I told him, listen, naturally most of the time when you're running your AA test, you're running it, whether it's Bayesian statistics or frequential statistics. You're going after 95% confidence. Guess what? There's a 5% chance of false positive, especially with frequential statistics. So you're falling within that 5%. And he's like, oh, maybe I should relaunch the AA test. And I'm like, no, no, no. Again, the goal there is not to validate, so just move on. Now, if I see the difference around 80%, 200 conversions versus 1,000 conversions. It's good that we've launched the AA test because that would probably have a problem. Now this problem could be in the tool, the AB testing platform that you're using. Most likely that problem has to do with the goal conversion, how goals, how conversions are tracked in the tool. And it's good that you've found this problem out with an AA test as a poston test that you've developed and designed. And now you're waiting for the results. That's another way. Again, it goes back to the goals and tracking the goals, that is really like, hey, there's an 80% difference. We have a problem here. Lots of times I see this, by the way, we're not triggering conversions in one of the variation because we're not tracking visitors correctly in the AB testing software. It's good to find that out before we get serious into AB testing.

That's good. So are there like any sort of challenges that you face when trying to launch AA test, or it's just like the same as a B testing?

No. AA tests should be, in theory, much easier, by the way, than a B testing, because if you think about an A B test, you pull up your A B testing software, you say, here's version A. And then you need to design sometimes code version B correct in a B test. So that takes a while. In an AA test, you don't have to deal with that. Here's version A. What is version B? It's version A as well. I'm not making any changes to it. Done. The tough part about setting up an AA test, and that's the reason I see some people struggle with it, is setting up the different goals. I have seen somebody setting up an AA test with 20 different goals, and I'm like, really? I'm all about tracking data for an experiment. Don't overwhelm yourself with too many goals. Correct. Oh, here's 20, 25 different goals. And I'm like, who's going to sit there and analyze 20, 25 goals? And five of them are showing you a lift, and seven of them are showing you no imp, and 18 of them are. This is being silly. Don't go overboard. Whether it's an AB experiment or an AA experiment, we like to set up anywhere between. I would say we max out at five goals. That's a good way to measure. We understand that there's one primary goal. There are secondary primary goals where people struggle with an AA test, because typically you're launching an AA test as the first tests you're launching on a platform is defining those goals and coming away to really clearly define them. That's where people struggle. And I tell them, listen, you're going to have this problem, whether it's an AA test or an A B test. Correct. This problem is going to exist. So might as well find out about this problem and solve it at the very beginning of the project when you're launching your AA tests, done, you're done for all, and you move on from.

There as I listen to you, like, explain this. I picked out, like, some rules of AA testing. The first one, I would say it should not run for more than a week. That's number one. Number two, your AA test must not have more than five goals and the goals have to be the same.

What do you mean, have to be the same? No, like, you have to explain to me that.

Now that you have said it, I understand. Okay, okay, okay.

You should not have more than five goals. I would say yeah, you shouldn't have.

More than five goals. So are there any other, like, rules when it comes to AA testing?

I would say the only other thing, don't worry if there is a little bit of difference between the test results. To me, 5, 10, 15% is within the acceptable range. Do worry if you are seeing a 30, 40, 50% difference or more in the conversion rates and conversions between the different variations in AA test, because that tells you that there is a major problem. And thank God you've launched the AA test to find out and discover that problem prior to getting serious about AB testing.

That makes a lot of sense. So I think those are the questions that we have today. I was going to ask you, how do you interpret the AA test results, but you kind of, like, touched on it.

Well, yeah, to me it's like, I know how many visitors, I know what's the conversion rate and guess what? Whenever. Now, since you have that data and you go to set up an a B test, an actual a B test, you need to use a duration calculator and it's going to ask you, hey, do you know the original conversion rate? Guess what? Yes, I do, because I ran my AA test and I know that my product pages have a conversion rate of 8% and I know that they get 40,000 visitors, like in a week or in a month. So you have that data. As opposed to scrambling and saying actually not sure what the conversion rate is, you have the data. So your tests, your duration calculators. Using duration calculators will be much easier.

In most of the CRO programs, people, they always say you should start by doing research and then creating hypothesis and then launching tests. Do you have to follow the same kind of process when it comes to AA tests? I know. Maybe you don't have to do research because those AA tests are research on their own. Right?

Exactly.

How about in terms of a hypothesis? Do you have to come up with something like that?

No, no, no, no hypothesis. You're testing the page against itself. When a company onboards with invesp, I think within the first couple of days, we launched the AA test. It's as simple as that. And the only reason we don't launch it the first day because we're just figuring out, okay, what are the different goals we need to be tracking, how we define those goals in the A B testing platform. So we launch AA test really quickly just to get an idea, because ultimately, we want to understand how many experiments I can launch on a product page, on a category page, on a collection page, on a cart page, on a subscription page. So I need to have that data. Yes, I can pull that data from analytics. But again, analytics data and a B testing data are a little bit different. So I want that data from a B testing platform. So we launched the A test right away. No hypothesis, just, hey, set it up, launch it. The only hard part is make sure that you have the goal definition done correctly.

Okay, I think those are the questions that we had for today. Awesome.

Well, thank you everybody for listening. Would love for you, by the way, to leave us a comment. And like, if you're listening to this on Apple Podcasts or wherever you're listening to your podcast, by the way, we're doing rather well, actually, with the listenership just growing week after week. So it's super exciting. Awesome. Okay, Simba, until next week. Take care, my friend.

Ciao. Bye.