Bonus Episode 5 – How To Choose An A/B Testing Tool

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live, our podcast today, wherever you listen to your podcasts.

Hello. Hello, everyone. Welcome to yet another episode of the CRO Live Hour podcast. I'm your host, Simba Dube. So in today's episode, we'll be focusing on a B testing tools. What are the features of a great a B testing tool? How do you choose an a B testing tool? We know that there are many A B testing tools in the market, but which one is good for you specifically for your business? What exactly should you look for? So I reached out to Khalid Saleh, who has used dozens of these A B testing tools, and he also had to create one called figpi. It's actually like a good a B testing tool. You should go and try it. So how do you choose an AB testing tool? This is what Kalisali had to say.

If you are going to be running AB testing, for example, on mobile apps and on your websites, that automatically will remove some a B testing tools from the running. But you have to be realistic. Sometimes I see companies say, well, you know, we have a mobile app and we have the website and we are going to be doing AB testing on both. And that's the reason we're going to commit to this a B testing tool. And three years later, they've paid a ton of money for that a B testing tool. But they never ran a single test test on their mobile app because a B testing requires a commitment, a resource commitment, and not everybody is ready for that. If you are a fundamental believer of multivariate testing, that case that excludes some Tools for us, for example, we have not run a multivariate test in probably seven or eight years. Our testing has focused on doing a B tests. But there are companies that want to do multivariate testing. In that case, you want to pick a tool that has multivariate testing capability. If you want to do server side testing, then again you want to look for a tool that supports that. Your business requirements there will drive at least the initial set of tools that you want to evaluate. A second thing that I would look at is the statistics engine for the tool. And how do you determine a winner? There's two different schools, Frequentist or Bayesian. If you're a frequentist, you probably want to pick a tool that's frequentious. If you're Bayesian statistics, you want to pick a tool that is Bayesian statistics. There's no magic bullet there. It just depends on the statistics and what you believe in and what you like more, who is going to be implementing the experiments. And this is very critical. Most of the tools nowadays give you a visual editor. A visual editor gives you the ability to move things around and test things around. If you talk to a CRO agency, most CRO agencies will never touch that visual editor. The A B testing companies try and push people to use the visual editor. But the reality, visual editors have major limitations in them. You cannot implement a complex or sophisticated AB test using a visual editor. But if you say, you know what, My experimentations For the next six months, 12 months are going to be just very simple. I'm changing headlines, I'm changing images, I'm switching banners. So maybe like, you know, maybe you want to look for something that has that visual editor. And finally you want to look at the cost. Do you want to do something for free? So Google Optimize, for example, gives you the ability to run up to five A B tests concurrently at the same time for free. Absolutely free. Or do you want to pay? And there's plenty of tools that are available to you. You can spend literally 10, $15,000 even more. Some of our clients use some tools that actually will charge them 25 and $35,000 a month for a B testing. I see those tools, by the way. I see them in action and I think to myself, I'm like, yeah, they have some nice features, but is it worth 20 or 30,000 do a month? If it was my money, I probably wouldn't, wouldn't spend the money. But usually those companies already signed annual contracts. And you know the saying, no one ever gets fired for hiring an IBM. Sometimes you want to go for the best tool out there and it's just safer. Majority of our clients nowadays, just kind of, to give you an idea, are on one of two platforms, either using FigPi or they're using VWO. There's quite a cost difference between the two different platforms. Functionality wise, from a CRO perspective, what the CRO needs, they're very close to each other. Price wise, they're quite different than each other. I don't think I have a client nowadays using Optimizely or some of the other the other tools out there. But I always tell people, like when you peel the onion, two things happen, correct? You cry, and then you discover that most tools are very close to each other. There's nothing very unique that sets them apart.