Bonus Episode 7 – 5 Big Challenges Faced By CROs

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today. Wherever you listen to your podcasts, what are the five biggest challeng that we face whenever we are trying to run an experimentation or conversion optimization project? There may be big categories of things that if you've done experimentation long enough, you're going to run into one of those issues. As you do more and more experimentation, you'll have to figure your way around them. So different organizations, different companies, different individuals, deal with them differently. The very first challenge is culture, experimentation and conversion optimization is built counterintuitive to how many organizations, how many companies do business naturally? Think about a typical company where you have a development team, you have a design team, you have some analysts, and maybe you have management. Typically, somebody in management will say, hey, we have a problem here. And analysts will dig a lot deeper to find out the issues, come up with a solution, a designer will come up with new designs on how to implement that solution. The developer implements it, we rolls out, we're done, we move. It's a very clearly defined process. There is a start, there's an end to it. You go to experimentation and conversion optimization, you throw all of that out the window. There's no longer a single individual who's saying, oh, well, we have a problem here. There is a process around discovering the issues that we need to address. But fundamentally, instead of coming up with a single solution to the problem that we've identified, we come up with multiple ways of fixing that problem, and we say, we don't know. We're taking our best guess on how to fix this problem. But we're not making that decision. It's actually our visit visitors, our customers. They are the ones who are going to help us decide. We run an experiment to say, hey, visitors, please tell us which design you like. We know that there's a good chance we'll throw away most of those designs if people don't like them. And even when we have a winning design, we don't stop there because we are going to go back to the drawing more. So you go from, I've designed something, I fixed it into, oh, I'm making a fix today and I'm going to be making a fix a week from now, and I'm going to be making a fix a month from now. So there is this constant evolution that a design and a functionality needs to go through. Designers and developers and even management struggles with that because they're used to, oh, I need to get a design perfect and roll it out. When you switch to an experimentation mindset, you say, you know what, it doesn't have to be perfect because there is a good chance that people might not like it and we might throw it away. We go from, I have to get every little detail right, crossing my T's, dotting my I's into, oh, let's get this out, let's roll it out now. We're not going to do something silly, but we know that we're going to be making changes to it a month from now. Another challenge that we see is selecting the right technology for the program that you have. Google Optimize is very popular nowadays. Many companies that are just starting out with conversion optimization jump into Google Optimize. And I think Google Optimize helps the industry move forward. However, there is always a caveat. Google Optimize, by its nature and just the way Google wanted to price things, allows you to run five concurrent tests at the same time. If you are serious about experimentation, if you are serious about conversion rate optimization, five concurrent tests tests is laughable. People say, really, I might be running two concurrent tests. Let's take a simple E commerce website. You have your homepage and you have your category page and you have your product page and your cart page, and we have many other pages. But this is at a very high level. I'm going to be running a test on my homepage for a mobile test and I'm going to be running a desktop test for my homepage. That's two slots. Two tests also for my category pages, one for mobile, one for desktop. That's four slots. And I want to run two Tests for my product pages, mobile and desktop. Oops. No, you have to stop because guess what? You already have four slots. Felt choosing the right technology is critical. There's different statistical methods that different AB testing software uses. Most people don't pay close attention to statistics and the statistical methods that their software uses. There is AB testing software out there that uses different style of statistics that will actually show you that most of your experiments end up losing, although you have winners. So it's very conservative AB testing software. And you have the other end of the spectrum where you have AB testing software that shows you that a good number of your experiments are winning. So both ends of the spectrum are winning. Really not so great. Another issue with technology is the flicker effects that can cause an issue. Some people struggle with it tremendously. Some people cannot accept the fact that there's a flicker effect. Another challenge whenever you're running an experimentation program is the time commitment that it requires. Most people jump into conversion optimization because they've seen a case study somewhere. Oh, we've tested this and it generated a 500% lift in conversions for us. They come to conversion optimization absolutely excited, thinking that we're going to come in, we're going to smash it, we're going to take our conversion rate from 5% to 10% and things are going to be great. And they start running experiments and they have a loser. Another losing experiment, Another losing experiment. There is a time commitment in terms of the analysis that you need to do prior to running any tests. There is organizational commitment to continue with the program even when you have losing experiments. And then there's a time commitment to do post test analysis. Not everybody is ready for that. You cannot imagine how many companies come to us and they say, I'm at 5%, I'd like to go to 7%. We are in July right now. I'd like to do it by December. And I tell them, great, that you want to do that. My job is to tell you the truth, whether you like it or not. That's not my business. Another challenge is the complexity of the experiments that you're running. Some experiments are very complex. That means that they require a lot of time to implement. That means that you have to have the right team that knows how to implement those experiments. Everybody can dream up different variations like, oh, well, we can fix this problem using this solution. We can fix it during that solution. Well, ultimately you have to have the right strong developer who's able to implement those experiments. You cannot imagine how often I'm part of groups and I See business owners and I see executives who say, well, we just like, you know, run experiments using the wysiwyg, like, you know, the, the visual editor. And I sort of chuckle. Most of the time I don't say anything because it's good that they're trying to run a B experiments. But if you're merely implementing experiments using the wysiwyg, which is the visual editor that's available on the AB testing software, you probably are not doing this real, meaningful, deep AB testing. And then the final challenge is the analysis that is required prior to running an experiment. Impose running an experiment, you skimp on any of those two garbage in, garbage outs. Don't expect that you're going to have meaningful experimentation if you're not doing a lot of analysis. What are the data points that are making you think that we should be running this experiment? And those data points could come from analytics, from heat maps, from session recording, from customer interviews, from competitive analysis, from expert reviews. But I don't want you just to randomly say, oh yeah, well, I dreamt this last night. You can, but I don't know if you're going to run a successful experimentation program doing that. And then the other area is you've run an experiment, whether you have a winner or a loser. The bigger question is what did you learn from the experiments? I'm a person who loves winning. Correct. I get excited by winning experiments, but I get even more excited when I see post this analysis, what did we learn? How can we implement those learnings in other marketing campaigns? That is meaningful experimentation. That is meaningful, called conversion optimization.