Bonus Episode 4 – How To Build A Culture Of Experimentation

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man. I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour Podcast today wherever you listen to your podcasts.

Hello. Hello everyone. Thank you for being welcome to the CRO Live Hour podcast. This is a short episode. We usually have long and short episodes. So for these short episodes I'll be hosting them by myself without my co host, Khalid Saleh. So the idea behind these short episodes is to focus on one topic at a time or on one question at a time. So in today's episode we are focusing on the culture of experimentation. How do you build a culture of experimentation? Or what are the components of the successful culture of experimentation? So I had a chat with Khalid some few days ago about this and then this is what he had to say.

There needs to be a shift in the mindset and the culture of the company in order for experimentation to really take its roots within the organization. Nowadays we think about management as something at the very foundation of how businesses run. While 100 years ago all these concepts were new. If you've taken any business classes, you see kind of the evolution of organizational structures and introduction of management. In 2022, who's going to run a company without having some sort of management structure or team structure? I think it's the same thing for experimentation nowadays. It's new. We're trying to talk to people about it. But I think you fast forward 50 years from now is going to be just taken for granted. It's going to become a way of doing business. A successful culture of experimentation has six different components. First, it starts with a complete buy in and understanding of what it would take to Have a culture of experiment. And that's complete buy in needs to happen from top of the organization on bottom of the organization. I have seen organizations where only top level management believes in experimentation and teams don't believe in it. It didn't work out. I've seen organizations where teams and marketing teams and experimentation teams are really excited about experimentation, but top management is not excited about it and it also fails. So you have to have that buy in from everybody in the organization because it's really going to require a shift. Which brings me to the next point of a successful culture of experiment is the team structure. Most organizations, whether they're software or products or E commerce, you have your analyst teams, business analysts, marketers, whatever they might be, but the people who are doing the analysis of the business. And then you have your design teams and then you have maybe your development teams. That's kind of a typical structure. And there are silos between these different teams. In order for you to have a culture of experimentation, you're probably going to have to change the makeup of those teams. You need to break down those silos. These things need to be communicating. And it's even better. Instead of having an analysis team, a design team and development team, you actually need to have a team that consists of people from those three different teams. What I like to call as optimization pods. So now I have an analyst and I have a designer and I have a developer working together as a team as opposed to like, I have nothing to do with designers or developers. A third one. In order for you to have a successful culture of experimentation, you have to really come to this realization that there is a big difference between perfection versus making progress. Traditional design, traditional development is all about getting the piece that you're working on perfect. We don't roll out anything until we've tested it completely. We make sure that we have all the details, all the T's crossed and the I's dotted. That's not experimentation. And people struggle with that. It's a very hard shift in the mindset, but that's what experimentation is. Fundamentally, you're saying, I don't know whether people are going to like this or not. Most of the time people are going to hate what we're doing. They are going to like it. Our conversion rate might drop. All I'm trying to do is to say I have enough data to show me that there is a problem or a potential opportunity in this particular area that I'm going to be testing. I'm experimenting with different hypotheses or different Implementation of those hypotheses. And then people are going to judge. I don't have to get things done perfect. I need to roll things out quickly. And then if people send me the signal that they like a particular solution, then I can take it and make it perfect. But that shift in the mindset is challenging for many teams and many companies. You have to encourage a culture of curiosity, of asking questions. And for that, I would say that there's two types of questions that you always should be asking if you're running an experimentation program. The five whys. Why are we doing this? Why? Why, why? If you only ask why once, you're going to get the top of mind answer. Top of mind answers are not helpful when it comes to experimentation. But if you ask why five times, and I think that's the kind of a Japanese business culture is like, ask why five times. Why are we doing this while we're doing it? Because people struggle here. Well, why do people struggle over here while people struggle over here? Because the usability is not good. And why is the usability not good? Another question that I love to ask is, so what? I'm looking at this metric and my bounce rate is at 80%. So what? By the way, people get really frustrated when you start asking them, so what? So what? So what? Because it forces them to think and go from these top, very superficial answers into digging a lot deeper. And that is hard, that is challenging, that sometimes people feel very uncomfortable with. Number five, you need to have a shift in the mindset from I like this is very subjective, into data driven decisions. And I hate the word data driven decisions. Just because we've used it so much as marketers that, ugh, you know, it's one of those terms, but really, your decisions should be based on data. Sometimes you have to say, you know what? We don't have enough data to make a decision. We need to collect more data, more insights before we make a decision a certain way. That means that people have to give up their opinions sometimes. That means that me as a CEO, I'm willing to say, oh, man, I think that this design absolutely stinks. But people love it. I'm willing to say, well, we will go with what people love. It is challenging, by the way. I cannot imagine the number of times where we have a design that wins that absolutely kills it in an A B test. But an executive who hates that design comes and looks at it and is like, no way initially. Lots of times what those executives do is they initially accept the winner, but they're still struggling with it. And they look at the metrics again and again, and a month later they're like, oh, can we tweak it? And they keep on coming back and the data shows them that, you know what, you got to leave this alone. This is what people like. And this is not strictly for executives. I've seen data scientists who struggle with that, which is sort of funny because part of your job as a data scientist is to look at data and analysis data and leave the subjectivity of your opinion aside completely. But people sometimes struggle with that. And then finally, successful culture of experimentation requires you to shift from being conservative and to take a more risk, being more aggressive. If you're going to run an experimentation program and you're doing one experiment per month, I don't even know if I would call that to have an experimentation program as much as when you're doing some experiments here and there. If you want to have a growth mindset and experimentation culture, that means we're going to get a little bit more aggressive, that some of our testing is going to make us feel uncomfortable. Not all of it correct, maybe 10, 15%, but we're not going to be as conservative. If we're only testing things that we're comfortable with, then maybe we shouldn't be testing altogether. But if we're coming at it to say, you know what, we're on this journey that is fueled by curiosity to find out things that are going to make our customers life better and it's going to increase our revenue, then we are going to take some risks, risks that are driven by data sometimes. And we're going to do post test analysis, post experiment analysis to really understand what happened and take on the next correct course of action. So all of these together will help you develop a good culture of experimentation.