#3 – Culture of Experimentation, KPIs for Experimentation, Sales Funnel vs. Marketing Funnel, A/B testing tips

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Stack Simba will be bringing all the questions.

Oh man. I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today wherever you listen to your podcasts. Simba, how was your.

It's been an interesting week. We're starting to work on our okrs for next quarter.

Yeah.

Today I was speaking like with everyone in the team and trying to see if we can get like everything done by tomorrow. Because one thing about our marketing team, we are good at content production, but I felt like we were kind of like maybe slacking or dragging when it comes to content promotion. So that's one of the things that we are also like looking at going into next quarter.

Yeah.

So it's been, it's been, it's been a good work.

Yeah. It's interesting to me because every content producer, every content marketer nowadays understands the importance of content promotion. But understanding the importance of content promotion and actually knowing how to promote content are two different animals. And it's not easy, by the way. I mean, not to say that content creation is easy. I guess it depends on the topic. Correct. If you're writing about something very generalized, it doesn't necessarily like, you know, it's not necessarily very complicated. But if you write about something highly specialized, then content creation can be challenging. But with that, even content promotion becomes even harder. You need to find the right communities. You need to share your content in a non spammy way, which is not always easy because lots of times people think, and when we've been guilty of this, by the way, content promotion, okay, take a link to the article, throw it in a whole bunch of Facebook groups. No, you're just spamming the groups and it's a sure far fire way to get kicked out and to annoy. Annoy people. So what are we doing in terms of content promotion?

In terms of content promotion, we're doing like a couple of things. One thing that we started like doing recently is to create like those TLDR sections for some of our best articles. Like we, if you have been like on the invesp blog, you know that we write like long form content so we had to create those TLDRs and then we take those TLDRs and we also promote them on different platforms like Medium. And we are also like looking into LinkedIn articles going forward and also creating like different segments of that one piece of content and sharing it like as a Twitter thread and also sharing that as a LinkedIn post and also on Instagram. So that's one of the things that we are also like going to be focusing on like next quarter. We also have like a guy who's good at SEO who's been like doing our SEO. So he's been like coming in with a lot of suggestions. So that's also some of the things that we are taking into consideration as part of like distributing our content and stuff. But it's going to be an interesting quarter.

So if I'm producing a content piece and I'm coming here saying Simba, like you know, here I'm going to produce this content piece, what is the breakdown of the time between the research, the writing, slash, editing versus promotion kind of. Those are maybe the three components that I think of. Unless you can think of other. What are your thoughts on those?

I think when I look at it like from like the perspective of how we have been doing things this quarter, we spent like maybe I would say like 80, 20, 80% of the time writing, producing the content and then maybe just like taking a bit of it and sharing it on LinkedIn on Twitter. I mean but we're not like just putting enough effort on that of which I'm a big fan of. Rose Ross Simons from Content foundation and he always like speaks about like you can create one piece of content today, but you can always be distributing it. So that's one of the things that we are also going to be doing one piece of content and then like come up with at least maybe 6, 7, 8 posts for different social media platforms.

I like that. And sort of interesting. I've been looking at like in some of our LinkedIn feed. By the way, it's really fascinating to me how Invest the Invest page and its followership has grown. We're at 13,000 followers. And I'm like, wow, look, all these people are just 14.

We're getting to 14.

Oh, okay, okay, 14,000. And even, by the way, Figpi Fitpai is doing really well because you get the growth there. So it's just. It's just fascinating. And of course, when you're below a th000, by the way, let's say, like now Fikpi, I remember 200 followers. You go to, let's say 300 followers. That's a 30% growth, which is impressive if you think about it. But if you're at 14,000, growth is like, you need to get to 20,000. So it's a lot, a lot harder. But it's just fascinating looking. Looking at those numbers. But okay, so a lot of exciting things.

One other thing before we start.

Yeah, tell me.

We also launched, like, this CRO Live hour and turned it into a podcast this week.

So how is that, by the way?

Yeah, so we launched it yesterday, and the listenership is. It hasn't. Is not bad, but it can always be improved since it's like the first episode. But people have been, like, tuning in, downloading it, listening to it. So whoever is like, maybe watching us again can also tune in at some point and follow us and subscribe to our podcast on Spotify, Apple Podcast and different whatever. Like, they get their podcast. Yeah, that's awesome.

That is awesome. So I'm super excited about that, actually. It's funny, I'll tell you. So I'm recording also the podcast boot trapped with IATs. Correct. And yeah, sometimes, like, you're like, oh, you're recording, but you don't get feedback and you don't know if you're doing well or not. But I've gotten several messages, and Ayat was in a meeting. Like, you know, a couple of people, they're like, hey, we've actually been like, you know, listening to your podcast, and it's so amazing. And there's one lady who actually wrote a blog post about it. I'm like, whoa. Okay. I mean, in order for somebody to write about it, I'm like, oh, must have, like, you know, must have been, you know, we've done well, but what questions do you have? You have for us? Okay, so today, by the way, we are so behind. I know.

Yeah, yeah, I know, I know. Yeah, true. So today we have about six questions to answer. So the first one is, what are the components of a successful culture of experimentation?

Fascinating question. And then it's an Interesting question. If you think about, if you think about the culture of experimentation and what that takes. And before I answer that question, let me say that if you work with Diff, as you work with different organizations, what you will discover it is one thing to run an experimentation program. It's a completely different animal to have a successful experimentation program. And there needs to be a shift in the mindset and the culture of the company in order for experimentation to really take its roots within the organization. I always joke and I tell people, nowadays we think about management as something taken for granted. Team structure, how we run business at the very foundation of how businesses run. Well, 100 years ago all these concepts were new. And then if you've taken any business classes, you see kind of the evolution of organizational structures and introduction of management. But in 2022, who's going to run a company without having some sort of management structure or team structure? I think it's the same thing for experimentation nowadays. It's new, we're trying to talk to people about it. But I think you fast forward 50 years from now is going to be just, hey, it's taken for granted. It's going to become a way of doing, of doing business. I think a successful culture of experimentation has six different components of it. First, it starts with a complete buy in and buy in and understanding of what it would take to have a culture of experimentation. And that's complete buy in needs to happen. Happen from top of the organization on bottom of the organization. I have seen organizations where only top level management believes in experimentation and teams don't believe in it. It didn't work out. I've seen organizations where teams and marketing teams and experimentation teams are really excited about experimentation but top management is not excited about it and it also fails. So you have to have that buy in from everybody in the organization because it's really going to require a shift. Which brings me to the next point. Successful culture, experimentation, which is the team structure. Most organizations, whether they're software or products or E commerce, there's certain team structure, right? You have your analyst teams, business analysts, marketers, whatever they might be, but the people who are doing the analysis of the business. And then you have your design teams and then you have maybe your development teams. That's kind of a typical structure and there are silos between these different teams. Well, if in order for you to have a culture of experimentation, you're probably going to have to change the culture, the makeup of those teams because you need to break down those silos. These things need to Be communicating. And it's even better. Instead of having a, an analysis team, design team and development team, you actually need to have almost team that consists of people from those three different teams. So what I like to call is optimization pods. So now I have an analyst and I have a designer and I have a developer working together. Correct. As a team. As opposed to like, oh, well, I have nothing to do with designers or developers. So that's second one, a third one. In order for you to have a successful culture of experimentation, you have to really come to this realization that there is a big difference between perfection versus making progress. Traditional design, traditional development is all about getting the piece that you're working on perfect. We don't roll out anything until we've tested it completely. We make sure that we have all the details, like, you know, all the, all the T's cross across and the eyes dotted. That's not experimentation. And people struggle with that, by the way, because it's a very hard shift in the mindset. But that's what experimentation is. Fundamentally, you're saying, I don't know whether people are going to like this or not. Most of the time people are going to hate what we're doing. They don't like it, our conversion rate might drop. Thus, all I'm trying to do is to say I have enough data to show me that there is a problem or a potential opportunity in this particular area that I'm going to be testing. I'm experimenting with different hypotheses or different implementation of those hypotheses. And then people are going to judge so I don't have to get things perfect. I need to roll things out quickly. And then if people send me the signal that they like a particular solution, then I can take it and make it perfect. But that shift in the mindset is challenging for many teams and many, many companies. You have to encourage a culture of curiosity, asking questions. And for that, I would say that there's two types of questions that you always should be asking if you're running an experimentation program. The five whys. Why are we doing this? Why, why, why? If you only ask why once, you're gonna get the top of mind answer. Top of mind answers are not helpful when it comes to experimentation. But if you ask why five times, and I think that's the kind of a Japanese business culture is like, ask why five times. Why are we doing this while we're doing it? Because people struggle here. Well, why do people struggle over here while people struggle over here? Because the usability is not good. And why is the usability not good? Well, because. Continue asking those questions. Questions. Another question that I love to ask is, so what? I'm looking at this metric and, you know, my bounce rate is at, you know, 80%. So what? Well, you know, you all. And then, by the way, people get really frustrated when you start asking them, so what? So what? So what? Because it forces them to think and go from these top, like, you know, very superficial answers into digging a lot, a lot deeper. And that is hard, that is challenging, that sometimes people feel very uncomfortable with. You need to have, and this is number five, you need to have a shift in the mindset from, wow, I like this. It's very subjective. Into data driven decisions. And I hate, by the way, the word data driven decisions. Sort of funny just because we've used this so much as marketers that I'm like, ugh, you know, it's just one of those terms. But. But really, your decisions should be based on data. And sometimes because your decisions are based on data, you have to say, you know what? We don't have enough data to make a decision. We need to collect more data, more insights before we make a decision a certain way. That means that people have to give up, correct their opinions. Sometimes that means that me as a CEO, I'm willing to say, oh, man, my opinion, or I think that this design absolutely stinks, but people love it. And I'm willing to say, well, we'll go with what people love. It is challenging, by the way. It is challenging. You cannot imagine the number of times where we have a design that wins that absolutely kills it in an A B test. But an executive who hates that design comes and looks at it and is like, no way. You know? And then initially, lots of times what those executives do is they initially accept the winner, but they're still struggling with it. Correct. And they look at the metrics again and again, and a month later they're like, oh, can we tweak it? And they keep on coming back and the data shows them that, you know what, you got to leave this alone. This is what people like. And this is not strictly also for executives. I've seen data scientists who struggle with that, which is sort of funny because part of your job as a data scientist is to look at data and analysis data and leave your opinion and the subjectivity of your aside completely. But people sometimes struggle with that. And then finally, successful culture of experimentation requires you to shift from being conservative. Correct. Into taking more risk, being more aggressive. Correct. With your. With your Testing. If you're going to run an experimentation program and you're doing one experiment per month, I don't even know if I would call that you have an experimentation program as much as when you're doing some experiments here and there. But if you want to have a growth mindset and experimentation culture, that means we're going to get a little bit more aggressive, that some of our testing is going to make us feel uncomfortable. Not all of it correct, maybe 10, 15%, but we're not going to be as conservative because if we're not, if we're only testing things that we're comfortable with, then maybe we shouldn't be testing altogether. But if we're coming at it to say, you know what, we're on this journey that is fueled by curiosity to find out things that are going to make our customers life better and it's going to increase our revenue, then we are going to take some risks, risks that are driven by data sometimes. And we're going to do post test analysis, post like, you know, post experiment analysis to really understand what happened and take on the, the next correct course of action. So all of these together, working together will help you develop a good culture of experimentation. So here's another question that I got, and this question is interesting. What KPIs do you use to track the success of experimentation? What KPIs do I use to track the success of experimentation? So there's several KPIs that we use to track the success of experimentation. And I think they come in three different categories. There's financial KPIs, there's usability KPIs, and then there is cultural KPIs. And it is important that you keep track of all those three KPIs or three different categories to make sure that your program is running smoothly. The first one, when it comes to financial KPIs, this will depend on the type of project that you have. So you could be tracking overall revenue for the property or the platform that you are working on. You could be tracking metrics, as much as I don't like it, conversion rates. We kind of talked about conversion rates and how conversion rate is a difficult metric to track. I like to track revenue per visitor a little bit better. Even more interesting, customer lifetime value, customer retention. Those are really interesting metrics that you want to track in terms of financial impacts of a project. Another set or another category of KPIs that you track is about the user experience on the website because you can do something in the short term that helps you generate more revenue, but in the long term, it's just not good for user experience and might leave a bad taste in the mouth of your visitors and that would kill your company. And I always mention a story of a company that worked with long time ago where they made a good chunk of their money on the, in the shipping and handling charges, which they insisted on hiding from people. They had it there during the checkout, but it was really hidden. And we went back and forth with them, back and forth with them about, hey, this is probably not ethical. And we were saying, we were trying to be nice by saying probably not ethical. We knew it was not a good way to do business. They wouldn't listen to us. They end up going and they came back to us after a few months when although their conversion rate in the short term helped go up, but guess what? Every time somebody bought from them and then they noticed afterwards after they received the order, the amount of money that they paid for the shipping and handling charges, people were getting pissed off and they would not buy from them again. Well, eventually the conversion rate tanked and that really killed their business. The business almost died as a result of this. So you want to look at usability and user experience metrics. And there's many metrics that you can track. You can track to evaluate where the site is. You can look at things like task success ratio. You want to look at like, you know, cognitive load. And you can do some analysis around that you can do, you can basically do use lots of metrics to track the user experience on the website. You can use NPS scores to track also how users feel about your site and about your service. And then finally, the last group of metrics that I use or KPIs that I use to track the success of our project is about experimentation. Are we able to have this mind shift? And people are adapting experimentation as a way of doing business. I always give the example of one company, the most expensive AB test, where they hire us, this is now a long time ago, maybe eight or nine years ago, they hire us to do experimentation. It takes us six months to launch the first A B test, 23 revisions. And the A B test did not generate any results. So we're sitting there, we're talking with the CTO who was our sponsor, and he's like, one AB test in six months. And basically we're like, yeah, that's basically what happened. We had one test in six months and it failed. Okay. So that was kind of the initial culture of that company. And Then we had a major shift where now their teams have changed. It's no longer development, design, quality control and project management. Now every. They have optimization pod and now authorization pod consists of different team members from different teams. Every single feature that gets released on their website is actually tested, A, B tested. And that's how they make the decision whether they want to move forward with feature or feature or not. That is a complete shift in culture. And that's another KPI that perhaps a little difficult to measure, but it's very important to use to judge the success of an experimentation program.

Yeah. Just to recap what you said, or maybe let me just go ahead. We are actually like writing an article about how to build a culture of experimentation. And I was interviewing like a bunch of people and one is from booking.com and the other one is like, used to work at God Daddy. So like, they know how to, how to talk about this topic of experimentation. So one thing that they also spoke about is like the democratization of this experimentation as well as something to do with the transparency as one of the major components of a culture of experimentation. The guy, like, from booking.com, his name is Jordan, he was saying that what they usually do is they are very transparent when it comes to experimentation. Like each and every test, each and everyone in the team knows what's happening, like, who's testing what, you know it. And he was saying that the reason for doing that is they are trying to. They try to improve like the cow. The. The quality of tests that they run. Because, like, if you know that, like, everyone knows what kind of tests that I'm running, you try to aim higher, you know, you'll be like, pushed to do that. So. Yeah, I think that's the point that I wanted to add.

Yeah, I love that. I love that. Yeah. What do they usually say about democratization of testing? Because that's, that's an interesting concept that I have some thoughts around, but I'd love to hear. What, what are some of the thoughts that you've heard about?

Yeah, one of the things that they actually mentioned about the democratization of experiment is people have to take ownership of experiments, you know, like, it's not one, like, as in the person who come up with the hypothesis is the one who's supposed to own it and not leave it, like to data scientists, you know. So, yeah, I think that's one major issue that they mentioned about. And the rest people would have to read the article to get it.

I like that. I look forward to reading the article, but I Mean, it's an important article because it's funny. By the way, I wouldn't have worried about culture of experimentation, by the way, 10 years ago. I just simply did not worry about it, didn't think about it. Nowadays, as a matter of fact, I have a call in about 30 minutes. With a large company, they're thinking about bringing Invesp on board to help them with experimentation. But the main focus is about the culture of experimentation. How do you take an organization that is very well established with this processes and development team, and they even have their own in house, like, you know, AB testing software, how do we actually help them change the mindsets and move into an experimentation mindset? It's not easy. And I was talking to their cmo. I told him that's perhaps going to be the biggest challenge. Process easy, we know process. How do we run with high velocity? We know it's in and out. How do we make sure that testing will deliver results? We know how to do that. The biggest hurdle is the culture and having the shifts in mindsets of what it means to be able to run experimentation. Not. Not very easy. And perhaps the hardest part of doing all the work that we do. Yep.

So let's move on to the next question. What's the difference between a sales funnel and a marketing funnel?

What is the difference between a sales funnel and a marketing funnel? Gosh, man. The difference between a sales funnel and market funnel is so fuzzy because both are invented by people thinking of selling to people. But they're thinking. Thinking of it from their own perspective. Correct. They're not thinking of it from the buyer perspective because marketing teams have their own marketing funnel. And then sales team have their own sales funnel. But guess what? Your customer has their own funnel. And guess what? The customer is the person who controls the buying process. Whether you want to acknowledge that or not. Marketing funnel usually is very focused on that initial intrigue that people are aware of the brand. They know that you exist. Correct. They know that you're here and you're there at top of their mind. So as marketers, we always think three different things. Correct. Top of, top of funnel, contents or literature or assets. Middle of the funnel and the bottom of the bottom of the funnel. Correct. And the goal is we want to move a particular person who doesn't know about us. So that's top of mind. Correct. Top of the funnel. So, hey, there's this thing that exists out there. Your prospective clients are. They're not even a client at that point in time. They may or may not Be aware of the problem that they have and they might be aware of the problem but they don't think it is worthy of finding a solution. For I have many problems that I'm dealing with and I'm like, you know what, it's fine. The pain is not large enough for me in order for me to find a solution that's top of the funnel for me in marketing. And then somebody at some point the person who's buying the buyer says okay, maybe I'm struggling a little bit, I need to find a solution. So they move on to mid funnel and then at some point they say, you know what, I really gotta evaluate that at the bottom of the funnel I gotta evaluate different options to solve this problem. It is painful enough for me to find a solution for this. So this is what I think of as a marketing funnel. Think of the sales funnel and see how they're like, you know, they really intertwined so closely. So every time we think about the sales funnel we think about ada correct awareness, interests, decision and then action. Those are awareness. I'm aware of the problem, I'm aware of the brand. Correct interest. I am interested in finding a solution decision. Okay, now I'm decided that I need to act or I've decided to like select this and the action I'm actually converting. It's almost, it is almost as if the marketing funnel is about the first two stages, correct, the awareness and the interest. That's what you know, what the marketing funnel does is like the brand exists, the problem exists and then marketing takes this person, they're ready to make a decision and it takes them from well hey, here, here's an mql, a marketing qualified lead. I'm going to hand them to sales. Now this is an elite gen type website, correct or lead gen type business. And now the sales team has to take this MQL marketing qualified lead. That's yeah, they have interest and now we're going to say oh well you know, know, do we accept them, they're really qualified and then we'll take them as an SQL sales qualified lead and take them through our sales funnel. There is high intersection between, between Those, between those two funnels in a lead gen type business, a SaaS type business. It's very clear. Correct Marketing does a whole bunch of activities to generate that initial interest to let people know about the problem and then at some point hands them over to sales. Sales. Sales manages to close the process. In an E commerce website, it's a little bit different. Marketing is all about brand awareness. That we exist, you might have a problem. And then the minute somebody lands on the website, you're an E commerce website. There's no sales team in traditional sense. Correct. The website is their sales team. And now it's like oh well you brought me somebody. Are they really qualified? This traffic that's marketing is bringing me, are they qualified? And how do I judge? Are they even going to my product pages or are they not going. Let's say they're not going to the product pages. Then I'm asking why aren't they going to the product pages in my sales funnel on the website? Am I stopping them because I'm throwing a whole bunch of barriers in their way or no, they're getting there and they're clicking on the ad so they really starting the process of conversion and what do I need to do to convert them? Again it will really depend on the type of business and the lines are so fuzzy between them that I always and maybe just because driven that with the fact that we work with E commerce. I always think of the mix between the two. I always like to think of ada, you know, and creating for me top of the funnel. Marketing campaigns are very focused on the awareness and the interest. And then the middle of the funnel is about like okay or bottom of the funnel, the decision and the action stages. It's a fuzzy answer. Because it's a fuzzy question by the way. Not, not an easy question to answer. Yeah.

Now that you have answered it like the way you answered it. I was like listening. So some people, they also like, like to talk about the conversion funnel. Is it any different from those or it's just the same thing?

Same thing, same thing. We listen. I'm sure there is somebody out there who's very smart who can really create the distinction between those. I'm sure you know, that's not going to be me sometimes splitting hairs. Correct. If you think about it really is splitting hairs. And I guess it depends on the organization. I mean if you're a large organization, you have a marketing team and a sales team then usually like, you know, marketing is very focused on getting those MQLs and all the activities that they do. And there's different types of activities. So there's the brand awareness campaigns and then there's the conversion campaigns which are bringing people just to generate leads and then the SQL, the sales team has to work with those leads and take them through the different stages of the sales funnel and that each one of those is very well defined. I was looking at the sales funnel for Adobe, for example, I mean, the sales funnel, after marketing says somebody is qualified and they hand over that MQL that's marketing qualified lead to the sales team. The sales team has to go through 13 different steps before they convert somebody. You know, initial discovery, pain, like, you know, analysis of need. Oh my God. So many different steps. So that's, yes, they have a very well defined marketing funnel and a very well defined sales funnel. But, and in this case, because it's such a large organization and there's two distinctions, you know, you can define those clearly smaller organizations, you see that like in the small, smaller the organization, the more marketing and sales become combined. Although they're different. Correct. And there's different stages that the buyer goes through. But yeah, a conversion funnel eventually to me is focusing on the buyer and how do they convert and what information they need. So maybe it's a combination of the marketing and the sales.

Yep. So moving on to the next question is.

Thank you. Thank you. I saw it and I'm like, man, you know, there's nothing.

Yeah, sure. So you have had loads of clients, like in the past few years. What makes you say no to a prospective client, to someone? Suppose you're in the sales call. What makes you say no?

So this is a really good question. And by the way, this is one of those questions that I always tell clients during our calls, prospective clients, I tell them, let me walk you through our deal breakers. There are a few things that basically, if you don't, if you're not comfortable with them, then we're not the right company for you. And I'm very clear about that. And it's very important to me to spell out the deal breakers and make sure that we're on the same page. And if somebody is not comfortable with one of those deal breakers, it's better just to say, you know what, we will walk away. Because if you try and accommodate somebody and change your process and culture, eventually they're not going to be happy, you're not going to be happy, and the relationship becomes really short. Instead of a one or two or three year relationship, it becomes a three or four months. And I'm like, okay, that's not interesting. The amount of effort that you put in the first three or four months in the relationship is tremendous. Correct. And then somebody decides to walk away, or you decide to walk away, then you've lost, they lost their time and money and you've lost your time and you could have done, you could be doing something better. So what are some of the deal breakers Some of the deal breakers that we have. First we tell them that we have an MVP approach. Everything that we introduce on the website is not perfect. Our goal is not to produce the perfect design and the perfect code. We come at every experiment saying that we have enough data and we have good analysis over here that tells us there is a possible issue. We are going to create designs to solve that issue. And then we're going to introduce an experiment and a B test for people to judge which one of those designs is better. There's a good chance that people might hate all of our designs. That means I don't really need to get the designs perfect. Correct. I'm just testing out the theory. And people will give me enough signals to say, you know what? Okay, this design is good. Now I need to go back and get it in a better shape. Some companies struggle with that. They want to get everything perfect. Those companies will struggle with investment. My goal is not to get everything perfect. My goal is to get everything to 90% perfect. Now, at the same time, I'm not doing something that's going to embarrass you as a brand, and I'm not going to go outside of your brand guidelines. But if we're going to sit there and discuss pixels, 10 pixels versus 5 pixels, no, we're not the right company for you. Number two, I tell them, if you're gonna hire us, if you're gonna work with us, we're gonna do it our way. And people always look at us like, what do you mean we're gonna do it our way? I tell them, listen, we know conversion optimization really well. We've done it long enough. We have more a B test than any other company worldwide. And I'm very comfortable saying that if you're coming to us, you gotta say, I trust you guys. Every once in a while, we run to a company that's, hey, well, you know, my cousin or my uncle runs a digital digital agency. Or I read this article like, you know, and I want to test that. I tell them, listen, if you're going to hold me responsible for the results, you got to let me do my own thing. Now. People struggle with this. They're like, oh, well, does that mean that we don't have a say? No, of course you have a say, but you got to trust us as experts in what we do. A third thing that I tell them that we definitely want to take is that 10% of our experiments will make you feel uncomfortable. Not right away, not in the first six months. Because the first six months for us on a project are all about establishing credibility and trust and showing you that we can deliver really good work, really good analysis, really good results. But after the first six months where you say, you know, those guys are really good, I trust them, they know what they're doing. This is the best decision I've made to hire them. Some of our testing is going to make you feel, oh man, really? This is what we're going to be testing? Yes, this is what we're going to be testing. Because if our testing and experimentation is strictly in the areas that you're comfortable with, then we're not going to produce the results that you, you hire us to produce. So that's another thing that we tell them. People sometimes are comfortable with this. Every once in a while you run to somebody who says, no, I don't want to do this, I want to tell you what to do. And I tell them, listen, if you're going to come there and you want to run testing yourself, there's probably a lot cheaper ways of doing experimentation than hiring. Invest. You can just hire some developers, front end developers and designers, and let them do their own thing and listen to you and do your own thing. Then finally I tell them the time commitments. There are agencies out there that will tell you, oh, we'll work with you for three months. And we don't do that. We ask every company that we work with to at least do 6 months or 12 months work with us. And then some people struggle. And I understand why they struggle. Correct? Because they've been burned by other agencies that sign them up and don't deliver results and then they feel that they're stuck. And I tell them, my goal is not to have you stuck. My goal is to deliver results. Six months relationship is not very interesting. One year relationship is not very interesting. Companies with us are for multiple years at this point in time and they're with us for that many years because we maintain an amazing relationship and we deliver good value. And this is not an advertising way to invest, but this is the way kind of after doing conversion optimization for so long, we know what we can deliver. We know the value that we bring on board. Some people don't appreciate it and it's fine. We're not the right fit for them. There's other agencies probably that will be the right fit. Those are the things that make me say no to somebody. So we have about 10 more minutes. Simba, your last question for the day, by the way.

Okay, so what are five best tips for running a B tests?

Really? Such a long. Yeah, just give us five.

Just give us five.

The standard answer. Five tips to run successful AB testing. Understand your markets, what they're looking for and how you can satisfy that need. And it's interesting to me because I always say, every marketer says or talks about the importance of understanding customers. Very few marketers takes the step necessary to really understand their, their markets. I love to attend Jobs to Be Done interviews, and yesterday I did the Jobs to Be Done interview for one of the companies I'm working with, invesp right now. And they sell. Basically, this company sells earplugs. And I'm interviewing a musician. And she explains specifically how the earplugs that she bought from this company first stop all the ringing that she was getting before using those earplugs. She's still able to hear the details of the musical instruments and the beauty of it. She said, when I use it in concerts. She was attending a concert, listening. She had her earplugs and she's like, you know, my brother was standing next to me and it turns out that there's somebody standing behind us who was talking during the concert. And my brother was really annoyed because, because he could hear that guy talk. And she's like, because of my earplugs, I didn't even hear any of that conversation. That is gold mine. Because we did not even know that feature. We didn't focus on it. Understand the customers, give them what they want. Use their words. That is very, very critical. Another rule for successful AB testing, Steal. Don't copy. Don't copy blindly what your competitors are doing, but be inspired by it. Some optimizers say, oh, don't look at your competitors. No, I look at my competitors, but I don't copy what they're doing. I take the ideas and I say, hmm, this is interesting. I wonder how people react to it. How can I make it better? How can I test it on my website? So still don't copy is another important rule. Track results. We just Talked about the KPIs for successful conversion optimization. Track results. Track them very closely. What is the financial impact? What's its usability impact? What's the cultural impact of my experimentation that really helps you deliver a successful experimentation program. Build credibility within your organization. Correct. The hardest part of an experimentation or a B testing program are the first three to six months, by the way. All eyes are going to be focused on what you're doing, the work and the analysis that drives you to test something. How you are addressing those problems. How are you creating designs, what are the test results? We win over losing. What are the insights that we're getting? Be very transparent and share and get people excited about the program. Fundamentally, be curious, ask questions, good experiments, good a B test. Start with really good questions and they end up with even better questions. You have to ask questions, you have to be curious. There is no such rules. The best AB experiments that I've done, the best experience I've done is when I hear somebody says, well, we've always done it this way. I'm like, oh, perfect. That means there's an opportunity for improvements there. But also it means, by the way, that I have an uphill battle sometimes to convince people to do things differently, to test things out. I know you've done it this way for the last six years. Why don't you allow us to show you a different way of doing business that might be better for your customers and for your bottom line?

Yep, that was a good one. So I think for the sake of time, we will stop here today, right?

Or you have, because I remember I told you I have a call with a company that's going to be doing this presentation. So I have called them. 10 minutes. I've already prepared. Like, no, but it will be an interesting, an interesting call. Well, Simba, my friend, we shall meet next week. Thank you everybody for listening. And as usual, if you have any questions, we get your questions from you. Simba's on LinkedIn, I'm on LinkedIn, just drop us a message. And since the CRO Live Hour is live as a podcast, make sure that you search for it wherever you listen to your podcast. Search, find it, subscribe, and until next week, take care.

Bye bye.