#55 – Questions To Ask Before Hiring CRO Agency

Every experiment has multiple ways, multiple solutions that we think will address the problem. Okay, so what are the insights that we are driving from knowing that this particular medicine actually worked to solve this problem while the other two medicines did not work? What can we learn about our website when we say what's using the number of people who have placed an order with us using that in our headlines actually increase trust and increase social proof and it increase our conversion rates by 10, 15% while using influencers as a social proof in our headlines and our hero image actually reduce our conversion rates. What are we learning from this? My name is Khalid Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions to dropping insights like it's no big deal.

Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we'll talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today wherever you listen to your podcasts. What a start for 2024. Simba, happy new Year. To start with. How have you been?

I've been good. CRO live hour still going from 2021, right? Yeah.

Is it 21 or 22? Yeah, 21.

Yeah, it must be 21.

Man, we're going like 21, 22, 23, 24. Okay. Yeah, like, this is funny math, but yeah, we've been going on and recording since November 2021. So that's when we started. We're still going strong and until now, every few episodes. I tell you, I'm like, simba, we really should figure out how many podcast episodes we've recorded. How many of those LinkedIn Live. And you tell me.

Yes, I think we've posted about 50 something podcast episodes. But it's just that, like, we started posting podcasts later on.

Yeah.

That we've studied.

Yeah, but you should be able to tell us exact numbers. We are an experimentation agency, an experimentation software. We should be like, we posted exactly that many here. But anyways, I'll let you figure this one out. How many LinkedIn lives we've done, how many podcast episodes? Podcast episodes I think we can figure out easily because we use one Software. But that LinkedIn lives, that's a little bit more complicated. But okay, so new year, I'm sure there's a million things going on, more experiments, more things to try out. What's the latest and greatest?

The latest and greatest. We had some marketing plans that we had like, that we started preparing, like, the marketing plan sometime last year, around November, December. So right now we're starting, like, to implement some of those ideas that we have. I really can't tell right now what will work, what won't work. Maybe I'll have to come back after about five weeks and just, like, maybe give an update of what. What have we done so far and what's really happening.

I love that. I look forward to seeing basically what worked, what didn't work. I love what works. Yes. We learn from what didn't work. Like, when you're paying for it, I'm like, oh, that was painful lesson. Guess you always learn from those painful lessons.

Yeah. How about you at your side. What's happening?

Oh, gosh. It's been an interesting few weeks because we went. We spent December planning our sales and marketing activities. I'm shifting my focus lots of more into thinkpi, trying to figure out what's going on with fitbuy. We've been growing at FITBI tremendously, to the point that I'm like, okay, I need to really focus on this growth because it's moving a lot faster than we anticipated. So it's exciting. Makes me feel anxious, which is an interesting feeling. I'm like, huh? Why would I feel anxious? But when you start achieving success, I'm like, okay, what do I need to do to maintain that success? And I have seen some of the reporting that INVEST was starting to put out for its internal clients, and it's been just very impressive. So overall, happy, anxious, excited to get started with 24 typical feelings an entrepreneur will have.

And hopefully, like, this year, we'll be back on LinkedIn. Posting often, I think, like, sometime last year, like, we slowed down on LinkedIn. I'm not sure about you, but for me, I know that, like, I was super busy because for me to come up with a good LinkedIn post, sometimes I have to think and then I have to take time to respond, like, to the comments. I don't just want to post like something just for the sake of posting.

So yeah, yeah, I'm with you. It's funny because I did post something this Monday and then we're doing the LinkedIn live and I'm like, khaleed, you need to be a lot more consistent. I mean, we know the formula. It's just a matter of figuring out the time and also trying to. And this is the challenge that I have. What do I post about? Because I can write about running ThinkPy, running Invesp, experimentation, dealing with clients. And it's like, you need to focus on a niche correctly. Here's my thing, here's what you're going to know me for. And the problem is, like, whenever you write whatever comes to your mind, that's not a good idea for any of those platforms, any of the social platforms. I will figure this out and see how it goes.

Okay, that's cool. So today we have about six questions that people often ask before they hire, like a CRO agency. But I'm sure like some of the questions they do apply to people who want to hire, like maybe any marketing agency. You just have to think about those. You just have to ask those questions, find those answers to those questions before you engage, like any company. So let me know when you're ready, then I'll ask the first one.

I am ready. Shoots away, my friend.

So the first question that I have is, what is the most challenging part of doing conversion optimization project?

What is the most challenging part of doing a conversion optimization project? I think each stage of doing a conversion optimization project comes with its own set of challenges. And I think it starts starts before even launching that first experiment, before even engaging in a conversion rate optimization project. Think about this. You are a business owner, you are a VP of marketing, you are a director, you are a co founder. And most of the time when people raise their hand and they say, hey, I think we should focus on conversion optimization, two things have happened. They identified that, hey, we have a problem over here. So yeah, our conversion rates are just going down. And at the same time they're saying to themselves, you know, I've seen this case study, I've heard somebody talk about conversion optimization. One of the challenging parts of conversion optimization is that many companies, many stakeholders, many business people come into conversion optimization with the wrong expectations. They think is the silver bullet. It is the one solution that is going to solve all of their problems. I was talking to a company late last year and they were doing 30 million in annual sales. Great talking to their VP of marketing. And he came to me, he said, next year I want to go from $30 million to $45 million. That's huge growth. That's 50%. 50% growth. When you're doing $100,000 in sales, to go to 150 easy. But to go from 30 million to 45 million, that's quite a bit of growth. And I'm like, okay, so let's talk about the plans. What are you going to do in order for you to achieve that? He says, well, yes, conversion optimization. In my mind, I'm like, okay, that's too many red flags because you are expecting way too much from any conversion optimization team that is going to work with you. So that was worrying. And here's another funny part. I'm like, how much money do you want to spend to get that 50 million growth? What kind of number do you think he threw at me? Just give me a number, Simba, what do you think?

I would just say 500,000 to a million.

Okay, so if you spend a million and you get 15 million, that's 15x return. Correct. Aggressive. Usually I'm happy with 5 to 7x return, maybe 10x return he wanted to spend. You know, he's like, oh, my budget is $100,000. I'm like, okay, I just don't even understand this. I was like, trying to do the math. I'm like, so you're expecting about 150x return? He's like, yeah, I didn't think of it this way. I'm like, no, you need to think of it this way. Because if you can figure out any business activity that's going to generate 150x, I'm like, tell me about it, because I will do it in a heartbeat. And it's just so unrealistic to have that expectation. So that's one of the challenging areas of conversion optimization, Walking into the project with the wrong expectations. One of the other areas that are challenging about conversion optimization is identifying different problems on a website. Now, people use different methods. Everybody talks about expert reviews. This is when somebody who's an expert reviews your website, comes up with ideas based on his or her experience, that's fine. The other two areas that we don't delve a lot into, and I think many people pay them just lip service is analytics and customer research. Everybody tells you, oh, of course we look at analytics. Everybody tells you, oh, of course we talk to our customers. That's fine. That's what people say. But let's talk about reality. Most People, when they look at analytics, they deal with it at a very simplistic level. I look at it, I'm like, really? That's the analytics analysis that you're doing? You're looking at a couple of numbers and you're trying to uncover major problems on a website? Probably not a good approach. You talk to VPs of marketing and you ask them, how happy are you with your analytics implementation or the insights from it? 93% of them say, we're not happy with it. But you talk to marketers who work for those VPs of marketing and they tell you, oh, yeah, we are using analytics. We are drilling through the data. Guess what, your boss doesn't think that way. We all have analytics, but that doesn't mean that we have the insights. Every marketer out there, every marketing book tells you that you need to focus on customer research. You go to conferences and you hear that, okay, so that's great. Lip service again, what does that mean? What does it translate into? How often are you actually talking to your customers? If you're having those conversations, what questions are you asking? Are you guiding them and asking them questions that guide them to specific answers? Or are you asking the right questions so you can take insights from, from those questions? To me, the part about coding and a B test is the easiest part, by the way. I'm like, okay, that's with the right developer, with the right designer, you can come up with that. But let's talk about perhaps as you engage in a conversion rate optimization experimentation project about one of the harder areas, which is finding the insights from an experiment. An experiment is trying to solve a problem. Hey, if I may use the expression there's a disease, the website, let's try and fix it. We're going to give it this medicine, we're going to give it that medicine. We're going to maybe take some X rays by looking at the analytics and we're going to give this medicine, that medicine. Let's see which one worked. Every experiment has multiple ways, multiple solutions that we think will address the problem. Okay, so what are the insights that we are driving from knowing that this particular medicine actually worked to solve this problem while the other two medicines did not work? What can we learn about our website when we say what's using the number of people who have placed an order with us using that in our headlines actually increased trust and increased social proof and it increased our conversion rates by 10, 15% while using influencers as a social proof in our headlines and our hero image actually reduce our conversion Rates. What are we learning from this? Maybe we're learning that people don't trust influencers, in this case, and they trust more the other human connection. Now, imagine if you're running 50, 100, 200 experiments, finding those insights and saying, oh, we've learned this from here. How can I take this insight and actually apply it across many channels, across many experiments? And then the final piece, I think this is the third time I say this is the final piece. But, okay, how do you actually draw a relationship between experiments that you've implemented and money in the bank? I've launched 15 experiments. I've launched 40 experiments. How much money did I make as a result of these experiments? Now, most CRS will tell you, oh, it's a little difficult to establish a relationship between the number of experiments or an experiment that you've launched and a dollar value. Okay. If you want to justify to a stakeholder, to a CEO or a VP of marketing or to a founder, if you want to justify to them why they should be doing experimentation and why they should be doing marketing, you got to show some ROI at some point. I don't even call it ROI anymore. We call it roc. Return on change because we've made some changes. What was the return on those changes that we've made? It's very complicated math. It is not very easy math. And the core requires a lot of work. We've had almost five people working at it at invesp, just with so many different complicated formulas, but you don't arrive at an exact dollar value. But we definitely come very close. I would say with an 85% confidence that, okay, here launch that many experiments. We've generated about $1.5 million for you. We've generated $7.1 million for you. I think that really helps you guide the program and continues the excitement around the program. So those are some of the challenges when it comes to running a conversion optimization project.

I was saying, like, that's a good explanation. And as you were explaining, especially like in part two, I was thinking of the book that you often recommend. I finally had a chance to. Not really had a chance just out of, like, maybe buying it and listening to it. They wrote less stupid, I'm sure. Like, it's one of your favorite books because you talk about it, like, all the time. So as you were explaining, I was just thinking about when you were saying that it's difficult to find the insights, especially maybe even from analytics or something like that. Then I was just thinking of, like, maybe finding the unexed questions as in, it's one of the difficult things to do because, like, most of the problems that people do have, like on their website, it's just like those questions, those are just like questions which are unexed. Or sometimes maybe people might just go like on their analytic tools. I think in fact, the main problem that people have is they go to the analytic tools sometimes looking for the answers instead of coming app with more and more questions from them.

Yeah, it's funny because I was talking to a friend yesterday and we're working on like a side project, just kind of volunteer work. And he's like, you know what? Here's the data. I have this point of view. And he's like, can you look through the data and just give me the data points that supports this point of view? And I'm like, hold on. It should be the other way around. Tell you, like, what conclusions? He's like, I know, can we just do that? I'm like, no, I don't think this is the way it works.

Yeah, that makes sense. So moving on to the next question is, what is the worst conversion optimization project that you have worked on?

I don't know if I have the worst conversion optimization project.

How about the one that, like, took about six months? Can you say was it the worst? Like the experiment that took about six months to implement?

Yeah. I'll mention three different projects that stand in my mind. I'm like, I don't want to do that. The one that you mentioned. So a large enterprise company hires us. They believe in conversion optimization. They have not done it before. Hey, let's go ahead and can you guys help us? And we want to run some experiments on our product pages, large, maybe the fifth largest at some point back then, run some experiments on their product pages. Great, we'll do that. And, you know, we do our analysis, we come up with our hypothesis, we make some changes and we show it to them, and we start getting feedback. We make some other changes, we start creating other variables, so on and so forth, so on and so forth. It took us six months of almost like 23 rounds of feedback until we end up with like, you know, four variables, four variations. We launch it and they all lose. Now it's, this was a hard pill to swallow, even for their cto. Really. He's not the single winner after six months. I'm like, not only that, not the single winner after six months and not the single winner after 23 iterations. There is a lesson there, correct? A lesson that the whole point of conversion optimization is you are taking the approach of mvp. I just need to find something that works. People send me a signal, and then I can spend some more time making it better and better. But many companies come at conversion optimization and they say, we have four variations and they need to be pixel perfect. Guess what? There's a good chance that your four variations that are pixel perfect, people are going to hate them and they're not going to increase your conversion rate and you're going to throw them out there. So instead, let's try and get them to the point that is acceptable and let's just sit out and people will raise their hands. They will give you enough signals that they are actually interested in the design. So that's one project. I don't know if that's one of my worst conversion optimization projects, but I'll give you a couple of other ones that just stand out in my mind. We signed up one company, and this is back in 2012, 2011, 2012. Sign up with us. And we always tell companies it will take us three to four months before you start seeing results of the experimentation. Our goal is to produce results a lot faster, but you want to set expectations correctly. Hey, it's going to take us a little while. We launched the first experiment 99%. Back then, the experiment had 99% confidence, basically to win the variation, 35% lift in conversion rates. We ran it, we're like, you know what? This is too good to be true. Let's run the experiment again, pause the experiment, relaunch it completely. Fresh data again. About the 33% chance to beat original or 33% confidence. Sorry. We launched a second time. There was a 33% lift with a 99% chance to beat the original. So now we've launched a couple of times, so we're super excited. When you see something like that, I always say whenever you're doing a B testing, you are digging for oil, you're looking for those big wins. I am very happy with small, consistent wins. But every time I get some of those big wins, I'm like, oh, we've done it. I called their CEO. I'm talking to him. I'm like, hey, super excited. Look at the results. And they were following with us. They looked at the second iteration and he's like, yeah, I absolutely love it. Thank you. But we'd like to cancel the contract. And I'm like, what just happened? I've just delivered to you from my first experiment a 35%. I have told you it's going to take about three to four months to see any results. And just by the way, when you measure the impact of that experiment while running it, I think it generated about a couple of hundred thousand dollars for them, which was incredible. Just by looking at the variation revenue versus the control revenue. And we're like, oh, look at that. We've done $200,000 more. Great. So I'm talking to him, Jeremy. I'm like, jeremy, can you please explain to me because this is very unusual that we deliver such result and you'd cancel right away? He said, yeah, no, no, the result is really good. But I was hoping for 65% lifts and conversions. Well, okay. It just threw me off because I was expecting many answers, but that's answer I did not expect. And I'm like, can you please explain to me? Did our sales guys tell you that we're getting delivered 65%? And they tell you that we're getting delivered 65% within the first month? He's like, no, no. They told me, like, maybe 30, 35%. I don't remember the exact, but they told me it's going to take three to four months. I'm like, so why are you expecting a 65%? He said, My business really needs that 65% lift. I'm like, okay, thank you for talking to me. We'll wrap up the project at the end of the month. Goodbye. Good luck. That was not a very good project where a person, no matter how often you tell them and you try and set expectations, they don't walk into the project with the right expectations and they're not being very honest with you. And then one of those projects that we walked away from, we get hired by an enterprise SaaS company. Hey, VP of marketing. Really nice, really sweet. She hires us. She's like, hey, can you help me? And one of the projects, we wanted to redesign the homepage. So whenever you redesign the homepage, we have a copywriter who's going through the homepage. We've done a ton of customer interviews. We come up with new copy sheets. Absolutely loves it. I'm like, okay, let's go ahead and implement it as an a B test. He's like, no, no, I actually need to run it by the board first. So now I'm like, oh, I didn't know that the board is going to actually review the copy on the homepage. Now in my mind, usually we avoid testing anything on the homepage because I know the political battles that you go through when it comes to homepages, but too late. You know, I Thought I was working with the right stakeholder. It turns out that there's other stakeholders and. Oh, my God. Within that week, we got almost 13 different pieces of feedback. One board member loves it, the other one hates it. And I found myself and the team found themselves stuck in the middle of a battle between board members. It was like, you fix one thing for one board member, and I'm like, okay, can we even get on a call with them? Nope, can't get on a call with them. We end up canceling that project. And it's rare for us. Usually we push through and we make sure, like, we dealt with so many different companies, so many stakeholders, but in that case, the fact that we could not get access to the actual decision makers did not make sense. So we walked away. Funny part. Couple months later, they end up hiring an agency to redesign the homepage. And I look at the copy that they did and I'm like, oh, it's almost the same copy. You barely changed anything. And about a third month later, that VP of marketing moved on and got hired somewhere else. I think she just recognized what kind of company she was working with and was not a good fit. So those are different projects that we struggled for different reasons.

Yeah, those were pretty good examples. Usually, like, how many experiments do you usually, like, launch a month for one client? Let's just consider. Maybe they have, like, the numbers.

So we have different levels that we work with companies. At some companies, they are just dipping their toes into the ocean of conversion rate optimization experimentation. So those companies, we might do anywhere between two to three experiments per month with them. Now, we set expectations very clearly with them. We tell them, hey, you're just trying to figure out if experimentation is a good idea, if you have the culture and the resources, and if you're going to be able to click with us, that's fine. We'll do that for three months. We'll give you a flavor of the work. But don't expect results because we are not going at the speed that we usually go at, especially if we're doing two experiments a month. Most of the companies that we work with, I would say about 70% of them, are around four experiments per month. Four or five experiments per month. Our next level is the six to eight experiments per month. And sometimes we've had companies that worked with us doing 14 and 15 experiments per month. Very unusual, because you have to have the traffic and you have to have the analysis, and you have to have the team to be able to do that many experiments. I would say the Larger companies usually are doing anywhere between six to eight experiments. Six is the standard. What's fascinating, we have all this customer data, client data, and we can track several metrics. One is the testing velocity. And we track that and we compare it to the percentage increase in revenue that you are gaining as a result of the experimentation program. And we try and correlate that to the quality and the complexity of the experiment that you are launching. And it's interesting because you can see more experiments you're launching, higher impact and higher increase in your revenue. And remember also experiments stack one after the next. I was just looking at data for one company that we're working with. January we started with a couple of experiments. By the time they got implemented, it took a while, but you're building. By the time you got to December we had high winning experiments running on the websites. And we always keep the control. And we can see that with the control you would have generated about a million dollars. With those nine different experiments you would have generated 1.5 million. So half a million dollars in one month. But those are, that's the beauty of experimentation where experiments are building on top of each other.

That's good. So who usually takes care of maybe the implementation of the test? Is it like from the agency side or from the client side?

So most dedicated CRO agencies, and I say most will have an implementation team that specializes in implementing AB experiments. They do the coding and the implementation and the development that is required for an A B test. Having good developers in a company does not mean that you have good A B test developers. As a matter of fact, most good solid developers don't make good solid A B test developers. They're two different creatures. And no matter how many times I try and explain that to CTOs and to VP soft marketing and they struggle with that. And they say why? If they're a good developer they should be able to figure it out. And I tell them a good developer follows certain standards. They're ingrained is in his DNA encapsulation abstraction. You basically are trying to write code that's going to exist maybe for 15 years. No code is going to exist for that long. But that's the mentality. Write code that you can look at it four or five years later and it's going to be readable, you're going to understand it. It's well documented. An A B test developer on the other hand is a hacker. I'm writing code that I know it's going to be thrown away in a week from now. Can you imagine the mindset between I'm writing code that's going to last for many years versus I'm writing a code for a week or two and then we're going to throw it away. Two completely different creatures. One of the things we used to do in 2006, 2007 is we used to ask our clients like, hey, if you have a developer, can they implement the AB test? And what we found out is developers get busy, priorities shift, and then instead of developing an a B test, oh, can we work on this feature? Can we work on that feature? And you're giving them ideas and suggestions of a B experiments and nothing gets implemented. That's the reason we went into the model and this happened now over 12 years ago. We said, you know what, we have our own specialized development team. They do experimentation day in, day out, as a matter of fact for us. And that's probably unique for us as we have an AB test framework because we've done tens of thousands of experiments and sometimes as you do tens of thousands of experiments, like we've done the same experiment, the same logic in another with another client. So we have a framework that allows our developers to implement experiments a lot faster. But having a dedicated team, whether if you're hiring an agency you want them to have, you want them to handle the implementation, if you are going to do something internally, have a dedicated implementation team that just only and focuses on a B testing and experimentation.

I was going to ask you, I know like we've touched on this question before, but maybe just for the sake of. Someone was listening to the podcast for the first time, how long does it take from ideation to implementation of the test? Because like I've talked about the difference between a B testing developers and stuff. So I just curious to know like how long will it take from ideation to implementation?

So an average AB experiment for us and again, we're probably a lot faster than anyone else. And let's talk about the medium complexity experiment, not oh, I want to test single step checkout versus a multi step checkout and oh, I don't want to test a headline, I'm just tweaking a headline. No, a medium complexity experiment for us takes on average about 14 hours from ideation to hey, the experiment is launched on your website now again, this depends on the complexity of the experiment. I think the standard, by the way, in the industry is around 22 hours. Again for us, I would say 12 hours is the average. I think for us a complicated experiment might take anywhere between 16 to 24 hours. Now, how long does it take to implement versus the duration for implementation? Correct. Because sometimes you have to get approvals. The designers have to get feedback. So that will take usually around 10 days. That's the reason we aim at launching four experiments a month. And you have to have a queue running. One experiment is in the ideation. The next one is actually in the design. The third one is in the coding. So continuous movement to be able to produce four experiments launched on the website at any point in time.

Okay, sure. I think that was the last question that I had.

Awesome. Thank you, Simba. Thank you, everyone, for listening. Super excited to be in 2024. And I would say let's ask our listeners. We've had thousands of downloads for the podcast, so it's super exciting. But I would say I would ask you to share the podcast with somebody who's gonna benefit from it. That's a favor to us because we want to grow our audience. And until next week, happy testing shift.