#63 – Building An In-house CRO or Experimentation Team

So in terms of budget, how much do you think? Let's say just like one e commerce company needs to build like a solid in house CRO team.

It's rough. I remember reading an article and I forget who the author was who argued and it was very popular. It was published on CXL2 three years ago where he was arguing that it makes perfect sense to hire a CRO agency. But then they were just releasing adding CRO to their services and I'm like really? Thank you. We appreciate the vote of confidence hiring an agency. By the way, whether it's SEO, CRO, ppc, there's cost is definitely a component of that. Correct. CRO agencies are not cheap by any means and a good CRO agency will cost you.

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of your conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today wherever you listen to your podcasts.

Another week, another CRO Live hour. And since people are slowly gonna trickle and since Simba is wearing his and United shirt, he gets to gloats this week. So they beat us. Feeling good. Go for it, Simba. I will be on the receiving end.

The truth is I had to make sure that like this week whenever I'm having a call with you, I'll be wearing the Winchester United jersey. And I'm sure you've noticed and I don't know like when this is going.

To happen was white color jersey.

No, I don't know like when this is going to happen again. We're going to play against your Liverpool in the next three weeks. We might. There's a high possibility that we might lose that game because we don't usually beat Liverpool twice in a season or twice in a year. Yeah, I'm just enjoying it while it lasts.

You deserve it. You guys played well. You deserve winning. It was a good game. I enjoyed it and I knew that we're going to lose, like, you know, second half. I was like, no, I know exactly where this is going. It's just like when we played. So for those of who are joining us, we are still. This is CRO Live hour, but we're giving people a chance to join. But we always talk soccer at the beginning. You know, when we played Chelsea in the. Which cup? The Caribou Cup. Correct. I knew they were going to win because you look at them this like, you know, in the extra time and they're just not hungry and we're hungry and the hungry team always wins. It doesn't matter how big or small you are or who do you have on the field? So, no, I thought it was a good game. I think we did not do you guys a favor by like, we didn't let you win. I don't think you did yourself a favor by winning. But we will talk about this in a couple of years from now. That's funny. I was watching what is a sport talk in the uk. That's one station I follow. They'll have lots of discussion around the Premier League. Absolute waste of time. But I like to waste my time on this. And this one Man United fan was just, he's like, I wish we lost. And one of the anchors is a United fan. I can't. This is so disrespectful. And he had the same theory as me. That's like, by man, you winning, things are just going to get worse because you don't get rid of your manager. So we're going to tie this into a company before we talk about company culture or anything, like before we talk about creating a CRO team, because I know that's what our episode is all about. I do think sometimes thinking about team members and employees, whatever you want to call them, it depends on your culture. Sometimes you have your top performers and it's very clear you want to keep them. They'll go through ups and downs, but you know that they are a top performer. When things matter, they'll come through and through for you. And then you have your poor performers and that's fine as well because you can get rid of those. It's the ones that wishy watchy. You never know whether you can rely on them or not. They're just absolutely Horrible. They can kill your company and culture. And I figured since we're talking about CRO, that's what we are going to jump into it today. So you chose an interesting topic today. Building an in house CRO team. Yes.

We once spoke about this. I think maybe it was the first episodes of the CRO live hour. We spoke about building an in house CRO team business building or hiring an agency, which one is better? But today, like I just thought we can dig a little deeper into building an experimentation in house and what goes into that? What should you do if you're planning to do so?

I think that that's an interesting topic to talk about. It should be an interesting discussion at least.

So let me know when you're ready. We have about four questions on this one.

Okay. Yeah, let's go for it.

So the first question is what does it effective in house CRO team look like and how do you suggest structuring.

It for maximum impact whenever you establish a new team in your organization, Let it be CRO SEO, ppc. Let's say you've worked on an agency and you say, you know what, now I want to bring this in house. Or you've never worked with an agency and you want to bring it in house. There are certain things that you have to think about and I especially for startup smaller companies, but I think at least in CRO, I've seen even bigger companies fail at establishing in house because they just did not understand what it takes. Because it's not only about creating a team and tell them, go ahead and run experimentation, go ahead and run CRO. The whole company, the whole organization has to be ready to run CRO because it's a different mood of running. And I don't mean this lightly. If you think about, let's take something like an SEO or a PPC SEO, hey, go ahead and get us some rankings. They're going to structure the websites in a certain way, get you some backlinks, some mentions and you start ranking. They're really not fundamentally changing the way you do business. It's okay, just do whatever you need to do. Same thing when it comes to ppc. Hey, create some ads. Google, Facebook, Reddit, Twitter, whatever platform drive us or YouTube, drive the traffic to the website and let's just figure out like, you know, how we can convert that traffic. That's the easy part about creating those teams. You just need to hire the right person who understands your goals, who's actually have the ability to deliver on those goals. Experimentation in CRO is a Lot different. Experimentation in CRO changes the way you do business. Guess what? Let's say you're an E commerce company or you're a SaaS company and your dev team has a development skew that they're working through. And your typical process like, oh, here's what we're going to be working on in this Sprint two week cycle, four week cycle, let's go ahead and build it and just release it and we'll build qa, release, so on and so forth. CRO comes in and if CRO is done correctly and experimentation done correctly first, they'll tell the development team is like, oops, put things on hold. We need to rethink how we release new features. What you're used to in terms of, oh, let's sit here, have a discussion, complete the feature requirements and build it and then release it and then move on to the next one. It's not the way experimentation works because now everything is going to be an experiment. That's the ideal scenario. Correct? You might not start with that. If you are somebody who just got hired to run experimentation, you come into your CTO and tell them, hey, no more releasing features. I don't know if you're going to have your job for a long time or you might just be put in your place very quickly. But ideally what you want to get to that everything that gets released on the website goes through two phases. There's an MVP phase. That MVP phase is surrounded by an experiment to say, okay, what are the metrics that matter to judge whether people like the MVP or not, get the data, take the feedback and then decide, do we pivot, do we iterate, what do we do based on the data? So again, it is really a major shift for dev, is a major shift for design, and it's a major shift for top management because no longer top management comes and says, I want this, let's get it done. Now we're going to experiment and we're going to come back to you and show you the results. I was talking to one of our senior CROs yesterday and we're looking at the results. We do quarterly business review for our clients, but we also do an annual review. And I was looking at the winning experiments and how much revenue we generated and what are the learnings. But then he's like, khaleed, there is a bigger thing that we're not capturing over here. And I'm like, what is that? He's like, those last experiments that did not achieve results, but they were all for features that our clients had asked for and their management had asked for. So we said, you know what, let's go ahead and run this as an experiment and see. And he said, what experiment? Where the CEO had insisted on one feature, and we pushed back and we went back and forth and we end up testing. It would have caused them about 14% drop in conversions on product pages that are an e Commerce Company. 14% is significant for a company that does about $35 million. Now, I'm not saying that they would have lost $3.5 million or actually maybe $4 million, but they would have probably lost $2 million if they had released this feature. They did not think about that. They're like, oh, let's just go ahead and push it now. We can come back with data, you can release it, but you will lose 14%. Do you really want to push it? And the answer was very quick, no way. We don't want to deal with that. So that's the very first thing that you want to look at. I think, again, this is what we aspire to get to. That, to me, is an effective CRO team. Do you get there overnight? No, you get there over time. And the overtime is an interesting journey because again, it's different from one company to the next. With most companies, what the CRO and experimentation team has to do is earn a seat at the table. You can demand a seat at the table, but earning is a lot more useful. Heads will turn when you speak. If you had to earn the seat at the table, how do you do that? Winning experiments, deep analysis, pre experiment, post experiment, getting everybody excited, not only your team. It's about evangelizing, experimentation and CRO within the company. That's probably your very first step, pulling other resources from other teams to participate. Slowly, slowly you get to the, here's the state that we aspire to get to. And the minute you do that, you achieve the maximum impact. And we hear about companies like Amazon or Facebook or Google where everything is tested correct at any point in time. Well, I forgot how many experiments, if you land on Facebook, you're part probably of some people say 20, up to 50 experiments that are running on Facebook at any point in time. They have enough visitors and volume to justify an experimentation at that level. And I always like to mention the story, when we did our first test with ebay, first experiments, they've never done experimentation before. Was painful, was painful, was slow. 23 iterations, six months, and we ended up losing. And it's just basically the team was not structured. We had an Absolutely amazing person who believed in experimentation, but the rest of the org did not know what experimentation is. And to them, it was just another thing that we're releasing on the website. So we had to get the design perfect. Designers had a say, branding team had a say, the marketplace guys had a say, the VP had a say, the CTO had a say. It was just slowed down the process, and then the experiment lost with all the different designs and the iterations. So we worked with ebay for a little while and then we stopped working. And then I'm like, ah, man, this is painful. If you are an agency, you want to show your value. I've learned this from a friend and a mentor recently where he said it, and I'm like, you know what? We're the same thing. We're not shy about the fact that we want to make our clients a ton of money. That's our goal. And our cost should be minute compared to how much money we're making our clients and the companies that we work with. When you see that experimentation programs and going well, you're like, this is really does not make sense. If I were them, I would not keep us. And that was the case initially with ebay. Then they came back after a couple of years and like, hey, we want to do experimentation again. And I'm like, oh, gosh, man. Guys, it was not a very good experience the first time. It was not a good experience for our team. It was not a good experience for your team. And what they said is, well, no, no, no, this time it's going to be different. And I thought to myself, really, is it going to be different or not? And they said, come and see how we restructured the team. So previously they had dev and design and they're working together in qa. So it's like small pods. And now they've actually instituted a policy where everything gets released. 90% of the features that gets released are actually AB tested. So although my initial assessment of the project was a complete failure, in all honesty, at so many different levels, I'm like, man, we've managed to change the culture of how the company does experimentation and how they release things. We've done really well. The second engagement was absolutely amazing. And I do think people say, why would you say this? But I will say it. I think the best engagements for CRO Agency are engagements where you are making your clients a ton of money. They are learning, and you're learning as well. So everybody's growing together, and that's exactly what we, what we achieve.

Yeah, that makes a lot of sense. I'm just like assuming here, but I think it's true because like at some point when you and I had started invest, probably it was just the two of you handling like all the CRO work. And I remember like at some point you mentioned that there was a point when I had more than 10 clients.

She had 18. 18. Yeah, 18.

That's it.

I was so our scar was, I'm like, you think four clients is an overload? Imagine doing 18s, which was crazy by the way. Would not recommend that you start making stupid mistakes. I don't like stupid mistakes. I hate stupid mistakes.

So do you think there's a possibility for CRO to be like a one man team?

It depends. Because I think of a CRO, of a way of doing business. You will have to have the designer and you will have to have the developer and you have to have the qa and there's some form of project management. Now, do I need to have a dedicated designer? Do I need to have a dedicated developer? Do I need to have a dedicated qa? That's a big question, correct? Because again, in an ideal scenario, if 90% or 95% of the features that we're releasing on the website are going to be AB tested, so guess what then? What we're talking about is that your development team, your design team, your QA team, start to play as part of the CRO team. Now you as a CRO, you're almost quarterbacking the process. You're the captain of the team. What are the features? Why are we like testing this feature? What's the hypothesis behind it? So they become a center clearing house for all the different ideas, all the different features for the pre test analysis, the post test analysis, that's not an easy job. However, the design dev QA is the design dev QA that you have. You will struggle there a little bit, but this is an ideal scenario because you need to explain to your designers that guess what, in the past you designed something, you got feedback on it and we rolled it and you were very happy and very proud of it. Moving forward, instead of creating one design, we're actually going to create maybe three, four different designs for the same and you can have a good explanation behind them. We're not just changing a couple small things. What is the rationale behind each design that we're introducing? And my friend, we are going to AB test it and instead of taking just feedback from management and from your co workers, you're going to get feedback from your Visitors from your site. Visitors. So if you're used to 10 people voting on your designs and using the word voting in a very relaxed way, guess what? Now we have about a hundred thousand people who are going to vote on it. Oh, and may God have mercy on your soul. You have to develop that thick skin, that design that you absolutely loved. Ooh, it was horrible. Horrible. And the fact that it reduced our conversion rates. So designers have to shift, correct? Yes. Use your existing design team. But they need now, instead of producing one design, you're producing multiple designs with deep thought process. They should be able to defend their designs and the thought behind them. I don't want to just change a couple of colors and saying, okay, yeah, here are the designs. No, I can change the colors. I want you to show me because I want to achieve an XYZ function or functionality on my side. You can achieve it through four different ways. Show me the designs that are going to achieve that and then we'll let people judge, get the data and then we'll analyze. So that's a change for the design and that's a change on how designers need to think with your developers. They're used to, and they have to. Whenever they're releasing code, whenever your developers are releasing code to production, they have to think about maintainability. They have well documented code, they have to think about a million things. And now we're telling them we're going to go through two stages. One of our writing code, there is the AB test code and then there is the production quality code. And 90% of your A B testing code is going to be thrown away. It's not going to be useful. We're just really hacking the functionality, developing an MVB to see whether people are going to like this functionality or not. If we get enough signals from people and they actually like the code, guess what I want you to do now? I want you to forget about that AB testing mindset and go ahead and write production quality level code with abstraction, with documentation, all the nice fancy things that developers are supposed to do when they release quality level code to production, when they receive the production quality code. So they have to make that switch. Now I can tell you that designers, if you actually give them the task of creating multiple designs for multiple fasts, they usually enjoy that because now you're allowing them the creative freedom. Most designers enjoy that, at least in my experience. They might not like the feedback they get from visitors and the fact that their beautiful designs aren't good for developers. Switching back and forth between a B Testing code and the skill set that are required there versus production quality code can be challenging. With one, you are a hacker, you're writing code to be thrown away with the other. No, you're putting on your solid developer who is writing code that's going to be there for a while. Two completely different sets. And some developers get it right away, some developers struggle with it. And if you find that your developers are struggling with it, then you might want to add a B test development team that just specializes in writing that very fast hacky code. Some CROs hate it when I say that. They're like, no, it's not hacky code. I'm like, dude, would you ever release your AB testing code to production? And if they're honest, they'll say no. If they're not honest, they will say yes. I'm like, okay, well you have a problem because if you say yes, then you don't know the whole purpose of AB testing. It's there to validate an idea. And now if I have four variations of your writing, four different code bases that are production quality, like now we slowed down our processes even more. So no, you should not be writing production level quality code. Now. By no means does this mean that AB testing code should be really horrible and it's going to slow down the website because again, that will defeat the purpose of AB testing in the first place.

Yep, that makes sense. When you like maybe when you do a quick Google search just looking at articles about how to build a CRO team, most of the people, they recommend hiring an agency because they say building a team is quite expensive. So in terms of budget, how much do you think? Let's say just like one e commerce company needs to build like a solid in house CRO team.

It's rough. I remember reading an article and I forget who the author was who argued and it was very popular. It was published on CXL2 three years ago where he was arguing that it makes perfect sense to hire a CRO agency. But then they were just releasing adding CRO to their services and I'm like, really? Thank you. We appreciate the vote of confidence hiring an agency, by the way, whether it's SEO, CRO, ppc, there's cost is definitely a component of that. Correct. CRO agencies are not cheap by any means. And a good CRO agency will cost you a good one. I would say at least $9,000 a month and they can go, I mean, I know some CRO agencies, they're charging $30,000 a month, all power to them. And by the way, it's just funny, I have to say, I've seen the work of the $9,000 CRO agencies and I've seen the work of the 30,000 CRO agencies. If I show them to you, you can't tell the difference. So sometimes you're paying a premium for the brand. Nothing more than that. But let's leave that aside. Again, the first question is, if I'm going to put the CRO team together, do I need to hire a designer, do I need to hire a developer, do I need to hire qa, do I need to hire project manager if I already have those resources? And I'm thinking that, hey, now, moving forward, we're going to rely on the internal resources and we're going to change our processes to fit experimentation. The new resource that you're adding is just a CRO. Correct. And you need to hire a good CRO. A CRO probably cost you as much as an agency, sometimes less. Of course there's benefits and there's the headache of managing a CRO. But I think even a bigger question to me is, do you have enough work to hire a conversion rate optimization specialist? That's the real question. Because the CRO work could be 40 hours a month or could be 160 hours a month. Two completely different. If you're only going to be launching two to four experiments a month, yeah, I would hire an agency. I cannot justify a full time person working on this because four experiments a month does not require 160 hours of analysis, pre or post test analysis. Maybe it's 40, maybe it's 50 hours. Correct, you develop the muscle. Sometimes I like to hire agencies and we do that, by the way, ourselves. We hire agencies that help us run certain parts of marketing. You've seen us do that. We learn from them. Our philosophy is we like everything in house. Sometimes I like it outsourced because it just makes life easier and I don't want to add another resources, another resource. And I'm like, you know what, we don't want to manage another person. Let's outsource this and it's fine. And sometimes it depends on the way we think of things. I think of us as a marketing company. Yes, we are a CRO agency, but we are really a marketing company as well. So anything that relates to marketing, we hire an agency, we learn from them and we bring it in house. Fundamentally on figpi, we are a development company. I don't outsource. I get so many calls like, oh, would you outsource I'm like, nope. We fundamentally do not outsource any development work. We deal with billions of records every day. Our scripts are on 3,000 websites. I want to hold my team responsible so we don't outsource that. There are certain functions where we outsource. So again it depends on the company, how you think of yourself and does it make sense to outsource to an agency or full time? You really have to understand your specific circumstances.

That also makes a lot of sense. We spoke like about the budget, right. And the way like you were breaking it down, you were looking at like maybe the skill sets that I needed there. And also how much like maybe it usually costs to hire like a CRO agents. How about like maybe the tech involved in this? How much do you think, like it's needed?

Lots of companies that are interested in doing experimentation, they start by the way with the tech question, which is fascinating to me because the tech is the easy piece. It's not cheap necessarily, but it can be. It's the easy piece. There are different options. So if you think about what's required for me to run a CRO or experimentation, I need to have analytics, which anyways you should have some sort of analytics. Everybody's on GA4 or some people are still on GA360, Google Analytics 360. They're making the switch I think now in six months. Sometimes if you don't like both options, you might be on an option like an amplitude or heap, but you need an analytics solution. Or I have to mention Adobe analytics, there's still companies use that and it's funny because it's usually the bigger companies and I have not seen Adobe analytics in a while but my experience with it not very positive in the past. And I used to say this, but again this is probably. I used to say it very comfortably back in 2020 and before where I used to say I have not seen a single Adobe analytics implementation go correctly from the first time. It was always something wrong with it. Now mind you, and it's fascinating, I just did a survey and we have access to over a hundred analytics accounts just for different clients. And we're talking about D2C brands. That's what I focused on. And it's a minimum they do about the smallest was about $10 million annually and they go up from there. So you look at about a hundred of those and I was looking through their GA4 installations. So everybody dropped the GA4 scripts on their sites. But gosh, the implementation is embarrassing. I would say 70% of the time where I'm like, you don't. You're not even capturing the right data. And sometimes even certain admins, very basic setup that you can just turn on and off and your admin is just not done correctly. So that's scary. That's scary for those companies, that's scary for GA4. Publish anything about the adoption rates. And it's one thing to install script, sell other things to actually get used to how you use GA4. So that's analytics. Correct. And you'll need to have analytics. You'll need to have some sort of a B testing software. Now, some companies like to build in house a B testing software. I'm going to just put that aside. I think you're wasting your time if something's available in the markets, unless you have a very specific scenario, but 90% of the time for 90% of the companies, or maybe even 98% of the companies, you don't need to build your A B testing software. You're not in the business of building a B testing software. I'm not in the business of developing databases. Correct. I'm going to use MySQL, I'm going to use Mongo, I'm going to use Oracle, I'm going to use DB2. There's enough solutions and there's companies that spend a lot of time on that. So I'm going to leave those who want to build the software all power to you if you want to do that. But the CEO and the CFO will really have to question that about the strategy. Let's say you want to go for a B test software. There's costs associated with that. Now, if you take a site that generates, let's say we've taken like a smaller site, 100,000 visitors a month, your cost to run AB testing software is probably anywhere, just for the 100,000 visitors. Anywhere from $400 a month up to a few thousand dollars a month. And it ranges. It depends. Do you want to go with the Rolls Royce of ab testing or do you want to go with, okay, here's our core functionality and let's find the software that's going to satisfy our core functionality and our requirements. We worked with a company and they came up with a list of humongous list of requirements and they did the whole evaluation. They pick a piece of software and I'm like, gosh, man, they picked a very expensive piece of software. And I was talking to their product manager and I'm like, why'd you choose this? He's like, because we want to run a B testing on our mobile app. We've been working with them now since 2020, so it's been almost four years. They have not touched their mobile app, they have not done any a B testing on their mobile app and they're paying a ton of money for features that they don't use. So let's be realistic. Pick something that you're going to use at least in the first six to 12 months you can. I follow this rule in life that seems to have served me well. If I want to pick a new hobby, something that I haven't bought before, I usually go with a cheap option. By the way, if I use it continuously for a period of four to six months, I'm like, you know what? Yeah, I like this. So I am going to invest more and get something more expensive. That was funny because that's for example, how I started with getting cameras. Like initially I'm like, cheap camera, cheap mic, let's see if I'm going to be recording videos regularly. And then after a while I'm like, oh, I think I am recording videos regularly. Let's upgrade. And then I'm like, oh yeah, it's been like two years, let's upgrade again. Correct. So I'm like rewarding myself after I've proven to myself that I actually use the software. Can't imagine how often people buy a piece of software, don't use it, and then, and sometimes they forget about the fact that the contract renews automatically. It's expensive and they're stuck for. Or sometimes they're stuck in a multi year contract. Then there's this one company, I don't know if they changed their policies, but somebody was commenting Zoom Info. People use it, Sales guys use it to get phone numbers for the different leads that they're going to reach out to. You sign the contract with Zoom Info. And we did this, by the way, at some point and I knew that the team is not going to use it. But you know, you don't want to be the barrier in front of the team. I'm like, okay, a few thousand dollars, we buy it, they use it for the first month and then no one uses it after that. I'm like, really guys? No one is using it, but in order for you to cancel with Zoom Info, I think you have to give them notice at some point, 60 days or even more. So I'm like, is this really a good way to do business? But that's their business model. I'm not commenting on that. But you have to be very aware of that. So that's the second piece of software that you need. You will need some sort of heatmaps and you can, you need some sort of session recording. That's another piece that you have to think about. Heat Map software, you can go as free. Microsoft Clarity is free now. There's nothing free. There is nothing free on the web. If you think something is free, then your data is the price that you are paying for it. And I appreciate that. That's exactly the case with Google Analytics. Correct. You're giving them access to your data. They know exactly everything about your website. They know more about your website than you know about your website because they have algorithms and they have AI. This just constantly monitoring and looking at trends within your website. With heatmap session recording you can go with something like Clarity, which is absolutely free. You go on the other extreme. You can go to Cotton Square. Continent Square can cost you probably it's a some one year contract at least and a few thousand dollars the contract there is probably anywhere between 7 to 10 to $15,000 a month. Now a friend was asking me, we work a lot with Content Square. Absolutely love the team over there. Somebody was asking me and it was not about Content Square but he was asking me, actually was telling me because he used to have a SaaS product. It's a no code product where you can create your own app. Very popular. And this friend used to be a partner at investment. He told me, you know what it's like the difference between software that you offer to SMB and software that you offer to the enterprises. Yeah. There are some features that you are specific to the enterprise, but the enterprise customer, why they're paying very heavy price tag is because of the support and the customer success that they expect. So what do you need? But you have to have that software if you can do experimentation. And then another piece is polling. You want to do some form of on site polling. Some people don't like that. I think it's fundamental to experimentation because I can look at the website, I can make assumptions that this area that's broke. There's two things that give you concrete data points that there is something broke. Analytics tells you that something's broke. But it doesn't explain the why. Polling or surveys on your website are powerful because now you're getting actual customer feedback. The problem with it is sometimes you get that feedback from one or two customers that they're the only ones who mention the problem.

Okay.

So you need to validate that through data. Now you can get to other fancy stuff. But I think, you know, some people love to do like, you know, BI and business intelligence and lots of analysis. I think to start with, that's a fundamental. Okay, here's the core functionality or, sorry, here is the core software that I need to have to run an experimentation program. Anything more than that is just icing on the cake.

Yep, that was like an insightful one as well. And we've spoken like about the skill sets right. Within the CRO team, the budgeting, the tech needed. How about maybe where do you think like CRO should sit within a business? Because like I bet some people, let's say like the branding team should, some say marketing, some say like a different department. What do you think is the right place for a CRO team to sit within and let's just focus maybe on an E commerce company.

To me, again, it depends on how you can do that. Different companies are run by different teams. We work with some E commerce companies where brand leads the process. Branding is really a big deal. We work with other companies where UX design leads the process and we work with other companies where development leads the process. To me, again, change that and think that CRO is almost the glue that connects all of those pieces. Now it's the clearinghouse. I'll mention a story about Amazon. So at some point, Amazon is one of the first companies that focused on AB testing tremendously. And they used to allow. A developer will come up with an idea and Sundays, I'll take 50,000 visitors from the site. I get millions of visitors every day and I'm going to test this feature. But you need clearinghouse because then everybody's just running in the wild. So they've created a clearinghouse, a committee that everybody who has an idea goes to them and validates. Why are you running this? What's the data behind it? Okay, if there is approval to run it. So to me, a CRO is doing that correct to say so they're involved with all of those teams and a CRO who does their job correctly will manage to break the silos. So it depends. I'll mention the case with ebay where now you have pods and the pods will have the developer will have the designer or the QA might support multiple pods. But again, the CRO is supporting all these different pods. Who do they report to now? They could report to either a VP of E Commerce or a VP of Marketing. To me, where it makes more sense is that the CRO reports a VP of E Commerce or a Director of E Commerce. Lots of times people. And again, depends on how you do business. Sometimes people confuse a VP of Marketing versus a VP of E Commerce. I'm like, no. The way I think of them as two completely different functions. Now you might merge them. Correct. Because sometimes you say, I don't need to have two VPs, I just merge them as one. But their functions are very distinct. Marketing. Think about it. Brick and mortar marketing is about bringing people to the store. We're going to run ads, we're going to figure out what channels, we're going to do billboards, TV advertising, this online advertising. You will never tell the guy who's doing the ads and the brick and mortar to say, hey, so how do we lay out the store? What do you think is the best layout? No, no, no. That's a function for merchandising. That's a function for VP of E Commerce and their team. Since CRO is working internally on the change of the websites, the flow of the website, the functionality of the website, I think they should sit under Director of EE Commerce or VP of E Commerce. But again, they should be integrated with the different teams. Because the whole point of doing CRO is you're breaking down those silos and the teams are all working together, learning from the losers and celebrating the winners.

Yeah, that's a good way of putting it. I've seen like a lot of people talk about this, like on LinkedIn, and some people, like, we're just saying that you should build like your own. CRO should be like a center of excellence for the whole business and stuff. But the way that you put it, that maybe it should sit under like a VP of E Commerce. I think that makes a lot of sense.

Yeah. I mean, to me, and that's what I would tell anybody who is in a position of experiment and who's going to be running experimentation. Lots of times people also get into fancy titles and the research and the learning. You're there to make the organization more money and have no shame about saying that. Just be very clear. So, yeah, we all want to be excellent and we want to improve the organization. But guess what? You're there to make them more money. The closer you are to the money, the harder it is to let go of you. And the faster you learn that in your career, the easier it is. And the more important you are, the easier your career will be and the more important your position will be. I like to sit under a VP of E Commerce. I like to influence them because that's a person who's generating money. Who's responsible for money? Who's looking at the money? I put it that way, so I like to do that. Now, your job, your actual job, entails excellence and entails breaking down this house. So just think of it this way. Yep.

Those are the questions for you today. I'm not sure if you have anything else to share.

No, I am done. Thank you, Simba. We're on a football break for the next two weeks, and then we come back at it again. Only two months until the season finishes. This exciting season. So awesome. Thank you, everyone. Wherever you're listening to this podcast, we would appreciate if you can share it with somebody who would enjoy it and like it. And of course, we always appreciate a review. Until next time. Happy testing.