#46 – Should You Choose In House or Agency For Experimentation?

Because we're expanding the CRO team. That's always, by the way, an interesting exercise. This relates to what we're going to be talking about today, the in house versus the agency CRO. And what do we look for when we hire a CRO? Because we are an agency. Because probably whatever I'm going to say is going to apply to SEO, PPC is going to apply to other types of digital marketing or even traditional marketing. But the challenges that you deal with in house versus the challenges that you deal with as an agency are completely different set of challenges. My name is Karl Slade. Love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal.

Well, pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today. Wherever you listen to your podcasts, I know we are live, but I'm just messaging back and forth with PEP from Spiro CXL Winter. I have to mention this before we jump into CRO Live Hour because they have the CXL event that is coming up and I do plan on going this year. The 16th 17th of October. Hold on. Let's make sure that I'm not mentioning the wrong dates in case I mentioned the wrong dates. Yes, October 16, October 17, CXL events. If you're doing experimentation, I mean it's a must attend event and then it's a shame because I have not attended it until now. Many of the industry friends go and attend and they all talk about how great of an event CXL Live is. So here you go. Like, you know, free promotion for CXL event. It's just when somebody puts on a good event, you just have to mention it. So I'll mention it to Pep. I don't think he's very involved. That's the reason we're talking about the events. He's not very involved in organizing the event. But I wonder if we can get a coupon for people listening to the CRO live hour. I don't know if they're doing coupons or not, so we can't promise anything, you know, but. Simba, how was your week?

Cuba? Business has been one of the busiest weeks so far, maybe this quarter. We were working on different things. We have like a number of articles that we have on our website that need to be pruned. Also the Bootstrapped podcast was coming back so. And to also embody to pick some videoings to post. But it was one of the business. It's been good, it's funny.

So for those who are listening, one of the things that we do at Invest and Fig Pie is the last Friday of every month we're off. That's the reason tomorrow we're recording this on a Thursday and that's the reason we're saying tomorrow everybody's going to be off. Which I think for the level of stress that sometimes we go through is much needed. To me, this week has been more reasonable. You run a business, so you go through the ups and downs. And I would say on average my stress levels around 30 to 50% last week made up for it. It was at like, you know, 500%. So, you know, a lot of stress between me and my ads. It's good though because we had to make some tough decisions. We ended up making them. I like those. We're gonna be talking about it. We talked about it in the Boot Trap podcast about some of the decisions, tough decisions that we've had to make. So it's been good now. Like, you know, this week has been a lot more stable. We've been going through interviewing a ton of people because we're expanding the CRO team. That's always, by the way, an interesting exercise, sort of relates to what we're going to be talking about today. The in house versus the agency CRO. And what do we look for when we hire a CRO? Because we are an agency. But that has been an interesting exercise because I must have interviewed 20 people and we post our ads everywhere in the world. I think we don't go beyond Europe. I've never tried to hire in India or Pakistan. Not to say that there are no good people over there. However, the challenge is the time zone difference. We mainly work with US clients and I just cannot imagine somebody as a long term project being Able to hold meetings at 1:00am, 2:00am with US clients, even if you think, hey, I can do it, yeah, you can do it for six months. But as a way of life, I don't wish that's on anybody. For the same reason we have not worked with many companies in Australia over the years, maybe like three, four companies. Because we tell them, hey, the time zone difference is challenging. One of the benefits of interviewing lots of people is kind of also talking to them about the processes that they do. We've ended up making an offer to somebody to join our team so she will be starting next week and we're adding another CRO person to our team as well. Maybe within the next couple of weeks. So we're expanding the team some more. It's fascinating because especially when you hire somebody with experience, you try to see like what do they bring to the table. Because we have our processes, we have our way of doing things and I'm always hoping when you bring somebody senior it's like, hey, how can we do this better?

I was going to ask you about which country is kind of like surprised you the most then I had to also kind of like think about it myself. My experience, I think is we work with people from different background, different countries, different nationalities, different backgrounds, different cultures. Everyone is kind of like different in their own way. Which like, that's interesting.

Yeah, no country surprised me. In all honesty, people surprised me. Or Lisiar was from Portugal and he came almost like 20 years of analytics experience. So it's been just fascinating because I love playing with analytics. I mean, you give me analytics, I'll play with it. And I've always thought I was good at nautics. I can configure tags, I can slice and dice the data and export it and do a million things with it. But here's a whole other level. So that has always been fascinating. I think every team member brings something new to the table and that's what I love and that's what I enjoy.

So as you have already mentioned today.

We'Ll be talking about in house.

This is agency experimentation. I'm sure like you've worked, obviously we're a BNC company and you have like interacted with a lot of people from in house. Just top of mind, before we get into the questions that we do have, what do you think are the main differences between an in house experimentation team as well as like an agency experimentation team?

So I don't know if there is something necessarily particular about a CRO in house or agency versus other Types of marketing. Because probably whatever I'm going to say is going to apply to SEO, PPC is going to apply to other types of digital marketing or even traditional marketing. But the challenges that you deal with in house versus the challenges that you deal with as an agency are completely different set of challenges. There's the core challenges of I'm doing some analysis and I need to work with team members and I need to make sure that I'm pushing enough experiments out of the way and doing the process analysis. You'll get it with both.

Correct.

You'll get it whether you're in house or you're outside in house. You're dealing more with the politics of the company. And I know people don't like to talk about that, but CRO could be very. This is one of those things that could trigger people experimentation. Correct. Because you're making changes to the website, like a whole bunch of stakeholders who are constantly looking at the website. Correct. So you have design teams and you have branding teams and you have marketing teams. So away from your core job of actually coming up with experiments and doing the analysis, you have to manage those stakeholders. And stakeholder management when you're in house is a little bit different than agency in house. If you know how to play your cards, the management of stakeholders could be easy as an agency or always an outsider. So how do you actually establish yourself as an authority? That's a challenge for an agency. And every time we bring on a new company to work with, I always tell our teams, your challenge in the next three months is to go from all other external. Regardless what the stakeholders tell us, there's always everybody's looking at you. Oh, you're somebody external. Do you understand our business? Do the experiments make sense that you're suggesting? So that's one of the challenges that a CRO working for an agency has to deal with. I think, by the way, working as somebody within an agency is a lot harder than working in house. Because also the amount of pressure that you get as somebody who's working for an agency is probably three to four times or folds what you get in house. People might not like this. I think even as a CRO, by the way, not all disciplines of online marketing are created the same. Let me give you an example. CRO deals with a lot more stress than what an SEO deals with. Two completely different ways. SEO by its nature is long term. Yeah, I have deliverables, you know, but okay, I'm going to deliver articles and I'm going to do some, maybe some link Building. It's a very well defined process and I'm going through the exercise with a CRO. Especially if you are working for a high velocity testing program, guess what you're doing? Constant evaluation, constant analysis from one experiment to the next. You're seeing the whole process of launching the experiment and that is very involved. Oh, we did the analysis, now we're coming up with the design, working with developers to implement. So there is a lot of pressure. So that's the difference between a CRO and an SEO. When I hire, by the way, I could consider people who, well, we've dabbled a little bit in CRO, but we have main experience in SEO. I'm like, huh? They're probably not going to understand the pressure that we put them under. Somebody who is running PPC campaigns, they probably have a little bit more pressure than an SEO. I hope I'm not starting like a battle by saying all these things.

Now.

A CRO that's in house versus a CRO that's working for an agency. When you're working in house, you're single. Focus on a single entity. If you're working for an agency, you might be managing anywhere between two to three to four to six clients. It depends on the complexity of those clients. I was just interviewing somebody. He works for a CRO agency. He manages three to four clients. He's launching two experiments per month. So I did the math. I told him, so in a month, like, you know, maybe when it's really busy, you're launching ACE experiments. He said, yes. I told him, well, our CROs and our senior CROs are launching anywhere between 20 to 30 experiments per month. So effectively our CROs are doing three to four times what you are doing. That's a little scary. I mean, it's one thing when somebody tells you, yeah, I can do this, no big deal. But then there's the reality. Can they do this or not? So that's one of, I think the bigger challenges and one of the bigger stresses of working as CRO for an agency. Now those are the stresses. The advantages that you get working as a CRO for an agency is you get exposed to a ton of clients and a ton of problems. The growth in your career is tremendous. We work with a company that is $1.2 billion. I mean, humongous. And on our weekly status calls, that's between our CRO team and that company. The CEO of that company is there now, mind you, love the fact that he's attending our calls. But sometimes I wonder. I'm like sure, you have better things to do than discussing crs, but it's such a high important priority project for him that he is there and he's discussing ideas. He wants to see the results. And when I tell our CROs is like, you don't get the opportunity. I mean, people work in that $1.2 billion company. They would struggle to get a meeting with him. And they might get lucky to schedule something once a month. You are getting him for half an hour to an hour every week and discussing ideas and exchanging them. The amount of growth that you get out of that, there's no dollar value that you can put next to it. So, yeah, there's the stresses, there's positives and the negatives. One of the CROs I interviewed absolutely liked him, but he was working for an organization and it's considered one of the top 100 places to work for in the US and one of the questions I asked him like, hey, you've been working there for three years. Why are you thinking about changing? I mean, it's one of the top places. We're not one of the top places to work for. There's a lot of stress that comes with working for us. And he says, I'm bored. He's like, I launch maybe three, four experiments a quarter. Not really very intrigued. There are people who survive in that atmosphere. There are people who cannot. And I know I'm like mentioning all these stories, but I remember when I worked, I came, graduated college, went into work for Sapience, which is now Razorfish. I worked there for two and a half years, and then I switched for another consulting company. And then I decided that it makes sense for me to go work in house Simba. From the very first day, I decided, I'm like, oh my God, I've made a mistake. And I called the recruiter who helped me get that job, and I'm like, can I leave? She's like, no, you can't. You just started today. Give them a chance. I'm like, I know this is not the right place for me. And I died in that place. I stayed for six months because I'm used to fast pace. Correct. I'm used to, let's make the decision, let's decide, let's roll it out. I remember the first month I worked for that company, they were still doing code review for a class that I wrote. A month later, I got it reviewed by a senior engineer and then an architect and the VP of development wants to review it. And I'm like, this thing is 50 lines of code. And now this is the fourth or fifth meeting that I'm actually reviewing it. I just could not handle that. I was not built for that. I ended up leaving them, you know, because just was not a good fit.

That makes a lot of sense. But now I'm beginning to question or not really question, but I'm trying like to think what's the reason why most emails experimentation team or some of them launch fewer tests than agencies.

I think that also depends on the organization because is experimentation a way of doing business in the company or isn't it? It's just like, oh yeah, we have to do it, so we might as well do it. I think that's kind of the difference. And then the execution of that in house could be external. One of the people that we are thinking about bringing on board and we're kind of in the process, he worked in house now, in all honesty, and I told him, I told him from the get go, I'm like, hey, so and so. That worries me. Everything else on your resume I absolutely love. But you've worked in house and you've worked with a single entity for the last year and a half and then before that about three years, another entity with an agency. It's a lot more fast paced now in the last place that he worked on and maybe next week he'll ask me whether we end up bringing him on board or not. He was launching himself, he was doing 30 experiments per month for a single entity because they had the traffic, they have the conversions, it made sense. And that organization is built around, okay, Experimentation is a way of doing business. If you're working for an Amazon or Facebook or Google, there are many other companies. Experimentation is a way of doing business. I remember one company who we worked with back in 2000, 2020, they brought us on board and we worked with them I think for about nine months. And it's very unusual for us to work with a company for nine months and then they walk away. And what they have said, hey, we've built our own CRO team, we want to take it in house. I'm like, great, we'll help kind of support you and make that transition. So that was 2020. They came back in 2023 and they said, hey, we need your help with CRO. I'm like, hey, so what are the results? Because you have an experimentation team. This is an extreme case. But between 2020 and 2023 they did not launch a single experiment, although they have a CRO team. And I'm like, I don't want to put them on the spot to say, what happened here? Why three years? Sometimes the internal team doesn't have the mandates and then the strength to be able to push experiments. Correct. And then that organization was very particular because there was lots of political battles. That's something that we felt as an agency look, as outsiders, the discussions with every experiment. One of the things that we emphasize with every company that work with design is never final, it's just a step. Things are going to change, Things are going to change. Not every organization is built for that. And I think a CRO, as somebody who's going to run experimentation before you get hired in house, if you are going to go the in house route, you really have to understand the culture of the company. Because not every culture is built for experimentation. And CRO in itself is just difficult to do. Experimentation is difficult to do. AB testing is difficult to do. If you add on top of that, having to fight political battles and then having to justify the essence of why we're doing experimentation. Croatia, you might as well walk away now. I'll just put the last caveat to that. I understand. I'm looking. Sometimes I hear people say, oh yeah, you should walk away. I'm like, yeah, salary also matters. Correct. What I'm seeing, by the way, and this is interesting going off of a tangent, but one of the things I've seen, I've seen in the last two to three months is lots of people are losing their jobs, unfortunately. It's unfortunate. But I can tell you what people expect in terms of salaries have dropped by 20 to 30%. People are staying without work now for three, four months. Even really good people, sharp people. But just the way the market is going and everybody is anticipating that things will get worse.

So let's talk about the structure of the teams and in house team and then agency, is it the same or maybe they are kind of like some disobedience.

There is a difference. There's different agencies. But I think the model, you always have a CRO, you have a designer, you have a developer, qa, project management on and off. You might have somebody from the team manage that. If you're small scale, if you're doing large scale experimentation, you'll have a dedicated project manager. That is the agency model. Now if we talk about the different model for in house, it's completely different because now you need to look at your development teams because really you do not launch a new feature, you do not add anything new without actual experimentation. So we don't have necessarily a Dedicated CRO team. But CRO is part of every team that you do. And I've kind of the best example I've seen with that is with eBay, specifically eBay Turkey, because we worked with them for a while. They went from, oh, let's hire Invesp in the very first iteration, 2016, and took us forever to push anything through into okay, now we've changed the structure of the team so everything that gets released is thought of okay. It's an AB experiment and now we're doing analysis. Now that doesn't mean that you have to hire 10 CROs or something like that. Sometimes there is 1 CRO or 3 CROs or 4 CROs who are supporting these different teams. Correct. But fundamentally, when you're in house, you're changing the way you release features, you're changing the way you do business, you're changing the way you develop things. And then the CRO team could be integrated within those development slash design resources. You could have a clearinghouse. Because sometimes it's not about new features. It's like, oh, I want to test this. A CEO says that a VP of marketing, there's some analysis ux. So you have a clearinghouse that says, okay, give us all the ideas and we're going to vet them. And then we can assign them to different teams to say, okay, here's how we're going to test. And then this team might do the post test analysis, say, okay, yeah, xyz, maybe a VP of marketing wanted to test this feature. We added it to Roadmap. We've run the experiment, now we have some test results and based on that, we're having a conversation with the VP of marketing, with the product team to say, hey, does it make sense to add this? Do we build on top of it? Do we pivot? Do we iterate? What are the next steps that we need to make?

That's a good one. What are the key benefits of conducting experimentation in house? And what advantages may agencies also bring to the table?

This is perhaps a very long time question. We work with some clients who just outsource all of their marketing. So they'll work with a PPC agency, with an agency, with a CRO agency. And we work with some clients who have large marketing teams. So it's a philosophy of doing business. The beauty of doing things in house is if you're able to keep your people, you have this knowledge that you've done and accumulated as a result of doing experimentation continuously. When you're relying on an agency, guess what? There's a lot like, okay, so the agency is doing work, but you need to make sure that you've hired the right agency, that they are delivering what you are asking them to deliver and what you're expecting from them. But how do we capture all of that data? I was looking at one of the companies I work with. We've launched almost 400 experiments for them in the last two years. It's a challenge. It's a challenge they recognize as a challenge. We recognize this challenge. What did we learn? The winnings are really great. You know, you've grown tremendously, you've done a ton of experiments. But how do you capture the learnings from 400 experiments? How do you capture the learning, by the way, from 50 experiments is different than from a hundred experiments, from 400 experiments, from a thousand experiments, because you need a framework around that. And that's one of the problems that we didn't have to deal with when we're dealing with clients who are doing two, three experiments. Kind of a slow pace because in a year we might have done like, you know, maybe in a year we would have done, I don't know, trying to think 50, 60 experiments. Correct. So it's kind of a slow pace versus if you go to that many experiments in a couple of years, putting that knowledge in a meaningful way where it's not like, oh, I've just done an A B experiment, no, what did I learn? How does that impact the different channels? That's the reason I like the in house sometimes as a model where you have somebody there. Now, one of the bigger challenges that most organizations are dealing with is one keeping their staff. Because, okay, so you're doing it in house. So that means it's higher cost. Correct. And outsourcing it, but you want to keep the staff. And how do you capture all this knowledge where it's not in somebody's head, it's always there, institutionalized. If I may say that everybody can benefit from it.

Okay, so can you give us like maybe some examples of scenarios where in house experimentation might be more suitable? And also situations where an agency will be like the better choice.

If you're just starting out, you have two models. If you're just starting out, I would recommend doing an agency. So you'd learn the ins and the outs. I mean, perfect example of that. We're starting out with SEO. We hired an agency to help us with SEO.

Right.

We started out ppc. We're like, hey, like, you know, doing PPC for FIGPI or invest. We're like hey, let's hire somebody because we can learn all this stuff ourselves, but we'll spend the time and the money learning this stuff. And I did not want to hire somebody because I want to validate the concept. Same thing with CRO. I want to validate the concept. I want to learn what works and what doesn't work. And then at some point you have to think to yourself, how are we doing business? Is CRO fundamental? Is experimentation fundamental to how we do business or is it just another team? If it's just another team, just like how we have an SEO team, how we have a PPC team, how we have a design team. Okay, so you can make it work, by the way, both ways. Either an agency and it's fine, we can outsource this or we can have the resources inside. I think there's advantages and disadvantages and you can think about your business model in that case. But if CRO is fundamental to the way we do business, not a single feature will be released on the website without actually a B testing it. That's more of a fundamental approach. Correct. That's different way of doing business. And that I think in house makes a lot more sense. Now, I might still use either a CRO consultant or an agency to help guide the teams to some extent, but I think if I'm doing something where every development, every new feature can get released as an A B test and I'm going pre test analysis, post test analysis, that should be in house. Sometimes people reach out to us and they say, hey, do you guys want to outsource some development for FigPi? It's constantly on LinkedIn and sometimes it's friends. And I tell them, no, we don't think by is a technology company. I can't outsource the core fundamental way of doing business. I want the researchers who work for Config Pai to be full time employees. For us, I might use, and we've used consultants, correct. To come and help us solve this particular problem. But it's not fundamental to us. If something's fundamental, then I'm bringing it in house. And sometimes we start by outsourcing until we learn certain things and then we say, okay, now it's time to bring it in house. At some point we're hosted on the cloud, so we're on aws. We outsourced some of that because we didn't have the skillset to build the infrastructure. But guess what? We brought it in house where we said, you know what? Okay, we've learned enough and now we have the skillset and now we've hired the right person, so now we can bring it in house. It's funny, now we're thinking about moving from the cloud to actual servers and we're like, oh, do we have the skill set to do that? I think we do. But again, it's a business philosophy on how we do business and people might agree or disagree.

Yeah, that makes a lot of sense. So how long does it take maybe for an in house to build their own CRO team?

Hiring is not a problem. Correct. There's enough talent out there who are looking for jobs. Hiring the right talent, that's the big challenge. And the problem is sometimes I'll give the example of developers. People sometimes might say, you know, okay, we'll hire a CRO and we have a designer, we have a developer and we'll manage. And then maybe like, you know, if this is good and it's producing results, then we'll add a designer and we'll add like, you know, a developer. And what you find is that the efforts of the CRO are being held back. He's not producing the specialists who you've hired, who you've dedicated because the developer is getting pulled in a million directions and the CRO is doing amazing analysis, coming up with ideas, maybe even working with the designer, but then he turns to the developer and the developer says, yeah, okay, yeah, add it to the list. I'll get back to this, I'll get to this when I get to it. And sometimes it takes a month, sometimes it takes two months and then things slow down and then you think to yourself, oh gosh, the CRO is not producing results. No, the CRO is doing the work. But it's a chain. Everything has to operate correctly to produce the results that you are looking to produce. That's the reason I say if you are going to build an in house team, start by saying, you know what? XYZ Resource is going to be the CRO. This person is going to be the designer who's dedicated to CRO. Here's the resources can be dedicated to experimentation to start with now, you might integrate that and change the way you do business later. But you have to have the dedicated resources if you are going to be serious about experimentation. If you are serious about a B testing and CRO, dedicate the resources or otherwise don't do it. Can't imagine how many times I talk to companies and they think that they're doing CRO but they're launching one or two experiments a quarter. I'm like, okay, I don't know if I would call that. You really have a CRO? It says now CRO just became a checkbox. Yeah, yeah, we're doing it. No, you're not doing it. I was talking to a company. They reached out to us, which is sort of funny. They're like, hey, like, you know, we need help. And I look at their site, and I'm like, oh, my God, this site is built from the 90s. That was my first impression. And they gave us access to their analytics. So we're looking through the data, and they're converting close to 5, 6%, which is absolutely incredible because they sell unique product you can't find anywhere else a whole bunch of unique products. So they're converting despite their efforts of designing a poor website. So we did analysis for them. We're presenting in front of the CTO and the CEO, and he's like, yeah, we know this. Oh, yeah, we know that. Oh, yeah, it's on the roadmap. And at some point I had to stop and say, okay, so it seems like you know everything, but you're really not doing anything. Why are we even having this conversation? A CRO is not only about knowing the stuff. It's also about pushing things and getting things done. And again, by the way, it goes back to the culture for me, that company. And they're like, oh, yeah, let's move forward. I'm like, yeah, I don't think they have the right culture. I think they'll give us a hard time if they think they know everything, but they haven't done it. And they said, oh, well, come help us do it. I'm like, yeah, I don't think so. I think it's going to be a battle, and I'm not interested in battles. There's enough companies that are interested in working with us. I'm like, okay, yeah, there's other CRO companies who will be better fit for them to work with them.

So I ran out of the questions about in house and agency. But I just want to ask you one question before I let you go. I remember, like, at some point last year, during the Black Friday event, we helped, like, a lot of clients. I think we generated, like, was it 37 million or over 37 million of revenue for both our clients? I understand that, like, it takes a lot of time to prepare for Black Friday, right? What advice can you give to some e commerce clients right now going towards the Black Friday? Anything to do, like fermentation.

We're starting the conversation. We actually Started the conversations mid August with companies to talk about Black Friday. Sometimes people wait until the end and I'm like, no, if you're serious about it, there are conversations that we start in August talking about, okay, what are the offers, what are the partnerships that we're going to have, what kind of experiments? We try and finalize a good chunk of this, I would say by mid October at the most. Now we always have a couple of companies we work with who are going to wait until the end. They're going to make their life difficult, they're going to make our life difficult. But you know, we're talking about, okay, so what are the offers, what are we going to be testing, what are the creatives, what kind of bundles that we're going to have, what kind of experiments that we're going to be running? If Black Friday is significant and for most of the companies, the e commerce companies we work with, Black Friday is significant for them. Black Friday until Cyber Monday, Correct. That week. It makes or break. It will make or break the company. You can't just walk into it. It's like, oh yeah, let's figure out the offer. You gotta have the structure in place to run the experiments that you need to run with the offers, with the designs. And waiting until two weeks before makes no sense. So we're starting those conversations. We're thinking about all these different experiments that we're going to be running. We like to finalize everything by the end of October. So when it comes Black Friday, we're just turning a button on or off and just monitoring results. So don't wait, do not wait. It's a team effort, by the way. It's not somebody sitting somewhere. Yeah, we're going to run this, we're going to run that. No, it will be huge. And then we're talking about, you know, companies. I mean, Z Gallery is a perfect example. And the reason I mentioned Z Gallery, because their VP of marketing was really nice to us. The did a case study with us where during Black Friday he said, oh yeah, the revenue lift that they saw during Black Friday, Cyber Monday, I think they did. And Zigato is a multi. Hundreds of millions of dollars. Correct. So there may be a unique case, but almost like in a 19, 20% lift in revenue, when you're talking about a company that does 500, $600 million, you can imagine how much of an impact that had on them. But regardless whether you are small or large, Black Friday preparation should start today. And your goal is by end of October. Sorry, by end of September we have. Most of the major decisions have been made and now working, we're working through the creatives. Everything's ready by end of October. So then it's just turn it on, turn it off, monitor results and whatnot.

I know, like, we can really, like put a number on this, but are we going to kind of like maybe go above the 37 million this time? Considerations and everything?

We'll do better because we have a lot more clients that we're working with now. Yeah, we're So, I mean, just naturally when you work with more clients. Correct. You'll do better because, you know, like, and it's funny because we've changed the business model. We typically worked with eight to nine companies. Correct. That's. We're like, we'll pick and choose. Now we're going to 20 different companies. So just naturally you're gonna do a lot better in terms of revenue. What's fascinating is last year, and that was kind of a big experiment for us, is doing what we call multi arm bandits on figpi, where in an AB experiment you have the control, you have one variation and you're trying to test three, four different variations based on hypothesis to see which one performs better. Multi armed bandits, a completely different philosophy where we don't care about like learning. In all honesty, we're saying let's maximize the dollars in the bank between those different branches. So we use fake buy for that. So last year was kind of a new experiment for us with that and now everybody's just looking forward to it because there are some companies who have tried it and experimented with it during Memorial Day, during Mother's Day, during Valentine. So I'm super excited to see how that actually operates with the millions of visitors that we're going to be testing during that period.

That's good. That's good. I don't know, maybe just by any chance or anything like that, maybe there might be like an E commerce testing that are listening and they want to really maybe consider engaging in this. Do you still have like space for few more clients or.

We have. Yeah, I mean, I don't know if we have space. It depends on also when we publish this podcast. Sort of interesting because we had said, hey, we maxed out at clients a couple of weeks ago, but at the same time now some of our CRS are saying, hey, I can, I can take on some more clients. But at the same time we have old clients were coming back and they're like, hey, what's the plan for Black Friday? Yeah, if somebody's interested, reach out. We're not promising that we can take on new clients. It changes day by day. And it's funny because as I finish this in half an hour, I have a meeting with our sales team and they were annoyed with because I had told them, I'm like, hey, like, we can't take any more clients. And imagine this is, by the way, the worst thing that you can tell a sales guy. We cannot take on clients. He's like, so what am I gonna be doing here for the next two to three months? So I told him, I'm like, by the way, potential clients are going to hate this. I told them, raise your prices so you can justify to me if you're going to bring on a new client, that's like, okay. I mean, if there is such demand. So yeah, if somebody's interested, reach out to us. We'll have a conversation. If we're going to be able. If we think we can help you, we'll tell you how. We'll figure out something. If we can't help you, we'll tell you, no, we can't. Or sometimes, like, you know, by the way, we say, hey, talk to so and so from this other company. There are some companies that are better than us in certain aspects and we'll say, hey, talk to them. It depends. There's some industries we don't work with. So we say, hey, yeah, XYZ companies is better fits and if we feel that we can help you, okay, just come on board.

So I guess that's what I had for today. I think like we must dedicate at least like a week or two talking about Black Friday because I think like there's a lot to talk about there, that we can help some of the business.

Yeah, I'm always excited to talk about Black Friday, you know that. So, you know, let's start.

Anything to add?

Well, I would ask our listeners if you enjoy the podcast, share it with somebody who you think will like the podcast as well. I love kind of the weekly reporting when we look at how many podcast downloads that we're getting and the fact that we this now for almost two years. By November it will be almost two years. It will be two years. Not almost, you know, of Scaro Live power. But until next time, happy testing.