#34 – How To Scale Your Experimentation Program

Having an experimentation program that you want to scale requires you to change the mindset. We might be committed to a quarterly release plan, but we're not releasing finalized, finished features. We are testing those features. We are getting feedback from our customers, and we are saying, okay, do people actually like this feature? My name is Karl Sladech, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live hour, a show show all about a B testing, experimentation, and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today, wherever you listen to your podcasts. Another week, another CRO Live. Super, super exciting. Simba, how are you doing? How's your week?

It's been a good week. My Manchester United has been doing well, even though we lost last week to Sevilla, so. But it's been a good week. Some of the marketing numbers are beginning to. We're seeing, like, an increase or a steady, like, increase of the graph. So it's good. How about you?

Good. Soccer has been good. I'm dreading the fact that the summer's coming and I'm not going to have any soccer for two months. I'm like, really? Two months with no soccer? And there are some games, but nothing as interesting as the Premier League. And of course, for our audience who are not following soccer, they're like, what are you talking about? I'm like, you know, there's football in the US There is basketball. I don't know many people who follow baseball. It's just, sorry, guys, it's not a very interesting sport. I might just watch the World Series if I'm really up to it. But otherwise, I find soccer a lot more fascinating. So that's one in terms of investment. Figpie, things have been interesting. I think this is the busiest time for invesp so that's always good. And figpi as well. I mean, you look at the number of subscribers who are starting to use figpi. I have to say something, though. Sometimes what's been happening lately is I end up having to eat my own words. So for many, many years, I've made fun of AB testing companies that end up adding personalization. I'm like, oh, yeah, you ran out of things to do. You stop innovating when it comes to experimentation. A B testing, so you add personalization. Oh, gosh, I can't believe I'm even saying that now. But guess what? I'm having to do so many demos for figpi, jumping on calls, and consistently what I hear is like, hey, so how do we do personalization? And I'm like, oh, man, after I was asked that question, I know 50 times, I'm like, okay, maybe we have to do personalization. So I'm like, I can't believe I'm doing that now. But I guess that's what happens when you have two approaches correct to handling this. One approach is to do what Basecamp does. You know, they listen, and then they'll wait for people to ask again and again. And at some point they might say, you know what? We're not going to do this. This is not us. You can find another piece of software. Well, Basecamp is probably doing $100,000,000 in annual revenue. FIGPI is not doing $100,000,000.00 in annual revenue. $100,000,000. So guess what? We are really driven by requirements that come from our customers. So we are adding personalization, and I think we're rolling it out in the next six to eight weeks. Fundamentally, we have all the features. We just need to package them in a way that makes life easier for people who are really interested in personalization and they're not interested in a B testing and then something else. FIC Buy was built on the premise of single line of code. You get everything for conversion, rate optimization, you get a lot more and more companies who are saying, well, I just want a B testing. I want session recording or user behavior analytics session recording, heat maps, or I want polling. And after you hear this again, I don't know, 30, 40 times, I'm like, oh, gosh, guys, maybe we have to separate them. Not everybody wants to pay for the whole package, mind you. That was kind of the premise of Fake Buy. You can still get everything but that custom pricing, and now it seems to be a requirement. So we're also rolling this out. It's fascinating because before I would have maybe four or five conversations in a month with customers of figpi. Nowadays, I can have four or five conversations every day with customers of figpi. So I love it. I love hearing them. And most of the time you're hearing the same requirements. So that has been my week for the last probably eight, ten weeks. So just kind of listening to people, listening to potential customers, adding more features, working with the team. I'm loving this process.

I think like what you just said about adding a personalization feature to FIG by it ties back to what we were talking about last week, customer feedback. You know, like at the end of the day you get to listen to what customers want and you build it whether you want it or not. But yeah, they have the final say.

But you want to hear something funny? There was this one module in FIGPY that was previously not used. And of course when you have a module in figpy, you have to maintain it. Correct. And every time you roll out version, you have to update it. Last year we're debating whether we should drop it or not. We're like, it's not really heavily used. Maybe we should focus on other modules. Well, things have shifted, I would say, in the last three months where it's like, oh my God, usage, and it just increased tremendously. So again, not only through talking through people, but also behavior on the platform is driving the new features on what to focus on. So, yep.

Speaking of growth, it's also interesting to see like the numbers for this podcast. I was looking at it today, getting to see like this podcast getting a lot of traction. That's so interesting.

So here I have to say, if you are one of those listeners who are listening to us and then downloading the podcast every week, thank you. We really appreciate that. And we thank you also for the questions that you asked. But a couple of things. One, if you have questions, send them to Simba, send them to me, you know, and we'll make sure that we answer them. That's number one. Number two, we would love to have a review from you. I don't think we have any reviews on Apple podcasts or any of the podcast platforms. So we want to get a whole bunch of reviews. So please leave us a review. We'd appreciate that. Today we're talking scale. You think everybody wants to scale, but to scale or not to scale. And we are specifically talking about experimentation. Scaling and experimentation Program another episode of CRO Live hour. So Simba Floor is yours.

Yeah, sure. So I have a few questions about experimentation. So the first one is how do you decide when it's time to scale your experimentation program. Maybe first give a brief definition of what an experimentation program is for, just for the benefit of those who might not have a clear understanding.

So let's talk about what an experimentation program is. In a typical website platform app, you figure out, okay, here's a new feature, and you add the new feature, and you're driven either by the people who are running the program, people who are running the platform. Correct. It's like, oh, we would like to add this feature. Hey, it would be cool to add this feature. With experimentation, with a B testing, you change that process where you say, you know what? Let's a B test this new thing that we want to roll on the website. How you run an A B test, very simple. I'm getting 5,000 people, I'm getting 10,000 people, I'm getting 100,000 people. Sometimes I'm getting a few million visitors to my site every month. Well, what you do, every change that you have is you split the traffic. Let's say you have 100,000 visitors. So 50,000 visitors will see the old design, 50,000 visitors will see the new design. And based on a specific metric, typically conversion, you'll say, okay, so how many conversions did the old design generate and how many conversions did the new design generate? Oh, we've rolled out a new design for our homepage. We are an E commerce website. How many orders did the old design generate and how many orders did the new design generate? And based on that, you judge whether you want to keep the old design or you want to roll out the new design to every visitor. So that's really what a B testing is. Now you start with AB testing. Eventually we want to say we want to graduate into having an experimentation program. What is the difference between doing AB testing versus having an experimentation program? Well, a B testing is when you're testing, you know, running an a B test now and then maybe once a month sometimes. I've seen companies that run an AB test once every two, three months. An experimentation program is a shift where you say, you know what? A B testing and experimentation, and I use those terms interchangeably, is a way of doing business. So now we go from doing a single AB experiment every month or every quarter into doing multiple experiments every month. So that's when you actually have a program. Now graduating from a single experiment once a month or once a quarter to multiple experiments. It's like graduating from sixth grade into going to college. Correct. There's a big difference between the two in the requirements and the process and the mindset. So there is a lot there. Is there a benefit to scaling an experimentation program? You bet. You want to start in primary school, elementary school, but eventually you want to go to college and you can finish your bachelor's degree and you can go to Master's and PhD. That's really what having an experimentation program is. So how do you decide when to scale your experimentation program? First off, you cannot scale an experimentation program, of course, without having started with AB testing. You have to have the mindset, you have to have the process established, and you have to have the team. We've done a little bit of AB testing, we've dabbled in it, and now we want to commit to running an actual program. Now, I always say you need to crawl before you walk, and you need to walk before you run. Scaling experimentation program is like running. Correct. And it is not a short sprint, it's not a sprint race. Correct. It's a marathon. You are fully committed to it. You understand all it takes to scale such program. You have the resources and the process. But let's talk about what it takes to scale that experimentation program. First, you need to have the mindset. Typical mindset that people have. Whenever you have a website, whenever you have a platform, whether it's E commerce, whether it's SaaS, whether it's lead gen, is to say, well, here are new features that we want to release. When we might have a release calendar, usually of features that you plan either monthly or quarterly, or preferably even maybe annually. Here are the features that we are going to be working on. You build them, you release them, and you move on to the next feature. Having an experimentation program that you want to scale requires you to change the mindset. We might be committed to a quarterly release plan, but we're not releasing finalized, finished features. We are testing those features. We are getting feedback from our customers and we are saying, okay, do people actually like this feature? Are we having people adopt this feature and use it regularly? Is it having a positive impact on our bottom line on our key metrics, on conversion rate, on retention, on revenue per visitor? And based on that very quantitative feedback that we're getting from our customers, we decide, do we iterate on the feature, do we enhance it, or do we completely drop it? There's a huge shift in mindset that needs to happen because you are no longer in control. You come up with your best guess on what will work best for visitors. But your visitors have that final say and they have that final vote, and they do that through Their actions on their website. Not every company is ready and prepared to have that mindset. So that is a challenge. Now everybody for the most part will tell you that they are ready to have that mindset. But in reality, I have seen many, many teams that just struggle with the fact. What do you mean? I've worked on this feature for a month and now you're telling me we need to scrap it or. I've worked on this amazingly beautiful design and people don't like it. Yeah, really. Ultimately, your visitors should be the judge. So what else we need to talk about? When it comes to scaling experimentation program, do you have the traffic to scale an experimentation program? Not every site can scale an experimentation program. Now lots of times people talk about the number of visitors that are coming to the website. I completely ignore that and instead I look at the number of conversions that the site or the platform is generating. If your site is only generating 100 conversions per month, you probably want to stick with slow scale, small scale, A B testing. You don't have a full program. You cannot justify having a full program running multiple experiments. I was talking to a company out of Spain and they were running 30 experiments per month. Very impressive. I'm like, okay, that's really good. And I had to ask just the basic question. I'm like, well, how many conversions are you getting per month to run 30 experiments? And they're like, oh, we're generating about 25 to 30 conversions per month. And I'm like, hold up. I'm a conversion optimization guy. I'm an experimentation guy. You should not be running 30 experiments per month, mind you. You might be able to run an experiment or two per quarter, if even that. And naturally I asked them a second question. I'm like, well, how long does it take for your experiments to conclude? And they're like, oh, they never conclude. We just run them and then we run more experiments and then we run more experiments. So they have scaled their experimentation program, but they really should not have done that. They're just running experiments for the sake of running experiments. That is just the wrong thing. Number two, you need to understand, let's say you have the numbers, do you have the resources to scale an experimentation program? And by resources I mean people who are going to do the analysis. In our world, we call them CROs, Conversion Rate Optimization specialists, you might call them anything. But there is somebody who is actually running that program doing the analysis to figure out where to create experiments and how to run the experiments and doing the post test analysis. Do you have the Designers who are going to be running and creating those different designs. Do you have the developers to implement the experiments? Do you have the QA quality control, quality assurance team? All of these things need to be in place in order for you to scale the program. And number three, do you have the process? If you're struggling to launch a single AB experiment, you're probably going to struggle if you start and you say, oh, we're going to go from one AB experiment per month or one AB experiment per quarter into doing four or eight experiments per month. If you do not have a solid process, you're going to end up with spaghetti AB testing. No one wants that. You're going to hate your life and things are going to get very confusing. So you need to have a solid process that you've tested and you slowly increase that velocity, the number of experiments that you are launching. If you have all of these things set in place, then I would slowly increase the number of experiments. Okay, so now we're doing, let's say, one experiment per month. Well, okay, so next month I might do two experiments, knowing that in a quarter, like three months, I'm going to get into four experiments per month. And you keep on adding number of experiments, checking your process, modifying it, and thinking, can we add more? Are we getting the value from the program? The goal is not to scale the program for the sake of scaling it. The goal is to get the insights from the program and to see more conversions from the program as well.

That's good. So you mentioned about four components that you need to scale. The optimizer, the experimentation program. Just to recap, you said the mindset, the process, the team, and also the traffic. Right. In my mind, like, thinking of those, having to put all those components together comes with a lot of challenges, obviously. So what are some of the challenges that you have seen when someone is trying to scale their experimentation program?

So some of the challenges are part of experimentation, team, process, mindset. Some of the challenges are marketing challenges. If I'm not generating enough traffic to the sites, that's a major challenge. We're talking to a startup that was referred to us by one of our SEO partner companies, and they said, hey, they just broke an SEO. They're getting a ton of traffic and we think that they're ready for CRO. Super excited. Always love to get a referral from a company that we've worked with on many different projects. And again, the first question I asked, like, well, how many conversions are you getting per month? And they said, well, we're getting 30 conversions per year. I'd love to help you and do actual experimentation. You're not ready for it. I was talking to the CEO, absolutely brilliant guy, and he's like, well, what do you mean? Like, you know, we're ready to pay whatever you guys ask. I appreciate the fact that you want to pay us, but you can hate the program. 30 conversions per year. Now, what you can do is not an experimentation program. You can do a CRO audit to fix glaring conversion optimization issues, but it's really based on heuristic analysis based on a little bit of data. You cannot run actual meaningful AB testing. So that's more of a marketing challenge. Do you have the traffic? And that CEO that was just mentioning, he said, you know what? Okay, well, it sounds to me like we need to scale our AdWords program. Not AdWords, Google Ads. I keep on calling it AdWords. Just shows how old I am. And I told him, yeah, if you're able to scale your Google Ads program, then at some point it will make sense for us to work together. So that's a marketing challenge. So let's put that aside completely. Between mindsets, resources and process. The hardest, by the way, is the mindset, and it is the one component of scaling of the program. Everybody thinks that they have, but eventually everybody finds out that it is difficult for them to achieve. One of the things that we do, whenever a company talks to us, we tell them, hey, deal breakers. If there are a few things that we require of every company that we work with and if you don't have it, it's fine, you'll find another CRO agency who's willing to work with you. But for us, this is a deal breaker. And one of the deal breakers that we have is we will do it our way. Some companies struggle with that. Some companies are saying, we'll say yes. And what do I mean by we'll do it our way? We're not going to do something crazy with your brand. We're very conscious of that. But at the same time, we have the expertise. You're hiring us because we want to do conversion rate optimization. You are no longer releasing things that are just random on your website. We're testing everything. We're thinking, we're doing the pre test analysis, we're running the experiment, we're doing the post test, post experiment analysis and we're coming back to you with our findings. Data drives our decisions. Now, most of the time, I would tell you, even the companies I work with all the time, they'll tell us, yeah, of Course, that's the reason we're hiring you guys. But when we start the work, that's when the reality happens. We have to remind every company that we work with that here is how the process works. It doesn't matter how much time you spend on a design and how beautiful you think the design is. If your visitors actually hate it, you don't have any interactions. They're struggling with it. Visitors are leaking from your website. We have a problem. We're trying to fix that problem. You have to surrender that control to your visitors. Not everybody's ready to do that, and they struggle with that tremendously. So Mindset is really, really hard. The other challenge that we see is around process. The process that works for you when you are running an experiment per month or an experiment per quarter is a lot different than the process that you rely on when you run four experiments. It's different than the process when you're running eight experiments per month. And we see that, by the way, for us, our process, and we've been doing this for 16, 17 years at this point. We've had a process when we were just doing one experiment, two experiments per month, we had to modify that process when we went into four experiments per month, when we were working with different companies, we had to modify it again when we reached 10 experiments. And we're kind constantly optimizing that process. I just had a conversation with Ayat earlier this week talking about the process, and I'm like, okay, we need to modify this, we need to modify that. Talking with our CRO team to say, hey, how can we improve? Because when you're moving really, really fast, you want to make sure that you're doing things that make sense. It's almost like, you know, you're riding a bike, going downhill, a mountain. You're always looking, trying to avoid any potholes, but at the same time, you have to raise your head and get a view of where you're heading. And that is really the case when you're running an experimentation program. And then finally, resources. I think if you understand what it takes to run experimentation, I think that's kind of the least challenging part, especially when it comes to dev. When it comes to design, when it comes to quality control, finding a right conversion rate optimization specialist is not easy. That's something that we struggle with. I mean, when we hire, we hire globally. We're looking for CROs all over the world, but also, by the way, finding somebody who has a ton of experience. There are a few of us who have done experimentation for 10, 15 years. I remember posting an ad to hire senior CRO and I had so many people apply and they call themselves senior conversion rate optimization specialists. Don't mean to offend anybody, but if you've done conversion optimization for less than, I don't know, five years, if you've only done it for a year, you're not senior. Just like, if you're a developer, you cannot call yourself a senior developer. If you have like, you know, one year experience, it's like, oh, I've done this project and that project. No, just like how with development, it will take five to seven years to get to that senior developer. And you're just starting as a senior developer, you're a lot different than somebody who has done development for 15 years. The same thing. Finding that, right? CRO, because he is running the program, he is thinking through the program. That is also a hard part of running conversion rate optimization.

That makes a lot of sense. So someone listening to us right now might be thinking that, oh, I don't have enough traffic. I have maybe about 30 to 50 conversions per month or per quarter. Yeah. You say that person can do experimentation program. How about an optimization program? Are the components needed to do an experimentation program the same as the components needed to do an optimization program? What's the difference between the two?

So to me, by the way, even if you have low conversions, I'm going to contradict myself, but I just kind of want to correct this. You can run a B testing, but you have to relax your standards tremendously. If, let's say you have 30 conversions or 50 conversions per month, should I just completely ignore a B testing? Remember, there is cost with a B testing. Correct. You need to do the analysis. So you have to have the CRO and you have to have the developer and the designer and the quality control person. And now one person might service to those roles, but there's cost there. So let's say you're getting 50 conversions and let's say you run the program and it's like, oh, it doubled our conversions. We went from 50 conversions to 100. Well, it depends on the value of a conversion. If the value of a conversion for you is $50, that's $2,500 more conversions. Yeah, I don't know if I would want to do AB testing, but we work with companies where a conversion value is close to half a million. Yeah, half a million. Multiply that by 50. You're talking about significant increase in value. So you need to look at your average conversion value and you need to relax some of the standards. Yes, I'm going to run AB testing, but maybe I am aiming for an 80% confidence. I'm not aiming for that 95, 97, 99% confidence. I want to slowly every month run some experimentations, relax the standards and really kind of our North Star. Correct. You're going down the hill, but you're keeping an eye. It's like, okay, we're trying to improve conversions because even going from 50 conversions to 70 conversions will have major impact on the business. So kind of keep that. Now, the other thing for me, an experimentation program is fundamentally based on experimentation on a B testing. I am going to be AB testing. Sometimes you might say, you know what, okay, yeah, going from 50 conversions to 100 conversions is great, but the conversion value is low. Just like the example I gave. I still want to improve the user experience on my website. And in that case, I'm not necessarily doing AB testing, but I'm doing CRO audits, taking the feedback and after I run the CRO audit, implementing those and taking some measurements around the website, maybe usability measurements. How long are people sticking when they come to the website? How many pages are they navigating, how many contacts even or conversions that I'm getting? It's not very scientific, but there are some metrics that I'm using to be able to actually see whether we're heading in the right direction or not.

Right now, as you're explaining, I'm thinking like going from one experiment to five experiments per month is also different from going from maybe five to 15 experiments maybe per quarter. Right. Or per month or something like that. I'm sure, like, it's easy to say that we went from one experiment to 10 experiments per month, but I'm sure it's a bit complicated. So how do you kind of like ensure that there is consistence and there's standardization in your experiment, in your experimentation processes?

Yeah, let me put it this way. Going from one experiment per month to four experiment per month. For experiments per month, it's doable. By the way, most of your processes, you'll need to tweak some of your processes. Now, the minute you go from 4 to 8, the minute you go from 8 to 16, that's when things get a lot more complicated. Because your site has limited pages that you can test. Correct. Think about the typical E commerce website. I have the homepage collection pages, my product pages, cart checkout, maybe some support pages, mobile versus desktop versus maybe tablet. If you're getting enough traffic on tablets, let's say you have the conversion so you can run experimentations, but there is a limited set of experiments that you can run. So mentioned, let's say four different types of pages, mobile versus desktop. That's just right there. Eight different areas of the website that you can test now. Well, no, I want to run three experiments on my product pages at the same time. And I have enough conversions. Let's say I'm generating, I don't know, let's go wild. You know, like some of the companies we work with are doing 50,000 conversions. So, yeah, in terms of conversions, they have that. But now you need to standardize the process we actually do with some companies run multiple experiments on the same page. You have to have the software correct that allows you to do that. And at the same time, you need a centralized clearing team where it says, okay, yeah, we know the experiments running on the product pages. We know the experiments are running on the homepage. What's your idea, why you're trying to test this? And how do you make sure that your experiment is not going to cannibalize the other experiments running on the page? How many traffic, how many visitors you want to take? Let's give you that. So you need that standardized centralized team where you bring your ideas, they look at everything. So you're really managing things. I'm running an experiment on my product page, but guess what? We're running also other experiments on the product pages. We're running other experiments on the category pages, collection pages, on my cart page. So we need to make sure there needs to be a coach that looks at the full board and makes sure that things actually make sense. Last thing you want is, okay, I have a winner on the product page that looks so much different than the collection page, that looks a lot different on the cart page. And the site looks just very funny. That's the reason you need to have that clearing process where everybody looks, everybody discusses, and definitely communication becomes critical components of that process. When we work with companies that we're launching, let's say sometimes with the larger companies, we're doing anywhere between 12 to 16 experiments per month. A good chunk of our process away from the actual experiment work is kind of looking at the full board and having discussions with the company's team to say, here's what's happening, we want to keep that. It's like all the experiments are running. And then really when you have winners to sit back and say, okay, well, yeah, here's how we orchestrate the user flow throughout the website. And when you have Experiments that don't generate results. What did we learn? You know, we make an assumption about our customers over here. That's the reason they didn't like this new design. Do we have other experiments that validated that assumption? So that's coach correct. That sits and looks at all the data points. This is very critical really to the success of scaling or scaled experimentation program.

One last question before you go. Oftentimes, I often hear people saying that experimentation should be the foundation of every successful company or of every company that wants to succeed. Right. What's the relationship between marketing experimentation and other marketing disciplines like PPC and content marketing? Because usually people say, like your experimentation program has to feed other marketing discipline. How does that work?

Think about this. Everything that experimentation team does will impact other marketing channels. Everything that other marketing teams do will impact experimentation. Now, sometimes it's very closely related. You cannot decouple them. What your performance team is doing in terms of bringing people to the website will have a direct impact on the site conversion rates, the site's revenue per visitor, all those metrics that an experimentation team is going to be looking at. You're bringing me poor quality traffic. Guess what? My conversion rate without me even doing anything is going to drop. And you might be bringing the best traffic to the site. But if I'm doing some stupid, silly experimentation, I might hurt the traffic that you're bringing. So those two teams will work very close with each other. Now, some other teams, yeah, they're working together. So let's say content versus experimentation. Content team typically is bringing traffic to the website, to the blog and content pages. Naturally, by the way, most of that traffic that the content team is bringing is what we call top of the funnel traffic. It does not convert well, but it is important traffic because it helps in establishing the brand in the space that it operates in lots of times. This is a typical complaint that I hear from my SEO friends or people who are running SEO programs like, oh, we're getting like, you know, half a million visitors, a million visitors, but they're not converting. We're getting only like 15, 20, 100 conversions for that many visitors. And I tell them, yeah, because you have the wrong understanding of what an SEO team wants to do. SEO teams bringing that traffic. It helps build the brand long term. It's very powerful. Don't look at it and try and measure its success in a short term scale. Now, at the same time, when I'm running an experimentation program, I want to come to that SEO traffic or that content traffic and say, okay, so we Understand that this traffic is not going to convert. But think what think about other things that we can do with that traffic. We want to capture that traffic, we want to capture the emails, we want to encourage it. There are many things. It's not pure conversions, but it's very early trying to learn. So I'm trying to educate them. That's when an experimentation team works. Now, outside of this, all these different marketing teams cannot operate in silos. The performance team learns something about the customers they need to inform the other teams. Hey, here's what's working really well. Here's the message that works really well. Experimentation. The fundamental aspect of experimentation is that you're learning, having hypothesis about your visitors and how they interact with the website. That hypothesis is validated through an experiment and you're always learning. So you want to feed that to other teams. Hey, content team, here's what we've learned. Here's what people love. Hey, performance team, here's the messaging that works really, really well. I'll give an example from experimentation teams. We run jobs to be done interviews, we run them very regularly for every company we work with. The insights that we get from those one on one customer interviews, 45 minutes to an hour with each customer, not only impact our work as an experimentation team, but really they help the email marketing team, they help the performance team. So we say, hey, here's the messaging. Use some of this messaging, here's some of the insights, here's some additional targeting that you can do. All these teams need to break down the silos between them, meet regularly, discuss things regularly and provide share those insights. And ultimately the goal is one, we want the company that we're working with and working for to succeed.

Those are the questions that I had for you this week.

Awesome. So next week, by the way, I'm going to be out. So we're not going to have CRO Live. I just remember it because I'm actually invited to speak at a private event. Should be interesting because I think there's about 150, 200e commerce brands. So I'm super excited about that. So I'm going to be out. I think I'm going to be speaking at that event around the same time we're doing we do the CRO Live. So we'll come back in two weeks.

Nice. Awesome.

Thank you Simba. Thank you everyone for liking. And again, if you have any questions, send them to either myself or Simba and we'll be glad to cover them in the CRO Live hour. Until next time.

Sure. Bye.