#62 – A/B Testing Prioritization (ft. Ayat Shukairy)

It's hard to necessarily decide when I'm prioritizing which ones are going to be the short term, which are going to have the long term impact. As you work with the client a little bit more, you'll be able to have a little bit more sense of what type of an impact it has. We do a lot of kind of analysis also in terms of, for example, those types of experiments. Hey, what was the LOE for this one? What type of an impact did it have on the bottom line? And what does that mean for the client overall in terms of the strategy that I'm going to put forth?

My name is Kalislade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high, testing velocity? How do you measure the success of your conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today. Wherever you listen to your podcasts. Another week, another CRO Live Hour. Yes, we do have a guest today. I convinced Ayat. I didn't really do any dragging. I'm like, hey, you want to join us? We have a very interesting topic to discuss. Simba, Ayat, how are you guys doing?

I've been good. I thought, like, maybe for the podcast listeners, before you say Ayatolling, we're going to have some drum roll.

We can always have a drum roll, but I'm so horrible with these drum rolls. So I'm gonna even, like, put myself on the spot over here.

Yeah, I'm glad to be here. Thank you for having me. I'm always from afar looking at the CRO Live Hour. And finally, I've been invited to join.

Yes, it took a while. It took us only about I don't know how many episodes. Simba. I keep on asking Simba. Every month. I ask him that question. I'm like, how many episodes we have? And we don't have an exact, exact number because we've been doing it now for how long?

More than two years.

More than two years. Wow.

So two years later, Khaled finally decides, let me tell my co founder to join.

Not that I ran off things to talk about because I can always talk about anything CRO related, but I thought like for this episode, you probably have more insights at his tradition. Like, you know, CRO Live Hour. Before we get into CRO Live Hour, I'm not sure I have to tell our audience. I'm not sure if Simba is going to be joining us next Thursday for CR Live hour or not because the.

Next one next week.

But one one of us is not going to join, I think because Liverpool is playing Man United. Oh, all of us is going to be very upset.

What I've seen in the past is that you guys completely cancel CRO Live Hour when there's a Liverpool game happening.

No, no, we've always been, especially when Liverpool wins, I make sure to gloat. I'm like, Simba, I don't want to do. I'm like, come on man, the whole company depends on this. And then we start and I'm just talking in soccer. He's like, liverpool looks like Liverpool, but okay, away from Liverpool. Simba, how was your week?

My week has been good. Busy as usual. Learning some new things and trying to implement like those things that I'm learning to our marketing. So yeah, it's been good about you.

You always keep it vague, like learning new things, implementing them into marketing. I'm like, okay, tell us more, man, tell us more.

Like I used to say when I joined invesp, about five years now, maybe the next two weeks. So that's interesting. The marketing that we used to do back then is different from the marketing that we are doing now. So for me it's always like a new learning process. Right now, like we're focusing on an ABM campaign that we've been working on like for a couple of weeks now. It's something that we didn't do before. So it's like a learning curve for me at the same time, like I can see some of the initiatives within that ABM program, some of the parts that we used to do, but we didn't really call it abm. But right now that's the new thing.

That is very cool. Yeah, it's just fascinating. Jimmy, by the way, every once in a while we pick up something new. We're always improving our marketing and sometimes I'm like, we really need to do this. I'll give an example of figpi. For a long time I knew we need to do content for FigPi, but I'm like, it's not a priority right now. And it's funny coming from the fact that we built investment based on using content, but I'm like, oh, we don't have the energy. And then eventually we decided. It took me like three years to decide. And then I'm like, oh man, I should have done it three years ago. Same thing with abn. I was like, so at some point it's like you start with something and you think it's just that initial start that is really rough and tough and you're not going to get everything perfect. It's learning and it's exciting as well. And your guest, Ayat, how is your week?

Week has been all right for me. It's always more on like managing the team and we have onboarding some new people. There's always excitement there because obviously you're getting to know new members, growing, all that good stuff. Yeah, my week has been pretty okay. And it's the first week of Ramadan, so just also navigating that.

Yeah, me and Ayat usually do Boot Trapped, which we did not record this week, but hopefully we'll record it. So that's our episode where we give an update about our journey on the invest side and on the fig pie side of things. So you can always catch Aya there. And she appears on so many different podcast. But by the way, I, I do hear through the grapevine that you might be having your own podcast or something. I don't know. So we'll find out.

You know, this is an interesting grapevine. I'd like to be tapped into the same grapevine that you're tapped into many grapevines.

So, Simba, what are we talking about today?

Okay, cool. So today we have about five questions about a B testing prioritization. The first one is, are there any specific frameworks or methodologies that you find particularly effective for prioritizing a B test? Maybe. Like, before we get to this, let's just give like an idea of what a B testing prioritization is.

I'll jump in here.

She takes the easy question.

Okay, so you know, again, like for any type of a B testing program to be successful, you need to be able to conduct research, right? Any type of research. This includes analytics research, user research, getting a little bit more information about what are the problem areas throughout the site that I can really enhance and create a better experience overall. And I always like to even segment those to say, okay, they're like the bug fixes which aren't really a B test, they just need to be fixed in order for the site to just be more functional and better. Then there's more like usability aspects. Again, you want a user friendly site. So those are following the known NNg 10 heuristics and ensuring that, hey, we're addressing all of these on all the various pages of our site. And then finally the last tier, which is the most difficult, most complex experiments, where the CRO really needs to think a little bit more is the CRO fixes, where it's more persuasive, getting into the psychology of the visitor, trying to really just enhance the experience and elevate it even more than what it currently is, taking it the next level from like a standard website to an even better functioning site that really enhances what your value is all about, what the company's all about, all that good stuff. You're collecting all this data and you have 150 different ideas. How do you decide what to do first and how to go first? So there's so many frameworks out there. Every single CRO company has developed their own. We of course at Invest have developed our own. And of course I'm biased, but it's the best framework out there. And what we did actually what we looked at, some of the other frameworks that competitors and other companies follow. So we took, there's some that are very simplistic, some that are super complex. And we tried to find a middle ground in terms of ensuring that we're really addressing the issues. What is this targeting? What is this particularly targeting? And you fill out this form and in order for us to figure out which one will actually go first, the one that scores the highest essentially is the one that should be targeted first. So that's a overview of prioritization and how we've approached it.

I'll ask you, I had the question here and this is fascinating to me because I've spent some time now we have access to so much data at investors that sometimes I'm like, oh, it's there, we just need to look through it. But just based on your experience, and I almost know exactly where I'm walking into, but nonetheless, I will ask the question, how many conversion opportunities between UX issues or conversion issues do you usually uncover on a website? Is 30 normal? Is 50 normal? Is 150 normal. Before we even get into the prioritization. What's a normal number?

I mean, it depends. Then they say all marketers. Simba, I don't know if you feel this way as well. I forgot who I talked to. I think they're somebody from tenuity. This is what I remember. But it. I could be completely wrong. I forgot who it was exactly. But it was a marketer, digital marketer that has literally a shirt that says it depends. They like printed it out on shirts. Because it always depends on the site and always depends on the company and always depends on so many different factors. But yeah, what I've seen, in my experience at least, we would be able to uncover anywhere between 70 to 150 different opportunities. Again, and this is also considering we have something called the conversion framework, which is something that we're digging in a little bit deeper to understand. Where are those trust factors? Value proposition, continuity. So those are taking into account a whole other angle of a B testing that not all companies necessarily follow. So yeah, that's like the. Typically when we finish any type of prioritization, that's the number that we usually end up with. And then of course, like as you a B test, you're really uncovering more opportunities. So you're adding a lot of those opportunities to your prioritization sheet and rescoring everything to ensure that you're still targeting the highest priority first.

I like that.

Okay. I actually think, like you mentioned something about, like the scoring of the test that you have. Are there any certain factors that you'd look at when you're doing the scoring? As in any elements that you look at to do what you call this the prioritization. When you're prioritizing a test, you say like, you'll be scoring, you'll be looking at each hypothesis and scoring what elements of that prioritization framework that you used to do this coding.

So there's a multitude of elements that we look at, but we'll look at, for example, how the particular experiment was discovered, whether it was discovered, for example, through just simply heuristics. That's usually like a lower score. Or was it actually user research or was it analytics? It could be a higher score as a result of that. Again, we also look at the complexity of the experiment or the concept. Because if it's something that is going to be higher complexity and complexity can vary. Meaning is it going to be difficult to get their higher ups to agree on it as well? The client? A lot of times there's a little bit of a political battle that you have to consider as well. And then also the implementation, like, how many hours is this going to take? If this is going to take 20 hours to implement, I really need to take a step back and figure out does it make sense to put this as a high priority at this point in time? So there's just a slew of different things that we'll look at also, like the page value. Like where depending on the page value, experiment is going to fluctuate in terms of score. If it's higher value, it'll be probably like higher priority.

Let me pick up from here and just talk about something because at least for invesp, the prioritization framework has evolved over the years. At some point, we used to score an idea against 18 different metrics, including how much traffic the page gets, where is it in the funnel. And then there's the how do we uncover the issue? Is it based on the conversion framework? One of the elements of the conversion framework is it's based on what kind of research actually helped us discover the issue. So those are all things that help us score an element. But then what we discover, let's say you have a page that has high value. So let's say you have the CART page and it has a really high value. It's, you know, bottom of the funnel, it's getting a ton of traffic, and you end up with the first 10 ideas that you want to work on are on the cart. Guess what? I'm not going to be able to run 10 experiments at the same time unless you're on ebay or Target. But even with those companies, what we've worked with, we did not do 10 experiments at the same time because you can run something called exclusivity groups. But we like to run one experiment per device, sometimes as responsive. So it's made more sense in the next iteration of the conversion framework, which has not made it public because we're still testing it with several clients where we said, you know what, let's calculate the page value separately. And for that we look at several metrics. How many visitors come to the page, what kind of conversion rate for the page, what kind of conversion rate the page has, how long it will take us, what kind of MBE we would get if we want to run experiment for three weeks with a certain number of variations, and then based on that, we do some revenue calculation that helps us understand that what's the homepage? If we have a winning experiment that generates a 10% lift on the homepage, it's probably like, you know, $15,000. But if we have a 10% lift on the cart page, it's worth six, $6,000. So we start looking at those numbers to really prioritize the pages and then we say, okay, the CART page is the top priority and then maybe it's the checkout and maybe it's the homepage and then it's the category pages. Now, within the realm of the CART page or the realm of the checkout, what are the different priorities? So this has evolved over time.

So yeah, absolutely. Again, when you're doing the prioritization, you might end up with two CART page experiments or three that are like consecutive. That's where I always say that the CRO plays a really critical role within prioritizing things. Because you cannot be pulling three experiments for the CART page and just running those for your client. You need to be looking at, no, I want to run on multiple different pages. So I'm going to look at what are those top ones for cart, but then next tier, what's next after that in terms of pages and that way it'll give you an idea of, okay, this is the structure that I'm going to run with for this particular month. Again, depending on our clients, Sometimes we'll run 4, 6, 8 experiments depending on how large our contracts are. Depending on that's how we're going to look at the prioritization and determine which experiments we're going to be running during that month. But having that overview and deciding on, okay, these are the next experiments for the next eight week period really helps first of all, the client to ensure that even business goals are aligned with because that's a huge consideration for us that are these experiments aligned with the objectives and business goals of this company? And then additionally ensuring that we're really targeting all the various pages because we don't want to really be optimizing only one page, we want to be looking at that entire funnel and improving it across the board.

That was a lengthy answer for one Simba. It's like, hey, let's start here. God, man. Guys, we have five questions.

Sorry. That was a detailed one and it was good. So the next question is, how do you balance short term wins versus long term impact when prioritizing A B testing initiatives?

Ayat, since you've jumped on the first question, jump on this one. Let's see.

Well, yeah, it's a kind of like an estimation, right? Because at that point in time you don't necessarily know what type of an impact this is going to have. We always say, for example, Whenever we start a project that we are actually discussing this just yesterday, that we're going to be targeting kind of those low hanging fruits. Even in that particular case, it really is highly dependent on the solution. So even if it's a low hanging fruit that you found and you've uncovered, perhaps the solution that you put forth isn't one that actually impacted the visitors positively and caused actually a decrease in conversion rates. So again, it's hard to necessarily decide what when I'm prioritizing which ones are gonna be the short term, which are gonna have the long term impact. As you work with the client a little bit more, you'll be able to have a little bit more sense of what type of an impact it has. We do a lot of kind of analysis also in terms of, for example, those types of experiments, hey, what was the loe for this one? What type of an impact did it have on the bottom line and what does that mean for the client overall in terms of the strategy that I'm going to put forth?

I think one of the areas that we still can do better ads is having those strategic conversations with clients. Every client comes to us, he's like, hey, I want to improve my conversion rates. Okay, Mr. Or Ms. Will help you improve your conversion rates. But really we're not helping them improve their conversion rates. Flat out, I'm helping you make more money online. That's exactly what I'm trying to do. And hopefully the money that you make online is tons more than we charge you for the service. It's just very straightforward. We love to see you win. But okay, move away from trying to improve conversion rates into what are your business goals for this quarter. What are you trying to accomplish? I think I've seen this with some of the testing that Ayat's team does. Sometimes you're running an experiment because there is a business objective that you are trying to accomplish. And sometimes even though that experiment loses, the client and the team would recommend to go ahead and implement the experiment because there is a direction. But okay, so this is not perfect. We did not have the win that we were expecting, but we're still going to implement it because there's a strategic business interest in a certain implementation in a certain way. Now, as we're making that decision of deploying that experiment, I'll go back to the experiment. Guess what? Now our decision is informed by data. Okay, yeah, this experiment might cause us 3, 4, 5% loss in revenue. We've ran into issues like that. But guess what? Strategically this is important. So There will be a second, there will be a third, there will be a fourth iteration trying to fix some of the losses that we've had, because strategically, this is exactly where we want to head. It reminds me of a story of Netflix one day deciding that, you know what, we're going to drop DVDs, the hard DVDs, completely, and we're going to just change to something just all digital. When your business, when your 99% of your business comes through the DVDs, that's a little risky, correct? Now, of course, they had money in the bank and they had investors and they had connections. So, yeah, there is a mitigation strategy, but sometimes you say, you know what, the business is heading in that direction and we need to test this out, we need to make it work. So what does it take to make it work? So I think that's really the fun side of conversion optimization.

I would also add to that. So when you're making that strategic decision, a lot of times the client will say it really depends on what type of a negative impact this has. So if it does have a negative impact on our business, we want to see what is the extent of it and if we can actually still operate and function knowing that we're going to have this hit, we're going to take this hit. Those are those discussions that need to be had with the client and ensure that again, there's that communication piece and everyone's on the same page of what it's going to have, what the impact is, and what are those guardrails that the client really like. If it exceeds this, we're not going to be implementing it and you have to move into a different strategy. We've had kind of some situations where again, it might may have been us looking at some sub goals of the experiment in order for us to determine what was it that didn't resonate with this particular solution, updating it accordingly and relaunching with a better win or maybe perhaps not as much of a loss.

Yeah, those are good answers. I was getting like a little bit shocked because we had gone this far without Kanye telling a story. He always like, I'm stepping forward, I.

Want everybody to say a story. Here we go. Simba, you asked for a story. Two things. First, I actually, since we're talking about stories now that I'm plugging this storyteller tactics, I actually bought this. So I will tell you guys whether it's good or not. I've been impressed actually with some of the stuff I've just watched the tutorials on how to use it. And I'm like, oh my stories can be so much better. So we shall see if the stories will improve or not. That's number one. Number two, one thing that's always interesting, talking about like testing and making a decision and a data informed decision. I hate using those jargon words, but. But okay, it is what it is. Every time a website owner, CEO, I don't know, president comes and talk to us or a co founder and they say, hey, we're upgrading, we're migrating from this platform to this platform, we've learned. And when I say we through a very hard experience that usually you think that you're getting a trip around the world and instead you're getting an atlas or something like that. So to be careful what to expect for. So I've learned every time a client tells us that is like, hey, so what is the cutoff point where you say, you know what, okay, this is too painful, we need to revert back to the old site. Now every time I tell this to somebody, they're like, nah, we invested too much and we're not going to do that. And I'm like, I understand, I understand you've invested too much. However, what if you launch the new website and it causes you a 20% drop off and conversions there no way this is going to cause unlucky. Okay, if it causes, let's say theoretical. Because they're like, oh, it's not gonna cause. I'm like, if it causes 50% drop off, oh, I'm gonna roll it back. I'm like, okay, so we agree that there is a threshold. We just need to talk about this threshold. Correct? Because 50% is not acceptable to you. And they're like, yeah, okay. They're like, we'll decide when we go live. I'm like, no, when you go live, you have about a million things going on and it's gonna be too stressful. Let's have the conversation right now. Let's draw the line in the sand and let's say maybe it's 10%. I remember. So we're having the same conversation with Z Gallery. They were migrating from custom build. They attempted to migrate to another custom build, went bankrupt because of a migration, which is a whole other story. The amount of money that they were spending left and right, unfortunately. And then they got acquired by one of our clients. And we knew the clients really well because we've worked with them for a long time, on and off actually since 2011. So having a conversation with Jordan Commission and I don't think he was convinced. But then somebody on his board, he says, you know what, I am the poster child for failed migration where I almost lost my business. Let's put that cloud line. And we went back and forth until they agreed, okay, anything more than a 15% drop off in conversions, we're going to roll it back. It's just kind of reality. Correct. So we have the discussion. Setting up a sidewide test, old versions versus a new version is so horrendous, wasn't it server side, Server side lambda. And this is many years ago where, oh my God, it took us almost a month and a half to implement. And then server side testing, especially site wide, is never perfect. And then they decided, you know what, we're just going to go live and we're just going to watch the data. So we went from doing an AB test site wide into monitoring analytics and saying, okay, yeah, there's a little bit drop in conversions. Not huge. It was not horrendous. They said, we're going to go with the new site. But just as a reminder, New Balance 2009, I think that's when they went. Just before Black Friday, a week before Black Friday, they went on, went live with a new website. VP of E Commerce is super excited about it. They launch it and their conversion rate drops by 70, 80%. They rolled it back, it took them about a month because they didn't have contingency plans and many people lost their jobs. So there you go, two stories, man, not for the price of one.

Yeah, yeah, those were good ones. The next question is what are the most common pitfalls or challenges encountered when prioritizing a B test and how do you overcome them?

I'll jump in over here. I think one of the biggest challenges that I see is there are frameworks out there. Correct. And each framework I think to some extent has a weak point. The biggest challenge that I see and pitfall is how subjective framework is. I've uncovered 150 issues. I take those 150 issues, I give them just three of my CROs and I tell them use the same framework. What is that match between those three different CROs? Are they matching 85% like in all categories and the prioritization is very similar or is this CRO telling me here the top five and for second CRO, those top five are like at the bottom of the list or in the middle of the list, then you have problem. Subjectivity is an issue. And I see that as a major issue with some of the frameworks actually one of the more popular frameworks, eyes, it uses so little data that end up with very subjective prioritization. Maybe even worse than that, which is very funny, is people who try to use eyes and they're like, it's so inconsistent. Oh, we're not going to use any prioritization. And who cares about prioritization? There's somebody actually who had made a post about that. I just thought, I'm like, oh, my God. He's like, oh, there's no place for this. And I'm like, yes, that's just like me trying to build the system. And I look at my engineers and I only have four engineers and I have enough work for a year. And I'm like, you know what? Why don't you randomly just, here's a list of 300 different tasks that we need to finish. Just go ahead and pick something and then let's just see. The fact that you've used a prioritization framework and it did not work is a problem either of the prioritization framework itself or how you've used it. Correct. If you think that there's no problem with the prioritization framework, then there's probably an issue with how you use this. So you want to fix that before you say, I'm going to throw what's the baby with the water? What's the expression? I forgot.

Okay.

No one knows it.

Never heard of that expression. You're coming up with new expression, throw.

The baby with the bath water. You know, where you just say, oh, sturdy's just going to throw both. You need to prioritize. Conversion optimization is just not going to exist by itself. It needs project management. We're not going to reinvent marketing. We're not going to reinvent project management. They are there. They've been there forever. So figure out the issue. That's the second one. The third issue that people might not notice is I think we've had that with our previous framework where it was bloated. There are just many things and many things get added. It's like, oh, let's rank. Let's use this as ranking factor. This as ranking factor. So the first iteration of our prioritization framework was almost 24 different elements. We had to step back and say, hey, when people are filling out the prioritization framework and you can do it after, like you pull your prioritizations from 6 months, 12 months. So hopefully you've done, I don't know, 20, 30 different prioritizations, you can see are there Fields within the prioritization that are always getting filled. Yes. 100% of the time, yes. Probably is a useless field in that case. Correct? If everybody's filling it.

Yes.

You know, constantly, I'm like, okay, this is really not very helpful. But again, always question, is it a problem of the framework? Is it a problem of the people filling out the particular field in a particular way? Because the field might be perfect, but people are just lazy and they're just saying, yeah, yeah, we discovered this through analytics. Yes, we discovered through this analytics. I'm like, show me the analytics. Oh, I don't have analytics.

Okay.

It's not the framework problem. It's a Joe Schmo problem. So let's fix that. And then the final thing that I would say is after you prioritize, like I had said, a CRO or a digital marketer or a project manager needs to sit back and say, okay, so here are the top priority items. Do they actually make sense? That's not going to be on auto drive. You have to sit back, make sure that it all makes sense. Are all the items getting very close score? Everything is getting 75, 75, 75, 75. I'm like, okay, either you have a problem with the framework or how it's getting filled out. So those are just kind of things that I look at as pitfalls that people don't pay close attention to.

I'll add a couple of things. I think what I've found in my experience, and again, I think it always goes back to like human error or the ability to kind of add things and be really proactive. I think that's the thing with frameworks and CRO in general. You really do need to be very proactive. And if you don't add, for example, the wins and the, even the losses and the information and the learnings from that back into the framework, there's so many missed opportunities there and you're not going to be able to prioritize them accordingly. And you're really basing a lot on, again, just the intuition of the CRO, which sometimes is just not enough necessarily. So I think that's a key thing. It's making sure if you're conducting a CRO project, that research is always included and then that research is also part of it, is the A B tests that you're conducting, that's part of your research. So adding those opportunities back into the framework and then rescoring it all is a really good way to ensure that you can build on the wins and keep that prioritization rich with ideas. The second thing is, and Khadid mentioned this one is just the CRO. When you're putting together your eight week plan, really being looking at it and making sure everything does make sense, it is still something where you're scoring manually and you have to make sure that it makes sense for the client for their business objectives. Aligning that with the business objectives that the client has, the last one I'll say is where we see some pitfalls is very often we have our clients recommending certain experiments. The hippo in the room is deciding, I want to do this on the site. So making sure that those are also added to the prioritization and then showing the client, hey, the opportunity that you actually mentioned is really low. It's like a low priority. We have some other priorities that we want to move forward with before we do. Because I think making sure that it is, you're not just taking something the client is saying and implementing it and you're actually feeding it through this prioritization again makes the whole prioritization concept a lot more meaningful.

I give you thumbs up in the working on LinkedIn Live, I'm like, I like it.

Okay, that's good. So the last question that we have is when working with the team, how do you ensure alignment on the prioritization of A B testing ideas?

Well, I mean, I think it's just about building the right processes within your organization, ensuring everybody's on the same page, ensuring there's buy in from the entire team. We've recently been doing a lot of work also aligning the type of trying to categorize all the different experiments that we're conducting within certain categories just to ensure that there is a streamlined process, a streamlined way that we're looking at problems, a streamlined way for us to actually pull data. Again, I think it's just about making sure that you're always optimizing the processes and you're having discussions with the team to ensure that everybody's on board with whatever it is that you're doing.

I'll add something there because you create the prioritization plan and the prioritization plan. If you're prioritizing 150, think about this. If you're prioritizing 160 items and you're saying like, no, in a month I might do four experiments. Four experiments, 160 items, that's not 14 months, that is three and a half years worth of work. It's way, way ahead. You look at the business objectives. That helps you. I Think that's one of the criteria that you want to look at. If there's a strong business objective driver, the client wants a feature that gives an item a priority, but still it's ranked within the framework. Okay, so I have this three year plan, one year plan, two year plan. I think what Ayat had just mentioned, where we like to take the plan and say, okay, what am I going to be working on in the next eight weeks to 12 weeks, right here is my, I have my long term plan, I know where I'm heading, I have the map. But in order for me to get there, okay, what are the next three moves that I need to make? Okay, so that's eight to 12 weeks. And now I'm having both internal discussions and discussions with the stakeholders. If it's a client or if you're internal resource. Here's my plan. Now, although it's an eight week plan, it's subject to change, correct? I've ran some experiments. Things change. Guess what, it's flexible. What I like to say is if you're holding a weekly call with the client team, if you're holding that weekly call, you say like at the end of the call, hey guys, here's our next eight weeks plan. We're going to work on that. That's what we're going to be changing. That's what we're going to be testing. Any changes, any recommendations? So it's always about making sure that really there's visibility and then there's alignment. Correct. Everybody knows. Not everybody's like, hey, what happened to this plan, this experiment? I thought we talked about it. No, we've changed, we've shifted priorities. Everything needs to be very well communicated so we're all on the same page. And I think, by the way, that's the very first step of getting people excited about experimentation. That's one of the things that the CRO needs to do. You need to get everybody excited about the experiments, the research, the experiments and the results. And then finally, if things go well, the business, how we help the business actually achieves the goals that it was, it was trying to achieve.

And that's really kind of like full circle to what you mentioned at the beginning of the podcast, Khaled, about storytelling. Because I believe that CROs need to be really good storytellers and they need to be able to tell the clients through their story of why this is a really great experiment, why it's going to have a positive impact. What type of an impact did it have? All of it is about how you deliver that information and it's so critical. And I'm going to just say to what you were saying about the framework is I always tell people your prioritization sheet is a living document. We've mentioned this throughout the podcast, is that you're always constantly updating it. You're always constantly reviewing it. Even after that eight week plan, I'm feeding in a lot of information. I've run experiments, I've run some more data, I've looked at analytics again, heat map, some recordings, whatever it is, I'm feeding more information back into my document. So it is a living document and it is subject to change. And of course, just making sure that you're in alignment with the client is the key thing.

Yep, that makes sense. So those are the questions that we have for today. Unless maybe you want to share more stuff, I'm here for it.

I have nothing to share, man. I am fasting, operating a little sleep. But I the discussion was good so I enjoyed it. So, you know, block prioritization, it's always good. Thank you AAT for joining us. What do you think of CRO Live hour? Fun. We're always talking about interesting, fun things. Thank you everyone for listening and until next time, happy testing. Sam.