#56 – Program Management in Conversion Optimization

How can program managers cultivate a culture of continuous learning within their teams without pushing them hard?

Let me challenge the premise of this question. Is it the responsibility of program management to cultivate the culture of learning, or is it the responsibility on all the team members? My name is Kalislade, and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal, while pretty.

Much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we'll talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man, I bring tough questions like, do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of your conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today, wherever you listen to your podcasts. Another week, another CRO Live Hour episode. I don't know what number 60, 70, something like that high up there.

On the podcast side, it's 60 something, but like on the LinkedIn side, it's more than that.

Yeah, it's double that on the inside, but it's all good, man. Simba, how are you doing? How is your week? How is the Q1 going for you?

My week has been good so far. Q1 Also, I think so far, like, we're beginning to see some signs of stuff happening because at the end of Q4, we spent it, like, planning our marketing. And then just like about a week or two weeks ago, we then implemented it. So we're starting to see some signs of good things happening. So hopefully. I always also look at some metrics to do with the website traffic, which was stressing me the whole of last year. I was doing like, all my best. And I was also working with Adam, who helps us with SEO. Like, we're doing all that we can, but still there seemed to be like a deeper problem to that. But right now we're beginning to see some uptick of which, like, that's a good thing. My goal is to get to at least 10k or something by end of it's. Funny that I'm now like aiming for 10,000 users.

It is.

Per week. Because 10,000 users used to be like something that we got like in one week. So.

Yeah, I know. Like, now we're like, okay, what happened? So basically we did a blog updates. We went to a better template, faster templates, and that was mid August and we got hit really hard. So a couple of things, by the way, happened in August. We did the upgrade and there was a Google algorithm also, like in a content, useful content updates. But I think our content is useful. But maybe, I don't know, it's coincided. And it's funny because I ran an audit today in the morning and I noticed some problems and I'm like, okay, actually, I probably shared the Google sheets, but here's good news. We're doing it on the podcast. You don't have to do anything with it. We're just going to have one of our developers just write one query, fix it. So we continue working with SEO. What else have we focused on?

The other thing that I've been focusing on, like, we've rolled out a few campaigns on Google Ads and also like some campaigns on LinkedIn for us. Google Ads. Like, we used to use it sometime last year, in fact, all of last year. So we know, like, that it really works. We just have to get our advertising right, our landing pages right. But then there's also like the LinkedIn. I know LinkedIn works for B2B. I've seen like a ton of case studies about it. It really works. So we started, like, also experimenting with it. We just rolled out, like, the first campaign sometime last week. So far, like, the plan of the initial step of that campaign is doing well, but it's too early to, like, judge if it's a success or not. So maybe in the next two, three weeks, that's when we'll see if it's really working well or not.

I'm anxious, cautiously optimistic to see what happens with LinkedIn. So, yeah, it'll be interesting. I'll ask you about it. Tell me.

Yeah, so far the signs are good, though they are not that bad. Right?

The signs are all dead and we can share. So we've taken a video from one of our customers, put it out there, and people are watching it. They're watching the full video. How long is the video? It's not very long. Correct.

The video is about 45 seconds long and we've had about 45,000 plays. That's a good thing.

Oh, that is an excellent thing. I wouldn't have imagined. So let's see though, that's just a play. There's a lot more that gets built on top of that. So I am cautiously optimistic. Awesome. Awesome. So super exciting.

How about you?

Just many things, actually, it's been fascinating because one of the things that I have access to is a lot of experiments data. And for the first time I was working with the INVESP team, pulled all the experiments that they've done last year, pulled the hypothesis, the classifications of them. So it's this very impressive way of 2,000, 3,000 experiments. And it's not about the experiment itself, but you start looking at the rationale behind the experiment and looking at certain patterns. So that has been just absolutely fascinating. And then it's very powerful when you're able to say, we've ran this same concept. We've ran it about, you know, 22 times, and of those 22 times, 18 times was actually successful. It is very powerful as you convince new companies, say, you know what, we need to pay close attention to this. And we've ran this concept on the product pages for E Commerce. What can we do for the PLPs for the category pages? What can we do for, you know, just so many things based on the data that's just extremely powerful. So it's been exciting trying to make sense of all of the data. And it's been an exercise that I think there's about eight people working on. So it's just fascinating though, and we're going to be starting rolling out some insights from that. So it's just really nice. I'm enjoying the process.

That's a good thing. One of the interesting things is, like, today we're talking about a different topic. I don't think we've touched on this, like, in the two years of doing LinkedIn life, have we?

No, you tell me. I don't remember us.

I don't think so.

Yeah.

So like, that also goes into showing, like, how broad CRO is. Let me know when you're ready, then I'll shoot you the first question.

No, go for it. Yeah.

By the way, we're talking about program management in conversion rate optimization. Just for people. Maybe know what we'll be talking about?

Yeah, I'm ready. Shoot away, my friend.

Okay, so the first question is just like an easy one. It's easy compared to the other questions which I'm seeing here. So what role does program management play in conversion optimization for us?

And based on our experience, program management at some point used to be very critical. It's the glue that brings all the team members together. It ensures that things are running on time, within the budget, within. We're allocating the right resources. And I think program management becomes more and more critical, the higher velocity you aim for. If you're running a single experiment every month. I don't know if you need a program management, but I wonder if you're doing a single experiment per month, if you're doing conversion rate optimization or if you're doing experimentation. Yeah, you are doing AB testing, but that is just too slow to see any benefit of that. The minute you start getting to 4, 6, 8 experiments per month, you need program management because now you need somebody to keep an eye on all the different team members and making sure that they execute correctly. In the past, and here's how we do it a little bit differently. In the past, we used to have a CRO pod, and that contains the conversion rate optimization specialists, that contains the UX designer, the developer, the QA resource, and the project manager. And the project manager and program management was leading our pods. We've changed this in the last couple of months where we said, you know what, we're going to do this a little bit differently because the relationship that the client has is mostly with the CRO specialist or the senior CRO specialist who runs the pod, who understands the business, the client business, and whatnot. So we've made the CRO specialist the leader of that pod. And everybody in the podcast reports to that CRO. The CRO is almost, I like to think of them as the CEO of the pod. They're responsible for the client, they're responsible for getting the work done, they're responsible for delivering, they're responsible for maintaining the relationship, doing the analysis and whatnot. The project manager is there to support the CRO. So they still, by the way, play the same role, making sure that everything is getting done on time, things are moving correctly. They. But they are there to assist the CRO to make sure that the program runs. And it might sound like a small, slight difference between the two approaches, but I do think that having the CRO, since they are the center of the universe of that POD and they're responsible for it leading the pod makes a lot of sense because ultimately sometimes the CRO will have to make decisions that, hey, we're running these experiments, does it make sense to run this other set of experiments or do we need to wait? That's an experimentation decision. That's not the project management decision. So the CRO decides the initial roadmap and then they work with the project manager to lay that out over three to six months and communicate that to the client and to other team members. That's the way we've done it. Seems to be working well, I'm sure other people have done it differently because I know, by the way, many of the agencies. Let's talk about the agencies. Many agencies don't actually have project management as part of their structure. For us, that worked at some point. But the more experiments we were taking on, the more sophisticated clients we're working with, the more we felt that we need to have project management as part of the program.

Yeah, that makes a lot of sense. I want us to dig a little deeper into the role of program management because are there any checks and balances that project managers should incorporate into workflows to ensure that teams deploy high quality experiments? What exactly do they do? If you can shed more light on that.

Yeah, it's difficult to judge in all honesty what a quality experiment is. It has nothing to do with the complexity of the experiment. Correct. Because you can create the most complex experiment, but it could be completely useless experiment. So the way we measure the success of program management in a CRO program is in the output that the team is doing. We have commitments that we're making that the CRO pod is making in terms of, oh, here's the number of experiments that we are going to launch. Okay. So the program manager needs to make sure that we are delivering that output. And that seems to be very basic. Correct. But if you're committing in a quarter to deliver 20 experiments and you've only delivered 15 of them, we're going to be raising questions. Hey, what happened? You've missed your goal by almost 25%. That's going to be a big question mark. So that's number one. Now you need to balance this out because there are some program managers who are going to push really hard and they're going to push their team members. They're constantly chasing them. And then you have the morale of the CRO pod of team members dropping. So really you need to be doing this very carefully. So we do quarterly reviews where we're asking the CRO and other team members to assess each other and we're watching the morale of team members on the pod. That's, by the way, a good way to measure that. Hey, is everybody doing what they do in a comfortable way? Because you want to make sure that they have the space to do the work that's required of them and at the same time they're delivering IT on time. So it's this fine balance. So how do we measure quality? Because, okay, you might be able to deliver the 20 experiments, the team members are happy and everything. That does not necessarily mean that you're delivering quality. We measure that through two different ways, and different companies do it differently. But this is what seems to work for us. So we measure that first through the client MPS score how satisfied the client is with the work of the pot. So that's extremely important. And our goal is to get nine or 10 out of 10. If we drop, we have some questions to ask. Hey, guys, what's happening over here? Because it's not only about the experiments corrected. The clients will get really upset if you're delivering to them experiments that are of no good quality and they're questioning them. It's like, hey, what am I getting over here? But at the same time, also, this adds another dimension, which is I'm also measuring the relationship between the POD and POD members and specifically the CRO and the project manager and program management with the clients. So that's something that matters to us. And then second thing that we measure the performance of the POD and specifically, by the way, measures the program manager and the CRO is the win rates. Okay, so you've watched 20 experiments in the last three months. What was your win rate? Now, what win rate do we aim for? This is actually different from one company to the next. Completely different. If we're working with a company that's been doing CRO and experimentation since 2010, win rate around 15 to 20%. This is really amazing. Correct. But if we're working with a company that has not touched their sights in the last 10 years and we see just so many things in it, we, we're aiming for a win rate at least of 33%. Our average is around 42% at this point. So that's a different way to measure this. Is this perfect? No, I think it's work in progress. We are, I joke about this. We are croing our own CRO process and our project management process, trying to discover ways, trying to optimize ways to really deliver the value from all the testing that we are doing.

Yep. I like the metrics that you use to measure. As you were explaining, I was also like thinking, does it also make sense since you already measured the win rate, does it also make sense to measure maybe the number of experiments that you launch during a certain period of time? Or it will be like a vanity metric. Like, considering that we're an agency working with the client, does it make sense to measure the number of experiments that are launched within a certain period of time?

So we measure all of that by the way we measure our velocity. We measure velocity versus the percentage revenue impacts. Like, hey, I've increased my testing velocity to 6. Am I seeing a revenue impact, percentage revenue impacts that corresponds to some of that increase in velocity. We don't hold the POD responsible when it comes to these numbers because I think if you're not delivering the commitment that you have to the company, to the stakeholders, to the client that you're working with, guess what's going to happen? Your NPS score is going to fall. Correct. So we're considering the NPS score as a way to measure multiple things. Is the client happy with the quality of the experiments? Are they happy with the number of experiments that we are delivering? There's secondary metrics that we look at that velocity, return on investment velocity versus percentage and revenue insights. There are lots of other metrics that we're measuring and I think when you look at all of these metrics, they help you tremendously in doing two things. First, showing the value of the program to the stakeholders. And number two, if you show the value correctly, then you're keeping the client for a long time.

As I was reading like also about this topic, I also noticed that some other people may be for us it happens, but maybe mostly from the client side. As in they like to measure the decisions made from insights. As in like how many impactful insights resulted in action by maybe stakeholders. Something like that.

Yeah. It's funny because I was reviewing one of the companies been with us now for about three years. I was looking at the number of experiments, ton of experiments that we've launched for them in the last three years. And by the way, it's one thing to find insights when you're running a program for six months. It's a whole other animal when you're looking for insights after doing something for three years. Because just the number of experiments, just sheer volume of experiments that you have to deal with, this is a little scary. But what's fascinating to me is there is a decision. There's insights post every experiment. We call it a PTA document, post this analysis document. But we start by saying there's two types of insights. There's an insight from the winning variation. Correct. I've run an experiment, I have a win and there's insights even from the losing variations within a winning experiment. I've launched an experiment that has four variations, variation. One beat the original. Great. Our hypothesis correct variations, 2, 3, 4 have lost. What can I learn from each one of those variations? What are the insights that I'm drawing from each of those variations? We also look at, did the client at the end deploy the experiment or not? That's an important factor because you might have owning experiment, but the client doesn't believe enough in the win. And maybe the impact of the experiment is so minimal that the client says, no, we're not going to implement. That's very important. Correct. Because that will tell you at least at a high level, did you have something valuable that you've learned from this experiment? It's a way to measure the insights that you're getting from an experiment. Is it perfect? Again, man, I don't know if it's perfect, but it's definitely one way that we look at insights.

I like the explanation that you just gave when you were talking about the pta, the post test analysis that you also consider like insights from losing experiments. Because a lot of people, I usually see it within like experimentation spaces. Most of the people, they love to glorify the winning experiments, to talk about the experiment that won and the lessons probably that they got from there. But talking about the losing experiments, not everyone does that so well. Interesting that you mentioned that.

Yeah, no, not only that, Simba, think about this. Most people, even if they're like, hey, we're going to learn from losing experiments, I tell them I love an experiment that wins. And anybody who tells you that they don't, and I'm like, okay, I'm sorry, you probably were not responsible for money in a company and the P and L statements, you know. But I love the insights even more because I think they are fascinating. But even not at the winning experiment level. Correct? Because okay, so that's one thing. But even within, oh, sorry. Not at the losing experiment level, but look at the winning experiments. Because a winning experiment will most likely have multiple variations. Let's do the analysis. Hey, what was it about V1 that actually helped us score a win? And what is it about V2, V3, V4? What is different about those? And why did they lose? And what assumptions am I making about customers based on the fact that these variations have lost? And you start asking questions, you start forming more theories. They're just theories. Correct. You still have to validate them through more experimentation, through more testing. So what if maybe there are about.

Three variations that won and then they performed better than the original and they're all very different? What does that tell you? As in someone like, who's Running the experiment.

Yeah, to me I'm looking at the top variation to see, hey, what kind of lifts? Oh, this top variation did the 17%. Second one did 4%. Okay, so there's something there. What did we do in V1 versus V2? Why did it deliver double digit lift versus single digit lifts? Okay, what's if they all deliver something very close? This is 8%, this is seven and a half, this is 9%. Sometimes people say, oh, it's obvious we're going to go with the nine. I tell them, a lift that you're getting in an A B experiment is not the same as the lift that you will see when you deploy this experiment as a winner on the website. Anybody who says, We've had an 8% lift over here in this when we're doing the A B test, that means my site's conversion rate is going to go by 8%. I tell them, you have not done enough A B testing in your life the minute you say that. So that's the reason to me, 8 and a half to 9% to 8, they're very close. So how do I make a decision that's number one? What do I learn? I say, okay, the three elements, they're completely different and they all have an impact on my visitors. Guess what? I will take each one of those elements and I'm going to test it out. I'm going to test it out on the same page. I'm going to test it out on other pages of the website to try and see and maybe it's a combination between those different elements. That's what I'm going to try and achieve. That's number one. Number two, within those three variations, I'm going to say, hey, which one actually matches where the business is going strategically? What direction is the business going in just to even just add a whole other dimension. Because sometimes people like to think of experimentation and as a one dimensional. I tell them it's not even two or three dimensions, it's probably seven or eight dimensions that you have to consider. Because really experimentation to me is about making the right business decision. And the right business decision in the short term sometimes does not make you money, but in the long term does make you money. So we're not looking for overnight success. Those are nice, but we're looking for long term success. So sometimes you might say, you know what, I ran the experiment, I have three variations at 1 and the fourth variation have lost. And I will go with a fourth variation. I have seen that done.

Why?

Because the fourth variation is the direction that we strategically are taking the business in. Okay, so we're going with the losing variation. Do I stop there? First I'm going to say, hey, how much money am I going to be losing by deploying this fourth variation? So I'm going to measure that very carefully, and then I'm going to say, hey, based on what I learned from V1, V2, V3, those winning variations, what adjustments can I make to my losing variation to make better? Correct. So we end up winning eventually. So there's lots of moving parts to an experimentation program that you're looking, you're analyzing. And by the way, when you do that, I think it excites everybody. It excites the team that's working on this. It excites decision makers as well.

As I was listening also, like to your explanation, I was also thinking in your explanation. So does it mean that sometimes you launch tests which are not aligned to the objectives of the company, as in, like, the direction that they want to take? What if that kind of an experiment wins and the one that, like, aligns with the company objectives loses? So what's next?

So what's next to start a CRO experimentation question? It's a business question at that point in time. And what you do, as somebody who's running an experimentation program, a conversion optimization program, what you do on it is provide the data points to the decision makers to say, hey, this direction that you are going in, at least based on the experimentation that we've done, based on the experiments that we've done. And we'll probably want to provide that to multiple experiments, not a single one to say, hey, one experiment. You need probably to do multiple. But let's say multiple experiments have proven the point that within those experiments that losing variation is what's where you're taking the business in. The decision maker will have to make the decision at that point. But here's the other thing. Lots of times I hear CROs and experimentation program, they complain. They're like, oh, decision makers, you know, we gave them this data point and they don't understand and they decided to go in a different direction. I tell them, no, no, no, no. Guess what? You're looking very specifically within the context of your experimentation program. That's one input that the decision maker has. They have probably a ton of other inputs, the market condition that the competitions think, something specific to the company. So lots of times people make decisions like that. And you look at it in the short term and you say, makes no sense. And in the long term, it's extremely powerful. That's the reason I would say I'm like, hey, experimentation is there to support decision making process, but it's not the only way to do decision making process. It's an input to decision makers. Correct. And then they need to look at the full picture and then based on that, they need to make a decision.

Okay, yeah, that makes a lot of sense. Sorry for taking you off like the main topic of the day and just labeling.

I like it.

So taking a different direction. So, okay, I'm just going to go back to what we're talking about. Program management. How can program managers cultivate a culture of continuous learning within their teams without pushing them hard?

Let me challenge the premise of this question. Is it the responsibility of program management to cultivate the culture of learning or is it the responsibility on all the team members? It's not the duty of one team member. And if somebody's going to carry the heavy burden when it comes to that, it's going to be the Croatia. Because they are again, in our universe. The CRO is the leader and program management is there to support them. How do you do this culture of experimentation? Through continuous discussions. Experimentation is not a top down approach. Experimentation is a team sport. We're having a discussion around why we think that there is a problem. We're having a discussion around why we're producing those different solutions to the problem. And then we're looking at the data to say, oh, this solution worked. This solution did not work. If the only person who's concerned about this finding the problem, coming up with solutions is the CRO, then you don't have an actual CRO pod. What you have is one person doing the thinking and then just delegating the details of execution, coding the task, qa, designing it to other team members. You basically limit the effectiveness of that podcast tremendously. So to me, that's constant discussion.

So.

And it starts, by the way. So we do regularly what we call design review sessions. But although they are design review sessions, they're really problem review sessions. Because it starts with the CRO saying, here's the problem that I notice on the client website. Here are the data points that support that problem. Okay, hey, Mr. UX Designer or Ms. US Designer, what are the solutions? So it's a very active and engaging discussion and you can tell whether somebody's engaged or not. If I see a team member who's just, okay, so doesn't participate? Unlikely. It's fine if you do it once, but if you're consistently not participating, we're losing on the value that we want to get from you. And that's usually extremely important for me that I say, you know what? Okay, yeah, you might produce amazing designs, but more importantly, you might be producing amazing code. But more importantly, I want your mind thinking about the problem and are we producing the best solutions for that problem? Because then the effectiveness of my pod goes from 20% goes up to 80% and 90%. And that's my aim.

I think one of the things that most CROs won't agree about is maybe the approach to CRO, because I've had like some people saying or defending that CRO has to be like a top down approach. Some are saying like a bottom up approach. So I don't think like there'll ever be like a consensus when it comes to that.

I think it's a mix between the two. There you go. Like sometimes people sometimes ignore the top down. They're like, oh, it's not important because it's not based on data. And I tell them, you know what, you have to understand the top is what pays the bills. So it's important to keep the top happy and satisfied. Now what you want to avoid is that the top is dictating 100% of your experimentation. It's like, oh, here's what we're going to do for the next six months. Here's why I want this and just random ideas. Because then, okay, the top is killing the whole program. But you need to leave space in your testing program for ideas from the top. And where this is powerful, by the way, is when those ideas fail. Yesterday I was talking to a company absolutely amazing. They do about $55 million annually. I was talking initially, talking to their VP of marketing and he said, hey, can we get on a call with the CEO? Love talking to CEOs. We're discussing the process and all that. And then just by the way, side note, she's like, yeah, I'm going to be flying to Hawaii for the next four months. And I'm like, wow, that's a rather lengthy vacation. And then the VP of Market is like, now she has a house there, she's going to be working from Hawaii. I'm like, that's a good thing. Besides the point, she was telling me because they have run an experimentation program and she's seen the value of it because she said there's multiple experiments where she asked the team to implement because she was so sure that they will produce a positive impact on the website. And she said people simply did not like the solution and she was shocked by the results. So there are two things that I told her. One is good. Now you're learning as a CEO, as a C suite person, that not all of your ideas are great. Sometimes people absolutely hate them. That's number one. Number two, although some of those experiments have lost, it doesn't mean that the idea is bad, by the way. It just may be. What had lost is the implementation of the idea. And we need to evaluate and we need to look at it and maybe we need to test multiple times before we come to the final conclusion that the hypothesis is incorrect and that there's no problem over here and that we just need to move on.

Yeah, that's a good one. I think it's important, like when the stakeholders also put like an input and show some interest in the CRO program. One last question that I have for you. Are there like any elements of program management that need to be like, maybe in place in order to scale the experimentation across different teams? Yeah.

There are certain things that you have to think about and some of them are very simple. It's going to sound silly saying them, but good project management software is extremely important because you have typical project management. Okay, so we have a ton of software out there and I might have a ton of people working on it. That's fine. The difference between an experimentation program and a traditional software development program or marketing program is with an experimentation program, you have a series of mini experiments that you're launching. So you have a whole bunch of small projects, maybe 200, 300 of them, that you're launching over a year. And within the project there are five or six people working. Correct? So five or six people basically doing something, handing something to the next person, jumping, taking something from the queue, doing analysis, doing the work. Project management software becomes extremely critical to be able to move correctly. That's number one. Number two, you need to be able, if you are going to scale the program, you need to be able to look at insights. You need to be able to look at. There are just so many metrics from a project management perspective, away from all the CRO ing that we do, there are insights, project management insights that we need to look at. Hey, how much time for 1 hour of CRO time? How much are we spending in designing QA and development? Correct. That's important for us. What are the metrics? How many times are experiments going back and forth between dev and qa? Again, that's critical to have those metrics before you scale the program, because if you don't you end up with a hot mess at your hands and it's like, oh, I don't know what's going on, too many things going on and you just lose your mind doing all of that. So that's in terms of the processes and metrics. I think culturally it is extremely important to know that you need to be able to have two characteristics that are very important to project management. One is transparency. So you're able to give direct and honest feedback to team members. Especially because you're going to be working closely with each other. I've led software development programs where we had 120 engineers and I thought that they were stressful, by the way, because I've run a project with 120 engineers. Three months. We're working constantly. The fact that was running over three months gave us a breather and the fact that we didn't have the delivery for three months gave us a breather. So it was like you fall behind in one week. You're able to make it up. Correct. Because you have enough time to make it up. Your whole project is within a week. No, the stress is much, much higher. So transparency is extremely important. Being able to give feedback and take feedback is very important in order for you to run successful project management program. Those are the things that come to my mind and I'm sure some of our listeners can add more things based on their experience and what they've done. So would love to see by the way in the comments or you can always reach out to us via LinkedIn on your thoughts if there's anything that you want to modify or based on your experience in terms of project management for experimentation programs.

Yeah, that's good. Those are the questions that I had for you today. Unless maybe you wanna I have time to listen to you too?

No, you always ask me that. I don't think I have anything to add. But I tell you it's funny because sometimes, and maybe it's just the nature of the beast, I wonder how we managed to survive for about eight years of Envisop's life without having program management in place. And maybe because we were not just working on really high scale high velocity programs. And then when you move to high scale high velocity, we understood that there's need for project management. And then even as we worked with project management, it changed how we envisioned it initially and then how we evolved it over time. You constantly have to be optimizing your process as an experimentation company or as an experimentation team.

Yeah, I think that's like how it happens because I also remember, like, the time when I joined invesp, the team was starting to implement jobs to be done.

Yes.

You know, but right now, like, they are one of the most important elements of the CRO process. So maybe back also, like in 2008 or like I had in 2008, 2009, you look back to this time and say, how did we not do this again?

Yeah.

You're always in optimizing.

Yeah. But I tell you what's even more fascinating to me, because you Google my name and you look at the podcast that I've done. Jobs to be done is something that I've spoken a lot, and now it's a fundamental block in every program. The innovation that we have now is in prioritization and putting the roadmap in strategic CRO, which is part art, part science. You have to mix the two in order for you to arrive at the right input. So there's so much innovation that we're doing in that space that I love Jobs to be done. But I'm so eager to learn more about strategy and CRO strategy versus business strategy and how they support each other.

Yeah, that's good stuff. Awesome.

Thank you, everybody, for listening. As usual, we always ask if you are listening to this podcast. If you find it useful, share it with somebody wherever you listen to your podcast. We would appreciate that. And until next time, happy testing Shop.

Bye, Sa.