#5 – Cost Of Hiring A CRO Agency, Price Optimization, Machine learning and CRO, Psychographic Profiling, etc

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh man. I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are. But we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today wherever you listen to your podcasts. So how is, how is your week? I mean we're, we're finishing up the quarter. Can't believe today's the 30th of June by the way.

Yeah, tomorrow is actually like, tomorrow is actually like the first day of the third quarter. So we started like working on our OKR as sometime last week. So it's been good. From what I can tell from the meeting that I had with the team today, the marketing team, like everyone is excited like and they are also waiting for the new quarter. It's good that we have like this long weekend. Everyone is going to go and refresh and then come back.

Yes, yes. Yeah.

With a refreshed mind.

So we're off like you know, of course this coming Monday because of the 4th of July and then also we have an invest holiday the Monday after. So we'll have two long weekends which is nice, a nice way to start, to start the quarter. So yeah, it will be interesting to see how the marketing team performs by the way. Just as a side note, I didn't see the marketing team okrs. Not, not that I have the time to see them as you can see from the messages that I'm sending you. But I'd love just to get an idea at some point and it would be interesting by the way and I don't know if you're up for it or if the marketing team is up for it to get some of the marketing team next week on our CRO Live hour. That would be Interesting okrs. You know, let's talk about, like, you know, so it'll be interesting. So.

So. Okay.

Okay. Ours is one thing. What else is going on with you? What have else. What have kept you busy?

One other thing that has been that I've been thinking about, like, a lot or that has taken much of my time is there's this one article. I've been talking about it, I think, for the past two weeks, you know, about building a culture of experimentation. And I've been working with different people who have been, like, helping me, like, create something which is better than other articles which are out there. Not to say that other articles that are out there are bad. No, we're trying to create, like, something which is more practical. And I have, like, this product manager, Jordan from booking.com. he's very good when it comes to experimentation. So we've been like, exchanging messages, like, I think since last week about this article. And I also, like, had to show him, like, the first draft of it and then he had some comments. So we've been working on those comments, reviewing them, everything. So it's going to be an interesting article, I think. I have written so many articles that I can tell if an article is going to be hit or not. So I think this is one of those articles that I think, like, more people will love.

I am eagerly waiting for that article. And I saw. I saw Jordan's comments, by the way, and I'm like, oh, this is good. You know, you can always look at comments and you can ask somebody to review an article. And the truth is, when you. When you write an article, you interview somebody, I mean, they're very generous with their time in the first place to do the interview, then to ask them to review and give feedback. It's a lot of work, correct? I mean, I can't tell you, of course, like, a few times where I committed to, like, do the interviews, and then I don't even.

Don't even.

I can't even show up to the interviews. And I feel really bad. But then to like, you know, for Jordan to come in, give us like, you know, like, looking at the time for the interview and then to go in and give feedback on the article. I was like, okay, this is a really good piece of content. I am super excited. That's all I'm trying to say. Looking forward to it. Awesome. Awesome. So that, that.

How about you? How about you? How was your week?

Busy, busy, busy, busy. I mean, it's interesting because invesp is at full capacity, you know, in terms of projects. And as a matter of fact, the team doesn't know, but we're kicking off also next week with another client. I mean, Invest capacity as a conversion optimization agency is a lot less, by the way, compared to some of the other agencies in the field. And just it's an intentional decision that we've made where with the staff that we have right now, other agencies do probably three times or they carry three times the amount of clients that we carry. So if we max out at 18 clients with the staff that we have, with the team members that we have, other companies use the same staff to do probably close to 60 clients or 54 clients. And I know that because, I mean, I talk to other agency owners. But that's an intentional decision because there's a certain standard that we want to give. Not to say that other agencies, they might have figured something out, but when people come to invest, they're looking to hire usually, like, they're really coming after. I mean, like, they want the CRO queen. They want her to be on projects they want. You can imagine that I cannot really talk to Ayat for the last couple weeks, but there's a certain standard that the companies that we work with are expecting. And that means we need to spend a lot more time kind of investigating issues, thinking through issues. So it's rare for me to be able to do work like more than 30, 40 hours and say, okay, I'm done. Usually we're doing 100, 120 hours, 150 hours. So it's a lot more work. It's very busy. But some of it is, like, you know, very interesting because we're reaching that level where like, okay, we're near full capacity. I'm having to do things that I'm like, oh, this is interesting. So just before this call, I was just doing some statistical analysis for the revenue impact of a test. So there's a certain method that armchair had introduced 14 years ago, 15 years ago, some companies use it. Many don't even use it and don't know how to do that analysis. And we've been using it for a while, but the method is outdated, so it needed changes. And over the years, I've been kind of creating different, new, different methods. And now I'm like, okay, let me sit down and write all these formulas. I had created this humongous Google sheet to do the calculation, and I just sent it to iat. I'm like, okay, the team can review this and we can have a meeting to talk about it. So it's interesting work. At least that's what I'm trying to say. I guess.

Talk to us about what do you think is the amount or the load or maybe should I say the number of clients that one CRO specialist can handle?

That's a good question. It depends first on how knowledgeable that CRO person is. You're talking about somebody, let's put juniors aside, somebody who's just starting out and conversion optimization. Let's talk about somebody who's been doing CRO for two years versus somebody who's been doing CRO for 10, 12 years. Completely different standards. So that's one thing that you want to consider. Another consideration that you want to think about is the support that CRO person gets. Because CRO doesn't happen in a vacuum for us. A CRO usually works with a designer, a CRO works with the developer and then there's a quality control person and then there's a project manager. Now if the CRO, regardless to how much they know, CRO will have to do, for example, the quality control so they are validating that the test is actually implemented correctly and that there's no bugs. Well, that's a lot of work. Correct. And that's a lot of testing that needs to happen to validate that there's no issues with the test. And when I say it's a lot of time, I mean sometimes running through the quality control for tests might take a couple of days. When you test, when you run the test and experiment through different browsers and different operating systems and different. If the CRO has to do that, imagine that will eat up two of his days or her days per week in a month. That is almost 30% of their time. So that's the reason I say the support that the CRO gets is very important. Does the CRO have to do the follow ups and check whether things are getting delivered on time or whatnot? For us at Invest, we have a project manager who handles that. So the CRO is really just focused on CRO work. So let me take that case of the CRO just focused on CRO work. I think somebody who has a couple of years of experience in CRO work probably can comfortably handle three to four clients again. Also, it depends and probably should have mentioned this one, depends on how much traffic and conversions and how many experiments you're going to run for the client. But let's take a typical client where you're running four to five experiments per month. In that case, I think the CRO will be able to Handle three to four companies that they are working with. That means they're able to Design anywhere between 12 to 16 maximum 20 experiments per month. Now, when I say you design an experiment, that means you do the ideation correct. Where you like, you know, digging through the data to say, oh, here's a page that we should be optimizing. Here's the problem. Here's like, you know, the hypothesis, and then work with the designer to create the design for that experiment. An expert CRO, by an expert CRO, somebody who has a ton of experience, somebody who's been doing CRO for a long time, can easily probably handle six to eight clients. Good example is Ayat. And I always tell her, be careful. You're not the standard. I mean, she looks at the page and she come up with so many different ideas, and they're in her mind. She can prioritize them so quickly, it became second nature to her. And if you read the book, blink where somebody looks at something and they're able just through their experience, make that judgment call. They're looking at the statue. You know, experts have ran a whole bunch of tests to say, yeah, the statue is actually authentic. An expert walks through the statue and looks at it and says, oh, this is not authentic. Is this a fake? And they're like, what? We've done all this testing, and they do a little bit more testing and they discover it is actually a fake. Well, so how did that person who's been kind of doing this work for 30 years, how did they. How were they able to make that call? Well, because, you know, it's second nature. They've done so much. There's so much stored in their brains that they're able to make those decisions seamlessly. And that's what happens with an expert CRO, where they're able to run six to eight different companies, think about their tasks, and they're able to think through 24, 30 different experiments a month.

I remember, like, one of the stories that you once said, you once told me, I think you were just, like, sharing it with the team. I'm not sure if you still remember it. You said, like, in the early days of Invesp, I had used to handle about 15 to 18 clients on their own.

Like, yes, that's exactly what I was going to say at some point. And this is back in 2012. So we've been doing conversion optimization for a while. And at that point in time, by the way, there were very few companies doing conversion optimization. I mean, really specializing conversion optimization. And there was so much demand. And I'm like, oh, great. More sales, more revenue. Let's take on more clients. And at some point, I was handling 18 clients. That was stupid, I have to tell you, because at some point, you start making silly mistakes. Correct. And trying to keep up between different. Between different companies. And it's funny, at that point in time, we thought to ourselves, we should hire. So we brought in a designer, Absolutely amazing designer. But we hired a person to help as a CRO, and we didn't know how to hire. That was one of our first hires. And this lady I can mention, M, goes by the first letter M came to us from a company. We were based in Michigan. She was also in Michigan. She was in Ann Arbor. She came to us from a company in Ann Arbor who I knew. And I remember when I hired her, I'm like, hey, how many clients are you handling? She's like, oh, I handle 50 clients. She didn't do conversion optimization, but she was doing Digital Marketing. Wow, 50 clients. That's impressive. Well, if you're managing 50 clients, at best, you might be giving a client two hours per month. Correct. Like, you know, 120 hours. You still need some time to meet with people, talk to his 120hours by between 50 clients. So we take her and we put her on four clients. And for her to go from, oh, I'm doing two hours per client per month into, oh, I need to be doing about 40 hours per client per month. She did not know how to manage that. Completely overwhelmed. Completely thought that we are very stupid for the way we run invest. That was kind of the funny thing. You've just joined us, like, two weeks ago. Just learn our methods before you tell us that we're really stupid. And she struggled, and then. Which was sort of funny because it's one of the first time I had to fire somebody. Turns out that she talked to one of our clients to like, oh, those guys don't know what they're doing. Why don't you come work? Like, I'll work with you directly. The client came to us because they know Khalid and they know Ayat. So the client turned around and said, hey, I don't know this lady that you just brought on, but this is really inappropriate conversation. Next day, I fired her. But it was just kind of an interesting kind of load and time at investment and how we manage things.

Yeah, that's an interesting story. You see, like, every time, like, you share, like, this kind of stories, I always remember them. I don't forget.

Yeah, yeah. It makes I mean, like, you know, living through them. Oh, my God. Maybe, like, one day we should, like, just have a whole episode just talking about hires and, like, the good hires and the bad hires. I mean, the good hires help you build your company. Correct. And they really, like, you know, add a lot of value. Bad hires, man, they make for funny stories, especially a few years later. I mean, sometimes they feel such a bad taste in your mouth, and it's like, God, man, what was I thinking back then when I hired this person? And you. You learn. Correct. I mean, like, you know, nowadays we have such like, you know, a process established for hiring people that, you know, like.

Okay.

It's rare for, like, you know, for us to hire the wrong person. You know, at least we do lots of validation to make sure that we are bringing the right person on board.

Okay, so let's get to the questions that we have today.

Oh, I thought we're almost done with our time, but. Okay,

we still haven't started. Yeah, yeah. So we have, like a bunch of questions today. I think there are about seven questions. So just like, try to run through them as fast as we can. So the first question that we. We got is psychographic profiling, a part of the CRO work that you do.

Psychographic analysis, part of the work that we do. So the analysis that we do is impacted, and when it comes to conversion optimization is impacted by psychographic data. But it's probably a little bit different than the traditional psychographic data because lots of times when traditional marketing companies or marketers look at psychographic data, they try to create Personas and then they think about attitudes, background circumstances, and they create these general trends. Correct. Like general guideline backgrounds for customer profiles. And then they try and use that. They try and use that in their marketing campaigns. Yeah, we tried to do that long time ago whenever we used to create Personas. And we will look at psychographic data, and then we'll utilize that psychographic data and creating the different Personas and campaigns. The problem that we faced is you can spend a lot of time creating Personas for sites, but then when you start optimizing the website to improve conversions, you forget about those Personas to a good extent. Most of the time, our teams were not using them, which sort of defeats the whole purpose. Correct. Because here you've done all this work, but this really a good chunk of it was work for the sake of work. I remember a big company who hired US back in 2014, 2015, and at the start of the project, they had hired this company to Do a whole bunch of psychographic analysis, Persona development, customer journey analysis. And I'm like, oh, this could be interesting. Can you give it to us? And they had spent almost $250,000 on that project with that big consulting company. And it took them about a week to find out the data because no one knew where it was. It was there somewhere, stored somewhere, but they couldn't find it. And the first time they found it, they're like, oh, we just have like a hard copy of this. And I'm like, surely. I mean, we're talking about 2015. Surely somebody sent a PDF, a PowerPoint to you. They're like, yeah, but we don't know where it is. And we have to dig through the data. That told me how much they relied on that data that told me that the $250,000 that they've spent was not very useful. They just took it, spent it, and then they didn't do anything with it. So how do you use psychographic data? Whenever we start a project, we love to do jobs to be done interviews. And when it comes to jobs to be done interviews, this is where you select a smaller set of people. That smaller set, sometimes five, seven people, sometimes 10 people. I don't think we've done more than 10 people unless we are dealing with a large company with lots of offerings. And in it, we're focused on asking the customer, hey, what was going on in your life when you decided to pull the trigger and buy this product? And then we go backwards. So we're not forward moving, but backward looking helps us build the customer journey. And as we do the interviews, we are looking for the circumstances that push somebody to make a decision. We're looking for looking for the attitude and then we're looking for the background. Those are the three things that we analyze. I love to give an example of me being stuck at home during COVID So me and Ayat share a car. We work from home. We don't exactly need more than one car. Just fact of life. Covid happens. We're even more stuck at home. Not a whole lot going on in our life. One day she has something to do to meet her friends and I need to drop the kids off well, so I'm preparing the kids. I go out and the car is not there. But where's the car? What happened? Call her up and she's like, oh, I'm going to meet my friends. I'm like, okay. My mind started thinking, I really need the car. This is unacceptable. You know, I tell the kids Sorry. And then looking at this cycle, and I've been thinking about buying a car for a while, but then this was the trigger event that made me go into active drive, active mode to say, I need to buy a car. So I'm sitting there that night, sitting, Talking to it. I'm like, you know what I've been thinking? She's like, yeah, I need to buy a car. She's like, really? I'm like, yeah, yeah. I've been thinking, and maybe. Maybe I'll buy, like, you know, maybe I'll buy a Camry or something like that. She's like, okay. She didn't say anything. You know, she's like. Because every once in a while I get an idea, and she's just, we'll let it happen. Now, next day, I start thinking to myself, and I'm talking to one of my friends. He's in San Francisco. And I'm like, yeah, I've been thinking about buying a car. You know, like, what do you recommend? He's like, well, like, you know, maybe like, Camry. He's like, come on, you know, Camry. You're like, you know, can't you afford something better? I'm like, of course I can afford something better. Watch this. Like, you know, in my mind. Then I hang up with him, and I'm like, oh, yeah, Camry, no, no, this is not good. I'm the CEO of a company. I need to buy something nicer. And what if I go to a client site and, like, you know, they'll see me? Like, you know, I need to have something really nice. I need to buy Alexis. That's what I need to buy. Now, this is me, myself, just playing game, like, you know, on my own self. So I'm now having conversations with Ant. I'm like, you know, I thought about it some more. I think I need to get me Alexis. She looks. And she's like, what? I'm like, yeah. I mean, like, you know, go to client sites and, like, you know, they see me and, like, you know, it's important. And she's like, how many times you've gone to client sites? Like, you know, if you go to visit clients, they're in a different city, and you rent a car from whatever car rental agency, so it doesn't really matter. In Chicago, we don't have any clients. I'm like, yeah, but she's like, ah, man. And I'm thinking to myself, well, my friends buy these nice cars. So now correct my background, and my friends and my attitude are really shaping that up like, okay, let me start looking at Alexis. And then I'm sticker shocked by the price, but it's okay, maybe I can afford it. And then I looks at me, it's like, well, you know what, we do need to buy an SUV because we have the kids and now we're taking them to games and this and that. And I thought to myself, it's very funny how my decision started with a Camry, went to Alexis, now I went to an suv, impacted by my attitude, by my background, by my circumstances. All of this to say, you do this analysis and then you try and look for trends. What is it that people are saying? How can you use those trends? How do you validate them on a website to look for general guidelines? And more importantly than all of this, and this is where we start with, how can I actually use that data to actually improve marketing, to generate more revenue, to find more insights about my customers? Because trends, just for the sake of trends. Psychographic data. For the sake of psychographic data. It's not going to take you anywhere. It's nice to have, but most marketers don't have the luxury to work on nice to haves. We need to work on things that will impact the bottom line. That's very critical definitely at this point.

So do you like, do the segmentation, like the psychographic segmentation?

So we do some of it, yes. So we're looking for trends. Correct. And we're trying to understand what are the, the trends that we're finding. And the bigger question for us, is there enough data there to say, yes, this is actual insight that we can actually use. So for example, I'm working with a company that sells hearing aids. So hearing aids, that means that you're going after a specific segment. Correct. They're six years or older. It's rather an expensive, an expensive investment. Correct. Like no, 4,000, $5,000. Like my hearing aid is like very expensive. So what segments do you go after? You know, there's kind of the well off and like, you know, what are the circumstances that they are going to choose to use hearing aid and what is the particular messaging that you can use for that different segment? Psychographic segment. So again, define the different segments, psychographic segments and then come up with campaigns, come up with messaging that applies to those different segments. Have people self identify or you categorize them whenever they land on a page and then use the right messaging for the right segments.

Nice. So for these, like the next two questions that I'm going to ask you, I love the topic. But I've never, like, seen many people talk about it a lot. We have, like, a blog on our website about it. It's about machine learning, you know.

Okay.

I've never, like, seen so many people talk about it in the CRO space. I don't know why. Is it because it's complicated or it's not? But. Okay, let me just ask you these two questions, then you can answer them both because I think they are related. Okay, so the first one is, what's the impact of machine learning on price optimization? The second one is, what's the role of machine learning in CRO?

Let's start with the second question, the role of machine learning and CRO. And then we can get into the price. The price, yes, because there's two different things. Let's first define what machine learning is, because lots of times people confuse machine learning with AI. AI is a hot term. Correct. Nowadays. And people use those two terms interchangeably in a funny way, although they are different. So machine learning as a concept, I think was introduced back, I'm tempted to say, in the 60s, but I might be, I might be wrong. And I think it's Professor Tom Mitchell who talked about improve automatically through experience is a crucial point since. Or since, like, you know, we can improve the learning. So as we're learning, new inputs are come. Are coming to the machine and the machine is learning. What does that mean? So basically you. You have an algorithm that is running in a loop. It has specific rules that says, you know, we think if these rules happen, then this particular outcome will happen. Imagine the perfect example for machine learning is the mri. You scan an MRI and you're able to determine whether there's cancer or not. You start by defining specific functions, specific data points to say, if there is data points like, you know, that match 1, 2, 3, 4, 5 things, then there is cancer. You put that through the algorithm. So a human sets that function correct, says the parameters. The machine or the algorithm runs, evaluates an image and says, okay, let me try and look for pattern. And we start with what we call a data set. The data set is like, okay, here's a whole bunch of MRI scans that we know for a fact have cancer. Here's the functions, here's the inputs, analyze and say, yeah, okay, well, if they match criteria number one and 20% and criteria number to a 30% and criteria number three, then there's good, like, no, there's a good chance that there's cancer. So the machine tries to use the function to identify the pattern. After you're done with the learning sets, then you take another set and you say, you run it through the machine and you say, okay, well now you've learned, you've established some patterns. Why don't you go through this test set to determine whether there is cancer or not? The machine runs and then we're able to compare the output of the machine now using the standards that developed to say, yes, there's cancer here, there's cancer there, and then we compare it to the actual data. And then we say, yeah, hey, machine, this is wrong, this is right. So the machine again learns and tries to fine tune the algorithm a lot, a lot more. There's an involvement from an analyst, correct. Who builds the initial model or function to describe the relationship between all the different factors that you can see on an image. In the case of mri, in the case of conversion optimization, the pattern that we're looking for is the certain behavior of visitors and what they see on the website and the ultimate output, for example, and conversion optimization, somebody converting. So we're trying to analyze, you know, we say it might be price, it might be where they landed, it might be the layout of the page, it might be different factors. The machine learns and then we test like, okay, well somebody converted. So now when somebody's landing on a site, do we present them with certain information in certain places and then see whether they convert or not? There's two types of machine learning algorithms. Some, you know, they learn once and then they're done. Some there are constantly learning is what we call reinforcements. Reinforcement process. In the reinforcement process, basically, the algorithm is always looking at the accuracy of its prediction. It looks and says, oh, I predict that in the case of MRI that this MRI actually has cancer or it doesn't have cancer. And then it compares and says, oh, I made a mistake here, let me try and improve. So the machine is learning and improving itself. The use of machine learning in conversion optimization has been limited in digital marketing. Its benefits, machine learning for the most part. Just like I gave you an example, is a classification problem correct based on certain factors and criteria. Do I classify, you know, certain output in a certain, a certain way? For example, let's say I'm going to analyze customer reviews sites, or a product has like a thousand customer reviews. Are they favorable or not? They aren't they favorable? I can use machine learning to learn correct. And give a final judgment for the, for the review. I can analyze social media and how social media is talking about certain topics to determine the sentiment correct towards a specific topic or Specific brand machine learning as a field is used, semi used in a b testing. The problem there is trying to determine based on a set of variables, you know, what are the changes that I'm going to make in a layout of a page or visitor journey across the website to improve conversion optimization. The problem in using machine learning when it comes to conversion optimization is that you need a large data set in order for you to be to effectively use the machine learning algorithm to make a determination and classification that automatically limits the number of sites that can use machine learning. I'll give you another example from machine learning and its use in digital marketing. And this is actually a software piece of software that we've worked with a company that we worked with and absolutely amazing. So they only work with apparel companies. You come to a website as female apparel. You know, you look at a particular item, below the item, they show you some influencers who might have worn that item in different styles. You're looking at the shirt. Well, you can match it maybe with like, you know, pants, or you can match it with a small skirt, or you can, whatever it might be, you're clicking, you go to Instagram and you come back, they can look at which influencers you clicked on over three or four pages. And they can actually classify you to determine what kind of clothing you like based on a couple of clicks. And based on that, now that I know which clothing you like, guess what, the category pages and how they are displayed to you look a lot different than how they're displaying to somebody else. And all of a sudden you're thinking to yourself, I'm like, oh, wow, this site, really? I love their items. Well, yeah, you love their items. Except that the fact that they understood your shopping preferences and your style, your dressings, your dress style, and now they're showing you items that appeal to you using machine learning, which is very powerful. I think eventually there will be space for machine learning conversion optimization. It's not heavily used yet. There is distrust between conversion optimizers and kind of like machine learning software because we think that, well, I still need to use my brain to do the analysis. I think there's place for machine learning conversion optimization. It's not going to replace conversion optimizers. It can help them make their jobs, make their performance better. So eventually we'll get to that point. Not yet, but we'll get to that point. Price optimization and using machine learning and price optimization is an area where machine learning is very powerful. Setting the right price for a product or even a service for that matter. Is challenging. Correct. And actually there's so much literature about it when it comes to economic theory. Ultimately the goal of a company is to maximize its profits from any unit that it sells. Now a company, let's take an E commerce company, we think to ourselves as we are pricing an item, we're thinking, well, what price we should set the item, what price we should sell this item for if we want to sell certain number of units in a week and what is the right price for that item Given all the different circumstances that play into pricing an item. There's the economic situation. Correct. There's the competitors. There is, there is a supply chain issues. An item could be priced at $100 today because it's available. You can buy it from other competitors. All of a sudden there's supply chain issues. And Webcam goes from $100 to $250 because just there's issues and there's so much demand for it. Machine learning can look for the impact of all these different factors to help us understand how we actually optimize the price, help us understand how customers would react to different pricing points. Should we be pricing like, you know, this webcam, $250 or $180 or $100. Correct. Giving all the different circumstances that are going on in the economy at some point and given how customers demand this particular product. The quality of a machine learning output depends on the inputs. Correct. We talked about the function and analysts actually programming that function, looking at all the different inputs to make a determination about the right price. When it comes to looking at an MRI and determining whether it has cancer or not, while we have like kind of the medical field and like the image, all the factors are there. When it comes to price optimization and using machine learning for price optimization, there are things that are really difficult to manage. How is the economy doing? How are my competitors doing? What, how is the weather is doing? What are some of the events and holidays that are coming up? What are the supply chain issues? There are many factors that are sometimes difficult to come up with a formula for. And companies who are successful, they're able to come up with an initial price for an item, a best price, a promotional price. There's different prices that you can, you can come up with using machine learning. Now there's other pricing strategies that are out there. There's dynamic pricing. There is sometimes like no, like no set pricing in terms of based on profit, dynamic pricing. You're always looking at your competitors, you're looking at, looking at their pricing and you're trying to match which is by the way what we see with comes to airline tickets most of the time. Like you know, pricing goes from one company, it goes from like, goes up from another company. That's a lot different than machine machine learning. The application of machine learning and pricing can be very interesting. It's challenging, not easy. The cycle analysis around it is very fascinating. It can help retailers maximize. But the software that does that in a good way is too expensive for the small retailers. And that's the other problem that you have with using machine learning when it comes to pricing. Software that does this really well is too expensive for the smaller guys. Lots of the software that's available for the smaller retailers claims to use machine learning, but it really doesn't. Just to give you an idea about the cost of machine learning, remember the company that I mentioned using machine learning to determine the, to determine the style. Each run to do an analysis on a data set cost them $5,000. Each run sometimes cost upward of $25,000. Imagine how expensive that is. Not every retailer can afford. And this is just the cost for the software company. Correct. They still need to pass that and they still need to make profit. That just tells you how expensive the software can get.

So the fact that like maybe the topic of machine learning isn't talked about a lot in conversion optimization, is it because of maybe most of the clients or maybe E commerce companies don't can't afford it or maybe it's something that is still like growing?

No, I mean I think, I think the larger players can afford it and they probably have the teams to do something like that. Whether they do it or not. That's a whole other question. Sometimes some of the big players, by the way, I'm always amazed. Some of the big players, they've been doing things for a certain way for a long time that they haven't developed and evolved. And it's interesting too because sometimes I work with two companies that are competing against each other and one of them is very rudimentary approach to marketing and data analysis. And the other company, same space, different markets, a lot more advanced data sciences team that is very advanced in their, in their analysis. I think using machine learning for retailers is still in its infancy. And it's funny to mention that after E commerce being around for almost what, 22 years or even more, 25 years. Years close to that. But it's still new. And I think there will be a lot more use of machine learning in the future. You'll see it's more and more as the costs of running machine learning algorithms drop, as the cost of AI drop. Think about this. The use of AI, which is completely a different topic than machine learning. But using AI to generate content is still sentencing. It's actually right now laughable the content that he gets. But I think it will develop. I think AI generated content 10 years from now is going to look a lot different than what we are getting nowadays. I think the demand for it is going to increase. It becomes more sophisticated as the, as the output becomes more, more interesting.

Okay, so the next question is, what's the cost of hiring a CRO agents? I think maybe the person who asked this question may be like a potential client. I'm not sure.

Oh man. What's the cost of hiring a CRO agency? You can hire a CRO agency for as little as $2,000 a month and you can hire a CRO agency for as much as 30 and $40,000 a month. And I can assure you one thing, what you get out of a $2,000 company is a lot different than like, you know, $30,000 company per month. Correct. Two completely different animals. It depends on what you are looking for. And let's talk about looking at different levels of hiring as conversion optimization or experimentation company. There are some companies that will come in and do turnkey solution. Correct. I'm going to do the ideation, I'm going to figure out the problems, I'm going to do the ideation to figure out solutions to those problems. I'm going to create the design, I'm going to handle the implementation, the quality control, I'm going to roll out the experiments, I'm going to monitor the results and I'm going to make a recommendation. That is a lot of work. And it depends of course on how many experiments I'm running in a month. If I'm running a single experiment per month, it's a lot different than if I'm running four or five experiments per month. The smaller agencies usually will charge, I would say, anywhere between 4 to $5,000 a month. Successful agencies probably like you know, spending 9, 10,000, $11,000 a month and can go up from there. Agencies, generally digital marketing agencies, play this game. We all know it's, but we don't talk about it where we don't like to share the number of hours that we put into a project. Correct. The idea there is, oh, let me try and do as little work as possible. Hopefully produce amazing results and hopefully charge my clients a lot of money. What does that mean? Well, maybe I can charge my clients $7,000 a month. I do 10 hours worth of work and just hope that those 10 hours worth of work are going to produce some massive amazing results. The client is happy and my hourly rate is $700 an hour, and everybody's happy. That works in some strategic campaigns and some strategic work. Most of the digital marketing work requires a lot of effort. Think about this symbol. How long does it take you to produce a piece of content? Throw a number at me. Like a good piece of content that you're proud of.

45 days.

So you're talking about like, you know, 30 to 40 hours, correct? Yeah, easily. Between interviews, between edits, between, like, you know, producing. If you're able. Let's say somebody comes and hires you and you say, I'm going to write you four pieces of content in a month and I want to charge you. Let's throw a number. I'm going to charge $10,000, my friend here for the four pieces of content. It sounds good. Now, if you're doing. Take this scenario. If you are doing 100 hours to produce those effectively, you know, your hourly rate is $100 an hour. Great. If you're able to produce the same pieces of content in 10 hours. Correct. Your hourly rate now is a thousand dollars an hour. There's a huge difference between the two. Can you actually produce amazing pieces of content? 4amazing pieces of content in 10 hours? Is that even doable? What do you think?

In 10 hours?

In 10 hours for articles? They're really good.

No, I don't think so.

There you go. And that's what I think the trick when it comes to conversion optimization. Conversion optimization is labor intensive. There's a lot of work that goes into it. There's the analysis, the ideation, the design, the implementation, the quality control. I can't imagine somebody doing a conversion optimization project in less than 40 or 50 hours a month. With most of the companies that we work with, we are doing probably anywhere between 70 to 100 hours. And that's kind of on the low end of things. Sometimes we're doing 200 hours a month. So you want to understand what the hourly rate is and you want to understand what kind of output you are getting. It's important to see two things. One is, okay, the effort that's getting put and the output that's getting put. And then you also want to look at the results. So two companies Both can charge $1,000 a month, but the output that you're getting is completely different. Correct. One company is producing four or five Experiments a month. Another company is producing one experiment per month. And even comparing experiments by the way to experiments is not comparing apples to apples because one experiments could be a lot more complex than the four experiments. So you want to kind of drill a bit deeper to understand what you are signing up for, what you are getting, what's the final output, what's the track record, what's the history. There's just so much there that you want to think about. When you're hiring a CRO agency. Yeah.

Talking about like hiring a SALO agency. I have like this other question that I just thought about right now. I know that at Invest, most of our clients or the clients that we prefer working with, they sign like a contract for maybe one year, like maybe going up. So suppose like a client wants to sign a contract, but it's a short term contract, let's just say three months. What kind of results do you think you can produce in a space of three months? Do you think like it's enough, enough time on it?

We do produce results within three months. I know that we've done enough testing. When you work with Invest, we're like, you know, a fast machine. I mean usually by the second month you're up and running and like, you know, the testing is there and you're seeing the results. And I also, every company that we sign with, after three months, there's a call that I'm going to give you to talk about the results. But I'm talking to clients and companies I will work with a lot earlier than that. If somebody comes and says, hey, I only work with you for three months, I tell them, okay, well there's plenty of other conversion optimization companies out there that will work with you for three months. Have at it. Are they going to produce the same results? Some of them might produce, might produce the same results. It's rare. There's probably like three or four other agencies worldwide that can match the quality of work that we do and the results that we do. We do ask for a one year commitment. Now lots of times whenever companies talk to us, they're like, oh, one year, why are you trying to tie us up? I tell them in all honesty, a one year commitment is not very interesting to me. Two years, three years commitment, you know, relationship. That's what I'm looking for. I'm looking for somebody who's going to stick around with us for a long time. They say, well, what if it's not working? Very normal question to ask. And I love for them to ask that. I tell Them, okay, first off, if it's not working, it's not like I'm going to disappear. So we're talking constantly. So you're always emailing me. You have my phone number, you call me. I'm the president of the company, I'm the CEO of the company. So we're going to be talking. We're not going to come to like, you know, three months into the project and you're like, oh man, I'm discovering that's not working. No, we're going to have that conversation. Second thing, we want to understand why it's not working. Is it not working because we're not following our process or is it not working because of something else? It's very important to understand why the process not working. And I tell them I've done this long enough. 24,000 AB tests and counting. I am surprised if our process doesn't work. Every once in a while I'm surprised, but it's rare for me to be surprised. And that is a very honest conversation. My goal is not to rob somebody. The worst thing that you could do somebody is to keep them in a relationship because the contract says you have to stay in a relationship, you have to deliver results, and they have to be happy with the relationship. So I tell them, hey, friendly relationship. If something happens and we're on the same page, you can probably walk away. During COVID we've had two clients who have had annual contracts with us. One of them had sponsorship from Nike and Adidas, and overnight their sponsorship got canceled. And one of the partners called me and said, khalid, we can't do this. I'm like, it's fine. Don't worry about it. It was as simple as that. I'm like, don't worry about it. We're looking at, we'll get other business. It's. He knew that he doesn't have to think a lot. And I knew I don't was like, what's the point? Have him pay us and then have him go bankrupt because he has to pay us. It's just stupid, stupid way of doing, of doing business. Now. At the same time, what I'm trying to avoid is companies that change their mind. Because I've also dealt with CEOs who today's trend is conversion optimization to Morris Trend is a SEO. The next day's next week's trend is ppc. I don't want to deal with a company like that. We're not a good fit for them. So I'm very clear with them that first, if you're hiring us because that's the latest and greatest and you might change your mind, don't hire us. There's plenty of other companies who would love to take your money if you're hiring us, but you're not going to. Let's do our own thing. Don't hire us. There's probably better, easier companies to deal with than investors because when you hire us, we will do things our way. Hold us responsible for the results, but let us do our thing. So whenever you're starting an engagement, pre engagement, there is this dance, this buyer seller dance. And I'm very clear within a company that talks, I tell them I'm not interested in the buyer seller dance. Let's just have a very honest and transparent conversation. And I will tell you if we're not a good fit, actually during our sales process, I tell them, here's our deal breakers. I'm very clear. If you're not comfortable with any of those, walk away. There's plenty of CRO agencies out there who would love to take your money and you might find us a good fit with you. We might not be the good fit for everybody. I know we're not the good fit for everybody.

Yeah, that's a good question. That's a good answer. I know when we look at the ship process, the CRO process that we have at invesp, the first stage is to scrutinize. That's where we do a lot of research. I've had like, I know that, like, that's where like we spend much most of our time, like doing whenever, like doing convention optimization. Right. Suppose you've done the research, you uncover 100, 250 conversion issues. How do you prioritize that? What I'm trying to ask is what. Which metrics do you use to score conversion issues during the prioritization stage?

So there's different prioritization frameworks out there that take all the items that you've identified on a website and help you prioritize those items. There is the ICE method, which is very popular. It's very simple. It looks at three different values to prioritize. There is a method that's developed by CXL which looks at eight different metrics. To me, when we evaluated those methods and we've used them, we've. The question was if I take those hundred issues that I see on a website, I give it to a conversion optimization specialist and then I give take the same items and I give it to a second CRO and a third CRO, are they going to give me the same prioritization or each using the same methods to prioritize, Am I going to end up with different prioritization order with eyes? The problem that I've had is once CRO comes up with a list, I compared to the CRO from the list from the other CRO and they don't even match. Maybe like, you know, like the match, the area where they match is about 20%. Like, you know, everything else is all over the place. So there's lots of subjectivity and the in the analysis and then how you prioritize an item, the more subjectivity you introduce into a prioritization method, the less accurate it is. And then you're really depending on the person who's analyzing the CRO and how good they are and how they understand the business. So we've developed a method at inves where we use 18 different metrics, we score each one of those problems that we've identified against 18 different metrics that help us really understand the priority of an item compared also to the rest of the item. Within the set that we're looking at. There's I think two items that are subjective. The other 16 are very, very objective items. The two items that are subjective is what kind of impact do you think this item will have is high impact in terms of increasing conversion rates, medium or low? So that's subjective. One CRO might say, oh, I think this is really medium. And the other CRO says it's high, another CRO says it's low. Expert CROs most likely will agree, but if you have different CROs at different knowledge level, they might disagree. And then the other item that is subjective to some extent is what is the level of effort? That's what we'll take to implement this item. Now there's two different types of levels of effort. There is the actual design and implementation coding of an A B test and then there's the political level of effort. When it comes to a test, we can estimate to a good extent how long it's going to take us to implement an item. The political level of effort, that's a little bit harder sometimes that's where the subjectivity comes. There's other factors that we evaluate. So when we discover a problem, you know, what does it impact? Is impacting trust? Is it impacting fuds? Is it impacting scarcity? Is it impacting incentives? Where do we, for example, did we identify this item in a high value page, In a low value page? Is it on top of the page? Is it middle of the page. How did we identify the issue? Did come from an expert review. Did come from an analytics analysis. Analytics research or data research that come from customer interviews that come through multiple efforts. So all these different things we put together to score an item. Now if you take 100 items, you score them, then this item gets a score of like, you know, the maximum score an item can get is 100. And the higher score it gets, the higher priority it gets. So now I can look at all my items and sort them and say, okay, well, here's the items. But that's not enough, by the way. There's other things that you need to do. One CRO run through the item, scores them 18, like, you know, 18 different metrics against each item. Another CRO takes the same items and a third CRO. So we have multiple people scoring, and then they have to agree on a final list, which is complicated. But again, that's the reason conversion optimization is very labor intensive. It takes a lot of thinking. Then it's not enough. Now I have a final list. Then you start grouping some of those items to say, well, here's the items that are around value proposition, maybe on the homepage. Let's group them together. That helps you create what we call initial hypothesis on how to fix an item. There's a lot further analysis, by the way, when you're developing a test. Oh, I need to. As an example, my hypothesis is emphasizing value proposition on the homepage. Well, I can do that in so many different ways. So a hypothesis can have multiple solutions. And now you prioritize the solutions and you come up with what we call more complex hypothesis. So the prioritization process, then the scrutinized phase is very critical, and it takes a lot of effort. And the quality of your testing program and your experimentation program is dependent on the quality of your scrutinized phase.

Okay, so I have, like the last question for the day. Go for it. Yeah. So when optimizing the price of your client's products, do you go with the Bayesian perspective or the frequentist model?

Oh, man. So now we're talking about price testing. Yeah, there is. We don't do a whole lot of price testing. Just some thoughts around price testing. There is very little a B testing software that does price testing and statistical analysis in a meaningful way, whether it's using frequentist or fusing or fusing Bayesian bias on statistics. Most of the testing engines might reports for different variations to the RPV revenue per visit, or they might report the aov. But determining A winner is not based on prize, not based on the RPV or the aov, or based on this variation generated this revenue for me, and this variation generated this revenue for me. Determining the winner using Frequentious or Bayesian usually is looking at conversion rates. Correct. Whether somebody converted or not. What's the conversion rate for this variation? We usually rely on conversion metric. A conversion metric determine a winner. We look at the revenue metric, but it's a secondary metric for us. The only time we're very concerned about the revenue metric, if the variation and the control have at least 20% difference between them at that point in time, we're like, huh, I wonder why this happened? But that's not enough either, because at that point in time, we want to look at the actual raw data for the A B test, remove what we call extreme orders. You're running a test, a control versus a variation, and somebody the average order value is 100, but somebody decided to come to the control and place an order for $100,000. Guess what? The AOV for your control just shot up and you look like your variation just lost horrendously. Well, no, it's a single order. It's an extreme order. That's the reason you see that huge variation in the average order value or the revenue per visits. That's the reason you have to be careful. You have to be able to download the data, remove those extreme orders on both sides, above the average value, below the average order value, and then do your analysis. So is it based on a frequentist? Not really. It's more a secondary, secondary metric that we use to help us make the decision. But it's not the primary metric. Yep.

So those are the questions that I had for you today.

Awesome. Well, Simba, those. Those were some tough questions. By the way, we had lots of people attend, but machine learning is not an easy topic. And I think we've lost quite a few people when it comes to machine learning. As I see people were attending but then slowly dropped off. So anyways, if you have any questions, myself and Simba are always available. And until next time, thank you everyone.

Bye.