#4 – Split Testing Limitations, Inconclusive A/B Testing Results, Copying Competitors, Predictive Analytics In E-Commerce

My name is Khalil Slade and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Stack Simba will be bringing all the questions.

Oh man, I bring tough questions like do a B testing results fade over time? How do you go from low to high testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live hour podcast today wherever you listen to your podcasts. How's your week, Simba?

My week has been good. We've been focusing on working on our, on our okrs for next quarter. So yeah, I think like, that's where like the major focus for me has been with the team. So we are making progress. We still haven't like, we know like our goals, we know like the objectives and everything that we are aiming for. So we just have to put it into action and see if like the, the plans that we do have workout. How about you? How was your week, huh?

Crazy busy. Good, busy. But at the same time, like, you know, I mean, it's funny because there are weeks where I'm just not busy and I get very antsy. I'm like, oh, what's going on? What's going on? What's going on? But actually it's been good because on the investment side, things are stable. You know, we've added enough resources. The POD system that we've introduced is working, is working. Not as well as I'd like it to work, but, you know, sort of expected. It's funny because. And then we can get back to the pod system in a second. I don't want to like no branch. So that was interesting. Project management is stable. Fig PI. Like, you know, I've just spent some time with, with our team, like, you know, last couple of days reviewing all the progress that we're making. So we're making good progress, but I tend sometimes to maybe expect a little bit more. So I'M like, I'm looking at some areas and I'm like, oh, we can improve here. But at the same time, I mean, you want to push people, but you don't want to push too much, and you want to know the right balance. So it's been. It's been an interesting week. Let's leave it at that.

Yeah, that's also like. Reminds me of a book that I've been reading. Like a book about Netflix. The one Reed Hastings. I think the guy's called Reed Hastings. I'm not sure.

Yes.

Yeah, the book called no Rules. No Rules. I think it's one of the good books when you want to learn more about team project management and how, like, Netflix has been working, like, internally. So I've been like, drawing some lessons from. From how they've been doing things. And one of the things that they say is they don't portray, like, their employees as, like, family. You know, like most companies, they usually say, oh, we are family. You know?

Yeah.

What does that entail? You know, family. It means, like, even if someone performs bad, you can't fire him because he's family. You know, families stick together even when things are bad. So they were, like, advising that you should, like, isn't that portray yourself as a team? You know, because a coach has to make sure that each and every position is the best player in that, and each and every player playing in that position has to know that if I don't perform very well, I will be substituted. So I think that's also, like, a good way to also think about it. It might sound harsh, I get it, but that's how it is. I'm not sure.

Yeah, no, it is the reality, by the way. You know, I mean, sort of like finding that right balance because, you know, the rule, you know, higher, slow fire, fast. I don't do that. I know I break that rule, by the way. And it's funny because I think. I think every time I've decided not to let go of somebody, it hurt us. Because I know when I have to let go of somebody, but I wait and I use it. It's funny because over the years I would have this conversation with a person, and this is the exact conversation, hey, so and so, you're not performing well. I've had. This is what I would tell the person. I've had this conversation with endless number of people who did not perform well. I usually tell them, you need to fix everything within two to three weeks. You know, maybe within a month. Usually for the first two, three Days after I tell them to fix their behavior or to do better, they do better buns. Usually within a week, they fall back into old habits and I have to come and fire them. And I would tell the person, please be the exception. Now I can tell you over the years, I've let go of many people because they were not a good match. Sometimes we're not a good match for them, sometimes they're not a good match for us. Of all the different people, I'll let go of one person, you know, was an exception. After I had that conversation, he picked up, he fixed things. I'm like, oh, okay, so there's always an exception. But guess what? He fixed his behavior for about six months and then we're going back into the old behavior. So you probably heard me say this, by the way, in, in the morning, where I'm like, I have a simple rule. I'm like, if I have to discuss an employee multiple times with Ayat, with you, with. I'm like, okay, they're not performing well. So because usually when I'm discussing an employee, it's not because of like, you know, them not understanding is usually they're not a good fit for the job. So it's a bit harsh, but it's reality. Correct. Because you keep the person well. No, I'd rather bring somebody who's like, you know, maybe more skilled, more dedicated. And this way everybody wins. Otherwise you keep the wrong person. And guess what it brings down, it brings down the company. And we see that, by the way, more on client side. Because somebody who's not performing well on client side, clients are not happy. Okay, so yeah, and I'll finish with this. The level of expectations between an American client and what somebody sometimes is working in outside the US is tremendously different. In the US there's high expectation and delivery, especially with the type of clients that we work with. So it makes for interesting discussions.

Yeah, you are so right. I think again, one thing, like when you start like working at invesp, I'm speaking this like from experience, you know, like at first, if you're starting out, especially for me, I didn't know like what conversion optimization was, you know. So at first, I don't wanna lie, I was kind of like struggling, you know, and every time I would write like articles and then I submit them to, for review to you, to either you or I had. And then like, I'll be just like maybe working on something different. Then I'll just see like those pop up notifications on slick about like the Comments that you are making. Yeah, so it was so scary, you know, because I would have done like, my level best. You know, for me, that was like my level best because I didn't know much about in terms of details. So I'll say, like, maybe just in one article, just to maybe an article of about 2,000 words, I would have like more than 20 comments. And that was so scary for me, you know, but with time, I think it also takes, like a lot of dedication from the employee side and trying to learn and improve yourself.

Yeah, yeah, it's interesting. And here, here's the other funny part. Now reflecting back and like, you know, I remember the first few articles that you wrote were like 5,000 words. You know, guess what? You're still. I mean, to me, those initial articles that you. That anybody writes, unless they're an expert in conversion optimization, they're just trying to understand and to put comments on a 5,000 or 6,000 words. I mean, the review process is painful. The writing process is painful. Fixing and editing is painful. So I remember telling you one time, like, simple, no articles more than 2,000 words because we're. We're dying over here. So. Yeah, and it's just one of those things, I mean, you've gone through it yourself, where you really have to learn the feel. I mean, you have to. In order for you to succeed in what we do. Like, as a, as a content writer, you have to have really good writing. Correct. And that's the first. And that's what most people would bring to the job. But then you have to be willing to learn conversion optimization. And conversion optimization is not always easy to learn. So it takes, it takes a little bit of effort to be. That's so true.

But that also, like, brings an interesting conversation about what can you outsource and what, what you shouldn't outsource. Do you think, like, it's a good idea to like, maybe hire a freelance writer to come in and write about conversion optimization? Seems like we all know that it's a tricky. It's a tricky subject because my fear is like, maybe most of the people might end up copying what the next article is saying. So what's your take on that?

It really depends on the person. You can hire a good content writer to write about conversion position. Let me tell you my experience. If anybody hired content writers to write about conversion optimization, it's me. Sometimes it was one of the worst decisions I've made. Sometimes one of the good decisions, better decisions that I've made, you get for the most part what you pay for. I remember hiring somebody, and I was just impressed with their writing. And I'm like, oh, like, you know, there is so much information. And this is probably from about 10 years ago. Didn't really think for a second to check whether they are copying content from other places or not. Completely my fault, by the way. All I know is I was too busy. We have a lot going on. And here's this person. His writing is really good, came recommended, and I'm like, oh, well, just start producing articles. She starts writing articles. And I was writing for a publication, famous publication, unfortunately. And then they sent me an email. They're like, hey, you're plagiarizing some content. And I'm like, what? I didn't even think about that. And for me, that was absolutely horrible. I'm like, okay, you can hire many. Like, you know, in all honesty, many busy people end up hiring ghostwriter. And I'm like, oh, he understands conversion optimization. So then I go and I open all the articles, and he had written 15 articles for me at that point. I go back and I start searching, and they are so plagiarized. And I'm like, what the heck was I thinking? Why didn't I check whether it was plagiarized or not? So I just. And I'm like, hey, dude, what's going on? What have you done? You're copying this. You're looking for other. And guess what happened? What? He tells me he was not writing the articles himself. Guess what he was doing.

He was also getting.

Horrible. It was this chain, and everybody's plagiarizing, and what I end up doing. Some of those articles were published on our blog. Some of them were published in other. I literally had to go to every place, and I'm like, can you please take this down? Take this down. Take this down. Because it was just an embarrassment. Wrote an apology to the publication. Wrote multiple apologies. And can you apologize? Like, you know, it's like, there's no good explanation for this. I've hired agencies. So that was one. One experience. I've hired agencies who specialize in writing content. And some of them are very successful agencies. And I thought to myself, I'm like, okay, I've tried the ghostwriter, and that wasn't very successful. Let's hire an agency. And one agency was very expensive. They will charge almost a thousand dollars per article. And I'm like, oh, my God, I would die. And this is back when we were just starting out, and then they come back with their articles, and I read them and they were so what's the word I'm looking for? Rudimentary. So basic that I'm like, okay, this is not conversion optimization. And the information that you're writing in this is. It's not something that I'm going to put my name or even bother to publish it on our blog. So they take the article, they do a second round of rewrites, and it was just like, you know, it's like, okay, you're not going anywhere. By the fourth rewrites, it was obvious to me and it was obvious to the agency owner. That's what I'm expecting. And what they can deliver are completely two different things. So that was an absolute, absolute failure. Try to. We also try to hire writers who come really highly recommended, but again, the field is so highly specialized that they struggled. By the time they get feedback from me, they get feedback from Aya, they're like, okay, this process, you know, is just way too long. And I think it's also demoralizing. Some of those guys who come and think that they know what they're doing and then it's like you slap them left and right and it's like this count and that comment and it's like, you know, after a while, they're like, this one lady who basically left us such a horrible review, she's like, oh, I can't. Like, you know those guys and like, you know what their level of expectation. I'm like, hey, you come setting yourself up as an expert. Like, unfortunately, maybe the level of expertise. And I think, by the way, that's the challenge. People who are really good in conversion optimization, guess what, they make their money. And by doing conversion optimization, they don't make them writing articles about conversion optimization. There is one exception. There's a couple of fractures that we hire who come from a CRO background, but they're not interested in doing CRO. So they've worked for on the agent side, they worked on the looking on the platform side. So they were with the AB testing platform. And those guys are actually good. They write really well, they're in high demand. But those are rare. For every hundred writers you're going to outsource the writing to, one will be at that level and the other 99 are just not worth the money. Which brings, by the way, a last point. Like, you know, this is my fourth last point.

Yeah, sure.

There's now there's this AI correct. I mean, there's plenty of AI tools that write articles. And it's fascinating to me because, like, you Know, we subscribed to a couple of them just to test them out. And you have all these freelancers who now use the AI tools to generate articles and then they turn around and sell them. And some of those Facebook groups that there's one particular tool, and I'm not going to mention it's because. But I joined their Facebook group and I see like all these like, oh man, I can write 30 articles a day using this AI. And I look at those articles and I'm like, okay, I'm sure there's a client somewhere paying you for those 30 articles. Would I even touch those articles? No, but that's, that's an industry. And the articles are just useless articles. And I think Google, by the way, came out and said, hey, AI generated articles are just not good. That's a whole other topic that we can talk about, but it's just not there. So you need somebody who's really good, who knows conversion optimization, who knows digital marketing and who's willing to not do it. Not to do conversion optimization, digital marketing, but actually write. That's how you can hire somebody who's good. Yeah.

Okay, so let's jump into today's questions.

Yeah, we spent like, how much did we spend talking about? Like, you know, almost 15 minutes. But I'm all yours, my friend.

Yeah, sure. So we have like a couple of questions to answer today, and I think five questions, but they are difficult questions. I can see.

Go for it. Let's see. Okay, sure.

So the first one is, what are the limitations of split testing?

What are the limitations of split testing? See, I have to move my hand to think about, about that one. Let me start by saying that split testing is not to solve our bullets. It's not going to be the one thing that you do that is going to save your business and is going to make things just magically, really just go well in your business. As a matter of fact, every time a company reaches out to us and they are thinking to themselves, hey, this is the one thing that's going to save us, I usually avoid that project. I tell them, you know what, we're probably not a good fit because conversion optimization and experimentation, split testing as part of that is an ingredient, correct. In a full strategy to help you succeed. It's not going to be the one thing that's going to make you succeed. And having said that, we know that there that that split testing experimentation comes with limits. And let's talk about some of those first. If you have a site with limited traffic, limited conversions, then effectively you're not going to be able to run as many experiments as you should or would like to. Experiments take a lot longer. To conclude results are really slow. And that's a limitation. Just due to the nature of the beast, things are just not going to move as fast. And I think moving fast and having high velocity and experimentation and split testing is going to help you succeed and see better, better results. So that's an issue that many companies struggle with. Now that doesn't mean that you cannot do conversion optimization. Yes, you can, but you can definitely. But you cannot do split testing. Or if you do it, you have to be very strategic on how you do it. Correct. So we've talked previously about doing micro testing as opposed to macro testing. So macro testing, I'm testing two different versions of the page. Let's say I'm an E commerce website. I'm testing two different versions of the page to see which page, which version increases my sales. Micro testing, I'm testing two versions of the page, but I want to see which version increases the clicks on the ad to Cart, for example, people say, oh well, that's good enough. Well, a click on the add to cart doesn't mean an order. And eventually you discover, you discover that. So that's one limitation. Another limitation is the statistical analysis that needs to go into conducting split testing correctly. You don't have to have a master's degree in statistics, but at the same time you really need to understand at least basic statistics statistics to do correct split testing and AB testing and experimentation. Otherwise you will do stupid things. And stupid things have a way of polluting your data and your results and you think you have a winner and you don't have a winner. And some of this stuff requires really like it's a basic understanding of statistics so you understand that what confidence level, what the confidence level is, what's on going significance is how do I collect the sample size? When do I stop my test? Can I stop the test early? Should I continue running the test? If I look, you know, if I determine if I reached a certain sample size, a certain timeline, understanding all of that and being able to really look at the data in an intelligent way can be a limitation. And maybe this is not the limitation of split testing itself, but it's a limitation of the people who are going to be running split testing. Can you separate the two? Most likely not. Another limitation of split testing is split testing is a tool and the goal from the tool. We've talked about this many times is two things. First is you want to increase your revenue. You want to increase. You want to make more money. Correct. That's, that's the reason people do marketing. Lots of times people say, well, we're learning. And I'm like, yeah, learning is good, but guess what? The whole organization you work for is there to make the stockholders, the owners, make more money. Correct. And marketing is just another activity that we're doing. A limitation of split testing is sometimes people focus on the results I love, but they forget about the question of what did we learn? What are the insights from the experimentation, the split testing that, that we ran. If you recall earlier, we're talking about as methodology that we follow whenever we present results. And I've said, okay, there are certain things that you have to cover in every split test. What are the data points that told you you need to run the experiments? What is the hypothesis? What do the numbers tell you? Correct. And then what are the insights? The insights? People skip the insights because sometimes the insights are really difficult. And guess what? The insights are even more difficult to come by when you have a losing experiment, when you have a winning experiment, it's obvious. Oh, so here's this. Like, we've done this hypothesis and people actually responded to it. But it's a lot harder when we had a hypothesis, we've generated designs and we don't have a winner. What happened over here? So a limitation is on the learnings. And sometimes this happens because the analyst or the CR working on the project doesn't have the experience to decipher the information. Sometimes it happens because they're testing way too many things. And then they wonder to themselves like, oh gosh, what happened over here? That's the reason the experiment design from the get go matters a lot. And then a final limitation is split testing is laser sharp focused on a specific area on a specific page that you are testing. And sometimes you can lose track of the big picture. What are we trying to accomplish over here? An experiment can increase your conversion rates, but at the same time, it can hurt the lifetime value of your visitors or customers. Is that really what you want? An experiment can take you in a certain direction, but the business strategically might be going in a different direction. So you need to be aware as somebody who's running split testing is like, okay, what are we trying to accomplish over here? What are the KPIs that we're tracking to this quarter? Does my experimentation support those KPIs and does it help us achieve those KPIs? Those are all different things, different limitations that you have when it comes to split testing.

Good one. I thought when I saw like this question for the first time, I thought I'll have like two or three follow up questions. But you answered it very well.

Intentional, intentionally, it's funny. So I'll give you an example which is fascinating to me about experiments. So we have this one client, large company, it's a marketplace and we were looking, it's an E commerce marketplace. We're looking at their product pages and when people clicked on the add to cart, there was nothing on the site that told people that the item was added. Session recording shows that there's a little bit of confusion. People clicked and you can see a certain behavior where people trying to click on the cart and then they come back to the page because they're trying at least you're saying based on this behavior that you see consistently that people are not sure whether the items were added to the cart or not. Usability testing also told us that people struggle with that. We're like, oh, this is an easy win. Hypothesis was if we show our visitors a prompt after they add an item to the cart, that's going to really remove that uncertainty and it's going to help us increase conversions. And I think that we estimated 10%. We introduced four designs. One of them, you can hardly even notice, there was a little bit of animation showing that the item was added on the button. One of them also showed a small, small layover pop up that shows that basically the item was added. One of them was showing a mini cart. And you see those now in E commerce where you show the item and there's another design that showed a confirmation. We started with a really good hypothesis, the design. Some of them are very drastic. Correct. Adding a mini cart like that slides one of them like, you know, even the pop up, some of them are so slight. We run the test and all variations lose. Not one of them, not two of them. I mean even the variation where there's a slight change on the page that is difficult sometimes to notice. Lost and lost by about 5 to 6%. And we're like, this is, this is interesting. Why did this happen? Why did this lose? And that's by the way when I get dragged into, into meetings like hey, we didn't expect that. And I look at it and of course the first thing that I do in this case I'm like, oh, validate that there's no implementation issues, that there is no software issues that we're running into. So we validate all of that. We're like, okay, well, we validated. That makes sense. And then what the data is telling us, at least in this initial stage, is that people trust the brand people. Yes, there is a small percentage that was uncertain about this. Like you know, was the item added to the cart or not? But for majority of people that was not an issue. They were not concerned about that. Now this is our way of deciphering the information. Is it correct? We don't know because guess what, we're going to be running other experiments, but keeping that insight in mind to say, okay, we need to validate whether this is a correct insight or not. And after we validate it, probably after multiple experiments, it becomes a principle on how we run experiments and a B test testing with, with that site. So is it easy? No, it's not easy. Is it fun? Yes, it is fun and it's challenging by the way, because all you can do in these cases is make an assertion. And the beauty of an assertion is it could be false or wrong and could be false or right. Correct, true or false. And people, it actually encourages discussion within like, you know, within our group as a company who's doing conversion optimization and it encourages discussion with the company's team member, the company that we're working with their team members to have that discussion. It's, that's really how you establish a culture of experimentation.

Okay, so moving on to the next question. How do you interpret the results of an a B test that has inconclusive results?

Oh, I just gave an example of that. Correct. I mean, to me, how do you interpret the results of an a B test that's inconclusive? What does that mean? In the first place, we did like, you know, we reached the sample size, the required sample size and we did not hit the confidence level that we were aiming at. Well, okay, so let's say we're aiming for a confidence level of 95% or a chance to win of 95%. And we ran the test and Jimmy, Inconclusive is an 80% or less or 85% or less. By the way, above that I start saying, okay, well you know, the whole 80, 85, 90, 95 or 95, that's the standard that everybody follow. That's an artificial standard that we've created or statisticians have created. But there's nothing set in stone less than an 85 or less than 80. That's when I say we might have a winner, but it's also a non inferiority win. What does that mean? It means that I might have a winner, but that Winner might have just made the life of people coming to the website better, created better user experience. But that winner also doesn't translate into money in the bank. That's from the, from the get go. And to me you take. So you need to decide do I have a clear winner or do I have a maybe a winner, maybe, maybe a loser. And then I want to look at also at the statistical analysis that I'm doing. So not to get too deep, am I doing one tail or two tails? Statistical analysis, because that matters. One tail. I'm only looking whether this variation helped me increase conversion rates. Two tail. I'm looking at did it help me increase conversion rates? Well, maybe if it didn't, this actually hurt my conversion rates. So you want to look at, look at both. And I think in the case of, of inconclusive results, you want to look at the detail analysis to say okay, so this particular design did not increase my conversion rate. It has an 80% chance of winning. 80% confidence. Should I deploy it on my website because maybe it has better usability. Well then the next question is did it actually hurt your conversion rate? We know it didn't help, but did it hurt? So that's an important question. And then you want to look at the hypothesis that you started the whole experiments with. What was the hypothesis and what do the results tell us? Now remember, a hypothesis usually will have multiple implementations. Correct. Within a single AB test. My hypothesis is if I add social proof that will help increase the visitors trust and that will increase my conversion rate by 15%. How I at Social Proof? It could be in four possible ways. Four different designs. And now the question is analyzing each one of those designs, knowing that each of them is an implementation of that hypothesis to say did one of those designs perform better than the others? Can I actually combine some of those designs? Did some of them did do absolutely horrible? Not all of them are inconclusive. But sometimes I'm trying to draw, draw and do some analysis and make some assertions about the data. Two more points. One point is you might decide based on inconclusive results that you need to run another test to do more validation. Sometimes that test is strictly the same test that you started with, but by removing certain challenges to the original to say, you know what, okay, we started with five challenges to the original data was inconclusive. Couple of them, you know, two of them did better than the others. Maybe if we remove those three that last completely in very obvious way, keep the ones that were inconclusive but did okay and run the Experiment again, that's an option. Sometimes you say, well, I'm going to run two of those that have inconclusive results. I'm going to create the third one that actually combines some of them. That's another way to do it. And there are instances where you have to be very comfortable to say, I just don't know. People see through bs, by the way. And I always tell people, I'm like, you know, always assume that the person sitting on the other side is intelligent and you want to have a very honest conversation. That, huh, this was surprising. We don't know. What do you think? Be very comfortable asking that. Now you don't want to come like, you know, six months into every experiment. It's like, I don't know here, I don't know there, or that. Then why did I hire you in the first place? But every once in a while trial, you have to be very honest to say, I'm not sure. Actually, I need to dig deeper. I need to run more experiments to see what's going on.

So let's talk about copying competitors. So one of the questions that we had we have is, should you copy your competitors a B test?

I mean, I don't know, should you copy your competitors a B test? I don't know if you're fully aware of all the AB tests that your competitors are running. Sometimes easy, sometimes it's not easy. But I think a bigger question, a more important question, should you copy some of the features that your competitors are running? That's an interesting concept. I know there is a standard in the industry where we say, no, just ignore competitors. I think that's being silly. So that's one extreme and then the other extreme stream as well as copy competitors. And I've worked for companies that just copied competitors constantly and did not achieve any type of results. What you want to do is you want to look at your competitors and see features that you say, this might be interesting to test on our website. I might choose to take this feature and create an MVP version of it to test on my website. I might take it and say, you know what, for my particular, particular client base, customer base, this is how I need to modify this feature so it works better for the people who are coming to my, to my website. That's the approach that we take. A little bit of competitive analysis never hurts and it can guide. So the idea is not to copy but to steal ideas. Correct? Let's steal some of the ideas, let's modify them to work better for our market, market space and Sometimes remember, your competitors might be just testing a feature out. You know, they don't know and you're copying them. You don't know, and something that might not work for them, they're testing it out. You might find out that works well, really well for you. Data will only guide you. I think looking at what competitors do can inspire some research on your site to decide, oh, we should test this or we shouldn't test that.

Yeah. I remember sometime, I think in 2020, we had like an. A webinar with Jacob Lynn. I don't want to say his name.

Yeah, yes, he just emailed me. He just messaged me yesterday, actually.

Yeah, like you had like a webinar with him talking about the same. Same thing how copying competitors, a B test. I think the conclusion was like the same as you are saying today, that it's okay, like to get inspiration from your competitors, but not just like to go in blindly and just take everything that they are doing because it might not work for you, Jacob.

And he's really fascinating, by the way, because, I mean, I love people who don't follow just what everybody else says in the industry. And you don't get to that place unless you have a ton of experience. Correct. So Jacob has that ton of experience. And I think his site is good ui. I hope it is. I hope I'm not butchering this because, you know, but in it, he looks at patterns, design patterns, and he says, okay, well, here's a design pattern. Netflix tested it, Amazon tested it, Facebook tested it, and it worked. And here's the data that shows. So now if you're dealing with a client in a particular space, you can look at that pattern and say, hey, well, you know, data shows me that there's like 10 other companies that have copied that pattern or have used that pattern and it worked for them. Let's test this out. I think that's very fascinating. And I think we've actually at invest decided to start giving them some of our case studies to say, hey, here's another pattern that we see. Because by the way, this is something that we do as well internally. We look at patterns tremendously. Something that worked for a client. We worked with eight out of 10 clients or 10 companies that we worked with. We are going to test it. You bet. That we're going to be testing on the 11th client. Now we might modify it a little bit. Correct. But we're looking at that pattern as well.

Okay, so what do you think is the importance of predictive analytics in E commerce?

It's funny because in conversion optimization, we don't think a lot about predictive analytics. Correct. Predictive analytics is very powerful in E commerce, is very powerful in conversion optimization. And it used to be, by the way, a lot more. As a term, it used to be a lot more popular, I would say about five, six years ago. Nowadays the popular term is AI, Right? That's, that's what everybody's talking about. But really, if you think about it, ultimately the goal of predictive analytics is using different methods to try and predict certain behaviors in the markets and your visitors that are coming to your site. You typically, whenever you're creating, whenever you're using predictive analytics, you're either using machine learning or you're using AI as well to try and predict the future. There's this one company that we work with, they're strictly an apparel space in the apparel space. And what they do is they do this analysis and they combine analysis from all these different brands and how people buy from those brands and an analysis of Instagram, people on Instagram, influencers on Instagram to predict. And this is where it's fascinating what fashion trends are going to be popular in the next six months. So powerful. Imagine if you are the merchandiser and I come to you and tell you, guess what, this particular, you know, type of shirts especially, they do it really well for ladies. That's what I saw. It's going to be puff. Like, you know, with this, okay, I'm not going to embarrass myself, but like, you know, they describe like, you know, the color and the pattern and the. Yeah, and initially I have seen them when they do the demo and people are like, we don't know whether this is correct or not. But then three months later, they're like, oh, my God, that was so accurate. You know, how did they predict that? Well, they're able to figure this, they're able to figure this out for you as an E commerce website. This is very powerful. Correct. Because now you're talking about merchandising as opposed to like, you know, what the merchandiser is deciding while you have data to start placing orders. So that's one side that is very powerful. Think about it on the customer side of things. Correct. Being able to predict whether somebody is going to churn and based on certain behaviors. This person didn't, and I'm going to oversimplify it over here, but they don't log, then log into our website at a certain time. We expect anybody, like, not to place an order within three months. This guy had not logged in over here. So what do we do? Like, you know, how do we encourage them? Otherwise they will churn completely. That can help you keep customers a lot longer. Correct. That can help you actually increase your revenue as well. You can use predictive analytics in modeling certain patterns that actually encourage people to convert on the website. That's also another area that's very powerful. Not all companies can afford predictive analytics. Not all companies have the teams to run predictive analytics. But I do think there's enough software out there to help you do some of this work. I mentioned the company that we work with at the same time, it's a very expensive piece of software. It's like a $10,000, you know, monthly subscription. It has many different modules. But not every company can afford to spend 120, $200,000 on just the software. Correct. You still need to do a lot of analysis. You need to hire somebody to run that software. So you have to be careful where. And let me finish with this. Where we love to come in as an experimentation company is to say, well, here's what you're predicting. Let's go ahead and ab test that and see what the predictive analytics software is telling you. Let's see whether that actually helps you increase conversion rates or not. An example of a recent test is based on. Somebody comes into a website. It's an apparel website. We show them for the. Let's say they viewed a couple of items from different categories across the website. I can predict what fashion pieces they like. Imagine you just clicked on two items, you looked at them. And I can actually Predict with a 90% confidence here are the items that you. Here's house and bed dresses. And I'm going to show him on the category pages certain sorting that matches his preference. And by doing that, guess what? Now Simba all of a sudden falls in love with my website because, oh man. All these items that you're displaying for me on the category page, I love them. I just need to place an order with you. So we're testing that. We're running a B experiments around that and we see how well, how well it works.

I was trying like, as you're explaining this, I was also like, trying to think about the recommendation feature on FIGPI on how it also, like, is there like, any similarity between like, predictive analytics and that kind of a feature?

It's. It's a little bit, A little bit different. The, the recommendation feature on FIGPI uses machine learning to look at certain visitor behavior of those who converted. Yes. And it tries to predict that, you know, what are they doing? They're interacting a lot more with images. What are they doing? You know, like they're using coupon codes as an example. What are they doing? They're always scrolling this part of the page. They're always clicking on this item. So it analyzes that to say, you know what, here is the behavior that I've learned from those who converted, and here's the behavior of people who did not convert. They always struggle with this area. We then, based on that analysis of those who convert or would not convert, we can start to make somewhat good judgments of what changes we need to be making on the page. Would I call, would I call that predictive analytics? Probably not, because predictive analysis is about the future. But it definitely uses machine learning to help you improve your site usability and conversion rates.

Yeah, I can see, like, that's a smart feature. Like, considering that, like, it does all the heavy lifting for you. All you have to do is like maybe implement what it's telling you then.

That's exactly, that's exactly the idea that your visitors coming to your websites, few of them are converting. Look at that pattern, look at what's making them, what's persuading them to convert and start implementing that. Because that in theory should help you increase conversion rates.

Do you use that like in some of the clients that we have at Invest, like the recommendation feature?

We do not use it yet because until now, like, you know, we separated fake PI from envesp. So there's two completely different entities. And until now, the analysis, in all honesty, that the team at invesp does machine learning is very powerful, but it's looking at small sets. The analysis that the consultants do is a lot more powerful, looks at so many more elements. The machine learning analysis is the driven by the visitors behavior who converted. So it's different type of analysis. So, yeah, sure.

So one last question. I think, like, this is one of the questions that it became like an advice. Like in the CRO space, like, everyone tells you to always be testing. Like, always be testing, always be, always be experimenting. So should you always be testing?

I have to tell you, historically, the whole phrase always be testing, I always attribute that to our friends and their leaders in this space, the Eisenberg brothers, Brian and Jeffrey. They're the ones who are pushing that. But this phrase came out back in 2006, 2007, 2008, where no one was testing. Correct. I mean, that's what we spent a lot of time trying to convince people to do you need to be testing? There are times where you should always be testing. So I'm a firm believer that no new feature should be rolled on the website unless it should be tested. I believe that because you want to know that this feature achieved this KPI or this metric and then you can choose to roll it out or not roll it out. That's a set of discussion. It also forces you, if you're a B testing everything, guess what? The mindset changes. Now I don't want to wait until the features fall to roll it out from AB testing it. I need to have an MVP version of the feature and then test it out. So if I, if the AB test come back and says, no, this feature is not good, it doesn't increase your conversion rates or doesn't increase this metric, then you're not emotionally connected and you're not invested in that feature and you can say, okay, we're going to drop it. That's a tough one. Now, as much as I believe in that, I think there are times where you need to slow down and you need to say, you know what, instead of moving at a very high speed, let's slow down, let's do further analysis, let's dig through the data. And, and by the way, this comes from a company that's, you know, we can probably run 8 to 12 to 16 experiments on a site and it has the traffic. At the same time, there are times where we say, you know what, we got to slow down because the analysis piece is more important than the experiments piece. Now if you're only doing one experiment a month, okay, yeah, you need to, you need to up that. But sometimes you slow down from 12 experiments, from eight experiments to three or four, and that strategically can help you. Also, instead of like, you know, splitting your effort between different experiments to say, okay, let's focus on one, ship it out and see the results. And this way there's not a lot of interaction between, between, between experiments. The final thought that I have is the idea of looking, testing should be driven by an insight that tells you, yeah, we should, we should test this. So if I'm rolling out a feature, the feature was developed because of some data, correct data, things that people are struggling here. We're going to roll out the feature. And that's the reason we are going to be a B testing that feature. It irritates me when I hear somebody works in the field and their answer to everything, you should test this, you should test that. And I tell them, well, why do I even want to hire you. If your answer is always, you should test this, you should test that. Maybe the answer, a much better answer will be like, let's dig through the data and let's see what. Whether we should test this or not. Test it. Correct. And the minute we have data, then we can do the. Run the test, as opposed to, oh, you should test this, you should test that. And I don't know, that's what they're saying. Correct. Okay. I don't really need to be talking to you. I will test everything.

Yeah, I think those are the questions that we had for today. We managed to keep it under 15 minutes today.

Yeah, no, that's. That's a good thing. That is a good thing. I think our next. I wonder. Let's see how it's going to happen next week. I might be a little out of town, but we'll try and keep looking. I love the fact that we're doing the live CRO hour every. Every week, so.

Yeah, cool.

Thank you, sir. Tell me.

Yeah, and maybe, maybe, maybe I was supposed to say this off camera, but.

Okay.

Or maybe we should, like, try to. Yeah, we should try to. We should, like, experiment with doing it like, a bit earlier, you know, maybe an hour before, two hours. Because we never tested to see, like, which time works, like, best. Of course we have, like, people who are watching right now, but you never know.

Yeah. I'm always up for testing it earlier, testing it later, by the way. So, you know, we should do that. So we'll see. Yeah. Awesome. Well, until next week. Thank you, everybody, for watching. If you have a question, me and simba live on LinkedIn, so just shoot us a message and we'll be glad to answer it. Thank you.