#41 – CRO ROI, A/B Testing Surprises, Experimentation Setup

The old me used to say, before I jump into a marketing, I'd like to understand the investment. And if I'm hiring an agency, I'm asking them those questions. Hey, what kind of return on investment nowadays? And you see us, we hire different agencies. I don't really care for the return on investment. I mean, it matters a lot to me, but I don't care what the agency says. My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba, and I love asking questions about conversion optimization.

This is CRO Live hour, a show all about a B testing, experimentation and conversion. Conversion rate optimization.

Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal.

While pretty much every episode will take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like, do a B testing results fade over time? How do you go from low to high, testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts. Another week, another CRO Live hour. Simba, how are you doing?

I'm good, but we've been off for some time now. We haven't been doing, like, serial level. I think it's been two weeks, right?

Yeah, yeah. It's funny, the week before it was, all the questions that we got were about G. And I'm like, in all honesty, I'm not prepared to talk about GA4. Every time I log in, it looks a little different. They're changing things. Filters are still now working. Date is interesting. So I'm like, okay, I don't think I know everybody's asking questions about GA4, but we're leaving that for probably another week just for things to stabilize. And then last week, I was ready, up and running, and like, around, I think, 9:30am My electricity went out. I'm like, really? Electricity out? I mean, that rarely happens. Sorry, not electricity. Internet went out, you know, So I kept on waiting, and initially they told me I was going to come back in half an hour, and this half an hour kept on getting pushed. So I'm like, okay, we're not doing CRO Live. So here we are back CRO Live Hour. July 13th, 2023, halfway through the year. You know, fascinating, by the way, Simba, I'll just mention something and then we'll jump into the questions and whatever talk. What's fascinating is GA4 and what's happening with GA4. So until now, Universal, although officially it's been sunset, it's still collecting data as of this recording, as of this episode. We don't know officially when Google will kill it. I mean, they can do it at any point in time. And people are struggling with GA4. You cannot imagine the number of companies that are struggling with GA4. It's interesting because I have seen this and we've seen it's coming. For months, people thought that installing GA4 is similar to installing Google Analytics, Universal. They're two different animals, two different creatures. And I'm part of the Shopify plus group and literally some people there were saying, oh, I hired somebody on Fiverr to install GA4 scripts for me. And I'm like, oh, my God, that is a disaster waiting to happen. What do you mean you hired somebody on Fiverr? I mean, if it's just a matter of dropping the GA4 scripts on your website, don't even hire somebody. You can do that probably quickly yourself in like two minutes. But the configuration that's required for GA4 is a completely different animal. And the minute you told people that, they're like, oh, you're saying that because you're going to be making money off of GA4. I'm like, no, actually we don't, we don't sell GA4 services. It's not our thing. Now they found out that, oh, our data doesn't match. Even if there's like, you know, especially for e commerce companies, oh, purchase data is not there. If purchase data is there, it's not matching what's in ga. And it's just a mess. And now I think we're working with several companies where we're like, okay, we have to jump in and help you do the implementation because we've done it for our clients, we don't charge for it. But now we've reached a point where I had to tell some companies, I'm like, oh, we're fully loaded. We can't Even look at GA4 until next month, until August. But yeah, I mean, it's just fascinating to me how people put themselves into that.

Yeah, it's interesting. I see a lot of people talking about GA4 in the past few weeks. In February, everyone was like, google Optimize. BY now it's J4. So we're waiting for August again. Just a month.

Is it a month?

Just before, like Google Optimize also like a lot of posts about Google Optimized and it's interesting to see like all those things happening.

Yeah, I mean, that's the other animal that people are like now with with Optimize. It's interesting. So optimizing, Google Analytics, they're the same family. My theory on analytics is everybody installs analytics. Not everybody looks at it. You install because everybody tells you you should have analytics. Many people look at analytics maybe once a week, sometimes once a month. There are companies that don't even look at analytics altogether. That's a lot different than Optimize. People who are actively using Optimize who are actually creating a B experiments, they're looking at it every day. They want to see the results, they want to do analysis. So the panic that came with the fact that Google Optimize is going to go away, we saw that and I think especially in the next 45 days, so many different companies are just going to jump the boat. And we've seen it. By the way, with figpi, we've just recorded the second episode of Boot Trapped and figpi is after years never really shared this publicly, but figpi was not breaking even. We get more and more customers signing up, but our hosting fees increase. FakeFi is actually making money finally. And I'm like, oh, it feels so good. It feels so good that like, you know, we've broken. Our sales are up, the number of signups are up, enterprise clients on figpi are up, which is really fascinating. So I love that.

Yeah, that's a good thing. Before we jump straight into the questions that we have, it's funny that like most of the time when we start like talking like this and when you are explaining like some of these things, like a lot of questions then start popping up in my head.

Right. Pop them away, my friend. Pop them. Yeah, okay, sure.

I just want to hear like your thoughts. How do you think this Google Optimize going away will impact the industry? Do you think like, it's a good thing because people now get into like maybe more advanced tools like TPI or it's a bad thing. A lot of people were just like, okay, cool, Google Optimize is going away. So we won't be doing like any optimization or we won't be using like any tools like that anymore. What's your take on that?

I think two things are going to happen. Companies that have been committed to optimization, truly committed to optimization, some of them have been doing Google Optimize, but they're really dabbling into it. And now they have to make a decision. Is this a marketing activity that we need to focus on or is this something that we need to drop completely? That's going to be an interesting dilemma because really the cost of the tool is not very high. Unless you're going to go with an optimizely or a vwo.

Those.

Yeah, those can run you. If you're a large website, they can run you $10,000, $15,000 a month. There are a ton of tools out there. Fakepie is one abtasty. There's lots of tools out there that are not going to cost you a lot of money. So now you have to make the decision, do I continue optimization? Am I committed to optimization or not? So that's for the companies that are doing optimization. And I think you'll see a certain percentage that will drop out because they've tried it. They probably did not commit the resources to make it succeed. And they're going to say, you know what? Okay, we'll just drop this entry into a B testing is going to be an interesting challenge. And I don't have an answer to that. One of the good things about Google Optimize is that it made entry into a B testing and conversion rate optimization really easy. You don't have to spend money. The tool's available for free. It's there in your analytics. Jump on it. And I think that's, by the way, if I want to look at the industry and the growth of a B testing Google website optimizer, which is the very first child of Google to do AB testing, and that was released back in 2007, that's helped the industry tremendously. Here's a tool. Go ahead and use it. Done. And then became content experiments and then became Google Optimize. So for companies that are not committed to smaller companies, they're probably not going to do AB testing in the first place because there's no tool. It's fascinating because until now, one of the things that I see we're talking to a company earlier this week and one of the things they brought up is like, oh, like, no, we're looking at their sites, kind of like discussing different elements. And their CTO said, well, you know, it's like, oh, we know all this stuff. I'm like, okay, great. You know all this stuff. He's like, well, it's like some of it we fix some of it, Some of it we fix. Some of it is on our plans to fix. And I'm like, well, okay, so let me stop you. And to me, Khalid in the past would have answered this a lot different than Khalid right now. And I had to be very blunt with him. And I told him, like, you knowing the things that need to be fixed on your websites doesn't change the fact that they need to be fixed. That's number one. Number two, even more important, we don't call it fixed because really we want to ab test we can. The way we do business is we say, oh, there's a problem over here. I have enough data that shows that there is a problem. Let me come up with a fix. But I will ab test that fix. The reality, when I looked at the sites, they're making a ton of money. And it's funny because some people are making a ton of money despite all their efforts not to make money. Because you look at the site and I'm like, oh gosh, this is a site that's been built probably in the 90s. And that's also. I asked them like, so when was the site built? And he said, oh, we just did a new site redesign a year ago. I'm looking at it. I'm like, oh my God, if this is a site redesign, we have a problem. And I was looking at their analytics. I'm just not judging purely by looking at the site design. But their numbers are absolutely horrible. High abandonment rates, the clicks on the add to cart are not good. So many different metrics. But the mindset for a company like that is still not there to understand what it takes to improve conversions. They're still thinking dev teams run the projects and run how things are. It's not marketing. Correct. So is it dev or is it marketing? Because if a marketing person and somebody who understands marketing will say, you know what, we need to, to probably ab test this and find the right solution. Dev, on the other hand, is not thinking that the dev mentality is, I'm going to come in, give me the list of things, I'm going to make the changes, move on to the next thing. It's a checklist. Oh, I've checked this. I'm going to move on. CRO is never a checklist. And I think our friend Rishi wrote a tweet about this that CRO is not a checklist. It's a constant work of improvement, improve measure, improve measure, so on and so forth.

That's a good point.

I'm Ready?

I'm not sure if you are to get, like, to the questions of the day.

This was fascinating because Kobe created this chat, I guess, within our group. Never seen it before. So maybe I'm looking at haven't been a lot on Facebook. But then we got all these questions and I'm like, oh, okay. So cool. I'd like another way to send us questions, but yeah, let's go for it, my friend.

Okay, so the first question that I have is, before doing a conversion rate optimization research and audit, some prospects may want to understand the potential return on investment. How do you usually tackle this question?

I think this is an important question because the old me used to say, before I jump into a marketing, I'd like to understand the investment. And if I'm hiring an agency, I'm asking them those questions. Hey, what kind of return on investments nowadays and UCs, we hire different agencies. I don't really care for the return on investment. I mean, it matters a lot to me, but I don't care to what the agency says. What I care about is hiring the right person, knowing the process and whatnot. Because maybe I've learned over the years that marketing is really hard and conversion rate optimization is really hard. So I don't expect them. I'm like, okay, do you have a process? Do I think that you will follow up on the process? And in my mind, I'm like, if they do all of these things and they do okay, I calculate the roi. That's what I do. So you remember, like, we hired the Facebook agency and I hired them. We watched a couple of YouTube videos for their founder, and I'm like, hey, let's go ahead and hire them. We hired them. It was sort of interesting because the videos that I watched I thought was interesting when it came to the actual work. They were not doing what they were saying in the videos. Correct. The people that were doing the work, they're like, nah, we don't do that. I'm like, what? That's the whole point. I hired you because I like the philosophy and you're not doing that, and then you are working with them. They were responding via, like, loom videos. So they didn't have that personal interaction. Many things anyway. So it didn't work out. We learned a few things from them, but we just moved on. And I don't think we've ever really asked them about the roi. In my mind, I'm like, oh, if we can get a lead for fake buy at, let's say, 55 or $65, we're doing well and we actually managed to hit that, I think. What were we doing? 35 or $40 per subscriber. Now that doesn't really answer the question, by the way, because the question was if I'm the agency guy and I'm selling CRO services and people want to know the roi, how do I answer that? So few things that I want you to keep in mind. First, as somebody who is practicing CRO and this by the way, could apply also if you are somebody who just got hired as a conversion rate optimization specialist and the CEO is asking that question, your director of marketing is asking that question first. What you need to do is you need to look at the sites. What are the platform that you're going to be optimizing? What state is it in? Now? You don't have to do deep research. So half an hour to an hour looking at analytics, looking at the different pages of the website and you can give it a rating. Is it in a state? Like would I say it's at 50%, 70% is at 90%. So those are kind of like just general things. I will look at the site conversion rate. Sometimes we talk to e commerce companies and from the get go they come in with a conversion rate of 11%. I'm like, okay, at 11%, unless you are selling something that people need right away, 11% for traditional E commerce is fairly high. And anybody who tells you otherwise probably does not understand E commerce. Now if you are one eater on the flowers. Yes. Especially like on Valentine's Day, Mother's Day. Yeah. You'll have those spikes even outside of those days. Usually when somebody's buying flowers, they need them to be sent tomorrow or the day after. But if you're selling, let's say apparel and let's say you don't have a very strong brand. Yeah, the conversion rate is probably going to be anywhere between 1 to 2%. If you're doing okay, like you know the conversion rate. I mean we work with some apparel brands where their conversion rate is around 4%. So if I look at the website and they're doing 8%, they're doing 11%. I know that conversion optimization will always be effective because to me I can still increase conversion rates but it's going to be harder. Right. Because most likely they've done the things that lots of the low hanging fruit are fixed and they moved on beyond that. Now lots of times people look at the website and it has, let's say a 10% conversion rate and they say, oh, I can still fix this and I can still fix that. A conversation also with the, with the stakeholders, whether it's a client or a boss, to understand what has been done. That's also important. Have you done any AB testing? How do you make changes? All of these things will help you understand what kind of results that you'll deliver. So that's one piece of it. The other piece of it is kind of looking at your track record. What kind of average lift and conversion rates do you deliver now? Invest. We do hundreds of projects. We've done like, you know, tens of thousands of experiments. 32,000 experiments at this point. And it's actually fascinating because we're pulling some data around those experimentation and how successful they are. The first 10 experiments, 10 to 15, 50 to 100 with some companies we've launched. I was looking at one company earlier today. 361 experiments in two years. That's a lot of experimentation and analyzing the data. So we know on an average what we deliver. We know on average whether we deliver anywhere. It depends on the sector. It depends on, like, is it B2B, B2C? Is it E, comm, Is it lead gen? But I would say we deliver anywhere between 45 to about 55, 60% lift in conversions. We don't tell that to any company that comes to us. We tell them, hey, yeah, we work together for 12 months. Most likely we might be delivering anywhere between 15 to 20%. That's what we tell companies. And we do. Even conservatively, we do better than that. But it's better to kind of set expectations at that level and then deliver much better results. And some companies are not going to be happy with that. So we had a company reach out to us and they reached out to us. I think it was. It's been like about eight months. They reached out, talked to us and they said, hey, can you help us? And we said, well, yeah, sure. We looked at the site. I'm like, oh, there's just huge opportunity. Their conversion rate was about 1 1/2, 2% if I remember correctly. They're talking to different agencies and then they end up hiring an agency. They disappear for about three months. They email us again. I'm like, okay, hello. Like an old jump on the call. I'm like, what happened? You guys just disappeared. They're like, oh, we hired like kind of a full digital agency. They do also conversion optimization, but really it didn't quite work out. We think we need somebody who's a specialist. And they're like, you know, you guys were on top of our list. And there's this other CRO agency that's on top of our list as well. Kind of the second after you guys. And then they asked the questions like, what kind of results? I'm like, oh, we'll probably deliver 20, 25% lift. And then the lady we're talking to said, oh, but the other agency is promising to double our conversion rate. And I told her not to mention her name if that's the case. And I'm telling you like, you know, 15, 20%. I'm like, why are you even talking to us? Why are we even continuing the conversation? Somebody's telling you that they will double the conversion rates and you're still talking to us. And I'm telling you 15 to 20%. And she's like, well, I want to just explore my options. Okay, that's fine. We explored the options. I think she knew by the way in the back of her head, doubling the conversion rate is not real. She goes and she comes back and she says, oh, you know what, we decided to hire you. I'm like, okay, that makes no sense. Again, somebody's telling you they'll double the conversion rate. And by the way, this somebody who's promised to deliver the conversion rate is a good agency. I have to tell you, sometimes there's some like, you know, fly by night operations. When I'm like, ugh, those guys. Oh man, I feel bad because I'm like, you're gonna get ripped off. But it's self serving to say that. So I'll tell them, hey, go with this other agency. But in this particular case, the other agency is actually a good agency. I know them, I know the quality of the work that they do. And I'm like, oh. I'm like, I would go with them. She decided to come and work with us. Now she start putting conditions like I want to interview the team and I want to interview the project manager. And I told her, I'm like, oh, I don't think we're a good fit. We don't do that. And she's like, what do you mean? I'm like, listen, we don't do that. You know, I don't know who's going to be working on the project. Now this is a little different because some agencies will let you interview and I don't know who's going to be working on the project. In all honesty, it depends on when you sign up, my resource availability. And really the way we do work, we're probably a little bit different. There's multiple people on the project. Yes, you have a lead. You have a senior CRO specialist who's working on the project and they only work on three or four other projects. But who's that? I don't know. And I can't commit somebody until you're ready to sign. So she got really annoyed and walked away. I hope she didn't sign up with this other agency. And actually he's a friend, so I'll probably ping him and ask because. Oh my God. If a client signs up with you and they have the experience expectation that you're going to double their conversion rate, oh my God, your life is going to be hellish on the project. Now, mind you, I understand that you might still sign up that client. Yes, they have their own expectations. Yes, you're not going to be able to deliver on those expectations. But sometimes you have to take those clients.

Correct.

We have a major company that we're presenting in front of tomorrow and interesting because they generate about 10 leads a month. That's it. We don't do ab testing at 10 leads a month. It's too small. But a lead for them is about 1.5 to $2 million. So they're very large enterprise sales. And I was talking to our VP of sales and I told him, I'm like, you explained to them that there's no AB testing. There's lots of strategy that goes around this loss of thinking, but no AB testing. He said yes. I told him, does he have the right expectations? Because without AB testing, we're not exactly measuring. Because I had signed a client a while back at Enterprise Clients, and although I was very clear with their VP of marketing, she had turned around and told the board that, oh, we're going to be able to double our leads. We're going to go from 20 leads a month to about 40. And I'm like, how? I never promised that. And we end up in very awkward conversations with their board showing them like, you know, it's like, oh, here's the conversation, we're very clear about it. So long story short, set expectations low of something that you can actually deliver. If you do better than that, you're going to be a lot better. Understand where the site is and make sure that as you are doing AB tests, this perhaps is the last thing. Get the company that you are working with, whether it's a boss, whether it's a client, get them excited about the work that you are doing, get them involved with ideas. I think that can guarantee that somebody will stay with you. For a long time.

I like how you answered that question. It was a good way. It was like a diplomatic way. But for some people, they might think when you ask them about return on investment and maybe as a prospect client, right before you hire them, they might see it as a red flag because it also shows that maybe you don't understand the value proposition of CRO. But I totally understand how you answer these.

So I would disagree with that, by the way. Simba, so. Which is sort of fascinating because some CROs like, oh, if a client, then they don't understand like CRO. And I'm like, no, no, no, no, no. You don't understand business when you say that. I'm like, you don't to understand business. What you're offering is a business service. Now how you position that and the ROI on it, it's up to you and your understanding of business. A VP or a CEO, VP of marketing or CMO or a CTO doesn't have to really understand CRO. They have about a million other things going on in their lives. And you're just one out of those million things. Correct. You need to be able to position yourself. And unless you're getting hired by some R and D company, usually when companies are doing marketing, there's some ROI on it. Now, some of it is measurable, some of it is not measurable. It's almost like you're taking them on a journey. Here is the map on this journey before having Google Maps and whatnot. Okay, but I'm going to take you from point A to point B. It's not necessarily a straightforward line, but here are the signs that we are doing the right thing. Most companies, companies think about that like, you know, they want to see. I'm like, okay, if there's no ROI over here, you cannot even like, you know, tell me like, oh, it's not about the roi. I'm like, oh, well, no, sorry, ROI is how the business is run. Correct. Now what's the ROI of having a project manager? What's the ROI of having like, you know, some of it is difficult to quantify. Is it important? Yes, it is important. So you position that it's not necessarily one to one. Just position it in a smart way and forget about your perspective. That's an issue with lots of people who work in experimentation. There's this attitude where I know and it's like, holier than thou. And I'm like, no, you don't know. I'm sorry. Running a company is hard and you're one piece of you're the one element of many, many other things. So just understand your place in helping the company execute that growth strategy. I don't know, that's just my thinking because I just see so much. So many CROs are like, oh, this person doesn't understand or that person doesn't understand. And I'm like, I think you don't understand business and what it takes to run a business.

Yeah, that's a good point. The second question that we have is any recommendations for tools that can help tell you exactly which fields people drop off from a form? I think like we were talking about reform or something like that.

Yeah, I was going to turn this question to you because we've done a couple of like joint efforts. Any tool that you recommend for like a kind of form analytics, I think.

Like they have tools. I've never used them myself, but for that kind of a question, I think like maybe what comes into mind, of course, like it's ZQO analytics. That's number one. Number two, I think maybe session replace. They might also show you where people are getting off. It might not be like exactly like a formal analytic tools, but you can see like which fields that people are not serving. And I'm not sure about Google Analytics, but I think there might be a way to also do that because something like when I check like Google Analytics, especially for Flickpy, we can see like what people are dropping off. So yeah, I think those are the top three that comes into mind.

I would add maybe heat maps as well because like, you know, especially if a good heat map tool is configured well, it's going to show you the different fields. Correct. And how people are just clicking on the different fields. So you'll see where there are clicks. So that's one. It's okay. And it depends on how the heat map is. And again, there's tools specialized. It's fascinating because hotjar used to have form analytics as part of their suite and then they dropped it off because people were not interested in that, at least for their markets. The other way that you can do it is through Google Analytics. You can fire events as people are going from one field to the next. You can do that if you have a developer and that will show you exactly how people are navigating outside of that. Really. I think those are kind of looking at the ones that come to my mind. I wish I had a lot more. I don't think our team, which is actually horrible maybe, but we'll say it. I don't think our team spends A lot of time looking at forms and form fills, but I might be wrong. So I'll. Tomorrow we have the CRO education 7:30am on Fridays. So I meet with the other CROs and I'm going to ask that question just to see because they might shed some light on some areas there.

Okay, cool. So moving on to the next question. What are some surprising A B test results we have seen?

Oh man. The rule there is two things I always tell people. One is I've done enough A B testing that I've been humbled way too many times where I think that this is just so obvious. We are fixing a problem over here. And then you analyze the results and people absolutely don't like the solution. And I'm surprised. And at this point I get surprised by the results, but I'm not surprised that I'm surprised. If I may say that I'm like, oh, okay, well, I guess people didn't like it. Now, over the years, perhaps there are a few A B tests where I'm like, really? Okay, so I'll just mention a couple of them. So one thing that I used to say quite a bit is that having the add to cart button on category or collection pages doesn't make sense. I used to say that all the time. And I still, still say it probably, but I probably should put kind of an exception. You need to test that. Now in the US we've tested this many times where you move the add to cart button and it helps actually increase conversions, actually increases the flow to the PDP because people don't make the decision to buy an item from the collection page. At least that's the way I think about it. I want to look at the item, I want to examine it. I'll look at the details and then I'll make the decision to click on the add to cart. One time we were testing that because we think it's like, oh, so an easy, quick test, testing that. And this is kind of one of the easy tests. The first couple months usually for us are not exactly sophisticated or complicated tests. So that's an easy one. It's a nice small win. It might give you 4 or 5% lift. Nice nonetheless. And we tested that and actually caused a major drop in conversion rates for the particular company. And I'm like, what do you mean major drop? I'm like, you must have. My first reaction was there is a bug in the A B test. If there isn't a bug in the A B test, there's a problem with the way we're claiming collecting data. I blamed everything on data collection, on the software, on analytics, and lo and behold, no, everything was configured correctly for that particular retailer. And that's a very unique case. My God, People were clicking on the add to cart a lot more from the collection pages than the product pages, although they were selling electronics. And I'm like, what? I was just shocked. I looked through analytics, looked at the funnels, looked at the events, and I'm like, oh, my God. There's always an exception. Now, of course, mind you, by the way, we probably should have looked at that data prior to launching the A B test as opposed to just saying, oh, such an obvious thing. Live and learn. This was a few years back, but then as I speak with conferences, some conferences I speak at quite a bit, I came back and I said, you know what? Yeah. I used to say, that's add to cart on the collection page. Doesn't make sense. I guess sometimes it makes sense. So that's kind of a surprising a B test where I'm like, oh, okay. Another AP test that actually surprised me. It was not from us, but somebody kind of sent me the test. And he's like, what do you think of this banner? Also, banner tests are really easy. Correct. You just throw them together and the banner actually gave about an 8%. It was across the website. And I love experiments that are across the website because now you really. You do an 8% lift in that a B test, you will see a nice lift in your revenue on your conversions. 8% lift. Love it. The problem with the banner is that it had a spelling mistake and a very glaring in your face spelling mistake. He's like, what do we do? I'm like, just fix it. He's like, oh, the minute we fix it, the variation loses. And like, you know. And I'm like, what? Go back again? I'm like, just test it again, man. He probably didn't give it enough time. And he just sent me the data from three separate instances where they tested this. Every time the banner with the spell mistake won. And I'm like, oh, gosh. He's like, how do you explain that? And it was kind of like a funny attempt. I'm like, maybe trying to explain something like that. I'm like, maybe like another spell mistake. Stop people. Otherwise we're banner blind and we're not looking at the banner. He's like, what do I do? I'm like, that's a very honest conversation with, with the company you work with to show them that there's like, hey, do you want this spell mistake mistake is worth like, you know, 8% or do you want to like, you know. And by the way, at the end they kept it for about two, three weeks and I think the VP of marketing came back and said, I don't care. I refuse to have a spelled mistake this big across my website fix it. So that was a surprising a B test. The last one that I'll mention is the showing the shipping date. And this is a very recent experiment that we've done. Do you show especially like, you know now and this was a lot more prevalent, maybe a year ago, but still, still it's an issue. Do you show when the item will be shipped out? You know, on e commerce website? Yes, you should show it. Do you show it on the product page? Do you show it on the cart page? Do you show it in the checkout? I mentioned this experiment before and the first response like, oh, you should show it everywhere or not show it everywhere. It turns out, at least for the particular site that we tested, we show it on the pdp, we don't show it in the cart, we show it in the checkout. Less prominence. So it's interesting because we tested so so many different variations of this because it's an experiment that ran across three different pages with three different possibilities. So there's I think 27 different combinations. Now you have to have a website correct that gets the traffic and gets enough convergence to be able to run an experiment like that. But that was kind of a fascinating AB test and although I told you that was the last one, I'm just going to add one last thing. It's always interesting when you run an experiment and you're not running a responsive experiment. So you're running the mobile versus the desktop. Two separate experiments and it's the same concept. It wins on mobile, it loses on desktop or vice versa. And I understand because the visitor who's coming to the website using a mobile device is different than the visitor coming on desktop. They're different. But that also poses other set of questions. Okay, so what do we do? Do we go with the winning design? Do we combine them? Do we actually create separate experiences? So those are kind of the interesting things that I love kind of to dig into.

Yeah, those are good points. What's better? Setting up experiments with a visual editor or using simple JavaScript?

This is a question being asked for somebody who runs figpy. It's fascinating. Every A B testing platform has a visual editor. I do not like visual editors. They have their limitations. In theory, Visual Editors will help you. You make changes to the site that are very quick. In reality, sometimes those visual editors add too much code on your website that automatically slows down your variations. So while you as a marketer, you think you've done something really good, in reality the code that you produced is really not very well optimized. So to answer the question, most of the time we try to avoid using the Visual Editor, although it's there by the way, for people to use. And I tell people, look at the type of experiments that you have if you're changing images, if you're just changing headlines, something really simple, yes, Visual Editor probably makes sense because maybe it's like five minutes in the Visual Editor, something more complex. I'm changing the layout of the page, I'm changing multiple things on the page. It's JavaScript. Have a good front end developer go in and change it Sometimes, by the way, I'll make the change using the Visual Editor, then look at the code and kind of change the code from there as well. Kind of combining Visual Editor helps me, but then I'll do some cleanup. So that's kind of the answer, I think most companies, I would say looking at Figpy, about 90% of the experiments that are run on Figpy every month are actually using JavaScript and CSS code, not using the visual editor. About 10% of the experiment experiments are actually using the Visual Editor. Mind you, we've spent quite a bit of time improving our Visual editor in the last couple of months, cleaning it up, cleaning the code that it generates. We actually have two developers who are just kind of looking at it, making sure that it looks really good still with all of that, Most companies using Figpy are still using the JavaScript, the CSS, but I think the Visual Editor is a marketing tool. That's it.

Yeah. I just want to know usually how long does it take to come up with an A B test idea and to implement it like that? Time.

So for us, on average, from ideation to the point of launching an A B experiment, it takes an average of about 13 and a half hours. But we're fast. We're probably three times faster, four times faster than anybody else for multiple things. We've done a ton of experiments. Correct. So you have a team. It's just like when you have somebody who runs every day, runs in the morning, runs at night, it's a lot easier for them than somebody who runs once a week, than somebody who runs every now and then. Correct the process, the team, I mean, that Helps us move really, really fast, kind of in launching experiments that fast. And some of those experiments are fairly complicated. We also have frameworks, developments and code frameworks that help us code a lot faster. You don't get to launch 32,000 experiments by writing the code every time again and again. So that's one thing that we do. I think on average, what I see is that most companies will take anywhere between 30 to 40 hours to launch an AB experiment. The same experiment that takes us about 13 hours. So that's one. Now, the other piece that people don't think about is those 13 hours are not necessarily happening. I'm going to start today. I'm going to launch tomorrow. The duration. Correct. Those 13 hours might take four or five days because there's different team members and there's kind of a queue. The ideation, like the CRO brief, the design, the approval, going back with feedback. Now it goes to development, goes to qa. QA probably sends it back to development, always ready to launch. Launch. So that duration is four or five days. Sometimes it's longer. Correct. That depends on the business processes and whatnot. So this distinguish between duration versus an actual time that it takes to implement something.

I think you explained it well. Those are the questions that we had for this week at least. You want to add some insights?

No, I'm super excited about the fact that also Boot Trap is coming up. I think the plan is to release four or five episodes all together and then, like, we'll go back on Weekly Cadence. So, you know, super excited to see how that goes.

Yeah, that'll be great.

Awesome. Okay, well, until next time. Happy testing. Thank you, Simba.

Bye.