#42 – Building High-Converting Category Pages

Yesterday, I decided to subscribe to Liverpool TV just to watch the preseason games. It's almost like when you watch a basketball game where people are not keeping track of the points and it's not competitive. The minute you start to keep track of the points and you have competitive people, the games get a lot more interesting. And I think, by the way, the same thing applies in business. If you don't have specific goals, people are just doing their thing. The minute you have specific goals and specific timelines that you expect everybody to accomplish and they're accountable for those, the performance just increases. So maybe it's the same thing in soccer.

My name is Khalil Slade, and I love to talk about conversion rate optimization.

I'm Simba and I love asking questions about conversion optimization.

This is CRO Live Hour, a show all about a B testing, experimentation and conversion rate optimization.

Each episode Khaled amazes me, answering some of the difficult CRO questions, dropping insights like, it's no big deal.

While pretty much every episode will take on a new set of conversion rates optimization questions that are not easy. We will talk strategies, we will talk process, and we will talk tactics. Simba will be bringing all the questions.

Oh, man. I bring tough questions like do a B testing results fade over time? How do you go from low to high, testing velocity? How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah, Khalid, these are very, very tough questions.

Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour PodC today wherever you listen to your podcasts.

Another week, another CRO Live hour. We are back. We did it last week, too.

Yep, we did it last week.

We're trying to be a little bit more consistent. And you know what else I've been very consistent about? This Boot trap managed to record up to four episodes of the podcast until now. Three, Sorry, the fourth one coming next week. And then we release the first three. Maybe. Or do we release the first five? What do you think?

I'll have to take time to think about it. Then I'll let you know what I think is the best way to go about it.

How about you?

How was your weekend? How was this week?

Week has been very busy. You know, some are supposed to be slow, but it's been busy personally. And it's been busy business wise, personally. Lots of family. I mean, I think for the past month and a half, there is family. In and out, in and out. Usually we travel in the summer. This time we did the other way around where family is visiting. I don't know which is better. I think when family is visiting, you're just being the host. So there's just a lot of demands there. And I didn't really appreciate school until this year. Oh gosh. The fact that kids are at school is. Makes life so much easier. We still have about another month before the kids go back to school. And I'll be a very happy man when they do. I mean, I'm not sure if they will be happy, but I'll be a much happier man. And business wise, things have been a lot busier for investors. So usually traditionally summer is slow. Most e commerce brands are kind of prepping for Labor Day and the Black Friday cyber Monday. And traditionally 95% of the companies that we work with are e commerce companies. This summer has been interesting because we have had a lot more demand from lead gen companies. I would say it started about a year ago. So we went from 95% E commerce, 5% lead gen to right now. The breakdown is 75% of the companies work with are E commerce, 25% are lead gen. And it's fascinating because typically I'm very involved in the sales process, at least maybe not the initial discovery, but as we kind of like hone in, analyze a company, make sure that they are a good fit. I would say for the last month or so, companies are signing up with us and I'm not involved at all. So that has been a very interesting progress for invesp where I'm like, oh, I don't know anything about the companies. I know they match because our sales team is very careful to make sure that any company would bring on board matches our requirements. But I have not been involved. So it's been fascinating. I mean, kind of that growth stage. And I think, by the way, I don't know if you know this, but if you think about your role, you were the first to kind of take on that lead role.

Correct?

It's like, okay, now you're leading the marketing and I'm less involved in the day to day of the marketing. Same thing happened now with sales, same thing is happening with operations. So we're going from initially me and Ayat doing the work. That was kind of the very first stage of envesp's life. And then the second stage, oh, we're doing the work with the team. Now we're at the third stage where the team is doing the work so it's interesting. The calls that I have are a lot more strategic. I love it. It's challenging because I think the questions that come up when you're dealing with strategic questions are a lot harder. But sometimes I think for, hey, let's try and just make a decision quickly. And I think that's one problem that we're struggling with where we don't make decisions quickly. Analysis paralysis, making sure. It's like, oh, is this the right decision? And we spend a week, two weeks sometimes analyzing and then sometimes we just need to make a decision, move forward because I think we can make anything work. Now you can make anything work with the same level of effort. But would you have been a lot more effective by taking a different path? That's always the question that I ask. So a lot of video recording tremendous amounts. I bunker. It's funny because the type of videos I do are rather lengthy. They're 10, 15 minutes sometimes, sometimes a lot longer. So beginning of this week I said, you know what, I'm just going to sit down and I'm going to record maybe 20 YouTube shorts, Instagram Reels, TikTok videos. It's a lot harder to record those shorter videos because you have to be so precise and concise. It's a lot different than recording a 10 minute video where I'm comfortable, I can talk about a topic. Now I need to pick a point, be laser sharp and then focusing on the points, make my point quickly in a minute, minute and a half and then move on. So it's been interesting. By the way, let me ask you this because I spend a lot of time analyzing YouTube shorts, successful YouTube shorts. How long do you think a successful YouTube short is?

How long do I think it's successful?

Yeah, what's kind of the average?

I would say 50 to 60 seconds based on like some of the channels that I follow. They do very well, but I think like in values, at the end of the day, it's all about maybe how interesting the content is. Right.

So I've done analysis now. This is on a single channel. I love Alex Harmuzi. We've talked about him and his channel. I bunkered down there and I pulled his last 50 YouTube shorts and did the analysis on them. And he averages about 35 seconds to 37 seconds. If I thought one minute was hard, 35 seconds is so much harder. But I can tell by the way that his approach is not necessarily sitting there and recording the YouTube shorts. I think they're part of a longer questions that he has and Then somebody goes in and picks up those shorts and cuts the longer videos to say, oh, here's an interesting point, but also like kind of picking a point that is like 30 seconds to 40 seconds. That's excellent as well.

Yeah. Speaking of like how difficult or challenging it can be when it comes to recording, I remember the moment when we were recording the intro for the CRO Live hour podcast. The duration of the whole trailer. Yeah, it was supposed to be about 54 seconds, but we spent about over 34 minutes just trying that 54 second thing. So yeah, so now imagine like the number of videos that for a 10 minute video, I'm sure like you keep on like repeating or I'm not sure. Or doing like a lot of retakes. That's challenging.

I'm struggling to hear you, by the way. Simba. Yeah, it's fascinating to me because one of the things I think about is just sitting there picking up one of our LinkedIn lives, the CRO Live Hour, and choosing different spots from it as opposed to going to YouTube and picking different spots from it. Might be interesting, might be fascinating to see, like, you know, can find things there that actually will be really excellent for YouTube shorts. So. Excellent. So how about you? How was your week?

It's been a good week. I'm always like having good weeks. I'm not much of a complainer, so I try to make everything so interesting or maybe everything so exciting. We've been working on our okras from August to November, so we have worked on that with the team and I think we're good to go. So I think it's been a good week. No complaints.

Yesterday I decided to subscribe to Liverpool TV just to watch the preseason games. It's almost like when you watch a basketball game where people are not keeping track of the points and it's not competitive. The minute you start to keep track of the points and you have competitive people, the games get a lot more interesting. And I think, by the way, the same thing applies in business. If you don't have specific goals, people are just doing their thing. The minute you have specific goals and specific timelines that you expect everybody to accomplish and they're accountable for those, the performance just increases. So maybe it's the same thing in soccer.

We're talking about high converting category pages.

I see a lot of people, like.

When it comes to E commerce, they usually focus on product pages, cart pages, and also maybe the checkout flow, also like maybe category pages. But they don't get like, people don't talk about them that much as much as they do like product pages. Why do you think like people focus much more on product pages and not really like category?

Most people focus on product pages because they think the minute I got somebody to the product page, I have a better chance of converting them. They have shown me the visitor had raised their hand and they said, hey, I might be interested in this product. And because the add to cart button is there, you think, hey, if I can just get them to click on that add to cart, I have a better chance of converting that visitor. That's the thinking that most people have. And that's the reason a lot of focus is on the product pages, on the cart, on the checkout. You always have to keep in mind that the rest of your site, that is your landing pages, your homepage, your collection or category pages, if they do a good job, then the job of your products card checkout pages becomes a lot easier. Because conversion is not a check mark that a visitor has. You know, it's kind of think about the human process, the human mind. And how do you actually persuade somebody to convert? Just doesn't happen by getting, just looking at a product and saying, oh yeah, this is the right product for me. It starts much earlier. In reality, the conversion of somebody from a typical visitor to an actual customer on your website starts even prior to them coming to your website. Somebody has to recognize that, oh well, I have a problem in my life I need to solve or I need to make this game, make this progress in my life. If somebody doesn't recognize the need or doesn't have the desire for the pleasure that your product provides, benefit that your product provides, your chance of converting them is minimal. And as a matter of fact, the more somebody has a need, the easier it is for you to convert. And that's the reason I always focus a lot on the landing pages. I focus a lot on the collection pages because I think we can do so much in telling the visitor, yeah, you are in the right place. Yes, you can trust us. Yes, we have all these products. That's really what I think the collection pages, even the landing pages provide one.

Of the things you do. I've noticed. Suppose like when you're going through like some of the e commerce websites, is there like something or key elements that distinguish a high converting category page from.

A average wise to evaluate whether category page is high converting or not, you have to understand the job of the category page. Category pages are what I call funnel pages. Their job is to funnel visitors to the right product. That Meets their requirements. If the visitor looks at your category page and they are confused because you present them with way too many product options, then the category page did not do its job. How do I help visitor get to the right product? That should be the main concern that you have when you are designing a category page. Now, one of the things that I see all the time with e commerce companies is they just throw together a whole bunch of products on category pages, on collection pages, and they just move on. They might add a little bit of filtration and we are done. And sometimes the products look very similar. One of the things that we like to evaluate on a website is what we call cognitive ode. Fancy word, but it means something very significant. Does the visitor struggle as they are trying to find the right product for themselves? This struggle could be a result of many different things. You could be selling apparel somebody wants to buy. Let's say a lady wants to buy a top. She goes to the category pages, woman tops. And she sees. And you have like, no, 200 different tops. There is a cognitive load that happens over there. And that's the result of having many different products. And now I need to find out the right product for me. Is it going to fit me? Do they have the right colors? How is this item going to look at me? So those are questions, correct, that are a result of you offering many competing products that could match what somebody desires. I'll give you a completely different problem that e commerce companies struggle with. We were talking to a company, they sell pesticides, pest control, you know, so if you have, you know, ants, bees, you know, mosquitoes. And one of the problems I struggle with in the summer in Chicago is that we have ants. We have ants all over the place. And I'm like, oh. I'm like, they sell something that's going to help me control the ants in the house. So I go, I select that I want something inside the house. And I'm dealing with ads. And their product page had about 20 different items. Sounds fairly simple. Except now I'm looking at the product page and I'm struggling. Which one of these products is the right one for me? Why would I pick product one versus Product three versus Product five? And as a matter of fact, product one and product two that they have on the category page were using the same image. And I'm like, okay, is it the same product? Do they have a problem in the way they categorize their products? Why are we using the same image? And the cognitive load happened there because I start clicking on each One of those products trying to read the information, I had to open multiple tabs. And I'm still not sure. Now I am somebody who raised their hand and said, I have a problem. They said, hey, we can solve your problem. It looked like they can solve my problem. I went to the category page and all that category page did is just increases my cognitive load. I could not pick the right product. And the failure at the category page didn't matter that I went to the product pages, you know, because I went like, you know, 10 of those different product pages that solved the problem that I have. I could not figure out which one is the right product for me. So there's multiple failures. But it started with the confusion that I had at the collection page where I was not sure what to pick. I've given this example multiple times. North Face, which sells jackets and they only sell 12 different jackets. Fairly simple and straightforward. I go to their category page, I'm looking at the 12 jackets and I still remember, by the way, their VP of E Commerce or Director of E Commerce saying, oh, how complicated would it be to design an E Commerce website when you're selling 12 different jackets? Oh, it can be very complicated because I'm looking at the jackets and I'm thinking to myself, oh, okay, which jacket is right for me? Now, mind you, I have decided to buy the jacket in the summer. Not exactly the best time to buy a jacket because I cannot try it on. But I had many different questions. Which jacket is going to keep me warmest in Chicago's cold winter? How well do these jackets fit me? Can I even wash the jackets or not? And I look at it at the collection page and there's no filtration. There is nothing. And I literally, again, cognitive load. I open all the different product pages, correct. I know 12 of them and some of them provided guidance on how warm you will feel. Some of them did not. I'm like, oh man. And the funny part is that the jacket that I thought I'm like, oh, this does not look, it's going to like, it's going to keep me warm. They had a scale on that product page that shows how warm it will keep you that said the warmest. So now I'm looking at the jacket that seemed very light. And on their product page they said it keeps you really warm. And the jacket that looked to me as a consumer potential customer that, well, you know, this probably is the jacket that's going to keep me in the warmest did not have that scale. And I go back to the category page. And I'm like, oh, can I even filter? I mean, this is the very basic thing that I want as a potential customer. I want to be able or sort the products that you have on the collection page by how warm they keep me couldn't find that information. I've been looking at their category or collection page at the North Face now for the last three months. I can make a decision. And I'm just looking at it and I'm like, okay, well, I don't know, maybe I need to make a trip to their store somewhere and hopefully I find a salesperson who knows how to explain this to me. But I would give their collection page 0 out of 10. And I don't easily give 0 out of 10. I mean, it's nicely designed.

Correct.

But I don't really care about nice design. I trust the brand. I mean, it's a well known brand. I don't care about nice design. I expect you to design the category pages really well, that they look nice. I need the functionality. That's what I need. I need your category collection pages to guide me to the right product. You have not done that. You failed. And we did talk about high, well, very well converting landing page collection pages. We did not talk about high converting category pages. But I think by talking about the pages that don't convert well, that are poorly designed, that don't provide the functionality, doing the opposite of that will give you highly converted collection pages.

Yeah, that's a good point. That's a good asset.

And this is not specifically to North Face, but I think every brand has an e. Commerce department. Correct. I mean, it's funny where some brands, they confuse e commerce and marketing and they're kind of like clumped together and I'm like, no, no, no. They're two different functions. And I think the fact that we're doing it online causes people to get confused. But think if you have a regular store brick and mortar store marketing guys are responsible for getting people to the store. Oh, maybe we'll do flyers, maybe we'll do advertising. Maybe we'll do billboards, maybe we'll do TV advertising, radio advertising. I don't care what you will do. That's the job of marketing. You're not going to have your marketing team decide how you lay out the store. The marketing does not get involved in the store layouts. It's your merchandising team. It's a kind of a different skill set. And the same thing applies online. By the way. The marketing team is responsible for bringing people. Maybe they will work closely with the E commerce and merchandising team to figure out, okay, here is the best layout. Correct. But they're two completely separate functions. And not to pick on any particular store because I am sure that those teams are really busy. Sometimes people say, oh, they don't know what they're doing. I'm like, oh, I'm sure they have their own list of priorities that they come up with items that they want to fix on the collection pages, product pages, car checkout and whatnot. But at least to the specific customer who I am, those priorities that I have as a potential customer and the priorities that the E commerce company has might not match, and that causes a lot of conversion. Now they might say, you know what, Khaled, you're not our target market. Our target market is completely different. That's the reason we're not focusing on your priorities. That's fine, as long as the priorities are driven by specific market segment that is important to the business.

So that's a good point. Most of the times the category pages, they have a wide range of projects. How do you strike a balance between providing enough options for users and avoiding overwhelming them?

Whenever you're designing the category collection page, the further down you can drill and create subcategories, the better you are. Now, don't create subcategories with one or two products because at that point in time, it doesn't make sense. I do like category pages with at least, I would say, 10 to 20 different products. So I was@macy's.com and I was looking at their polo shirts. I mean, they had like, you know, almost like, you know, a thousand different polo shirts there. That's a bit much. Now, they do provide filtration, so I can filter it out. If you're going to have that many products in a collection or a category page, filtration becomes critical. And even more critical is that filtration works correctly. So I'm like, okay, I'm looking for a polo shirt. Great. Let me filter it out by brand. Okay, let me pull, you know, boss, for example. Okay, great, there's still 170 shirts. I'm like, oh, gosh, that's a lot of polo shirts to look through. Let me like, you know, this is funny talking about a consumer and where the purchase process starts. I was on Quora and I wanted to see kind of the research about polo shirts for Boss, and somebody says, oh, look at where they're manufactured. If they don't make them in France or Italy, Then they're probably making them, like, you know, in Vietnam, and it's not the same good quality. And I'm like, oh, maybe I need to pay close attention to that. Oh, do they have that as a feature? No, they don't. You cannot filter out by polo shirts that are made in a specific country. And I understand sometimes you don't have that information, but again, you having or not having that information is a lot different from what the consumer wants. And maybe also I'm a very small segments. Then I said to myself, I'm like, okay, well, let me go to this polo shirt. I'm gonna sort by price, guys. We're a little cheap, so I'm gonna go by price. I, like, start looking at the items, and I'm reading through the product page and says, hand wash only. And I'm like, whoa, whoa, whoa. Hand wash only. Who's gonna do that? So I'm like, oh, I wonder if I can go back to the collection page and I can filter those out. Because I really. I'm not interested in hand washing my shirts. I just throw them in the washer or take them to a dry cleaner. Didn't have that information. So now I'm left with 170 polo shirts from Boss. And I'm like, okay, maybe we need to, like, you know, sort them by color. And I'm like, okay, I just want plain. I don't want anything like, you know, with stripes or anything like that. The filtration there fell. Did not work. Again, if you have a collection page with a lot of item, I would think about how do I break it down further. And now you can do this in two different ways. One, through providing filtration, making sure that that filtration one, matches the criteria that your visitors have. Two, that it works really well. And you need to spend time testing this out. You need to spend time talking to customers to understand how they filter. And you need to spend time looking at the filtration, making sure that it actually works well. And by the way, this is kind of a holistic view. Looking at heat maps for your product pages. We're talking about collection pages, correct? But looking at heat maps for the product pages, seeing high interactive areas, people, where are they clicking? What if you have expandable menus? Figuring this out and saying, you know what? Our heat maps are telling us that the information that's most important on the product page is xyz. Do we take that XYZ information and provide it on the category pages as filters? Maybe sometimes provide it with, like, you know, with a little additional information on the collection pages. So information from the product pages helps us design our category and collection pages. Correct. Because again, we're trying to help consumers find what they do. If I'm running Macy's.com and I look at the PDPs, and I look at the heat maps for the PDPS, and I notice that people are always expanding the care instructions for the item. It's like, oh, there's maybe 15, 20% of our visitors are all expanding this. Maybe they're moving their mouse around this. That tells me, maybe I need to include that information on my collection pages. Now, sometimes people tell me that, well, yeah, okay, so we noticed that people are looking at that information on the product pages, but we don't have it. How do we provide it on the collection pages? One of the biggest projects that we've done when working with ebay was doing this kind of looking at the product pages and seeing, oh, what features matter? And we need to include them the collection pages. And that was a huge project for ebay. It was almost like a year, year and a half. Because, yes, we figured out the required information, but we didn't have it on the collection pages, on the category pages. And it was not easy because amacis.com has vendors. Correct. So they can talk to their vendors about getting that information. With ebay, the vendor is you and I, somebody who wants to list some items even a lot more challenging. Ebay has over like a couple million items getting listed, if not even more every day. How do we create that nomenclature where somebody listing an iPhone7 sitting and in New York and somebody sitting in Midland, Texas? How do we make sure that the data is consistent so we can provide that filtration on the category page? So that was a huge project. And after we implemented that project, we saw a huge uplift as well.

I like the idea, like the point that you make about alteration. Do you have, like, any thoughts on implementing dynamic content on category pages based on user behavior and intent? And if you do, how can AI and machine learning contribute to this process?

In all honesty, I think the. And maybe this is because I don't appreciate how much information is required on the collection pages. I am a big fan of using AI on the product pages and dynamic contents based on signals that the visitors sends you. You can actually use AI. You can use dynamic content to provide the right information to visitors. I think AI can be very powerful if you have the right AI tool to figure out the structure of the collection pages. Now, I'm Sitting in Chicago, you're sitting in Istanbul, Turkey. I bet you if we pick any random e commerce website, let's say an apparel website, and we go both to it, I will see the same category page that you will see. For the most part, personalization has not kept up with. I shop a lot differently than you, Correct. A lot differently than how I ad shops. And when it comes to apparel, I have seen some really amazing use of machine learning and AI to customize the collection pages. So the collection page that I see is a lot different than the collection page that you see. A lot different than the collection page that the third person and the particular AI software that I saw, what it did, it actually on the product page it gave you, using Instagram influencers, how this particular item could be, what goes together, what else you can wear it with. And it's fascinating. As people click on Instagram, they look at Instagram and then they come back to the site by looking at which images you've clicked at, the AI is able to figure out with up to 85% accuracy that this is the right design for the collection page for Khalid. This is the right design for the collection page for Simba. So through three clicks on particular influencers, oh, we can figure out the style that Simba enjoys and it can build this very accurate model. And it's also fascinating knowing that if you look at the population, 85% of the population shops this particular way and then there's 15% that shops completely differently. But you can be like, I would say within 85% accuracy. Now, this is not dynamic content per se, but this is kind of the layout of collection pages. And again, going back to eBay, eBay had implemented this and it's very powerful. I've seen also some stores in Europe, Germany in particular, that have also implemented that AI and it's been extremely powerful. Kind of like, you know, where it's not personalization, it's using AI and machine learning to do that personalization, as opposed to very simplistic personalization, which is really what you see for the most part when we talk about dynamic content or using personalization on e commerce sites.

Okay, do you have like any sort of KPIs that maybe you use or that you have seen other people using when it comes to maybe trying to measure the success of category page optimization.

So every time we run an experiment on a collection page, there will be several metrics that we track. We are always tracking the order confirmation, although we are just merely testing the collection page. But we want to See again, the job of the collection page is to guide you to the right product page and by presenting a new design for the collection page, I want to see if that design works well. You're going to select the right product and you're going to click on the add to cart and you're going to finish the checkout. So I'm always measuring as a main driver, main KPI, I am always measuring the order confirmation. Did this design generate more orders? That's number one. Then I like to measure the click through rates because again, the job for me, and this is maybe the direct metric that you can control at the category page is how many people am I able to get from the category page to the product page? Second metric. Now, third metric has nothing to do with the category page itself. But I do measure the clicks on the add to cart on the product page and I think that impacts which design of the collection page I'm going to go with. Some people say what does that have to do with the category page? Again, if the category page presents you with a whole bunch of products and you're opening multiple tabs and you're looking at those products and you, you don't know which product you want to click on the add to cart, the category page had failed. Yes, the product page failed, but the bigger failure was at the category page level. So I do measure the clicks on the add to cart as well. I'll probably measure also the different steps of the checkout, but the three primary ones, order confirmation, the click through from the category page to the product page and the clicks on the add to cart for the product page, those are kind of my three primary metrics. I probably look at starting the checkout, sorry, from cart to checkout and finishing the checkout as well. But the three prime metrics are the ones I just mentioned.

Okay, so basically I think those are the questions that I had for you today. Awesome. Unless maybe you want to add some stuff on temporary pages or anything to do like with CRO.

I think we talked quite a bit. One of the things, I've had an earlier conversation today with a company that we're working with for probably now. This is the fourth month that we're working together and I think during this period we've launched about 20 different experiments. Of those 20 experiments, I think six or seven of them were successful, 14 were not successful. So success rate is around 30%, 35%. And to me, success of a conversion rate optimization program is measured through two different ways. One is through the process and the questions that's. Are we asking the right questions? Are we, as a result of asking those right questions, are we learning more about the customers? And then I like to look at success, and I've emphasized that I like to look at success. And our job, as we work with every company, is to emphasize, are we learning about our customers? And that was the question that I had. I told him, we'll talk about the numbers in a little bit. But tell me, in the past four months, have you learned about your customers? How much did you learn about customers? They said, oh, we've learned many, many things about our customers. We've learned that they like XYZ and they absolutely hate this other thing. And I'm like, oh, wow. Like, you know, most of their marketing was focused on the fact that a person can come and do the shopping on their own. They've discovered through the A B testing that we've done is for the type of product that this company sells. People want assistance. It's not an easy purchase decision, and it's quite an expensive investment. It's a 1500 dollars investment right off the bat. And then you are buying multiple times, three to four times a year. So, you know, you're talking at the end of the year, somebody has spent seven to ten thousand dollars with you. Sort of makes sense. Correct. When I say it out loud, it's like, yeah, it's a major investment. It's not like buying a shirt. Correct. It's not like buying a polo for 20 or $30. No, this is a $10,000 investment for a year. So they need that guidance. And I told them, so what's the biggest challenge right now is, okay, so we recognize that people need that guidance. We also recognize that that guidance is expensive. Correct. We can't really, like, you know, we have 150,000 visitors coming to the sites. So how do we pick the right customer to give them the guidance? And how do we lower our cost when it comes to the guidance? So those are kind of interesting questions. I think that's what experimentation is about. You start with good questions, you end up with more interesting questions. And kind of you're going through this learning process.

I like the idea of, like, are we asking the right questions? So are those, like, the kind of questions that you usually, like, ask maybe on each CRO program or a deep side based on maybe the client project that you're working on?

No, I think every experiment, by the way, has questions. So there is the hypothesis, but then there's questions that you say, as a result of this experiment. Here's what we're trying to measure. Does introducing this reduce this? Does it increase that? There's five or six questions that you want to get answers to as a result of your experiments and if you're able to get answers to those questions, as you answer them, you add more, more questions. So that's how the process is. We're always asking more questions and coming up with different answers, so on and so forth. So it's not only the hypothesis but also the questions that are surround the hypothesis.

Okay, yeah, I think I get it makes a lot of sense and I hope that it makes a lot of sense for some of our listeners, for all of our listeners.

Sounds good.

Awesome.

Well, thank you everybody for listening. We would really appreciate if you leave us a positive review on wherever you listen to this podcast. And until next time, Simba, everyone happy testing.

Sure. Bye.