Ethan Smith explains how to test SEO changes on a large marketplace. Use random buckets stratified by traffic level, run title-tag tests for 1-2 weeks, and check individual URLs for outliers. Zero-traffic pages need their own bucket since they behave differently.
Questions this video answers
Use split‑test experiments with randomly selected buckets, but first clean the buckets by removing or separating zero‑traffic URLs. Stratify buckets by traffic level (high, medium, low) or by city tier to capture different effects. After running the test, examine both aggregate results and individual URL performance to detect outliers.
No. Instead of discarding them, place zero‑traffic pages into a separate experimental bucket so they can be tested independently, acknowledging that results from high‑traffic pages may not apply to them.
Title‑tag changes: 1‑2 weeks. Internal‑link adjustments: about 1 month. Content‑block or major content changes: 1‑3 months. Adding new pages (e.g., 10,000 new URLs): roughly 1 month.
Chapters
Hey, Dimitri, do you want to come on?
Thanks, Mike. I have a question for you about the best way to quantify how well, like, a set of SEO optimizations is performing for the marketplace. And if the answer is experiments, can you talk a little bit about how you would set up the experiment? Would you randomize at kind of these canonical URL pages? Would you do a geographic experiment where you optimize all the pages in a single, like, say Cleveland? And leave like the rest of the country alone? And how long do these typically experiments take to run?
Yeah, so I think Thumbtack would be a good example. Like, how do we add— adding a content block? Thumbtack has 100,000 pages, plumbers in San Francisco variants, and we want to add a content block and we want to see what happens. Is it— is that what you mean? Cool. Yeah, so some things— oh, okay, so title tags. For that, I would just randomly select buckets. However, the common mistake with this is to include URLs with no traffic in your buckets. So let's say you have 100,000 pages and you split test buckets of 10,000, but each of your buckets, there's actually only 10 pages in your 5,000 URLs that are getting the traffic. So you have all these noise URLs. And then let's say that one page in your bucket dramatically increased. And then you say, okay, well, the, you know, it's very statistically significant, but it actually, if you look at the URL level, it's one URL that went up. So I would, for a, for a thumbtack, you know, the earlier you are, the more that you have this issue. So I would manually go in and probably like randomly select buckets, but sort of clean them up, remove the URLs that have Or, you know, maybe do like high traffic, medium traffic, low traffic. We did this for a company where we did popular city, popular cities, LA, New York, mid-tier and lower tier, because there might be some different effects. So we bucketed the different cities. We only selected URLs that had traffic. And then we tested title tags, let's say. And then We looked at the results, but not only the totals, like the traffic went up in this bucket more overall, but we looked at the individual URLs to see if there were one or two that were skewing the whole bucket. And then you'll have cleaning data. So that would be a split test. And that is, so for title tags, you get results in 1 to 2 weeks, which is short versus something like adding a content block, which actually takes a longer time. So that I would do 1 to 3 months. Something like internal links would be like a month. So it's these 3 different types, which is short-term wins. Short-term wins would be typically title tags or like rich snippets. Medium is going to be things like internal links. Longer term is going to be content changes. And then the 4th bucket would be I'm adding new pages. Like I'm going to add 10,000 new URLs. In buckets, that would be like a month or so.
Your suggestion to remove the zero traffic pages, isn't that problematic in a sense that the lessons that you learn from your high traffic pages may not apply to those zero traffic pages? And so you don't know, like, if it's worth optimizing those at all, or if you just sort of abandon and never come back to them.
Yeah. And for the zero traffic pages, I said you should remove them. I, I, I'll modify my answer. I don't think you should remove them. You should just separate them into a separate bucket because they might actually get traffic. eventually. So I would test the zero traffic pages and for sure the changes for the high traffic ones might not carry over to the low traffic ones. So I would test those low traffic ones as well. And for the low traffic ones, you know, they're low traffic either because there's zero search volume and you could never perform, or it's that there's a lot of search volume and you could perform. So then you'd wanna test how do I get from position 80 to 15 to 5, but separate that into a separate experiment so it's not noise within the, you know, the main bucket.
