SEO Testing: How to Run Successful SEO Tests

Welcome to a new edition of Crawling Mondays. In today's edition, I have 2 people who are going to share with us a lot of insights, actionable steps to make your SEO tests meaningful, reliable, accurate, to start making better SEO decisions. I wanted to do this episode about SEO testing in general, and for that, I have invited Emily Potter, who's Head of Customer Success at SearchPilot. Hello, Emily. How are you?

Hi. Yeah, good. Thanks. Thanks for having me.

Amazing. And then on the other hand, I have Nick Swan, who's the founder at seotesting.com. Hello, Nick. How are you?

Hi, Aleyda. Yeah, I'm really good. Thank you. Thanks for having us on.

I am so very excited to have you both here and very looking forward to our conversations. Before starting, I just want to take the opportunity to thank To our sponsor today. ContentKing is a real-time SEO auditing tool that in case that you haven't yet tested, you totally need to take a look. You can find a link below this video. So thank you very much ContentKing for today's sponsorship. And now we're ready to start the conversation. I believe that we're all very used to A/B tests in usability, conversion optimization too. And then since a few years ago, we have seen the start more tools in the SEO world that try to replicate that type of functionality, right? Towards the SEO process. SEO A/B testing, split testing, time-based tests. So there's so many different ways to test and different types of tests. Based on your experience, what is the right test in each scenario? And what are the most common types of tests that you have seen in the market until now?

At SearchPilot, we have like Strictly enterprise customers really. So our experience is a little bit different from Nick's, but I think will be insightful for people with smaller websites. So for us, we kind of interchangeably use A/B testing and split testing. What we're doing, we are splitting up pages. The idea being that we're testing, unlike CRO testing, you're testing for users and you split them up 50/50 as they come. We're actually just testing for one user, which is Googlebot. So we can't split. Googlebot up, which is why we're splitting up pages. But to do that in a way where you get like reliable results, you need a lot of pages, you need a lot of traffic. So that's why Search Pilot is inherently limiting in that sense to enterprise. That's kind of the gold standard, I guess, of SEO testing, but it's not available to everyone. So there's definitely other things you can do.

From a starting point of view, I think we've been doing SEO testing ever since we've been doing any kind of optimization. We've always made changes to websites to try and get them to rank better. Previously, we looked at rank trackers. So we'd have a page, we'd have some content, we want to rank that page better for a particular keyword. We'd go and make changes to that page and we'd watch a rank tracker over a 2 or 3 week period. And hopefully our rank would go up. If it did, you know, that's great. If not, it goes down, we perhaps go and optimize it further. I think what's different now is that we've got more data available to us through Search Console, through the API. Emily and our companies and other businesses are doing the tooling's got a lot better around it as well. In terms of coming up with a more process-driven way of being able to make changes, collect the data, and analyze the results. So from a basic level, what we do from our tool, we have 2 types of tests. There's time-based testing and there's split testing. So with time-based testing, what we're doing is we're comparing one period against another. So before we've made a change compared to after the change was made. And this is more akin to what I was saying about the rank tracker. So this is like watching a rank tracker after you've made a change, but we're collecting all the data like clicks, impressions, position, and so on. And then there's the split testing, which our tool does and which Emily's tool specializes in, in terms of where across the same time period, we're looking at 2 groups of pages, one group of pages that we've made a change to compared to a group of pages that we've left the same and seeing how those 2 groups fare against each other. Hopefully the test group will go up and then you can compare against the control group to say it's those changes that have had that effect. So they're the kind of 2 main types of SEO tests that you can run, I think.

I think that is one of the characteristics of SEO since At the beginning, nothing was written or even pretty much documented. It was always about testing what works and what not. But I believe that the difference now is that the methodology and the framework is much more, let's say, formal, well-structured, to follow certain experimentation best practices that were not before, with a little bit of more sophistication. So what you mentioned, Emily, before, that you require a certain number of traffic to have statistically significant results?

So we usually say, yeah, like roughly 1,000 sessions a month to the group of pages that you're testing on. But you also want those pages to be templated. So you're thinking like ecommerce product pages or category pages, that sort of thing. And part of that is because like we are splitting up pages into control and variant, but what we're doing and what that like the newer technology allows us to do, we use a neural network model, which is a newer form of analytics is we're actually forecasting variant sessions. So you can never perfectly split up traffic in the same way you can perfectly split up users for a CRO test or close to 50/50. So that's why we use the control group for controlling, like with seasonal, is it seasonality, all that stuff. So that's what our control data is doing during the test. But ultimately, we're actually measuring against what, what the variant sessions would have done had we not made that change. It's that extra level of sophistication, but— Yeah. It requires those big sample sizes and lots of pages and all that sort of thing.

And then also what I also see is that before we tend to use a lot of tests, but it was like very time-based, pretty much. Like I changed the title or I tweak the heading in this way, or I add all of these extra links to this group of pages. And now we'll wait for a few weeks to see if it has improved or decreased. And based on that, like make a decision. But realistically, the issue with that is that There are always updates that we don't expect. It's not only up to us. Maybe our competitors have released something new or there is—

Algorithm updates.

Indeed, updates going on that can influence. And I can definitely see how it's much more difficult to identify the actual reason of the impact if it is time-based rather than an SUA/B test. So I wanted to ask you, what would you say are the typical mistakes, errors that you see people doing when developing tests? I can definitely see how there are still the typical challenges of not developing the correct hypothesis in some cases.

I think, well, it's related to what Emily was saying in terms of traffic. Well, and I'd say as well, in an ideal world, all testing would be split testing because it gives a lot better result. You know, like you say, you can reduce the impact of algorithm updates and seasonality and things like that on your results. So in an ideal world, we'd all be doing SEO split testing, but it's It's just not available to everybody. Not everybody's got enough traffic and so on. And so one of the issues that I see coming across with our tool when people are running split tests is that they're just trying to run tests with pages that don't have enough traffic or don't have any traffic at all. And in particular, we do an average, we do a daily difference between the 2 groups and we average out the test group and the control group. And we find people are putting like 1,000 URLs into each group. And lots of those URLs have got zero clicks on lots of days. And so that's heavily affecting the average. So you want to be picking pages that are getting traffic and decent amounts of traffic. And that makes people nervous because they're kind of saying, well, we don't want to be running tests on our pages that, you know, are converting and are getting traffic. But you're going to see a lot better results and decent results if you're testing on them, on URLs that are getting traffic.

Yeah, I mean, we've seen that problem too, actually, of customers like, well, we don't want to test on our like money-making pages. The good thing about Search Pilot is we're a server-side solution, so we can kind of pull things back really quickly. But yeah, we forecast variant sessions. So we're actually comparing aggregate sessions to aggregate sessions. So I think you don't— we get a lot of zero-click pages, but that's kind of accounted for in our methodology. So it's different. The thing that I see a lot with our enterprise customers is like not doing iterative testing. I mean, one thing is formulating a really strong SEO hypothesis. So that's kind of still something that I think a lot of people don't do. They formulate things that are actually more like user changes and like expecting it to impact SEO, but there's no clear way that would impact search. But then beyond that, yeah, iterative testing. So by that, I mean, you test something, it comes up with a result that surprises you, you go back to your hypothesis, and you're like, okay, we thought it would happen this way because of this. But this is what we saw. Maybe if we now test this thing, we can further learn something. And that's, I think, where some of the most valuable lessons in testing come from, is those follow-up experiments. And it takes some time to get customers kind of on that. idea because they want to just like test something and move on.

What would you say are the building blocks? Many things will highly depend on the context, of course, but which are the, the must-follow steps that should be there for a meaningful, good, reliable test in an SEO process?

I mean, step 1 is going to be your hypothesis regardless, and that's free. You don't need a tool to create a good SEO hypothesis. So that I think regardless of how you're testing, that needs to be the first thing that you're doing. I'm working on a blog post about this also, but one of the things we talk about is like, we don't think of SEO traditionally in this way, but actually if you drill it down, there's only 3 ways you can impact SEO traffic, which is improving your existing rankings, ranking for more keywords, or improving your organic click-through rates sort of independent of rankings. So that's things like getting rich results, that sort of stuff. So an SEO hypothesis should clearly influence one of those 3 levers or oftentimes a combination, but that's kind of like the ultimate foundation. From there, you know, you need to figure out how you're gonna measure things. Are you gonna split test it? If you're gonna do that, how are you gonna bucket your pages? And then you need to come up with a methodology for measuring it.

Same as Emily said, you know, start with a hypothesis, be descriptive of that hypothesis as well. You know, the more detail you put in when you come back and look on it in 6 weeks' time or whatever. I don't remember what I was up to last week sometimes in terms of SEO and what I've been changing. So yeah, the more detail you give, the better. And then be brave when selecting URLs. I was going to ask Emily actually from her experience, like how do you— the patience when tests are running and so on? Because one of the issues and, you know, common mistakes is making other changes while the test is running. Like how do you manage the expectations and patience of the people that you're running tests for?

That is one of the things I had written down before as like a mistake is yeah, like for us commonly what happens is we're at big enterprise companies. They have a product team that are running CRO tests on the same pages and they don't communicate. And then you find find out, yeah, something might've made that test invalid for that reason. So a big part of— that's a bigger and bigger challenge that we face. So really finding a way to get our points of contact to have a point of contact on the product team where they can coordinate is a big thing. But yeah, it's hard also just getting people to run tests for long enough. The lower traffic you have, so a really high traffic site section, I'm talking like 80,000 sessions a day sort of thing, those you can get a result in like a week really quickly. When you start dropping that closer to that 1,000 a day threshold, or sometimes we can model things lower, but it gets less and less reliable. You're looking sometimes at like a 6-week runtime. So that part is about setting expectations from the start, not just like early on in the engagement, but in the sales process and figuring out, you know, okay, this is a challenge. We need you to address this and just kind of making sure that that's spoken about at the beginning. But it is a challenge and each of our customers, I think, handles it differently. And this is where like a lot of project management tools come in really handy. Like a lot of our customers use Jira or Monday, like different ways to coordinate with the other teams.

Especially with the duration. If you have enough traffic and you can run it just for, let's say, a couple of weeks, then it's okay too for you to not touch anything on these pages during this couple of weeks. But if it takes a lot more than that, then especially if these are important pages in your customer journey, right? You need to be always updating, always changing things. What are the things that you have tested that you have been really shocked that have worked? Because there are certain types of things, for example, that I have tested in the past that I completely believe, like adding the numbers, especially high inventory websites showing that yes, they have dozens of thousands of items or products available. that I expected to have an impact on clicks and they did. There are other types of changes that you don't expect as much to have an impact and they did. And I can definitely see how, again, it's important to take this with a grain of salt because it will highly depend on that particular website. Which are those, like, let's say, counterintuitive results that you have gotten or you have seen in the past?

One of my favorite, like, surprising test results that we've had was we had an ecommerce customer. We ran a test early on in their engagement with like implementing breadcrumb schema for them. And that was positive. And then maybe a year or so later, there was like some sort of encoding error that happened in the schema markup that broke. So we ran a test to fix it. I can't even remember exactly why we decided to test that instead of just fixing it, but we did. And then the test was negative and we were like, what happened? And what had actually happened is Google now, if you have a clean URL structure, Google will create breadcrumb for you. And actually, when we looked back at the test results, their URLs were more descriptive of what the page was about than the actual breadcrumbs were. So I think there's a lot of things where you don't realize changes that Google has made, how that's going to impact things that maybe previously were like no-brainers. So those things tend to be surprising a lot. I mean, Google overwriting title tags, for example, we're doing a lot of tests now changing headings that is positive that weren't before. So I think anything that kind of touches on Google changes. tends to be interesting.

I did some analysis a little while ago of— this is for time-based tests, so the results may differ for the split testing side of things. So I was kind of interested how good the people were at SEO that were using SEO testing. So I collected the data for all the time-based tests that had completed running on the platform. And from all the tests, 18% of them had no meaningful change. So there's no real difference in the clicks up or down. 34% of them had a decrease in clicks. So they actually dropped in traffic. And so about 40% of tests had an increase in traffic. And so SEO testing is predominantly used by people, I would say SEO professionals. So a lot of people who spend most of their time thinking and doing SEO and reading and reading about SEO. And so there's about a 50/50 chance of the changes that these SEO professionals, people are making, leading to an increase in clicks. So that's like a toss of a coin really, isn't it? And that's with all the best practices, all the case studies that we've got. So it kind of shows how important it is to track these things and track the results. Because as Emily has said, you know, one thing that can work one year for a site, you go back another year and it kind of all changed. One thing that works for one niche might not work for another, even down to keywords, you know, and search intent. One thing that works for one particular keyword, the rankings could be completely different how they calculate them for different keywords. So, and this again is where the tooling helps a lot. I'm guilty. I used to do the testing in terms of just trying to track it all in a spreadsheet. And I'd often forget to, you know, mark down the changes that I'd done. So going back in 12 months' time, trying to remember what changes and things you made to a site and stuff is difficult. And I say these are my own sites rather than client sites. But by having an SEO testing process and creating tests for all the changes you do, content optimizations, anything even on single pages, this is where the time-based testing comes in use. You can then go back and have a look over your changelog on your site, and it will tell you what you did last year. It will tell you what you did 6 months ago, and you can just click on the results button. and it brings it up straight away so that you can see. So being able to create that changelog is kind of one of the good outcomes from doing all the SEO testing. And as people come in and as consultants come in and out of businesses and so on, you know, they can go back and look on that as well. They can see exactly what has been changed and done to a site. That information isn't lost as people come in and go.

Building on what Nick said, I think we have a— we say like most SEOs are worse than a coin flip at predicting because on top of like 50/50 will be statistically significant. I think we have an even higher percentage of inconclusive test results. But of those percentage that are positive, like how many of those were expected to be positive? Like there's another layer to that where you get surprising results all the time. Like we're doing those Twitter polls now before we release case studies. And more often than not, people got it wrong.

Realistically though, since there's so many factors and the context can vary so much, even within the same website, right? I can definitely see how a hypothesis is working well for an example, commercially focused queries of listing pages, than when you come up with the same hypothesis for a very top of the funnel guide type of pillar pages that will be ranking can vary so much. It's also because not only the type of features that can be shown in search results, but also who you are competing against too. So—

Google shows FAQ schema now in some industries and not others. Like we're having tests where we can't get FAQ schema to show up and they used be our FAQ snippets from schema. But I definitely don't say that to be like, SEOs are bad at their job. It's more like, this is why testing should be part of your arsenal because even us, we will tell customers like, oh, you should test this because it was positive here and it's not. So it's just, you can't ever possibly know all those different things.

Yeah. I was talking to some CRO split testing people and I said about the 50/50 thing and they were quite actually impressed with that. They're like, if we could get anywhere near 50/50, then we'd be quite happy.

At one point, I feel like it was definitely over 50% were inconclusive.

Interesting. We mentioned also how there are so many more tools that people can use. And of course, SearchPilot, seotesting.com. We have also, I think, all the larger enterprise-level platforms already supporting also some sort of SEO testing. So Vryde, I know that supports it. Now, we have also other tools like SplitSignal from Semrush and Zippy from Cyrus Shepard that sort of does this at the metadata level too. So you have plenty of options to go ahead and start trying new tests taking into consideration all of the tips that have been shared today by Emily and Nick. And before wrapping up, I wanted to ask you for a last tip. Something important that you want the audience to take into consideration? Something that potentially we haven't mentioned yet in our conversation that you think is fundamental, that is critical?

Maybe something we haven't talked about is what SEO testing can do for you internally, particularly for people in-house. Like the business case that you get from SEO test results is one of the most valuable things. Like one of the things, if nothing else, like we've known things that work as SEOs for a very long time. It's very new that we're able to put a monetary amount to changes that we're making. So that's one huge benefit of doing it. And like, if you can't like actually test yourself, we release all our case studies. We've gotten feedback from that of people that are using our case studies even to make business cases internally. So there's lots of people sharing stuff now out there on results, even if you can't run them yourself. But I'd say, yeah, the biggest thing is just even if it's not a surprising test result, you now have Data that you can use and say, get this implemented to your devs or engineers. And that's incredibly valuable.

I am with you there, especially because it allows us to say, it depends, and we can run this test that will show us the best way in this particular context. So it gives us this tool in order to eliminate uncertainty and also to show the value of our work, eliminating this layer of complexity that there has been always there. And before, we didn't have all the resources that we needed to simplify them and actually validate what was the best way for the business in a very straightforward way that was also easier to understand for decision makers and non-technical people in the organization, for sure. Also, I do believe that at some point, maybe in the past, the fact that we couldn't give a straightforward answer without having a way to back it up, right? Like now we can back it up with data. And even if it is something that, well, that is counterintuitive or not the best practice, but at the end of the day, this is the beauty of our work, right? If it was just a matter of a best practice, there wouldn't be a need of an actual specialist and SEO advising for that particular scenario at the end of the day.

Well, just to touch on the business case as well. So my background is in development, software development, as well as SEO. And so I know one of the bottlenecks of getting changes rolled out to be able to do split testing And so on is developers. So if you're relying on developers to code up stuff to switch titles and things like that, it can be a bottleneck. But the good thing about all the tooling that's coming out now is that these solutions can manage those changes for you. So Search Pilot, Split Signal, Cyrus's tool as well—they can manage the. So from a marketing point of view, you don't need to get involved with the developers quite so much. You can run and roll out your testing and collect the results and so on. And then if you get the good result, then you can get the developers in, you know, to fix up the solution, do whatever you want to do from a long-term point of view. So there's certainly less barriers to getting things done and being able to experiment and run tests quickly, which has always held people back before. From a tip, from a really basic point of view, you know, you don't need to pay any money. You can just be using Google Search Console. So within Search Console, you can select the page that you want, you've been testing, or you are testing from a time-based test point of view. You can select the URL. You can configure the dates that you want to be running the test to and from. And that will add all the data for that particular filtering into the query string, which you can then copy that URL out. Put it into a spreadsheet along with your hypothesis and what you've changed on that particular URL, so that you can just click through and reuse that URL each time from within a spreadsheet to be able to quickly get to to the data you want to be able to see. So you know there doesn't have to be a big cost layout to get started with SEO testing. As we kind of we both all said, we've all we're all doing it anyway in terms of trying to make our rankings better. So we might as well make use of the data that's out there and build it into part of our process. And what's useful as well if If you've been making changes previously with Search Console as well, and with SEO testing, you can backtest as well. So if you made a change 6 months ago, you want to check up and see how the data worked out, you can go in and plug the date change to 6 months ago and it will pull the data for you, as will Search Console as well. So just get started. You're doing testing anyway. Let's just make use of better data, build it into part of our process, and have— be able to, as we've all said, you know, be able to build better business cases for the work we're doing. and the impact we're having on businesses.

That's amazing advice indeed. And something that you mentioned, and it's completely true, like a lot of the complexity, especially for SEO A/B tests, right, is development support and the technical complexity of doing it. So, but I have to say, if you're willing a little bit to experiment yourself a bit, right, if you don't have the development resources or the budget yet to hire a tool to implement it, I believe it was a Chris Green presented last year in Brighton SEO how to run your own SEO A/B test using Google Tag Manager. Then we can start having a conversation of how reliable and how we can be completely sure that Google will render the—

Yeah.

—is implemented through the Tag Manager every time. But there's that, right? That's definitely an option. We can start developing and building a case for testing within our organization to move forward with tools for which we will require actual budget, right?

Yeah, we can talk about server-side. A lot of the other tools are doing server-side solutions still, which aren't as good for SEO testing. But another thing Chris Green has talked about more recently that's coming out is like edge technology. SearchPilot is technically an edge technology, but also, I mean, if you are more technical, you can get them— if you have Cloudflare or Akamai, you can get enabled to start doing that stuff. And that means you're doing server-side testing. So that's another step above.

100%. You can leverage now Cloudflare Workers to implement this type of test too. So hopefully we will see many more solutions like this that will be much more easy to integrate and start running with any website using a popular CDN. So thank you. Thank you very much for your time, Emily and Nick. I really appreciate it and all the insight that you have provided today. If you have any questions, if you have any doubts, you can see how savvy Emily and Nick are about this topic, follow them over Twitter. You have the handles here and ask them any questions. They're super nice. I'm sure that they will be able to help if you need anything. And if you like this episode, of course you need to like the video, like the video, subscribe to the channel, and make sure to pay attention to the next one that will be as good as this one. So thank you very much for your time and see you in the next Trolley Monday. Bye. Bye-bye.