SEO Auditing Large Websites

Welcome to a new edition of Crawling Mondays. Today's edition is about SEO auditing large websites. Is it the same to do an SEO audit for a small versus a big website? Not necessarily. And to cover this topic, I have invited 2 people who have tons of experience with not only small, medium, but huge websites too, who can share a lot of insights about it. So I'm I have on one hand Jamie Indigo, technical SEO consultant slash GPT-4 hallucination based on what she told me. Hello, Jamie. How are you?

Good morning. Well, good afternoon.

So excited to chat with you as always. Very excited too. And on the other hand, I have Patrick Stocks, who has a very exotic position, product advisor and technical SEO at Ahrefs. Hello, Patrick. How are you?

Hey, doing great. Excited to be here.

Very excited too. And before starting our conversation, I need to thanks to our today's sponsor, who's ContentKing. ContentKing is that SEO alerting tool that will be able to tell you when something goes to hell, even on a big website. They are able to crawl in real time to identify SEO issues like if Canonical tags were changed without you knowing, title tags, internal links, whatever. Go and take a look at content team to avoid SEO challenges that are completely unnecessary. And let's go back to our topic today. SEO auditing for bigger websites, right? It's interesting because I am more used nowadays to do SEO auditing for larger, especially websites like e-commerce, marketplaces. These are Websites that tend to have many, many URLs. So I want to start with that. What differentiates doing an SEO audit for a large versus a small or medium website, right? The first thing, just to spark conversation here, is that I remember many years ago I was mentoring a person and she only had had experience with smaller websites. And the first thing she told me is like, oh, this website that I'm auditing now is like, is an e-commerce that has a million URLs and it's like, how can I crawl it all? And I'm like, you don't need to crawl it all. So you will tend to use, yes, SaaS-based tools and you don't need to leave your computer running the whole time or Screaming Frog, that's more targeted to a smaller website and try to crawl every single URL, but with rules and segmentation, you can configure it to identify issues at different levels, different type of pages following patterns, right? So that then it made me realize that we don't tend to think many times at the difference.

Well, first off, you can make a lovely slow roasted pot roast if you leave a crawl of a million pages on Screaming Frog going for a nice 8 hours, you know, just check the temp before you serve it. I personally think the big gap here is when we get to these much larger sites We get further and further away between what we think exists and what actually exists. We tend to be dealing with a lot of dynamically generated content with PIM systems. Even the identification of all the assets becomes a little bit more tricky.

I think you just don't know what's out there on these large sites. You know, you can ask a company how many pages you have and they might say like 70,000 and you go look and like there's 370 million. It's like, what? Macro view and like dig in. Some people will use site queries. That doesn't really work with large websites. GSC will have good data. Third-party tools will have data to show you things like that are indexed that may, they may not even know about.

Mm-hmm.

Patterns, you nailed it. Like break it down either by section, by what you're working on there. It may even be you brought in specifically for a certain section because they suspect an issue or. They have trouble with area or they just might want like top X pages even looked at. You never know what this—

Oh, and different teams work on them.

Yeah.

Like there can be entirely different teams who, you know, are warring factions in the coffee break room every day. Don't communicate.

I think that has to do that it correlates that bigger sites tend to be tied to bigger organizations. So this additional layer or aspect of doing an audit comes to you, that communication coordination. Sometimes you don't have even access necessarily because of NDAs and restrictions to many data sources that sometimes you take for granted when you are doing an audit for a startup or a small website, right? I think you're on, onto something important here. Yeah.

I worked for a major e-commerce site. We had at least 4 million products at any given time, but the exact number that were actually in stock Getting that source of truth. No one had that.

This is when actually something that we tend to hear about a lot becomes important. Crawl budget. Right? Where Googlebot's actual resources and efforts go. Is it wasting their time going through a lot of URLs that don't make any impact that you are generating by default, or is it actually going through the right information that matters to you to rank, to, to get traffic to convert with.

Yeah, filters, parameters, variations suddenly become a lot more meaningful when, you know, you have, you've got one parameter and it has 5 different values for it. Okay, you've made 5 variations. What if you have 10 of those and each have 5 and you've got 5 million products? Suddenly we're getting a lot more complicated. Those little things that we kind of take for granted breed like bunnies.

A question that I should have already asked before. What would you say that is a small and what you will consider already a big website and will start treating as a big website? What is the threshold there? Like a million, 5 million URLs, hundreds of thousands URLs?

I don't—

What would you say?

Have a personal scale for that. I would say it's more to deal with complexity. I mean, you've got. You know, maybe 3 patterns on a site and 5 million pages, that's an easy audit. You've got 15 different CMS systems and 500 different patterns of pages, different templates, then that becomes super complex.

Oh, is it doing reverse lookups? Does everything respond, or does it only respond sometimes based on which CMS you're querying through? That's where you can get into some Fun, deep rabbit holes.

I think that you mentioned something important, Patrick, regarding that beyond sites is actually about the complexity of it. That let's say your knowledge and, and certain type of tools, you could easily or try to easily gather the data from the website to analyze and recommend on when you're dealing with a website that indeed has so many facets. filters, so many URLs to handle, and then challenges to gather all of the information, putting together a puzzle many times in order to have the necessary and the complete vision to recommend and to prioritize a recommendation that actually makes sense based on all of these challenges because of the complexity of the web structure.

Corporate stuff too. Like you're going to run into things where some of those systems, like they're just legacy. They're not going to fix them. It might be years before they sunset. Like there's no point in making a recommendation. And as an outsider, you may not know that, but like they might be like, well, that's off limits. Like they're not gonna spend any resources or any time fixing this. We're gonna get rid of it in like 2027.

Don't delete the JPEG of a coconut cuz everything breaks. That's what you end up running into when we have a lot of these really large, cuz one of the largest sites out there are built on legacy code. If anybody's trying to pitch to you. Optimizing crawl budget and you're 1,000 pages, you need to end that call early. Something came up. A better use of your time.

100%. So what would you say are the steps that you think that should be there? And I understand that context matters. It depends as usual in SEO, right? But what would you say that are the not-to-be-missed steps that you tend to follow whenever you're doing an audit for large websites in order to maximize its impact?

Well, I have thoughts and opinions. My favorite game is let's play find the patterns because I can't audit everything and analyze it equally when they behave very differently. If you have an opportunity to go sit down with a QA engineer, they're my favorite oracles for this pre-analysis portion.

I'm just gonna try and focus on generally where the biggest impact is, or if people come to me, it's usually cuz they have a certain problem with a certain part that they just can't figure out. So like that's, that's an easy place to start work on what the people that are there want you to like figure out and work on. You know, it might be a certain BU that they're like, we need this section. So that can help you like figure out, okay, I need to focus here. But large sites, especially large companies, there's always going to be certain things. Fixing redirects is almost always one. And not just on the site. They've done migrations over the years. They've done mergers and acquisitions. They've killed off a ton of content. They like didn't do the redirects or they did like overly broad redirects where everything from the site that had 5 million visits a month now just goes to the homepage. There's going to be stuff like that.

I think that what I try to do is to try to gather as much data as possible from different data sources because of what you mentioned before. It's like a treasure trove that you always end up finding more URLs because somehow like when you like 3 different SaaS-based crawlers and through Google Search Console. And I tried to register Google Search Console profiles for each main category to try to gather even more in-depth data as much as possible. And you end up even like digging more, even like Ahrefs and Semrush and Sistrix and like give me all the link pages that have 404s with backlinks too. And then I end up somehow getting even more than the ones that I had initially pulled with a single tool. So I think that in this case, in this particular scenario, I think that having many data sources will pay off much more than with smaller website audits, right? Also, it is possible ideally to have access to web server log files in order to be able to see their real behavior. We know how hard That is many times too, but yeah, ideal. A lot of CDNs will provide it by default and DevOps will have access to it. So it's about even give access and configure a little report and filtering to be able to see the Googlebot and search bots. And that's pretty much it. Not build the whole thing just for us.

Yeah. You don't need the 5 billion row spreadsheet, but having some aggregate reports to just show you like, hey, you know. What are my, where are my top 404s at? What are my, what's my crawl by first and secondary subfolders? What URLs being, are being hit the most? And sometimes I can point you towards crawl waste because perhaps it's an internal analytics tracking tool, but it's being requested 5 billion times a day and you have a very large site as an opportunity to zhuzh up some crawl space. So be it.

So speaking about platforms and tools, what would you say are your to-go or favorite tools? that you wish that in every single project that you work with, you will have access to. Many, many times it happens. Many, many other times, you don't, for certain circumstances too. I have been also many times, unfortunately, especially with big websites, like, oh, we need to enable you crawl access through this URL and you need to configure your hosts files in order to be able to do it. Things like that, right? Because of how restrictive it can be.

I mean, what am I going to say? Honestly, like we're—

Patrick, please.

Please, beyond the obvious.

Of course, beyond the obvious.

Our audit, like, I work on that. Like, people underestimate it. Like, it's not like most all-in-ones that are simple. Like, it's flexible. You set your own issues. You have all these like custom search things. I'd say we do hreflang better than any other tool out there because no tool did it well, in my opinion. So I built what I needed for that, basically. Besides that, GSC in the past, I've used, I like Content King. Their, their real-time crawling is great. Catches things that like scheduled crawls won't. Screaming Frog, DeepCrawler, Blumar. Like I've used everything. Like they're, all the tools are typically useful for one thing or another, you know, business cases or something. Botify is pretty good on that.

I have to say something, right? With Screaming Frog, it's true. Like we tend to think about it because it's a desktop-based tool. Yes, of course, more websites. And of course it's very like the basic, most obvious use case is that, but it can also be quite handy even with larger websites when a release is done and you set a SaaS-based crawl. It will take a while to finish. You can do a very quick validation with a list crawl with Screaming Frog for redirects, for example, things like that. A very quick validation like that. And also like, I think that there's this tool that I, that I love, Cybo, because of the blend of big versus small that it provides. It allows you also to run on the cloud if you want. It's not only desktop-based and has all of these features and filtering that will allow you to quite easily gather the data that you need. That makes it very powerful.

Oh, I love me a good crawler. I love having an enterprise crawler, but I also want one that gives more detailed information. So Having a price crawler to go and do that large scale, having something like Screaming Frog where I'm getting the full stack trace and getting all those really juicy details that would fry or cost exorbitant amounts of money to do at a large scale. Absolutely adorable. We need that GSC access. That is our source of truth. I always want to find out what my client's source of truth is and use that as a basis when we're doing reporting, we're doing audits and analysis. We set KPIs, we set the sources of truth as we start off that way. We're not moving goalposts on anyone. There's good expectations when it comes to, uh, modular nature of a really good crawler and, you know, being able to go up and down and do what you want to do. If you haven't checked out Chris Johnson's Headless Horseman crawler yet, go ahead and give that one a peek. Really a big fan of that. Very customizable.

That is a great heads up too. What would you say if I asked you, like, if someone starts doing an SEO audit for a large website? for the first time, what is the top one error that you will tell this person? Like, don't do this, please. Like, so many ways, these tools that we just mentioned, all of the steps and criteria, but don't fall into this error or this mistake that will cost you a lot of time and resources.

Checklists are really like out the window. Like, they're, they're for you, but if you were like give them a checklist audit, it's like you need to rewrite 80 million meta descriptions. You're fired. Like, they're not going to change their URLs. Like, so much of the stuff on these checklists that you're like, oh, I should point this out. No, like, you really shouldn't. There's just no way they're going to do it. So it's really like, you, you really have to prioritize the things that are going to have an impact. Like, pick your top 5 or 10 things. Like, that's really what you need to be giving them. Like, what is, what is going to help them the most, whether that's with like traffic or business value. It's just the, the checklist, the, the like 180-page audits, none of that is, is going to solve anything, right? You need better H2s. Yeah.

I will say that is also good advice for smaller websites, but it's especially true for bigger ones because you will tend to find many more errors across many more URLs, so it can easily become unsustainable, right? So prioritization here, knowing what actually matters and And summarize very well in an actionable way that on one hand, decision makers care about. And on the other hand, stakeholders can execute in an easy, straightforward way. These things are fundamental for sure in this particular context.

So my biggest advice, if you're moving towards auditing, you know, very large sites, there are a million and one tools that have already told them something is broken. Your value prop comes with, and here's how you fix it. That doesn't have to be a full user story. If you're well enough versed to write up that as blank, I want to blank in order to goal, fantastic. But giving acceptance criteria, you know, we all want to do good work, but what does good mean? Define that. Provide the resources. Be part of that process. Everything out there is like, for just $12.95, I'll tell you your site is a complete dumpster fire.

Cool.

No one wants that. What can you bring here?

Yeah, everything's on fire like it has been for years. It's not going to be fixed tomorrow. A lot of the stuff may be years before it's actually fixed or just never fixed.

I will say that in this particular context too with large websites is that strategical SEO becomes much more important. Like with very small websites, you can get away implementing different set of tactics that are not necessarily even well connected, like a lot of low-hanging fruits. But in this case, you really need to understand the why and to have a very clear business marketing related goal in order to like connect the dots of your actions so they actually make an impact and they are actually meaningful to make it matter. So 100% that you need to eliminate all of the noise, be very strategical, and recommend and follow up to maximize the execution of what actually matters and forget about the rest. Well, this has been a wonderful conversation already, but to start wrapping up, I want to ask you a last tip in case there's something that you think that you haven't mentioned before that is worthy to share with the audience before finishing.

One, listen to like the in-house team. If you're working with someone there, like they're typically going to know what they want to work on. They've probably said it a million times already. They just need someone outside to say, it also to like sell the project. And I would say like, if you're in-house, just look bigger and like look beyond one audit. Like if the same issues keep popping up, figure out why, like root cause that, or put in like automated checks before, you know, you can actually launch it live. That way, like it stops it from happening again. Any, any kind of like future-proofing you can do or You know, like if you're gonna change all the URLs on the site or whatever, or in a CMS, make sure you're gonna keep that structure and not change it again a year from now, 2 years from now. You have to start thinking about that kind of stuff because the more things change, like the more, the more stuff is just gonna break.

Absolutely. And that's such a good point. Half the time when you're brought in as a consultant for a large site, you're doing these, you know, enterprise-style audits. Most of your job is repeating what the in-person house has said. And that's okay. You know, you've made some allies, you've helped reinforce it as a good strategy, and that's beautiful. My personal tip, because I do love the technical flavor of SEO, is when you are auditing these large sites, if you see issues that have a tendency to exist that you cannot reproduce— oftentimes when you're at that large of a scale, you can have multiple load balancers, server clusters, CDNs— be curious, see which servers you're hitting when you get the issues. Because oftentimes, depending on the replication method, so how code is pushed, if it's pushed to cluster 1 and then cluster 1 copies to 2 and then 2 to 3, starts Xeroxing and we get little mistakes in between.

Amazing. 2 things that we mentioned quickly, but I think that I would like to highlight because of how important it is in this particular case. On one hand, patterns. You definitely cannot implement changes one by one. You need to establish patterns, easy-to-follow patterns, patterns that actually make sense across different types of pages, use cases, targets, etc. So it's important that on one hand you're able to identify them well, on the other hand you communicate with teams that run the website, the developers, the product people, so these are integrated as functionalities that can be easily implemented on the website and are not as hard next time. Secondly, I will say filtering too. Knowing how to filter well the different type of URLs, the structure of the website to gather the data that you actually need is fundamental, is critical. And finally, I will say that being able to prioritize well on what actually matters, not only because you think it's important from an SEO standpoint, but what is actually aligned with other areas, the product area, the development area, the content team, the UX team that will tend to exist across this type of organization. Is fundamental to move things forward because as both Patrick and Jamie mentioned before, many times they are very well aware that these problems exist, but it's just a matter of prioritization and how critical and how impactful they think they are for their own areas and for their own goals. So aligning them as well to what actually matters to them and the organization is fundamental for sure. So thank you. Thank you very much for your time, Jamie and Patrick. It has been a wonderful conversation, a fantastic talk. And of course, if you have any questions, any doubts regarding auditing big websites, just let us know, let Patrick and Jamie know. And they are very active on social, especially on Twitter. I will tag them in case there are any questions here on the video. Thank you very much for watching. If you liked the video, like the video. If you liked the channel, Subscribe to the channel so next video that comes, you don't miss it. Bye-bye.