How to Prune your Website Content in an SEO Process

Crawling Mondays. In today's edition of Crawling Mondays, I am going to focus on a very, very important topic, especially when doing an SEO process for big websites, all websites too: content pruning. Content pruning in SEO, well, you will see that this is a question that is asked again and again and actually I got the idea for this topic from Cyrus Shepard. Heads up to Cyrus and if you're still not following him, you should definitely do it over Twitter. He's always sharing really, really good SEO stuff. He did this poll. When pruning old content, is it better to noindex/remove, 301 redirect to a relevant better page? Assume that the content cannot be refreshed, improved. We should say, okay, if it is old content that at some point we wrote, should it be worth it to update, right? Well, not necessarily. Maybe it was very, very long ago. Maybe we have another page that is very, very similar targeting the same queries that might be much more worthy to keep. So there are so many different criteria and factors here that the usual answer is it depends and with everything with SEO. And here my answer is, if you see here, there were many SEOs who replied, if that old content is not driving value, traffic, ranking, links, and it's not worthy to improve update, not that relevant, not good potential topic, remove. But if driving traffic, rankings, links, or could drive value if improved, then keep improve update or redirect to a new version. Realistically, a content pruning process is something that will usually drive value to improve the perception from a content quality, content relevance perspective of a website. Also end up positively impacting the website crawl budget in case we have a very, very large old website that Google spends a lot of time crawling pages that are not really meant to be necessarily ranked anymore. It will end up having many, many effects in our SEO process. So today I would like to share with you a little flowchart that I actually did to try to simplify all of this if-thens and it depends, facilitate for you to go through the different type of scenarios and outcomes that this question usually has. So hopefully it will be highly valuable for you, as I expect it is. And I will be actually sharing this flowchart also over my blog, so you can, you can share it, you can save it. And I will be also linking to blog posts from the video description too, so it's far easier for you to get it too. So don't worry. And I will also be sharing how to actually go through all of this steps that I share here with the help of a few tools, gathering the data from all of these metrics that we are going to use as criteria by using certain crawlers. In this case, for example, I will be using DeepCrawl and Screaming Frog too, and using many other metrics that we will be gathering to do the best possible content assessment to identify if we should prune or optimize a page from our websites? Let's take a look. Let's start replying to questions and going through the flowchart. Should you prune or optimize a page from your website? This is the first question. Is the content driving SEO, organic traffic, ranking links, social referral, or overall user engagement value, right? So this is the first question that we should answer. From all of these pages, how many of this— like, which are those actually delivering any value, not only from an SEO perspective because they are ranking, getting traffic, or attracting links, but that have an important role delivering any type of value from a user perspective. If they are not driving value, then we go to the next step. Is the content of that useful, relevant topic with search potential worthy to be improved? Maybe they are not driving any value right now, but is it a very worthy topic? It has a Good search potential. If not, again, if the topic is not good, somehow we ended up publishing something that is not necessarily that well relevant at this point. Does the content have a business, customer, legal related goal that requires it to be kept in the site? Like for example, the legal notice and or the privacy conditions. If not, right, no, I don't even know why I have this content here because it has nothing to do. Not potential topic, but not legal requirement, nothing to do with the business or legal side, operational side of the website. So in this case, yes, we can eliminate the page. We will show 410 HTTP status and delete any internal links going towards it and take it out also from the XML sitemaps because this content is just consuming our crawl budget. On the other hand, we may Have found that instead of replying with a no here, we reply with a yes. Yes, it is content, a legal notice that maybe it is not necessarily as specific from a business perspective, etc., but it's important from a legal criteria. The next question will be, is it required that the page content is findable to keep it indexable, right? Because maybe I should have it on my website because it fulfills some role some type of role from a legal perspective, but it's not necessarily, it's not required that it's actually findable by search engines. Users won't actually be looking to search for this type of content. And if not, then it's okay to noindex this page. We can add a meta robots noindex follow to the page and nofollow any links going towards it because realistically we don't want to send any value to these pages. That if these pages are not fulfilling any business, important business role that we want that is findable and they are not targeting any important query also from a business perspective or important in the customer journey, so we very well can noindex. And after its new status is updated and search engines noindex this page and we see it's not indexed anymore, we can very well block it. With the robots.txt so Google doesn't, or search engines in general, they don't spend time going through these pages in case there are many, right? Maybe this is not even worth it if there is a very small number of them, so it's not necessarily a huge consumption of crawl budget. However, if we are in a very specific scenario that there's a high number of this, we might just stop crawling them to stop consuming our crawling budget. Then on the other hand, We might answer with yes, the content has a business, customer, legal-related goal, and yes, it is required that it is findable, right? So we should keep it indexable within our website. If that's the case, then we come here and we ask the next question, that is the content of the page to be kept too poor, thin, not updated, not relevant enough, or not formatted to satisfy the user needs? So even if it is not a page that is meant to be ranking for a top query from the business side of our website or from a product service perspective, we want it to fulfill very well its role. If it is too poor, too thin, then it's important to improve page content by updating, expanding, reformatting the information to satisfy users' needs and make the most out of the SERP features also if that's a possibility. And then of course, Validate that page to be kept is correctly crawlable, indexable, internally linked, and included in XML sitemaps. Since it is meant to be findable, it is meant to be indexable, it's meant to be searched, we should deliver the best possible experience, right? In case that we find that the content is not poor, not thin, it's good enough, updated enough, it's relevant enough from a content perspective, no, then we should only validate, verify in this case that is technically correctly configured within our website, then we go to the other side of the flowchart. Is the SEO content actually driving value of any kind? And the answer is yes. So is the page content too similar, offering the same information and/or targeting the same similar queries than other pages with SEO traffic user volume? If not, this page is targeting something very unique, featuring very unique information, content, and targeting a very unique query. If not, then we go directly and ask, is the content of the page too poor or thin? Can it be updated, improved? And we follow the same steps. If yes, we improve the content, we expand the content accordingly to make sure that provides the best possible experience to the user and fulfills the user needs and validate that it's correctly configured from a technical perspective, that is correctly included in the navigation, XML sitemaps, etc. And if not, we just validate the technical aspect, technical configuration of it. If the page is actually targeting very similar terms than others that are also indexable, that are also driving value, then we should ask if the page has a specific role than the other. If this page individually is targeting something unique, relevant, unique popular queries to make it worthy to differentiate and optimize its content. Maybe we can see that the content at this point is very, very similar, but realistically it could be differentiated because the nature of the pages are actually different or can be differentiated very easily. This other page can be tweaked to be actually improved and differentiated well enough to target another topic that is also, might be also very important for our website, right? If that is the case, then it is worth it to differentiate the content of these very similar pages and optimize the page to target its own specific queries and update it, format it in a way that it will actually fulfill the user needs and to make the most out of the SERP features. And once that we do this, we will again verify that it's technically technically correctly configured again. That is correctly crawlable, indexable, internal linking, and included in the XML sitemaps. If we identify, however, that the page doesn't have a specific role within the website, so it is featuring very similar content or duplicate content, duplicate information at this point, and it doesn't make sense to differentiate it because it actually has the same role than another already very popular, another page that is also driving value, Then the question that we ask here, is it necessary to keep the page enabled as it fulfills a specific functionality purpose towards the user experience or business within the site? And this is the type of role that, for example, some filters or facets might have. If these are just to change the number of items that are shown in a page, they will generate a new URL, doesn't necessarily connect with an actual query need, but it does play a role within with the user experience of our website. So we need to keep it enabled as a specific URL and that is actually accessible from a user perspective. But we don't necessarily want to go into this other site because it's not targeting any unique specific queries that are worthy to rank for. So in this particular scenario, we will canonicalize these pages toward the original page URL version that we have. Identified. This is usually the one that is more prominently linked from within the website navigation, bringing more SEO traffic, user value to you, that has a more prominent role also in the site web organization. So we will canonicalize back to this original URL that we set and also link accordingly. We are going to nofollow the links going towards these non-canonical URLs because they are not worthy to get any value because they are not the ones meant to be ranked, but the original ones as much as possible, eliminate them from the XML sitemaps. And once that this is done, then we should ask again the question, is the content of the page that is kept, in this case not the one of the pages that are canonicalizing but the ones that where they are canonicalizing to, the original canonical URL, is the content of the page to be kept too poor, not updated, not relevant enough or not formatted to satisfy the user needs? If yes, then we should again improve and expand, update the content to fulfill the user needs and to make the most out of SERP features, etc. And then again validate from a technical perspective that the page is correctly crawlable, indexable, internally linked, included in XML sitemaps, and so on. On the other hand, if the answer here is no, if it is not necessary to keep the page enabled, it doesn't fulfill specific functionality that is really required to be kept on the website. Then we can consolidate these pages to a similar one and we can redirect them to the one that we are going to select as original, new, updated one that we want to keep. So consolidate the similar pages by 301 redirecting them to a single URL version to be kept. The one that usually drives the highest SEO value, user engagement, value, showing the same similar content while fulfilling the same similar role from a topical functionality, business, and user experience perspective, right? And we should replace and eliminate any internal links going to these URLs that are going to be redirecting. The same, eliminate them from the XML sitemap. And again, once that we do this, with the content that is actually kept with the page where these other URLs are redirecting to, We then should ask, is the content of the page that is kept, the final destination of this redirect, is too poor, thin, not updated, not relevant enough, not formatted to satisfy the user needs? If so, we improve again and we then verify, validate that all the technical configuration is okay. So as you can see, I think that most of the most common scenarios in this content pruning journeys are included in this flowchart. So we can act accordingly. Of course, as I mentioned here in the note below, remember that there might be exceptions to this criteria as happens with everything. Apply with care and after validated within your own website business goals and this is just the first version. If you see that there's anything here that is actually worthy to be included, an important scenario that I haven't included, please let me know. I'll be more than happy to include it and I'm going to show you now how you can actually gather all this data and reply to these questions very easily thanks to a few tools. Of course, I want to show you how you can answer these questions from the flowchart in a very straightforward way by using SEO crawlers to crawl our website and integrate the additional metrics that will tell us what's the value that they're bringing to our website. So for this, we can use crawlers like Screaming Frog, as I mentioned before, or deep crawl that I have here. And then of course complement this data with the one that we can obtain from the Google Search Console, for example. With the coverage report here, we can obtain which are those pages that have been excluded from the index, right? Why they have been excluded if they are indexable. These are pages that are important for us to check. And of course we can gather data like this from the Google Search Console and do specific list crawls with any of these SEO crawlers to validate these specific pages, it is also recommended that we do a full crawl from all of our website content to identify and assess all of the crawlable URLs that we have on our site. This is what I have done already here with Remoters, which is my website project here. And as you can see, we can easily integrate external data metrics that will allow us to answer the questions from the flowchart. So from a traffic perspective, Google Analytics, the Google Search Console to be able to identify what are the rankings, the current rankings of pages, the impressions that they are getting, etc. Then Ahrefs for the link-related metrics to understand which of these pages are actually being also externally linked. Not only with Screaming Frog, but also I'm going to show you with DeepCrawl too, we can integrate data from the Google Search Console, Google Analytics, the backlinks, data from the sitemaps too to identify the gaps too and assess these pages again to prioritize the evaluation of these pages and identify these lower performers. Is that if we go here for example with a Deep Crawler or Screaming Frog or whatever SEO crawler that you use and look for the thin pages, those pages that are showing less content. In general. And we can include and identify which from these pages have actually any organic search traffic. Pages with duplicate content or very thin content and identify which are the ones driving less traffic or with zero links, for example, that are indexable at the same time. Another important aspect is that any of these tools will allow us to export these pages along all of the metrics that we have gathered for them. For example, here I have the export from DeepCrawl. For example, I can start and select these pages and tell me all of these pages are actually indexable and that are also duplicates. So in this case, none is duplicate thankfully. And that have the lowest content to HTML ratio or that have the lowest text count in any of them. And exactly the same, with the data that we can export from Screaming Frog. From all of these pages that I have exported, that I have crawled, which of these are actually indexable and which of these have the lowest word count? So here I will reorganize a bit like this and tell me of which that are indexable and have a very low word count, which are the ones getting zero GA sessions, zero traffic, and zero links because again, I have the Google Analytics data here and I have the Ahrefs backlinks data here, zero links. So I can see all, despite being indexable, have very little content, may have content duplication issues and cannibalization issues and are delivering zero value from an organic traffic perspective and rankings perspective. Start filtering and at the end we can end up having with this very specific set of pages that we can start here like the worst performance. We can see which are this very low performance and unfortunately these are the jobs type of pages that have indeed very little content because they have been generated in a very automated way, very little information. Take a look, so these are the lowest for performance and it makes sense. So in this case I should decide like what do I do with this lower performance. I will follow the flowchart then. Is it worth it for me? Yeah, they are not driving much value right now and the content is also too similar with the ones with the other job pages unfortunately. Yes, they do have a specific role within the website and they should target individual queries, should be worthy to target and so ideally in this case I should differentiate, include and feature more content to fulfill this type of query that we want to address with these pages. Hopefully with this couple of tools and integrations, you have been able to see how to gather this information and to prioritize which should be these pages that you should be improving and which should be those that it's not worthy for you to keep and you can eliminate in a very straightforward way like this, following these flowchart rules. And of course, if you have any questions, any doubts, if you want me to include any other particular type of use case that you think might be valuable, just let me know. I'll be more than happy to hear your thoughts about this. Leave your comment, don't forget, or tweet me @aleyda over Twitter. Follow me in case you still don't have followed me for any reason. I'll be more than happy to reply. Thank you very much. Have a good day. Chronic Mondays.