SEO For Paginations: How to Optimize your Paginated URLs for Search

Welcome to the pagination edition of Crawling Mondays. Today we are going to discuss about the best way to deal with pagination, how we can optimize paginations to avoid eating our crawl budget, to make the most out of paginations, to help our products discoverability in our e-commerce or marketplace website. What is the right way to configure pagination? What we should avoid? And for this, I have invited today 2 SEO specialists who have lots of experience. On one hand, I have Orik Mutznik, who's the Head of SEO at Silfred. And on the other hand, I have Patrick Stoxe, Technical SEO and Product Advisor at Ahrefs. Thank you. Thank you for joining me today. Pagination is super important to understand how they work, to deal with them, because realistically these are not pages that we are going to look to rely to rank for anything new, unlike categories or subcategories or facets, right? But these are pages that will exist on the websites and can help us or not to maximize our products' discoverability on one hand. And then on the other hand, if we don't configure them well, they can end up damaging our website crawl budget. They can also end up generating cannibalization issues. So can you please share a little bit about scenarios that you have found in the past to show why it is important and what we should be thinking about pagination.

In my case, before I started to look at it, when I first started, so I discovered that 70% of our products were only accessible through pagination. And pagination was blocked to Google through essentially via any single method that you can block a page. So it was canonicalizing to main, it was generated through JavaScript, client-side JavaScript with fragment identifiers. So pretty much every way to block indexability of pagination. And that, when I crawled the website, it was 70% of the website actually that was blocked from Google. And needless to say, what's the importance of losing 70% of your website, basically disabling that from Google? You can't really avoid it. Well, you can choose a different method, of course, that's not pagination, but you got to find a way to show all of your products to search. In my In my case, it was pagination and I had to make sure it's optimized so all the products show to Google. That's what we want eventually. So the first page was showing, it had 40 products. And even if we had lots of categories and subcategories, it was always the 40 first products out of thousands that Google was seeing. So even though you have internal linkings a little bit and different categories, Google would never be able to see everything without pagination properly accessible. So in my case, I saw it as a necessary step to make it accessible. And I saw it more beneficial than the risk of having pagination indexing was much, incredibly much smaller than having 70% of your website basically inaccessible to Google.

That's a great answer. It's— I try not to overthink pagination generally. But it's mostly you want things crawled, you want things indexed, and you want a way for users to be able to find your products.

So what are those worst scenarios that you have found, right? Like those typical errors that you see people doing again and again. So for example, like I have a particular pet peeve because I think that this is actually quite in line with user experience, right? Let's say like overusing paginations, but literally like only featuring 6 products per page.

Yeah.

And I'm like, why? Like, please give me 2.5. And I have been involved and I have also heard stories of SEOs doing A/B tests. Because from an SEO perspective, whenever I am asked like, how many products should we feature per page? The most as possible to maximize the content relevance and facilitate the user to go through the page while also not putting in risk the loading speed. Of the page. So it's about a balancing act, right? Balancing act of showing as much as possible while also not hurting the speed of the website because it will never load because it has so much. And this will also facilitate to not rely so much on paginations to go through additional products. We have also JavaScript configurations that rely completely on scrolling. Of course, the Googlebot cannot scroll, things like that. I don't know, what else have you seen? What are the craziest stories that you can share?

So I've been really lucky to, in addition to make these fixes and build my case study on, you know, our website. So I use this opportunity to look at the 150 top UK retailers, my competitors. So I essentially witnessed all of the different mistakes that other websites are doing. So for example, 67% of the e-coms in the UK basically point the pagination to main without providing any see all options. So that would be fine if you'd have a way for Google to see all of your products together somewhere. Like there were some websites where you would disable JavaScript and you would see like all products in one page, which is actually really cool. But most websites, like 67% of them who are pointing to main, didn't have any other way for Google to reach them. Also, 50% of brands, for example, strictly noindex and robots.txt the pagination. And again, no alternative. It's not like they have it somewhere else where the products are. Loads of JavaScript-related issues. Like for example, there is no JavaScript-disabled version, which is something that is a best practice by Google when you have a fast-changing website to provide a static HTML version, like a see-all version. So no, basically when you disable JavaScript, many of these websites are simply like blank. And block pagination is generated by fragments. And I think the worst area is wherever a website implemented infinite scroll. So 11% of those implementing infinite scroll were actually indexable. And of course there is no alternative. So of course you can block your pagination, you can do whatever you want, as long as you have another way for Google, for the crawler to reach your products. If you don't, I see that as a big mistake.

Quite a big list of errors. It's also a little bit disheartening to see that in this study that you did of the top ecommerce sites, in the UK, you ended up finding a lot. Do you know what is the risk of that also? That I see a lot of new websites when they are just starting and when they don't have also sometimes even proper SEO advice, that they go to the bigger ecommerce websites to see how they are configured and they just copy, right? And then they come like, oh, but we did it like this because we saw this big retailer doing it like that. Yeah, because they have it like this Because of a lot of restrictions that big websites have. But they are ranking. But yes, they are ranking not thanks to that, but despite that, because they have all the content, all the products, all the links, and many other factors. We need to be careful with those.

Don't copy the big brands. They make tons of mistakes in that area.

Big companies, they're going to be dealing with a lot of legacy systems, a lot of people involved that have differing opinions on things. And some of this stuff is complex because we don't really know what is right or what's wrong in some cases. You mentioned canonicalizing to the first page. Well, yes, don't do that, but why? The pages should still get crawled through, but maybe not be passing PageRank. You could test that. It's something where they could also learn that all these canonical, and maybe they actually stop crawling the pages. If they see a pattern that says all these paginated pages just canonical to the main, they might just stop crawling them. They may not see the other links to the products. And so you could end up with a lot of things not indexed because of that. The noindex, you know, we've got conflicting advice from Google there. I like to point this out, but John says noindex pages, they don't pass signals after a while. They're treated as nofollow after a period of time. Gary says they do. So we don't know like what's really right there. You might be able to get away with noindexing stuff. You might not be able to.

Yes. This is the thing. It's tricky. And at the end of the day, I think it's, It's very, very important that we test for ourselves. It's interesting because I think that the other day we saw this story that was published about a test, which I thought it was amazing that we are testing this stuff. And I think it's important and necessary, but just make sure to follow a proper methodology when testing, right? Like for example, when you're testing this, don't do a lot of other work at the same time or configuration changes at the same time that will impact these pages too, because it will be very difficult to isolate. And also you want to be able to compare and have a version that is not changed at all to be able to compare and see if the evolution in rankings or whatever that you're testing, crawlability, indexability, whatever that you're testing, the metrics that you're testing, will change differently than the other versions that haven't been touched at all, things like that. So following certain best practices when— and testing methodology, right, is super important. Let's move on to the other side of the story, right? Which are those best practices and guidelines that we know that at least if we follow them, we should be like safe, right? That they set the base of what are the good practices to deal with paginations to minimize issues and maximize opportunities with them.

So I think since the rel=next previous deprecation, so Google have stated in numerous opportunities that you should treat paginated pages as standalone pages, that is having a unique URL that is indexable and crawlable. That's something that John said really briefly after this announcement. So essentially, if you're talking about a unique URL, so make sure that your paginations have like parameters or like a friendly URL canonicalizing to self, uh, if that is the content of that page as you want seen by the crawler, and link it from clean hrefs from the website's architecture. But essentially, if you come to the conclusion that the only way to reach a product, for example, is from the pagination and you have concerns about making it indexable, which is a concern, then you could block the pagination and just provide a see-all page. So you could canonicalize to main, noindex, robots, all that, as long as you have a see-all version that is available so the crawler sees all the pages. Or you could also make sure that all the products are covered beyond the pagination itself. So make sure that you have a very strong interlinking mechanism through various places. And of course, make sure that you crawl the website with and without pagination and see how much you're losing or gaining and then make those decisions.

Internal linking of pagination is critical. I have seen so many ecommerce websites—

100%.

The only way that they link to paginations are with parameters that are not even self-canonicalized.

I would say a lot of CMSs actually are fine out of the box. Leave stuff indexed. It's sometimes just SEO's kind of getting in the way. They have an idea that they should do something like canonical everything to the main page and they end up like kind of breaking stuff. So it's not, it's not always, uh, an issue. I would say with JavaScript, uh, one of the things I've run into a bunch is people tend to not actually build links as like Ahrefs. Like it works when you click it, but It's not a real link, it can't be crawled, etc. So that can be a problem, uh, something you want to look out for. Well, we're talking about things to do, so in that case, add that, make sure it's actually a link.

Yes, not even only on scrolling, but actually, like, you see, like, the, the actual navigation supposed to be navigation, uh, but that is completely implemented with obfuscated JavaScript that cannot be that cannot be crawled indeed.

Yeah, Google doesn't click. They're not, they're not going to see that. If it's not a link, they can't crawl it.

Exactly.

I would say for a lot of sites, a lot of blogs, for instance, they should take some hints from news or e-commerce. They typically have one long, really long paginated series, might be hundreds of pages. Whereas if you look at e-commerce or news sites, they split things into categories. You know, here's my world news, my politics, my travel stuff, et cetera, or e-commerce, different product categories or even subcategories. And so they've done a good job of kind of breaking things out and making sort of valuable pages actually by having categories. And unfortunately, like blog systems like WordPress don't really do that out of the box. They probably should on the blog homepage be like, here's my list of categories. And then you click in a category and it's like, here's some popular blogs and then a list of all the blogs. and stuff like that. So you can do, you know, Oren talked about internal linking, but some of these systems aren't designed the best for this out of the box, I would say.

Yeah, 100%. I know that a lot of people sometimes is even scared to index facets. They are very scared of content cannibalization, et cetera. But for me, it's a matter of supply and demand. And if you make sure that you differentiate and this provide a unique value, unique products, and then by linking to them in a hierarchical way, consistent way along your taxonomy, and then paginating these pages correctly too, you split all of this in a way that will be found in a much more natural way by Google, replicating much more real user search experience, right? And the way that the users actually search with colors, with characteristics, with whatever terms, instead of having like 20 paginated pages, then each one of these will have 3. And it will be much easier for Google to find the products in the right relevant way. Another important aspect I think that we should mention is like, yes, indeed, a lot of people are scared of paginations because they think that this will end up like eating their crawl budget and not necessarily like providing an extra value from a specific ranking perspective, right? By themselves. It's true that they are not going to rank by themselves, but they will help Google to discover the additional products of content from that series. And what is important that I think that we also do is to make sure that we'll give enough hints also from a content perspective to Google about that, the metadata, right?

100%.

That we don't have exactly the same metadata, but that we add the pagination number in the title, in the descriptions, along the content, that we give additional signals. And because I think that this is the thing before with the rel=next, prev tags. I think that a lot of people say, oh, we add those tags and that's it. We don't do anything else. Google will sort out. Sometimes we rely like that on specific configurations instead of aligning all the signals. We need to always align the signals to facilitate Google's work, right? Are there any other recommendations, tips that you will give to make sure that pagination is not only avoid issues, but also like Google can understand that these are part of the same series of content.

This is something that's sometimes hard to do. Like it wasn't in my phase 1, unfortunately, just because it's just so technically complicated to implement. But exactly what you said in regards to providing those signals. So the first thing that I needed to do as soon as pagination was live in a more friendly way was to de-SEO the paginated series. So that is only the first page will have the content. The meta tags would have like the page number as you said and some, they would probably not have a meta description for example. Only the main page would have it. Like I was trying to look for all those ways to de-SEO those pages. I have no cannibalization issues because in that way like it's very clear That the first page is the one that's supposed to rank and all the other pages are there for crawlability.

I have a weird view on that because I feel like if a page is cannibalizing another, like really it's just one page is showing instead of a different one. And if they're showing that one, it may actually rank higher than the one you want to show because maybe the product selection is better on page 2 than page 1, etc. Why?

I think that we should be identifying these sort of issues. And 100%, like one of the things that I do, and I think that this ties very well with the troubleshooting questions that I have, like what do you do to troubleshoot, to identify opportunities? What tools do you use, functionalities that you find useful for this? One of the things that I do whenever I am doing ranking research to identify how the website is already ranking and with which pages, I go and take a look if there are any paginations they're ranking instead of the proper first page that should be ranking and identify why. Is it because of a lot of backlinks? Is it because from an internal linking perspective, the first page is not linked at all? Sometimes I have discovered that there is a bug in the pagination navigation and it's the second one that is linked instead of the first one, things like that. So I do think that, of course, I really like to identify those low-hanging fruits.

When going to my current rankings opportunities, I think, well, my main go-to is directly to crawl the website with DeepCrawl and Screaming Frog to see the different ways that they understand the website and understand what exactly the problems are with pagination. Like what are all the like 5 different types of blockers that I have to take care of? And then after that was resolved, then I think Google Search Console is great to see where like, is there a page that slipped somewhere? Is there a page that is ranking? Semrush as well, like beyond Google Search Console to see any page that slipped in the cracks and understand why. In e-com, we have tested the number of products per page, for example, and surprisingly the lower number of products won. So this is why I have no cure to the issue of Pagination.

I love the recommendation of the Google Search Console because I think that it is not as leveraged as it could, especially with pagination because you can go and take a look at if they are left out, if these are some of those pages that are excluded from the index, right? And we know that we have some limits. So for that, I really like to register every category and subcategory of the website independently. To be able to gather more URLs that Google will end up showing, to go through them and see how many paginations are there from those excluded pages. And why is Google excluding them? Is it because they are choosing another one as the canonical one? Which one is that? Is it another page, another pagination page, or is it the first page that I actually want to endorse? Any other tool, any other functionality, Patrick, that you like to use and you would like to advise on?

Well, of course I'm going to say Ahrefs. Like, our crawler is actually good. Not a lot of people use it, and, uh, it's weird. It's, uh, it's—

But yes, please talk more about the functionalities that you have, uh, today.

I mean, it's totally free if you sign up with, uh, Ahrefs Webmaster Tools. You can go to ahrefs.com/awt and that'll get you there. Um, yeah, I wrote an entire blog on pagination and how to use our Site Auditor to identify different issues. The one issue that we were talking about though, where another page shows, that one, yeah, you can easily find, but that might be more of a business decision type problem to actually solve. Like if it's a problem where it's a mix of the products shown, maybe change the sort order, because if you're showing users like all the popular products, well, the The products that people are looking for or buying may all fit into one particular facet. If it's lawn furniture, maybe it's the wood lawn furniture everyone's buying, but maybe to have a better result, you need some wood and some metal and some plastic ones. So maybe page 2 has a better mix like that. So you may not even want to worry about it because doing changes like that, you might make page 1 rank instead, but you could knock it down a couple positions if you do stuff like that, or try and change your own sort order or show a better mix of products. It's hard if you're all popular and that's what people are buying, you think that's the best search results or that's what's best for Google, but it might not be. So you might have to do a few little hacky workarounds to show a few products from each particular category. Here's popular ones for Wood, for metal, for plastic.

So to start wrapping up, I want to ask you a last tip. Something that you think that we haven't covered yet, something different, something extra that you would like to advise people watching this to optimize their pagination.

So I think that there's a lot of debate about this subject. You don't, you won't always reach the same conclusion. You won't always decide I'm going to make this indexable, I'm going to block everything. So you first have to have a set of considerations. So I think you need to think about it as a process, like as one of Aleyda's flowcharts. So you first start by crawling the website with or without pagination and see what's the difference. What percentage of the website are you blocking from search without it, with it, like for example. So if you're happy with that percentage, then you can keep it. That's fine. If you're not happy, then You can make your pagination indexable. That's one option. Or you can use a display all option to avoid pagination if that's not something you want to do. Or you can make, you can make sure that everything that's lost from pagination to be linked otherwise internally through different like facets, for example, like smaller categories, like landing pages with like the color in it, the length in it, like a little bit more specified, which would link to all of the products. But at the end of the day, you gotta make sure that everything is covered.

Yep. And I would say, you know, just start off with the basic way. Just leave everything indexed, leave everything self-canonicalized, and then see what happens. Like, you're— it's probably gonna do pretty well. And then you can run tests against that. Like, if you have an idea, I wanna say, like, you know, maybe I should just index the first 5 and noindex everything else. Well, at least you have the baseline to go on to say, Did this actually have a real impact or like I just launched it like this and I hope for the best? So I think you really need that baseline. I'd say you can never go wrong with that setup anyway, unless you have other issues like infinite facets and stuff. But yeah, there you'll probably have to adjust if that's the case, block some crawling, et cetera. But you generally want to start with things indexed and crawlable.

The one that I don't think that we have mentioned, there is a very big difference sometimes with the pagination navigation. Some websites literally linking only the 2nd and 3rd page and then page—

Dot dot dot.

Dot dot dot and page 100.

Yeah.

The number of hops that the crawler needs to do in order to find everything will be completely different if you link the first, the second, the third, then the fifth, then the tenth, then the twenty, twenty-five, then the fifty, fifty-five, and finally the one hundred, right? So try to balance. Do tests with this. Try to balance to also well facilitate the way that Google will find these pages to reduce the number of hopes also because yes, crawl budget will also depend on on this at the end of the day is different to having to do 100 hops to reach a page, then have 1/3 of that, of course. That on one hand. And then on the other hand, another way also to minimize the reliance on pagination is how many products are you able to show per page without hurting your user experience, your web experience, the loading times, et cetera, of the page, right? And you should definitely do tests for this, not only from an SEO perspective, but also from a user experience perspective. So again, you need to think that at the end of the day is about facilitating the users to find the products that they actually want to buy from you and you want to sell, of course. Yeah. So thank you. Thank you very much, Orit, Patrick. It was a pleasure to have you today. It was a great conversation. And if you have any questions, any doubts regarding these topics, please leave your questions in the comments. I'll make sure to give the heads up to Oridan or Patrick to ask them to. And if you like the video, remember to like the video. If you want to watch more videos like this, subscribe to the channel. And also, Oridan and Patrick, before I forget, they are also very active on Twitter. You can follow them on Twitter too. Definitely follow them. They're always sharing good tips. They have also tons of resources regarding these topics. So thank you very much. And until the next Crawling Mondays, bye-bye.

Crawling Mondays.