How to Do Successful (Non-Spammy) Programmatic SEO

Welcome to a new edition of Crawling Mondays, this time dedicated to programmatic SEO. In today's session, we are going to discuss what is programmatic SEO, when it is relevant to use it, is it always spammy, is it always bad, and how you can ensure that your programmatic SEO output is really relevant, insightful, high quality, connects with the user search behavior, and allows you to really scale your organic search results. For today's session, I am bringing a couple of people who I respect a lot and have tons of experience in this particular area. On one hand, I have Lazarina Stoi, who's an SEO and machine learning consultant. Hello, Lazarina.

How are you?

Hello. Hello. I'm good. Very happy to be here. Thank you for inviting me.

Thank you for joining today's episode of Calling Mondays. On the other hand, I have Rossana Estingu. Rossana is the head of search and SEO at Alamy. Hello, Rossana.

Hello, Alida and Lazarina. Thank you for having me. I'm great.

Thank you for coming. Before starting, I want to thank our sponsor, Content King. Content King is that tool that you want to configure and have in every SEO process in order to identify right away whenever something goes bad, when a misconfiguration is triggered or launched in your key pages. Something is noindex, something is canonicalized. Content King is great for that. Go and take a look at Content King in the description below. And now we are ready to start. So programmatic SEO, what is it and what is not? And is it always spam, right? Unfortunately, there's this misconception about it. Coincidentally, a few weeks ago, Google updated their guidelines, and one of these guidelines have to do with scale content abuse, right? And unfortunately, because of the nature of programmatic SEO that allows pretty much to generate information and content at scale, many websites have used it in order to generate low-quality content, spammy content, many times leveraging machine learning, AI-related functionalities, right? But it doesn't need to be like that. There are plenty of examples out there of websites currently making the most out of programmatic SEO in order to generate information at scale, information that they have in their own business because of all of the insights and information that they store from users, that they have from their industry, that they have from their day-to-day that can be helpful to enrich existing content Or to generate even more content to connect with the way that the user search. I believe that a great example for this will be Zapier connectors pages that are generated automatically to help users to find out about the different connections that they provide between different tools. Another good example is Wise currency conversion. These pages are generated also at scale, I believe, uh, pretty much combining different currencies, giving insights about day-to-day conversions. And also making it easier for their actual customers to understand when it is a good moment to use them or not. So there are plenty of programmatic SEO out there showing how this can be actually useful for not only websites, but for users to look for information or to access to information and how it can leverage by businesses to scale their content approach. Having clarified this. I would like to ask Lazarina and Roxana to tell me a little bit of like those misconceptions that you have found in your day-to-day about programmatic SEO and when it is that you consider it to be more meaningful, more useful to actually use it in your processes.

So you asked initially, right? What is programmatic SEO? And it's just SEO, but once you automate it, it becomes programmatic. That's it. So if your SEO practices are spammy and might get you in trouble, then your programmatic SEO practices might do the same. So it's not about using automation that makes it spammy. It's just about the end result of it. So I think you already kind of touched on this. As long as you, you create useful pages or you create useful information on your pages and you can get quality out of using automation, it doesn't matter that it's automated as long as it's useful. And especially when you get to scale, how, how are you going to write titles for millions of pages, for instance, one by one? That's never going to happen, right? You need automation in that case. And not just that, pretty much any change you want to do on millions of pages will have to be automated.

In any e-commerce marketplace out there right now, there is some sort of programmatic SEO component already being used or has been used Historically, right? And people tend to overlook that and thinking about it of only those pages are generated from scratch at scale.

I share the same view as Roxana, but my definition of programmatic SEO, uh, based on all the content that, you know, different case studies and all of the examples that exist out there for websites, they have several things in common. And you don't necessarily have to have all of these things in order to call a project programmatic, but typically there are a few shared characteristics. The first one. Is that you have different content sources that you're pulling data from and you have a content database. Whenever you're organizing the creation of new pages, you have to have a database. Of course, this database can include elements like canonical tags and title tags. It could also include the content. It could include design elements. When it comes to design, all programmatic projects typically have a scalable page template. That means that the database is connected with the template. And so if we take all of these factors together, the content sources, the database, and the template, and we combine that with a programmatic publishing approach, that's typically what defines most of these projects. I think the misconceptions and more of the myths that are, that exist around this topic, there are a few. The first one is, like you mentioned, programmatic SEO is spam. Uh, it's only spam if it's low quality, if it's created without the user in mind, if you are replicating information. And I think the, the biggest misconception about programmatic SEO came recently within the past couple of years because people are confusing programmatically generated content with programmatic SEO. And those are totally different things. You can have a programmatic SEO project that uses as one of the content sources a programmatically generated content, but if it only stops there, then it's a surefire way to get a manual action and to create spammy pages. So I think that's one of the misconceptions. Another misconception is that you can only use programmatic SEO for niche sites and, you know, your pages are not really going to get a lot of traffic. A lot of the examples that we see out there like Zapier, Wise, I would say also Forbes author pages. I would say also G2 as well as another example. Companies are creating content for their blogs with programmatic SEO. I think UserPilot is a very recent example that they started experimenting with comparison content. Most of these pages are not really that niche in terms of searches, but they are niche for the industry that they exist in. And I think also another risk and a misconception is that you can't really get a lot of traffic for the pages that you're creating. And that's again another misconception because the traffic is actually dependent on the quality of your pages, the quality of your processes. If they're high quality, they're going to get indexed. If there's interest for the searches for which you're creating pages, of course, there's going to be traffic to them. If they're, you know, consistent systems and processes, programmatic SEO is not going to hold you back. On the contrary, it can amplify the efforts that you already have in place.

I believe that this is key in general for any SEO process, right? But more obvious when you have the ability to do it at scale. So not because you can't target everything at scale, you should, but in some ways it's actually needed, relevant, and actually very profitable. So knowing when to focus, when it actually makes sense, makes all of the difference. This brings me to the next question that I have for you. What would you say are the top dos and don'ts for programmatic SEO, which are those mistakes that you see people end up making. In my case, what I have seen many, many times, unfortunately, is not establishing rules to avoid duplication, to establish a minimal level of insights or data to index a page, to generate and then even index a page, or internal linking considerations as an afterthought. And then seeing how they have generated hundreds of thousands of pages many times that are pretty much silos. And don't cross-link between them and connect with the rest of the website in order to end up helping the conversion journey, which at the end of the day is the ultimate goal of all of it.

A lot of the don'ts are around the content because that's where it's so easy to get it wrong. A lot of people will just go with a template and just replace some words here and there. And then I think that's good enough to create new value. And usually it isn't. We see it in local SEO quite a bit where there's a whole page about somebody's services and then they have hundreds of pages with the exact same text, just different locations. And then they wonder why their locations don't rank when we've been told so many times, don't do this. Where's the value to the user in this case, right? And pretty much just content spinning in general. If you're thinking, oh, it's not going to be exactly the same because now I can use AI and just spin my content a little bit and it's not going to be word by word, it's still the same topic, it's still the same idea, it still serves the same intent. So then it's duplicate Because of that, it doesn't bring anything new. So content is definitely a big one. There's also a risk when you automate things that it gets a bit out of hand and you forget to measure what exactly you're creating. So don't just create excessive pages continuously. You need to have a bit of a plan when you, when you start automating things. You, you don't just want a page for every single word out there or keyword. You want to target things that have a search volume or interest or are relevant for your niche, or you end up just Creating a mess on your website. And then the more pages, the harder the internal linking. How are you going to create valuable internal linking when you have so many pages without ending up with blocks of internal links everywhere just to be able to create a few links for all the pages?

So my don'ts, I 100% agree on like content on autopilot. That's a very bad idea. Like copying others, being unoriginal, especially even now that you can do it automatically. That's I think the biggest don't is don't start with programmatic SEO unless you have the time and the budget to do it correctly, because this is the most important defining characteristics of the difference between successful and non-successful projects. People think that because programmatic SEO is kind of like a, an entryway to creating a large website fairly quickly, then it must be easy to do. But instead it's not, it's absolutely not easy to do. So in terms of the dos, how do you actually do it successfully, even with time, even with budget, my dos are first of all, invest in a good content system. So it could, you could be incorporating some level of automation within the content, but that should not be unoriginal content. You can still create systems involving experts, having editors on board, pulling data from your own company database to feature on the website, and that way enriching the information within the search landscape overall, instead of just copying what others have done and rebranding it with your own company brand. I would also say try to imagine what a perfectly optimized page looks like before you even start organizing your database, because your database needs to contain all of the different components that are going to make a page good and visible and crawlable and indexable and high quality, high traffic, helpful content, all of these things. The good thing about this is that we have actually very good guidance from Google when it comes to what makes content helpful, what are the signals that, you know, they're looking for when it comes to quality content. So you can actually reverse engineer that and incorporate all of these elements within the database design process so that you can make sure that each page has these elements. I'm a very big proponent of automation, but the question is, how, how can you learn to do that in a not only ethical way, but also in a way that is actually quality. So a good thing to do is actually understand how machine learning models work and when to use which one.

I think that you have actually touched something very, very important here. The ethical, let's say, matters that shouldn't be an afterthought because I have seen many mentions out there regarding programmatic about where do we get the data? Oh, like there is a database sold out there. With data about, I don't know, the weather, the latest trends in whatever sector or industry that you don't even know that if it is accurate, who have created it, that, I mean, will be your core input. If this is going to be your core input and you have no idea about the level of quality or accuracy or ownership, you cannot create something of value if you don't know where it comes from or how good it is. And then another very important thing I will say is to also Double check if the type of content that we want to create with a programmatic approach is actually the one that is ranking the top positions. Because I have seen many times clients asking us, oh, we have all of this data from our customers. Can we create some directory on our site providing all of this insights? And then I go and take a look at which are the top rank pages. And all of them are pretty much informational content. Because for this type of query, Google has identified that the need is informational content at the end of the day, right? So, I mean, great if you want to generate it from a programmatic standpoint, but it cannot be a directory-like presence. It needs human information added or blended in with a different sort of format. Then a final thing, it is more important than ever in an era in which a lot can be automated now. Everybody seems that it's leveraging AI content, right? Like to ask, what is your unique selling proposition? If all of the pages already ranking in top positions for the queries that you want to target at scale are already doing the exact same thing, what extra can you do? What is the additional insight, meaningful insight that you can provide in order to make your content outstanding? What would you say are the must-follow steps that you highly, highly recommend people to take in order to ensure that we avoid all of these mistakes and that our process output is the right one?

I have one that ties nicely into what you just said about having a look at what other people do and try to stand out somehow, right? You don't just want to be a copy of the already thousands existing websites out there. You want to do something that makes you unique for your users so they prefer you over other people, over and over again. And programmatic SEO actually can be very creative. Uh, and that's because, as you said, it does work with data and data is the best at telling stories and it can say so many stories and all you need to do is find that story and find a way to then use that data on your website and automate it to create new content that your users will like. And to give you an example, a few years back there used to be somebody, and I'm so sorry, I can't remember her name, but she had a lovely presentation on programmatic SEO before it was even cool. where she talked about a beauty website and how they looked at the data they had available in their database alongside the products and decided, well, we know ingredients for every single product. What if out of ingredients we derive a story about the smell of the product? So then can add, you know, for face creams and perfumes and stuff, not everything has a smell, but if something contains citrus, then we can say it has like a fresh smell of oranges or something. Like they would have all these rules. Saying if it's this ingredient, here are some phrases you can use. And then those would generate into sentences that would describe the smell of the product. And that was something that everybody else selling the same product didn't have because everybody else would just list ingredients because that's what you get as information. So you can always get very, very creative with these things and find new ways of automating the data you have into showcasing it in a new way that might be helpful for the user. But I think ultimately you should A/B test it. Test everything. And especially with programmatic SEO, you have hundreds, thousands, millions of pages to test with, which is the best when it comes to SEO A/B testing. Because you can always just grab a few thousand here, a few thousand there, have a test, have a control, have multiple tests and controls. Why not? Because you have so many pages, you can actually do that. And then run your changes or your new programmatic modules on some of the pages, compare with the others and see, are users actually enjoying this? Your, your new programmatic creation. Are they finding value in it? Because if they're not, maybe you shouldn't release it as a site-wide thing. But if they are finding value and you are seeing users interacting with it, then you should release it. And most likely ranks will follow as well when users are happy.

I do have a little bit of a structured approach. It's a very data-driven approach. Uh, so the first thing is niche selection. So within every content strategy and any project that you do that relates content or new page creation in any way. You should have at least a general idea of your niche. What's crucial for programmatic SEO, but this is, I think, quite general for any content project, is you have to have an audience for which you are creating this content. If you don't have that and if it doesn't have business value, then why would you pull this information to the surface? Another thing that you should check, what you mentioned, the SERP. Love that. Absolutely. 100%. If you're going to create content just for the sake of having it on the site and just boasting about having thousands of programmatic pages, then there's no need. Align the content format with the SERP, understand what you're trying to create, and also see what information exists out there. So starting with a very automated SERP analysis, just to understand what are the different entities that are populating this niche that you're trying to target, how are they organized, who are your main competitors, how are they organizing their pages so that you actually know what you're up against. I think this first step in the niche selection is actually going to pave the path towards understanding how you with your company and your brand can actually enrich the information that's out there. So if you cannot say yes to this first question, are you going to make the search landscape better? Are you going to bring new information? Is it going to be unique to the users? Is it going to be high quality? Then this is the first step where you can actually drop out and say, okay, programmatic is not the way to go here. The second thing, if you do understand you have a good niche and a good enough understanding of how you can bring new information to the surface, even if it's regardless how competitive it is, my argument is even in the most competitive spheres, you can still do something new and original. You can be extremely creative. But the question is, can you do this at scale? So the second thing is actually doing keyword research to validate whether there is enough search for the different entity attribute keywords and combinations. So are people actually searching enough for your main topic? And are there enough combinations of different keywords and different things that they can know about this topic for you to actually create variants within different markets, within different combinations of keywords? You know, thinking about the Zapier example, nobody knew that there was a market for this until Zapier was created. So that means that And here you don't really have to focus on search volume because you might be working in a field where search volume is not really relevant because there's no search volume yet for this, or the tools are not recording it and so forth. But what you should be absolutely focusing on is the business value. If there's no business value, there's no need to create this page. So lots of steps, but I think the first 2 in understanding the niche, understanding the market, understanding the target users. And selecting the right entities to target is the most important one. And also just one last thing on this, what you mentioned about building pages ethically. One thing that I really want to highlight is we already have a problem with too many pages being created. And this is not only bad for the information landscape, Google search results are getting worse no matter what anyone says. And they obviously don't have like such a clear handle on things. Even though they're trying. So we also have kind of a responsibility whenever we are in the room to make these decisions to say, okay, this is not only bad for the information landscape that goes far beyond Google and like any search engine to say, but it's also bad for the environment as well. Because imagine Google continuously trying to scale their systems, crawling, understanding the web, indexing pages. And obviously we know what impact that has on their environment as well. And not only that, but if you're only publishing pages without business value, without targeting users just because you want to publish them, then what kind of internet does that create? An internet for nobody, right? There's just low-quality content.

Beyond that, it's bad for your crawl budget because I believe that crawl budget is a concept that a lot of people talk about when it's not necessarily that meaningful or relevant, but in this particular case, it is. If you are going to be creating hundreds of thousands, even millions of URLs out there programmatically, automatically, you want to ensure that the quality and meaningfulness is high enough. So those are actually meaningful for Google to go through and index them because of all of the challenges that you have mentioned before, Lazarina, we all know that they are not indexing pages or they are not indexing content as fast as they did before. What are the tools that you tend to leverage to develop a programmatic SEO approach? For example, you mentioned starting with, of course, really good keyword research. For that, I really like this particular functionality that the Semrush keyword research section has, the Keyword Magic Tool that allows you to see all of the different variations from a very specific topic and the search volume from each of the main combinations. So that is great. Then also Keyword Insights, it also allows you to create content clusters. Really easily. So you can validate very easily also the ways that it actually makes sense to combine the content and to connect the content with the search demand out there and validate what is doable, what is not, what is not that meaningful and where to start also to assess the value of this content that can be generated later on at scale too.

I find that with programmatic SEO, technical SEO becomes so much more important. So then I focus a lot on tools that can help with this. At scale. So a large website, you'll probably need some sort of log analysis that can scale up with your website. So you're still able to understand which pages have value, which pages have not. Is Google maybe crawling something that you don't want crawled and you're wasting crawl budget? Because when, when you're talking about millions of pages, a bit of waste somewhere in a corner of your website might actually become quite a lot of waste and you want to solve that quickly. So enterprise-level log file analysis, enterprise-level crawlers. Now, don't think you have to crawl a website that has millions of pages. That would be insane and probably so costly it's not worth it. And there's that 20/80 rule that works here as well. You can crawl a part of your website, not necessarily 20%, depends how big it is, but even a smaller percentage of your website can tell you the full story for everything else. Because don't forget, with programmatic SEO, it's all about templates. So as long as you have a good sample of pages from each template, you're probably going to find 80% or more of all issues on your website. If you just crawl those, then you need data to know that your pages are working. And I would, I would always recommend go back to Search Console, go back to Bing Webmaster Tools. You were talking before about quality of pages, and it's always something so generic when we talk about it because it could be personalized in each situation and mean something different. And this is why we don't really touch on what quality is because it is going to be different for every website. But for instance, Bing Webmaster Tools crawls your website and gives you a detailed report of pages it thinks are not quality. So if, if 90% of your programmatic pages are in there, you're doing something wrong. You need to find a new way of doing it because search engines don't necessarily like it. And then with Search Console, this is where Search Console, it is one of the most important tools because if you have any errors, any problems, if search engines can't get to your pages, they won't be able to get to your pages in mass, right? Big numbers. So I would monitor Search Console like my life depends on it because if something goes wrong, it goes truly wrong. When we talk about millions of pages. So that's another one. And then I would say looking at individual keyword ranks when it comes to programmatic SEO, it is just a waste of time. We rank for so many keywords. I don't even have a finite number because it's probably going to be new keywords every day. So instead of looking at specific keywords, I would recommend looking at topics, trying to see traffic trends for parts of your website, if you can break them into to parts and not focus on specific keywords because you're going to spend all your money monitoring millions of keywords and that's just going to be a percentage of all the keywords you have. So it's not even going to be potentially as relevant. And lastly, regex is the best tool you have when you talk about programmatic SEO. It's free. There are courses out there everywhere. Anybody can learn it. It's quite interesting. But when you work with so many pages, you need to be able to find patterns and find them fast. You need to be able to segment your analytics, segment Your log files and your crawls, you need to be able to do all of that. And without regex, you just won't be able to do so. So I'd say these are my main top tools with, without which I could never do programmatic SEO.

If you're not at an enterprise organization, 2 things. Data for SEO is a suite of APIs for different uses, including SERP analysis, including Google Trends, including competitor analysis, keyword research, anything that you can think of. Highly, highly recommended, especially for these 2 steps that I mentioned, like validating your niche, understanding the topic, understanding your competitors. Uh, of course, uh, crawling, uh, competitor content, crawling your own database. You need something that identifies entities. I tend to fall back on Google's Natural Language API, uh, to do that. And of course, in order to work with any kind of scale in programmatic SEO, you need to have at least a general understanding of BigQuery because, uh, Google Sheets is going to fail you. Quite fast. So you need to be able to merge together data, you need to be able to analyze segments. So knowing a little bit of SQL can go a very long way. And also BigQuery is great because you can analyze your datasets directly with some of the APIs that Google has, which most of them, by the way, are extremely good and extremely undervalued in our community. Zapier for no-code automation. Zapier is also very good If you are building systems, Alida, again, I was very happy to be in the audience whenever you gave a presentation at Brighton SEO a couple of years back about building systems who don't have any fuck-ups or something like that in SEO. So highly recommend it. Just to build with programmatic SEO, you need to have the systems that tell you this page, the content is not good, the title is not good. This is falling in terms of traffic. It's fallen by 10%. It's fallen by 5%. Half of your website has been deindexed. If you don't have the systems for that, it's going to slow For smaller sites, automation tools in terms of the generation of pages. For WordPress, there's a plugin that's very popular that's called WordPress Import All, and it allows you to actually download the content, upload it, make changes, and so forth. So manipulations that are— we're talking here, maybe the scale is below 10,000 pages, but that still can qualify as a very good programmatic SEO project if it's done correctly. And for internal linking, I recommend Link Whisper. Whisper, especially because there are automated features there that you can create rules for mentions and related posts and so forth. So you have a lot of customization on the actual internal links that are created.

And Lazarina, you mentioned before you were doing a programmatic SEO course. You meant you were creating it, right?

So the course was created for Traffic Think Tank. The course is published now and it's live. So it essentially has more than 4 hours of video lessons and more than 10,000 words created. It takes you through a step-by-step process.

This is great. So thank you. Thank you very much, Lazarina. Roxana, to start wrapping up, what would you say is the top programmatic SEO example that you would love to leave the audience with to go and take a look at? In my case, because I am also very into the remote work industry, I will say Nomad List is one of the websites out there that are showcased as a great example of programmatic SEO. In this case, the whole programmatic project is the full site in itself, but of course it doesn't need to be like that. It can be an area, a section of your business, already existing business, of course. But I think that this is a great example of how you can bring value into a sector.

It's really hard to pick just one. Pretty much any marketplace website you look at today, if you ever looked at one and thought, well, I like this website. They really know their SEO. It's programmatic. So that's your best example, I guess. Um, because I come from the stock photography industry and we rely heavily on the data we get from our photographers, I think it's good just to look at stock photography websites in general. My website included, but also my competitors. Everybody's doing a great job in this area. So study any of them. Try to kind of guess what rules do they have to maybe noindex pages that they think they're low quality value. If they're using nofollow selectively. Maybe think why and how that helps. Like there's a lot to unpack just looking at these websites and trying to kind of understand how they're applying programmatic SEO in a good way.

So I have 2 examples. So I have Canva. Canva uses programmatic SEO to create the Facebook templates or free logo templates and so forth. So they have a template that they use and they just replace it with the kind of resource that the person is looking for. And because their site is so visual, then they don't really need that much text to address the query. And because the query type is also with the intent that doesn't need that much text, they still have some at the bottom of the page, but that's another story. So have a look at that. Another company that I really like is G2 because they do an incredibly good job at pulling out information from their own database of companies. And actually doing really cool visualizations and kind of like a positioning map for the best companies out there.

I love it.

You have provided so many ideas, insights, input. I really, really appreciate it. Rosanna, Lazarina, thank you very much for today. It has been quite insightful. I'm sure that the audience will like it. If you like this video, remember to like the video, remember to subscribe to the channel to avoid missing them out. Have a wonderful day. Bye-bye until the next Crawling Mondays.