SEO Automation: What Can Be Automated in SEO? Status, Trends, Scenarios & Tools!

Welcome to a new, very special edition of Crawling Mondays. Today we are going to talk about SEO automation and all of the opportunities that automation offers. And for this topic, I have invited 2 people who I see pushing the boundaries of what can be automated in SEO on a day-to-day basis. On one hand, I have Ruth Everett. She's a technical SEO analyst at DeepCrawl. On the other hand, I have Andrea Volpini from Worldleaf. He's the CEO and co-founder there. Before starting, I would like to start today's episode with a very heartfelt tribute to Hamlet Batista, who unfortunately passed away. It was very shocking news in the SEO community, and of course we need to remember him Whenever speaking about especially SEO automation, he was very well known in the SEO industry. Well, for being a great colleague, very kind, always helpful with everybody, and especially for being the leader of Python SEO. He was the force behind this big trend that opened the eyes to many SEOs to a lot of automation opportunities.

I think for me, Hamlet was like a special A really special person. He showed everyone what was possible and inspired a lot of people to try something new. And he had a way of explaining really complex things, but he made them sound so simple and so accessible and got everyone really excited about things. And he was always happy to take time out of his day to help other people. You could message him about anything. And he never made you feel stupid or like that you weren't any smarter than anyone else because you didn't understand something. And he really loved amplifying others and like showing, sharing what people were doing and sharing everything in the community. So I think it's something that we need to carry on doing for him and like carry on talking about it, carry on sharing stuff, carry on inspiring each other and lifting each other up. I couldn't agree more.

And everybody, I think, that agrees and recognizes him for this, because he was so smart. He was there to help other people to learn more and to grow professionally. He was so selfless into putting his efforts on other people's development. Indeed. Andrea?

Yeah, it's a huge loss personally and for the community. I mean, Amlet has started really a movement. So we saw one person that became kind of the driving force behind a tendency that, you know, has changed the, you know, the business and, you know, the daily work for so many of us. And I was, you know, lucky to exchange so many weekly, daily, sometimes, you know, continuous messages about the work that he was doing, the work that I was doing. And I always remember is kindness and this amazing power of keeping things super simple. I mean, he was always saying, you know, Andrea, this is for the people that need to start learning. And it was so cool in doing this because that's how you create a movement. And he was terrific for this. I really, I mean, for me it was, it's really, it's been, it's a loss of a friend. It's a loss of a, you know, a source of inspiration. for my daily work. I mean, WordLift is a company that has been created for supporting SEO automation. I mean, I built the company on it. And we were so similar also because Rank Sense, it's a platform that automates SEO. It's probably the most interesting platform for edge SEO that you can find.

If you don't know about edge, you definitely need to take a look and watch The Edge SEO, who had Hamlet and Dan Taylor, because they explained what their platforms do. So I think the best way to honor him is by spreading the word, by sharing even more about what is possible, what is doable within this field. And this episode, it goes to him and hopefully it will allow that many people open their eyes even more about the opportunities within SEO automation, right? And for But before that, I want to ask you the first question because I think that this is the thing with SEO automation and why, as I was talking a little bit before, in SEO we always have this sort of black and white sort of stances with certain topics, right? And SEO automation, I think, is one of those, right? And I think it's unfortunately because of how some solutions or tools have been sold in the past, right? I have seen some enterprise-level SEO platforms selling themselves as not necessarily even having the need of an input of an SEO specialist to be able to work with them, things like that. So of course when an SEO sees this type of stuff, there's a little bit of concern. Then on the other hand, we have technology like GPT-3, right? Oh my God, we are not going to need copywriters anymore, right? Oh my God. So I think that this sort of concern— so I want to start the conversation today by asking you about SEO automation in 2021. What are realistically the opportunities? What is doable? What is not? How you will define SEO automation at this point? And which are those use cases and scenarios and opportunities where you see that it's actually a resourceful alternative or solution for certain time-consuming And resource-consuming task.

So for me, I guess why I love automation so much is that it kind of takes away the need for a lot of manual effort with work. And you can kind of spend that time then on other things which can't be automated or need a bit more of like a human touch to them. So you can automate repetitive tasks or like, for me, I use it a lot with data analysis. So being able to take a lot of data and like look at it, but like with efficiency and like at scale. So, and it can like back up the decisions that you're making and like give you a bit more confidence in the stuff you're saying. So kind of examples of how I've used it has been things like using APIs and Python to like automate things like using the PageSpeed Insights API. And then you can automate those, getting the insights from that rather than having to manually do it one by one. You can just upload a list and it can manually run through the API and give you like 100. And that's done it so quickly and it's given you those insights so quickly. And yeah, just being able to perform things that usually would take a lot of time or like you'd be kind of tearing your hair out. Trying to use Excel to do something and analyze something, but you can automate that process and just make things a bit quicker and a bit more efficient for you. That's kind of what I've— where I see automation for me personally. That's how I've been using that a lot and how I think is a good way to kind of get into starting doing automation with tasks that you do normally anyway, just seeing if there's a way you can automate them. And make your life a bit easier.

Automation in SEO, it's a big topic. And as you said, it's very controversial because I have a company and a product that is AI-powered. So I learned a lot of stuff and I have to make a quick disclaimer. I'm not from the SEO community. I mean, I got into the SEO community as I started the tool. I started to create a tool that resonated with the SEO community and solved all of a sudden I became part of this beautiful community. And so SEO automation, the way I see it, it's a tool that everyone can use to empower the strategy that you have in mind at every step of the process, whether it's about gathering the insight or it's about optimizing the on-page SEO or it's even about creating content that can glue one page with another. So automation, it's not, you know, a solution that replaces the human. Quite the opposite. SEO automation, it's something that will empower the human to do better, more things in different areas.

I think it's a wonderful opportunity for us to stop doing so many manual tasks that don't necessarily need our analysis. So we can actually focus on analysis. And speaking about different levels, I think that we can talk even about different levels or layers of automation, right? Like in SEO, we have been enjoying for a long time about task automations. So for example, whenever rankings dropped or traffic dropped more than a certain share or percentage, to be sent an alert that we can even glue or connect with Zapier or IFTTT. to be sent to our project management system and trigger a warning to someone to go and check out. Also tools also that have in the past allowed us to do things like integration of different tools, Google Sheets to put together and the generation of reports, for example, every month for our analysis, right?

Yeah.

So I think that this is like the first layer of automation that most of us have already leveraged and profited from at some point. But I think that the new layer of more sophisticated sort of automation that is coming is the one that, well, offers more opportunities. Of course, it's true that the prebuilt tools that allow us to do a lot of stuff already, but there are many other scenarios and it's important to have our own input. Like for example, the generation of titles or FAQs. Or internal linking, things like that that will need a little bit more training from our part, from our own particular context to make it meaningful to our particular scenario, right? So I would like if you could please share those use cases that you have found in the past. So for example, that you will say are the best examples of those new levels of more sophisticated automations, and in which scenarios you wouldn't advise to use.

One for me that's been quite cool to see and kind of follow how it's been improving is image captioning. So this was actually something that Hamlet introduced to me in 2019. And there's basically a model that can look at an image and caption it. It can see what the image contains and write a caption for it. And it's like something that's continually improving, so it's continually being trained on data in order to caption an image automatically for you, so you don't have to look through all your images if they don't have alt text, and it can caption it for you. And that's something that's like, still needs some human input, it still needs humans to help train it and to help it to learn, but like eventually it's going to get to a place where you can just caption your images, just put it through this and it will, it will give you a caption for your image. That's been a really cool thing for me. And then something that I saw, I think it was LinkedIn did for like a content quality thing, to basically automate like content quality checks. So to see, they took things like the word count, the traffic to a page, the internal links, the backlinks, and they're able to see automatically, give like a page a score on whether it's good for users and good for SEO, or if it's not and they need to work on it or they need to just get rid of it. So things like that have been really cool. And something that, I mean, DeepCrawl do, we do a lot of automate, like crawling and automating and reports, but you still need, like we have a tasks or an issues report, but you still need to go in there as a user and look at that data. Like it's not gonna tell you, like it's gonna tell you you've got this many broken pages or this many, whatever, but you still need to go in there and have a look at it. So I think it's, a case of like, there still needs to be humans going in there and checking things and being able to see, especially when you first get started with automation. I think it's good to have some kind of sense check and some kind of understanding of like how it's being automated, what's being done and like how it's working in order to understand it and help it get better. But I think, yeah, there's still a lot of things as well, kind of like strategy and like communication That kind of stuff can't really be automated. I mean, eventually I'm sure we'll get to a place where it's going to be automated, but it still needs us to have an understanding of it in order to be able to communicate it. And those, I guess, more softer skills that can't be automated. So we can let automation do all the boring stuff, and then we can come in and use that information that we get from it to do more human stuff and to be more communicative with each other and Explain things better than a machine can at the moment.

We talk a lot about the analysis or the problem identification area of SEO, that a lot of it can be automated, for example, with an SEO crawler and reporting tools, etc. But then SEO is connected with the business, it's connected with an industry, with an audience behavior that is changing all the time. And sometimes you cannot predict what happened last year. Even if you build like a model that will predict, for example, or project what is expected in traffic and already plan ahead. I don't know, the task in order to fulfill those needs. All of a sudden a pandemic happens. You can expect to not having to have a human interaction there to say, oh no, we need to analyze and change everything again to make it reasonable and to make it relevant for this particular scenario, things like that. And then on the other hand, more than understanding human search behavior is also influencing humans to implement stuff with the priorities that we have set. And of course, in an ideal world, everything should be super fast and efficient, and we should be able to show the data, this is harmful, please just click on a link or in a button in the CMS or the platform, and it will happen instantly. But it doesn't happen that way, so you need to build a lot of cases, a lot of analysis studies, projections, etc., to make that happen. And soft skills are particularly important. It's true that a lot of SEO tools right now automate the warnings or the identification of issues, but I do see people using these tools are not necessarily SEO specialists or are SEO specialists who are just starting, right? And don't have this know-how. And then all of a sudden they run this link profile analysis tool and they get this list of links coming from damaging domains or this list of toxic links that have been provided automatically by this tool. And they get freaked out and they go directly and disavow. All of these links without even validating or analyzing, checking if these are actually damaging. So we need to take this with a grain of salt. Tools are evolving, but it's always recommended for you to validate, to double-check if the output that you get is relevant for your particular context and is taking into consideration. Exactly.

First, I think that a lot of the advancement in what is called narrow artificial intelligence, so what we have at the moment, The more you follow it, the more you understand that you don't have to be scared about losing your job because AI is still pretty dumb. And as also Ruth was saying, I mean, we have to train systems on human skills and intelligence. So every time that, you know, I start thinking about automation, what do I do? I mean, how do I work? I mean, I look at the work that that people like you, Aleyda, do or that others have been doing. And that's my starting point because there's no intuition in the machine. The machine is not intuitive. So I don't see this problem of automation taking SEOs out of the picture. I do see that SEOs are becoming something else. So I see SEOs becoming data scientists. I see SEOs becoming strategists. I see SEOs becoming workflow managers, as you said. I mean, soft skills, yes, super important. So I do see a transformation in the profession. Yes, I have to be honest. And the reason is that, of course, things change very fast and technology allows us to do things in a more simple way. I like to call it generative SEO, meaning that it's a technology that does things for you. So we see a lot of the SEO tools when we enter the market, market have been always keen on analyzing and providing insights. This is very valuable. I think now there is a new wave of tools, as I was saying. I mean, Rank Sense is one, and definitely, I mean, we also do our part where we also try to act on the insight with small things. So I mean, something like image captioning. 2 years ago, we created the first extension for Wordlift to automatically describe the image. Using computer vision and natural language generation. So this is one task, but where we are really successful is in automating structured data. So the idea that a machine like Google needs curated data in order to provide better results to its user, it means that we can use the power of AI to create these structured data. Now, assuming that we have someone like Aleyda that does the strategy for where it's good to apply structured data, because I mean, we don't need to overdo it. We don't need to apply structured data everywhere, but we know that if we use machines to automate structured data, let's say on an e-commerce website, well, we can create a lot more sales. This is a fact. And there is no point in going through the process like we do now of dealing with the templates and Talking with the developer team. I mean, we can take these out of the organization and provide a very simple solution that with JavaScript on the page will automatically add, or with a plugin if you are running WordPress, will automatically create these without the IT team to start thinking about how to do it. We do it before them. Sometimes we do have this debate because they say, why can't we do it? And we say, yeah, try, do it. I mean, because it's also something that changes over time. So you want to automate things where also the change management is a cost for the organization. You want automation in this context, once you have designed the strategy for that client. And then, and then you want automation to do the things that otherwise would require, you know, complex interaction with technical teams, but also you want automation to keep things on track. over time. So you want to predict if things are going to go in one way or another. Now, can we predict SEO? Really not, because I mean, as Aleyda was saying, all of a sudden pandemic arrives and we don't fly. So there is no need for flight pilots or something else. So things change very fast, but automation allows you to keep things under control, on track. And that's, that's also when you want to use it. But remember that every form of narrow artificial intelligence will require humans. Artificial intelligence is the most human-intensive technology that we ever invented. That's a fact.

And at the end of the day, what is produced is only as good as what it learned from. So it will likely be biased or not necessarily as accurate as we want. So it's important to Do a little bit of validation, good training along the way, right? It's not a one-time type of—

It's fundamental, but it's also part of the workflow. I mean, right now we are, of course, working on automating FAQ and we're using GPT-3, and Google is coming now with a new, you know, billion-parameters language model that it's kind of the evolution of T5. It's going to be even better than GPT-3, but Nevertheless, you know, even if, you know, I'm able to train it with the best content possible that I have created in the past, which is what we try to do now, you know, if we don't plug in the human, so if we don't implement this human-in-the-loop type of approach, the value, it's not going to be rewarded. It's the interplay between the human and the machine that creates the magic. It's not the opposite. It's not the technology that replaced the human. No, it's the interplay. That creates the magic. And it's not easy because, you know, how do I engage an editor with automated writing? It's not easy. You know, I spend a lot of time talking with them and say, hey, you need to review these. Right now, for instance, what I do is I take the FAQ from Google and then we take the response that Google thinks is the best response for the answer, and then I use it as a priming for the model to learn How to produce an even better answer? Still, it's not enough. Still, it might not fit with the tone of voice of of the brand. So I'm just creating a first draft. It's important, but it's a first draft. Nothing more.

Thank you very much for bringing this up because I think it connects well with the next topic. Speaking about content generation, I have been in a lot of internationalization process when nowadays we are leveraging something that a few years ago I cannot. Expect to even suggest this to clients, like DeepL, for example, as a translator that is so, so, so, so much better because it uses neural networks, because it— how it works. So the quality of the translation is superbly good, and they offer integration with the CAT tools that are the ones that human translators will use when translating content. So they use that initial translation as an input for them to later on indeed align the tone of voice with the brand, include the relevant terms that we know that we want to rank for those particular topics. I wanted to ask you, what would you say are the first steps or the first, like the steps that you follow when you start with an automation process? How do you select the language? How do you You identify, oh, there is already an API for that, to make it a little bit more, let's say, actionable for those out there who are just starting and would like to start automating stuff.

I think for me, the journey that I usually take is like thinking about what can be automated. So especially when you're doing like really tedious tasks, like just think like, is there a way that this can be automated? And just, I guess, like, almost making yourself think, because if you don't realize that things can be automated, you don't think you could automate them. So it's kind of just like changing your behavior, like how you think about things of, can this be automated? Like, is there something you're trying to solve? Is there a problem you're trying to solve? Can this be automated? Can you get some data that can help you do this? And then, yeah, just look if there's— see if there's resources that you can use. like you say, APIs that are already built or something that someone's already done, a tool out there that can already do that. And I think, yeah, it's like we've been saying, it's still important to have that sense check and to have that understanding of what is happening. So you can see a tool that does all this really cool stuff, but without really understanding how it works, it's kind of like you're just letting it do its thing. Having that understanding is like, it's really cool to know how it's working for one, but also it helps you to understand how things can be automated and what you can do with it. And just, yeah, having that human touch on it still. Like we have a tool that you can automatically do like SEO QA before you push any code live, but we would still have to go in there and say like, if you have this many noindex tags or this many broken pages on the staging site, don't push it live. But we still have to go in there and tell it, like, it reaches this threshold and then it can't, and then it won't go live. Like, we won't be able to make this change. But we still have to know how it's working, know what it's doing, and be able to inform it of, like, where it needs to stop and how it needs to work. So I think, yeah, just having an understanding of how you can do it and playing around with things that other people have done. I think that's a really good way to get started because just seeing other people do it.

There's a lot of open source code there.

Exactly.

Yeah.

Yeah. Yeah. So yeah, I think that's probably my top tips for getting started, just playing around with what's already out there.

And about languages, I know, Ruth, that you particularly started with Python, right? I don't know how you feel about other programming languages or which was your particular case, right, for starting with Python in particular.

I don't— I was just like intrigued when I saw things that people were doing with Python, so I started learning it. But it's the libraries, I think, with Python. So the machine learning libraries, the natural language processing libraries, being able to like use Python as the language but have all of these libraries on top of it and do even more stuff with those have been really cool because they're still— they still run using Python, but they all have different things that they do. And you can use them in different ways. So yeah, I really only use Python, but it's the different libraries that do the different things on top of Python that I use for a lot of the automation stuff.

I think you always start with data gathering. So you always look at, you know, the insights that you have that can trigger, you know, some potential opportunities. So you want to seek opportunities and you want to do it, you know, with the eye of your SEO knowledge. So you want to use your SEO wisdom to look at the data and to seek for the opportunity. Then you want to kind of pilot the process on a small batch of URLs where you can use this technique which is called stratified sampling and kind of split a test group from a control group. So sort of an A/B testing where you can prove the effectiveness of the automation. Because there's nothing worse than scaling up something that doesn't work. So you want to test these on a smaller scale and then you want to scale it up if you see that it's successful to more pages or even more sites. So that's the way. Now in programming, it's whatever you feel comfortable. There is no programming language. I mean, programming, it's about talking with the computer. So if you want to talk with the computer using your natural language, there are options. I mean, I did, I think, a test with GPT-3 about writing schema markup using just your natural language. So you just tell the machine what type of output you want, and then it will produce it for you. So yeah, programming language is not really an issue. I mean, there's no there's no machine learning and data science. It's somehow Python-centric. But in reality, I mean, you have to use whatever you feel comfortable. Now, in our case, we typically have these VIP programs where we have some clients where we can kind of work deeply on the integration. And this is kind of our gym. That's where we test things out and we come up with new ideas. And as to be innovative, as to create kind of a repeatable success for these clients. And then we consolidate these into APIs for Wordlift that then we can bring into the product. That's a little bit of the lifecycle. So we now have different APIs for content analysis, for content generation, for structured data creation, for reading a web page and getting the attributes out of the web page. So we are creating this toolbox of APIs that soon will be available for other tools makers. In order to build their own Wordlift or their own tool. And I think you don't need to see yourself as a coder. I mean, you need to see yourself as an SEO strategist and you have to seek the opportunity and then you know that the technology will come and will do some of the work for you or with you.

Indeed.

What I think is important is always to be curious. Identify opportunities with the technology available sometimes. Things are not going to be pre-built enough or won't have like a user interface, so you will need to go into the code. And if this is something that is particularly attractive for you, because it's something that you really need to enjoy, of course you can definitely— there are so many resources out there that you can teach yourself to do that, or there are so many online courses for Python or whatever other language that you want to learn. And of course, like now there's even a website that Charlie Wagner is building, like SEO Pythonistas in honor of Hamlet, that has tons of resources with a lot of articles, all of Hamlet's articles and also articles from everybody in the community. So hopefully it will become a hub of learning of many new SEOs who want to learn how to do this themselves, right? So to start wrapping up, I wanted to ask you to share a last tip, a last message, or a last advice to anybody out there who is starting or looking to automate something in their SEO day-to-day?

I think kind of going off what you were just talking about, looking at what others are doing as inspiration. So just seeing what other people are doing, getting inspired by what other people are doing. People like Hamlet and this website that Charlie's putting together, that's just going to have like people share this stuff because they want to inspire and encourage others. So just look at what other people are doing and get inspired. Encourage others, see what's out there and see what you can do. Don't obviously copy what other people are doing, but just be inspired by them and use the kinds of things they're doing because they're putting it out there for you to use. So yeah, just get inspired by other people's work.

Amazing. Open source is a wonderful thing, right?

Exactly.

You can start building. I mean, you don't need to build a building just by yourself. You're already, already up to floor 10.

Yeah.

You just need to continue taking it from there. So yes, it can be really powerful indeed. Andrea?

Yeah. So I mean, just, just to conclude and to remember Amlet, he wrote a terrific piece on title tag optimization that was really inspiring. It was probably the last, if not one of the last articles that he's written about using T5 and Ludwig. This is a library that he liked a lot because it simplifies a lot of the integration and the fine-tuning of a language model. And I would recommend you to go through this article that Amlet has written. I read it myself a few times because there's so much in it and it's so simple the way he explains it. So if you want to start today, do something for bringing self-automation forward, read this article about title tag optimization by Hamlet Batista.

Amazing. Amazing. Thank you very much for your time. I really appreciate you sharing your journey, sharing your insights on learning, especially in a moment like this. I think it was going to be a very lovely tribute to Hamlet too, because this is the type of stuff that he was always looking to do, to share and Give enthusiasm to other people and empower other people to learn much more and to start automating much more. So don't wait, as he said, of course. And thank you, thank you very much for joining today. And if you like this video, please like this video, subscribe to the channel, watch more. And if you have any questions regarding SEO automation too, please follow Andrea, follow Ruth. They are very active on Twitter, in the articles that they are sharing everywhere. So look for them. and learn from them. And thank you very much for watching today. Bye-bye.