The Truth About AI Crawlers and Publisher Content

Hi everyone. Uh, welcome to, uh, the 5th installment of Beers and SEO with Barry and Steve. And I'm Steve.

And I'm Barry.

And we've got a lot to talk to you about today, haven't we, Barry?

Oh yes. Yes. We're going to be talking about AI and the impact on news and publishers with some tips and advice on how to handle optimization for AI search, if you're interested in that. And also maybe talking about AI bots, AI crawling, and what publishers might need to do to protect themselves from that.

Yeah, so it really is a bumper edition. And every week, me, before we record, me and Barry have a list of things that we're going to talk about. And I always front-load it with the entrée, some things that we discussed before the main course. And the main course is pretty substantial. So stay tuned and get your notepad out or Notepad LLM. Right, so Barry, first of all, the most crucial item is what are you drinking today?

Today I am drinking Brew House Hoppy Days. This is actually from a brewery in Belfast, in East Belfast, close to where I used to live before I moved to Cookstown. They have a lovely own brewery. They have their own tasting room. And basically it's a glorified pub and a bit of a hipster place, but who doesn't like hipster places? This is a hazy IPA and I haven't actually tasted it before. So this is my first sample of it.

Please feel free to complete the sample.

That's delicious, actually. That is really, really nice. I quite like hoppy beers, and this one is hoppy without being overwhelmingly so. Oh, this is gorgeous. I might have to drink this one more often.

I think we need to get you onto advertising, Barry. I think that's really, really good. Myself, I'm on a Northern Monk variety, so it's a classic. I mean, this Leeds brewery, You know, I probably late coming to this, but I discovered it by using one of those kind of like you know mailing beer. The convenience of beers being delivered to you. I've never really sort of like stopped appreciating that. But yeah, I've got a I had at the time a selection of Northern Monk beers and the Eternal One. I mean, it's a session ale. It's it's for me now. I'm You know, I think gone are my days of the kind of like European beers where they're sort of like 7 or 8%. And—

I can't do that anymore either. Like 5% is sort of my maximum nowadays. And after that, you're just taking the piss really.

I mean, it was fun discovering all that, but yeah, if you've got to go to work the next day, it's not so great. So, right. So the entrée. So there's been like some really, like every week, I mean, most of our jobs now working in SEO and AI visibility, right? I mean, a lot of it is just keeping up to date with everything that's being discussed and new reports and new analytics. It's, you know, I find my, you know, what used to be like 20 minutes before the working day just catching up. I now have to book in like an hour just to go through everything because particularly over the weekend when it drops stateside.

For me, it's about 20% of my time is just spent reading stuff, trying new tools, Staying up to date on the latest. I mean, SEO has always been an industry where change is just part and parcel of what we do. You know, we always have to adapt to new strategies, new tactics, new Google features, new context where search is important. But the last few years has been a bit more extreme in that regard.

Yeah.

And I find myself having to dedicate a lot more time to just staying on top of the latest things, which I don't actually mind doing because I like reading about this stuff. I like our industry. What I don't necessarily enjoy is the stuff that was cutting edge a year ago is now probably, you know, ancient history already. And you always have to redraw the lines about what advice you can give to clients. And for example, I have a rolling PowerPoint deck that I've presented several times to clients about the state of AI and the state of user behavior in search. And if I look at the first version of the deck to the current version, it's entirely different and it's only been a year old. Whereas My standard SEO training decks, how we actually do this stuff for day-to-day optimization hasn't really changed that much, but the context in which we do that has drastically changed. Yeah.

And we're going to come on to a bit on that. But looking at like the last couple of weeks, I think there's a couple of things that we might want to comment on. So obviously there was the March core update.

Yes. Now we had 2 updates. We had the spam update and the core update back to back.

Yeah. So, uh, how impactful do you think that's been, Barry?

I haven't really seen significant shifts off the back of it, neither the spam update nor the core update. I know some people said they saw some shifts off the spam update, and there were some minor fluctuations off the core update, but I can't really say that we can isolate the core update as being the culprit there, because those fluctuations tend to fall in the regular ups and downs of the news cycle and traffic deviations anyway.

Yeah, I was speaking to some news publishers, and without naming names, somebody said, reporting back to senior level, some of our sites went up, some of the sites went down. And yeah, we have these fluctuations, but there doesn't seem to be— I mean, I know others like Alayda, and there's been many other sort of in-depth analysis on various different sorts of outlooks on it. It's always useful to look at who, you know, winners and losers. It's always important, but it seemed that the winners didn't win by much and the losers didn't lose by much, although at least in the news publishing world. So we'll see.

Yeah, fairly uneventful one, which I think we should be very grateful for. It more or less bypassed the news industry in a meaningful fashion. So we can relax a little bit and breathe a bit easier, maybe.

Right. So on to our second item. And this is, do you know what, I normally have like little notes to support each of the, each of the talking points that we're going through. And I've just got in exclamation marks, the back button. So this is all about Google releasing, basically now saying that publishers who are hijacking the back button when you're visiting their sites. So by that, you know, it is It is a terrible user experience to call that out immediately. But, you know, for those that may not have ever experienced it, you, you attempt to leave the site and then it will throw up one additional page of you might also want to read this. It's so really, it's like 2 clicks before you have to get out of that site. So Google are going to come down heavy on this, aren't they?

Yes, they've given us 2 months notice before they start rolling out manual penalties. And I've seen back button hijacking appear in multiple different forms. Like the form you set, you click the back button, you get an overlay before you're allowed to leave the site. Other websites are even more aggressive where the back button doesn't actually go back at all, but it just sends you to an entirely different article. So basically inflating their page views by just making the back button do something else entirely, as if you're clicking on a link. And I'm actually a huge fan of this particular spam policy. I have had issues with Google's new spam policies before, like Site reputation abuse, I think, was a massive overreach and I do not agree with it. This Google spam policy, I am a huge favor of because it is actually bad user experience. It is a dark usability pattern that some publishers, some websites have embraced as a way to artificially inflate the time on site, artificially inflate pages per visit, and has no benefit whatsoever for the user. And it's only there purely for publishers to make more money. And I think Google with its power it has over the web can make the web a better place. And it has done so before by, for example, pretty much forcing everybody to go to HTTPS, secure connections instead of HTTP.

Yeah.

And this is one of those things where Google is using its power to actually make the web a better place. So for this one, I'm all in favor of it. Having said that, you also sort of have to question the power that Google then has over the web, that they can make these policies and basically force the entire web to adopt their rules. Because if you don't, you lose your Google traffic. And that's a situation very few websites are comfortable being in. So while I like the policy, it also highlights the fact that Google probably has way too much power over, over the web as a whole. But you know, that's not something we want to talk too openly about. Lest we bite the hand that feeds, right?

Yeah, which is a great song by Trent Reznor, Nine Inch Nails. But yeah, right. Yeah, loads of SEOs and loads of publishers came out on kind of like, well, maybe not publishers, right? Came out celebrating this. Because, you know, just reading sort of like some of the comments around when this was announced, it was, you know, Some publishers argued that you've got to make money somehow, right? And so it's by advertising methods like this or, you know, audience, inverted commas, audience retention methods like this, you know, it pays the bread and butter for the real journalism to happen, right? But I can understand somebody who might be in the commercial department arguing from that. But, you know, I think it's our job as SEOs or as editors, writers, is that you should always be defending the audience experience, right? Because if that's not happening, then who is?

I have very strong opinions on this and I have had arguments with people about this. You know, people say, oh, we have to do this because we have to pay for the journalism somehow. Whereas there are some websites that have gone so far with monetizing their audience that the website isn't a journalistic outlet anymore. It's an advertising vehicle. And yes, I realize we need to make money. We need to fund journalism in one way or the other. Slapping these sorts of hyper-aggressive ads on it, and I include stuff that has this built in like Taboola, Outbrain, RevContent. They have features in their software platforms that enable things like back button hijacking as well. I do not agree that those platforms offer any sort of positive user experience. And that if websites are content or even happy to host these sorts of platforms and go hyper-aggressive with monetizing every single visit, it shows an inherent contempt for your audience, for your readers. It shows that you don't actually care about the usability, the experience that you provide to your audience, but that you want just to extract maximum value from your audience, no matter what the cost is to that audience, to those readers. And we have to find a bit of a balance there. Yes, we do need ads. We do need monetization. We do need to find ways to pay for the journalism, but there are things that you do that go too far. And websites that go too far are websites that, in my experience, tend to lose in the long run because people start building These subconscious associations that that website is not fun to use, not pleasant to use, and people will try to avoid going to those websites. Whereas I think if you still focus on providing an optimal user experience within the context of what you can do with monetization, you tend to fare better in the long run. You find it easier to build audience loyalty. You find it easier to maybe to get people to subscribe, to pay for your content, to register for the registration walls, all those other things that you can do to make loyal audiences. So I think publishers need to be very aware and push back on their own commercial departments that they don't go too far with injecting these nasty dark usability patterns that serve no purpose at all except to maximize money and decrease the actual value of your website to your actual audience, the actual readers that consume your content.

Yeah, I guess it's really down to if audience teams or data teams or UX, you know, product teams, if they're empowered and have got an equal, you know, equal level with the commercial team, because, you know, sometimes it may just be that the commercial teams, they've got like the primary communication channel with the powers that be. And, you know, you really need to be listening to the other specialists in the room that really know the audience, right?

It's a leadership challenge. Leadership needs to set these boundaries and needs to build that internal company culture that makes sure that everything that the company does to improve and to grow doesn't go too far in building bad experiences for their audience.

Okay.

Agreed.

So yeah, so really there's been a couple of reports, particularly in the New York Times and something which Lily-Rae is also commenting on around AIO accuracy or lack of accuracy. Do you want to talk a little bit about that?

Yeah, there was a New York Times article where they did a fairly small study that showed that about 1 in 9, 1 in 10 AI overviews was inaccurate to various different degrees. And they wrote an interesting piece about that. Google fired back and said that these were not representative searches, part of the study, and they were deliberately crafted to elicit bad answers, inaccurate answers. And Lily Ray responded to that Google response saying, actually, no, these are fairly representative. This is normal user behavior or fairly reflective of normal user behavior. And the accuracy rate in AI overviews is about 90%, which means that, yeah, 1 in 10 AI overviews will have a bad response in it somewhere, which shouldn't really surprise anybody by the very nature of LLMs. The hallucination problem is not solved, can never be solved. It's inherent in the technology. So of course AI overviews are going to be inaccurate, but Google wants to safeguard its image. And personally, I don't think the media make enough of the fact that AI overviews are inaccurate. They don't show enough examples and people just take these AI-generated answers as gospel and make very important decisions sometimes based on the information provided to them by an AI overview or by a ChatGPT answer or by a Perplexity answer without necessarily double-checking it. So I think we probably need to give this more attention and tell people, yeah, okay, it's very useful, it's very handy getting an answer from an AI, but at the same time, double-check it, especially if you want to make any important decisions based on it regarding your finance, your health, your relationships, and don't accept it at face value ever.

Right. I mean, it's all about regulation, right? I mean, it's like using a food example, a fast food example, right? It's like to clearly label, it's Well, maybe not food, maybe cigarettes, right? It's kind of like late, but that took many years to put that kind of warning on it. But it feels, I mean, we're sitting now looking at, you know, looking at how the search results look like, which we're familiar with for many years. But across all these services, there should be some kind of warning, right?

Yeah.

Or treat with caution.

A health warning.

Yeah.

I also see the other side of the coin here where if you go to a social media website, TikTok, Instagram, Reddit, YouTube, and you see something said there by a human, you tend also not to necessarily take that at face value, at least you shouldn't. The difference is, of course, that we have been trained by the last 25 years of Google to sort of trust their responses, that there's a certain level of authority built into what Google shows you. And with AI, especially with the confidence that AI presents its answers, it tends to really throw people's bullshit detectors off and they tend to just see, oh, it's said with such confidence and it comes from Google, therefore it must be true. And I think that's something that we need to maybe teach people how to handle better, that okay, even if it's presented with confidence by a trusted brand, doesn't necessarily mean it's actually true and you might want to dig a little bit deeper before you do anything with that information.

Critical learning, critical skills. We all need to sharpen up on them, right? So kind of like, I mean, that brings us directly into AI, right? Into the main course, if you like. So, you know, it's been a huge learning curve for myself. And I assume you, as you've said, Barry, You know, there's something new to learn every day. But just from the top of my mind, I saw a post recently which didn't really comment on the kind of the latest report or the latest findings. It was just asking a selection of 10 to 15 experts how they are coping with all the reports which are coming out, right? Because even if you thought like, I remember when I first started studying SEO, studying Google, you know, there is no— I mean, obviously you can read Google documentation, right? But there is no central resource where you can learn everything. It's a much more complex thing. You've got to read the latest studies, you've got to network a lot, you've got to kind of like, you've got to test, see, you know, Google might say something, you've got to put it into play and see what actually happens. There's a holistic approach to it, right? And now you've got AI visibility. And someone is telling you every hour, every minute on LinkedIn what you need to do, right? So that's what we're here is to kind of like, kind of give you some guidance on that. And with that, I'll hand over to you, Barry.

I think there's 2 things we want to cover today, which is AI crawlers, AI bots and scrapers, and how publishers should maybe interact with those. And Then talk about whether optimizing for AI is a fool's errand or maybe worth doing in the first place. I want to start with crawling and how AI crawlers are currently behaving, because that's also a bit of a hot topic at the moment and is also an accelerating topic because the AI bots are getting much more aggressive. I've seen some interesting data, interesting reports, as per what you just said, because this stuff just keeps flowing out at like one report a day. It's hard to stay on top of, but this was an interesting one from Tollbit, which has a state of the bots report that they do every quarter. And for the last quarter of 2025, they had an interesting report that showed that there's very aggressive crawling happening by AI bots, primarily ChatGPT. And most of that is not actually training data. There's 2 types of crawling generally that happens, increasingly 3 types, but let's talk about the big 2 types, which is a crawling for training data, which is how AI tools gather content to either train an entire model or fine-tune and improve an existing model. And then there's retrieval augmented generation, RAG, which is when someone actually does a prompt on an AI system like ChatGPT and ChatGPT goes and basically searches the web for relevant pages and then retrieves them and summarizes them and provides them as an answer and shows them as citations. The training data crawling is less than 1/10 of the retrieval augmented generation crawling, whereas it used to be the other way around. But that also has to do, of course, with how many more users have now adopted ChatGPT and other AI bots. So there's much more usage. So there's less need for AI tools, LLMs, to train their models. They already have the data to a large extent and just keeping those data, those those training data is up to date isn't that intensive a process anymore. Whereas the sheer use case of AI tools means that, you know, there's tens of thousands, hundreds of thousands of prompts happening every minute. And therefore there's a lot of retrieval augmented generation calls being done. And then there's a third layer of calls, which is the AI agents, but it's still a very early stage. So it's not a huge activity there for now. So we don't really have to talk about that too much. What I find really interesting when I look at those, those sorts of reports and the amount of crawling that's happening in that it tends to outpace the crawling of classic bots like Googlebot and Bingbot. And more importantly, that crawl behavior from especially ChatGPT does not seem to align to the value that the crawler generates for the website. As in, yes, okay, you get crawled 150,000 times by ChatGPT and you get 12 visitors. Whereas if you get crawled 150,000 times by Googlebot, you get a couple of thousand visitors. Usually the value exchange is very different, which makes it very worthwhile for websites to allow Google to crawl them and index them and use their content. Whereas the value exchange for ChatGPT is less clear. It really means what the goal is of you being shown in a ChatGPT-generated answer. We can talk about that a little bit later as well. And you have methods of blocking certain crawlers on your website, but those methods are not necessarily foolproof because there's certain workarounds that the AI tools use. Again, to stay with ChatGPT for a moment, they have the main retrieval augmented generation crawler is ChatGPT user, which is what they call user instantiated, which means they only trigger a call when a user asks for a specific prompt. And that also means that ChatGPT feels it shouldn't necessarily respect your robots.txt disallow rules. Because when you browse web pages in a browser, you also don't respect robots.txt. If there's a link to a page, you can just visit it in the browser, whether or not a bot like Googlebot has access to it. And ChatGPT extends that to its ChatGPT user crawls as well, because the user instantiated, they basically see it as sort of a browsing session and therefore should be able to load the webpage. Of course, there isn't actually a browsing session. The page doesn't actually load. It's just ChatGPT downloading the HTML, extracting the content, regenerating it in an answer and serving it up to their users. So again, there's no value exchange. You don't get a visitor to your webpage, whereas in a browser, of course you do. So I think that's a little bit of a sneaky method. But you can still probably block ChatGPT users if you want on a hosting level. But then there's an extra layer, a really, really problematic layer, I think, is that there's a lot of third-party crawlers, like a lot of them, that will basically scrape the web, everything they can find, and using user agents that you probably have never heard of. And they probably change user agents all the time as well to mask themselves and become different crawlers in your log files, but basically all the same crawlers. And they gather all the data and they then sell it to AI companies. And there's a lot of examples of that. There's a few posts on LinkedIn I saw about it as well.

And so I mean, that's a key, that's a key thing, right? So you may have, you may have some established AI companies who say, we don't do this. But then there's the acquisition of that data. Where do they get the data from?

Yeah, the AI company says, yeah, you can block us in robots.txt and we won't crawl you. Great. But then they still get your data by just buying it from other crawlers, from third parties. So you're not really blocking anything. And again, the crawling behavior is very aggressive, tends to ignore robots.txt also because most robots.txt files will not have specific block rules for those user agents because they tend to change user agents all the time, or they might not even know that they exist. And some of them just plainly ignore your robots.txt file anyway because they don't care because there's no legal repercussion for them to just ignore your block, your disallow rules and call you anyway.

Yeah. And I found, you know, some clients have come to me and they're quite confused about what they should be doing with the robots.txt file, right? So, you know, it's, you know, it's a, it's It's not a directive. And as you say, there's many ways that the bots can bypass you.

Yeah.

Um, but I still think it's a useful thing for publishers to do. You know, if for those that do not know, please go and check out the robots.txt file. So it's literally domain/robots.txt. Um, many publishers, particularly who might be on WordPress or other kind of out-of-the-box, um, CMS solutions may never in their lives have to look at a file like that. But, you know, that's where you— that's why I think it's a really good discussion for publishers who may not have explored what they should be doing with AI bots is to start somewhere. And starting is looking at your own robots.txt file, or go to a publisher like the New York Times, you know, a well-established publisher, look what they've done and try to explore why they've taken those decisions. That doesn't mean that you copy and paste, because they're there's key decision-making beside each disallow entry, right, in there. But I found it a useful discussion when there's so many options. Well, let's just start with this. And then it's— then by discussing what's the difference in the bots, which are, you know, which are training bots, and which are the ones that we want to keep to our site, like Googlebot, right? We don't want to disappear from the SERPs. So treat with caution again.

Yeah. And I've always been a fan of just allowing the bots to crawl and then disallowing the ones that don't behave or that you don't feel have a positive value exchange. But now I actually think we probably need to flip it around because bot behavior is so aggressive. I think more than half of the web traffic is now just crawlers and bots. And the bots are so aggressive and basically devious to a large extent that I think we now just need to block them all by default. Uh, and yes, do that in robots.txt and then Have allow rules for the bots you do want to crawl, Googlebot, Bingbot, some specific other bots that you think there's an added value there, and then just block everything else. And that will actually stop most of the bots because they tend most bots do tend to be polite crawlers in the sense that they will adhere to robots.txt disallow rules. But there's also plenty of bots who just ignore it and crawl anyway, which is where hosting level blocking comes in, which is not necessarily. robots.txt, but your server, your hosting environment, your CDN detecting bot crawls and saying, you know what, you're a bot, I'm going to block you unless you're specifically allowed. And CDNs like Cloudflare already have these things enabled. And I think more CDN providers will follow. And it, again, it's one of those symptoms of how the web has changed from an open, free access ecosystem to increasingly walled gardens. And This is all another extension of that where, you know, and I sound maybe a bit old-fashioned or slightly depressed of this because I remember these old days of the web, Steve, like you do, where it was just a free-for-all. It was like the Wild West.

Fake memories.

You could build anything, launch it, and it was good. It was happy. If a little bit of a cowboy environment as well. Whereas now everything is going behind walled gardens, paywalls, logins, you name it. And even You know, access to content by machine systems is now being blocked very aggressively on the hosting level because we have to, because it's self-defense, it's pure protection, because the economics of the web have now been skewed in such a way that giving away your content for free without any safeguards is not commercially viable in any way, shape, or form. Unless you're happy to give it away. And there's plenty of people who are perfectly happy with giving that away. But if your business revolves around the production and distribution of content, you need to have safeguards in place. And hosting-level blocking is the next step after paywalls, after registration walls, all those things. We now also have to block bots in a very direct, aggressive way on a hosting level. And only, I feel, allow Bots to crawl your website where you know you have at least a good chance of a positive value exchange, and that means search engines like Googlebot. And we can complain about Google's diminishing traffic to the web all we want. It is still by far the largest traffic source to the web with with no no real competition. So yes, you should let Google crawl your your content and index it to show it in search results, to show it in the Discover feed, things like that. Some other smaller search engines Bingbot. Let's go into the Yahoo. Whatever other crawlers there are, DuckDuckGo, Baidu if you're in China, great. And then when it comes to the AI systems, you need to have that internal debate whether you want to be part of the training data and be shown as a cited source in their answers with retrieval augmented generation, whether you just want one or the two, no training data, but we do want to be cited as a source, or if you want none of it, And then block all the AI bots by default.

Right. I mean, just on that point about being in training data, right? I mean, this is not a quick fix, right? So, you know, so if you're in the training, if you're in training data, you've almost got like a pole position, right? Or visibility in there.

Not necessarily. The training data is useful to a certain extent, but because nearly all the prompts now also rely so heavily on retrieval augmented generation, the training data is basically just there to do determine the quality of the model in the first place, how good the LLM model is in terms of the quality of its output, the accuracy of its output. I'm actually more okay with being part of the training data than part of the retrieval augmented generation nowadays, because the training data, just because the LLMs model are out there, the technology is there, and we want these tools to be as good as they possibly can be. We're not putting Pandora back in its box. So, okay, let them be trained on my data. I think being part of the training data also means that you can build associations, that if you have a strong brand in a specific area, for example, SEO, and you're mentioned in the context of SEO very often, I think in that initial generated answer that is not augmented, that purely comes from the LLM, there's a higher chance of you being cited in it because that association already exists. Then the retrieval augmented generation where the crawler or the LLM actually goes and, right, has a generated answer. Let's see how accurate it is. Or some LLMs do it the other way around. They start by retrieving and then summarize it. That's a different sort of value exchange. Then you have to decide, right? It's a lot of crawl activity on my website. There's a lot of load being placed on my server by these retrieval systems to get me a mention in an answer in some context or another. And then you have to decide, is that worth it? Can I attach a value to being part of that answer that has a positive economic value for me somewhere down the line? And that's a much more difficult question to answer at the moment.

Yeah. You know, it's really interesting what you say because I was reading an article on Digiday by Sarah Gielloni, who was writing a response piece to this information out there that third-party bots are scraping websites like mad. She had this great quote, which I've got here, which is, she says, most brands have no idea whether they're being mentioned or ignored by ChatGPT, Gemini, and Perplexity when, when users ask questions in their category. There's no rank tracker for AI-generated answers, no impression data, no notification when a competitor gets recommended instead of you. The result is that most content teams are investing in AEO based on theory, not measurement. They're publishing content, optimizing for AI search, and hoping it's working with no reliable way know whether it is.

Yes. I mean, there's a lot of talk about prompt tracking and its accuracy. And the truth is prompt tracking is just not accurate in any way, shape, or form. There are some studies on this actually that show that you can do the exact same prompt 10 different times in the same AI bot and you get 10 different answers, 10 different set of citations, 10 different brands mentioned. Now, there are some consistency signals there for some brands, especially brands that have very strong positions in their market. And therefore they will always be mentioned in one way, shape, or form as part of a particular answer. But prompt tracking in itself, I feel, is such a colossal waste of time and energy because it doesn't really give you meaningful data. In fact, it gives you bad data and bad data, in my opinion, is worse than no data at all. So I think, you know, if you're currently paying a lot of money for prompt tracking and whatever, shape that that takes for you, I recommend stop that. Don't do that. Maybe do some experimental prompts now and again if you want to show some data to some senior-level executives in your company. But as a whole, the exercise is fairly pointless because of the inherent instability of an AI-generated answer.

Right. So, I mean, other options. We will go into a list of things that you can do from the skills that we have between us. But I would always call out, you know, Obviously, having a having a view, a strategy, an AI bot strategy is obviously a key thing for publishers to do. But it's not the only thing they can do. And so I would always argue that they should wherever possible publishers in general should join part of their collective bodies to have more of a clout. Now there are many options out there. We've got the PPA, the AOC. You've got the new Spur Collective with the likes of the Guardian. Financial Times, etc. All these are great steps forward. But I wanted to recommend here Ricky Sutton's podcast Future Media, where he was speaking to an expert that was previously a Danish MP who is part of a new collective there. It's a really fascinating podcast because Because it just demonstrates the power of working together as a collective. There are obviously less publishers in Denmark, but they've all got together and they're making a difference, partly because of how the political system is set up. But there just seems to be a moral agreement among them that we can't have this misinformation and our business is going to go if we don't do So we need to stand up together as a collective to do that. So I think we can all take a lesson from that. And that was a great listen to.

Absolutely. The classic tactic of divide and conquer is too powerful. And the only way the industry as a whole can get meaningful change done is by combining forces, which is one of the reasons why I like the Spur Coalition here in the UK so much, where they have such massive media brands associated with it. And I think that'll be absolutely key to get government regulation in place to protect intellectual property, to protect copyright and protect the sanctity of journalism in this country.

Right. So, you know, one of the things I wanted to pick up on is everything that you've just said about the low value exchange for publishers, which, you know, we obviously get particularly in news because what you click through is the product, right? So if that's not leading to a subscription and per click, that it's kind of like low, low value anyway, you know, it just taking, taking that away from publishers is a real kick in the teeth. But, you know, when you kind of look at LinkedIn and other sources of information and you see the big evangelists of prompt tracking or just AI visibility in general, I often think, well, actually that's because they are connected more to a brand capacity, right? Or to like an automobile, right? So if you're selling 30 grand, 50 grand, 100 grand cars, then obviously you don't want to lose any position. So, you know, that's why you're seeing in brands, you're seeing a lot of teams being built in-house, right? Where there might be more of an argument to battling against competitors in that space. But obviously, You know, if you talk to most news publishers, there doesn't seem to be this kind of assortment of teams assembling to build AI visibility teams, right? It seems to be more something that SEOs or audience teams are kind of looking at. But I mean, would you agree just because the value exchange is so low, that's why it's not happening?

Yeah, yeah. If you have a direct sale opportunity, or even an indirect sale opportunity and part of a customer journey through a funnel, then being visible in AI-generated answers has value because people do use it to do research. Sticking with your car example, you can imagine people asking what is the most reliable car? What is the best value for money car? What is a fairly good affordable luxury car? Things like that. And if your brand matches those sorts of queries, you want to be shown in those answers. So optimizing for AI visibility in those contexts, Very important. Don't underestimate it. You probably want dedicated teams focusing on that, even if you can't directly measure it. But that's the same as we have with classic advertising. You know, you don't know what part of it works, but you know that it works. So you just keep doing it. And the same with AI visibility. You sort of need to see it as indirect advertising in AI. If, however, your output, your product is content, you have a very different context when it comes to AI visibility because the output of AI visibility is content. That's what they do.

Yeah.

So if you are part of an AI-generated answer as a content website, there is no need for the user to go beyond that. There is no secondary conversion down the line. They just interact with your content through the medium of an AI-generated answer, and there is no need for them to visit your website, even if you're cited as a source. If you are a traffic-dependent website, which most publishers are, there is no value, none whatsoever, in being visible in an AI-generated answer because the traffic just isn't there. Even the highest click-through rate we see on AI-generated answers, which is in Google's AI overviews, it's 30 to 40% lower than a classic search result. Now that still means there's traffic potential and there probably is still some value exchange being shown in an AI-generated answer on Google. But then if you go to the next big boy, which is of course ChatGPT, it sends almost no traffic to the web. In fact, I think ChatGPT a few months ago even changed its citations and external links in its answers to keep people on ChatGPT more often because they felt too many people were clicking through. The highest percentage of traffic I've seen to a news brand from ChatGPT was, I think, the New York Times. And this is public data, so I'm not giving away any secrets because I don't have access to their private data anyway. But I think it's like less than 1% of the total referral traffic comes from ChatGPT. And I think on average, New York Times, like 30, 40% comes from Google sources. So we're talking like an order of magnitude of variance there. So So that makes it a very interesting conundrum for publishers. And then they have to maybe look at what secondary benefits there might be to still being cited as a response in various contexts. You can imagine, for example, a website like the Financial Times, which is a very hard paywalled website, that they still probably want to be shown in certain contexts for financially relevant queries in these AI tools, because it also shows them as a trust brand, as in, hey, you know, we have this information. You want to subscribe to us. Come to us if you want it directly from the source. And that works if you do have that higher level of brand perception or brand power and a product that you can sell in association with that, which in the Financial Times case is the hard paywalled content. But if you're more of a commodity news website and you have fewer distinguishing content types, it's very difficult to justify optimizing for AI, because there just doesn't seem to be any value in it for you as a publisher to do that.

Yeah, I mean, it's only going to be the bigger publishers who are going to say, we're going to invest in this because we want to fight misinformation, right? Because they've got the capacity to do it. There might be other reasons, like, I mean, ultimately, it's, you know, a lot of AI visibility is around brands' reputation, right? Who's citing you? Who's quoting you? What are the others saying about you? Okay, that's popping up in AI. We need to, we need to look at that, which you could argue we've done many times before in SEO. So I'm not just going to go into that. But, you know, I think we want to kind of like move on to kind of like things that we can do from a technical SEO to help publishers, right? So we've already had a comment podcast before, even in our first podcast, which was around, you know, at the source, there's a lot that you can do in technical SEO in order to advance your AI visibility. But Let's kind of like go through a couple of them, right? So I think we've got semantic HTML.

Yeah, semantic HTML specifically for retrieval augmented generation. It allows an AI system to very easily identify boilerplate content versus core article content and use that core article content then to be used as a citation in the AI-generated answers.

Yeah, we've also got, I mean, this is a classic one, subheadings as questions, right? I mean, we've done a lot of that, particularly in explainers for SEO, but these, these, these pieces still, they appear a lot. And I mean, on that, it's interesting about evergreens, right? Because some publishers have pulled away from doing that. But it's a way to demonstrate added value, right, to the users, because you're, you're, you're collecting, you're listening to the, to the audience about what it is that they want to know. But also, and you're putting it in one place, which is added value you sitting alongside the actual news story, right?

Exactly. Exactly. I think when you understand the questions that your audience asks and you put those questions in your actual content and then answer them in the subsequent paragraphs, you're helping your users because they just get the direct answer to what they're looking for. And you're helping the LLMs because they know, oh, you've got the answer to that particular question that the users asked. So I'm going to use your content as a citation.

Yeah. And structured data, right? That's been sort of like debated because, I mean, this is why everything flip-flops, you know, from day to day on what's the general consensus on this? But I think we got into some position at some point where some practitioners were saying, oh, you know, if you just concentrated on structured data, that's definitely going to advance your mentions and your citations. And then others were saying, well, wait a minute, what is actually the content on the page, right?

Yeah, yeah, yeah. The jury's still out on that one. There's one particular context where we know structured data helps. And that's again, semantic optimization. If you want to optimize for AI agents interacting with your content, they tend to look at structured data because it's basically a machine-readable summary of your webpage and allows them to extract the right information, make the right actions. For pure content-driven AI citations, the added value of structured data is not that clear. But structured data generally has positive effects, especially if it's a schema type that Google supports. So always a fan of implementing it when you Yeah.

Okay, so chunking, right? So chunking has been, I mean, we're in this interesting environment where we're all learning about LLMs. And then anyone that says or identifies a pattern will come out and say, you need to do this. And then, you know, the industry is just too young to latch on to anything with any concrete evidence. But we're still there looking for the main trends, the main patterns, right? But it's interesting about this element of chunking. So looking at paragraphs, even that has come under fire. And, you know, even Google spokespeople have said, you know, it's not a long-term strategy to chunk everything. However, from my understanding, you know, What chunking actually indicates is really that you should look at line by line. So really, for AI visibility, it's all about just getting to the point. So now, this is interesting, a point, right? Because you've got to write for users and write for the bots. And what do you do? We're always looking for a happy medium between the two. But sometimes, I want to read and I want to subscribe to copy which sends me on a journey, right? So often opinion writing or commentary or just humor writing, that is, or comedy writing. I'm sounding like a bot now. You know, it's just, it's meandering and like I'm talking now, it just, whereas for AI bot optimized copy, it's very much line by line optimization, get to the point, put the main facts in, the main entities, move on to the next point. No dilly-dallying. And that's the kind of approach that you take. And, you know, there is an art in that. But yeah, it may be less so for news publishers.

Yeah, yeah. The chunking is always a bit misunderstood, I think, by many SEOs. I don't pretend to understand it fully myself. But I do know that in the context of LLMs, it's about the context window that an LLM has to understand a sentence or a paragraph, which is not as big as people think. It's It's not a huge context window, which is why what you just said, get to the point, short, concise sentences, that is what the LLMs want because it allows them to understand the point of your sentence in that small context window that they have. But it doesn't make for pleasant reading. It doesn't take you on that journey, which is why I am a strong fan of not applying this as a direct tactic for human-focused journalism. to write for humans, to craft that narrative, to build that story and just make sure it's pleasant to read and engaging with the article is pleasant for your users. I think that is a much better long-term tactic than trying to artificially break your content up into short, snappy sentences that are just not fun to read.

Yeah. And also there's also a side note here on using, actually when you're using AI to help create copy, just be very careful. I think the general consensus there is, you know, it's fine to have AI-assisted copy to help you. But if you are using AI to just generate the actual idea of the article, be very careful of that because it's only sourcing what it's already read, right? So will it mention you if if you're just redoing something it's already got. So it's definitely worth being careful there. This is an old favorite of mine, crawlability and crawl depth and internal linking, right?

Yes.

I think that's our specialty, isn't it? Yes.

I think there's a lot of— this is actually a lesson that publishers can embrace because not a lot of publishers allow deeper, easy access to their article archives. And I think that's a missed opportunity in the context of search to a certain extent, but especially in the context of AI, where every article that an AI has access to, both for training data and for retrieval augmented generation, improves your presence in their systems. And there's too many publishers that just have like a category page with 20, 30 articles and that's it. And maybe there's some pagination if they're well set up to maybe 10 pages of articles, which again is fine for users, but there's very few publishers who allow deeper access to older content with archive links, HTML sitemaps, things like that. And I think that's probably a missed opportunity for many publishers. If you are a high-quality publisher and you have a whole archive of tens of thousands, hundreds of thousands of good journalistic articles, you should probably build a pathway for crawlers to find that in a very easy way through a date-based archive or some sort of HTML sitemap. So that is, it's not, it's only a few clicks away from your homepage. One example I will quote for this, for a website that I think does this really, really well is TechRadar. TechRadar has like 9 pages of pagination for any one of the tags and categories. And then there's a link to their archive, which is a unified archive date-based, which basically allows you to find every article published on their website, on the techradar.com website, in its entire history with maybe 3 clicks. And I find that a very elegant solution. Some people worry that that sort of solution creates a lot of crawl overhead for Googlebot. It really doesn't. Googlebot knows to focus its crawling generally on new pages that it hasn't seen before. And we'll crawl the old pages once in a while to check that they haven't changed, but usually only when it has crawl effort to spare. So we shouldn't worry too much about that. And I find that sort of a solution, a very good way to enable that older content to be crawled and indexed and also to be visible to the AI bots.

Yeah, it's interesting because when I've looked at particularly news publishers who have visibility in AI and ChatGPT or Perplexity and the Like I've often noticed that it's links to the archive, right? So that I mean, it's it's it's it's it's it's an AI bot dream, right? You've got everything there for it, and it's easily accessible. So yeah, I've noticed brands like the independents get great great visibility through that.

Yeah.

So yes, go and have a look at the archive system on on the on on publisher sites. Okay. Meta tags and Open Graph.

Yeah. Still used by both search engines and AI bots. You know, the title tag, canonical tags, meta descriptions, Open Graph title, Open Graph description, Open Graph images even. Because when ChatGPT shows a citation for a webpage, it generally has a thumbnail image associated with it as well. And that stuff is just classic SEO as well. You want to make sure that they're all fine in order and, you know, know, properly optimized to give meaningful descriptions and titles for your content.

Okay. I agree with that. And obviously with titles, particularly around optimization around Google Discover, right? Which is still, we know, a big—

Open Graph especially. Yeah. Open Graph. I think the last data I saw was that 60% of Discover headlines come from the Open Graph tag, not the title tag. So that's an interesting one to keep in Yeah.

Okay. LLMs, LLM text files.

Yeah. Again, jury's still out. I do see data that some crawlers, especially ChatGPT, do tend to crawl LLMs.txt files with some regularity, but it's not an officially accepted standard yet. So, and most of the implementations come off the back of just basic default functionality of like an SEO plugin on WordPress. So publishers might not even be aware that they have an LLMs.txt file.

Yeah.

If you don't have it enabled at the moment, I do not necessarily see value in creating one just yet. I think we need to wait and see if it actually becomes a standard for LLMs to use it. At the moment, I don't see that there is a good business case for it yet.

Yeah, I mean, like, there's even this week there was discussion around the benefits of doing that. But for the last year, there's been a consensus of not really. not really doing it. So again, literally things can change within days of, in this fast-moving world of AI visibility. So I think we've covered a lot of topics there.

I want to talk about one more thing because it's a bit of a hot topic as well at the moment, which is Markdown.

Go for it.

Markdown is basically a stripped-down version of the content of your website served in machine-readable format, which you can serve hypothetically to AI crawlers so that they don't get your full website, they just get the Markdown version of your website. And that is seen as a way to basically present an AI crawling optimized website to crawlers for AI machines and present your regular users with normal web pages. But again, it requires development effort, it requires investment and resources in that technology. And the benefit to cost ratio isn't really there because LLMs have since their inception been very, very good at just crawling HTML pages, standard web pages. That's what they're designed to do, to basically separate out the boilerplate content, separate out HTML tags and look at the actual content of the page. They don't need Markdown. They really don't. And what you're doing, if you're giving a Markdown version of your content to specific crawlers and a standard version to users, you're falling into the same trap we fell into with accelerated mobile pages. If you remember that AMP, you basically have an entirely duplicate duplicate version of your website for one specific user context. And that just creates an entirely separate development, parallel development track for your website. And that is a potential source of all kinds of conflicts and problems. And especially in the context of Markdown, there isn't really a value exchange there because the LLMs already are very good at parsing HTML. So I think when it comes to Markdown, that's a big no at the moment. And really not worth the investment.

Right. Steer clear. Okay. Right. So that's a lot to consider. I think that was a hearty meal this time.

Absolutely.

Okay. Right. So I think we're going to sign off there. Obviously, we would always cherish feedback and comments and recommendations for what to cover on this podcast. We will be attending Brighton SEO in a couple of weeks, won't we, Barry?

Yes, we are. Yes, we are. I think 2 weeks from now, 2 weeks and 2 days. I look forward to it. I tend to only go to the spring edition rather than the autumn edition. So the spring edition is coming up. And for me, Brighton SEO is all about the people who are there and the connections that you make and catching up with old industry friends. So I look forward to seeing you there as well, Steve.

Yes, I'll be there with ice cream in hand, although I think last time it rained, but anyway, we'd be down on the beach, our toes in the water, maybe with a beer in hand. Uh, but like, um, yeah, please remember to, I'm going to do that thing of, um, please comment, like, subscribe on our podcast, wherever you can find it. We're across all platforms. Um, and like I said, uh, really would look for recommendations to discuss on future episodes. But for now, thank you for listening.

Thanks very much, everybody.