News is at the cutting edge of SEO

Thank fuck you guys all made it back here, because I was afraid the whole room would be just people sitting in the pub instead of actually, you know, being here. I don't know why you're here anyway in the first place. I would be in the pub myself. Yeah, let's talk about news SEO for a little bit and why it's the best form of SEO. No heckling, Dixon. I want to give an advance warning because I have been doing this shit for very, very long, and I realized that the more I learn about SEO, the more I realize how little I know about SEO. This is a quote from Bertrand Russell. He's probably one of the smartest humans to have ever lived. And he basically says that the less you know, the more confident you are about your own ignorance, and the more you know, the more you doubt yourself. And some of— maybe a lot of what I'm going to be presenting in the next half hour or so will be, shall we say, speculative stuff that I've picked up, that I've learned from reading between the lines, from hints dropped or inadvertent words said in conversations by people who shouldn't really be saying things, and also by working with websites like these. Sorry, that's too fast. These, these are some of my publishing clients, not all of them. That's a sample of the ones I've worked with over the last several years. Different types of publishers with different needs, different requirements, different use cases, different technical setups and different target audiences and all bring something different to the ecosystem. And I learn something new from pretty much every project that I work on. And I think that's what I love about our industry in general, that you never stop learning. You know, the moment you think you know it, something will happen. Google does something funny and you realize you know nothing. You're like Jon fucking Snow all over again. So let's start with some basics. How does Google actually work? Loaded question there, right? But Google is still, still to this day, they haven't reinvented themselves entirely yet, still a web search engine. And web search engines have 3 main components. There's a crawling component, there's an indexing component, and there's a ranking component. Now Google have their own model that they use and a very simplified scheme like this one, and because I will be showing some exceptions to these rules, I've created my own version. There it is, exactly the same fucking thing, this versus this. But I'm gonna make some amendments to this because I think there are some interesting subtleties here that are missing from this particular picture. Let's start with crawling. That's, you know, the basis of everything a search engine does. It crawls the web to find content so that it can index it and rank it. In Google's ecosystem, of course, that is Googlebot. Googlebot, the hungry crawler. 2 types of Googlebot generally, a desktop user agent and a smartphone user agent, and pretty much all the crawling that Google does will have these 2 user agents. But I think there's actually 3 layers of crawling that Google does, 3 layers of Googlebot that is not desktop and smartphone but something else entirely. I think Google has 3 core queues that determine when they crawl things and how often they crawl things. And for me, the first layer is a real-time crawler, the crawler that crawls very, very, very fast to find new content. Then there's a second layer, what I call the regular crawler, and this is the workhorse Of Googlebot, and then I also think there's a legacy content crawler that looks at old content once in a while to see if anything's changed or updated. The real-time crawler, the fast crawler, the Formula One crawler, calls VIPs very important pages. These are pages that have a very high change frequency and or are seen as highly authoritative, and Google calls these very. Very often for mainly news websites, their homepages and key section pages, but also things like job portal homepages or property portal homepages have a high change frequency, high levels of authority, and high levels of trust. I have a little anecdote. I once asked one of my bigger publishing clients for server log files, and the question I got back from their lead developer was, How big is your hard drive? So we managed to narrow it down to Googlebot visits in the server log files on their homepage in 24 hours, and it was still a 20 gigabyte log file. Googlebot crawled their homepage every 5 seconds. Every 5 seconds. Ping, ping, ping, ping. Because it's a high-volume news publisher, and Google realized that, oh, I need to crawl that homepage very often to find new articles. And when you publish 800 articles a day, which they did, Google was very, very hungry for their content. The main purpose of the Real-Time Crawler is discovery of new valuable content, for example news articles, but not just news articles of course, but most of the content that's being out there, being published out there that Google would see as valuable and worthwhile indexing would be news articles from reputable Publishers. I've also learned that the real-time crawler rarely recrawls URLs that it's already previously discovered. If it finds an article, it'll crawl that article straight away, but it won't come back to it. I think the regular crawler does that, unless Google thinks there's a new VIP page, like a new section page or something like that, but even then it's a learned behavior. This is one of the things I run into with a lot of my publishing clients, that they publish an article and because of the speed with which Google crawls news websites, that article tends to be indexed within a few minutes, crawled and indexed very quickly. And then they realize that that headline maybe needs to be a little bit changed or they need a different featured image or whatever, and they update the existing article and then they realize, fuck, Google's not picking that up. Why not? We've changed it. You know, there's a new date modified attribute, the whole thing. But Google's like, no, everybody crawled that. Too late. Fuck off. The trick is to change the URL, because then Google thinks it's a new article, and then it'll recrawl it, and then you redirect the old article to the new article. But yeah, so when you have a news— when you're a news publisher, you basically get one chance at optimizing an article. You get one shot. And if you click publish and you realize you fucked up, you fucked up. You'll have to republish it under a new URL, which actually works really, really well. And then there's a second layer of crawler, the regular crawler, the workhorse, which I think does Google's primary crawling effort, which is most web pages on the web will be crawled by the regular crawling system. It probably also fetches the page resources as part of the rendering process. And like I said, this is a bit speculative, so I don't know exactly what the regular crawler does because that's the part I know the least about. But I do know that there's a third crawler, which is the legacy content crawler. It's called the VUPs, the very unimportant pages. These are pages that are buried very, very deep on your website, or old 404 pages that might have existed at one stage but now serve a 400 error message of some sort. And Google will still occasionally recrawl those. You see this all the time in Search Console when you have errors in your Search Console report for pages that haven't existed in years and years and years, and that's because Google hasn't recrawled them, so it doesn't know the new status of those pages. The legacy crawler is the one that will eventually crawl those pages at some stage and then update Search Console report after that. Which is an interesting one because sometimes you want Google to, you know, hurry the fuck up and crawl that a bit faster. Actually quite a simple way of doing that. Link to those older pages from a high authority page like your homepage. Very quick way to get Google to recrawl content that's deeper on your website, that's multiple layers of clicks away from your homepage. Just build better links to it internally, and Google will crawl them and index them and update its own status on that very, very quickly. One thing I've also learned about Googlebot as a crawling system is that robots.txt disallow rules, which we always assume are crawl management rules and only for crawl management, are not only for crawl management. Yes, we have disallow rules that tell Google not to crawl certain parts of the website, and Google obeys those. They obey them religiously. They are what they call a polite crawler, so they will obey your disallow rules. But there are some interesting things when you start digging into Googlebot and robots.txt disallow rules that make you think it's not just purely for crawl management. You can have disallow rules for specific Googlebot user agents, like Googlebot News or Googlebot Image. But when Google actually crawls your website, it tends to have either Googlebot Smartphone or Googlebot Desktop as the user agent. It doesn't have Googlebot News as the user agent. It hasn't had that in more than a decade. Same when it crawls your images. It doesn't use Googlebot Image as the user agent. But you can exclude them in your robots.txt file. You could have a rule like this that tells Google, don't include this in news results. But that's not a crawl system then. That's not a crawl disallow rule. That's an index disallow rule, or maybe a ranking disallow rule. That's a robots.txt disallow rule that has nothing to do with Googlebot crawling of your website. And everything to do with whether your content gets indexed and/or ranked in specific search verticals. So robots.txt is not just for crawl management, it's also for managing indexing and managing ranking, especially in specific search verticals. Which is why it annoys the fuck out of me that you can't tell Google to noindex pages with robots.txt. Well, you have this report in Search Console all the time that says Called or indexed though blocked in robots.txt. That really pisses me off because robots.txt is being used as an index management system already by Google for specific user agents. So why can't you just do that across the whole board? Anyway, don't get me started on that. It makes me very frustrated and very angry, and I think I'm angry enough in normal life without adding more to that. So that's crawling, and what can non-news websites learn from this? You can turn your key pages into VIPs, very important pages. Improving link value, both internal and external links pointing to your web pages, makes them more valuable, and therefore they will be crawled more often. Also, if you make meaningful changes to your pages more regularly, especially if you have a large e-commerce website, for example, your homepage should be constantly changing. Products listed there, or specific categories you want to emphasize. The worst thing you can have as an e-commerce website is a static homepage that barely ever changes. You have to use that real estate to constantly promote products, promote categories of products, to make Google realize it's worth them crawling that page very frequently so that they can find new products or updated product pages, updated product categories. And yes, you can use robots.txt disallow rules to prevent indexing and prevent ranking in specific search verticals. You can use Googlebot-Image user agent in your robots.txt file to prevent your product images from being shown, for example, in Google Image Search. Actually quite interesting, isn't it? Next part then is indexing. This is Google's— again, Google's models, how they visually present it. I like to reemphasize that a little bit because I present it like this: that render queue and rendering part, which is the little vertical fold-out that Google shows, is more of a race to the end in real life. A race to the end because that rendering happens in parallel to indexing, the normal processing of web pages. And for news articles, that rendering tends to lose most of the time. It doesn't win the race. It means the rendering doesn't happen because rendering takes time and news doesn't have time. If a news article has to wait for Google to render it, including loading all the client-side resources, all the JavaScript, all of that, before it gets indexed, it might not be indexed until several hours after it's been published, and that's too long to wait for news. News needs to start ranking straight away because news is very fast-moving. It's as close to real-time SEO as you can get. So indexing for news articles is initially based on the HTML source only. This is one of many shortcuts that Google employs in the context of news. It's not the only shortcut because the moment you start digging deeper into the news ecosystem, you realize that Google does a lot of quick thinking when it comes to news articles. For example, only the other day Google explicitly said for syndicated content, as in content you take from other publishers like Reuters or AP or Bloomberg and put on your own website, you have to noindex it. Google wants you to put a noindex tag on syndicated content that you take from third-party providers. And why is that? Because Google sucks at identifying duplicate content. They're very, very bad at that shit. And even if they're not bad, at least they can't do it quickly. It's a slow process. So they want us to help them along and just tell them, yeah, don't index this, this is duplicate content. To which I say, yeah, no, just pretend it's yours. Just pretend it's yours. Don't even canonicalize it to the original publisher. Just say, no, this is our article. Rank it, and you know what? Google will rank it because they suck at deduplicating that. Just pretend it's yours. Get all the traffic. At least if the original publisher of the content lets you get away with that, obviously. But it was hard enough to get syndicated— syndicators to canonicalize to the original publisher. then putting a new index tag on it, that's, yeah, unlikely at best. And of course, indexing itself is a multi-layered set of processes, a lot of things happening at the same time as part of Google's indexing of any particular web page. And that is also reflected in some way in the actual index itself. The index is not just one index of content either. Again, this is speculative, but I think there might be 3 different levels of indexes where I think the fast crawling is stored in a RAM storage index, the regular crawling is stored in SSD, solid-state hard drives, and the legacy content crawlers stores all the stuff on old-fashioned hard disks. RAM storage is memory, basically, like you have in your laptop. It's the fastest way to store any sort of information, and Google has part of the index stored in RAM memory in their data centers. These are pages that need to be served up very, very quickly, and they get served up very, very often, like news articles, but also other forms of popular content, very popular website homepages, for example. This is content that Google will update regularly. They have to update the index regularly, and therefore they need the fastest type of storage available to allow that to happen. Then we have solid-state hard drives. This is content that is regularly served up in search results, and solid-state hard drives are fast, but not as fast as RAM, but they're a lot faster than old-fashioned hard drives. So Google relies a lot on SSD storage for a big bulk of the index. And then the rest gets stored on old-fashioned hard disks. And this is actually the bulk of Google's index, because Google's index of the web is vast. But I think there was a study done not too long ago by Ahrefs which showed that 90% of web pages that are indexed by Google get zero traffic from Google. And those are the ones that get stored on hard drives, because those pages are rarely, if ever, served in search results, but they're still part of Google's index. So Google still needs to store them somewhere, and it does that on hard disks. It's probably a lot more complicated than that, but I find it interesting that there's 3 levels of crawling and also 3 levels of index storage for web pages. So what can non-news websites learn from this? You need to make indexing as easy as possible for Google. Don't throw any obstacles in their path. Don't do anything client-side that you can also do server-side. Make sure your critical content is in the HTML source. And for the hundred thousandth fucking time, there is no such thing as a duplicate content penalty. The problem is if you have duplicate content on your own website. That means you're competing with yourself, which you really don't want to do because that's stupid. But duplicate content from other websites on your website is really not as much of a problem as some people think it is. The last part, ranking, the last piece of the ecosystem puzzle. This is all about search intent. Google has put a lot of effort into understanding what people actually mean when they type in a specific search query. This started with BERT, B-E-R-T, and then MUM came along. I don't invent these fucking acronyms, but I do find them interesting because it's definitely something that Google has gotten quite good at. And sometimes Google is so fast at understanding a change in search intent that the industry as a whole hasn't yet caught up on it, especially specific SEO tools. For example, you may have noticed last week we crowned a new king in the United Kingdom, King Charlie III, and as part of every coronation there is now an official dish. With Queen Lizzie it was a coronation chicken, And with King Charlie, it's coronation quiche, which is the dumbest, blandest, and lamest recipe you can possibly think of. But you know what? Whatever. The problems you start recognizing is that, of course, in the lead-up to the coronation, there was obviously a big search demand for coronation quiche and recipes associated with that. And as you can see from the search result, Google was all over that. They served the right search result. The right recipes, the right information. Third-party tools like Semrush didn't make that connection. They still thought you meant coronation chicken quiche, which is a quiche based on the coronation chicken dish from the 1950s when Lizzie was coronated. Ahrefs, same problem, doesn't have any up-to-date monthly search volumes on this, doesn't— hasn't yet adapted to the new search intent. You can do that in Google Trends. You can see, oh, there's a spike in the trend for this particular keyword, but of course you lack the numbers there because it's just a purely trend based on 0 to 100, so you don't know the actual search volumes associated with this rising keyword. Some tools have access to similar data like SimilarWeb that show you more accurate numbers and also show you who's actually ranking for those, who's benefiting from those traffic numbers. So that better data allows you to make some better decisions on whether or not to chase after those keywords. But you saw in that search result, there was a recipe carousel there, and they don't have that either. Their data hasn't updated for that yet either. Scroll down further on that search result, you get a top stories box as well, which is where I play. That's where my news clients want to rank. And that of course has a very big impact on click-through rate as well. You know, if you rank organically below a top stories box, you might as well not fucking rank because you're never getting that click. You need to know which SERP features are active to make the right decision. Are we going to be able to chase after that keyword or not? Is it a recipe carousel we want to be in? Is it a top stories box we want to be in? Is it a video carousel? You need to know those things before you go chasing after a keyword, which means you probably just need to type in the search term I know, shocking, right? And actually see what Google is showing rather than look at it through the lens of a tool. And these SERP features can be very, very volatile, none more so than top stories boxes or any sort of news box. These come and go very, very quickly depending on the newsworthiness of a specific topic. I think top stories boxes are triggered when 2 specific conditions are met, which is an increase in search volume as well as an increase in publishing volume. If Google sees a spike in search volume but the news websites are not writing about the topic, it's probably not a news topic that's driving it. The other way around, if there's a big spike in publishing volume but nobody's searching for it, Google will probably show a Top Stories box, but it's not going to get a high priority because it probably doesn't match the intent of the search, because Because people aren't changing their behavior based on whether or not it's a newsworthy topic. But if both happen at the same time— spike in search volume, people are searching for this topic more often, and a spike in publishing volume, or news publishers are writing about this at a much greater rate than normal— then Google says, yeah, this is newsworthy. I need to change the search result. I need to show a top stories box or another news box. And that, of course, has a big impact on click-through rate. Interesting study done by Advanced Web Rankings a while back. They showed that different SERP features on a search result page have a dramatic impact on where the clicks go. And you can be number 1 organic in the 10 blue links, but if you're below 3 different boxes of SERP features, you are just totally fucked. You're not going to get any traffic. And you can get your agency to say, hey, yes, you're number 1, but we don't know why there aren't any clicks. Well, Look at the fucking SERP features. Why are boxes ranking above you? What's the intent of the search? Do you need to change your approach to content? On those SERP features, I want to talk a little bit about why Google ranks specific publishers in top stories boxes. And that's primarily based on things like topical authority. As in, is this website a reputable publisher on this topic? Authorship— are these real people with real expertise who are writing these stories?— and about the EEAT factors that we all know and love dearly. And what I find really interesting, which I don't think a lot of people are talking about, is that these are all expressions of the Knowledge Graph. You remember the Knowledge Graph? I think a lot of people are sleeping on this. This is something that Google introduced years and years ago as a method of understanding the web. Basic example, the knowledge graph is a connection of people, entities, things, places, and the strength of those connections between them. I like using this example because who doesn't like Arnold Schwarzenegger, right? And he has a knowledge graph entry in Google's Knowledge Graph. He's connected to things like Maria Shriver, his ex-wife. He's connected to the bodybuilding sport. He's connected to Ronnie Coleman, a similar Mr. Olympia winner, et cetera, et cetera. He's also connected to the 1987 Predator film, which is the greatest action movie ever made, and I will fight you if you disagree. Boo! I will fight you, Greg, and I will win because I'm bigger. But he's also connected to websites that write about him. If, for example, The Independent has an article on Arnold Schwarzenegger, Google recognizes that, will create a link between the entity of Arnold Schwarzenegger and the entity of The Independent newspaper. The Independent is a great example here because they are very, very good at internal linking. Every time a topic, an entity, is mentioned, they link internally to that category page for that person. And they do that consistently with every article that they publish. And as a result, they are building very strong connections between themselves and that particular entity. And the end result of that is, if you do a Google search on google.co.uk for Arnold Schwarzenegger news, the first result is The Independent's category page. These guys are fucking good at this shit. Internal linking, people, don't sleep on that. So what can you as a non-news site learn from that? Understand the intent behind the keywords that you're targeting. Don't try to rank content that doesn't match the intent. And look at the SERP features. What is Google actually showing as part of the search? Even more so now in the age we have these generative AI chat boxes. If one of those is above you in your keyword, again, you're probably fucked. And improve your Knowledge Graph presence. Category pages on your ecommerce website serve as topic hubs about a specific topic. Internal linking is your best friend, and schema markup, which I think has been mentioned once or twice today, really helps Google connect the dots. So mark up all the shit. So to summarize, why is news SEO at the cutting edge of SEO? It's because news websites are crawled the most, are crawled the fastest, are indexed the quickest, are ranked according to the latest signals, and are ranked based on the best interpretation of search intent. From news SEO, you can learn how Google crawls websites, how Google indexes content, how Google evaluates quality and authority, and how SERP features impact on rankings and traffic, and much, much more. If you like reading stuff about SEO for Google News and/or have trouble sleeping, I highly recommend my own newsletter on this topic because everybody has a fucking newsletter now, right? seoforgooglenews.com, which I think ranks first when you type in SEO for Google News, which is, you know, not surprising because it's an exact match domain name. That stuff never went out of fashion, people. Thank you very much.