Technical SEO in the Wild - Real World Issues and Fixes

Love it. Thank you very much. I hope everybody's awake now. Word of advice, there will be swearing. Those of you who have seen me speak before will sort of know my thing. And to sort of help set the scene a little bit, this is what a lead developer of a client of mine actually said about me in a Slack channel, not realizing that his CTO had the Slack channel open and was sharing his screen with me. So I saw this, and I'm like, yeah, mission fucking accomplished right there. I piss developers off. That's sort of my job. I go up and I look at a website and I pick it apart and find out all the shit that's wrong with it. And I will find something. That's what I do. There will be something wrong with it. I'm gonna find the developer responsible for that and ask them nicely to fix it. And some of them take that a bit personally, but hey. Let's talk about technical SEO a little bit. Who here has a technical background? Only a few hands go up, great. You might be able to follow most of what I'm saying. The rest of you, don't despair, the non-techies, the classic marketers among you. I'll do my best, but I can't guarantee that after this it'll be any clearer to you. In fact, it might be even more like holy fuck to you. As long as you sort of know the direction that we're trying to go into and what we're trying to accomplish here, it's not really that complicated and daunting, but to set the scene again a little bit, what SEO in general is, is information retrieval. Every single engineer at Google who works on the search aspect of it will have a degree in information retrieval. probably has read this book as well. This is the Stanford book that they use to teach information retrieval, and the 2 founders of Google, Larry Page and Sergey Brin, studied information retrieval at Stanford while they were building Google, when it was called BackRub. And all information retrieval systems, Google included, have roughly 3 main processes associated with them: the crawler, the indexer, and the ranker, or the query engine as we call it. The crawler crawls the web or any corpus of documents, and sends what it finds to the indexer. The indexer tries to make sense of it, extract content from it, see what those documents, those web pages rank for. And then the ranking system, the query engine, the front end, is about when you type in something that it serves the right documents, the right relevant results. This is what Google themselves, how they describe it themselves in their pipeline. This is a very simplified view of the Google system. But you see the same elements come to place. You have the crawler, you have the indexing system there, and then the search and the ranking engine at the front end. Front on that side. So when I do SEO, I have a 3 pillars of SEO approach: technology, relevancy, and authority. And it will not surprise you that this aligns pretty well with the 3 main processes of a web search engine. Technical SEO is primarily about optimizing for the crawler, on-page optimization, relevancy basically is about making sure the indexer understands what your content is about, and authority, the query engine That's when people type something into Google and you get the right results back. That's what we associate with doing a Google search. When we do certain SEO tactics, there is a bit of overlap, and I apologize for the inadvertent satanic imagery here. You know, you do a bit of technical SEO optimization, you might also help the indexer become a bit more aware of what the page is about. You build good links to a website, you also help the crawler crawl the website more efficiently, etc., etc. But generally speaking, Technical SEO focuses on one aspect with a bit of overlap to some of the other aspects. So when I define these 3 pillars, technical SEO is primarily about optimising for crawling, making sure the website's fast, making sure it's mobile-friendly, a bit of JavaScript, which will be a recurring theme today in my talk, not in a positive context, and international SEO signals as well. Relevancy, on-page SEO, we're talking about keyword focus, topical focus, Page structure, content structure, structured data as well, although structured data bleeds a bit into technical SEO because it is what can be technical to implement, because I will be mentioning it here as well. And then authority ranking, making sure that you actually show up as the top-ranked result in Google, about of course link building, internal linking, strong social signals, or basically just spamming the shit out of Google if you can get away with it. Any Googlers in the room, please don't penalize my websites. I would use a certain set of tools when I do technical SEO, and they tend to be the ones that I go to for almost every single site audit. DeepCrawler is my favorite crawler because it's a cloud crawler. You set it to crawl a website, you go home, and the next morning it's done. Gives you a nice little report. But it's on a per-credit basis, so you have to be a bit careful how you configure it so that you don't run through all your credits in one go, which is why I also use Screaming Frog, of course, which is installed on your computer, on your own desktop. You run it from there, you crawl your website from there, and you can do all kinds of very focused crawls, like, for example, crawl the XML sitemaps associated with the website, or crawl just a specific site section of a site and things like that. Google Search Console, this is the old one. Google has replaced it with a new one, and the new one is only half as good as the old one and is missing most of the reports, and Google says it's an improvement. Fuck off, Google. But we sort of have to deal with the cards that we're given, so yeah, we're stuck with this shittier version of Search Console. GTmetrix is one of my favorite low-speed measurement tools because it's very hard to get a good top score in it, and I like that. And developers always say, yeah, our website is fast, and then you run it through a tool like that, and it's like, no, it's not really that fast. Yeah, but no, no, no, work your ass off, get it faster. Lighthouse is built into Chrome. Great tool to run all kinds of audits on your website for SEO and accessibility and, of course, also load speed. And I use a whole host of plugins like AEM Redirect Path, which will be something I'll show you again later as well, to see more information about the pages I'm visiting and browsing through a site. And they help me identify specific issues that might occur and might not be immediately visible otherwise. So let's talk a little bit about technical SEO in the real world, because a lot of the stuff that you hear about technical SEO and you see written about it as well is pure theory, as in this is an ideal case scenario. And the same thing when you read through Google's developer documentation, that's, you know, it paints an ideal picture of what a website should be like and how it should work, but there's a lot of stuff that you don't find out can go wrong until it actually goes wrong. And you have to fix it. And some of these can be real fucking head-scratchers, like, why is this not working? Because it looks fine, but it's not working. Why? And I'll show you why. We'll start with structured data. I'm contradicting myself a little bit here because this is a technical SEO-focused talk, and I just put this in the on-page relevancy box earlier on, but because it's so code-dependent, I do consider it part of the technical SEO remit. To implement and optimize structured data snippets. Like Aleida said with the speakable structured data for voice search. I'm not going to talk about speakable, we're going to talk about star ratings. Who here works on a website that gets these in Google? Nice little review stars? A few hands go up. You lucky fucking bastards. They're nice, they're really, really nice and not that difficult actually. You know, you implement product markup or service markup and you add the aggregate rating attribute to your structured data snippet, you tend to get the review marking, which is awesome, because they help you get more clicks from the same search results. You might not improve your rankings with them, but if you have the star ratings and your competitors don't, people would probably be a bit more likely to click on yours, except of course when it says 1 star out of 5, in which case, why the fuck are you still in business, right? I used to have them. I used to have star ratings on my own website. The problem being I made these up. I put the structured data on my website, and the website itself didn't actually show any reviews from customers. And this was fine for 2 years until a couple of months ago I got a manual penalty in Google Search Console saying you have spammy structured data. And now I don't have it anymore. And that makes me an angry boy. And it motivates me to write shit like this. Seriously, it keeps me up at night sometimes. But enough about me. This is what that looks like in the wild. In the HTML source code of a page, you put the structured data in there. This is an actual example of a client of mine who had the structured data in the footer just below the closing body tag of the page. So way down in the source code at the bottom. And the snippet itself was fine, it verified if you threw it in the structured data testing tool, it came out fine. The problem was they did not get the star rating because this was above it. No stars. Sad face. So the fix was actually quite simple but not immediately obvious because if the structured data testing tool that you use to verify if your structured data works or not, shows it as accurate, then why isn't Google giving us the star ratings, right? Well, the only thing we really did was move it higher up to in the header section of the page with all the JavaScript and all the other stuff below it, and very quickly we got the star ratings. Yay! Apparently something happened in the rest of the code, and that's the next bit I'm going to talk about. That broke the parsing of the structured data for Google to give the star review rating. So the moment we removed that blocking system, we just put the structured data higher up in the source code, literally within a couple of days we had the star ratings. This is pretty awesome. But again, not something that you'd obviously think of because, you know, if it verifies, it should be fine. However, where it sits in the source code is quite important. So we learned a few lessons from this, primarily, of course, that JavaScript is evil. If you want the rich snippets, you need to put the markup in the head section and ideally quite high up in the head section so that Google sees no obstacles in processing and parsing it. And especially if you have JavaScript above the markup, it may break Google's processing of it. And the next bit will show you exactly why that is the case. And the next bit is hreflang tags. Who here has an international website, works on international multilingual websites? A few hands go up. Who here also implements hreflang meta tags or XML tags? I see the look of despair on these people's faces because seriously, this is really, really fucking difficult. Even Google themselves, this is John Mueller, a webmaster trends analyst at Google, even they themselves say that, yeah, this is probably one of the most difficult aspects of doing SEO. And it will go wrong. In fact, I have never found an hreflang implementation that is right the first time it's implemented. Never. It always goes wrong. The first time, probably the second time as well, and then you fix it a third time, maybe by then it will actually work properly. And there's all kinds of different reasons why it can go wrong, and this is one of the reasons I'm going to show you now. This is an actual hreflang implementation, anonymised of course, of a client I worked with where you've got the hreflang tags sitting there, they're perfectly valid, they sit in the header section of the source code pointing to all the different country versions of this particular website. Page. It's all the same language, it's all English, but one aimed at the US, one aimed at the UK, one aimed at Ireland, plus an x-default version for all the other countries that are not specifically targeted. But again, there we have it, there's some JavaScript above it. Yay. So, in the search console for this particular website, Google did not recognise any hreflang meta tags even though they were present in the code. Which again makes me an unhappy boy. You do not want to see me unhappy. Why is this a problem? The tags are there, they're perfectly valid. Why is Google not recognizing this? Why does that JavaScript break Google's processing of those hreflang tags? And to understand that, we need to go back to the 3 main processes of a web search engine: the crawler, the indexer, and the ranker. The indexer is the module that handles processing of hreflang meta tags and makes sure that everything is working and sees which page it needs to serve for which country and which language. But the indexer has a multi-tiered process. First stage indexing and second stage indexing. First stage indexing is based on the raw HTML source code and is really, really fast. I mean, this is nearly real time. The delay between a page being crawled by Googlebot and being indexed by the indexer is minutes nowadays. It's really, really fast. It was one of the main reasons Google launched their Caffeine infrastructure, to make this process so much faster and, you know, with less delays and less hiccups. But that's not the only thing that Google does. There's a second stage indexing, which is a lot more involved and tends to be delayed by a couple of weeks at least, because it's based on Google's web rendering service. And the web rendering service tries to render a web page as if it was a browser. In fact, it's Chrome 41 that they use, Chrome 41 rendering engine, which is an old version of Chrome. But still quite— supports quite a lot of features and will execute the JavaScript that is present on the page. And that's where the problems come from. Because the rendering engine will execute the JavaScript. In the unrendered HTML source code, the JavaScript looks fine and shouldn't really pose any hindrances. But when you render the page, the JavaScript executes and the JavaScript injects an iframe into the page. where the JavaScript sits. So there's an iframe in the rendered code, completely rendered page, above the hreflang meta tags. And that's a problem. This is a really interesting thread from Jamie Alberico, @jamervoltz on Twitter, who you should all follow, she's fucking amazing. And John Mueller responded to that and says, yeah, this is how HTML5 works, the moment you see an iframe, It should be in the body, not the header of an HTML page. So, the moment an iframe is there— iframe is supposed to be in the body. The moment an iframe is seen to be in the header, the parser, in this case Google's indexing system, assumes the header has ended and the body has started because an iframe only belongs in the body of a page, not in the header. But the JavaScript injected it in the header, so Google thinks, oh, wait, there's an iframe, right, then the rest of it must be the body of the page. page, not the header. And, of course, that breaks the processing because hreflang tags only work if they're in the header of a page, not in the body. Still with me so far? I see a few eyes glazed over. Like, yeah, this is way too fucking technical. I didn't come all the way here for this. Fix is very simple, actually. You just move the JavaScript lower so that it sits just above the closing header tag. In which case, having an iframe is okay because the header ends there anyway and the body begins there anyway. So, the moment we did this, literally the next day, Google recognised our hreflang meta tags, which made me a happy boy. What did we learn from that exactly? Still evil. You have to be very careful with JavaScript that inserts iframes in the header. The next thing that is probably on everybody's radar as an important element is, of course, load speed. Especially in this era of mobile-first indexing and mobile-first usage of search engines, having fast-loading websites is increasingly important. And of course, it's a ranking factor for both desktop and mobile search results in Google. Fast-loading websites get a slight boost in ranking. It's a very small signal as far as I could tell, and it's all relative on your competitive landscape, but it helps to strive towards faster loading websites. websites. But in the context of technical SEO, I look at load speed from a slightly different perspective. For me, fast-loading websites are primarily so that Google can crawl the website better, because Google is a time-dependent crawler, Googlebot. It's not going to try and crawl every single page on your website. It has a certain crawl queue of pages that it's prioritizing in its crawling, and it tries to do as much as it can in any given time period, but if your website is very slow, it can crawl fewer pages in, say, an hour than if your website is very, very fast. Now, I wish I had a before and after example here to show you, but I don't, because for some reason I never save the screenshots when I'm involved with this sort of process, but I should. But I'll show you 2 different websites who work in the exact same industry, and one of them has recently been improved and optimized, and the other one is still using an older codebase. This is the old website. This website has been around for a very long time. An average page on this website takes over 10 seconds to load, 2.5 megabytes plus of data with 154 requests, which I hope you will agree with me is not exactly ideal. This is what we call a slow-loading web page. Not even the worst I've seen. I've seen massively worse than this, but this is a very big brand and they think this is perfectly okay. Compare that to a competitor of theirs which has under 3-second load speed with less than 1 megabyte of data, still not ideal but much, much better, and uses only 42 requests. What do we mean by requests? Requests is everything. The HTML page, all the images, all the JavaScript, all the CSS, et cetera, et cetera. This is from webpagetest.org, another load speed analysis tool. Huge fan of these because you get graphs like this one which really visualize it for managers and the hippos in the room who make all the decisions. This is the slow-loading website. Yes, it's very image-heavy, but there's also 67 JavaScript files being loaded there, who take nearly 1.8 megabytes of data when they're not unzipped. So, when they're compressed, it's already 1.8 megabytes. When they're unzipped, uncompressed, it's nearly 6 megabytes of JavaScript. Just to load one single page. 5.8 megabytes of JavaScript. Do you need that much JavaScript? Compare that to the fast-loading website. Yes, they're also very image-heavy. Fewer images in total, but as a total of the code, again, images dominate. JavaScript, only 12 JavaScript files, which make up less than half a megabyte. That makes an awful lot of difference. An awful lot of difference. In fact, 55 fewer JavaScript files are being loaded on the fast-loading page. 55 fewer JavaScript files. It's 88% improvement in compressed data being sent over the internet. A 91% uncompressed savings right there, which is just fucking mental of a difference. So what happens if you optimize your website for faster load speed, aside from, of course, the obvious user experience benefits that you get? Well, there's a few other things that happen, and one of the things that happen— I do actually have a screenshot of this one, even though it goes back a while, because for me it really illustrated the power of fast-loading websites in the context of technical SEO. This was a client of mine. We launched a new version of their website which was significantly faster loading than the old one, and the on-page SEO wasn't really that much improved. It's still the same amount of pages on the site as well, but it was faster. And this is what we saw in Google Search Console. These are the crawl statistics in Google Search Console. You see exactly when the new website was launched because they went from pages crawled per day from roughly, what is it, 700 pages per day to more than 2,000 pages crawled per day. That's the only thing we did, faster loading website. And it's also because it took Google a lot less time to load any single page. You went from nearly 2 seconds load speed per page to half a second load speed per page. Now, it's not the only thing we did with this website. There was also some other SEO activity. So this is a case of Correlation, not causation, keep that in mind. But you see when this website, the new website version was launched, the growth track we'd already started, started really taking off. And there's a lot of other factors here that play part of that, also because of improved user experience, people were more willing to share the page and more willing to link to the page, and we started writing better content for the website as well, et cetera, et cetera. But it does go to show that load speed has more than just direct user experience benefits. It affects every single interaction that people have with your website, and of course it affects the interaction that Googlebot has with your website. So what did we learn from this? Fuck JavaScript. Faster websites means that more pages will be crawled by Google in the same amount of time, which is great in the context of technical SEO, especially if you have large, intricate websites. You want to make sure that Google sees the updates on your website fairly quickly. And of course, load speed has much more benefits than just providing a better user experience. Every single aspect of your website will be improved if you just make it fucking faster. And that can be a tough battle to win sometimes, because especially with news publishers— and yes, as Andy said in the intro, I work with quite a few news publishers— they are very dependent on on-page advertising. And And then I have this conversation with them and say, yeah, you have a 15-second load speed on your articles, and that's primarily because it takes 14 seconds to load all your fucking ad slots. And that's never going to change because that's their lifeblood, you know? So we go around in circles on that one. The last thing I want to talk to you about today about technical SEO is redirects and why they're bad, generally speaking. A redirect is basically when page A redirects to page B. This is a very simple redirect. The HTTP version redirects to HTTPS version. Which I sure as fuck hope we're all on HTTPS by now, people. If you're not on HTTPS, what the fuck's stopping you? Anyway. Who here has read this? I know Ian has, yeah. Dana, of course. This is where you separate the SEO amateurs from the hardcore geeks. This is Google version 1. This is the research paper written by Larry Page and Sergey Brin in 1998 in which they introduced Google to the world. This is a research paper in which they describe how Google works. And Google was the very first search engine, well, technically the second search engine because Yandex, the Russian one, beat them to the punch, but they didn't read a paper about it and Google did. And they're fucking Russians. They were the second search engine that looked at links and how pages linked to one another to decide which page was the most trustworthy and the most relevant and authoritative, and that page would be ranked highest. And there's a part of this research paper which is fairly underappreciated but very, very important if you want to understand how redirects and links in general affect how pages rank. And that is the PageRank damping factor, which they describe in the paper and is still used today in calculating PageRank for a page and where that page then should rank as a result of that. And I'll try to illustrate this. You have web page A, that's the page, which has a whole bunch of links pointing to it from other websites. And we'll just assume that because of all those links from other websites pointing to page A, page A has 100% link value. Now, when page A links to page B, the PageRank damping factor applies. So the 100% of the link value from page A, the damping factor applies to it, so page B gets 85% of the link value. And then, of course, if you have page B linking to page C, you get the damping factor applied again, so you lose a little bit of link value. You're down to 72% of page A's original value, et cetera, et cetera. It flows through the web like that. That's how PageRank is distributed and how link value is slightly diminished with every link that you go through a website with, or from one website to another. Now, the 15% is, of course, debatable. That was what Google said in the original paper, but it's probably one of the volume buttons they tend to fiddle with a lot. And I also think there's different damping factors from internal websites, internal links within a website, to, for example, one website to another website, or a domain to a subdomain, and things like that. But generally speaking, 15%, 0.85 of the link value from one page to another page is associated with it. So when we have a website like this, which is an utter and total clusterfuck of a website, a lot of problems, an ideal DeepCrawl graph is nearly entirely green, and that's a very tiny sliver of green on that particular graph, and they have a lot of duplicate content problems, but in this case I want to focus on the redirect problems. They have a lot of internal redirects. In this case, they're also 302 redirects rather than 301 redirects, which is an added clusterfuck on top of it. But, you know, that's a problem when a website has 20 times more redirects than it actually has pages to rank. That's not ideal. So when Google says this, there's no PageRank dilution when using 301s or 302s or any redirect anymore, you have to read very carefully and not just look at the headline. Because what the article actually says is that the amount of PageRank that dissipates through a 301, i.e., redirect, is identical to the amount of PageRank that dissipates through a link. And we know PageRank does dissipate through a link because of the PageRank damping factor. So a 301 redirect counts as a link from page A to page B. If page A redirects to page B, page B doesn't get 100% of page A's link value. No, it gets only 85%. Of Page Age Link Value, because PageRank damping is applied through a redirect. So when we turn the clusterfuck of a website into a slightly better optimized website, still a lot of problems there with duplicate content, but at least we got rid of most of the redirects, you see things like this happen. Again, very hard to isolate specific SEO factors because it's not the only thing we did, but it's one of the biggest optimizations we did on that website, and you see traffic go up, you see see rankings go up, you see sales go up, which makes me a happy boy. Of course, you have these Googlers like Gabi Ilya saying that, oh no, there's no actual PageRank damping factor for redirects, and yeah, be skeptical. Look at what you see in the wild versus what a Googler says because there can be a bit of a difference. Googlers are very good at giving you an answer that's not really an answer when you ask a question. You always have to try and read between the lines. So, lessons learned. Oh, wait, no, that's a different one. Redirects bleed link value. You need to minimize your redirects, especially on your own website. Ideally, your own website, every link points directly to the final destination page with no hops, no loss of link value in there. Because that's the thing you can't control. You can't control how other websites link to you usually, but you can control your own internal links, and those internal links need to point directly to the final destination page if you can. You can find that out by doing a crawl on your website with Screaming Frog or DeepCrawl or whatever crawler you use and seeing what kind of redirects you find and fixing them. And sometimes this is stuff that you don't actually notice. I worked with a client who had redirects in all of their navigation bar, but they didn't know because you click on the link and you end up on the right page anyway. anyway, and they had no way to see that there were redirects involved. But just fixing those top navigation links can give quite a big boost to the website because you stop bleeding link value through your top navigation. So what have we learned here today in technical SEO? What are the key takeaways? The devil is always in the details. It can be very little things that can really fuck up your website, and you can You have to look really, really hard to find out what the problem is because the root cause isn't always obvious. The code might look fine, it might validate in all the required tools, and still it's not working. And then you have to dig a little bit deeper. And it really helps to understand how search engines work to find out what the core problem actually is. And of course, it really helps if you know how the web works. what HTML is and how web pages themselves are built. And if there's one thing you take away from this today, I sure as fuck hope it's this. We all clear on that? Thank you. My name has been Barry Adams. I've been doing SEO for nearly 20 years. I have no sanity left, hence the swear words. If the talk didn't impress you, I hope these logos will. Thank you very much. Thank you, Barry.