Should We Do This Yet? Clear Advice On 5 High-Impact Technical SEO Decisions - Alex Halliday

What an intro. I wish I could have that every all hands. Pop forward. Okay. That's me. I'm in the unenviable position of sitting between all of you and lunch today. So, I'm hoping this is a high value, high nutritional value 30 minutes that we can get into. And so, I thought about how we could use the time today and decided that the best way to spend this 30 minutes would be to focus on 5 high-impact technical decisions that we get asked about a lot at AirOps, and share our perspective, some learnings, and please, after this, tell me if you disagree, or you have other opinions. We are all students of the space. Space, as we like to say. Uh, my intro— thank you for that, it was awesome— uh, but have basically spent 20 years building companies and growth teams, uh, huge focus on SEO, marketplace SEO. Zero to One at Masterclass, which is a big success story, along with Ethan Smith at Graphite. Um, but have been having the professional privilege of a lifetime building Airopps over the last 3 and a half years with so many people in this room. Um, we've done webinars with lots of you There's so many people here that I've met so many times on LinkedIn or on Zoom and have never had the opportunity to chat with in person. So it's been a thrill the last 24 hours being in this space with all of you. For those of you that don't have LinkedIn and haven't seen AirOps, we help companies prioritize, plan, and create content that wins citation and visibility and have had a really amazing year kind of leaning into the space. With some fantastic successes along the way. So let's make this a great 30 minutes for all of us. And I want to talk about 5 plus a bonus kind of interesting questions that have emerged. And the first one that I want to start with is the value of Markdown alt web pages. Do they drive value? Is it a kind of interesting strategy that we should be considering? Today. And so first of all, what are we talking about here? Well, we're starting to think here about taking existing HTML pages that are heavy. They potentially have a lot of boilerplate code in them, a lot of JavaScript, and creating an alternative markdown version of the page that is leaner. It's well-structured. It contains just the pertinent information for RAG. And we've seen some people create just alternate.md versions of pages. And just to completely dispel this one, hopefully no one in this room is doing that. There is no value to doing that today. It may become interesting in the future, but we can just skip over this. The one that's more interesting is real-time switching. So when a request comes in from a human or an agent user, there is a user agent passed in. At that point, the user agent will have a name similar to Chrome or GPTBot or ClaudeBot. And at that point, using something like a Cloudflare worker or Vercel middleware, we could actually show a different response to those users based on the user agent. And I think this is worth spending a little bit of time on because it's becoming for us a more recurring question that we are seeing. And so, but why would teams even want to do this in the first place? And there's 3 reasons that we want to kind of really I want to talk about quickly. So the first one is agent readability. And the theory behind this is quite good, actually. It's, you know, obviously the context window is finite. Focus with LLMs is very important. Can we strip down the context that we're supplying back to GPTBot in order to make sure it's able to find the information it needs and we're more likely to win that precious citation and mention? And this is how you would build your own RAG system. So if you were having your own chatbot internally or you were building anything, for your own customers, you would do this. You would prepare the context for the model to make sure that you were being efficient with that compute. You were more likely to get a high-quality answer out the back. The problem is that in this context, there's no evidence that this improves anything. And that is because the, the RAG pipeline and the retrieval pathways that Christian was talking about already do a fantastic job of converting well-structured, disciplined HTML into Markdown or equivalent format. There's a few alternatives coming down the pipe. And so both community tests show us that this does not move the needle. And Google has come out and said the same. So for pure agent experience today, this is not a great use of your precious time. There is another reason, though, that people have started to do this. And this is probably something that's only really present in much larger organizations, larger companies. But there are certain pages on the website for which making changes is a, let's just say, a political nightmare. And in those instances, there are product marketing teams, there are brand owners, there are lots of eyes on those pages. And we've spoken to a lot of teams that are starting to think about creating Markdown alt actually to be able to fast iterate on those pages, make more rapid changes, and ultimately test their way into successful content changes that they could then scale up. And then the third is like content expansion. And this is where potentially you get a little bit dicey because there are situations where folks want to expand on the content. They want to maybe format the answers differently for citation likelihood and retrieval, and they want to be able to do that free from worrying about updating the main page. And so our view here is like sometimes maybe this is a good idea, but do this should definitely not be a core strategy today. And once you started to create different content pathways between what humans are seeing and what agents are seeing, there's all sorts of landmines you could step on and risk that you can introduce. So for example, you know, in the traditional SERP world, there's very clear guidelines around cloaking and deceptive practices that you want to avoid. Now, those have not been updated to my knowledge to explicitly include this use case, but the, the spirit of that is to make sure there is consistency with the information that's provided to the user. And so logically, we should probably extend that to this. The second is user confusion risk. You see a wonderful passage in a ChatGPT response, you click through, it passes that passage through the URL parameter so that Chrome or equivalent browser can then go and highlight it on the page, and suddenly it's nowhere to be found. Um, and then the third is, is just general content parity and quality of experience. Experience. You want to make sure that the information the models are confidently returning as much as possible is then followed through on the existing page. And so our recommendation here is don't use markdown-only versions unless you have absolutely no alternative, and stick to the classic best practices that hopefully we're already doing to make those pages clean, make sure heading hierarchies are strong, remove div bloat and sort of the decay of the boilerplate code that's seems to happen when too many people are touching it. Question 2: LLMs.txt, should we adopt this? No is the answer today. And the reason isn't because the spirit of the proposal isn't a good one. We obviously want to make sure that we can provide high-quality information to models. We want to make sure that the models can find the best possible answer and grounding for the user's question. We're all aligned that that is what makes a great user experience that helps your brand get discovered. But lms.txt today in its current implementation is solving a problem that doesn't exist. So for this to be a helpful standard, it would obviously need to be adopted and supported. But it would also need to imply that the fanout queries are passing through the fact that the web pages are going to be returned to the context window of the model and therefore Markdown pages or lms.txt link pages would need to be prioritized, and that's simply not the case. When fanout queries happen in SERP, they are seeing the same results as humans today, and so file type doesn't matter. I've also seen some really crazy stuff like people putting, "Hey AI, click this," in their footer. Don't, don't do that. In 99.9% of cases, the retrieval process is one hop, so it's not going and doing deep exploration situation of your site. There are obviously things like ChatGPT agent mode, which is very cool, but I have never seen a real human being use that for anything that useful. So I wouldn't be spending your time optimizing for that. We'll come back to that one. But I think what's really interesting is, um, as we think about what happens in the future, you know, right now the way in which ChatGPT and co are finding information is really leveraging all the fantastic work that's happened over the last decade and beyond. On creating a fantastic content and authority graph. So SERP is currently the rails in which the models reach out of their own corpus, um, and they go and get additional information to include in the result to make sure customers get a fantastic response. And so, as we've heard multiple times today, that looks like construction of fanout queries, uh, page— pagelisters downselecting those pages and then retrieving chunks from those in the response to the customer. But there's sort of little emerging, I would say, sparks of things that could indicate what's going to come next. And none of us really know in this room, I don't think. If they do, they should come take the mic off me. But I think there's starting to be some additional interesting protocols and things emerging that will start to shed light on what the future optimization challenges that we're gonna have to deal with and we're gonna be asked about from our customers and, and our CEOs. And so OpenAI recently launched the Agentic Commerce Protocol, and baked into that was the ability for brands to directly submit rich product descriptions directly to ChatGPT. So instead of there being a RAG process to fetch the information, that's going to exist on the OpenAI side and can be retrieved upon at runtime. And, and there is a very credible reason to believe that we will move beyond products for those feeds into things like events and other entity types that can then be retrieved. The Apps SDK, which was launched, which allows for brands to build richer in-stream, in-feed experiences, is very interesting. It creates a real challenge for brands that need to decide the delineation between what they're prepared to build in-stream directly versus on their own website. And the optimization of what's going to trigger an app and how to think about the new App Store optimization there will, I think, become a core question within the community over the next year. So just on that one, we're expecting app submission to open end of this year in December. Initially, it will only be your installed apps that will be rendered. But then 6 to 9 months from now, you will be able to discover new apps. And therefore, there will be a new optimization challenge for us all to contend with. And then who knows what's going to come next. I think this is something to watch, and it will start to shift the way certain intents are served, how we as technical SEOs will think about serving those needs. And a lot of the work we do today will extend and mature to take advantage of these new pathways to reaching customers. Krishna wants to pull up the roadmap. He might be able to tell us when this is going to happen. I feel like this should say, do the damn schema at this point today. But a couple of thoughts here. So we know what schema is. It's really about creating clarity around the entities that a page represents. And there's specific pieces of the metadata on the page that are overweight more than ever. And so creating structure and clarity about those can only be a good thing. And so there's more schema than most people realize to pick from. But things like products, articles, question-answer pairs, all, you know, commonplace today and super worthwhile doing. And I think there's great unintended consequences of doing schema. So, you know, in traditional SERP, they can really assist with enhanced affordance on the SERP page. So better listing types. And although they may not necessarily improve ranking themselves, they can help you get more— a higher click-through rate, more of that SERP, and generally do a better job. So do that. And then I think the other thing is the Web Data Commons really uses schema as part of its core common crawl. And that's used in a lot of training material. So it sounds like we are hearing firsthand today that schema is included at runtime. But even if it wasn't, you still want to do it so that the Common Crawler is able to train on it. And there's just super clear, unambiguous entity definitions and things like freshness signal, pricing, ratings. These core data points that are used in response formation are crystal clear laid out to the model for RAG. So, so definitely do that. When I mentioned there are lots of different types, These are some of the priority ones that we try to push. And I think there's— the 4 at the top are slightly less obvious ones that we'd certainly encourage. The one that's not on here, actually, but I probably should have added is images. The nice thing about doing image schema tags is that you actually get a really fantastic Google image boost if people are searching for infographics or other things. We've seen tremendous success. In actually traffic coming through from Google Images by doing high-quality schema for images. But take a picture of that if it's useful to you. Happy to send it off. But as with all things, we all are very busy. We have many people yelling at us all day long for lots of things. So always prioritize and pick the best ones. I think I covered most of this. But I think as we think about the emergence of the next version of the content and authority graph, the more you can do to harden the definition of the entities that your content contains. So if that's products, people, places, FAQs, events, the more you can harden that, the more that future indexing and the models at runtime can understand those relationships. You can disambiguate things. It's about the quality of the content you're producing, making its way through to the customer, and it's about good content hygiene along the way. I did that one already. Question number 4, should you analyze server logs for AI foot traffic? Yes, you definitely should do this. I think this is an interesting sort of emerging opportunity now. And the— the— if AI is constantly spending time looking at your pages, you should probably be spending time updating them is the easiest way to think about that. And so your server logs will show the foot traffic of agents crawling your pages. The problem with traditional tracking methods like GA and other things is they rely on JavaScript. And as we heard today, these models are not triggering JavaScript, unfortunately. And so you have to rely on CDN-level or web server-level tracking for how these agents are traversing your pages. That gives you the bottoms-up view. So it gives you a heat map of crawl behavior to see which pages are being heavily trafficked. You can then use tools like AirOps or others to look at the tops down. So as you go through and check high priority queries, what are the pages that are being either read, cited, or both, and be able to understand the kind of connection of those two things. You're really trying to triangulate a picture of how both the agents are visiting your pages and how the agents top down are using your pages for RAG and those runtime retrieval pathways as well. Question number 5: Should you use answer blocks? Yes, you should. And so LLMs consistently extract short declarative answer chunks, and there's a lot has been written about this in terms of how to think about what makes a really strong, strong answer. They are looking for a good, authoritative, fresh answer chunk that they can include in their response. And so formatting pages with, with Q&A, TL;DRs, summaries at the top is a, is a strong strategy. And, and ultimately you're sort of spoonfeeding the intermediary. You can think of like the agent is sitting between you and your, your end customer and you want to make sure you're setting them up for success with prepared snippets that they can then go and take and include in their, in their responses. So this one's pretty easy, but but definitely powerful. So a few specifics here. So add first paragraph definitions, FAQ blocks, key facts sections, and step-by-step blocks. And make sure those are really well structured from an HTML standpoint, and they succinctly answer the question. To go even further, things like freshness signals on the answer, or baking in authority in the first answer— in the first sentence of the answer can help as well. And just general page hygiene is super, super important. Bonus question 6: Is there opportunity in LM URL hallucinations? So yes, so LMs love to, as we heard earlier, I think, love to make up URL paths and try things and run all over your site. Oh dear. But so studies have found that 1% of of URLs that ChatGPT tries to retrieve don't actually exist. If you're able to track those, they can provide opportunities to create additional pages of content that will just work. The issue here is that not everybody has sufficient traffic for this to be a worthwhile strategy, but you know, for those with larger footprints or with higher hallucination rates, this is a really good opportunity to just capture some traffic along the way. And the data supports that this problem is not really going away. So to summarize takeaways, use Markdown-only ARIA versions sometimes, but only when absolutely required. Do not adopt LM's TXD today, but keep an eye on it. Might get interesting, might get adopted in some future iteration. Go deeper with schema, but prioritize. It's good hygiene, helps you reinforce the entities you're talking about, make sure that the content you're putting out can be read, can be referenced by customers. Server logs contain gold, but on their own sometimes are only the bottoms-up picture, should be matched with the tops-down picture of how pages are being used in specific question and topic clusters. Answer blocks are good hygiene and definitely worthwhile, both top of page summaries and FAQs, and accidental traffic provides some freebie pages you can go create. So please please say hi. One final thing is we are hiring across the board, so if you are excited as I am about the intersection of brand, SEO strategy, and kind of emerging AI, please come chat to me. We have some really fun stuff going on, and look forward to meeting more of you today. So now we can go enjoy lunch. Yay!