JavaScript SEO in 2024 - A Survey and Crawl Report from Sitebulb

JavaScript SEO. We all know that it is fundamental to understand how Google actually indexes and tries to render the content, the configurations of our websites that rely on client-side JavaScript. It has been already a while since Google announced that they were supporting it and they published a lot of resources to help us understand it also. Many tools like Sitebulb, the very well-known SEO crawler, started to integrate functionality and a lot of resources to facilitate the troubleshooting, analysis, and validation in order to fix whatever issues we could find. But is it like this really? To answer this question, the team from Sitebulb recently did a very, very interesting JavaScript SEO survey. To release the JavaScript SEO 2024 report that highlights the top trends, not only what they are seeing, but feedback that people has regarding their understanding of JavaScript SEO, as well as how do they actually develop their JavaScript SEO audit. To announce these findings from the report that is going to be published today, and you can find the link here below. Patrick Hathaway, co-founder of Cybulb, is here with me. Hello, Patrick. How are you?

Hi, Aleyda. I'm very well. Thank you. And yourself?

I'm good. I'm good. Thank you very much for coming over. From what I could see when going through the survey, I found that were expected answers, like 59.8% mentioned that it's critical to get a clear picture of what the Googlebot sees, but then there was a clear gap because only 39.9% said that they understand well how Google actually crawls and renders and indexes JavaScript on their websites. And then also it took my attention connecting this with the functionalities that you provide also with Sitebulb to validate and see the difference between the rendered HTML and the response. And I saw that 60% of Sitebulb crawls choose to render JavaScript. So 40% done. And I'm like, it's just a click to select to do it. So it was a little bit shocking to see this gap between the understanding of the importance, then saying that they truthfully believe that they can validate well. And then the people who actually select to do it with a tool that already provides the functionalities. What were the major highlights or most interesting findings from the survey? And the analysis too.

I think this part that you've highlighted right here, you know, the JavaScript rendering documentation, to me as a technical SEO, is a sort of fundamental reading, I suppose. And the idea that less than 60% have read that doesn't, to me, align with the amount of people that understand how important it is according to what they've said. So that is interesting. And I'm not sure, we can only speculate really, as to the reasons for why, but I don't know if that's like due to a lack of trust in Google in terms of like historically, have we always really appreciated the things that they share and whether or not we actually want to believe and trust what they're sharing. This is to do with understanding and you've essentially got half of the survey respondents saying, I understand it.

well.

You know, I either understand it perfectly or understand it well. And, you know, I understand it perfectly is probably quite a difficult answer to give. Like, you have to be pretty advanced, I would say, in order to do that, to say I understand it perfectly. But I would have thought most people at this stage would be in the I understand it well category. And for essentially half of the respondents to feel at most that they understand the basics, Again, it just feels like there's quite a gap opening up between essentially almost everybody saying they think it's important and only half of them really feeling like they actually understand what's going on.

And acting on it. And I don't want to be harsh here, a little bit of like a lack of proactivity to start playing around, going through the documentation, even if it can be a little bit technical, maybe. My hypothesis here is that for some that don't come from the web development world, the documentation can be a little bit technical to understand. Although I have to say, Martin Splitt from Google has done a fantastic job, not only documenting in text, but also through video series, et cetera, highlighting the most important, let's say, areas or elements to check and validate. To go a little bit more with the findings regarding investigating JavaScript issues, they took a lot my attention that in one of the questions, besides asking if people know how to assess and the dependence on JavaScript of a website or not, again, like 15% say that no, that they didn't on one hand, which is still a lot. And then a little bit more than 30% say that they are not comfortable investigating SEO issues generated by the overreliance on JavaScript, which is a little bit aligned to those who said that they still don't feel comfortable explaining them to stakeholders, very likely because a lot of the stakeholders sometimes are also developers too, the ones that they need to communicate to fix these issues. So what do you think you can do to close this? I know that you provide tool functionalities. I've seen this data has made you to try to highlight a little bit more certain information that you provide potentially. Or steps to take? I know that you're also planning to do some trainings too.

Yeah, for sure. The first question here, how to determine whether a site has significant dependence on JavaScript. There are things that we could do within the software to help with that. And I mean, there are things that we've done already within our software to help with that. But I feel like that question in particular does highlight a training gap. Like that should be something which most SEOs should feel comfortable doing. And I was writing about how to essentially do this back in 2017 when we first launched Sitebulb. Like there's been, we've had enough awareness of this specific issue for a long time. And I don't think that particularly has changed. So I do find this a little bit more surprising. And then if you scroll down to the next 2, it does feel like these are linked, right? 30% are not comfortable investigating. Well, it makes sense if you're not really comfortable investigating it, you're not then going to also be comfortable explaining it. And that's a confidence thing as well, right? Isn't it? Because like. If you feel sort of armed to investigate the issue, I feel like at that point you would also be able to go ahead and explain what you need to explain to clients and stakeholders. I feel like this, this part is really interesting because it's that sort of always that fuzzy divide and some people fall on both sides of the question, but I've always been on the side of SEOs should not feel like they need to be developers or learn how to code. I know there are some SEOs that feel that way, but I'm certainly not one of them. However, I do strongly feel that SEOs need to be able to communicate with them. And like, that's really the bit in between. And I think that probably the most important bit in all of this is not necessarily to be able to say, okay, it's this specific code, or it's this part of this JavaScript file that's having all these impacts, but saying that JavaScript is having an impact on these pages and these are the changes and these are the ramifications of those changes. And this is what that actually means. And this is the impact. And it's all of that is connecting all those bits together that, that means you don't necessarily have to know, all right, well, you know, exactly what's happening underlying happening on the page, because that really is developer work. But being able to communicate with them and actually get through both to clients and to developers that this is actually genuinely an issue and influencing them to actually do something about it, right?

100%. It's being able to explain the implication from an SEO standpoint when Google tries to access to the content and indexing. So on this other section, when you asked about the actual auditing of a JavaScript-reliant website, that almost 31% said that they are all the time dealing with websites or working with websites that have a significant dependence on JavaScript, which I think it is more than normal nowadays. It took my attention that 57% say Sometimes, and only 4.3% said that never. But then when you ask something that I'm— in my mind is tied to it, it's like, if most of the times you are finding websites that are highly dependent or dependent, it was only 35% or almost 36% that said that they are using tools to assess or to analyze it all the time, right? So. Only 43% said that sometimes they use it. So if you are running into this sort of scenarios most of the times, why you are not checking this all of the times with proper tools that will also allow you to do it at scale or highly, highly facilitate the analysis? If you have access to so many SEO crawlers out there, it can be so much easier for you to understand if there's an actual gap. Or there's something to pay attention of, right? So again, I can see that there's like a little bit of a gap there on knowing that is important and actually doing the little step of selecting the option there on your favorite SEO crawler to see if there's actually something that is impactful and you should take care of.

I mean, you're dead right. Like part of the SEO auditing process is discovery. Like when you, particularly when you're dealing with a website that you don't know very well or you're looking at for the first time, you don't know what you don't know. You don't know how big the site is. You don't know what technologies it's using. You don't know how much the dependence is on JavaScript. So if even some of the time the sites that you're dealing with have got a heavy dependence on JavaScript or at least partial dependence on JavaScript, why we are not seeing like the all the time as a checkbox to just go, well, I'll check it. You need to, the first time you crawl with it or a few times, to understand actually that discovery question. Like, actually how much dependence is there here? And when there are tools that are readily available and affordable that can give you those answers on scale, like you can look at the homepage and okay, there's not much dependence on JavaScript on the homepage, but maybe on product pages or category pages. Or if you've got websites that are built literally using different tech stacks different parts of the site, you know, bigger websites, that's often the case. So you really aren't going to be sure until you've actually looked at the whole site, or at least a good sample of the site whilst turning these options on, which are helping you understand these things.

The advantage with Sitebulb and certain crawlers is that you don't charge by credits, right? Or per page crawl, which I understand that it might be a reason for some to not necessarily do it every single time because they are charged by credit or something like that. But, but yes, the So many ways, not necessarily to do a full crawl with JavaScript right away, but as you mentioned, a sample of every type of pages on, on the site to double-check the reliance of each one and then do a fuller, a bigger, more comprehensive crawl to, to see the dependence of this and the actual impact at scale.

I can certainly see why people would select the HTML crawler as the default. And like what you've just brought up now, the sort of site bulb data, I think this is, this is really interesting because it shows the changing landscape with how we all as SEOs have understood Google's willingness and ableness to render, how the technological landscape has changed, and how we've chosen as users to change how we actually audit websites. And I'll just try and give a little bit more context on this. So this is literally when you load up Sitebulb and you choose like your very initial settings, right? Okay. Am I going to choose to crawl with the HTML crawler, with the Chrome crawler? Those little like small bits of data that they get sent through as usage data to us. So we, we can track that, right? And so we've tracked that back from January 2020. And you can see that at the very beginning, 65% of people were choosing to use the, the HTML crawler, right? We call it HTML crawler in the tool.

Mm-hmm.

For this graph, we call it the non-JS crawls, right? And you can see that that pretty much tracks across until it's roughly halfway through 2021 where it kind of flips, right? So it kind of switches at that point. And that is because within Sitebulb, we made Chrome, the Chrome crawler, the default option. So you had to override that, right? So you then see about 2 years of people kind of fighting it, right? Users kind of going, well, no, they've, you know, preselected the Chrome crawler, but I'm going to still choose the HTML crawler. And then it isn't until 2023 that they start to diverge in the other way where people are like, Either actively happy that this the Chrome crawler being used, or they're deliberately choosing it themselves. And we're now at a point where it's it's almost completely flipped, where it's sort of 60 percent that the Chrome crawler, the the JavaScript crawler being used versus 40 percent with the HTML crawler. And there are still very good reasons why people would choose to to crawl with the with the HTML crawler. Like I said, for speed. But it's it's stark how different. It's, it's ended up being now, right?

Yeah, 100%. Here we have the response versus render adoption. And it's interesting to see that how it, it doubled since you released it at the end of last year, isn't it?

Yeah, we had that call with you. We also had a call with Arnoud Hellemans as well. And you basically both said the same thing, right? You know, add more features to your response versus render report. Cause we had one for a, for a, for a couple of years. And one of the things that we added was the ability to select at the beginning of the audit to turn it on or off. And if you turned it on, it would literally save the HTML. And so I, the first, probably the first 2 or 3 months of this, there's some usage adoption stuff in there, right? Like maybe people aren't up to the latest version. Maybe they haven't spotted it yet. They're not aware of the feature itself. So I think we can, it's fair to assume that the first few months are a little bit of usage adoption stuff, but the fact that month after month after month, it's carried, continued to grow. It does just show that, you know, this is something that people care about and something that people are getting value from that data, right? They're getting value out of the data that's telling them that difference. And so they're selecting to use it more and more.

Yes, I certainly do get a lot of value myself to be able to understand so quickly if there's something that is misconfigured that I actually need to take a look at. I have many ideas if you want even more to be able to understand the gaps. And now we have also the hints of data to understand that it might or might not necessarily be also critical or impactful to take action on this, depending on which are the pages that are suffering from these. Then we can see what I was highlighting at the beginning of our conversation, how there are many critical SEO-related configurations that are only found in the response or only found in the rendered HTML or modified by JavaScript in the rendered HTML? Which of these you'll say that you didn't expect that to be as high as it was or nothing impressed you here?

Well, no, there are. I do want to contextualize this because I don't know if this will make sense to people, like in terms of like this sort of data, because we only collect aggregated Like data, you know, literally all we're getting is top, like top level stats about like what hints have fired across websites. So this is literally if it's fired one time, even one time for any one website. So it's, this does not mean that 4.6% of all websites ever crawled have noindex in the response, right? It's just across all the audits, this hint was fired at, you know, at least, at least one time on 4.6% of URLs, but that one in particular, right? Because that's.

Yeah.

That's the only hint we have that we class is critical where, where literally, you know, if noindex is only in the response, it basically means that Google won't render it. And it then also means that when you are rendering the page, the noindex is no longer there. So presumably that means that whoever's designed the website didn't want it to be noindex, right? Or at least there's, there's this, there's this conflict where you're not sure. So I found that extremely surprising that would trigger on so many sites. And then H1 as well, like. being so high, you know, that triggered on so many sites. Like this is literally just, just data that was going into a database. So we knew if people were actually using the tool and if the tool had any value. So we hadn't really even looked at this, but it's, it's incredibly surprising to see the H1s having such an impact like that. Because content and link, they're the two that you expect. Like I expect some content to be changed by JavaScript and I expect links for sure to be inserted by JavaScript on almost every single site on the internet these days, but But key elements like H1, for me, that is very surprising.

Indeed. So to start wrapping up, and here I want to move to your final thoughts too, because I know that you also have, besides thoughts, a few resources, a few actions that you want to invite the audience to take, especially if they are unsure of where they find themselves here.

Yeah. The big takeaway for me and for us at SiteBob looking at this data was that There is a gap, right? It really does feel like there's a gap between how important people think it is and how able they are to investigate it and to actually explain the issues that they're finding to clients, to developers, to other stakeholders at their companies. And so the big thing that we think is important is that that training need is addressed. And so we have worked in partnership with Women in Tech SEO to put together a training course, which you can sign up for using That link right there.

Yep.

A totally free course available to absolutely anybody. And it will start at the beginning of September and is run by Sam Torres and Tori Gray, who are absolutely brilliant and know this stuff better than anyone. And they're extremely good trainers. And we have 3 key lessons, essentially training webinars going across everything that we've talked about here. So the first one, identifying and understanding JavaScript. The second one is how to audit it. And the third one is perhaps the most important one is how to actually go ahead and communicating these issues to people. And so. So if you do find that any of this strikes a chord with you, or you feel like you could do with some extra training, go ahead and register for this series and you'll have access to that completely for free. And in addition to that, we have a whole load of resources on, on the website ourselves, which is under javascript-seo, which you can get to either via that link. That's it. There you go. And so this is, this is our hub for everything to do with JavaScript SEO that you can learn from Sitebulb, including access to the report. There's a whole section in here. We've got the webinar that you were on, Aleyda, as well through here as well. So there's loads of stuff in there. help you through this journey with loads of third-party resources as well. So hopefully this does help bridge that gap. We hope to be able to do this report on a regular basis so we can kind of see how things have changed, if any of these trends have shifted, and maybe in a year's time it's done some good to actually making some progress in terms of closing that gap.

100%. Thank you. Thank you very much. I think this is fantastic because it's not that you have only found a gap, an issue, a challenge, but you're providing free resources for everybody to take action. The training on one hand that is free and open to all, and you can have access. Also, you have a free trial with the tool. If in any case, if everybody wants to just start testing it and troubleshooting it and getting hands-on with what they learn with the training. And I have to say something, the JavaScript SEO Hub that you have here. Mm-hmm. Is one of the most complete with multi-author resources, uh, also linking to other resources out there. So it's very, very comprehensive explaining also all of the issues that you find with the tool and why they are important. So you provide context. So if you start using Cyboop as an SEO crawler to assess this type of JavaScript-related issues, you will find also the why. in most of the scenarios or circumstances, along with a notification there, the report saying that there's something wrong in there. So it can help you to really, really accelerate the SEO learning process for sure. Thank you. Thank you very much, Patrick. I really appreciate it for you to come over and share with the fresh insights of the first JavaScript SEO report. I hope that it is read. I will certainly include it in the next SEO FOMO. Remember, if you want to keep up with the latest in SEO, subscribe to SEO FOMO if you haven't yet.

Thanks very much for having me on.

Until the next one.