JavaScript SEO: How to Optimize JavaScript Content for Search

Welcome to a new edition of Crawling Mondays. In this edition, we are going to cover a topic that I know that generates a lot of interest: JavaScript SEO. Can Google actually render JavaScript well? What are the tips? What are the best practices? What we should avoid? And for that, today I have a couple of very well-experienced technical SEOs. On one hand, I have Olga, SEO consultant and auditor at SEO Sly. On the other hand, I have Tomek Ruski from One Lead, The SEO Playbook, and ZipTie Dev. Hello. Hello, Tomek and Olga.

Hi.

Before starting our conversation, I want to thank ContentKing, who's today's edition sponsor. ContentKing, real-time SEO auditing tool that actually supports JavaScript-rendered content. They're able to identify whatever issue that happens to your websites. in real time, you can just configure the alerts to be warned whenever something goes bad. To start our today's conversation out, we know that Google since a few years ago have switched to an evergreen crawler. Google also started to share much more regarding how they are able to support JavaScript-rendered content, right? And Martin has been doing a great work evangelizing about the importance and best practices regarding JavaScript-relying content. But still today we can see, especially with certain frameworks that over-rely on JavaScript to render core elements of the content, that there are challenges. In fact, it was last week that Google published a little bit of a clarification in their JavaScript documentation saying how they didn't recommend dynamic rendering as a long-term solution, but more of as a workaround. So we can see that there are still challenges. In my case, one of my main clients at this moment, they rely on client-side rendering. But the thing is they actually didn't follow a lot of SEO best practices. It seems that websites that rely on JavaScript framework ended up generating, an example, hash-based URLs that are not independently indexable, don't implement links with href tags many times. Also because the overreliance on JavaScript, the speed is super slow, which ends up also damaging the capacity of Google to be able to properly render all of the content, et cetera, et cetera, right? So I know that there's room for progress. What is your experience with that? Have you seen Google to be able to render all the content that relies on JavaScript? And what would you say are those configurations that you tend to see that can make or break the capacity for Google to do so?

Google has the newest version of Chrome right now to render. So yes, Google Can render JavaScript. However, we as SEOs, we should remember about the limitations that Google has with that respect, which is that, for example, Google may not necessarily fetch all the resources. Google may not, Google won't click or scroll. Google won't interact with the site the way a user would. And we have to keep in mind those limitations and knowing what those limitations can cause. Like for example, that the main content won't be visible to Google, that the links won't be visible, that our content won't be indexed, uh, or the main content won't be indexed, or even the entire page may not be indexed. So we need to be aware of those limitations and find the solutions around them, knowing what those limitations can cause, because the list of those limitations is kind of not closed and we need to simply like go one step forward and try to anticipate those situations.

Google first informed us as SEOs that they are generally able to render JavaScript back in 2014. Then after 3 years, Bartek Guralewicz noticed that it's not the case. He showed that Google cannot properly render most popular frameworks. And then we go to the next part of this story when Google announced they using Chrome Evergreen. I believe it was 2019. And since that time, a lot of SEOs believe, okay, everything is fine. Google is using the most recent version of Chrome and everything is fine, but it's not the case. First, we need to ensure that there are no technical problems on the website because as you mentioned, webmasters do basic mistakes. But it's not only the question whether Google can render or not because yeah, they have powerful technology. But I have a very interesting case of my friend. Who has got a pretty car. The funny story is that he's not using that car for any trip beyond the city because he believes that it's too expensive. And I believe this is a good analogy to Googlebot. I mean, they have got new shiny Hummer H3, which is Chrome evergreen, but sometimes they decide, okay, it's not worth crawling and it's not worth rendering that website. Because JavaScript doesn't change that much. This is the first reason. And sometimes, let's say we cannot use that car, Hummer H3, for mountain trips because the roads are very narrow. Same with Googlebot. Sometimes they cannot properly render because crawl budget, that the limitations is too low. The problem is that rendering counts. our crawl budget. So you can, if you have 1 million pages website and a lot of JavaScript to render, that's really problematic. And going back to my friend's case, he needs to decide how to spend his car budget. I mean petrol budget to use his daily car, which is, and many times he decided to use that old good Volkswagen Golf instead of Hummer. And same here, Google. can use Hammer, but sometimes it's using like very basic stuff, not render it properly. But even if they decide to render, which is still pretty common, I believe most pages are rendered now, it really consumes your crawl budget. And then here we go again. We see a lot of indexing problems and this is like the vicious cycle. Okay. Google can render, but What does it mean that Google render? It means that you will probably end up with crawl budget issue and a lot of your pages will not be indexed. Just check Google Search Console statistics and what percentage of your pages are classified as discovered, currently not indexed. And there is the answer.

I think that you both have touched a couple of very critical issues. On one hand, the context, how you as usual in SEO context is everything. It's not a matter that Google can or cannot render JavaScript, but the matter is how is the rest of the configuration of your website for Google to render the content that actually matters for you. crawl budget here plays a role and how big is the website at the end of the day. One thing, if you have 100 pages or 1,000 pages relying on JavaScript, that takes a little bit to render. And another thing is you have millions of pages that will rely on JavaScript and there's only so much that Google will be willing to spend on your site. one hand, then another very interesting aspect of it. And you reminded me, Tomek, with your analogy here, the other day I saw over Twitter this meme. There was someone making a salad and there's a bottle of oil, but it was huge. Like the salad was this small, the little plate of salad, and there was this huge bowl of oil being added to the salad. And the salad was your website and the, the big oil was JavaScript. The overreliance on JavaScript by default. Right now by many developers when developing stuff that you can very well do with simple HTML and CSS, with web standards. And many times it's not only that there's an overreliance on those, but also like how the configuration is not even consistent between them. So many times the issue that I have found is not that Google cannot render JavaScript, but because the canonical tag in the raw HTML says something different than the canonical tag that is actually rendered, that the URL sometimes changes. The same with the meta robots, the same with some very important configuration. So this lack of consistency sometimes is appalling. I have also had situations with clients on websites saying, oh, I don't rely on client-side rendering, JavaScript rendering for my main content. And then you, you see in listing pages, category pages, pretty much 6 products loaded by default, and then the rest is pretty much hidden by this load more button that completely relies on JavaScript actions to be shown. And Google pretty much cannot see 90% of the products just because of that. So I know that you have seen also a lot, and I want to go through those most common issues and challenges that you tend to see that you want to alert viewers to avoid and what to do instead?

So I would say one of the issues I've noticed recently in one of the audits I was doing is that simply robots.txt was blocking JavaScript and CSS files. And as a result, this is like Google wanted to render, Google could render, but like the directives didn't let Google do so. So that's one of the mistakes I've recently seen. Also, you touched upon load more, like loading more products or articles this way. This is like usually something we think of like, this is not a good idea in terms of SEO, that it is better to just have regular pagination. But even with pagination that, uh, at first sight looks normal, there can be errors because for example, sometimes the pagination is implemented in such a way that the second page requires a click. There is an event and. And even though it may look kind of okay, Google still requires a click interaction. And this type of pagination in many cases works just like having this load more button, which basically loads content based on JavaScript. So I would say these are like the 2 examples I've seen a lot.

Thanks, Olga. Tomek, in your case?

It's dynamic rendering, which is quite popular solution these days. Dynamic rendering is not a silver bullet. It cannot solve all the problems because websites were thinking, okay, if Google cannot properly render or it's not that efficient, then we will do dynamic rendering. And the problem was that many websites didn't care what version Googlebot gets. And that leads to many situations like auditing that version of my website But Googlebot is getting totally different. The most famous example, I believe, is this WordPress commenting plugin. We told them on Twitter, hey, you've got problem. You are presenting empty content for Googlebot. And since that time, still didn't fix that issue. And we noticed that problem across many of our clients. And this is one of the most common problems these days. That's why Google decided, okay, it was workaround. Don't use it as a long-time solution. But also the problem, website don't use traditional href links either created by JavaScript or HTML in pagination, but also it refers to menu links that Google cannot access to main categories because it's loaded as onclick events. Nowadays it's not that common, but We still can see a lot of cases when additional content, which we believe is part of the main content, like parts of the product description, related car parts for car accessories companies, they're not available in the initial HTML, initial rendering. Users can click and then download that crucial content, but as we know, Googlebot Cannot click, so it cannot see the content. So when we talk about JavaScript SEO, it's not only if Google can render, but also if we implement JavaScript SEO correctly. This is the first step. And then if we are prepared for Googlebot to come and render, because we can invite any person to our house, but if it's messy. Then we see, okay, we go out and we don't go back. And same here, if Googlebot come to your website and thinks, okay, it's really slow, we cannot handle the rendering, then I believe that Gary was saying, okay, if that situation coming, it's on you, not Googlebot. It's the part of crawl budget. If your server is healthy, this is on your end, not Googlebot. And things like. crawl budget optimization, server optimization, website speed optimization is really on our end, not on Googlebot's.

What is funny is that actually it's a little bit like what is first, the chicken or the egg, because a lot of the configuration and the context around JavaScript end up impacting the ability of Google to actually render content. And we sometimes, because of the overuse of JavaScript, is that they cannot end up doing it well. So for example, I've seen many times that because of all the JavaScript that is used and it's loaded in every single page without actually correctly splitting the files or preloading certain assets is that the page in the first place is very, very slow and Google doesn't properly render many times thousands of them that are very, very hidden also in the web structure. So again, it's when crawl budgets end up hitting the case. But if they were not overusing JavaScript that they didn't even need it in the first place, it's like a little bit chicken and egg in many, many circumstances. I know that we thankfully have also many tools. Nowadays that allow us to troubleshoot, to validate. The first thing that I do whenever I'm talking with a potential new client is to even go to the web developer extension of Chrome and I disable JavaScript and I take a look at what is actually still visible there, what I can still do on the page. Then of course, when we actually are able to, to start validating and crawling, most SEO crawlers nowadays also support JavaScript rendering and allow us to compare every single element and area of each page to see what relies on JavaScript and what changes via JavaScript, which is also useful, especially on canonical tags, meta robots, all those elements of configurations that will actually impact the page's indexing too. What are your favorite tools and what is the process that you tend to follow to validate and troubleshoot JavaScript-based websites?

I would say we can divide those tools into the tools that allow us to assess how much JavaScript is actually used. The tools that can show us how JavaScript is rendered and the tools that can, can help us assess the indexation of JavaScript content. So I would say to assess the JavaScript use, I also do what you just said. I disable JavaScript. There is also different plugins that can also help you do that there. There is also the tool from Wanli, which I use, which is, I think, What Would JavaScript Do? That's the name of the tool. So once I know how much content is loaded, I can check like how Google actually renders that. If I have access to Google Search Console, then I would simply do a live test, view the rendered HTML, and simply see if, do like Command+F and, and look for the content that was loaded with JavaScript, whether it's actually present there. Of course, if I don't have access to Google Search Console, I can use the mobile-friendly test, which basically Does a similar thing and also lets me like quickly see what's, what's going on. If I want to run this test for the entire website, for example, then I can execute JavaScript using Screaming Frog. It is possible there to, of course, change the user agent to be Googlebot. Of course, this may different, this may not be like 100% accurate, but there is also, you can use a GSC API so you can get pretty close. You can view the screenshot that Screaming Frog does. There is also the, this option to compare differences between raw HTML and rendered DOM HTML. So you can really like do a lot with that. And I believe Sitebulb has the same functionality. And when it comes to indexing, so first I'm just checking if that page on which I want to check the content is actually indexed. Of course, I can do that with just site:. URL or with just inspecting that page in GSC. And once I know that the page is actually indexed, I can simply do site plus in quotation mark, like the entire sentence that was added with JavaScript. And if I do this site command with that sentence in quotation mark and it actually appears in Google, then I know that, okay, Google indexed that JavaScript rendered content. So that would be more or less my toolset.

I take a sample of pages from each of the templates. So the template is blog pages, product pages, category pages. And then I check in What Would JavaScript Do, what's the difference between version with JavaScript switch on and JavaScript switch off. And then I see, okay, this element is generated by JavaScript. Here my content is generated by JavaScript. Here related items. And then I noticed in Google Search Console if it's indexed or not. We also created a software for checking at scale so that we see, okay, in the case of 70 or 80% of pages, this JavaScript content is indexed, in 20% it's not. The process is easy to follow, just detect templates and check in Google Search Console. It's also really important to set up a JavaScript crawler, Screaming Frog, Content King, right? Whatever. Because I believe we do a lot of mistakes. Like there are some inconsistencies between HTML version and JavaScript version, like problem with canonicals, just because we look that part of the process of rendering JavaScript. Because we want to save time and resources there, and we set up Traditional crawling.

Those, these are definitely very actionable advice that I can see how they can be of value for users. There are a couple of 2 extra tools, right? The View Render Source, the Chrome extensions that also allows you to easily see the difference between the raw HTML and the rendered DOM of any page. And also this tool from iPullRank, from my king, PariTool. You can enter a URL and see what is the difference between What the Googlebot mobile JavaScript and HTML is, and the same with desktop too. So it's great to also validate the parity between mobile and desktop, which right now is not so much of an issue, but it still happens from time to time. And also with raw HTML and rendered JavaScript. To start wrapping up, I want to ask you for a last tip that you think that could be very valuable for our viewers to avoid JavaScript SEO issues and challenges. What I will highly, highly, highly, highly advise is a little bit of communication. Here, I have found that in many cases, developers have started to over-rely on JavaScript frameworks and develop many of these elements that should be very straightforward to do also with raw HTML and CSS too, just because they don't see the impact. They don't understand the impact, right? It's not because of lack of knowledge from their side. Evangelization here is key, and what I have started to do is that after. That I provide the recommendations, taking into consideration, of course, each website's own characteristics and circumstances. I do this little webinar to the web team involved and the product team involved evangelizing a little bit about the impact and the importance to follow SEO best practices. In the case that I see that there's a reliance or the usage of JavaScript frameworks for the website, what is okay to do and what doesn't impact SEO and all these extra configurations that It's true that they are able to do, but not because they are able, they should, right? And what are the type of things that should be avoided? Like even like single page applications usage, which today, thankfully, they are far less popular than a few years ago that when there was a boom of them, right? But there are a few very simple, and I think that easy to understand best practices in terms of what is reasonable or not, and also Something very, very important to highlight here is that a lot of the things that we have pointed out already regarding the misuse or the overusage of JavaScript are things that not only negatively impact SEO, but also negatively impact speed and page experience, UX, to accessibility many, many, many times. So these are things that developers should take care of, not only because of us, but because it has a a lot of impact in the web app that they're developing and how it is used and how accessible it is. And a lot of this non-functional requirements that are always critical to ensure in any software application out there, indeed.

So maybe I will share a tip for auditors who are auditing websites with JavaScript, because this is like the case with one of my mentees. Screaming Frog, for example, or Sitebulb will, once you execute JavaScript, will show you the differences, right, with words, how much content was added. And in most cases, there will always be some differences because the raw HTML code looks different than the rendered HTML code. And even if there aren't like huge differences in the content, like the main content is there, the tool may show you that there are differences. And you should always like check exactly what those differences are because in many cases it is just a matter, for example, this tag is placed in a different spot in, in the rendered code. And actually the site is doing okay. Like there is nothing to alert and the tool is alerting you, but this isn't really an issue. So that would be my final tip, number one. And number 2 is that we should remember that social media sites like Facebook, Twitter don't execute JavaScript. So if we want our sites to look okay when shared or embedded on those sites, we need to make sure that like this Twitter card or Facebook Graph placed in the source code, in the raw code, not in the rendered code. It's not placed with JavaScript. So that would be my 2 final tips.

My 2 final tips. The, the first one is if you don't want to rely on Google rendering your content, you can think of server-side rendering. Quite popular solutions for React, you can use Next.js. For Vue.js, you can use Nuxt.js. Most of the JavaScript developers are very familiar with that. It's a very trending topic, so you shouldn't have massive issues with implementing server-side rendering. Of course, not for every stack because we see sometimes it's problematic, but in overall, think of server-side rendering. This is the first recommendation. And second, it's not exact science, but this is Like approximation. If you want to see if Google will be able to render your content, just look at Google Search Console and what percentage of your pages are treated as discovered currently not indexed. If you see that this report shows many high quality pages, if many high quality pages are treated as discovered currently not indexed, you will see very likely That Google will have problem with rendering your content.

Perfect. That is definitely a handy and very useful advice. Thank you very much, Tomek and Olga. And thank you all for watching today's edition. I hope that you have been able to get a few extra tips or ideas of what you can do to avoid having JavaScript SEO issues for sure. And if you liked this video and you liked today's edition, please remember to like the video and follow Olga and Tomek who are super knowledgeable about this topic over Twitter. where they are also very active too. Thank you very much, Olga and Tomek, and see you all in the next Crawling Mondays. Bye-bye. Bye.