AI Crawlers Miss JavaScript Links: What Website Owners Need to Fix for Visibility
An experimental crawl study found that GPTBot and ClaudeBot did not reach pages linked only through JavaScript. Here is what that means for website architecture and AI visibility.
AI crawlers may not discover a website in the same way as Googlebot. A 41-day experiment published by Search Engine Land found that GPTBot and ClaudeBot followed standard HTML links but failed to reach pages available only through JavaScript-injected links. For website owners seeking visibility in AI assistants and AI-powered search experiences, the practical lesson is clear: important pages, navigation, and content should be accessible in the raw HTML.
The Search Engine Land crawl experiment tested crawler behavior across a 2,400-page website. Its findings do not mean JavaScript should disappear from modern websites. They do show that relying on JavaScript alone for critical discovery paths can leave valuable content unreachable for some prominent AI crawlers.
What the 41-Day Crawl Test Found
The test split pages into two discovery methods. One group was accessible through ordinary HTML links, while the other relied on links injected with JavaScript. Researchers monitored GPTBot from OpenAI, ClaudeBot from Anthropic, Googlebot, GoogleOther, Bingbot, and other crawlers over 41 days.
The most consequential result was the contrast between traditional HTML navigation and JavaScript-dependent navigation. GPTBot and ClaudeBot reached HTML-linked pages but did not reach JavaScript-linked pages. That creates a potential gap between what a human visitor can access in a browser and what an AI crawler can discover on its own.
| Crawler | Behavior in the JavaScript-link test | What changed after HTML links replaced JavaScript links |
|---|---|---|
| GPTBot | Followed HTML links but did not reach JavaScript-linked pages. | Recrawled and discovered about 250 new pages quickly. |
| ClaudeBot | Followed HTML links but did not reach JavaScript-linked pages. | The research highlights the broader benefit of HTML links, without a specific Phase 2 page total for ClaudeBot. |
| Googlebot | Reached about 5% of HTML-linked pages and about 2% of JavaScript-linked pages. | Remained comparatively slow. |
| GoogleOther | Crawled JavaScript pages more than Googlebot, but those pages did not feed into the index. | No comparable Phase 2 discovery total is provided. |
| Bingbot | Showed limited JavaScript activity. | Discovered about 212 pages after the links were changed to HTML. |
Googlebot's behavior also deserves context. In this experiment, it reached only a small share of pages in either group, about 5% of HTML-linked pages and about 2% of JavaScript-linked pages. GoogleOther crawled more JavaScript pages than Googlebot, but the study found that this activity did not feed those pages into the index. Bingbot showed limited JavaScript activity as well.
In the second phase, JavaScript links were replaced with HTML links. GPTBot then discovered roughly 250 new pages quickly, while Bingbot found about 212. Googlebot remained slower by comparison. The before-and-after result strengthens the case that link format, not merely page existence, can determine whether crawler discovery happens efficiently.
How to Make Important Content Discoverable
The appropriate response is not necessarily a full platform rebuild. It is an audit of what crawlers can see before browser-side JavaScript runs, followed by a focus on the pages that matter most for discovery: service pages, product or category pages, articles, documentation, and key navigational paths.
A practical workflow starts with four checks:
- Inspect raw HTML, not only the rendered page in a browser. Confirm that essential copy and links are present in the server response.
- Compare source code with the rendered DOM. If a link or section appears only after scripts execute, it may be less reliable for AI crawler discovery.
- Prioritize HTML navigation for critical paths. Main menus, internal links, category hubs, pagination, and links to important landing pages should not depend exclusively on JavaScript.
- Re-test after changes. The experiment showed that replacing JavaScript links with HTML links can lead to fresh discovery, but crawl timing differs among bots.
For teams using a CMS, the practical question is whether templates output meaningful content and internal links directly in HTML. A site can still use JavaScript for interactive elements, personalization, filters, and visual enhancements. The risk rises when a site uses client-side rendering to create the only route to core content.
Server-side rendering is the preferred approach in the study's practical guidance because it delivers page content and links as HTML from the start. Prerendering can also provide HTML versions of pages where full server-side rendering is not available. Dynamic rendering is presented as a fallback option for situations where a website needs a different rendering approach for crawlers, rather than as the first architectural choice.
The business implication is straightforward. If an article, service page, product page, or knowledge resource cannot be discovered, it has less opportunity to be surfaced when AI systems crawl the web. This is separate from any guarantee of inclusion, ranking, citation, or recommendation. Crawlability is a prerequisite for discovery, not proof that an AI assistant will use a page in an answer.
For many organizations, the highest-value work is therefore selective: identify commercially important pages, verify their HTML output, repair JavaScript-only links, and make future publishing workflows HTML-first by default. That is more practical than treating every interactive component as a technical emergency.
If your website relies on client-side rendering, visibility issues can remain hidden until key content fails to reach the systems people increasingly use to research products and services. Scalevise can help assess your crawlable content, identify gaps between rendered pages and raw HTML, and prioritize fixes that support AI search visibility without unnecessary rework. Use the AI Visibility and GEO Checker to start an AI visibility scan.
Frequently Asked Questions
Why do AI crawlers miss JavaScript links?
The study found that GPTBot and ClaudeBot followed HTML links but did not reach links injected with JavaScript. AI crawlers do not necessarily execute JavaScript with the same reliability as Googlebot.
Does this mean JavaScript is bad for SEO or AI visibility?
No. The issue is relying on JavaScript as the only way to expose important content or internal links. JavaScript can still support interactive features, while critical pages and navigation remain available in HTML.
What is the best way to make important pages discoverable?
Prioritize server-side rendered HTML for key content and internal links. Where that is not feasible, prerendering or dynamic rendering can be fallback approaches identified in the study's practical guidance.
Will HTML links guarantee that AI assistants cite my website?
No. HTML links can improve discoverability for crawlers, but crawlability alone does not guarantee indexing, ranking, citation, or inclusion in an AI-generated answer.
Conclusion
The Search Engine Land experiment shows that JavaScript-only discovery paths can create a real visibility blind spot for AI crawlers. Website owners do not need to abandon JavaScript, but they should ensure that the pages and links most important to their business are available in raw HTML. An HTML-first publishing and navigation strategy remains the most dependable foundation for crawler discovery across a fragmented search landscape.