Quick summary: AI crawlers like GPTBot, ClaudeBot and PerplexityBot don't run JavaScript – they only read your page's raw HTML. If you render content client-side, load slowly or send crawlers into 404 pages, you're invisible to ChatGPT and Perplexity. Technology comes before content.
Key Takeaways
It loads your HTML and reads what's directly inside it. Nothing more. An analysis of over half a billion GPTBot requests found zero evidence of JavaScript execution (Vercel, 2025). GPTBot processes around 569 million requests per month, ClaudeBot around 370 million. Both grab your page, take the raw text and leave.
The difference from Google is decisive. Googlebot renders pages in a real browser and waits until JavaScript has finished loading. AI crawlers don't. They see your server's first HTML response, exactly as it arrives. Whatever happens afterwards in the browser stays invisible to them.
What's telling is how contradictory the crawlers are with JavaScript: GPTBot downloads JavaScript files in 11.5% of its requests, ClaudeBot in as much as 23.84%, yet neither executes them (Vercel, 2025). They fetch the code but ignore what it produces. For you, that means anything that only becomes visible through that JavaScript does not exist for the AI.
If your website is built as a single-page application that loads content via JavaScript, it's largely empty for AI crawlers. This affects many modern frameworks in their default configuration. The content is there, a human sees it in the browser, but the first HTML response to the crawler often contains only an empty shell.
This creates a paradox. The same page can rank at the top of Google and be completely absent from ChatGPT. Google renders the JavaScript and rewards the page, the AI sees nothing. If you've only optimized for classic SEO, you often don't notice this gap, because your Google numbers still look fine.
The fix is technically clear: server-side rendering, static generation or prerendering. Your content has to sit in the first HTML response, not appear only afterwards. This is exactly where the technical basis of your site pays off. For how clean technology feeds into load time, see our guide to Webflow page speed optimization.
We saw what this means in practice at Tangiblee, a SaaS company for e-commerce technology with over 100 employees. During a full website rebuild, we accelerated mobile load time by 260% and cut code load time by 277%, partly through clear CSS conventions and less rendering overhead. Interventions like these work twice over: they improve the user experience and make the site easier for machines to read.
A big one, but not the way you might think. AI crawlers don't have an infinite time budget for your site. When they hit errors and detours, they burn their requests in the wrong places. GPTBot spends 34.82% of its fetches on 404 pages, ClaudeBot 34.16% (Vercel, 2025). So a third of the crawl budget runs into nothing.
For you, that's a direct lever. Dead links, broken redirects and orphaned URLs cost you attention that should go to your good content. The cleaner your site structure, the more of your valuable content the crawler actually reaches.
And the volume of crawling is growing fast. GPTBot's share of AI crawler traffic rose from 5% to 30% within a year (Cloudflare, 2025). These systems come by more often, but they're impatient. A fast, error-free website now helps decide whether an AI takes in your content fully. Load time feeds directly into your visibility, not only into user comfort.
Deliberately, because most sites don't. Only 14% of the top 10,000 domains address AI bots specifically in their robots.txt at all (Cloudflare, 2025). The first decision is fundamental: do you want to be read by AI systems or not? For most B2B companies that want to be found as vendors, the answer is a clear yes.
If you want to be found, first make sure you're not accidentally blocking the relevant crawlers. These include GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic and PerplexityBot. Many sites block these bots out of old privacy reflexes and later wonder why they're missing from AI answers.
A look at the ratio of crawling to actual benefit explains why clean control pays off: in July 2025, Anthropic crawled roughly 38,065 pages for every single visitor it referred back (Cloudflare, 2025). AI crawlers take a great deal and give comparatively little direct traffic in return. That makes it all the more important that every request hits your best, citable content and not clutter.
On top of that, the llms.txt file is emerging as a kind of map for AI systems. It sits in your domain's root directory and points, in a structured way, to your most important content. The standard is young and not yet officially confirmed by every provider, but the effort is minimal and the signal is clear. We set up llms.txt for webnique.de because it bundles our own content structure in a machine-readable form.
One that a machine can read without interpretation. Once the crawler reaches your HTML cleanly, structure decides citability. The research here is unambiguous: adding statistics raises visibility in generative answers by 32.8%, citing named sources by 27.6% (Princeton et al., GEO paper, KDD 2024). Structure and evidence clearly beat keyword density.
In practice, that means semantic HTML with clear headings, short paragraphs with one statement each, tables for comparisons and an FAQ section with schema markup. These elements sit directly in the HTML, which is exactly what a non-rendering crawler can process reliably. Structured data works like a translation aid for the machine here.
The content side, meaning how you write answer-first and prepare content for citation, is a topic of its own. We covered it in detail in our guide to AI search optimization for B2B companies. The order matters: first the technology has to let the crawler through, then content decides the citation.
Before you invest in content, check the technical basis. These five points make the biggest practical difference to whether AI models can read your website fully.
We know how much this groundwork carries from the classic search world. At a Munich consulting firm with about 30 employees (BelForm), a structured content and keyword strategy grew page views by 400% and leads by 220% within 18 months. The same discipline, clean technology and clear structure, is now the foundation for showing up in AI answers at all. To see where you stand, use our AI readiness check for B2B websites.
AI visibility begins with the HTML, long before the text matters. If your content only appears via JavaScript, your page loads slowly or crawlers run into 404 pages, you lose visibility before the content even counts. The good news: these technical levers are all controllable. Server-side rendering, clean site structure, allowed crawlers and machine-readable HTML are the foundation every content-level AI optimization builds on. Ignore the technology and you're optimizing content for a machine that never sees it. The next sensible step: check what an AI crawler actually sees on your website today, then close the gaps. You'll find concrete starting points in our guide on how to optimize your website for AI results.
