Everyone’s being asked some version of this right now, and it’s coming from every direction – executives, clients, designers, developers, the person who just saw a competitor mentioned somewhere they weren’t.
- “Why does ChatGPT recommend them and not us?”
- “We rank fine in Google, so why doesn’t Perplexity ever mention us?”
- “Is the new site AI ready?”
- “Are we doing GEO? Should we be?”
- “What exactly is it you want me to change?”
The answer almost everyone reaches for is content and strategy, and sometimes that’s the right answer. But many times, it’s not, and it’s worth ruling out something simpler before you spend a quarter rewriting things.
The question to ask: Can AI/LLM systems read your pages at all?
Google grew out of this problem. The AI crawlers haven’t yet.
About ten years ago, this was one of the most common technical problems in search. Front ends moved onto JavaScript frameworks, content began arriving after the page loaded, and crawlers were handed a shell with very little in it. Plenty of sites lost visibility without changing a word they’d written.
Technical SEOs made a living fixing that, and then the work dried up. The industry didn’t solve the problem. Google did.
Googlebot learned to render JavaScript, wait for content to appear, and index what showed up, and Google documents that behavior openly in its JavaScript SEO guidance. The problem didn’t get solved so much as absorbed by the largest crawler on the web, and it quietly stopped being something anyone had to think about.
Because Google is where the overwhelming majority of search traffic comes from, “Google handles it” became “it’s handled.” That was always a little generous. It was one enormous crawler getting very good at something nothing else was ever obligated to match, and as long as Google was effectively the only reader that mattered, the distinction cost nobody anything.
Now there are other readers, and they’re at an earlier point in that same evolution.
What the crawlers actually tell us
Start with what the companies publish themselves.
OpenAI maintains an overview of its crawlers, covering what each one is for and how to control it in robots.txt. Anthropic did the same, breaking its crawlers into ClaudeBot, Claude-User and Claude-SearchBot with separate purposes. Both are useful, and both are worth reading.
What neither documents is JavaScript rendering as a capability. That absence tells you something. Google publishes detailed guidance on how it renders and what to do when rendering goes wrong, because rendering is something it does. Nobody writes that documentation for a thing their crawler doesn’t do.
Independent log analysis fills in the rest. Studies of crawler behavior have found GPTBot fetching JavaScript files on roughly 11.5% of requests without executing them, and ClaudeBot downloading JavaScript on about 23.8% of requests and never running it. They take the raw HTML and move on. Between them, they account for something in the range of 12% and 9.2% of global bot traffic measured across Cloudflare’s network.
So a page can be excellent by every measure that matters to a person – well written, well designed, fast, accessible – and still hand these systems very little.
It cuts both ways, too. Crawlers sometimes see less than we do, and sometimes more: text tucked inside collapsed accordions or hidden tabs is invisible to a visitor and perfectly readable in the source. The point isn’t that they see a worse version of your site. It’s that they see a different one, and the only reliable source of truth is finding out which.
None of this means anybody messed up
The first reaction in the room is usually defensive, and it really shouldn’t be.
Content that depends on rendering isn’t a mistake somebody made, it’s a default. Modern frameworks, page builders, tag managers, personalization tools, lazy loading, tabbed and accordion content, infinite scroll – all of it delivers on the client side because that’s what’s fast and flexible for the people using the site, and nearly every one of those calls was made for good reasons by someone doing their job well.
A site can be beautifully built and still hand a non-rendering reader an empty room. That isn’t negligence, it’s a delivery decision nobody had to frame as a visibility decision, because for the last decade it hasn’t been one.
If you have Screaming Frog, this is a ten-minute check
This part will be familiar to anyone who has done technical SEO for a while, and it’s the kind of thing a lot of us spent years doing. If you have Screaming Frog or something like it, checking takes about ten minutes. If you don’t, it’s still worth understanding what the check involves, because it tells you exactly what to ask for.
Here’s the whole test. Crawl the site twice.
Once with Configuration > Spider > Rendering set to Text Only, which approximates what a non-rendering crawler receives, and once set to JavaScript, which approximates Googlebot. Export both, then compare word count, inlinks and outlinks per URL.
There’s no universal benchmark for what the gap should be, it’s site-specific, and that’s exactly why it’s worth running rather than assuming.
Here’s how I’d recommend reading into the results:
- Word count drops in the text-only crawl: Body content is being injected client side, and the size of the drop is roughly the size of the problem. This is the serious one.
- Inlinks or outlinks drop: Navigation, related-content modules or pagination depend on JavaScript. Individual pages may read perfectly well while the paths between them quietly disappear.
- The rendered crawl finds URLs the text-only crawl never saw: Something is only discoverable after rendering. When I ran this on my own site, the extras were all assets, images surfacing through lazy loading rather than pages.
- No meaningful difference: Also an answer, and a genuinely useful one. You’ve cleared the technical floor and can move on with confidence instead of suspicion.
If you’d rather do it in one crawl than two, use Configuration > Custom > Custom JavaScript with rendering on:
// Returns the visible rendered text length of the page body.
// Compare against a Text Only crawl's Word Count to find render-dependent content.
function extractRenderedTextLength() {
const body = document.querySelector('main') || document.body;
if (!body) return 0;
const clone = body.cloneNode(true);
clone.querySelectorAll('script, style, noscript, nav, footer, header').forEach(function (el) {
el.remove();
});
const text = (clone.innerText || clone.textContent || '').replace(/\s+/g, ' ').trim();
return text.split(' ').filter(Boolean).length;
}
return seoSpider.data(extractRenderedTextLength());That returns the rendered word count of the main content area as its own column, so you can compare it against a text-only crawl directly. Run it against a page where you already know the answer before trusting it across a whole site.
The gap between the two crawls is your answer
If the gap is real, the fix is work the industry already knows how to do: get the content that matters into the server’s initial HTML response. Server-side rendering, static generation, prerendering, or simply not hiding meaningful copy behind an interaction. Your developers will have opinions about which, and they should, because the right answer depends on how the site was built and what it runs on.
What matters is that the conversation is now specific. “Our content isn’t reaching these crawlers, here are the templates where it breaks” is a problem somebody can scope, price and fix. “We need to do GEO” is not.
If there’s no gap at all, you’ve ruled out the least interesting explanation, and the strategy conversation you have next is an honest one. That’s worth something by itself, because most of those conversations are happening right now without anyone checking this first.
And if you can’t run the crawl yourself, ask whoever maintains the site to compare the raw HTML against the rendered page on your five most important URLs. Same question, plainer clothes, no particular software required.
Either way, ten minutes buys you an answer instead of a theory.
Still have questions once you’ve had a look? Get in touch, I’d be happy to help.