GPTBot, ClaudeBot, PerplexityBot: What AI Crawlers Really See With Prerendering
TL;DR: „AI crawlers execute little to no JavaScript. If you don't prerender your SPA, you are effectively invisible to ChatGPT, Claude and Perplexity. Three months of edge logs show how often they come – and what they actually pick up."
— Till FreitagWhy This Analysis
We talk a lot in this cluster about prerendering being the bridge to AI search. But "LLMs prefer static HTML" is a claim. This article delivers the evidence.
We analyzed three months of Vercel edge logs from our own site (Playwright SSG) and a client site (classic SPA, no prerendering). Same traffic range, same stack setup minus the prerendering step. This is not a lab benchmark – this is production traffic.
The Three Relevant User Agents
| Bot | User-Agent (excerpt) | Operator | Purpose |
|---|---|---|---|
| GPTBot | Mozilla/5.0 (compatible; GPTBot/1.x; +https://openai.com/gptbot) | OpenAI | Training + ChatGPT Search |
| OAI-SearchBot | Mozilla/5.0 (compatible; OAI-SearchBot/1.x; +https://openai.com/searchbot) | OpenAI | ChatGPT Search live retrieval |
| ClaudeBot | Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | Anthropic | Training + Claude Citations |
| PerplexityBot | Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Perplexity | Search index + answer engine |
| Perplexity-User | Perplexity-User/1.0 | Perplexity | On-demand fetch on user query |
Important: GPTBot ≠ OAI-SearchBot. GPTBot indexes broadly for training. OAI-SearchBot fetches specific URLs that ChatGPT wants to cite in an answer. If you want to appear as a source inside ChatGPT, you don't optimize for GPTBot – you optimize for OAI-SearchBot and similar "user-triggered" crawlers.
What the Logs Show: Crawl Volume
Three months, one million edge requests, filtered by user agent:
| Bot | Requests | Share of all crawlers | Trend |
|---|---|---|---|
| Googlebot | 41 % | Reference | stable |
| Bingbot | 12 % | stable | |
| GPTBot | 9 % | +40 % QoQ | |
| ClaudeBot | 7 % | +120 % QoQ | |
| PerplexityBot | 6 % | +60 % QoQ | |
| OAI-SearchBot | 4 % | new | |
| Other (Bytespider, Amazonbot, etc.) | 21 % | mixed |
The three big AI crawlers together account for about a quarter of non-human traffic. Trend: steeply rising. What you could call "niche" in 2024 isn't niche in 2026.
What They Actually Fetch
This is where it gets interesting. We measured three values per bot:
- HTML request (the initial GET to the route)
- JS bundle requests (does the bot even load the app bundles?)
- Hydration indicator (any follow-up request that can only exist after hydration – e.g. a lazy image, an API call?)
On our prerendered site
| Bot | HTML | JS bundle | Hydration follow-up |
|---|---|---|---|
| Googlebot | 100 % | 78 % | 31 % |
| GPTBot | 100 % | 3 % | 0 % |
| OAI-SearchBot | 100 % | 0 % | 0 % |
| ClaudeBot | 100 % | 0 % | 0 % |
| PerplexityBot | 100 % | 1 % | 0 % |
On the client's non-prerendered SPA
| Bot | HTML | JS bundle | Hydration follow-up |
|---|---|---|---|
| Googlebot | 100 % | 71 % | 22 % |
| GPTBot | 100 % | 2 % | 0 % |
| OAI-SearchBot | 100 % | 0 % | 0 % |
| ClaudeBot | 100 % | 0 % | 0 % |
| PerplexityBot | 100 % | 0 % | 0 % |
That's the finding in one table: AI crawlers practically don't load bundles. They don't execute them. They see what's in the first HTML response – nothing more.
Googlebot renders JavaScript a significant share of the time via its Web Rendering Service. The AI crawlers simply don't. Not "sometimes", not "on important pages". Across three months of data: essentially never.
What That Means in Practice
On the client's non-prerendered SPA, GPTBot, ClaudeBot and PerplexityBot get this HTML:
<!doctype html>
<html><head>
<title>Loading...</title>
</head><body>
<div id="root"></div>
<script type="module" src="/assets/index-abc123.js"></script>
</body></html>That's all. No headline. No body. No JSON-LD. No OpenGraph with a real title. For the training set and the live retrieval index, this page is an empty document titled "Loading...".
On our prerendered site they get, per route:
- complete
<title>and<meta description> - all
<h1>to<h3>with real text - the full article body inline
- JSON-LD (
Article,BreadcrumbList,Organization) - OpenGraph with real title, description, og:image
That's the whole difference. Not "somewhat more visible". Rather "exists" vs. "does not exist".
Why AI Crawlers Don't Run JavaScript
Three reasons that show up in bot docs and in engineering posts from the providers:
- Cost. Headless Chromium per URL is orders of magnitude more expensive than a plain
fetch. At the crawl volumes we're talking about (billions of URLs per month), that doesn't scale. - Determinism. LLM training data should be reproducible. Hydration race conditions, A/B tests, JS-based login walls – all of that produces noise.
- Speed. For on-demand retrieval (Perplexity-User, OAI-SearchBot), the answer needs to come back in seconds. No room for 3 s of hydration.
Googlebot can afford headless rendering because it's been its own infrastructure stack for 15 years. OpenAI, Anthropic and Perplexity aren't rebuilding that from scratch. They take HTML as it comes.
What the Logs Also Show: Cache Behavior
A second finding that supports our SSR vs. pre-rendering argument for Mykeythai:
AI crawlers respect Cache-Control and ETag noticeably more consistently than Googlebot. On the prerendered site we see, for GPTBot and ClaudeBot:
- 70–80 % of re-crawls land on
304 Not Modified - Edge CDN answers them from a regional cache
- Origin is never touched
On the non-prerendered SPA: every request goes through (200 with empty HTML), because the HTML almost never carries a useful ETag. More traffic, less content. Worst trade-off possible.
What You Should Do Now
Concretely, in this order:
- Filter your logs.
vercel logs --output rawor straight from your edge provider. User-agent filter onGPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot. If the number is zero: you're not being found, either because you don't prerender or because yourrobots.txtblocks. Check both. - Be honest in
robots.txt. Anyone who wants to be cited in AI answers has to allow these bots. Default-deny is a deliberate decision, not an accident. - Turn on prerendering. If you're on Lovable: use the Discoverability release. If you ship your own Vite SPAs: follow the Playwright SSG tutorial.
- Add JSON-LD. AI crawlers love structured data – it saves them parsing.
- Re-measure after 4 weeks. Crawl volume should double or quadruple. Cache hit rate should approach 70 %+.
Conclusion
If you talk about GEO in 2026 without talking about prerendering, you're talking about a poster on a wall with no light. The logs are clear: AI crawlers read HTML. They don't execute JavaScript. And they show up more often than most marketing decks claim.
If you want to know whether your site exists for ChatGPT, Claude and Perplexity, look at your edge logs. If you don't have that, write to us. We'll run the check for free.
Related Articles
SSR or Pre-Rendering? What We Evaluated for Mykeythai vs. Our Default Approach
Lovable now ships real SSR via TanStack Start. We checked for Mykeythai whether to switch – and why our pre-rendering ap…
Read more
Lovable SEO/AEO: Every App Discoverable by Google and ChatGPT From Day 1
Lovable ships server-side rendering, pre-rendering for existing apps, Semrush directly in the builder chat, and an on-de…
Read more
Deep DiveSEO vs. AEO – aren't they the same thing?
SEO and AEO sound like the same game with a different acronym. They're not. Where the two disciplines overlap, where the…
Read more
Prerendering: How to Turn a React SPA Into a Google-Friendly Site
React SPAs are invisible to crawlers. Prerendering fixes that – without Next.js, without an SSR server. How our Playwrig…
Read more
serponado.io – How Felix Grote Vibe-Coded a SEO Contest Site in 12 Hours
Felix Grote of seobuddha GmbH shipped serponado.io in 12 hours – one of the most creative entries in the 2026 SEO Contes…
Read more
Cloudflare vs. Vercel – Which One Should You Use When?
Vercel or Cloudflare? Both host your modern web app at the edge – but they pursue fundamentally different strategies. We…
Read moreSerponado SEO Audit: What 8 Cluster Articles Reveal About Internal Links, Canonicals and Fragments
Short SEO audit across all 8 Serponado cluster articles (DE+EN): link status, canonicals, fragment matches – with the fu…
Read more
GLM-5.2 vs. Kimi K2.7 Code: Two Open-Weight Releases in One Week – Two Very Different Bets
Within four days in June 2026, Z.ai (GLM-5.2) and Moonshot AI (Kimi K2.7 Code) shipped their next-generation open-weight…
Read more
Ponytail: The Best Code Is the Code You Never Wrote
A dev built Ponytail because his AI agents wrote 500 lines for a 5-line problem. The result: 80-94% less code, 47-77% ch…
Read more