Can AI see your site?
AI Crawler Check tests any website against the 15 biggest AI crawlers and gives you a 0-100 AI crawlability score — a clear measure of how easily AI search engines like ChatGPT, Claude, Perplexity and Google's AI Overviews can crawl, read and cite your pages. It runs 35 technical checks and shows exactly what each crawler can and cannot see.
What it measures
Every scan scores five pillars:
- Access (30%) — whether each AI crawler is allowed by robots.txt and not silently blocked by a CDN or firewall
- Renderability (25%) — whether your content survives without JavaScript, since AI crawlers do not run it
- Extractability & Structure (20%) — whether headings, main content and clean markup let a crawler pull the meaning out
- Trust & Metadata (15%) — JSON-LD structured data, dates, authorship and other signals AI engines use to trust and cite a page
- Discovery & Indexes (10%) — sitemaps, llms.txt and internal links that help crawlers find every page
The 15 AI crawlers it simulates
Each one is probed with its exact, vendor-published user-agent, so you see real access — including crawlers that robots.txt allows but a CDN or firewall silently blocks.
- Search & answer engines: OAI-SearchBot, Claude-SearchBot, Googlebot, Bingbot, PerplexityBot, DuckAssistBot, Applebot
- Live user fetchers: ChatGPT-User, Claude-User
- Model training: GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot
Why this matters
Almost no AI crawler executes JavaScript. If your content only renders client-side, GPTBot and ClaudeBot fetch an empty page — so your site can be invisible in AI answers even when it looks perfect in a browser. AI Crawler Check catches that, measures exactly what percentage of your content is visible without JavaScript, and hands you a prioritised fix list.
Frequently asked questions
Do AI crawlers run JavaScript?
Almost none do. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and the other major AI crawlers fetch your raw HTML and never execute JavaScript. If your content only appears after JavaScript runs, those crawlers see an empty page — which is why a single-page app can be invisible to AI search even when it looks fine in a browser.
What is an AI crawlability score?
It is a 0-100 measure of how easily AI search engines can crawl, read and cite your site. AI Crawler Check runs 35 technical checks across five pillars — Access, Renderability, Extractability, Trust and Discovery — and weights them into one score with a plain-English fix list.
Which AI crawlers does it test?
The 15 biggest, each with its exact published user-agent: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); Googlebot and Bingbot; PerplexityBot; DuckAssistBot; Meta-ExternalAgent; Amazonbot; Applebot; Bytespider; and CCBot.
Why would a crawler be "edge-blocked"?
Your robots.txt can allow a crawler while your CDN or WAF (Cloudflare, Akamai, a bot-manager rule) still returns 403 or a challenge to its user-agent. This is the most common way sites go invisible to AI without realising it, because it never shows up in robots.txt.
Is it free?
Yes. Every scan gives you the full on-screen result and the 15-crawler matrix. A free account unlocks the downloadable PDF report and saved history; paid plans add a deeper 50-page crawl and scheduled monitoring with email alerts.