How we score AI crawlability
This page explains, in full, how AI Crawler Check turns a website into a 0-100 AI crawlability score. The model is deterministic: the same site scores the same way every time, and every point traces back to a specific, named check.
How AI crawlers read the web
AI search and training crawlers are not browsers. With the sole practical exception of Google's Gemini, they do not execute JavaScript — they fetch your raw HTML and read it. They respect robots.txt on a per-purpose basis (training vs search vs live user fetches). And the single most common failure is invisible: a CDN or WAF returning 403 or a challenge to a crawler's user-agent even though robots.txt allows it.
The five pillars
- Access (30%) — whether each AI crawler is allowed by robots.txt and not silently blocked by a CDN or firewall
- Renderability (25%) — whether your content survives without JavaScript, since AI crawlers do not run it
- Extractability & Structure (20%) — whether headings, main content and clean markup let a crawler pull the meaning out
- Trust & Metadata (15%) — JSON-LD structured data, dates, authorship and other signals AI engines use to trust and cite a page
- Discovery & Indexes (10%) — sitemaps, llms.txt and internal links that help crawlers find every page
The 35 checks
The pillars break down into 35 individual checks — from "is GPTBot allowed in robots.txt" and "does content survive without JavaScript" to "is there valid JSON-LD" and "are sitemap lastmod dates trustworthy." Each returns pass, warn, fail or not-applicable, with the evidence that produced it.
The 15 AI crawlers
Access is tested per crawler, grouped by what the crawler is for:
- Search & answer engines: OAI-SearchBot, Claude-SearchBot, Googlebot, Bingbot, PerplexityBot, DuckAssistBot, Applebot
- Live user fetchers: ChatGPT-User, Claude-User
- Model training: GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot
Grades
Scores map to four grades: AI-Ready (90-100), Good (75-89), Needs Work (50-74) and Invisible to AI (0-49). The grade is a fast read; the fix list is where the value is.