The State of AI Crawlability

The Index benchmarks how well-known websites score for AI search visibility, so you can see how ready the web is for AI search — and how your own site compares. Each site is scanned with the same 35 checks and the same 15 crawlers, so the numbers are directly comparable.

What the Index tracks

• The median AI crawlability score across the tracked sites.
• The share of sites that block at least one AI search crawler.
• How much of each site's content is visible without JavaScript.
• Which AI crawler is blocked most often across the web.

Common reasons sites score low

• A single-page framework renders content only in the browser, so crawlers fetch an empty shell.
• A CDN or firewall rule blocks AI user-agents even though robots.txt allows them.
• Thin or missing structured data, so AI engines can't confidently attribute or cite the page.
• Sitemaps with stale or identical dates that make freshness impossible to trust.

Why it matters

Many large, well-built sites still score poorly for AI crawlability. The Index makes those patterns visible and gives every site a public, shareable benchmark to improve against — and a way to prove progress over time as AI search becomes a primary way people discover the web.