HelpAEO
Log inScan my site
← All the checks
Access

robots.txt allows AI crawlers

  • Our estimate: high impact
  • Runs in the free scan

What we check

Checks whether robots.txt disallows any of six AI answer engines (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot) from fetching /, merging every matching User-agent group per RFC 9309 (the longest matching rule wins; an equal-length allow beats a disallow) and treating Content-Signal: ai-input=no as a disallow. AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) being blocked is reported but never fails this check. An absent or empty robots.txt passes by default — per RFC 9309, no robots.txt means everything is allowed.

Why it matters

If robots.txt disallows a named AI answer engine, that file is your site's own instruction telling it not to fetch this page at all.

More checks in Access

Can agents reach and crawl the site at all?

Scan your site to see if you pass.

Run the free scan →Free score · no signup · about a minute