What we check
Checks whether robots.txt disallows any of six AI answer engines (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot) from fetching /, merging every matching User-agent group per RFC 9309 (the longest matching rule wins; an equal-length allow beats a disallow) and treating Content-Signal: ai-input=no as a disallow. AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) being blocked is reported but never fails this check. An absent or empty robots.txt passes by default — per RFC 9309, no robots.txt means everything is allowed.
Why it matters
If robots.txt disallows a named AI answer engine, that file is your site's own instruction telling it not to fetch this page at all.
More checks in Access
Can agents reach and crawl the site at all?
HelpAEO