AI companies operate crawlers that fetch web content for two very different purposes: training future models, and answering live user questions by browsing the web in real time. Your robots.txt file can allow or block each of these independently, bot by bot — and most site owners have no idea what their current rules actually say once you get past the standard Googlebot line.
Why it matters for AI search: if a bot like OAI-SearchBot or PerplexityBot is blocked, your pages simply can't be cited in that platform's AI-generated answers, no matter how good your content is. Conversely, if you don't want your content used to train future models, you need to explicitly block bots like GPTBot, ClaudeBot, or Google-Extended — omitting them doesn't block them, since robots.txt defaults to allow.
How this tool works: enter your domain and the tool fetches your live robots.txt through a server-side proxy, parses the User-agent groups and Allow/Disallow rules, and checks each of the major AI crawlers — including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended, Amazonbot, Meta-ExternalAgent, and cohere-ai — against those rules. Each bot is reported as Allowed, Blocked, or Not mentioned (which defaults to allowed under the robots.txt spec).
Limitations: this tool evaluates access to your site root (/) using the robots.txt rules currently live at your domain — it doesn't crawl your full site to check path-specific rules on individual pages. robots.txt is also a voluntary standard: it signals your preference to well-behaved crawlers but can't technically enforce blocking against a bot that ignores it. If you want to generate or edit these rules rather than just check them, use the companion robots.txt Generator for AI Bots.