Loading
AI crawler comparison
The two biggest AI training crawlers, compared on documentation, IP verification, and what blocking each actually costs.
Short answer
GPTBot and ClaudeBot do the same job — AI model training — so the choice between them is not either/or. If you want to opt out of that use entirely, you need a rule for each of them; blocking one leaves the other free to crawl.
| GPTBot | ClaudeBot | |
|---|---|---|
| Operator | OpenAI | Anthropic |
| Purpose | AI model training | AI model training |
| robots.txt | Respects robots.txt | Respects robots.txt |
| Published IP ranges | Yes — verifiable | None published |
| Official docs | Yes | Yes |
| Blocking costs you | No traffic — training only | No traffic — training only |
GPTBot is OpenAI's web crawler for gathering training data. Content it collects may be used to improve future GPT models. It is OpenAI's broadest crawler, and the one most site owners mean when they talk about "blocking ChatGPT" — though blocking it does not remove you from ChatGPT's live search answers, which use OAI-SearchBot instead.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: GPTBot.
ClaudeBot is Anthropic's primary web crawler, collecting publicly available content that may be used to train the Claude family of models. It is the Anthropic equivalent of OpenAI's GPTBot, and the user agent to target if you want to opt out of Claude training specifically.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: ClaudeBot.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Allow: /
Both are AI model training agents, but they belong to different operators — GPTBot to OpenAI, ClaudeBot to Anthropic. GPTBot: Crawls content to train OpenAI's models. ClaudeBot: Crawls content to train Anthropic's Claude models.
No. robots.txt matches on the user-agent token, so a group naming GPTBot applies only to GPTBot. ClaudeBot reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
It depends which outcome you want. Blocking GPTBot opts you out of OpenAI's model training and costs you no traffic today, because training crawlers send no visitors. Blocking ClaudeBot opts you out of Anthropic's training data on the same terms. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
GPTBot: OpenAI documents GPTBot's IP ranges and states it honors robots.txt disallow rules. ClaudeBot: Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals.
Yes — they send different user-agent strings, so grep for each token separately: grep -i "GPTBot" and grep -i "ClaudeBot". Be aware that a user-agent header is self-declared and anything can send either string. For a check that cannot be spoofed, match the request IP against the published ranges: GPTBot at https://openai.com/gptbot.json.
Part of our directory of every known AI crawler. Last verified: 2026-09-13.