Loading
AI crawler comparison
Anthropic's own crawler versus the open dataset that also feeds model training.
Short answer
ClaudeBot and CCBot do the same job — AI model training — so the choice between them is not either/or. If you want to opt out of that use entirely, you need a rule for each of them; blocking one leaves the other free to crawl.
| ClaudeBot | CCBot | |
|---|---|---|
| Operator | Anthropic | Common Crawl |
| Purpose | AI model training | AI model training |
| robots.txt | Respects robots.txt | Respects robots.txt |
| Published IP ranges | None published | None published |
| Official docs | Yes | Yes |
| Blocking costs you | No traffic — training only | No traffic — training only |
ClaudeBot is Anthropic's primary web crawler, collecting publicly available content that may be used to train the Claude family of models. It is the Anthropic equivalent of OpenAI's GPTBot, and the user agent to target if you want to opt out of Claude training specifically.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: ClaudeBot.
CCBot builds Common Crawl, the open web archive that has served as foundational training data for many large language models, including early GPT models. Blocking CCBot is the single broadest AI-training opt-out available, since dozens of model builders consume the Common Crawl dataset downstream.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: CCBot.
User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: ClaudeBot Disallow: / User-agent: CCBot Allow: /
Both are AI model training agents, but they belong to different operators — ClaudeBot to Anthropic, CCBot to Common Crawl. ClaudeBot: Crawls content to train Anthropic's Claude models. CCBot: Common Crawl's crawler; its dataset is widely used to train LLMs.
No. robots.txt matches on the user-agent token, so a group naming ClaudeBot applies only to ClaudeBot. CCBot reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
It depends which outcome you want. Blocking ClaudeBot opts you out of Anthropic's model training and costs you no traffic today, because training crawlers send no visitors. Blocking CCBot opts you out of Common Crawl's training data on the same terms. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
ClaudeBot: Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals. CCBot: Common Crawl is a nonprofit with a long public record of honoring robots.txt and crawl-delay directives.
Yes — they send different user-agent strings, so grep for each token separately: grep -i "ClaudeBot" and grep -i "CCBot". Be aware that a user-agent header is self-declared and anything can send either string. Neither operator publishes IP ranges for these agents, so log entries claiming them cannot be verified.
Part of our directory of every known AI crawler. Last verified: 2026-09-13.