Loading
AI crawler comparison
Training crawler versus the index behind Claude's cited search answers.
Short answer
They are not interchangeable. ClaudeBot handles AI model training, Claude-SearchBot handles AI search and live answers. Blocking one has no effect on the other, so a robots.txt rule aimed at the wrong agent produces exactly the outcome you were trying to avoid.
| ClaudeBot | Claude-SearchBot | |
|---|---|---|
| Operator | Anthropic | Anthropic |
| Purpose | AI model training | AI search & live answers |
| robots.txt | Respects robots.txt | Respects robots.txt |
| Published IP ranges | None published | None published |
| Official docs | Yes | Yes |
| Blocking costs you | No traffic — training only | AI-answer citations |
ClaudeBot is Anthropic's primary web crawler, collecting publicly available content that may be used to train the Claude family of models. It is the Anthropic equivalent of OpenAI's GPTBot, and the user agent to target if you want to opt out of Claude training specifically.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: ClaudeBot.
Claude-SearchBot indexes pages to improve the quality of Claude's web-search answers and citations. Blocking it reduces your chances of being cited by Claude without affecting training data collection, which ClaudeBot handles separately.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Claude-SearchBot.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: /
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: /
ClaudeBot is Anthropic's AI model training agent — Crawls content to train Anthropic's Claude models. Claude-SearchBot is Anthropic's AI search and live answers agent — Indexes content to inform Claude's search-style answers. They are separate user agents with separate robots.txt rules.
No. robots.txt matches on the user-agent token, so a group naming ClaudeBot applies only to ClaudeBot. Claude-SearchBot reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
It depends which outcome you want. Blocking ClaudeBot opts you out of Anthropic's model training and costs you no traffic today, because training crawlers send no visitors. Blocking Claude-SearchBot removes you from the answers Anthropic's assistant shows its users. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
ClaudeBot: Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals. Claude-SearchBot: Anthropic states Claude-SearchBot respects robots.txt directives.
Yes — they send different user-agent strings, so grep for each token separately: grep -i "ClaudeBot" and grep -i "Claude-SearchBot". Be aware that a user-agent header is self-declared and anything can send either string. Neither operator publishes IP ranges for these agents, so log entries claiming them cannot be verified.
Part of our directory of every known AI crawler. Last verified: 2026-09-13.