AI crawler comparison
ClaudeBot vs Claude-User: what's the difference?
Anthropic split training from user-triggered fetching across two agents; blocking the wrong one has the opposite effect from the one intended.
Short answer
They are not interchangeable. ClaudeBot handles AI model training, Claude-User handles AI search and live answers. Blocking one has no effect on the other, so a robots.txt rule aimed at the wrong agent produces exactly the outcome you were trying to avoid.
Side by side
| ClaudeBot | Claude-User | |
|---|---|---|
| Operator | Anthropic | Anthropic |
| Purpose | AI model training | AI search & live answers |
| robots.txt | Respects robots.txt | Respects robots.txt |
| Published IP ranges | None published | None published |
| Official docs | Yes | Yes |
| Blocking costs you | No traffic — training only | AI-answer citations |
What ClaudeBot does
ClaudeBot is Anthropic's primary web crawler, collecting publicly available content that may be used to train the Claude family of models. It is the Anthropic equivalent of OpenAI's GPTBot, and the user agent to target if you want to opt out of Claude training specifically.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: ClaudeBot.
What Claude-User does
Claude-User fetches individual pages when someone asks Claude to look at a URL or search the web mid-conversation. Like ChatGPT-User, it represents live human attention rather than bulk crawling — hits from it mean Claude users are being pointed at your content.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Claude-User.
Block both
User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: /
Block ClaudeBot, allow Claude-User
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: ClaudeBot Disallow: / User-agent: Claude-User Allow: /
Check your own site
- AI Crawler Access Checker — see which of the two your live robots.txt currently allows.
- robots.txt Generator for AI Bots — build per-bot rules for both agents and the other 26.
- AI Bot Log Analyzer — paste your access log and see which of them is actually visiting.
- Should you block AI bots? — the decision framework behind the split above.
Frequently asked questions
What is the difference between ClaudeBot and Claude-User?
ClaudeBot is Anthropic's AI model training agent — Crawls content to train Anthropic's Claude models. Claude-User is Anthropic's AI search and live answers agent — Fetches pages on behalf of a user during a Claude conversation. They are separate user agents with separate robots.txt rules.
Does blocking ClaudeBot also block Claude-User?
No. robots.txt matches on the user-agent token, so a group naming ClaudeBot applies only to ClaudeBot. Claude-User reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
Should I block ClaudeBot, Claude-User, or both?
It depends which outcome you want. Blocking ClaudeBot opts you out of Anthropic's model training and costs you no traffic today, because training crawlers send no visitors. Blocking Claude-User removes you from the answers Anthropic's assistant shows its users. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
Do ClaudeBot and Claude-User both respect robots.txt?
ClaudeBot: Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals. Claude-User: Anthropic documents Claude-User alongside its other agents; it fetches single user-requested pages rather than crawling at scale.
Can I tell ClaudeBot and Claude-User apart in my server logs?
Yes — they send different user-agent strings, so grep for each token separately: grep -i "ClaudeBot" and grep -i "Claude-User". Be aware that a user-agent header is self-declared and anything can send either string. Neither operator publishes IP ranges for these agents, so log entries claiming them cannot be verified.
Other comparisons
Part of our directory of every known AI crawler. Last verified: 2026-08-14.