AI crawler comparison
Claude-User vs Claude-SearchBot: what's the difference?
Two Claude agents that both serve live answers, at different stages of the request.
Short answer
Claude-User and Claude-SearchBot do the same job — AI search and live answers — so the choice between them is not either/or. If you want to opt out of that use entirely, you need a rule for each of them; blocking one leaves the other free to crawl.
Side by side
| Claude-User | Claude-SearchBot | |
|---|---|---|
| Operator | Anthropic | Anthropic |
| Purpose | AI search & live answers | AI search & live answers |
| robots.txt | Respects robots.txt | Respects robots.txt |
| Published IP ranges | None published | None published |
| Official docs | Yes | Yes |
| Blocking costs you | AI-answer citations | AI-answer citations |
What Claude-User does
Claude-User fetches individual pages when someone asks Claude to look at a URL or search the web mid-conversation. Like ChatGPT-User, it represents live human attention rather than bulk crawling — hits from it mean Claude users are being pointed at your content.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Claude-User.
What Claude-SearchBot does
Claude-SearchBot indexes pages to improve the quality of Claude's web-search answers and citations. Blocking it reduces your chances of being cited by Claude without affecting training data collection, which ClaudeBot handles separately.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Claude-SearchBot.
Block both
User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Disallow: /
Block Claude-User, allow Claude-SearchBot
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: Claude-User Disallow: / User-agent: Claude-SearchBot Allow: /
Check your own site
- AI Crawler Access Checker — see which of the two your live robots.txt currently allows.
- robots.txt Generator for AI Bots — build per-bot rules for both agents and the other 26.
- AI Bot Log Analyzer — paste your access log and see which of them is actually visiting.
- Should you block AI bots? — the decision framework behind the split above.
Frequently asked questions
What is the difference between Claude-User and Claude-SearchBot?
Both are AI search and live answers agents, but they belong to Anthropic at different stages of the same pipeline. Claude-User: Fetches pages on behalf of a user during a Claude conversation. Claude-SearchBot: Indexes content to inform Claude's search-style answers.
Does blocking Claude-User also block Claude-SearchBot?
No. robots.txt matches on the user-agent token, so a group naming Claude-User applies only to Claude-User. Claude-SearchBot reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
Should I block Claude-User, Claude-SearchBot, or both?
It depends which outcome you want. Blocking Claude-User removes your pages from Anthropic's cited AI answers — that is a visibility loss, not a privacy win. Blocking Claude-SearchBot removes you from the answers Anthropic's assistant shows its users. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
Do Claude-User and Claude-SearchBot both respect robots.txt?
Claude-User: Anthropic documents Claude-User alongside its other agents; it fetches single user-requested pages rather than crawling at scale. Claude-SearchBot: Anthropic states Claude-SearchBot respects robots.txt directives.
Can I tell Claude-User and Claude-SearchBot apart in my server logs?
Yes — they send different user-agent strings, so grep for each token separately: grep -i "Claude-User" and grep -i "Claude-SearchBot". Be aware that a user-agent header is self-declared and anything can send either string. Neither operator publishes IP ranges for these agents, so log entries claiming them cannot be verified.
Other comparisons
Part of our directory of every known AI crawler. Last verified: 2026-08-14.