AI crawler profile
What is Claude-User? How to allow or block Anthropic's crawler
Operator
Anthropic
Purpose
AI search & live answers
robots.txt
Respects robots.txt
Official docs
Claude-User fetches individual pages when someone asks Claude to look at a URL or search the web mid-conversation. Like ChatGPT-User, it represents live human attention rather than bulk crawling — hits from it mean Claude users are being pointed at your content.
Does Claude-User respect robots.txt?
Anthropic documents Claude-User alongside its other agents; it fetches single user-requested pages rather than crawling at scale.
Block Claude-User with robots.txt
User-agent: Claude-User Disallow: /
Explicitly allow Claude-User
User-agent: Claude-User Allow: /
Should you block Claude-User?
For most sites, no. Claude-User powers visibility: blocking it removes your pages from the cited answers Anthropic's assistant shows its users — the AI-era equivalent of de-indexing yourself from a search engine. Block it only if you deliberately don't want that audience. Our guide on blocking AI bots covers the trade-offs in detail.
Check and monitor Claude-User on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Claude-User.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 15 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Claude-User hits, entirely in your browser.
Frequently asked questions
What is Claude-User?
Claude-User is Anthropic's agent for AI search and live answers. Fetches pages on behalf of a user during a Claude conversation.
Does Claude-User respect robots.txt?
Anthropic documents Claude-User alongside its other agents; it fetches single user-requested pages rather than crawling at scale.
How do I block Claude-User?
Add "User-agent: Claude-User" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking Claude-User hurt my Google rankings?
No. Claude-User is separate from Googlebot, which handles Google Search indexing. Blocking Claude-User has no effect on your traditional search rankings, but it does remove your pages from the AI answers Anthropic's assistant serves to its users.
How can I tell if Claude-User is crawling my site?
Search your server access logs for the string "Claude-User". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Claude-User, with per-path breakdowns.
More Anthropic agents
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-15.