GeoPromptTracker

AI crawler profile

What is ClaudeBot? How to allow or block Anthropic's crawler

User-agent string to match in robots.txt and server logs: ClaudeBot

Operator

Anthropic

Purpose

AI model training

robots.txt

Respects robots.txt

Official docs

Anthropic documentation ↗

ClaudeBot is Anthropic's primary web crawler, collecting publicly available content that may be used to train the Claude family of models. It is the Anthropic equivalent of OpenAI's GPTBot, and the user agent to target if you want to opt out of Claude training specifically.

Does ClaudeBot respect robots.txt?

Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals.

Verify it's really ClaudeBot

Anthropic publishes no IP-range file for ClaudeBot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.

Block ClaudeBot with robots.txt

robots.txt — block
User-agent: ClaudeBot
Disallow: /

Explicitly allow ClaudeBot

robots.txt — allow
User-agent: ClaudeBot
Allow: /

Block ClaudeBot at the server or CDN

robots.txt is the right first step for ClaudeBot, since Anthropic honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.

nginx
if ($http_user_agent ~* "ClaudeBot") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "ClaudeBot")

Find ClaudeBot in your server logs

shell
grep -i "ClaudeBot" /var/log/nginx/access.log | wc -l

Should you block ClaudeBot?

Blocking ClaudeBot opts your content out of Anthropic's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.

Check and monitor ClaudeBot on your site

Related reading

Frequently asked questions

What is ClaudeBot?

ClaudeBot is Anthropic's web crawler for collecting AI training data. Crawls content to train Anthropic's Claude models.

Does ClaudeBot respect robots.txt?

Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals.

How do I block ClaudeBot?

Add "User-agent: ClaudeBot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking ClaudeBot hurt my Google rankings?

No. ClaudeBot is separate from Googlebot, which handles Google Search indexing. Blocking ClaudeBot has no effect on your traditional search rankings.

How can I tell if ClaudeBot is crawling my site?

Search your server access logs for the string "ClaudeBot" — for example: grep -i "ClaudeBot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including ClaudeBot, with per-path breakdowns.

How do I use ClaudeBot?

You don't — ClaudeBot isn't a tool you run. It's Anthropic's own crawler, operated by Anthropic, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "ClaudeBot" as your user agent would be impersonating Anthropic.

Where is the official Anthropic documentation for ClaudeBot?

Anthropic publishes it at https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. That page is the authoritative source for the user-agent string and Anthropic's stated crawling policy.

More Anthropic agents

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.