GeoPromptTracker

AI crawler profile

What is Claude-SearchBot? How to allow or block Anthropic's crawler

User-agent string to match in robots.txt and server logs: Claude-SearchBot

Operator

Anthropic

Purpose

AI search & live answers

robots.txt

Respects robots.txt

Official docs

Anthropic documentation ↗

Claude-SearchBot indexes pages to improve the quality of Claude's web-search answers and citations. Blocking it reduces your chances of being cited by Claude without affecting training data collection, which ClaudeBot handles separately.

Does Claude-SearchBot respect robots.txt?

Anthropic states Claude-SearchBot respects robots.txt directives.

Verify it's really Claude-SearchBot

Anthropic publishes no IP-range file for Claude-SearchBot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.

Block Claude-SearchBot with robots.txt

robots.txt — block
User-agent: Claude-SearchBot
Disallow: /

Explicitly allow Claude-SearchBot

robots.txt — allow
User-agent: Claude-SearchBot
Allow: /

Block Claude-SearchBot at the server or CDN

robots.txt is the right first step for Claude-SearchBot, since Anthropic honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.

nginx
if ($http_user_agent ~* "Claude-SearchBot") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Claude-SearchBot [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "Claude-SearchBot")

Find Claude-SearchBot in your server logs

shell
grep -i "Claude-SearchBot" /var/log/nginx/access.log | wc -l

Should you block Claude-SearchBot?

For most sites, no. Claude-SearchBot powers visibility: blocking it removes your pages from the cited answers Anthropic's assistant shows its users — the AI-era equivalent of de-indexing yourself from a search engine. Block it only if you deliberately don't want that audience. Our guide on blocking AI bots covers the trade-offs in detail.

Check and monitor Claude-SearchBot on your site

Related reading

Frequently asked questions

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's agent for AI search and live answers. Indexes content to inform Claude's search-style answers.

Does Claude-SearchBot respect robots.txt?

Anthropic states Claude-SearchBot respects robots.txt directives.

How do I block Claude-SearchBot?

Add "User-agent: Claude-SearchBot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking Claude-SearchBot hurt my Google rankings?

No. Claude-SearchBot is separate from Googlebot, which handles Google Search indexing. Blocking Claude-SearchBot has no effect on your traditional search rankings, but it does remove your pages from the AI answers Anthropic's assistant serves to its users.

How can I tell if Claude-SearchBot is crawling my site?

Search your server access logs for the string "Claude-SearchBot" — for example: grep -i "Claude-SearchBot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Claude-SearchBot, with per-path breakdowns.

How do I use Claude-SearchBot?

You don't — Claude-SearchBot isn't a tool you run. It's Anthropic's own crawler, operated by Anthropic, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "Claude-SearchBot" as your user agent would be impersonating Anthropic.

Where is the official Anthropic documentation for Claude-SearchBot?

Anthropic publishes it at https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. That page is the authoritative source for the user-agent string and Anthropic's stated crawling policy.

More Anthropic agents

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.