AI crawler profile
What is CCBot? How to allow or block Common Crawl's crawler
User-agent string to match in robots.txt and server logs: CCBot
Operator
Common Crawl
Purpose
AI model training
robots.txt
Respects robots.txt
Official docs
CCBot builds Common Crawl, the open web archive that has served as foundational training data for many large language models, including early GPT models. Blocking CCBot is the single broadest AI-training opt-out available, since dozens of model builders consume the Common Crawl dataset downstream.
Does CCBot respect robots.txt?
Common Crawl is a nonprofit with a long public record of honoring robots.txt and crawl-delay directives.
Verify it's really CCBot
Common Crawl publishes no IP-range file for CCBot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.
Block CCBot with robots.txt
User-agent: CCBot Disallow: /
Explicitly allow CCBot
User-agent: CCBot Allow: /
Block CCBot at the server or CDN
robots.txt is the right first step for CCBot, since Common Crawl honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.
if ($http_user_agent ~* "CCBot") {
return 403;
}RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} CCBot [NC]
RewriteRule .* - [F,L](http.user_agent contains "CCBot")
Find CCBot in your server logs
grep -i "CCBot" /var/log/nginx/access.log | wc -l
Should you block CCBot?
Blocking CCBot opts your content out of Common Crawl's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
Check and monitor CCBot on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks CCBot.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 28 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real CCBot hits, entirely in your browser.
Related reading
- List of AI crawlers and their user agents — how CCBot fits among every other AI crawler, in one table.
- Can AI crawlers execute JavaScript? — whether CCBot sees content your site renders client-side.
- Should you block AI bots? — the full decision framework for Common Crawl and the rest.
Frequently asked questions
What is CCBot?
CCBot is Common Crawl's web crawler for collecting AI training data. Common Crawl's crawler; its dataset is widely used to train LLMs.
Does CCBot respect robots.txt?
Common Crawl is a nonprofit with a long public record of honoring robots.txt and crawl-delay directives.
How do I block CCBot?
Add "User-agent: CCBot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking CCBot hurt my Google rankings?
No. CCBot is separate from Googlebot, which handles Google Search indexing. Blocking CCBot has no effect on your traditional search rankings.
How can I tell if CCBot is crawling my site?
Search your server access logs for the string "CCBot" — for example: grep -i "CCBot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including CCBot, with per-path breakdowns.
How do I use CCBot?
You don't — CCBot isn't a tool you run. It's Common Crawl's own crawler, operated by Common Crawl, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "CCBot" as your user agent would be impersonating Common Crawl.
Where is the official Common Crawl documentation for CCBot?
Common Crawl publishes it at https://commoncrawl.org/ccbot. That page is the authoritative source for the user-agent string and Common Crawl's stated crawling policy.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.