Original research · 2026-11-11
Which AI crawlers do the top 10,000 websites block?
We fetched the robots.txt of every site in the Tranco top 10,000 and checked, for each of 28 AI crawlers, whether the site blocks it. 5,250 sites returned a usable robots.txt. The most-blocked agent is CCBot, which 16.3% of them block by name.
15.3%
block GPTBot (OpenAI training) by name
4.6%
block OAI-SearchBot (ChatGPT search) by name
11.2% vs 5.1%
average block rate, training agents vs search and user agents
What the numbers say
- Sites treat training and search differently. Training crawlers are blocked at an average of 11.2%, search and user-triggered agents at 5.1%. Blocking training while staying visible in AI answers is the common pattern, not the exception.
- GPTBot is blocked by name on 15.3% of sites, OAI-SearchBot on 4.6%. The two are separate opt-outs; see GPTBot vs OAI-SearchBot.
- Naming a bot is not the same as blocking it. OAI-SearchBot is named in 11.3% of robots.txt files but blocked in only 4.6%: most sites that mention ChatGPT's search crawler do it to let it in. GPTBot, by contrast, is blocked on 15.3% of 22.1% that name it.
- The most-mentioned agent appears in 22.1% of robots.txt files.
Full results
"By name" means the site has a group for that exact user-agent token and it disallows the root path. "Via *" means no named group exists, so the bot falls back to the wildcard group, and that group blocks everything. Percentages are of the 5,250 sites with a robots.txt.
| Agent | Operator | Blocked by name | Via * | Mentioned | Blocked in top 1,000 (any rule) |
|---|---|---|---|---|---|
| CCBot | Common Crawl | 16.3% (856) | 4.3% | 19.5% | 25.2% |
| Bytespider | ByteDance | 16.2% (852) | 4.4% | 18.0% | 26.2% |
| GPTBot | OpenAI | 15.3% (805) | 3.9% | 22.1% | 22.5% |
| ClaudeBot | Anthropic | 14.3% (749) | 4.1% | 19.6% | 22.7% |
| Google-Extended | 13.0% (683) | 4.0% | 18.6% | 19.9% | |
| Meta-ExternalAgent | Meta | 13.0% (680) | 4.3% | 15.3% | 21.3% |
| Diffbot | Diffbot | 12.5% (657) | 4.5% | 13.2% | 21.5% |
| Applebot-Extended | Apple | 12.3% (644) | 4.3% | 14.8% | 19.7% |
| Amazonbot | Amazon | 12.1% (636) | 4.2% | 14.7% | 18.1% |
| cohere-ai | Cohere | 12.0% (631) | 4.4% | 13.6% | 19.9% |
| PetalBot | Huawei | 10.6% (556) | 4.4% | 11.3% | 19.5% |
| PerplexityBot | Perplexity | 7.8% (409) | 3.9% | 14.1% | 18.3% |
| omgilibot | Webz.io | 7.0% (369) | 4.6% | 7.4% | 17.7% |
| YouBot | You.com | 6.8% (356) | 4.4% | 8.2% | 17.3% |
| ChatGPT-User | OpenAI | 6.5% (341) | 3.9% | 13.0% | 14.7% |
| Timpibot | Timpi | 6.5% (339) | 4.5% | 6.7% | 17.3% |
| GoogleOther | 6.1% (322) | 4.2% | 7.3% | 14.5% | |
| ImagesiftBot | The Hive | 5.9% (308) | 4.6% | 6.2% | 18.3% |
| Meta-ExternalFetcher | Meta | 5.0% (262) | 4.4% | 5.8% | 14.7% |
| AI2Bot | Allen Institute for AI | 4.9% (257) | 4.6% | 5.4% | 16.7% |
| Perplexity-User | Perplexity | 4.8% (251) | 4.3% | 7.6% | 16.1% |
| DuckAssistBot | DuckDuckGo | 4.7% (246) | 4.4% | 6.2% | 15.1% |
| OAI-SearchBot | OpenAI | 4.6% (244) | 3.9% | 11.3% | 12.9% |
| Claude-User | Anthropic | 4.6% (240) | 4.2% | 7.8% | 15.1% |
| Claude-SearchBot | Anthropic | 4.2% (220) | 4.2% | 7.7% | 14.9% |
| MistralAI-User | Mistral | 3.9% (206) | 4.6% | 4.9% | 14.7% |
| PanguBot | Huawei | 3.9% (203) | 4.7% | 4.1% | 15.5% |
| Applebot | Apple | 1.7% (91) | 3.7% | 5.2% | 6.6% |
Methodology
- Domains: the Tranco list Y895G, top 10,000. Tranco averages several popularity rankings and is the standard list for research of this kind.
- Fetch:
https://domain/robots.txt, falling back to the www host, on 2026-11-11. Responses that were not 200, were HTML, or had no User-agent line were discarded. Many top domains are CDNs and API hosts with no website, which is why 5,250 of 10,000 qualify. - Parsing: RFC 9309 group selection: a bot's own named group wins over User-agent: *. Blocked = the root path / is disallowed for that bot (longest match, Allow wins ties). Partial blocks (a Disallow on some paths only) are not counted as blocks.
- Limits: robots.txt shows intent, not enforcement. Sites can also block at the CDN or firewall, which this study cannot see, so real block rates are higher.
Use the data
The aggregates are free to reuse under CC BY 4.0 with a link to this page: robots-study.json. The crawler registry they are based on is at ai-crawlers.json. To check your own site against the same 28 agents, use the AI Crawler Access Checker, or generate rules with the robots.txt generator for AI bots.