Every major AI company operates one or more web crawlers, identified by a distinct user-agent string in your server logs and controllable independently through robots.txt. Here's the current list, what each one does, and why they're not interchangeable.
Why this list matters
Most site owners have a robots.txt file that mentions Googlebot and maybe Bingbot, and nothing else. But AI companies run separate crawlers for two fundamentally different jobs:
- Training crawlers collect content to train future model versions. Once collected, that data may be used indefinitely across model generations.
- Search/browsing crawlers fetch pages in real time to answer a specific user's question — closer in spirit to how Googlebot indexes pages for search results, except often triggered live, at query time.
Blocking one doesn't block the other, and most companies operate both under different names.
For a deep dive on any single crawler — including whether it actually honors robots.txt and copy-paste rules to allow or block it — see our AI bot directory, which has a dedicated page for every agent in this list.
The list
| User agent | Operator | Purpose | What it does |
|---|---|---|---|
GPTBot | OpenAI | Training | Crawls content to train OpenAI's models |
OAI-SearchBot | OpenAI | Search | Powers search-style results surfaced in ChatGPT |
ChatGPT-User | OpenAI | Search | Fetches pages a user asks ChatGPT to browse live |
ClaudeBot | Anthropic | Training | Crawls content to train Claude models |
Claude-User | Anthropic | Search | Fetches pages on a user's behalf during a conversation |
Claude-SearchBot | Anthropic | Search | Indexes content for Claude's search-style answers |
PerplexityBot | Perplexity | Search | Crawls and indexes pages for Perplexity's answer engine |
Perplexity-User | Perplexity | Search | Fetches a page a user asks Perplexity to browse live |
Google-Extended | Training | Controls use for Gemini/Vertex AI training, separate from Googlebot | |
CCBot | Common Crawl | Training | Its dataset is widely reused to train LLMs across the industry |
Bytespider | ByteDance | Training | Used to train ByteDance's AI models |
Applebot-Extended | Apple | Training | Controls use for training Apple Intelligence models |
Amazonbot | Amazon | Both | Crawls for Amazon's search and AI products, including Alexa |
Meta-ExternalAgent | Meta | Training | Crawls content to train Meta's AI models |
Meta-ExternalFetcher | Meta | Search | Fetches links for Meta AI on a user's request; may bypass robots.txt |
cohere-ai | Cohere | Training | Used for training and grounding Cohere's models |
Applebot | Apple | Search | Powers Siri and Spotlight; the crawler behind Applebot-Extended |
GoogleOther | Both | Google's generic crawler for research and internal products | |
MistralAI-User | Mistral | Search | Fetches pages for Le Chat when a user's question needs them |
AI2Bot | Allen Institute for AI | Training | Collects content for open research models and datasets |
DuckAssistBot | DuckDuckGo | Search | Fetches pages for DuckDuckGo's AI-assisted answers |
YouBot | You.com | Search | Crawls for You.com's AI search answers |
PetalBot | Huawei | Both | Petal Search crawler, also feeding Huawei's AI products |
Diffbot | Diffbot | Both | Structured-data extraction licensed to customers incl. AI companies |
ImagesiftBot | The Hive | Training | Crawls images for Hive's visual-AI products |
omgilibot | Webz.io | Training | Web-data broker whose feeds are sold to AI builders |
Timpibot | Timpi | Search | Crawler for Timpi's decentralized search index |
PanguBot | Huawei | Training | Associated with training Huawei's PanGu LLMs; undocumented |
This list changes as companies launch new products — check our AI Crawler Access Checker for the current set applied against your own site, and see our robots.txt Generator for AI Bots to build rules for all of them at once.
A note on Google specifically
Google-Extended is not the same as Googlebot. Blocking Googlebot removes you from Google Search entirely — almost never what you want. Google-Extended only controls whether Google can use your content to train Gemini and improve AI features; it has no effect on your regular search ranking. These are easy to confuse and worth double-checking in any robots.txt you inherit or edit.
Multiple crawlers, same company
Notice how many companies split training from search into separate bots. This is deliberate — it lets a site say "don't train on my content, but you can still cite me in live answers," which is a very different stance from blocking a company's AI presence entirely. If your goal is specifically to avoid training-data usage while remaining discoverable in AI answers, target the training-labeled bots only (see our guide on the trade-offs of blocking AI bots).
How to check and control access
- Check what's currently allowed with the AI Crawler Access Checker — it fetches your live robots.txt and reports allow/block status for every bot in this list.
- Generate new rules with the robots.txt Generator for AI Bots — presets for allow-all, block-training-only, or block-all, plus individual toggles.
- Re-check periodically. New crawlers appear as AI companies ship new products; we refresh this list monthly.
Limitations of robots.txt as a control
robots.txt is a voluntary standard. The companies listed here are generally compliant, publishing documentation confirming they respect it — but it isn't a technical enforcement mechanism, and it can't stop a crawler that chooses to ignore it, or an unlisted/unknown bot. Treat it as the correct first step for expressing your preference, not an airtight guarantee.