The bot directory · 15 crawlers · refreshed monthly

Every AI crawler, explained

This is a maintained directory of the AI bots and user agents that crawl the web — the crawlers behind ChatGPT, Claude, Perplexity, Gemini, and the training datasets feeding every major model. Each profile explains what the bot does, whether it actually respects robots.txt, and gives copy-paste rules to allow or block it. To check what your own robots.txt currently permits, run the AI Crawler Access Checker.

OpenAI / 3

Anthropic / 3

Perplexity / 2

Google / 1

Google-ExtendedTrainingControls use of content for Gemini and Vertex AI training, separate from Googlebot.

Common Crawl / 1

CCBotTrainingCommon Crawl's crawler; its dataset is widely used to train LLMs.

ByteDance / 1

BytespiderTrainingIgnores robots.txtByteDance's crawler, used for training its AI models.

Apple / 1

Applebot-ExtendedTrainingControls use of content for training Apple's AI models (Apple Intelligence).

Amazon / 1

AmazonbotBothCrawls content for Amazon's search and AI products, including Alexa.

Meta / 1

Meta-ExternalAgentTrainingCrawls content to train Meta's AI models.

Cohere / 1

cohere-aiTrainingCohere's crawler used for training and grounding its models.

Work with these bots, not just read about them:

robots.txt Generator — build per-bot allow/block rules for all 15 crawlers at once.
AI Bot Log Analyzer — see which of these bots actually visit your site.
The complete guide to AI crawlers — background reading on how these bots differ.