The bot directory · 28 crawlers · refreshed monthly

Every AI crawler, explained

This is a maintained directory of the AI bots and user agents that crawl the web — the crawlers behind ChatGPT, Claude, Perplexity, Gemini, and the training datasets feeding every major model. Each profile explains what the bot does, whether it actually respects robots.txt, and gives copy-paste rules to allow or block it. To check what your own robots.txt currently permits, run the AI Crawler Access Checker.

Respects robots.txt
Partially respects robots.txt
Ignores robots.txt
Compliance unknown

OpenAI / 3

Anthropic / 3

Perplexity / 2

Google / 2

Common Crawl / 1

CCBotTrainingCommon Crawl's crawler; its dataset is widely used to train LLMs.

ByteDance / 1

BytespiderTrainingIgnores robots.txtByteDance's crawler, used for training its AI models.

Apple / 2

Amazon / 1

AmazonbotBothCrawls content for Amazon's search and AI products, including Alexa.

Meta / 2

Cohere / 1

cohere-aiTrainingCohere's crawler used for training and grounding its models.

Mistral / 1

MistralAI-UserSearchFetches pages when a Le Chat user's question requires them.

Allen Institute for AI / 1

AI2BotTrainingCrawls the web for the Allen Institute's open AI research models.

DuckDuckGo / 1

DuckAssistBotSearchFetches pages to generate DuckDuckGo's AI-assisted answers.

You.com / 1

YouBotSearchCrawls and fetches pages for You.com's AI search answers.

Huawei / 2

Diffbot / 1

DiffbotBothExtracts structured data from pages for customers, including AI companies.

The Hive / 1

ImagesiftBotTrainingCrawls images across the web for Hive's visual-AI products.

Webz.io / 1

omgilibotTrainingWebz.io's crawler; its web-data feeds are sold to AI builders.

Timpi / 1

TimpibotSearchCrawler for Timpi's decentralized search index.

Commonly confused pairs

Work with these bots, not just read about them:

robots.txt Generator — build per-bot allow/block rules for all 28 crawlers at once.
AI Bot Log Analyzer — see which of these bots actually visit your site.
The complete guide to AI crawlers — background reading on how these bots differ.

Use the data in your own project:

This directory is published as an open dataset under CC BY 4.0 — grab it as JSON or CSV. Attribution appreciated; a link back to this page is plenty.