There's no universally correct answer — blocking AI bots trades away potential visibility in AI-generated answers in exchange for keeping your content out of model training data. The right call depends on your content type, business model, and how much you value each side of that trade.
The two things you're actually deciding
Every AI bot decision really breaks into two separate questions:
- Do I want my content used to train future AI models? Once ingested into a training run, content's influence on the resulting model is effectively permanent and can't be un-trained after the fact.
- Do I want my content surfaced and cited when someone asks an AI assistant a question? This is closer to traditional SEO — being discoverable now, in response to a specific query.
These map to different bots (see our full crawler list): GPTBot, ClaudeBot, Google-Extended, Bytespider, Meta-ExternalAgent, and CCBot are training-focused; OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, and Perplexity-User are search/browsing-focused. You can block one category and allow the other.
Reasons to block training bots
- Your content is your product. If you sell access to research, analysis, or writing, having it ingested into a competitor's free AI model undermines the business model directly.
- Competitive or proprietary information. Pricing pages, internal documentation not meant for wide distribution, or content you don't want summarized and reused elsewhere.
- No reciprocal value. Training doesn't send you traffic or citations — the value flows one direction, into the model, with no attribution back to you.
Reasons to allow training bots
- Brand and topical association. Being part of the training data may influence how a model describes your product, brand, or category when asked general questions — an indirect but real form of visibility.
- Goodwill and ecosystem participation, especially for open-source projects, documentation sites, or content whose whole purpose is to be widely known and reused.
- Practically limited enforcement value. If your content is already broadly available and indexed elsewhere (aggregators, forks, mirrors), blocking one crawler has diminishing marginal effect.
Reasons to allow search/browsing bots
- This is the AI-era equivalent of ranking in search. If PerplexityBot or OAI-SearchBot can't fetch your page, you simply cannot be cited when a user asks a question your content answers — full stop.
- Attribution and referral traffic. Unlike training, live browsing/search citations typically link back to your source, similar to a traditional search result.
- Low cost. Allowing a bot to fetch a page at query time costs you nothing beyond normal server load — there's no training-data permanence concern.
Reasons to block search/browsing bots
- Paywalled or gated content you don't want summarized for free even at query time, defeating the paywall's purpose.
- Content that's misleading out of context — if your page depends heavily on surrounding context, design, or interactivity that a text-only citation would strip away.
A practical default
For most content marketing, documentation, and product sites, a reasonable starting position is: allow search/browsing bots, block or selectively allow training bots based on how much you value long-term content control versus brand/ecosystem visibility. This gets you into AI-generated answers now, without permanently ceding your content to model training you can't retract.
For paywalled, proprietary, or directly-monetized content, blocking both categories is defensible — but understand that doing so also means giving up any chance of appearing in AI Overviews, ChatGPT, Claude, or Perplexity answers entirely.
Making the change
Use our robots.txt Generator for AI Bots — the "Block training bots only" preset implements the practical default above in one click, or fine-tune each bot individually. Once deployed, confirm the result with the AI Crawler Access Checker.
The enforcement caveat
robots.txt expresses intent; it doesn't technically prevent a non-compliant crawler from fetching your content anyway. Reputable AI companies generally respect it, but if your content is highly sensitive, robots.txt alone is not a security boundary — pair it with authentication or a paywall for anything that truly must not be accessed by an automated client.