AI crawler comparison
Meta-ExternalAgent vs Meta-ExternalFetcher: what's the difference?
Meta runs one crawler for training data and another for user-triggered link fetches.
Short answer
They are not interchangeable. Meta-ExternalAgent handles AI model training, Meta-ExternalFetcher handles AI search and live answers. Blocking one has no effect on the other, so a robots.txt rule aimed at the wrong agent produces exactly the outcome you were trying to avoid.
Side by side
| Meta-ExternalAgent | Meta-ExternalFetcher | |
|---|---|---|
| Operator | Meta | Meta |
| Purpose | AI model training | AI search & live answers |
| robots.txt | Respects robots.txt | Ignores robots.txt |
| Published IP ranges | None published | None published |
| Official docs | Yes | Yes |
| Blocking costs you | No traffic — training only | AI-answer citations |
What Meta-ExternalAgent does
Meta-ExternalAgent is Meta's crawler for collecting training data for its Llama models and Meta AI products. It is distinct from facebookexternalhit (which fetches link previews), so you can block AI training without breaking how your links render on Facebook and Instagram.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Meta-ExternalAgent.
What Meta-ExternalFetcher does
Meta-ExternalFetcher retrieves individual URLs that Meta AI products need for a user's request — distinct from Meta-ExternalAgent, which crawls for model training. Because its fetches are user-initiated, Meta says this agent may bypass robots.txt rules, making it one of the openly documented cases of a user-fetch agent that doesn't honor blocks.
Full profile, with every robots.txt, nginx, Apache and Cloudflare rule: Meta-ExternalFetcher.
Block both
User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: /
Block Meta-ExternalAgent, allow Meta-ExternalFetcher
The split most sites want when the two agents do different jobs: opt out of one without giving up the other.
User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Allow: /
Check your own site
- AI Crawler Access Checker — see which of the two your live robots.txt currently allows.
- robots.txt Generator for AI Bots — build per-bot rules for both agents and the other 26.
- AI Bot Log Analyzer — paste your access log and see which of them is actually visiting.
- Should you block AI bots? — the decision framework behind the split above.
Frequently asked questions
What is the difference between Meta-ExternalAgent and Meta-ExternalFetcher?
Meta-ExternalAgent is Meta's AI model training agent — Crawls content to train Meta's AI models. Meta-ExternalFetcher is Meta's AI search and live answers agent — Fetches specific links for Meta AI when users share or request them. They are separate user agents with separate robots.txt rules.
Does blocking Meta-ExternalAgent also block Meta-ExternalFetcher?
No. robots.txt matches on the user-agent token, so a group naming Meta-ExternalAgent applies only to Meta-ExternalAgent. Meta-ExternalFetcher reads the group that names it, or the wildcard group if none does. To stop both you need a rule for each — or a wildcard rule, which would also affect every other crawler that reads it.
Should I block Meta-ExternalAgent, Meta-ExternalFetcher, or both?
It depends which outcome you want. Blocking Meta-ExternalAgent opts you out of Meta's model training and costs you no traffic today, because training crawlers send no visitors. Blocking Meta-ExternalFetcher removes you from the answers Meta's assistant shows its users. The common choice is to block training agents and allow search agents, so your content stays citable without feeding model training.
Do Meta-ExternalAgent and Meta-ExternalFetcher both respect robots.txt?
Meta-ExternalAgent: Meta documents the agent and states it respects robots.txt rules. Meta-ExternalFetcher: Meta's own documentation states Meta-ExternalFetcher may bypass robots.txt because fetches are performed on behalf of a user.
Can I tell Meta-ExternalAgent and Meta-ExternalFetcher apart in my server logs?
Yes — they send different user-agent strings, so grep for each token separately: grep -i "Meta-ExternalAgent" and grep -i "Meta-ExternalFetcher". Be aware that a user-agent header is self-declared and anything can send either string. Neither operator publishes IP ranges for these agents, so log entries claiming them cannot be verified.
Other comparisons
Part of our directory of every known AI crawler. Last verified: 2026-08-14.