AI crawler profile
What is Meta-ExternalFetcher? How to allow or block Meta's crawler
User-agent string to match in robots.txt and server logs: Meta-ExternalFetcher
Operator
Meta
Purpose
AI search & live answers
robots.txt
Ignores robots.txt
Official docs
Meta-ExternalFetcher retrieves individual URLs that Meta AI products need for a user's request — distinct from Meta-ExternalAgent, which crawls for model training. Because its fetches are user-initiated, Meta says this agent may bypass robots.txt rules, making it one of the openly documented cases of a user-fetch agent that doesn't honor blocks.
Does Meta-ExternalFetcher respect robots.txt?
Meta's own documentation states Meta-ExternalFetcher may bypass robots.txt because fetches are performed on behalf of a user.
Verify it's really Meta-ExternalFetcher
Meta publishes no IP-range file for Meta-ExternalFetcher, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.
Block Meta-ExternalFetcher with robots.txt
User-agent: Meta-ExternalFetcher Disallow: /
⚠ Because Meta-ExternalFetcher's robots.txt compliance is unreliable, pair this with a firewall or CDN rule matching the user-agent string if blocking actually matters to you.
Explicitly allow Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher Allow: /
Block Meta-ExternalFetcher at the server or CDN
Meta-ExternalFetcher's robots.txt compliance is absent, so a robots.txt rule alone may not stop it. These enforce the block. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.
if ($http_user_agent ~* "Meta-ExternalFetcher") {
return 403;
}RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Meta-ExternalFetcher [NC]
RewriteRule .* - [F,L](http.user_agent contains "Meta-ExternalFetcher")
Find Meta-ExternalFetcher in your server logs
grep -i "Meta-ExternalFetcher" /var/log/nginx/access.log | wc -l
Should you block Meta-ExternalFetcher?
For most sites, no. Meta-ExternalFetcher powers visibility: blocking it removes your pages from the cited answers Meta's assistant shows its users — the AI-era equivalent of de-indexing yourself from a search engine. Block it only if you deliberately don't want that audience. Our guide on blocking AI bots covers the trade-offs in detail.
Check and monitor Meta-ExternalFetcher on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Meta-ExternalFetcher.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 28 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Meta-ExternalFetcher hits, entirely in your browser.
Related reading
- List of AI crawlers and their user agents — how Meta-ExternalFetcher fits among every other AI crawler, in one table.
- Can AI crawlers execute JavaScript? — whether Meta-ExternalFetcher sees content your site renders client-side.
- Should you block AI bots? — the full decision framework for Meta and the rest.
Frequently asked questions
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is Meta's agent for AI search and live answers. Fetches specific links for Meta AI when users share or request them.
Does Meta-ExternalFetcher respect robots.txt?
Meta's own documentation states Meta-ExternalFetcher may bypass robots.txt because fetches are performed on behalf of a user.
How do I block Meta-ExternalFetcher?
Add "User-agent: Meta-ExternalFetcher" followed by "Disallow: /" to your robots.txt file. Because this bot's robots.txt compliance is not reliable, enforce the block with firewall or CDN rules (for example a Cloudflare WAF rule matching the user agent) if it matters to you.
Does blocking Meta-ExternalFetcher hurt my Google rankings?
No. Meta-ExternalFetcher is separate from Googlebot, which handles Google Search indexing. Blocking Meta-ExternalFetcher has no effect on your traditional search rankings, but it does remove your pages from the AI answers Meta's assistant serves to its users.
How can I tell if Meta-ExternalFetcher is crawling my site?
Search your server access logs for the string "Meta-ExternalFetcher" — for example: grep -i "Meta-ExternalFetcher" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Meta-ExternalFetcher, with per-path breakdowns.
How do I use Meta-ExternalFetcher?
You don't — Meta-ExternalFetcher isn't a tool you run. It's Meta's own crawler, operated by Meta, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "Meta-ExternalFetcher" as your user agent would be impersonating Meta.
Where is the official Meta documentation for Meta-ExternalFetcher?
Meta publishes it at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/. That page is the authoritative source for the user-agent string and Meta's stated crawling policy.
More Meta agents
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.