AI crawler profile
What is Meta-ExternalFetcher? How to allow or block Meta's crawler
Operator
Meta
Purpose
AI search & live answers
robots.txt
Ignores robots.txt
Official docs
Meta-ExternalFetcher retrieves individual URLs that Meta AI products need for a user's request — distinct from Meta-ExternalAgent, which crawls for model training. Because its fetches are user-initiated, Meta says this agent may bypass robots.txt rules, making it one of the openly documented cases of a user-fetch agent that doesn't honor blocks.
Does Meta-ExternalFetcher respect robots.txt?
Meta's own documentation states Meta-ExternalFetcher may bypass robots.txt because fetches are performed on behalf of a user.
Block Meta-ExternalFetcher with robots.txt
User-agent: Meta-ExternalFetcher Disallow: /
⚠ Because Meta-ExternalFetcher's robots.txt compliance is unreliable, pair this with a firewall or CDN rule matching the user-agent string if blocking actually matters to you.
Explicitly allow Meta-ExternalFetcher
User-agent: Meta-ExternalFetcher Allow: /
Should you block Meta-ExternalFetcher?
For most sites, no. Meta-ExternalFetcher powers visibility: blocking it removes your pages from the cited answers Meta's assistant shows its users — the AI-era equivalent of de-indexing yourself from a search engine. Block it only if you deliberately don't want that audience. Our guide on blocking AI bots covers the trade-offs in detail.
Check and monitor Meta-ExternalFetcher on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Meta-ExternalFetcher.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 28 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Meta-ExternalFetcher hits, entirely in your browser.
Frequently asked questions
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is Meta's agent for AI search and live answers. Fetches specific links for Meta AI when users share or request them.
Does Meta-ExternalFetcher respect robots.txt?
Meta's own documentation states Meta-ExternalFetcher may bypass robots.txt because fetches are performed on behalf of a user.
How do I block Meta-ExternalFetcher?
Add "User-agent: Meta-ExternalFetcher" followed by "Disallow: /" to your robots.txt file. Because this bot's robots.txt compliance is not reliable, enforce the block with firewall or CDN rules (for example a Cloudflare WAF rule matching the user agent) if it matters to you.
Does blocking Meta-ExternalFetcher hurt my Google rankings?
No. Meta-ExternalFetcher is separate from Googlebot, which handles Google Search indexing. Blocking Meta-ExternalFetcher has no effect on your traditional search rankings, but it does remove your pages from the AI answers Meta's assistant serves to its users.
How can I tell if Meta-ExternalFetcher is crawling my site?
Search your server access logs for the string "Meta-ExternalFetcher". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Meta-ExternalFetcher, with per-path breakdowns.
More Meta agents
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-18.