AI crawler profile
What is Google-Extended? How to allow or block Google's crawler
Operator
Purpose
AI model training
robots.txt
Respects robots.txt
Official docs
Google-Extended is not a separate crawler — it is a robots.txt control token evaluated by Google's existing crawl infrastructure. Disallowing it opts your content out of Gemini training and grounding without affecting Google Search indexing or ranking, which remain governed by Googlebot.
Does Google-Extended respect robots.txt?
As a robots.txt token rather than a crawler, honoring your directive is its entire function; Google documents it officially.
Block Google-Extended with robots.txt
User-agent: Google-Extended Disallow: /
Explicitly allow Google-Extended
User-agent: Google-Extended Allow: /
Should you block Google-Extended?
Blocking Google-Extended opts your content out of Google's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
Check and monitor Google-Extended on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Google-Extended.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 15 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Google-Extended hits, entirely in your browser.
Frequently asked questions
What is Google-Extended?
Google-Extended is Google's web crawler for collecting AI training data. Controls use of content for Gemini and Vertex AI training, separate from Googlebot.
Does Google-Extended respect robots.txt?
As a robots.txt token rather than a crawler, honoring your directive is its entire function; Google documents it officially.
How do I block Google-Extended?
Add "User-agent: Google-Extended" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is separate from Googlebot, which handles Google Search indexing. Blocking Google-Extended has no effect on your traditional search rankings.
How can I tell if Google-Extended is crawling my site?
Search your server access logs for the string "Google-Extended". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Google-Extended, with per-path breakdowns.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-15.