GeoPromptTracker

AI crawler profile

What is Google-Extended? How to allow or block Google's crawler

User-agent string to match in robots.txt and server logs: Google-Extended

Operator

Google

Purpose

AI model training

robots.txt

Respects robots.txt

Official docs

Google documentation ↗

Google-Extended is not a separate crawler — it is a robots.txt control token evaluated by Google's existing crawl infrastructure. Disallowing it opts your content out of Gemini training and grounding without affecting Google Search indexing or ranking, which remain governed by Googlebot.

Does Google-Extended respect robots.txt?

As a robots.txt token rather than a crawler, honoring your directive is its entire function; Google documents it officially.

Verify it's really Google-Extended

The user-agent string above is self-declared, so anything can send it. Google publishes Google-Extended's IP ranges as JSON, which is what makes a rule verifiable: check the request's IP against the published prefixes instead of trusting the name. Requests claiming to be Google-Extended from outside those ranges are spoofed — useful to know whether you're allowing or blocking it.

Block Google-Extended with robots.txt

robots.txt — block
User-agent: Google-Extended
Disallow: /

Explicitly allow Google-Extended

robots.txt — allow
User-agent: Google-Extended
Allow: /

Block Google-Extended at the server or CDN

robots.txt is the right first step for Google-Extended, since Google honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — pair it with an IP check against the published ranges above for a rule that can't be spoofed.

nginx
if ($http_user_agent ~* "Google-Extended") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Google-Extended [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "Google-Extended")

Find Google-Extended in your server logs

shell
grep -i "Google-Extended" /var/log/nginx/access.log | wc -l

Should you block Google-Extended?

Blocking Google-Extended opts your content out of Google's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.

Check and monitor Google-Extended on your site

Related reading

Frequently asked questions

What is Google-Extended?

Google-Extended is Google's web crawler for collecting AI training data. Controls use of content for Gemini and Vertex AI training, separate from Googlebot.

Does Google-Extended respect robots.txt?

As a robots.txt token rather than a crawler, honoring your directive is its entire function; Google documents it officially.

How do I block Google-Extended?

Add "User-agent: Google-Extended" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended is separate from Googlebot, which handles Google Search indexing. Blocking Google-Extended has no effect on your traditional search rankings.

How can I tell if Google-Extended is crawling my site?

Search your server access logs for the string "Google-Extended" — for example: grep -i "Google-Extended" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Google-Extended, with per-path breakdowns.

How do I use Google-Extended?

You don't — Google-Extended isn't a tool you run. It's Google's own crawler, operated by Google, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "Google-Extended" as your user agent would be impersonating Google.

Where is the official Google documentation for Google-Extended?

Google publishes it at https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers. That page is the authoritative source for the user-agent string and Google's stated crawling policy, and Google also publishes Google-Extended's IP ranges as JSON at https://developers.google.com/static/crawling/ipranges/common-crawlers.json so you can verify requests rather than trusting the header.

More Google agents

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.